1.Context

Video streaming services are delivered through a complex ecosystem of multiple actors. Video service platforms (e.g., YouTube Live, Twitch, Amazon Prime Video) are responsible for delivering video content and managing end-user experience at the application level. Network operators (e.g., Orange, AT&T) ensure end-to-end connectivity and transport of video traffic across heterogeneous network infrastructures. CDN providers (e.g., Akamai, Cloudflare) optimize content distribution by caching and delivering video streams closer to end users, thereby reducing latency and improving scalability. Video technology vendors such as ATEME and ENENSYS provide critical enabling technologies, including video encoding, packaging, streaming infrastructure, and broadcast delivery systems that support large-scale media distribution.

Typically, video streaming services are consumed across a wide range of heterogeneous end-user devices through a diverse ecosystem of media players, including native applications, web-based players, and third-party streaming applications. Despite operating under identical network conditions and similar server workloads, these media players may deliver significantly different Quality of Experience (QoE) due to heterogeneous adaptive bitrate (ABR) algorithms, buffering strategies, and playback policies.

To mitigate QoE degradation, adaptation mechanisms performed by different actors require access to application-level parameters such as buffer occupancy, rendering state, playback speed, ABR decision logic, and user-perceived quality indicators. These fine-grained signals are essential for understanding actual playback behavior and enabling coordinated adaptation across heterogeneous media players. However, in practice, such application-level parameters remain largely inaccessible due to privacy constraints, encryption mechanisms, technical limitations, the lack of standardized interfaces across media players, and the absence of unified telemetry frameworks.

Use case:

We consider a live streaming platform for a major sports event (e.g., a football match) delivered simultaneously to thousands of users on mobile devices. Although users consume identical video content, they rely on a heterogeneous ecosystem of media players, where each player implements proprietary adaptive bitrate (ABR) algorithms and playback strategies, leading to heterogeneous QoE outcomes across users.

In this context, key actors typically lack access to application-level parameters such as buffer occupancy, internal ABR decisions, rendering pipeline status, or user-perceived quality metrics. This limited visibility into application-level states prevents optimal adaptation—such as bitrate shaping, traffic scheduling, or edge caching strategies—leading to inconsistent QoE across users and inefficient resource utilization. Indeed, adaptation actions rely mostly on packet-level indirect metrics such as throughput, packet loss ratio and patterns, delay, and jitter. This scenario highlights the challenge of optimizing video delivery under partial observability, where application-level dynamics remain hidden and system actors have asymmetric access to control and monitoring signals.

2. Problem and Objective

The receiver side in adaptive video streaming services over B5G and 6G networks plays a key role in ensuring stable and consistent QoE. However, the actors involved in delivering video streaming services have limited observability of what is actually happening on user devices. This is mainly due to privacy constraints, encryption mechanisms, heterogeneous media player implementations and the absence of standardized telemetry frameworks. This makes it difficult to accurately characterize and analyze users’ observed QoE scores, thereby leading to decision uncertainty and suboptimal adaptation across different management entities.

Objective

The main objective is to design intelligent and adaptive video streaming telemetry frameworks for heterogeneous users’ end systems operating under partial observability constraints. Specifically, the goal is to:

1. Develop AI-based models to reconstruct hidden application-level states, such as buffer occupancy, ABR decisions, and playback dynamics, from indirect and partial observations.

2. Enable cross-layer inference and coordination between application, transport, and network layers to improve global understanding of system behavior.

3. Design enhanced telemetry mechanisms that extend existing frameworks (e.g., CMSD/CMCD) to capture richer, interoperable receiver-side signals under real-world constraints such as privacy, encryption, and system heterogeneity.

Ultimately, the objective is to create enhanced video streaming telemetry frameworks—building upon standards such as CMSD/CMCD—and to integrate AI-driven inference techniques to reconstruct hidden application-level states and gain a more accurate understanding of end-user playback behavior. This, in turn, enables more reliable QoE estimation and precise root-cause analysis of performance degradation, whether originating from network conditions or application dynamics. By providing a unified and interoperable view of users’ end-system behavior across heterogeneous and partially observable video playback environments, the proposed approach paves the way for intelligent, data-driven adaptation strategies that improve user experience while ensuring efficient utilization of network and computing resources in emerging B5G and 6G multimedia systems.

3. Brief overview of the state of the art

QoE in adaptive video streaming has been widely studied, with models relating network and playback indicators (e.g., rebuffering, startup delay, bitrate switches) to perceived user quality [1][2]. However, these approaches generally rely on full observability of application-level states, an assumption that does not hold in real-world deployments. To improve observability of end-user playback systems, recent efforts have introduced standardized telemetry mechanisms such as Common Media Client Data (CMCD) and Common Media Server Data (CMSD), enabling lightweight and interoperable exchange of playback and delivery metrics between clients, CDNs, and service providers [3], [4]. These frameworks have significantly improved operational visibility in large-scale video streaming systems.

At the transport layer, QUIC provides enhanced network observability through richer congestion and latency feedback mechanisms [5][6]. Its user-space design and flexibility have also enabled extensions for media delivery, most notably Media over QUIC (MoQ), which targets low-latency video streaming by leveraging QUIC’s multiplexing and real-time capabilities. These developments highlight a shift toward transport protocols that increasingly expose richer signals for both network and media optimization.

Despite better characterization of video playback system, existing telemetry solutions remain partially decoupled across layers: CMCD and CMSD primarily expose client- and server-side telemetry, QUIC focuses on transport-level signals, and MoQ targets efficient media delivery over QUIC without fully exposing application-level playback internals. As a result, fine-grained application states—such as ABR decision logic, buffer dynamics, and rendering pipeline behavior—remain largely hidden, limiting end-to-end observability and cross-layer optimization in heterogeneous video streaming environments.

4. Research questions

This research work aims to address the challenge of telemetry in heterogeneous video playback end-systems under partial observability. In particular, the following research questions arise:

• RQ1: Telemetry design under partial observability

  How can we design enhanced telemetry frameworks that extend existing standards (e.g., CMCD/CMSD) to capture richer and more interoperable signals from heterogeneous video playback systems while respecting practical constraints such as privacy and encryption?

• RQ2: Reconstruction of hidden application states

  To what extent can hidden application-level states (e.g., buffer dynamics, ABR decisions, rendering behavior) be accurately reconstructed from indirect and partial observations collected at the network, CDN, or client-side telemetry level?

• RQ3: QoE inference and attribution

  How can we reliably estimate user QoE and distinguish between network-induced and application-induced performance degradation using incomplete and heterogeneous telemetry signals?

• RQ4: Cross-layer observability and coordination

  How can telemetry from different layers (application, transport such as QUIC, and emerging frameworks like MoQ) be unified to enable consistent and end-to-end observability of video streaming systems?

Collectively, these research questions highlight the need for new telemetry frameworks capable of operating under partial observability and bridging application, transport, and network layers. Addressing them is expected to enable more accurate QoE estimation, improved root-cause analysis, and more efficient cross-layer adaptation in heterogeneous video streaming systems for B5G and 6G environments.