Considerations on Test Framework Design for QUIC-Related Study Items

1. Introduction

This contribution addresses the coordination of test framework development for two QUIC-related study items: FS_Q4RTC_MED (QUIC for Real-Time Communications) and FS_QStream_MED (QUIC for Streaming). The document proposes a coordinated approach to avoid duplication of effort.

2. Evaluation Plan for FS_Q4RTC_MED

The evaluation framework for QUIC-based RTC is defined in objectives 2a and 2b of the SID:

Objective 2a - Evaluation Framework Definition

• Define evaluation framework for QUIC-based media delivery protocols in the context of RTC System (TS 26.506 and TS 26.113)
• Define application scenarios for evaluation, including existing 3GPP services and service enablers such as split rendering

Objective 2b - Performance Evaluation

• Define appropriate performance metrics and requirements for identified application scenarios
• Evaluate QUIC-based protocols against existing architectures (WebRTC and (S)RTP-based frameworks) under realistic 3GPP network conditions
• Consider existing performance evaluations from academia and other SDOs, with independent SA4 verification (analytical verification without necessarily conducting new simulations)

Objective 2c - Deployment Impact Assessment

Document potential deployment impacts on TS 26.506 delivery architecture, considering:
- Current architectures
- 3GPP core network architecture (TS 23.501)
- UE implementations
- Advantages/disadvantages including: efficiency, scalability, distributed deployment capability, impact on radio optimizations, flow control and management, security/privacy vs. traffic management, and implementation readiness

Notes:
- Evaluation framework may be based on open-source network simulator (e.g., ns3)
- Evaluation scenarios involve real-time audio and video communication using EVS/IVAS codecs for audio and H.264/AVC, H.265/HEVC for video (per TS 26.114)

3. Evaluation Plan for FS_QStream_MED

The evaluation framework for QUIC-based streaming is defined in objectives 1-4, with 3b and 3c being most relevant:

Objective 1 - Application Scenarios

Identify application scenarios and delivery characteristics for segmented media delivery services (uplink and downlink), including:
- Low latency video streaming
- Live streaming
- On-demand and short-form video platforms

Objective 2 - Technology Identification

Identify existing and emerging segmented media streaming technologies, particularly QUIC-based technologies from TR 26.804:
- DASH over HTTP/3
- MoQ
- MPEG-DASH over WebTransport
- MPEG-DASH Part 6 over QUIC

Objective 3 - Evaluation Framework Definition

Objective 3a - Metrics Definition

Determine existing metrics reflecting QoE (from TS 26.247, TR 26.944, ITU-T P.1203, CTA-2066, etc.):
- Playback time from live edge
- Start-up time
- Rebuffering events and duration
- Streaming quality
- Respective QoS metrics if needed

Objective 3b - Deployment Impact Assessment

Document potential impact of deploying QUIC-based streaming technologies on:
- Media delivery architecture (TS 26.501)
- Delivery protocols (TS 26.512)
- Codecs and formats (TS 26.511)

Consider: current CDN architectures, 3GPP core network architecture (TS 23.501), UE implementation, encrypted content. Identify advantages/disadvantages including caching efficiency, scalability, distributed deployment capability, and implementation readiness.

Objective 3c - Test Framework Design

Design test framework for collecting selected metrics to evaluate baseline (DASH over HTTP 1.1) against technologies identified in objective #2.

Objective 4 - Technology Evaluation

Evaluate selected technologies by collecting QoE metrics using the framework from objective #3 for use cases from objective #1, under 3GPP network conditions using mobile network traces. Develop network simulation setup and select network traces for relevant application scenarios.

4. Potential for Coordination

Joint coordination on certain topics could streamline test framework development and ensure effective evaluation:

QUIC Protocol Aspects

• Considerations on applicability of different QUIC features
• Choice of QUIC stack for the test framework

Common Protocols

• MOQT is expected to be the only protocol applicable to both streaming and RTC scenarios
• Joint discussions on MOQT protocol features and related IETF MOQ WG specifications would be beneficial

Evaluation Methodology

• Whether evaluation is conducted via network simulation or emulation
• In case of network simulation: model-based vs. real network traces

5. Proposals

The following agreements are proposed:

1. Joint Initial Sessions: In the initial phases of both studies, address test framework-related contributions together during joint sessions. This will help clarify coordination opportunities until specific methodologies for each study are established.

2. Contribution Labeling: Proponents of relevant contributions should indicate when their contributions target the test framework to facilitate session scheduling.

3. Common Base Framework: Strive to create a common base framework on which both frameworks can build and branch out, reducing redundant implementation work.

4. Separate Application Scenario Evaluation: Contributions targeting evaluation of specific RTC or streaming-related application scenarios can be handled separately at the respective SWGs.