# Considerations on Test Framework Design for QUIC-Related Study Items

## 1. Introduction

This contribution addresses the coordination of test framework development for two QUIC-related study items: FS_Q4RTC_MED (QUIC for Real-Time Communications) and FS_QStream_MED (QUIC for Streaming). The document proposes a coordinated approach to avoid duplication of effort.

## 2. Evaluation Plan for FS_Q4RTC_MED

The evaluation framework for QUIC-based RTC is defined in objectives 2a and 2b of the SID:

### Objective 2a - Evaluation Framework Definition
- Define evaluation framework for QUIC-based media delivery protocols in the context of RTC System (TS 26.506 and TS 26.113)
- Define application scenarios for evaluation, including existing 3GPP services and service enablers such as split rendering

### Objective 2b - Performance Evaluation
- Define appropriate performance metrics and requirements for identified application scenarios
- Evaluate QUIC-based protocols against existing architectures (WebRTC and (S)RTP-based frameworks) under realistic 3GPP network conditions
- Consider existing performance evaluations from academia and other SDOs, with independent SA4 verification (analytical verification without necessarily conducting new simulations)

### Objective 2c - Deployment Impact Assessment
Document potential deployment impacts on TS 26.506 delivery architecture, considering:
- Current architectures
- 3GPP core network architecture (TS 23.501)
- UE implementations
- Advantages/disadvantages including: efficiency, scalability, distributed deployment capability, impact on radio optimizations, flow control and management, security/privacy vs. traffic management, and implementation readiness

**Notes:**
- Evaluation framework may be based on open-source network simulator (e.g., ns3)
- Evaluation scenarios involve real-time audio and video communication using EVS/IVAS codecs for audio and H.264/AVC, H.265/HEVC for video (per TS 26.114)

## 3. Evaluation Plan for FS_QStream_MED

The evaluation framework for QUIC-based streaming is defined in objectives 1-4, with 3b and 3c being most relevant:

### Objective 1 - Application Scenarios
Identify application scenarios and delivery characteristics for segmented media delivery services (uplink and downlink), including:
- Low latency video streaming
- Live streaming
- On-demand and short-form video platforms

### Objective 2 - Technology Identification
Identify existing and emerging segmented media streaming technologies, particularly QUIC-based technologies from TR 26.804:
- DASH over HTTP/3
- MoQ
- MPEG-DASH over WebTransport
- MPEG-DASH Part 6 over QUIC

### Objective 3 - Evaluation Framework Definition

#### Objective 3a - Metrics Definition
Determine existing metrics reflecting QoE (from TS 26.247, TR 26.944, ITU-T P.1203, CTA-2066, etc.):
- Playback time from live edge
- Start-up time
- Rebuffering events and duration
- Streaming quality
- Respective QoS metrics if needed

#### Objective 3b - Deployment Impact Assessment
Document potential impact of deploying QUIC-based streaming technologies on:
- Media delivery architecture (TS 26.501)
- Delivery protocols (TS 26.512)
- Codecs and formats (TS 26.511)

Consider: current CDN architectures, 3GPP core network architecture (TS 23.501), UE implementation, encrypted content. Identify advantages/disadvantages including caching efficiency, scalability, distributed deployment capability, and implementation readiness.

#### Objective 3c - Test Framework Design
Design test framework for collecting selected metrics to evaluate baseline (DASH over HTTP 1.1) against technologies identified in objective #2.

### Objective 4 - Technology Evaluation
Evaluate selected technologies by collecting QoE metrics using the framework from objective #3 for use cases from objective #1, under 3GPP network conditions using mobile network traces. Develop network simulation setup and select network traces for relevant application scenarios.

## 4. Potential for Coordination

Joint coordination on certain topics could streamline test framework development and ensure effective evaluation:

### QUIC Protocol Aspects
- Considerations on applicability of different QUIC features
- Choice of QUIC stack for the test framework

### Common Protocols
- MOQT is expected to be the only protocol applicable to both streaming and RTC scenarios
- Joint discussions on MOQT protocol features and related IETF MOQ WG specifications would be beneficial

### Evaluation Methodology
- Whether evaluation is conducted via network simulation or emulation
- In case of network simulation: model-based vs. real network traces

## 5. Proposals

The following agreements are proposed:

1. **Joint Initial Sessions**: In the initial phases of both studies, address test framework-related contributions together during joint sessions. This will help clarify coordination opportunities until specific methodologies for each study are established.

2. **Contribution Labeling**: Proponents of relevant contributions should indicate when their contributions target the test framework to facilitate session scheduling.

3. **Common Base Framework**: Strive to create a common base framework on which both frameworks can build and branch out, reducing redundant implementation work.

4. **Separate Application Scenario Evaluation**: Contributions targeting evaluation of specific RTC or streaming-related application scenarios can be handled separately at the respective SWGs.