Summary of S4-260230: Experimental Approach and Test Setup for Media Transmission for AI Inferencing
Document Overview
This Change Request (CR) to TR 26.823 v0.2.0 addresses the currently empty Clause 6.5.1 by providing detailed experimental approaches and test setups for evaluating media transmission for AI inferencing scenarios. The contribution is part of the Study on dynamically changing traffic characteristics and usage of enhanced QoS support in 5GS for media applications and services.
Main Technical Contributions
Two Experimental Approaches Proposed
The document proposes two distinct experimental approaches corresponding to the AI inferencing in XR service scenario (Clause 5.6):
- Experimental approach #1: Based on commercially available generative AI applications
- Experimental approach #2: Based on a standalone generative AI platform for traffic measurement associated with QoE metrics
Experimental Approach #1: Using Commercially Available Generative AI Applications
Experimental Methodology
- Black box approach: Uses commercially available generative AI applications (e.g., OpenAI ChatGPT™, META AI™, Google Gemini™) to measure and characterize media transmission traffic in uplink and downlink
- Focus: Interactive or delay-bound AI inference responses as described in typical implementation and end-to-end procedures
- Example use cases:
- Meta AI glasses "ask Meta AI about what you see" functionality (voice + periodic images uplink, audio response downlink)
- OpenAI ChatGPT™ voice mode
- Google Gemini™ live mode with camera feed
Measurement Procedure
- Baseline measurement: First conducted on wired network (ideal network conditions)
- 5G network testing: Uses test channels or emulated 5G network for deterministic, controllable, and reproducible conditions
- Network conditions considered: Nominal, cell-edge, multi-UE scenarios with system load
Test Setup
- Observation Points (OPs):
- UE side: 5G network emulator ingress of AI inference input data and egress of AI inference output data
- UPF side: 5G network emulator egress of AI inference input data and ingress of AI inference output data
- Measurement tool: Open-source network protocol analyzer (e.g., Wireshark) for IP packet analysis
- Purpose: Measurements at both ingress and egress highlight network performance impact on traffic characteristics
Limitation
NOTE: Due to black box nature, QoE metrics identified in Clause 5.6.3 cannot be measured due to lack of appropriate Observation Points.
Experimental Approach #2: Using Standalone Generative AI Platform
Experimental Methodology
- White box approach: Standalone client/server generative AI platform mimicking commercially available applications
- Client side: Generation and transmission of AI inference input data
- Server side: AI inference, generation and transmission of AI inference output data
- Key advantage: Control over open-source AI model and generation/reception of AI inference data enables QoE metrics measurement (Clause 5.6.3) in addition to traffic characteristics
QoE Measurement Capabilities
- Per-packet metadata insertion: Enables latency QoE metrics measurement (e.g., time-to-first-token, end-to-end latency)
- Packet marking: Uplink packets (questions) and corresponding downlink packets (inference responses) marked with metadata for correlation
Measurement Procedure
Similar to Approach #1:
- Baseline: Wired network (ideal conditions)
- 5G testing: Test channels or emulated 5G network
- Network conditions: Nominal, cell-edge, multi-UE scenarios
Test Setup
Client Side (Custom Application, e.g., Unity 3D)
- Generates and transmits input data (prompt/text, image) with metadata
- Receives AI inference output data with metadata
- Renders AI inference results
Server Side (Custom AI-enabled Application, e.g., Unity 3D)
- Receives AI inference input data with metadata
- Performs AI inferencing using open-source standalone Multi-Modal LLM (e.g., Llava model from LlamaSharp based on llama.cpp)
- Collects QoE metrics
- Transmits AI inference output data with metadata
Observation Points
- UE and UPF side OPs: For uplink/downlink traffic characteristics (as in Approach #1)
- Client and server-side OPs: For QoE metrics measurement
Variant Configuration
NOTE: Test setup may be extended to mimic advanced AI-enabled AR devices (e.g., lightweight AR+AI glasses requiring remote rendering) by adding XR data (e.g., periodic pose information) to AI inference input data in uplink to evaluate impact on traffic characteristics.
Key Technical Differentiators
| Aspect | Approach #1 (Commercial Apps) | Approach #2 (Standalone Platform) |
|--------|-------------------------------|-----------------------------------|
| Control | Black box | White box |
| QoE Metrics | Not measurable | Measurable |
| Traffic Characteristics | Measurable | Measurable |
| Flexibility | Limited | High (customizable) |
| Realism | High (actual commercial apps) | Medium (mimics commercial apps) |
| Metadata Support | No | Yes (per-packet) |