# 3GPP Change Request Summary: S4-260233

## Document Overview
This contribution proposes the use of ML model formats as intermediate representations (IR) for the ULBC (Ultra Low Bitrate Codec) reference implementation, rather than a pure C implementation. The document is structured as a proposed Change Request (pCR) to TR 26.940, introducing a new clause 6.4.2.

---

## Main Technical Contributions

### 1. Problem Statement and Motivation (Goal Section)

The document identifies a fundamental question for ULBC standardization: whether to provide the entire codec reference implementation in C (including neural network components) or to define specific parts based on ML model formats (e.g., ONNX, PyTorch, TensorFlow).

**Key concerns with pure C implementation:**
- Limits UE vendors from leveraging custom architectures and optimizations
- UE vendors typically have custom optimization pipelines to port ML models to internal formats
- Pure C approach restricts full utilization of specialized hardware (NPUs, DSPs, TPUs)

### 2. Limitations of C-Based Reference Implementation (Clause 6.4.2.1)

**Issues with existing WMC (Weighted Million Operations) tool for complexity measurement:**
- Weights in Table 18.3 of G.191 do not account for vectorized implementations of matrix multiplications
- Theoretical complexity estimation does not reflect actual runtime complexity
- Does not account for diversity of target platforms

**Additional limitations identified:**
- **Hardware/platform dependencies:** C implementations may rely on platform-specific intrinsics and vectorization pragmas, limiting portability to NPUs
- **Unoptimized reference code:** May not be optimized for certain platforms
- **Compiler dependencies:** Intrinsics are compiler-specific
- **Maintenance burden:** Keeping C implementation updated with new ML operators and architectures is costly and error-prone

### 3. Definitions and Concepts (Clause 6.4.2.1 - Definitions)

The document establishes clear terminology:

- **Graph format:** Describes neural network as computational graph (structure only, no parameters)
- **Model format:** Combines graph representation, trained parameters (weights, biases), and metadata; self-contained and directly runnable
- **Intermediate Representation (IR):** Serves as bridge between high-level ML framework and execution runtimes

**Note:** PyTorch does not contain a graph format and requires model definition as Torchcode.

### 4. Advantages of Model Format Approach (Clause 6.4.3.2)

**Platform Portability:**
- Specifies what is computed, not how it's executed
- Framework-agnostic: models can be exported from different training frameworks
- Allows vendors to use custom toolchains for hardware-specific optimization

**Hardware Evolution:**
- Future-proof method to leverage latest AI processor developments
- Maintains compatibility with low maintenance effort

**Combination with Standard C-code:**
- ULBC can combine ML parts (as model format) with classic signal processing (in ANSI C)
- Backend runtime in C can integrate ML components
- Enables traditional 3GPP codec reference implementation structure

### 5. Comprehensive Model Format Analysis (Clause 6.4.3.3)

The document provides detailed comparison of major ML model formats:

| Format | Type | Key Advantages | Key Limitations |
|--------|------|----------------|-----------------|
| **ONNX** | Framework-agnostic IR | Cross-framework portability, wide runtime/hardware support, native OS support (Windows/Linux), dedicated C/C++ runtime | Operator coverage limitations, limited dynamic graph support |
| **TensorFlow Lite (TFLite/LiteRT)** | Edge/embedded-focused IR | Mobile/edge optimized, strong Android ecosystem, quantization tools, C/C++ runtime | TensorFlow-centric, partially vendor-specific maintenance |
| **PyTorch/Python** | Torch.nn.Module + checkpoints | Easy prototyping, highly optimized conversion tools | Suboptimal for real-world testing, Python dependencies, no C/C++ runtime without Python |
| **TorchScript** | PyTorch-specific serialized IR | Static graph without Python dependencies, supports custom ops, LibTorch C++ runtime | PyTorch-specific, **deprecated** (being replaced by ExportedProgram) |
| **ExportedProgram & ExecuTorch** | Two IRs: ExportedProgram and .pte | Replaces TorchScript, canonical PyTorch export IR, dedicated C++ runtime | PyTorch-specific, requires compilation to another IR, pipeline not fully mature |
| **OpenVINO IR** | Intel/CPU-centric IR | Strong Intel CPU/GPU optimization | Not suitable for mobile SoCs, extra conversion step needed |
| **Proprietary vendor IRs** | Vendor-specific internal IR | Highly hardware-optimized | Not portable, requires conversion from open IR |

**Key observations:**
- PyTorch format provides maximum flexibility and transparency but may have long-term compatibility concerns due to format evolution
- ONNX and TFLite are designed for inference deployment and cross-platform compatibility, representing stable industry standards
- ULBC ML parts will likely be based on PyTorch format, convertible to stable formats like ONNX or TFLite

### 6. SoC AI Engine Support Analysis (Clause 6.4.3.4)

**Hardware landscape:**
- Major smartphone SoCs include NPUs, DSPs, TPUs, GPUs, and CPUs
- Vendors provide specialized runtime environments and SDKs
- Vendors use native/preferred internal model formats optimized for their architecture

**Industry pattern:**
- All major vendors provide conversion mechanisms from popular open-source formats
- Common supported formats: **ONNX, TFLite, PyTorch, TensorFlow**
- References provided for major vendors: Qualcomm, Apple, Samsung, MediaTek, Google, Huawei

### 7. Summary and Recommendations (Clause 6.4.3.4)

**Advantages of model-format/IR-based reference implementation:**
- Decouples algorithm definition from hardware-specific implementation
- Leverages existing SoC vendor compilers, AI accelerators, and runtimes
- Significantly more portable, maintainable, and future-proof

**Recommended approach for ULBC reference implementation:**
1. Base reference on ML model-format with auxiliary signal processing in C
2. Include **both ONNX and PyTorch** as ML model-formats
3. Define neural network model-format including operator set and version
4. Specify I/O interfaces of ML models and auxiliary signal processing steps in C
5. Use reference implementation for integration illustration, verification, and testing

---

## Proposal

The document proposes:
1. **Discussion and agreement** on selection of one or more model formats for ULBC reference implementation
2. **Agreement on principle** of using model format as part of ULBC standardization reference model
3. **Documentation** of findings in TR 26.940 under new clause 6.4.2

---

## Key Technical Impact

This contribution represents a significant departure from traditional 3GPP codec standardization approaches by advocating for ML model formats rather than pure C implementations. The proposal addresses practical deployment considerations for ML-based codecs while maintaining compatibility with 3GPP standardization practices through hybrid approach combining model formats with C code for signal processing components.