Analysis on Complexity Evaluation of ULBC with WMOPS

1. Introduction

This contribution examines the use of WMOPS (Weighted Million Operations Per Second) as a complexity metric for ULBC (Ultra Low Bitrate Codec). WMOPS has been proposed as one of the possible complexity metrics and is traditionally used for evaluating 3GPP speech codecs complexity. The analysis focuses on the WMS tool used for automated WMOPS calculation with floating point C code.

2. Technical Analysis: Discrepancies Between ITU-T Documentation and WMC Tool Implementation

The source conducted systematic testing of the WMC tool against the examples provided in ITU-T standards documentation (specifically clause 18.12.7 and related tables in the ITU-T Software Tool Library 2024 User's Manual). Several discrepancies were identified:

2.1 'Move' Operator Counting

Issue: Extra MOVE operations are counted by the WMC tool

• Expected behavior (per Table 18.4): Division by constant b = a / L should count as 1 MULT (since 1/L is constant, operation becomes multiplication)
• Actual WMC output: 1 MULT + 1 MOVE
• Discrepancy: 1 additional MOVE operation

2.2 Increment Operator ('++')

Issue: Missing operations in WMC tool output

• Expected behavior (per Table 18.4): (*rnd_T0)++ should count as 1 ADD + 1 STORE (equivalent to *rnd_T0 = *rnd_T0 + 1)
• Actual WMC output: 0 operations counted
• Discrepancy: 1 ADD + 1 STORE not counted
• Note: This may not affect actual complexity on DSP implementations where pointer increment can be combined with other operations

2.3 Logical Operators ('AND/OR')

Issue: Missing TEST operation counting

• Expected behavior (per Table 18.4): if (a!=b || c==d) should count as 2 ADD + 2 BRANCH + 1 TEST
• Actual WMC output: 2 ADD + 2 BRANCH
• Discrepancy: 1 TEST operation missing

2.4 Indirect Addressing

Issue: Extra MOVE operation and incorrect INDIRECT counting

• Expected behavior (per Table 18.4): Double indirection Indice[0] = indirect_dico1[indice[0]] should count as 2 INDIRECT
• Actual WMC output: 1 MOVE
• Discrepancy: 2 INDIRECT operations not counted, 1 MOVE added instead

2.5 Instrumentation with Array Subscripts

Issue: Arithmetic operations inside array subscripts not counted

• Expected behavior (per Table 18.3): Operations like a[i*2+1] should count arithmetic operations within the subscript (1 INDIRECT + 1 MULT + 1 ADD)
• Actual WMC output: Only 1 MOVE counted
• Discrepancy: INDIRECT, MULT, and ADD operations inside array subscripts are not instrumented

3. Observations and Impact Assessment

The source identifies three key observations:

1. Systematic discrepancies exist between ITU-T standards documentation and WMC tool implementation, with both over-counting (e.g., extra MOVE operations) and under-counting (e.g., missing operations in array subscripts) observed

2. Potential significance for AI codecs: Some discrepancies, particularly the counting of MOVE operators and instrumentation inside arrays, could significantly impact WMOPS measurements for AI-based codecs

3. Need for clarification: If WMOPS is adopted as a complexity metric for ULBC, these differences must be carefully addressed and the calculation methodology must be clearly defined

4. Proposal

The source proposes to document the findings from Clause 2 and Clause 3 in clause 6.2 of the permanent document to ensure proper consideration of these WMOPS calculation issues in the ULBC complexity evaluation framework.