Summary of S4-260156: Non-BE Conformance of Metadata Output in IVAS

1. Introduction and Background

The document addresses conformance testing issues for IVAS codec as defined in TS 26.252. Current conformance requirements include:
- Bit-exact conformance for both fixed-point and floating-point implementations (audio signal and metadata output)
- Non-bit-exact conformance with tolerances for floating-point implementations
- Current limitation: Metadata output deviation is set to zero (no deviation allowed)

2. Observed Issues

Platform-Specific Non-Bit-Exact Behavior

Post-Release 19 experiments revealed that metadata output does not always remain bit-exact compared to conformance references across different platforms.

Specific example identified:
- Platform: Apple MacBook Pro with M4 ARM processor, Mac OS 15.7.3, Clang 16
- Issue: OMASA decoder conformance test vector produces minor numeric differences in output MASA metadata
- Specific differences:
- Direct-to-total and diffuse-to-total ratios affected
- 9 frames contain maximum absolute difference of 1/255 each
- Subjective impact is minimal

Broader Implications

• Similar differences expected on other platforms
• Encoder-side implementation differences may create larger differences in spatial direction metadata due to minor metadata value changes affecting encoding decisions

Analysis Tool Development

A MASA metadata difference tool has been developed as part of IVAS public collaboration to analyze metadata differences:
- Provides overall and frame-by-frame statistics
- Reports maximum and mean absolute difference values for spatial metadata parameters
- Includes difference reporting for descriptive metadata parameters
- Details provided in Annex A

3. Conclusions and Recommendations

The source proposes that non-bit-exact tolerance for metadata in Annex A.3.1.2 of TS 26.252 should be adjusted.

Next steps requested:
- Additional experiments needed to establish suitable criteria for non-bit-exact metadata conformance
- Invitation to IVAS codec implementors to share non-bit-exact results
- Goal: Set reasonable limits for both MASA metadata and ISM metadata conformance
- Ensure high-quality implementations across different platforms

Annex A: MASA Metadata Difference Tool (masaDiffTool)

A.1 Purpose and Scope

Informative tool designed to support evaluation of non-bit-exact behavior in MASA-format metadata produced by IVAS implementations. Primary use case: comparison between reference and alternative implementations.

A.2 Tool Functionality

Validation and comparison features:
- Validates both inputs use IVASMASA format
- Performs structured comparison of descriptive and spatial metadata across all frames
- Reports exact matches or identifies differing frames
- Computes aggregated difference measures (maximum and mean absolute deviations) for key spatial parameters
- Returns 0 only if metadata files are identical

Error handling:
- Stops comparison if metadata format doesn't match IVASMASA
- Classifies frames with mismatched numbers of spatial directions as entirely different and continues with next frame

A.3 Tool Usage

Command syntax:

masaDiffTool [options] refMetaFile cutMetaFile

Optional output formats:

1. Text report (--report <file>): Frame-by-frame summary including spatial and descriptive metadata differences
2. CSV report (--csv <file>): Structured per-frame table with key difference indicators and summary metrics

Standard output includes:
- Overall metadata difference status
- Count of frames with differences
- Summary status for descriptive and spatial metadata
- Aggregated maximum and mean absolute differences across all frames for:
- Direction
- Energy ratios
- Coherence measures

A.4 Availability

• Binary version available in IVAS codec public repository (3GPP Forge)
• Will be contributed to official TS 26.252 software package during next update cycle