Summary of 3GPP Technical Document S4-260143

Document Information

• Type: Change Request (CR 0002 rev 3)
• Specification: TS 26.966 v19.0.0
• Work Item: FS_AVFOPS_MED (Feasibility Study on Audio Video File Operations for Media)
• Category: B (addition of feature)
• Release: Rel-20
• Source: Xiaomi Communications

Purpose and Scope

This CR proposes adding a new scenario (Scenario #6) on Refocusable Video to TR 26.966, addressing objective 1 of identifying relevant new representation formats not yet documented in TS 26.265.

Main Technical Contributions

5.7.1 Overview - Use Case Description

The CR introduces the concept of refocusable video, which enables post-capture modification of depth of field effects (bokeh). Key points:

• Traditional portrait photography achieves bokeh through lens selection
• Digital photography can simulate this effect pre-encoding
• Growing user expectation (especially prosumers) to refocus already captured content
• Technical approach: record sharp video + depth map sequence, then generate bokeh effect per frame during playback/editing

5.7.2 Previous Work in 3GPP

Identifies gap: coded representation of depth maps as part of video bitstream has not been addressed in 3GPP specifications.

5.7.3 Review of Related Work

Comprehensive survey of depth map representation across multiple standards bodies:

5.7.3.1 ISO/IEC 23091-2:2025 CICP Video

• MV-HEVC enables depth map encoding via auxiliary layers
• Auxiliary layer identified as depth auxiliary layer containing depth representation information SEI message
• New color primaries code point (value 130) defined to indicate decoded picture represents depth map
• HasChromaticityCoordinates = 0 (one color component)
• Can be carried in VUI parameters for AVC/HEVC without relying on auxiliary layer design

5.7.3.2 ISO/IEC 23000-22 MIAF and ISO/IEC 23008-12 HEIF

• MIAF builds on HEIF for multiple images, groups, sequences with defined relationships
• Depth maps defined as auxiliary image items
• Identified using auxiliary image item type: urn:mpeg:mpegB:cicp:systems:auxiliary:depth
• Depth map interpretation out of scope; recommends including depth representation information SEI message as item property for HEVC-encoded auxiliary items

5.7.3.3 SMPTE ST 2087:2016 Depth Map Representation

Defines comprehensive depth map data representation with key definitions:

Terminology:
- Reference Camera: Camera corresponding to viewpoint (can be virtual)
- Depth Map: Array of depth values corresponding to image pixels
- Depth Value: Distance in meters from reference camera to object surface, measured parallel to optical axis
- Relative Depth Value: Offset and scaled representation of depth value

Two representations specified:

1. 32-bit floating point:
2. IEEE 754 single-precision format
3. Unit: meter (1.0 = 1 meter)
4. Max value: positive infinity (+INF, 0x7F800000)
5. Unknown value: NaN

6. Co-located sample mapping

7. 16-bit floating point:

8. IEEE 754 half-precision format
9. Relative depth values (unitless)
10. Max value: positive infinity (+INF, 0x7C00)
11. Unknown value: NaN
12. Requires metadata: DepthScaleFactor and DepthOffset

5.7.3.4 ISO/IEC 23008-2 HEVC / ITU-T H.265

• MV-HEVC enables depth map encoding as auxiliary layer
• Auxiliary layer identified as depth auxiliary layer with depth representation information SEI message
• References Clause 6.9 Solution #4.1

5.7.3.5 ISO/IEC 14496-12 ISOBMFF

• MPEG Systems WG03 developing Amendment 2 of 8th edition
• Specifies storage of depth map video sequences in auxiliary video track linked to main video track
• Covers: media handler, track referencing, metadata box for depth map interpretation

5.7.3.6 SMPTE ST 268-1:2014 DPX Format

Digital Picture Exchange Format v2.0 for moving pictures:

Depth component support:
- Code value 8: Depth (Z) component

Transfer characteristics:
- Code 11: Z (depth) – linear
- Code 12: Z (depth) – homogeneous (requires distance to screen and angle of view in user-defined section)

5.7.4 Functional Requirements

Outlines analysis framework based on:

1. Hardware impact assessment:
2. Option a: Reference existing hardware product-grade support

3. Option b: Describe expected hardware implementation impact with justifications

4. Codec capabilities: TBD

References Added

The CR adds 9 new normative/informative references covering:
- Android AOSP camera bokeh documentation
- JVET documents on CICP extensions
- ISO/IEC standards (MIAF, ISOBMFF amendments)
- SMPTE standards (RP 157, ST 268-1, ST 2087)
- Google Dynamic Depth specification
- Android MP4-AT file format

Impact Assessment

• Specifications affected: Only TR 26.966
• Other specs: None
• Test specifications: None
• O&M specifications: None