[Proposal] Add compressed depth storage paths for LeRobot datasets

[yuecideng]

Proposal

Add a space-efficient depth storage backend for datasets recorded by LeRobotRecorder, while preserving the current Python 3.10 compatibility baseline.

This proposal tracks two implementation paths:

Path A: compressed depth storage on LeRobot 0.4.4 / Python 3.10

Keep lerobot>=0.4.4 and implement an EmbodiChain-owned depth writer rather than patching LeRobot internals.

The proposed pipeline is:

float32 depth in metres / uint16 depth in millimetres
    -> configurable linear or logarithmic 12-bit quantization
    -> gray12le single-channel frames
    -> HEVC/libx265 video
    -> persisted encoding and episode metadata

The first implementation should:

• integrate with LeRobotRecorder without modifying the installed LeRobot package;
• write depth streams as sidecar videos with deterministic sensor, episode, chunk, and timestamp mappings;
• persist depth_min, depth_max, shift, use_log, input/output units, codec, pixel format, FPS, and lossless/quality settings;
• default to HEVC lossless encoding of the quantized 12-bit codes, while allowing a configurable lossy CRF mode when storage size is more important;
• retain the existing numeric Parquet representation as an exact raw-data fallback;
• provide a decoder or dataset wrapper that dequantizes frames into metres or millimetres;
• support monocular and stereo keys consistently;
• include codec-availability checks, interrupted-recording cleanup, and round-trip error tests;
• keep segmentation masks as exact numeric features rather than using lossy video encoding.

This path is an EmbodiChain dataset extension, not a claim of native LeRobot 0.4.4 depth-video compatibility.

Path B: migrate to the official LeRobot 0.6.0 depth pipeline

When EmbodiChain is ready to raise its Python requirement from >=3.10 to >=3.12, upgrade the dependency to LeRobot 0.6.0 or newer and adopt the official depth path:

• represent depth as a single-channel visual feature with shape (H, W, 1);
• use dtype: image or dtype: video under an observation.images.<camera> key;
• set info.is_depth_map: true;
• pass a configurable DepthEncoderConfig through to LeRobotDataset.create;
• use the official TIFF path for raw depth images or the official gray12le/HEVC path for depth video;
• preserve official depth units and quantization metadata;
• provide compatibility or migration handling for datasets created by Path A and for numeric depth features created by feat(dataset): preserve depth and mask camera frames #422;
• deprecate the custom writer once the supported Python and LeRobot baselines allow the official implementation.

The Python baseline upgrade should be treated as a separate compatibility decision rather than being bundled into #422.

Motivation

PR #422 fixes the immediate data-loss problem by preserving depth and segmentation masks as numeric LeRobot features with their original dtype and shape. This is the appropriate compatibility solution for LeRobot 0.4.4 and Python 3.10, but dense depth arrays can substantially increase dataset size during long, high-resolution recordings, and use_videos currently applies only to RGB frames.

LeRobot added a native depth storage and decoding pipeline in version 0.6.0. However, LeRobot 0.6.0 requires Python 3.12, while EmbodiChain currently supports Python 3.10. Requiring the official implementation in #422 would therefore expand the PR into a project-wide runtime upgrade.

Tracking both paths here allows us to improve storage efficiency on the current runtime while retaining a clear migration path to the upstream implementation.

Related work:

• EmbodiChain depth/mask recording PR: feat(dataset): preserve depth and mask camera frames #422
• Official LeRobot depth integration PR: feat(depth maps): adding support for depth in LeRobot huggingface/lerobot#3644
• Official LeRobot video/depth encoding documentation: https://huggingface.co/docs/lerobot/main/video_encoding_parameters
• Official LeRobot 0.6.0 release: https://github.com/huggingface/lerobot/releases/tag/v0.6.0

Additional context

HEVC lossless mode preserves the quantized 12-bit codes, but 12-bit quantization is not bit-exact relative to the original float32 or uint16 depth input. Applications requiring exact source values must retain a raw numeric or lossless image representation.

A full backport of LeRobot 0.6.0 depth behavior into LeRobot 0.4.4 would touch dataset writing, reading, metadata, statistics, image writing, video encoding, and episode/chunk handling. An isolated EmbodiChain writer is expected to have a substantially lower implementation and maintenance cost.

Suggested implementation milestones:

1. Standalone quantizer/dequantizer and codec capability tests.
2. Episode-aware streaming depth writer and metadata schema.
3. Decoder/dataset wrapper and stereo coverage.
4. Migration design for the future LeRobot 0.6.0/Python 3.12 upgrade.

Checklist

• I have checked that there is no similar issue in the repo (required)