[format][python] Add video keyframe index format for efficient random reads

[XiaoHongbo-Hope]

Purpose

Persist optional sparse video seek indexes in .video, so a cold random reader can fetch the video metadata and needed GOP range instead of probing or scanning the video. This PR establishes the Java/Python format and rewrite path; index generation and reader integration follow separately.

Design

.video (in file order)
|-- Complete video payloads A, B, ...
|-- NEW: Video seek index blocks A, B, ...
|   |-- Video metadata ranges: (offset, length) pairs
|   `-- Compressed keyframe entries: (frame ordinal, PTS, packet byte position)
|-- Video payload lengths
|-- NEW: Video seek index block lengths
|-- Consecutive-row counts / video references / starting frame ordinals
`-- Footer
    |-- Five index byte lengths: 5 x 4 bytes
    |-- Magic: 4 bytes
    `-- Format version: 1 byte

Each video has one block-length entry; zero means no persisted seek index. All seek-index offsets are relative to the first byte of that encoded video.

Stored value	Meaning	Random cold-read use
Video metadata offset and length	Location and size of video metadata; for MP4, the moov box containing track and timing information	Fetch initialization metadata directly without probing the video
Frame ordinal	Zero-based frame number in presentation order	Find the preceding keyframe for a requested frame
PTS	Presentation timestamp	Seek to the exact keyframe time without deriving it from a constant frame rate
Packet byte position	Byte offset of the compressed packet containing the keyframe	Start the target GOP range read at the keyframe

The keyframe-entry list is compressed as one block. A cold random read fetches the video metadata, finds the preceding keyframe by ordinal, reads an adjacent GOP range from its packet byte position, then seeks to its PTS and decodes forward. Readers may extend the range through the following GOP for reordered frames. GOP index and in-GOP frame index are derived from the target ordinal and preceding keyframe, so they are not stored separately. This keeps seek metadata proportional to keyframes rather than rows.

Payload length is already available from the video descriptor. The index describes the first video stream; its time base remains in the encoded video.

• All write APIs validate and copy indexes supplied by source descriptors without decoding unindexed videos.
• Frames sharing one video payload must reference the same seek index; conflicting descriptors fail the write.
• Java and Python validate indexes in bounded chunks. Before fetching index bytes, writers cap one block at 16 MiB and one file at 64 MiB.
• Pack rewrites preserve indexes. Size-based rolling counts buffered index blocks and still rolls only between complete video groups.
• Version 1 remains readable.

Current open-source Lance comparison

The current open-source LeRobot-Lance implementation stores each source MP4 and its seek metadata as one row in videos.lance:

Paimon seek metadata	Lance column
Video metadata offset and length	moov_offset, moov_size
Frame ordinal	kf_indices
Packet byte position	kf_positions
PTS	Not stored separately; the Lance reader assumes constant-frame-rate MP4

Paimon stores this metadata in the same .video file and compresses the keyframe entries.

Reader follow-up

A separate PR will integrate index generation and consumption into PaimonLeRobotDataset/PyAV. It will group requests by payload, merge byte ranges, cache container data and decoders, and select the prefetch size at runtime. The stored metadata supports low-request object-store reads without requiring a fixed 64 KiB over-read in every environment.

Reference

TorchCodec frame mappings also motivated the persisted positioning metadata.

Validation

• Real MP4 sparse-read loop: two video streams, multiple GOPs, B-frames, VFR, and trailing moov; six random frames matched full PyAV decoding after the index was written to and read from .video.
• Python format/index tests: 20 passed and 16 subtests passed; the no-PyAV path skipped the real-video test cleanly. Bounded index validation stayed below 1 MB peak memory.
• Java: 4 descriptor, 12 video-format, and 2 rolling/compaction tests passed.
• Python 3.6 syntax, flake8, Spotless, Checkstyle, and git diff --check passed.

[JingsongLi]

[P1] Avoid materializing unbounded keyframe indexes

This materializes the entire compressed index before the chunked validator runs, and _keyframe_indexes retains every block until the writer closes. BlobRef.to_data() issues a single read for the descriptor length, while rolling is checked only after add_element; therefore a valid or crafted large index can allocate beyond the worker heap despite the 64 KiB decompression chunks. The Java path has the same behavior in VideoFormatWriter.java. Please validate from a bounded stream into spillable storage, or enforce explicit per-index and cumulative limits, instead of materializing unbounded bytes.