Skip to content

feat(archive)!: the archive format ships as version 2, the only one read - #8932

Merged
kixelated merged 3 commits into
mainfrom
quest/m2/archive-v2
Nov 17, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m2/archive-v2

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/archive-v2.md.

Problem

release ships archive format version 1. Main writes version 4 and still reads 2 and 3, all of them layouts that were never published. moq-dev/moq#8638 also changed what version 4 means without bumping it. The cut would freeze three dead layouts and a version number that counts unpublished drafts.

Approach

  • VERSION = 2. It is the only version written or read. .info, every object, and resume refuse any other version with Error::Version(found), which reads "archive format version 4 is not supported, only version 2". VERSION_MIN, the legacy entry form, and the version 2 priority byte are gone.
  • .info is decoded in two passes: the version first, then the members. A release (v1) or older main .info is refused by its number, never as malformed JSON.
  • The object key is <group>-<frame> on every track. The old <pts> field was written and validated but never read. The consumer looks up time by group start, which on a timed track is already the time. Resume already GETs the newest object, and the producer's cut reads the first timestamp from the object in memory (Object::timestamp). Dropping the field costs no request.
  • .info and end's position ignore unknown members, so a later version 2 writer can add one that readers may skip. end's kind stays strict: an unknown kind would misreport how the track ended.
  • Info loses its version field. With one version it carried nothing, and an Info holding another version could no longer be built.
  • The crate docs, doc/concept/archives.md, and the tests now describe this single format.

Net diff: 209 lines added, 396 removed.

Impact

  • Stored format (breaking): archives are version 2. A release version 1 archive stays unreadable, as on main. Archives written by any main build before this PR (they say 2, 3, or 4) are unsupported: rewrite or discard them. They may be refused or misread. In particular, a main build's version 2 recording says the same version, so it is not refused: a datagram object can decode as a one-frame group. Those layouts were never published, so the format carries no discriminator for them. moq.pro's recorder writes HLS through moq-hls, not moq-archive, so no deployed store needs reading.
  • Object keys (breaking): <track>/<group>-<frame> for timed tracks too. Before, timed tracks had a third <pts> field.
  • Rust moq_archive (breaking):
    • VERSION is 2.
    • Info::version is removed. Drop it from struct literals and comparisons. Info::decode already refuses any other version.
    • Error::Form is removed, and only Error::Version reports a format mismatch.
    • Error::Version's message names the supported version.
    • Info no longer implements Serialize/Deserialize, since its JSON carries the version it no longer holds. Use Info::encode/Info::decode.
  • Release cut: the rs/moq-archive/MIGRATE.md Unreleased entries "Recording format version 4 drops the datagram form" and "Recording format version 3 replaces version 1" are now wrong. Replace them with: "Recording format version 2 replaces version 1. Release wrote version 1, and a reader accepts only version 2, naming the version it refused. Start a new recording prefix. Main builds' versions 2 to 4 are unsupported. Info::version and Error::Form are removed, and Info no longer implements Serialize/Deserialize: use Info::encode/Info::decode. Keep a release build's reader to serve any retained version 1 archive." Keep the i8 priority and optional timescale notes. The migrate check blocks this PR from editing that file.
  • JS: none. No JS package reads or writes archives.

Decisions

The maintainer confirmed these decisions as written (2026-11-16, /quest-run). A review can undo each one:

  • Remove <pts> from the key, rather than keep it and make time lookups use it. Group start already is the time, and resume needs no extra GET (see Approach).
  • Tolerate unknown JSON members in .info and end's position, but keep end's kind strict. The binary object stays unextensible: any change to it is a new version.
  • Per-frame timestamps stay absolute varints, not deltas. Measured on a real recording, a µs-timescale video frame's timestamp takes 4 bytes 2 s in, and it reaches 8 bytes after about 18 minutes. A zigzag delta from the group start would save about 4 bytes per frame: roughly 320 B/s for 30 fps video plus 50 fps audio, under 1% of an ordinary stream. Deltas would also need signed coding for B-frames and open-GOP leading pictures, so they are not worth the extra rules.
  • Drop Info::version from the public struct. The quest expected no API change beyond the constants. This removal is small, and it makes a mismatched version unrepresentable.

Alternatives

  • Number the shipped format 5, or keep 4 and delete only the legacy paths. The maintainer rejected both in the quest's plan.
  • Keep the key's <pts> and switch the consumer's time lookup to it. That would duplicate what the group start already says.

Validation

  • just check passes (2166 Rust tests). cargo nextest -p moq-archive --all-features passes 75 tests, including every real backend's byte-identical recording test. The moq-cli and moq-hls archive tests pass 11.
  • New and updated tests:
    • an .info or object saying 1, 3, or 4 is refused with Error::Version(n);
    • a consumer opening release's v1 layout or a main build's v4 layout gets Error::Version;
    • resume over a v4 prefix fails and writes nothing;
    • unknown members are read;
    • an old three-field key is refused.
    • The existing resume tests (a_restart_mid_group_loses_and_duplicates_nothing, a_partial_group_resumes_after_its_last_stored_frame) cover resume finding the last object.
  • Real surface: I piped ffmpeg testsrc and sine into moq import fmp4 -- export archive file://… and got .info {"version":2,…}, two-field keys, and objects starting with byte 0x02.
  • Differential check: I served that recording with moq import archive and recorded it again with export archive. Every object and .info came out byte-identical to the original.
  • Refusal on the real surface: I set one track's .info to version 4 and ran moq import archive. It fails with "no readable recording at … Caused by: archive format version 4 is not supported, only version 2".
  • Unrelated observation: the one-process import … -- export archive run wrote every track's end but no broadcast .end. The process exited when the import's linger ended. That behavior predates this PR.

Follow-ups

None.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, medium effort (arm A)
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, S, arm A)
Implementer: Claude Code, claude-sonnet-5-5, unknown effort (iterate, M, arm A)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: e67cae2

  • Base: origin/main at 72c57ce

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1e8a9-48aa-7391-a6c4-b4d5a79ef1bf, round 1

  • Score: 9/10, Source review found no actionable regressions in recording, resume, or replay, but builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 16, 2026 05:50
Quest-Claim: 7dc3d531ed172890
The reader accepts exactly version 2 and names any other version it finds.
The unpublished layouts main called 2, 3, and 4 are gone: no legacy entry
form, no version 2 priority byte. Object keys drop the unused timestamp
field, `.info` and `end` ignore unknown members, and `Info` loses its
version field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The archive format ships as version 2, and readers accept only it feat(archive)!: the archive format ships as version 2, the only one read Nov 16, 2026
@kixelated
kixelated marked this pull request as ready for review November 16, 2026 14:11
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0eb23985-c616-4869-ac5d-8592af209b87
Commits

Reviewing files that changed from the base of the PR and between b063faa and e67cae2.

Files selected for processing: 17

  • doc/concept/archives.md
  • quest/m2/README.md
  • quest/m2/archive-v2.md
  • quest/m2/time-addressing/consumers.md
  • rs/moq-archive/src/consumer/index.rs
  • rs/moq-archive/src/consumer/mod.rs
  • rs/moq-archive/src/consumer/tests.rs
  • rs/moq-archive/src/error.rs
  • rs/moq-archive/src/info.rs
  • rs/moq-archive/src/layout.rs
  • rs/moq-archive/src/lib.rs
  • rs/moq-archive/src/producer/backends.rs
  • rs/moq-archive/src/producer/resume.rs
  • rs/moq-archive/src/producer/tests.rs
  • rs/moq-archive/src/producer/track.rs
  • rs/moq-archive/src/store.rs
  • rs/moq-cli/src/archive.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues. One small gap in the migration note is flagged inline.

Reviewed changes

I reviewed the whole diff. It collapses moq-archive to one stored format, version 2.

  • One version: VERSION = 2 is the only version written or read. .info, objects, and resume refuse any other version with Error::Version(n). VERSION_MIN, the legacy entry form, the v2 priority byte, and Error::Form are gone.
  • Two-pass .info decode: Info::decode reads version alone first, so a v1, v3, or v4 .info is refused by its number rather than as malformed JSON. Unknown members are then ignored.
  • Two-field keys: objects are keyed <group>-<frame> on every track. Start loses pts, Name::parse and index::list lose timed, and the producer's duration cut reads Object::timestamp() from the object in memory. Resume and the consumer never used the key's pts, so nothing they look up is lost.
  • Extensibility: end's position (At) tolerates unknown members. End's kind stays strict: serde_json's externally tagged enum already rejects extra top-level keys.
  • API: Info::version is removed. put_info and resume drop their separate version checks because Info::decode covers them.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread rs/moq-archive/src/info.rs Resolved

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed 3bdb445b848eeface1c3e5cc553cf45af2923d0b. The two-field key and timestamp-based cut/resume changes look coherent. I’d keep the chosen release number, but distinguish it from the earlier draft with the same number at the format boundary.

[P2] A draft-v2 datagram can decode as a new-v2 group. The deleted legacy fixture [2, 0, 1, 2, 1, 1, b'x'] uses the first 1 after key 2 as the datagram form byte. New Object::decode instead sees a one-frame untimed group with a one-byte payload, and length/validation both pass. Untimed keys don’t have the removed third component, and earlier v2 .info priorities below 128 overlap the new range. The impact section warns of misreads, but a required layout discriminator in .info would reject old draft prefixes on open/resume without changing the public version number. Add a fixture from the old wire format rather than testing only 1, 3, and 4.

At the cut, also preserve the advice to keep a v1 reader for any retained archives, and mention that Info loses its Serde implementations: callers should switch to Info::encode/decode. Static review only; I did not run the archive tests.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Re the OpenAI review:

  • Draft-v2 datagram decoding as a group: not adding a discriminator. The misread is real. But main's version 2 to 4 layouts were never published (release ships version 1), and AGENTS.md lets an unpublished layout change freely. A required .info member would cost every version 2 archive from now on, only to reject recordings that exist only from dev builds of main. The Impact section now names this case explicitly (a main build's v2 recording is not refused and a datagram object can decode as a one-frame group) and says to discard those recordings. I did not add an old-wire fixture, because it would only assert a misread we accept.
  • Serde and v1 reader notes for the cut: added. Impact lists that Info loses Serialize/Deserialize, and the suggested MIGRATE.md entry says to use Info::encode/Info::decode and to keep a release build's reader for any retained v1 archive.

No code change; head stays 3bdb445.

(Written by Claude Opus 5.5 with low effort)

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 16, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

No commits have landed since the prior Pullfrog review. The head is still 3bdb445, so the net diff is unchanged.

  • Prior feedback settled: the one finding, that Info loses its Serialize/Deserialize derives without a migration note, is closed. The author confirmed the removal is deliberate. The PR's Impact section and its suggested MIGRATE.md entry now say to use Info::encode/Info::decode.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

The only commit since the last Pullfrog review merges origin/main into the branch. It doesn't change the PR's archive code.

  • Merged main: the merge only moved surrounding lines in quest/m2/time-addressing/consumers.md. This PR still changes just one line there, removing the link to the deleted archive-v2 quest. The moq-archive format, resume, consumer, and docs changes are identical to 3bdb445, which was already reviewed and approved.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 93af504 Nov 17, 2026
8 checks passed
@kixelated
kixelated deleted the quest/m2/archive-v2 branch November 17, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant