Skip to content

fix(transcode): a cold fetch finds a finer source's group by time - #9120

Open
kixelated wants to merge 3 commits into
mainfrom
quest/m1/precut-fixes/transcode-history-key
Open

kixelated wants to merge 3 commits into
mainfrom
quest/m1/precut-fixes/transcode-history-key

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/precut-fixes/transcode-history-key.md (deleted here).

Problem

A transcode rung keys its output groups in microseconds. A source keyed finer than that (a 1 GHz CMAF track) maps a run of starts to each output start: source group 1,000,001 ns is output group 1,000. The live path mirrors it there. A cold fetch of output group 1,000 inverted the mapping to the run's lowest start (1,000,000), fetched that exact key, got NotFound, and dropped a group that exists. Live and history disagreed on the same key.

Approach

  • source_starts returns the whole run of source starts that map to an output start, replacing source_start, which returned only the lowest.
  • A run of one (any source no finer than microseconds) is fetched by key, as before.
  • A longer run is looked up by time: the fetch subscribes to the single group holding the run's last start (Time(hi)..=Time(hi)). That group mirrors the output group when it starts inside the run; otherwise the source has none there and the existing gap drop takes over. This fetches nothing extra when the group exists: no group below it is pulled, and there is no fetch per nanosecond.
  • The subscription stays alive with the group until the transcode reads it.
  • A time lookup can wait on the source's media, so a retired rung's drain refuses the group (Cancel) instead of holding the retirement. This matches how drop_gap already handles its lookup.

Impact

  • No public API or wire change. Behavior change: cold fetches of a rung over a sub-microsecond source now serve the group live serves at that key, instead of dropping it.

Decisions

Made without asking, for review:

  • Resolve, not retain. The alternative was to give each rung track the source's timescale, so output keys equal source keys and the inverse goes away. Rejected for a pre-cut fix: it changes every rung's public track timescale and means accepting the track request only after the source's info resolves.
  • Two source groups inside one microsecond. The lookup finds the newest of them, while live writes the first that arrives. Whichever path writes the output group first wins, through the cache's Duplicate. Finding the lowest instead would pull the group below the run on every cold fetch. Per AGENTS.md, a publisher this degenerate (two keyframe groups under 1 µs apart) only hurts its own broadcast.
  • An unanswered lookup reads as a miss. FillLookup has no refusal, so a source handler that fails a lookup (an archive store error) looks to moq-net like "can't tell", and the read falls back to the groups held. Then this fetch drops the output group for the rung's lifetime. This is moq-net's contract, shared by every time-anchored reader, drop_gap included. The refusal it needs is already planned in quest/m5/archive-time-followups.md, and once it exists this fetch refuses the group (retryable) with no change here. Codex raised this in round 1, and the refutation stood in round 2.

Alternatives

  • Subscribe Time(lo)..=Time(hi) and take the first group at or after lo: this finds the lowest group in the run, but the start anchor resolves to the group holding lo, usually the previous group, so every cold fetch would also pull a whole extra group.

Validation

  • Reproduced first: tests::a_finer_source_fetches_the_group_live_serves runs two real run() transcoders (openh264) over a nanosecond source whose groups start 1 ns past their first frame. One serves output group 166,665 live. Before the fix, a cold fetch of 166,665 on the other failed with NotFound. After the fix, it returns the same group with the same frame timestamps. Fetching group 0 cold works too.
  • Independent check at the source boundary: rung::tests::a_finer_source_is_asked_for_one_group records what the source is asked. Exactly one time lookup at 1,000,999 (the last nanosecond of microsecond 1000), answered with 1,000,001, and nothing else. a_finer_source_without_the_group_drops_it answers with a group below the run, and the output group is dropped. a_retired_rung_refuses_a_finer_lookup_it_cant_place refuses rather than hangs. Removing the retire branch makes that last test hang, so it covers the branch.
  • just ci passes (382 tests in the touched set).

Caused by

  • moq-dev/moq#8638 - introduced source_start, which inverted a finer source's output start to the lowest source start and fetched that exact key (found by the 2026-11-17 pre-cut correctness audit)

Follow-ups

  • Edited quest/m5/archive-time-followups.md, which owns FillLookup::refuse (maintainer decision 2026-10-31). Its Plan now names this fetch as a consumer that needs it, plus the test it owes. No new decision.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, high effort

(Written by Claude Opus 5.5 with high)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 220230c

  • Base: origin/main at 4c8af4c

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1f0c1-e879-7f62-9cc6-8b67dffcdc2d, round 2

  • Score: 10/10, The incremental changes document an existing lookup limitation and its previously approved follow-up, with no new defects found.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 0, refuted 1, open 0

  • Findings: none

  • The last round's findings, at 5eb9c79:

    • Refuted: medium, rs/moq-transcode/src/rung.rs:887: Preserve retryability when a source lookup is unanswered; FillLookup's documented fallback contract in rs/moq-net/src/model/track/fill.rs:924 confirms the shared limitation, and quest/m5/archive-time-followups.md already records the maintainer's decision in the merge base to address it through lookup refusal.

kixelated and others added 2 commits November 18, 2026 14:04
A source keyed finer than microseconds (a 1 GHz CMAF track) maps a run of
its starts to each output start. The fetch path inverted that to the run's
lowest start and fetched it by key, so a group starting anywhere else in
the run read as missing and the output group was dropped, though live
serves it. A run is now looked up by time: the source is asked for the one
group holding the run's last start, which mirrors the output group when it
starts inside the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Fetch the same transcoded group live and from history fix(transcode): a cold fetch finds a finer source's group by time Nov 18, 2026
The fallback is owned by quest/m5/archive-time-followups.md, which now
names this consumer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 18, 2026 22:34
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: aa1d01e2-a9a0-4fa1-88f4-76c3b895869e
Commits

Reviewing files that changed from the base of the PR and between 71c6209 and 220230c.

Files selected for processing: 4

  • quest/m1/precut-fixes/transcode-history-key.md
  • quest/m5/archive-time-followups.md
  • rs/moq-transcode/src/lib.rs
  • rs/moq-transcode/src/rung.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: Sound. The fix inverts the output key to the whole run of source starts it covers, and only uses a single bounded Time lookup when the run is longer than one. Sources keyed at µs or coarser keep the exact-key fetch, and the lookup and the retired-rung refusal follow the same pattern drop_gap already uses.

Reviewed changes

All four files at 220230c were reviewed against moq-net's time-anchor resolution (Window::start_waiting, unresolved, Holdings::containing) and Timestamp::convert's rounding.

  • source_starts replaces source_start: The run [ceil(s·scale/1e6), ceil((s+1)·scale/1e6) − 1] is exactly the set of starts that output_start's floor conversion maps to s. An empty run returns None and replaces the old round-trip check. Scale-less and coarser sources still get a run of one.
  • source_group: A run of one is fetched by key, so NotFound still means a miss. A longer run subscribes Time(hi)..=Time(hi), which resolves to the newest group starting at or before hi. That group is accepted only if it starts inside the run, so it never pulls the group below or one fetch per tick. The Ordered stays alive in Source until transcode_group has read the group.
  • Retired drain: The retire.fired() arm is gated on placing, so only the lookup that can wait on source media refuses with Cancel. This mirrors drop_gap's retire arm.
  • Tests: An end-to-end openh264 test checks that live and cold fetches agree at output key 166_665 on a nanosecond source. rung unit tests pin the single lookup at 1_000_999 and cover the drop and retired-refuse branches.
  • Quest docs: The finished quest file is removed. The FillLookup refusal follow-up now names this fetch as a consumer that needs it.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated

Copy link
Copy Markdown
Collaborator Author

Decision from the maintainer:

  • ✅ Rework: give each rung's output track the source's timescale, so output keys equal source keys and the conversion goes away (rejected: per-microsecond time lookup, hold). This changes each rung's public track timescale; record the migration note in Impact if it is breaking.

(Written by Claude Opus 5.5 with high)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant