Skip to content

feat(net)!: writing a group needs no invented start, and its end has one word - #8961

Merged
kixelated merged 14 commits into
mainfrom
quest/m2/group-api
Nov 17, 2026
Merged

kixelated merged 14 commits into
mainfrom
quest/m2/group-api

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/group-api.md: a publisher opens a group at a stated Timestamp start, ends a group with finish, ends a track in error with a call of its own, and a fill handler matches a run of groups apart from a time lookup. The quest file and every link to it are deleted.

Summary

  • track::Producer::append_group(start: Timestamp): the start is required and is a Timestamp, not a bare u64. A timed track converts it into its timescale as write_frame does; an untimed track takes Timestamp::unscaled(key); a start that doesn't match its track, or that its scale can't hold, is TimestampMismatch, refused before any group exists. Production callers now pass the media timestamp they already hold: moq-mux's catalog, container, fMP4 and SI writers, moq-json's snapshot, window and stream producers, and moq-stats, which before passed timestamp.value() and so assumed the caller's scale matched the track's. moq-ffi's and moq-c's private next_start stays for their start-less append_group()/track_group, now returning the next key as a Timestamp in the track's timescale. Tests that count groups keep the test-only append_next (and the integration tests' AppendNext trait), which appends at the next key without a timestamp round trip; tests on timed tracks state their starts with from_millis/from_micros.
  • Groups: group::Producer::close() is finish(), and finish(end) is finish_at(end), matching track::Producer and what moq-ffi and JS already say. finish()'s docs say that readers cannot tell the group is whole, since only finish_at declares the end a media reader checks. About 1,300 call sites were renamed, guided by the compiler.
  • Track errored end: track::Producer::finish_at(end) is always a clean end. track::Producer::fail(self, err) -> Result<Finished> ends the track with err at its live edge, the errored counterpart of finish: readers get every group below the end, then err. A session forwarding a wire end that carries an error, possibly ahead of the live edge, uses the crate-private end_at(end, error). The three production callers (moq-mux end_with, moq-audio fail_track, moq-archive's replay End::Error) each computed the live edge by hand and now call fail.
  • track::Fill is an enum: Fill::Groups(FillGroups), today's run of groups minus time(), or Fill::Lookup(FillLookup), which has only time() -> u64, priority, max_delay, and accept. A lookup has no wanted, drop_groups, end, or is_wanted to misuse. moq-archive serves both kinds. moq-transcode, moq-ffi, moq-c, and moq-net's own lite and ietf sessions drop a lookup unanswered as soon as they receive it. Before, they pushed it into a fill run that ended immediately, so the outcome is the same.
  • moq-crypto follows its wrapped types: e2ee::group::Producer::close() is finish(), and e2ee::track::Producer::append_group takes a Timestamp start like the track it wraps, rescaled before it allocates so a refused start allocates nothing. This also settles that item of quest/m3/e2ee-format.md.
  • JS (@moq/net): Track.Producer.finishAt(end) drops its error parameter, and fail(error): Finished ends at the live edge. Track.Fill is FillGroups | FillLookup, told apart by kind: "groups" | "lookup", the codebase's union convention. The lite and ietf sessions close a lookup at once. Tests that needed an errored end past the live edge use the received-copy hook, the only path that produces one now.
  • Docs: doc/lib/rs/moq-net.md and doc/lib/js/net.md name the new calls. Quests that cited the old names were updated (refresh-interval, archive-time-followups, e2ee-format, the m2 README).

Impact

Public API: breaking in Rust (moq-net, moq-crypto) and JS (@moq/net). moq-ffi, moq-c, and every binding keep their API: they already served only runs of groups, ended groups with finish(), and their append_group() takes no start. Wire: none.

Migration:

  • moq-net:
    • track::Producer::append_group takes a Timestamp: pass the group's first frame's timestamp on a timed track (it is converted into the track's timescale), and Timestamp::unscaled(key) on an untimed one. append_group(key) becomes append_group(Timestamp::unscaled(key)) only where the track has no timescale.
    • On group::Producer, call finish() where you called close(), and finish_at(end) where you called finish(end).
    • Drop the error argument from track::Producer::finish_at(end, None). For finish_at(live_edge, Some(err)), call track.fail(err); it consumes the producer as finish does, so use track.clone().fail(err) to keep a handle.
    • Requests::requested_fill() returns track::Fill. Match Fill::Groups(fill) where you used the fill, and Fill::Lookup(lookup) where you checked fill.time().is_some(). FillLookup::time() returns u64. A handler that cannot resolve times drops the lookup.
  • moq-crypto: e2ee::group::Producer::close() is finish(), and e2ee::track::Producer::append_group takes a Timestamp as moq-net's does.
  • @moq/net: replace producer.finishAt(end, error) with producer.fail(error) at the live edge. Track.Fill is a union: check fill.kind === "lookup" where code checked fill.time !== undefined, and narrow to FillGroups before calling wanted, dropGroups, start, or end.

Decisions

Maintainer, 2026-11-16 (first, refined), reversing the group-api Plan item "append_group takes an optional start":

  • ✅ append_group(start: Timestamp), required, converted into a timed track's timescale as write_frame does, Timestamp::unscaled(key) on an untimed track, TimestampMismatch when it doesn't match (rejected: append_group(start: u64); an optional start, whether None meant the next counter on untimed tracks only, the previous group's declared end on timed tracks, or largest + 1 as first merged here; an optional frame timestamp to drop the Timestamp::unscaled(group.start) restatement). Reason: None meant the previous group's start + 1, a counter on an untimed track but neither the previous group's end nor a media time on a timed one, which the maintainer found confusing; and no Option<Timestamp> anywhere in the API.

Taken while applying it, open to review:

  • append_next stays test-only (cfg(any(test, feature = "fuzz"))), and appends at the next key directly rather than through a Timestamp, so it reaches the reserved key's BoundsExceeded on a timed track whose keys a Timestamp can't hold. Other crates' tests that used the next-key idiom spell it out (Timestamp::from_millis(track.largest().map_or(0, |l| l.group + 1))) rather than gaining a shared public helper, which would be the rejected implicit next start under another name.
  • moq-ffi and moq-c keep their start-less append_group()/track_group this PR: changing them is a breaking change in every binding. quest/m2/time-addressing/bindings.md, the bindings' breaking rename, now recommends giving them a required timestamp start.
  • JS appendGroup() still takes no start; JS has no start to mirror until quest/m2/time-addressing/js.md rekeys its groups by start.

Taken without asking earlier (the user was away), each open to review:

  • The errored end is named fail, ends at the live edge with no end argument, and consumes the producer to return Finished like finish. All three production callers wanted the live edge. The only caller that needs an end ahead of the live edge is the relay forwarding a wire end, and it uses the crate-private end_at.
  • The fill payloads are named FillGroups and FillLookup, not track::fill::{Groups, Lookup}. The same names work in JS, where Track.Groups is already taken by the subscription range type.
  • FillLookup exposes no start/end: the answer is "the group containing the time", which the handler finds from its source. A test-only span() checks what a lookup was asked over.
  • moq-ffi and moq-c keep dropping lookups (their MoqFill and fill handle now wrap FillGroups), and gain no fail: no binding has an errored end today, and nothing downstream asks for one. Exposing lookups to bindings stays with quest/m5/archive-time-followups.md, whose wording now names FillLookup. With no binding API change, no binding MIGRATE.md note rides this PR.
  • JS fail() is a no-op once an end is declared or the track closed, as JS finish() is. Rust's fail refuses with Closed, as Rust's finish does. Each mirrors its own language's finish.
  • rs/moq-mux/src/start.rs stays whole: every helper in it computes a timed track's start from a timestamp, and none existed only for the next-start idiom. container::Producer::next_start stays too, as the floor a seek clamps to.

Follow-ups

  • quest/m2/time-addressing/bindings.md (edited): its Plan now recommends a required timestamp start on the bindings' append_group(), mirroring Rust. Open decision: whether to take that in the bindings' rename (recommended) or keep the start-less binding append.

Validation

After the required-start change (merged with main through ec04746; just check reran on each merge): just check over the branch diff passes every module; cargo nextest run --workspace --tests --all-features ran 13,910 tests with 13,908 passing and two moq-audio PCM round-trip tests failing only under that full-workspace load (they pass alone, three runs out of three); just test interop --all passes. To find every timed track handed an unscaled start, a temporary build of append_group logged each such caller (with #[track_caller]) instead of refusing it, over the whole test suite and every bench in --test mode; each logged site now states its start in the track's unit.

Before it:

  • just ci over the branch diff: every module passes except test py. Its one failure, test_socket.py::test_abort_and_endpoint_close_are_errors, is a known flake (quest/m0/test-flakes-3/py-socket-abort.md, fixed by moq-dev/moq#8900). It fails 22 of 40 runs alone here, and this diff touches no socket code. Rust: 12,855 tests passed. JS: @moq/net's 3,327 tests passed. no-async clippy and rustdoc with -D warnings are clean.
  • just test interop --all: all checks passed. That runs the real Rust, JS (node and bun), Python, Go, and C clients against each other over every lite version and moq-transport, including the drop, tail, finish, track-replacement, and socket scenarios, whose Rust and JS clients this PR edits.
  • New tests: a timed track converts a scaled start and refuses an unscaled one, an untimed track takes an unscaled start and refuses a scaled one, and the reserved u64::MAX is refused; a group ended by finish() reads whole with end() == None, while finish_at declares its end; a track ended by fail serves every group below its end, then yields the error, and refuses a second end. In JS: a failed track keeps its history, and fail ends at the live edge and leaves an end already given.

Models

Size: L
Implementer: Claude Code, claude-opus-5-5, high effort
Implementer: Claude Code, claude-opus-5-5 (iterate, M)

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 5210fbb

  • Base: origin/main at ec04746

  • Risk: high, The branch changes public Rust and JavaScript track/group APIs, which can break downstream callers and require coordinated caller updates or an API rollback.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort

  • Session: 01a1e6df-6cef-7033-bbd3-500bc8c7cfa6, round 2

  • Score: 9/10, Static review of the incremental changes found no actionable regressions; builds and tests were not run, as instructed.

  • Raised the risk to high: The branch changes public Rust and JavaScript track/group APIs, which can break downstream callers and require coordinated caller updates or an API rollback.

  • Rounds: 5; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 6 commits November 16, 2026 08:33
Quest-Claim: 4eaa21678156a7a0
…inish, fail ends a track in error, and Fill splits groups from lookups

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rack

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ames

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Writing a group needs no invented start, and its end has one word feat(net)!: writing a group needs no invented start, and its end has one word Nov 16, 2026
@kixelated
kixelated marked this pull request as ready for review November 16, 2026 17:54
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 16, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

I reviewed the full branch at d4d9ec7. That covers the moq-net model changes, including track.rs, whose diff I pulled separately because it was too large for the PR diff file. It also covers the session and production callers and the @moq/net changes. The 200+ mechanical rename sites I spot-checked rather than read line by line.

  • append_group(None): this starts the group at TrackState::next_key() (max_key + 1, or 0). Removing the overflow filter is safe: create_group, append_group, and fill accepts all refuse u64::MAX, so max_key never reaches it. finish_shared already relied on the same invariant.
  • Group ends renamed: group::Producer::close() → finish(), and finish(end) → finish_at(end). An old call fails to compile because the arity changed. I found no remaining two-argument finish_at/finishAt callers and no references to append_next or Fill::time in code.
  • track::Producer::fail: this declares set_end(next_key, Some(err)) and consumes the producer as finish does. The wire-forwarding path keeps its received-copy tail handling through the crate-private end_at. The moq-mux, moq-audio, and moq-archive callers end at the same live edge they computed by hand before.
  • Fill enum: FillGroups keeps the run API, and FillLookup exposes only time, priority, max_delay, and accept. In moq-archive, the Asker that makes drop_groups/refuse_groups no-ops for a lookup matches the old behavior: owed() was always empty for a lookup. moq-ffi, moq-c, moq-transcode, and the sessions drop lookups unanswered, as the PR says.
  • JS: fail(error) declares at #received and mirrors finish()'s no-op guard. Fill is a union told apart by kind. The lite and ietf subscribers and the interop clients close or skip lookups.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI) · d4d9ec771ac6cf5e390dbbde5503c519d957efbb

No Rust runtime blocker found in the group/end changes. Choosing the next key and claiming it under the same lock is better than the old call-site arithmetic. fail preserves the drain-before-error path, and splitting Fill keeps the archive's time lookup while making unsupported lookups explicit at the wire and binding boundaries.

One migration safeguard worth adding: plain JS callers of finishAt(end, error) don't get TypeScript's arity error. They now silently declare clean EOF. At js/net/src/track.ts:2210, consider rejecting a supplied legacy error argument with a message pointing to fail(error) (or preserving the old overload for one release). Throwing is safer than silently claiming success when the requested error can't be represented at an arbitrary end. Static review only.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-16, /quest-run): append_group keeps a required start (rejected: None as the next counter on untimed tracks only; None as the previous group's declared end on timed tracks; merging None = largest + 1 as is). This reverses the 2026-11-16 group-api Plan item "append_group takes an optional start": None meant the previous group's start + 1, which is a counter on an untimed track but neither the previous group's end nor a media time on a timed one, and the maintainer found it confusing. The rest of the PR (finish/finish_at, fail, the Fill enum) stands, and the PR is picked to merge once this is applied.

(Written by Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-16, /quest-run), refining the one above: append_group(start: Timestamp). The required start is a Timestamp, not a bare u64: a timed track converts it into its timescale as write_frame does, and an untimed track takes Timestamp::unscaled(key); one that doesn't match its track is TimestampMismatch. Avoid any Option<Timestamp> in the API (rejected: append_group(start: u64); an optional start; an optional frame timestamp to drop the Timestamp::unscaled(group.start) restatement).

(Written by Opus 5.5)

kixelated and others added 4 commits November 16, 2026 13:05
The maintainer settled that a group's start is always stated: append_group
takes a Timestamp, converted into the track's timescale as write_frame does,
and an untimed track takes Timestamp::unscaled(key). A start that doesn't
match its track is TimestampMismatch. The optional start is gone, along with
its implicit 'previous start + 1'. Tests that count groups use the restored
test-only append_next (and the integration tests' AppendNext trait).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production callers pass the timestamp they already hold (moq-mux's catalog,
container, fmp4 and SI writers, moq-json's producers, moq-stats), moq-ffi and
moq-c compute their start-less append in the track's timescale, and tests on
timed tracks state their starts in milliseconds or microseconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ename

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Applied the refined decision at 712354c: track::Producer::append_group(start: Timestamp) is required, converted into a timed track's timescale, Timestamp::unscaled(key) on an untimed one, TimestampMismatch otherwise; e2ee::track::Producer::append_group mirrors it. The PR body's Decisions record it with the rejected options. The bindings' start-less append_group() is left to quest/m2/time-addressing/bindings.md as a recommendation, and JS appendGroup() takes no start to mirror yet.

On the earlier review's finishAt(end, error) note: declined. TypeScript callers get an arity error, and a runtime guard for an extra argument would be legacy-shape cruft in the API; the Impact section's migration note names fail(error).

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)
Head: 712354c5f4d4dcc0afb33b2da464b2047562cd16

No new blocker in the Rust timestamp migration. The mux and CMAF callers rescale the same timestamp they previously used to compute the group key; explicit pending starts still use create_group. The catalog and SI paths pass the bumped timestamp through, so same-tick updates still get distinct keys. The C and UniFFI compatibility shims retain their old synthetic numbering and the bindings quest now tracks removing it.

The earlier JS migration caveat still stands: untyped finishAt(end, error) calls silently declare a clean end. Consider rejecting the old argument rather than dropping an error. Static review; I did not run tests.

# Conflicts:
#	rs/moq-cli/src/play/video.rs
#	rs/moq-ffi/src/media.rs
#	rs/moq-video/src/decode/consumer.rs
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 16, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since d4d9ec7, append_group takes a required Timestamp instead of an optional u64. I checked the moq-net implementation and every non-test production caller. The test, bench and fuzz migrations I spot-checked.

  • Required Timestamp start: append_group rescales start through Info::rescale and passes the key to a private append_key. A start that doesn't match the track is refused before any group exists. write_frame now passes the rescaled timestamp, and rescaling it again changes nothing.
  • Test-only append_next: restored as pub(crate) under cfg(any(test, feature = "fuzz")). It calls append_key(next_key()) directly, so no Timestamp round trip is needed. The integration tests' AppendNext trait builds Timestamp::new(key, timescale), or unscaled on an untimed track.
  • moq-mux callers: the container producer, fMP4 import, catalog producer and SI writer now pass the timestamp, not the precomputed key. start::opens, start::after, start::next and start::before already return a timestamp that rescales to that key, so the group keys don't change.
  • moq-json and moq-stats: the snapshot, window and stream producers now take the converted key, and largest follows group.start. moq-stats previously passed timestamp.value() without converting, so this fixes a real scale bug.
  • moq-crypto: e2ee::track::Producer::append_group rescales before claim.allocate, so a refused start allocates nothing.
  • moq-ffi and moq-c: the private next_start now returns Timestamp::new(largest + 1, timescale), or unscaled on an untimed track. This exactly matches the key used before.
  • Quests: quest/m2/time-addressing/bindings.md now recommends a required timestamp start for the bindings.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

# Conflicts:
#	rs/hang/src/catalog/consumer.rs
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 16, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since 712354c, the only new commit merges origin/main. I checked its conflict resolutions, and that the code main brought in uses this PR's API.

  • Conflict resolutions: moq-cli play/video.rs and moq-ffi media.rs keep main's rewritten tests and end their groups with finish(). Main deleted the moq-video decode/consumer.rs tests that used the old append_group.
  • Tests from main migrated: the new tests now end groups with group.finish() in place of close(), and min_age_join.rs passes a Timestamp::from_millis start to append_group. That covers the min-age join tests in track.rs, the finished-track fetch tests in resume.rs, claim_continues' Worker::write and the new min_age_join.rs. None of the lines the merge adds call finish_at(end, err), finishAt(end, error) or Fill::time.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

# Conflicts:
#	quest/m2/README.md
# Conflicts:
#	rs/moq-boy/src/input.rs
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 17, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since bc478b2, the only new commits are two merges of origin/main. I reviewed how their conflicts were resolved and checked that the code main brought in uses this PR's API. I checked by reading the code only. The Quest check is the only CI that has run on 25c7454 so far, so nothing has compiled or tested the merged code yet.

  • Catalog consumer tests resolved: in rs/hang/src/catalog/consumer.rs and rs/moq-ffi/src/media.rs catalog_tests, main changed catalog consumers to return moq_json::Timed. The tests now pass a Timestamp::from_millis(ms) start to append_group, end each group with group.finish(), and read the payload through .value.
  • Quest index resolved: quest/m2/README.md drops the deleted group-api.md link along with main's removal of reconnect-off.md.
  • Merged code checked: none of the lines the merges add call group.close(), pass an integer to append_group, call two-argument finish_at/finishAt, or use Fill::time.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since 25c7454, the only new commit is a merge of origin/main at ec04746, which brings in #8936. I reviewed how its one conflict was resolved and checked that the code main brought in uses this PR's API.

  • moq-boy input tests resolved: main replaced the session-track send helper with a Viewer that publishes commands on a command track with a Timescale::MILLI timescale. The resolved tests call append_group(Timestamp::from_millis(1)), which lands on the same key 1 as main's append_group(1), and end the group with group.finish(). viewer.at = 1 makes the snapshot's next append land at key 2, after the stalled group.
  • Merged code checked: none of the lines the merge adds call group.close(), pass an integer to append_group, call two-argument finish_at/finishAt, or use Fill::time.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-16, /quest-run): ✅ no runtime guard for an untyped-JS caller still passing an error to finishAt(end) (rejected: a runtime throw for one release). Only TypeScript is supported, so the API does not check at runtime what TypeScript already makes impossible; that rule goes into js/AGENTS.md in its own PR. Keep the bindings follow-up in quest/m2/time-addressing/bindings.md. Merge.

(Written by Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Head 5210fbb already implements #8961 (comment): finishAt(end) has no runtime guard, and the bindings follow-up is recorded in quest/m2/time-addressing/bindings.md. No change is needed.

(Written by Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 128e4c2 Nov 17, 2026
8 checks passed
@kixelated
kixelated deleted the quest/m2/group-api branch November 17, 2026 02:00
kixelated added a commit that referenced this pull request Nov 17, 2026
Takes main's group::Producer::finish_at(end)/finish() names (#8961), with JS
mirroring them as finishAt(end)/finish(), keeps main's Announce Stream (0x3)
while dropping Padding (0x2), and drops main's edits to the re-offer and
judged-range tests this branch removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant