Specify dynamic (pattern) advertisements in moq-lite, plus a moqt extension draft - #3005
Specify dynamic (pattern) advertisements in moq-lite, plus a moqt extension draft#3005kixelated wants to merge 4 commits into
Conversation
A publisher can advertise a (prefix, suffix) pattern of paths it could serve on demand instead of enumerating them: WILDCARD_START (0x3) and WILDCARD_UPDATE (0x4) on the Announce Stream, sharing the Announce ID space and retracted by ANNOUNCE_END. A wildcard is a capability rather than an inventory, carries a hop list and one Route Cost (no Epoch, never warm), and forwards under the existing routing rules. Resolution for an unadvertised path consults only the most specific matching tier, orders it by cost, and distributes by a hash of the requested path; a new CAPACITY stream error (0x20) permits one re-resolution excluding the refuser, while every other reset stays terminal. Mirrored in moq-cluster as the WILDCARD_SUFFIX parameter. The Ended flag is removed from ANNOUNCE_REQUEST, ANNOUNCE_START, and ANNOUNCE_UPDATE: announcing recordings is per-recording announce state, the growth wildcards exist to stop, and a recording is discovered out of band and read via FETCH with the wildcard routing the request. Path matching is respecified as segment-aware, matching the implementations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c529de518f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| The original publisher seeds the value with its production cost: 0 for content it is already producing, higher for content it would have to spin up on demand, such as a standby transcoder advertising everything it *could* serve. | ||
|
|
||
| ## WILDCARD_SUFFIX Parameter {#wildcard-suffix} | ||
| A publisher MAY advertise a pattern of namespaces rather than an exact one: a PUBLISH_NAMESPACE or NAMESPACE carrying WILDCARD_SUFFIX matches every namespace that starts with the message's namespace and ends with the parameter's fields, both possibly empty and matching whole tuple fields. |
There was a problem hiding this comment.
When a wildcard is carried in a NAMESPACE response, the message's namespace field is a suffix relative to the enclosing SUBSCRIBE_NAMESPACE prefix, not an absolute namespace. For example, under a subscription to tenant, a NAMESPACE suffix of worker must describe the prefix tenant/worker; treating worker as the complete pattern prefix matches the wrong namespaces and validates authorization against the wrong scope. Specify reconstruction against the subscribed prefix while keeping PUBLISH_NAMESPACE absolute. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| |:--------|:----------------|:-----------------------------|:--------------| | ||
| | 0x40B57 | HOP_PATH | PUBLISH_NAMESPACE, NAMESPACE | This Document | | ||
| | 0x40B58 | ROUTE_COST | PUBLISH_NAMESPACE, NAMESPACE | This Document | | ||
| | 0x40B59 | WILDCARD_SUFFIX | PUBLISH_NAMESPACE, NAMESPACE | This Document | |
There was a problem hiding this comment.
During a rolling upgrade, an older peer can successfully negotiate RELAY_HOPS and then receive this newly registered parameter without supporting it. The existing moq-transport message-parameter decoder (decode_params! in rs/moq-net/src/ietf/parameters.rs) deliberately rejects unknown message parameters, so the advertisement or session fails instead of degrading to non-wildcard behavior. Add a separately negotiated SETUP capability or a protocol-version boundary, and send WILDCARD_SUFFIX only after the peer advertises support. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L204-L204
Useful? React with 👍 / 👎.
|
|
||
| A request for a namespace nothing advertises MAY resolve against pattern advertisements ({{wildcard-suffix}}), after the same origin exclusion. | ||
| Only the most specific matching tier is consulted (the longest literal match, prefix plus suffix in tuple fields; equal-specificity patterns form one pool ordered by ROUTE_COST, then distributed by a deterministic hash of the requested namespace against each advertiser). | ||
| A reset with a capacity error permits one re-resolution within the tier, excluding the refuser; every other reset is terminal, so probing unserved namespaces costs one round trip each. |
There was a problem hiding this comment.
In moq-transport, requests are refused with REQUEST_ERROR, while a stream reset does not provide the typed application response this rule needs; this draft also assigns no numeric capacity code or REQUEST_ERROR mapping. Consequently, a receiver cannot distinguish retryable capacity exhaustion from a terminal refusal, making the specified one-time re-resolution non-interoperable. Define and register the exact REQUEST_ERROR code and its carriage rather than referring to an unspecified reset. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
| A receiver MAY resolve a SUBSCRIBE, FETCH, or TRACK request for an unadvertised path against its wildcards. | ||
| Only the most specific matching tier is consulted: the longest literal match, prefix plus suffix in segments, with equal-specificity patterns forming one pool. | ||
| Within the tier, order by lowest accumulated Route Cost, then distribute by a deterministic hash of the requested path against each advertiser, so a set of paths spreads across a pool of advertisers while one path always resolves the same way. | ||
| A resolved path competes on that same cost against a concrete advertisement of it; nothing ranks the two kinds differently. |
There was a problem hiding this comment.
The first sentence limits wildcard resolution to an unadvertised path, but the following rule requires the resolved wildcard path to compete with a concrete advertisement. When both an exact publisher and a matching standby exist, conforming implementations can therefore either ignore the wildcard or compare their costs, selecting different publishers and potentially starting duplicate production. State one eligibility rule and cross-reference it consistently. (Written by GPT-5.6 Sol)
AGENTS.md reference: drafts/AGENTS.md:L69-L72
Useful? React with 👍 / 👎.
A wildcard is routing capability, not content discovery, so it no longer rides the Announce Stream: WILDCARD_REQUEST/WILDCARD_OK then WILDCARD_START/END/UPDATE mirror the announce mechanics with their own Wildcard ID space. The separation also buys the deployment story for free: a peer without support resets the unknown stream type, which the spec already defines as the non-fatal extension fallback, so broadcast announcements are never at risk and no shared implicit-id counter forces an ignoring peer to count messages it does not understand. The doc test pinning the stream table's row count learns the new row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reworked in cdbafd7 per @kixelated's review: wildcards now live on their own Wildcard Stream (0x7) instead of riding the Announce Stream. Beyond the API-shape argument (routing capability vs content discovery), the shared design had two concrete problems the separation removes. Announce IDs are implicit — derived from message order — so a peer that merely tolerated unknown wildcard messages would still have had to count them to keep broadcast ids aligned; each stream now has its own counter. And a new message type on the shared stream would have killed an older peer's announce stream outright, forcing the accept-first fleet staging; the spec already defines unknown stream types as a non-fatal reset ("the fallback when an extension stream is opened against a peer that did not negotiate it"), so the new stream degrades gracefully and announcements are never at risk. The cluster mirror is unchanged (parameter-based). The cross-kind rules the shared design needed (ANNOUNCE_UPDATE-on-a-wildcard-id violations, the Active Count contortion) are gone with it. (written by Fable 5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cdbafd7b63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| The remainder of the pattern's prefix after the stream's requested prefix, like ANNOUNCE_START's Broadcast Path Suffix. | ||
| A pattern belongs on a stream when it can match a path under the requested prefix: its prefix extends the requested one (encode the remainder) or the requested one extends its prefix (encode empty, since every path on the stream already satisfies it). | ||
| Empty means every path under the requested prefix. |
There was a problem hiding this comment.
When the requested prefix extends the wildcard prefix, encoding an empty remainder is not equivalent if the requested prefix already contains the wildcard suffix. For example, pattern (foo, bar) matches foo/bar because its halves are adjacent, but on a stream requesting foo/bar this rule reconstructs (foo/bar, bar), which cannot match foo/bar under the non-overlap rule. A valid scoped subscriber therefore silently loses the exact path; define an encoding or reconstruction rule that preserves this overlap. (Written by GPT-5.6 Sol)
Useful? React with 👍 / 👎.
The mechanism routes dynamically produced broadcasts and namespaces; "wildcard" named the matching shape and implied glob syntax the design does not have. The lite stream becomes the Dynamic Stream (0x7) with DYNAMIC_REQUEST/OK/START/END/UPDATE and Dynamic IDs. The moq-transport mirror moves out of moq-cluster into a standalone extension, draft-lcurley-moq-dynamic: a DYNAMIC setup option (0x40B5B) negotiates support, the DYNAMIC_SUFFIX parameter (0x40B59) marks a PUBLISH_NAMESPACE or extended NAMESPACE as a pattern, and the capability, specificity-tier selection, capacity-refusal retry, and authorization semantics travel with it. Cost ordering references moq-cluster as an optional integration, so the extension stands without the mesh machinery; moq-cluster reverts to untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kixelated
changed the title
|
Two more review-driven changes in 307422a: Renamed wildcard → dynamic throughout, per @kixelated: the mechanism routes dynamically produced broadcasts/namespaces, while "wildcard" named the matching shape and implied glob syntax the design deliberately lacks (the message carries two literal halves, no wildcard character). The lite stream is now the Dynamic Stream (0x7) with DYNAMIC_REQUEST/OK/START/END/UPDATE and Dynamic IDs; prose says "dynamic advertisement". Split the moqt half into a standalone extension draft,
(written by Fable 5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 307422ac85
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| When the stream is closed, the subscriber MUST assume that all broadcasts are now unavailable. | ||
|
|
||
| Path prefix matching and equality is done on a byte-by-byte basis. | ||
| Paths are sequences of `/`-delimited segments, and all matching is segment-aware: a prefix (or suffix) matches only whole segments, so `foo` is a prefix of `foo/bar` but not of `foobar`. |
There was a problem hiding this comment.
When an untrusted path contains leading, trailing, or repeated slashes, this rule does not say whether empty segments are retained, collapsed, or rejected. The Rust and JS decoders currently normalize those forms (rs/moq-net/src/path.rs and js/net/src/path.ts), while another implementation can reasonably preserve the empty segments, causing peers to disagree on advertisement matching and authorization scope. Specify canonicalization or reject noncanonical paths before applying the new segment-aware comparisons. (Written by GPT-5.6 Sol)
AGENTS.md reference: drafts/AGENTS.md:L10-L13
Useful? React with 👍 / 👎.
| # Acknowledgments | ||
| {:numbered="false"} | ||
|
|
||
| This document was drafted with the assistance of Claude, an AI assistant by Anthropic. |
There was a problem hiding this comment.
This source is translated into a /doc/draft/ page, so the acknowledgment publishes a source-level AI attribution that repository guidance explicitly prohibits because those markers rot. Remove the attribution from the draft. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L111-L111
Useful? React with 👍 / 👎.
The pattern prefix now travels whole - the requested prefix filters which patterns a stream carries but never rebases the encoding, since rebasing was lossy and corrupted specificity, re-forwarding, and pattern identity. The ended-broadcast lifecycle is stated: a broadcast that ends but remains readable retracts with ANNOUNCE_END (which never disturbs in-flight subscriptions) and serves stored groups over FETCH, discovered out of band. CAPACITY moves to 0x30, clear of every value current implementations emit as placeholders, the reserved range admits assigned exceptions so the session space keeps 0x20 reserved, and a receiver that spent its one re-resolution refuses downstream with a terminal code so the retry cannot compound per hop. Selection pins the rendezvous FNV-1a hash (seed, input, advertiser identity), states that spreading among cost-equals is the design, and defines the mixed dynamic-versus-concrete comparison (single cost as both halves, kind carries no rank, concrete tie-break chain applies). Authorization is receiver-side in both documents with discard as the action. Path grammar and equality are defined. moq-dynamic becomes implementable: cluster is a normative reference whose HOP_PATH/ROUTE_COST rules apply unchanged, multi-hop resolution requires it, DYNAMIC_CAPACITY is named and registered, and the codepoints move to 0x40B5C/D/F, clear of moq-broadcast's 0x40B59/0x40B5B. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Ran a high-effort review over this PR (8 finder angles, adversarially verified); ten findings, nine confirmed, all fixed in eec6991:
Implementation of the Dynamic Stream in rs/moq-net is underway on a stacked branch. (written by Fable 5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eec69912fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Only the most specific matching tier is consulted: the longest literal match, prefix plus suffix in segments, with equal-specificity patterns forming one pool. | ||
| Within the tier, the lowest accumulated Route Cost wins, and a deterministic hash distributes among advertisements tied at that lowest cost. | ||
| Spreading among equals is the design: a co-located worker pool ties on cost and shares the work, while a costlier advertiser is deliberately overflow rather than an equal peer. | ||
| The hash is rendezvous-style FNV-1a: for each candidate, hash the requested path's bytes followed by the advertiser identity as a 64-bit little-endian value, seeded with 0x420C0DECB00B, and select the highest result, so one path always resolves the same way and a pool member arriving or leaving moves only its own share. |
There was a problem hiding this comment.
When equivalent noncanonical spellings such as /foo//bar and foo/bar/ reach a dynamic pool, the comparison rule normalizes them into one path, but this line hashes the received bytes and therefore permits them to select different advertisers and start duplicate production for the same broadcast. The fresh evidence beyond the earlier slash-handling comment is that normalization is limited to comparisons while the newly specified rendezvous input remains raw bytes; require the normalized bytes here as well. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L111-L111
Useful? React with 👍 / 👎.
| When several patterns match, only the most specific tier is consulted: the longest literal match, prefix plus suffix in tuple fields, with equal-specificity patterns forming one pool. | ||
| A refusal from that tier is the answer; it never falls through to a less specific pattern, so a request the winning tier will not serve costs one round trip rather than a walk down the candidates. | ||
|
|
||
| Within the tier, the lowest accumulated ROUTE_COST wins, and a deterministic hash distributes among advertisements tied at that lowest cost, so a set of namespaces spreads across a co-located pool while a costlier advertiser is deliberately overflow. |
There was a problem hiding this comment.
When peers negotiate DYNAMIC but not the cluster extension, direct single-hop resolution remains permitted by lines 79-80, yet no advertisement carries ROUTE_COST and this rule still requires the lowest accumulated value to win. With multiple matching direct advertisers, implementations cannot determine whether missing costs are zero or whether local policy should choose; define the no-cluster ordering/default or require Relay Hops for all resolution. (Written by GPT-5.6 Sol)
AGENTS.md reference: AGENTS.md:L111-L111
Useful? React with 👍 / 👎.
1 participant
A publisher can advertise a pattern of paths it could serve on demand — a prefix and a suffix, either half empty — instead of enumerating every broadcast. Three workloads motivate it (planned in moq.pro's wildcard questline): a transcode worker advertising
("", "transcode.pro")once for a whole fleet instead of a standby announcement per matching broadcast, a chat backend advertising("<pid>/chat", "")for rooms that exist independently of any broadcast, and an archive advertising("", "")— "if nobody is publishing this live, I have it".moq-lite
Endedflag from ANNOUNCE_REQUEST, ANNOUNCE_START, and ANNOUNCE_UPDATE. Announcing recordings is per-recording announce state — the exact growth wildcards exist to stop. A recording is discovered out of band (e.g. a catalog, which also carries the generation identity) and read via FETCH, with a wildcard routing the request to whatever serves it. The ended-broadcast content state survives in the Subscribe/Fetch prose; only its announce-layer representation is gone./-delimited, whole segments only), matching what the implementations actually do; the draft previously said byte-by-byte.moq-cluster
Mirrored as the WILDCARD_SUFFIX parameter (0x40B59, odd = length-prefixed) on PUBLISH_NAMESPACE and the extended NAMESPACE: the message's namespace is the pattern's prefix, the parameter the suffix. Same selection tiers, capacity-refusal re-resolution, and standby seed floor; ROUTE_COST never takes the actively-carrying discount on a pattern.
Endedwas added within the unreleased lite-06, so its changelog bullet is deleted rather than a removal logged; the wildcard, CAPACITY, and segment-aware changes get bullets. The doc-site table test learns the stream table's new row.just drafts checkandbun run --cwd doc checkboth pass.🤖 Generated with Claude Code
(written by Fable 5)