diff --git a/quest/m0/nightly/lite07-route-hang.md b/quest/m0/nightly/lite07-route-hang.md deleted file mode 100644 index 33f46ec00d6..00000000000 --- a/quest/m0/nightly/lite07-route-hang.md +++ /dev/null @@ -1,34 +0,0 @@ -# [M] lite-07 sessions stop hanging on a dropped token's route - -## Goal - -The lite-07 benches that hang on `main` complete: moq-net `auth/waiting/moq-lite-07-wip/1/1`, -moq-relay `session` lite-07 delivery credit, moq-relay `relay_auth_refresh/1/1`, -and the browser bench smoke ("control kept 8 broadcasts after concluding"). -The root cause is fixed, and a deterministic test (a kio simulation or a -model-checked case) fails without the fix. lite-03 to lite-06 are out of scope. - -## Plan - -Found by the triage of Nightly run -[44127032469](https://github.com/moq-dev/moq/actions/runs/44127032469) -(2026-11-17). Bisected over 7 steps and reproduced locally: caused by -[moq-dev/moq#8835](https://github.com/moq-dev/moq/pull/8835). In -`auth/waiting` under real tokio, the client adds a token, the server grants -everything, the client announces a hidden route, then drops the token. On the -13th repetition the server never hears the route. It is deterministic in the -bench, but a kio simulation of the same steps passes 60 rounds, so a lost -wakeup or a timing-dependent ordering in `rs/moq-net/src/lite/subscriber/route_stream.rs` -or `routed.rs` is likely; `Running::poll`'s permit, ask, and edit path around -the 13th grant is the first place to trace. The relay and browser hangs share -the range and the lite-07 routing and auth paths but were not bisected. - -Decided 2026-11-17 (maintainer, /quest-plan): - -- Goal: ✅ fix every bench above with one deterministic regression test - (rejected: only the bisected bench, with the others getting their own quests). -- Order: ✅ re-run the benches on - [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987)'s head first, - since it rewrites `route_stream.rs` and `routed.rs`. If they still hang, fix on - top of #8987 after it lands; if not, add only the regression test (rejected: - fix on `main` now, which #8987 would likely throw away). diff --git a/quest/m0/nightly/resume-history-control.md b/quest/m0/nightly/resume-history-control.md new file mode 100644 index 00000000000..d5cd87a68b1 --- /dev/null +++ b/quest/m0/nightly/resume-history-control.md @@ -0,0 +1,28 @@ +# [M] The browser resume-history control releases its last route + +## Goal + +The browser bench smoke's `resume-history` microbenchmark stops failing its control +intermittently on Nightly: once every front concludes and the serving route ends, +nothing of that route stays reachable. The cause is found and fixed at the root, with +a deterministic test that fails without the fix. + +## Plan + +Seen on Nightly runs +[43601796176](https://github.com/moq-dev/moq/actions/runs/43601796176) (2026-11-14, +"8 fronts, 1 names, 2 routes, churn, subscribe") and +[44127032469](https://github.com/moq-dev/moq/actions/runs/44127032469) (2026-11-17, +"8 fronts, 1 names, 2 routes, stable, fetch"), each with "the control kept +{"sessions":1,"transports":1,"broadcasts":8,"bytes":0} after concluding": the serving +route's client session, its transport, and its 8 broadcast handles. The 11-15 and +11-16 runs passed. Split out of the lite-07 route hang quest: the bench speaks +lite-06 (`ALPN_06`) and shares no cause with the lite-07 hangs. + +It did not reproduce locally: three runs of +`bun micro.ts --bench resume-history --rounds 4 current=../.. aa=../..` in Chrome, +about 36 full sweeps, all passed. `drain` in `js/net/bench/resume-history.ts` returns +once two consecutive collections count the same, so teardown still pending on a host +task or a real-time timer after `conclude` would read as kept on a slower runner. +Trace what still references the client session after `conclude` (a heap snapshot +from the failing case) before changing the harness. diff --git a/rs/moq-net/benches/session/mod.rs b/rs/moq-net/benches/session/mod.rs index e0d45ac8a42..774cb1cd46a 100644 --- a/rs/moq-net/benches/session/mod.rs +++ b/rs/moq-net/benches/session/mod.rs @@ -752,9 +752,10 @@ impl Cluster { let (origin, pool) = self.origin(CLIENT_CACHE); let mut options = MockConnectOptions::new(self.version); options.transport.stream_buffer = self.window; - // Draft 22, the one IETF version here, keeps its SETUP stream open beside the groups. - options.transport.max_concurrent_uni_streams = - self.credit.map(|credit| credit + usize::from(!self.version.is_lite())); + // Draft 22 keeps its SETUP stream open beside the groups, and moq-lite-07 the + // relay's Announce Stream: neither may take the one grant the groups share. + let persistent = !self.version.is_lite() || self.version.as_str() == "moq-lite-07-wip"; + options.transport.max_concurrent_uni_streams = self.credit.map(|credit| credit + usize::from(persistent)); options.client_discovers_all = self.discovers_all; // Hidden included, as the dash's `.stats/` scope sees its subtree. options.server_publish = Some(B::origin(&self.relays[relay]).consume().with_hidden(true)); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 2291b3cc62d..a985fd0e714 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -9330,6 +9330,70 @@ mod tests { ); } + /// A grant that widens to cover a claim the answer already held back sends it at once, + /// though the node queues nothing behind it to flush it out. + #[kio::test] + async fn a_widened_grant_flushes_the_claim_it_shows() { + use crate::model::ProduceTest as _; + const VERSION: Version = Version::Lite07; + let publish = origin::Config::new(std::num::NonZeroU64::new(2).unwrap()).produce(); + let route = publish.route("room", crate::origin::RouteOptions::hidden()).unwrap(); + route.announce().unwrap(); + // The peer asks for the whole tree, hidden paths included. + let request = |w: &mut crate::coding::Encoder<'_>| { + lite::AnnounceBroadcast::Request { + more: false, + remove: Vec::new(), + add: vec![(Path::empty(), true)], + } + .encode(w, VERSION) + .unwrap() + }; + let session = crate::lite::test_transport::ScriptedSession::new(announce_script(VERSION, &[&request])); + // Only the half answering the peer is limited, so a change to it is all that happens. + let auth = crate::auth::Link::new(false); + auth.handle().authorize(&crate::auth::Grant::default()); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: crate::time::sim(), + session: crate::driver::Transport::new(session.clone()), + origin: origin::Config::new(std::num::NonZeroU64::new(1).unwrap()).produce(), + recv_bandwidth: None, + version: VERSION, + peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Address::Setup), + cost: Some(1), + peer_node: None, + going_away: Default::default(), + auth: crate::auth::Link::new(false), + }); + subscriber.peer_setup.set(lite::Setup::default()).unwrap(); + let recv = crate::transport::boxed::RecvStream::new(session.recv()); + let _ = (subscriber.announce_streams).try_push(route_stream::Reader::new(recv, VERSION)); + let mut routes = route_stream::RouteStream::new( + subscriber.clone(), + route_stream::Publish { + origin: publish.consume(), + auth: auth.clone(), + withdrawal: Default::default(), + routing: false, + }, + ); + let written = || session.log.writes.lock().unwrap().len(); + // Run until everything parks: the request answered, the claim held back. + let settled = kio::clock::timeout( + std::time::Duration::from_millis(1), + kio::wait(|waiter| routes.poll(waiter).map(|err| panic!("{err:?}"))), + ) + .await; + assert!(settled.is_err(), "the session ended"); + let held = written(); + + // The grant change is the one wakeup: the pass it runs must send what it shows, + // since nothing else is queued to wake the session again. + auth.handle().authorize(&crate::auth::Grant::all()); + assert!(routes.poll(&kio::Waiter::noop()).is_pending()); + assert!(written() > held, "the claim the grant now shows stayed buffered"); + } + /// A peer streaming announces gets a budget of them read per poll, so the session's /// other tasks run between them, and the announce behind the flood still arrives. #[kio::test] diff --git a/rs/moq-net/src/lite/subscriber/route_stream.rs b/rs/moq-net/src/lite/subscriber/route_stream.rs index 2c60af64d54..e7db7897006 100644 --- a/rs/moq-net/src/lite/subscriber/route_stream.rs +++ b/rs/moq-net/src/lite/subscriber/route_stream.rs @@ -371,6 +371,11 @@ impl Running { match self.answer.poll_mounts(&mut self.writer, waiter) { Poll::Ready(res) => res?, Poll::Pending => { + // A grant change can buffer claims with nothing queued behind them, and the + // wakeup that brought it is spent: flush before resting. + if matches!(self.sending, Sending::Open | Sending::Ending) && !self.writer.is_flushed() { + continue; + } // Everything the node queued is served, so what the peer synced shows. self.routed.release(); break; diff --git a/rs/moq-relay/benches/auth.rs b/rs/moq-relay/benches/auth.rs index f7b2a04dc53..9caab6f436f 100644 --- a/rs/moq-relay/benches/auth.rs +++ b/rs/moq-relay/benches/auth.rs @@ -39,14 +39,11 @@ impl Fixture { let (send, mut receive) = tokio::sync::mpsc::unbounded_channel(); let decider = tokio::spawn(async move { while let Some(admission) = admissions.next().await { - let prefix = admission - .request - .query - .as_deref() - .unwrap_or("initial") - .strip_prefix("jwt=") - .unwrap_or("initial") - .to_owned(); + // An in-band token arrives as the request's token; the connection has none. + let prefix = match &admission.request.token { + Some(token) => String::from_utf8(token.value.clone()).unwrap(), + None => "initial".to_owned(), + }; let (producer, consumer) = lease::Producer::new(grant(&prefix)); admission.grant(consumer); if send.send((prefix, producer)).is_err() {