From e4408d217289aec1b8e0f8604d7d3ca14b0e44c5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 11:54:48 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m0/ci/mbx-failed-puts.md Quest-Claim: 2e66d5a9fe8a38da From c3311d7157e28763672635870ccdeec5b97f0a7a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 11:54:57 -0800 Subject: [PATCH 2/3] quest(ci): record the first night after the 20 GiB mbx budget, plan the prefetch-cap bump Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/mbx-failed-puts.md | 41 +++++++++++++++++++++++++++++++-- quest/m0/ci/mbx-prefetch-cap.md | 33 ++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 quest/m0/ci/mbx-prefetch-cap.md diff --git a/quest/m0/ci/mbx-failed-puts.md b/quest/m0/ci/mbx-failed-puts.md index 48b413cb425..d9788126bd7 100644 --- a/quest/m0/ci/mbx-failed-puts.md +++ b/quest/m0/ci/mbx-failed-puts.md @@ -62,6 +62,39 @@ night runs under it; note that beside the comparison: budget exceeded" or "cached action is missing blob" warnings, from its log. - Disk: no hosted job fails for space with the larger store. +Measured 2026-11-14 for scheduled Nightly +[43601796176](https://github.com/moq-dev/moq/actions/runs/43601796176), the +first night after #8631 and the first under the 7-day expiry, over the same +12-hour window as the baseline (all four writer workflows): + +| Night | PutObject 412 | PutObject 200 | GetObject 200 | GetObject 404 | +| ---------- | ------------- | ------------- | ------------- | ------------- | +| 2026-11-12 | 151,043 | 12,259 | 317,569 | 6,534 | +| 2026-11-14 | 86,387 | 44,906 | 429,428 | 17,783 | + +- Puts fell 20% (163,302 to 131,293) and 412s 43%. The new 200s and 404s + are most likely the 7-day expiry deleting what the night then re-stored. +- Hits rose in 24 of the 38 Rust-cache jobs, often doubling (doctest 165 + to 421, Rust benchmark tests 156 to 638), and misses fell in 29. Hits fell + in 12, mostly where hundreds of units were "not looked up" (features 590, + x11 648, moxygen 342, Rust bench targets 321): mbx found no prediction or + prior inputs for them, which the budget does not explain. Find why before + judging hit rates on the second night. The Windows `cargo check` job went + from 92 to 621 hits, as posted on + [jdx/mr-boxington#950](https://github.com/jdx/mr-boxington/discussions/950). +- 20 Linux jobs still downloaded the full 20 GiB, nearly all of it prefetch + (about 600 objects), and 25 still logged "cached action is missing blob" + (mbx stops at 128 per job). The budget still binds on Linux, so these jobs + keep re-uploading recompiled units. +- No hosted job failed for disk space with stores up to 21.7 GiB. +- Upstream merged + [jdx/mr-boxington#951](https://github.com/jdx/mr-boxington/pull/951) on + 2026-11-13, unreleased as of 2026-11-14: prefetch stops at half the budget + and a refused restore names the budget. It skips the HEAD-before-PUT check + until the 412 counts after it are posted. Its adoption is + [its own quest](/quest/m0/ci/mbx-prefetch-cap.md); raising the budget past + 20 GiB first was not tried, since no job records its free disk. + If 412s stay high once restores succeed, size the remaining sources (repeat stores of units mbx did not look up, and blobs every job's session sends once) and judge again whether cutting writers is worth the hits it would cost. Plan @@ -71,9 +104,13 @@ Public API: none. Wire: none. ## Waiting -- GitHub Actions: the first two scheduled Nightly runs after #8631 merges - supply the after numbers above. +- GitHub Actions: the 2026-11-15 scheduled Nightly, the second after #8631, + confirms the 2026-11-14 numbers above. +- jdx/mr-boxington: a release containing #951, which + [mbx-prefetch-cap](/quest/m0/ci/mbx-prefetch-cap.md) adopts before the + remaining 412s are judged. ## Related - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - its Windows misses may share this cause, since Windows also reached its 10 GiB budget +- [Adopt mbx's prefetch cap](/quest/m0/ci/mbx-prefetch-cap.md) - the upstream fix for the budget that still binds on Linux diff --git a/quest/m0/ci/mbx-prefetch-cap.md b/quest/m0/ci/mbx-prefetch-cap.md new file mode 100644 index 00000000000..6a0d9962686 --- /dev/null +++ b/quest/m0/ci/mbx-prefetch-cap.md @@ -0,0 +1,33 @@ +# [S] Bump mbx to the release that caps prefetch + +## Goal + +The Rust cache runs the first mbx release containing +[jdx/mr-boxington#951](https://github.com/jdx/mr-boxington/pull/951), so +prefetch no longer spends the download budget a job's own restores need. On +the first scheduled Nightly after the bump, no hosted Rust-cache job logs +"cached action is missing blob", each job keeps or raises its hits, and +`mbx-moq-dev` takes fewer PutObject 412s than on 2026-11-14 (86,387). + +## Plan + +Bump `version:` in `.github/actions/rust-cache/action.yml` (and the action's +pin if the release needs one), keeping `MBX_GC_MAX_SIZE=20GiB`, which upstream +recommends until then. On 2026-11-14, 20 Linux jobs still spent the full +20 GiB, almost all on prefetch, so a half-budget cap leaves 10 GiB for their +restores; check that budget refusals, now named as such, are gone. + +Measure the night as [mbx-failed-puts](/quest/m0/ci/mbx-failed-puts.md) +does and post the PutObject and 412 counts on +[jdx/mr-boxington#950](https://github.com/jdx/mr-boxington/discussions/950), +as upstream asked, after re-reading its CONTRIBUTING.md and AI policy. + +Public API: none. Wire: none. + +## Waiting + +- jdx/mr-boxington: a release containing #951 (`gh release list -R jdx/mr-boxington`). + +## Related + +- [Failed puts](/quest/m0/ci/mbx-failed-puts.md) - judges the 412s left after this From 2b797b4a13eb3af53f33f08ff1229d60c96a7d69 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 12:17:17 -0800 Subject: [PATCH 3/3] quest(ci): record the maintainer's budget and follow-up decisions Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/mbx-failed-puts.md | 5 +++++ quest/m0/ci/mbx-prefetch-cap.md | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/quest/m0/ci/mbx-failed-puts.md b/quest/m0/ci/mbx-failed-puts.md index d9788126bd7..9ac6c453a62 100644 --- a/quest/m0/ci/mbx-failed-puts.md +++ b/quest/m0/ci/mbx-failed-puts.md @@ -95,6 +95,11 @@ first night after #8631 and the first under the 7-day expiry, over the same [its own quest](/quest/m0/ci/mbx-prefetch-cap.md); raising the budget past 20 GiB first was not tried, since no job records its free disk. +Decided 2026-11-14 (maintainer, on #8763): ✅ keep the hosted budget at +20 GiB until #951 ships (rejected: raise it now). ✅ Keep +[mbx-prefetch-cap](/quest/m0/ci/mbx-prefetch-cap.md) in m0 (rejected: move it +to m4, drop it). + If 412s stay high once restores succeed, size the remaining sources (repeat stores of units mbx did not look up, and blobs every job's session sends once) and judge again whether cutting writers is worth the hits it would cost. Plan diff --git a/quest/m0/ci/mbx-prefetch-cap.md b/quest/m0/ci/mbx-prefetch-cap.md index 6a0d9962686..cf13304ac34 100644 --- a/quest/m0/ci/mbx-prefetch-cap.md +++ b/quest/m0/ci/mbx-prefetch-cap.md @@ -13,7 +13,8 @@ the first scheduled Nightly after the bump, no hosted Rust-cache job logs Bump `version:` in `.github/actions/rust-cache/action.yml` (and the action's pin if the release needs one), keeping `MBX_GC_MAX_SIZE=20GiB`, which upstream -recommends until then. On 2026-11-14, 20 Linux jobs still spent the full +recommends until then (maintainer, 2026-11-14 on #8763: keep it at +20 GiB, rejected raising it now; keep this quest in m0, rejected m4 or dropping it). On 2026-11-14, 20 Linux jobs still spent the full 20 GiB, almost all on prefetch, so a half-budget cap leaves 10 GiB for their restores; check that budget refusals, now named as such, are gone.