This page collects the measured numbers for GenClass Runtime (@genclass/runtime) and its model
(@genclass/runtime-model): model quality, false interventions, detection rates, demo and real-app runs, the
compatibility matrix, first-load size, browser cost and the measured visual checks. The READMEs and the model card
quote these numbers; the raw reports they come from are listed below. The current releases are runtime 0.2.2 (npm
latest) and 0.3.0-beta.0 (beta) with model 0.2.0, but the measurements span runtimes from before 0.1.0-alpha.1
to 0.3.0-beta.0, release branches and unreleased work on main. Measured on 0.2.2 or on the branches that became
0.3.0-beta.0: the CI first-load and telemetry-volume numbers (§8 to §11), the dashboard counts (§12) and the visual
checks (§13, §14). The demo, real-app, compatibility-matrix and browser-cost numbers come from earlier runtimes and
were not re-run; the model numbers are held-out evaluations of each model, with model 0.2.0 the current one. Each
section says what it measured and on which model, runtime version or branch.
Updated 2026-10-22 (the summary below and sections 16 and 17 added, and the numbers that were recorded only in the
READMEs and the model card added to sections 1, 2 and 8; the newest measurement is section 15, 2026-10-18). Model
numbers are measured on held-out data. Every recall or fix rate is reported next to its
false-intervention rate (FIR): the share of cases where doing nothing was best but the gate fired anyway. The raw
reports are in training/EVAL.md, sim/SEPARABILITY.md, realapps/README.md, demos/results.md and
compat/RESULTS.md.
The headline numbers for the current releases, each checked against the section it names. On npm (dist-tags on
2026-10-22): @genclass/runtime 0.2.2 (latest) and 0.3.0-beta.0 (beta), with the model
@genclass/runtime-model@0.2.0 (training run r17-v2dT) at its default profile, balanced. Rows further down for
round 1, r17-v2a to r17-v2c and model 0.1.0 (r17-v2b) are superseded and kept as the record.
| measure | number | section |
|---|---|---|
| Observe (the default): problems detected with the right diagnosis, held-out real apps | likely tier (report gate 0.90 at balanced; what rt.on("detect"), report listeners and the console's default level get, and all that 0.2.2 reports): duplicate submits 75%, stale overwrites 30% (situation-v2.3 slice; 21% on situation-v2), genuine broken state 6%. 0.3.0-beta.0's observe default (detect: "possible") also records possible detections, at the eager report gate 0.70, to console.debug, the devtools and telemetry (tier: "possible"): likely and possible together 77% / 45% (40% on situation-v2) / 13%, the eager profile's measured rates, not measured end to end on 0.3.0 |
§1, §11 |
| Observe: false detections | likely tier: the 0.2.0 fit held them to at most 2% (real apps) and 3% (simulated) of clean decisions, met on held-out test; the exact rate is not published. Possible tier (the 0.3.0 observe default): at most 5% / 6% for likely and possible together; the share of possible detections that are false is not measured. Compatibility matrix (likely tier, balanced): 49 false findings in 880 runs on correct apps. Bench demos (observe, model 0.2.0): 2 of 30 clean trials had a detection at balanced, 3 of 29 at eager |
§11, §16 |
| Guard (opt-in): false-intervention rate (FIR) | 0.13% of held-out simulated cases where doing nothing was best; 0.00% on held-out real apps. cautious 0.005%, eager 0.54% |
§1 |
| Guard: recall, at the FIR above | acts on 7.8% of clear simulated cases and 7.5% of actionable real-app cases | §1 |
| Heal (experimental): FIR and recall | FIR 0.59% simulated, 0.00% real apps; acts on 7.1% of clear simulated and 10.4% of actionable real-app cases | §1 |
| Project targets | guard FIR ≤ 0.1% and heal FIR ≤ 0.5%: cautious meets both, balanced is slightly over both (it was fitted to guard ≤ 0.3% and heal ≤ 1%, and meets those), eager is well over |
§1 |
Six demos under network chaos (model 0.2.0, runtime 0.1.0-beta.4, balanced) |
bug rate 72% without GenClass, 70% observe, 72% guard, 73% heal: no mode lowered it (differences under about 5 points are noise); 0 false interventions in 30 clean trials each in guard and heal | §5 |
| Correct apps left unchanged (always-passive model in heal mode, compared with observe mode, not with running without GenClass; runtime as of 0.1.0-alpha.1, not re-run since) | 0 of 396 clean runs changed (66 real apps × 6 seeds); with network chaos, 3 of 198 | §4 |
Compatibility matrix (7 apps: 6 frameworks and plain HTML; 15 data layers, 8 scenarios, 10 seeds, every mode; 2026-10-10, runtime commit d04eff3: the 0.1.0-beta.4 code plus the "Model ready" log fix; 0.2.0's audit trail, protect presets and hold-budget ceilings and 0.2.1's first-request hold were not run through the matrix; it was not re-run on 0.2.2 or 0.3.0-beta.0) |
0 bugs introduced in 3,510 runs; 0 actions on correct apps in 1,760 guard and heal runs; 10 double submits prevented, all by heal block on one data layer |
§16 |
| Browsers measured | Chromium only (the demos, the compatibility matrix, the visual checks); Firefox and Safari are not measured | §5, §13, §14, §16 |
| First-load JavaScript, minified and gzipped | 0.3.0-beta.0 (the published build, tag v0.3.0-beta.0): 108.25 KB for /auto, 100.13 KB for the main entry, 128.56 KB for the script tag. 0.2.2: 99.17 / 91.70 KB. With the interaction-analytics stub (branch feat/interaction-analytics, train VM, 2026-10-18, since merged into main and released in 0.3.0-beta.1; the later builds' sizes are in the limit rows further down): 108.529 / 100.395 KB |
§9, §10, §15 |
| Model files and speed | q8 10.16 MB (WASM), fp16 13.57 MB (WebGPU with shader-f16); 8.8M parameters; about 182 / 330 / 599 ms per pass at 500 / 780 / 1,170 tokens on one WASM thread (a least-squares fit over runs of onnxruntime-web 1.30 in Node on the training VM, not in a browser; training/ortweb/validate.mjs) |
§2 |
| Browser cost on a pilot app (0.1.0-beta.3, model 0.2.0, 4× CPU throttling) | +190 to 280 MB renderer memory while the model is loaded; 12.7 MB downloaded on a first visit on WASM (the READMEs say about 13 MB); no measurable change to the load event, LCP or long tasks | §17 |
Telemetry volume, one tab at the model's maximum decision rate for an hour (scripted decider, observe mode, situation text off as by default; release review commit 9d567a0, not the published build) |
251.9 KB in 165 requests when every decision is routine; 2,068.5 KB in 361 requests when every decision is a detection | §10 |
| Measured visual checks (deterministic, not the model) | fixture pages: 35 of 35 planted faults found, 0 false reports. Bench: 0 false reports and 0 fixes in 128 page views of the demos, templates and compat apps; recall 1.00 on 20 of 23 planted fault kinds (run 4), 19 of 23 (run 5); public sites: 8 findings in 96 loads in run 4 (5 confirmed real, 0 check errors, 3 unconfirmed), 6 findings in 97 loads in run 5 (after the review; not looked at) | §13, §14 |
| Flow expectations (opt-in) | false-report rate not measured yet | §5 |
Coding agents: genclass_explore and genclass_replay on one real page (train VM, headless Chromium, 2026-10-20; one page, one bug class, not a recall figure) |
explore found the stale-results race in 5 of 6 reordered runs of the buggy page and reported nothing on the fixed page (0 of 6 differ: no false report); replay reproduced it before the fix and not after; the generated Playwright test failed before and passed after | §18 |
| Stage | Data | Diagnosis | Action | Guard FIR | Heal FIR | Recall on clear cases (see the note below) |
|---|---|---|---|---|---|---|
| GenClass 0.1 (no runtime training) | none | 8.0% | 37.6% | ~0% | 0.0–0.33% | ~0% |
| Stage 1: curriculum only, R17 (tested on simulated apps) | 1.2M curriculum rows | 44% | 48% | 9.7% | 18.4% | n/a (over-intervenes) |
| Stage 2 pilot, R17 | 224k pre-freeze sim rows, mushy labels | 91.1% | 78.3% | 0.00% | 0.08% | ~1% (heal recall on every row where acting was best, not clear cases only: 1.4%) |
| Final round 1, R17 (format v1) | 448k sim rows (phase A) | 90.5% | 81.9% | 0.05% | 0.24% | 7.7% |
| Final round 1, R32 (format v1) | same | 89.7% | 81.8% | 0.05% | 0.22% | 6.8% |
| r17-v2a (format v2, first round) | 2B tokens of v2 sim gold + v2 curriculum | 84.4% | 77.9% | 0.00% | 0.46% | heal 11.3% (all clear cases, fixed gates; training/EVAL.md, sim2e) |
| r17-v2b (its gates were refit at 10:30 and that refit shipped as model 0.1.0: guard FIR 0.01%, heal 0.07%, "Model 0.2.0" table) | from r17-v2a, +1B tokens: sim v2 gold 70%, real-app gold 18% (eval-set rows left out), curriculum replay 12% | sim 84.2% / real 83.6% | sim 77.8% / real 80.0% | 0.02% sim, 0.00% real (first derived gates) | 0.23% sim, 0.00% real (first derived gates) | heal 6.0% clear, real-app action cases 9.5% (derived gates; training/EVAL.md); at the fixed heal gate 0.8 (heal FIR 0.50% sim2e, 0.31% real2e, 0.00% on the eval set's clean rows): real duplicate 8.2% (14% on held-out apps) |
| 150M teacher (t150-v2a) | 1B tokens of v2 sim gold and real-app gold | sim 81.8% / real 82.0% | sim 76.5% / real 78.1% | – | – | lower than r17-v2b everywhere, so distillation was skipped |
| r17-v2c (its q8 export, in the head-to-head for model 0.2.0) | + on-policy round a, v2.2 retry labels (training/configs/mix_v2c.json) |
sim 85.2% / real 84.0% | sim 77.8% / real 79.9% | 0.016% sim, 0.00% real | 0.21% sim, 0.00% real | guard 1.8% clear; heal 6.0% clear, real-app action cases 9.8% |
r17-v2dT (gain gate), shipped as @genclass/runtime-model@0.2.0, profile balanced |
from r17-v2c, 1.5B tokens in all (the same data and schedule as v2d): sim v2 gold (sim2r), the situation-v2.3 sim gold, on-policy round a, real-app gold (v2c plus the v2d and v2.3 top-ups), curriculum replay (8% of the mix, training/configs/mix_v2dT.json), both eval sets left out; action labels are gain distributions (softmax(gain/τ = 1)) |
sim 85.4% / real 85.5% | sim 73.9% / real 77.8% (gain labels: not comparable with the rows above) | 0.13% sim, 0.00% real | 0.59% sim, 0.00% real | all clear cases: guard 7.8% (real-app action cases 7.5%); heal 7.1% (real 10.4%); all three profiles in "Model 0.2.0" below |
Recall column: the share of clear cases (the gold label puts at least 0.9 on one permitted action other than doing
nothing) on which the gate acted. The two final round 1 rows count the clear stale and duplicate cases only (R17 7.7%
of 220 rows; on all clear cases R17 acted on 4.2% in guard and 6.1% in heal, R32 on 3.5% and 5.9%). From r17-v2b on,
the cells count all clear cases in the mode named, with real-app recall where given (training/EVAL.md). r17-v2a's
cell (heal, 11.3%) counts all clear cases at the fixed gates of its row. The r17-v2c row is the q8
export measured in the head-to-head (training/EVAL.md, "Head-to-head for @genclass/runtime-model@0.2.0"), with
gates fitted on dev at 0.8 × each limit; an earlier PyTorch evaluation of r17-v2c, recorded only in commit f9509b8
without its report, is not used here.
| Set | Diagnosis | Action | Guard FIR | Heal FIR | Recall at shipping gates | Recall at gate 0.5 |
|---|---|---|---|---|---|---|
Sim test (sim2e) |
84.4% | 77.9% | 0.00% | 0.46% | heal 6.8% | heal 45% on clear cases (harm 1.8% of rows) |
Held-out app features (sim2f) |
81.0% | 77.4% | 0.00% | 0.75% | heal 6.8% | – |
| Real apps, eval set (16,600) | – | – | 0.00% | 0.00% | duplicate 0.6%, stale 1.0% | duplicate 35% (64% on held-out apps), broken state 10%, stale 9%, with FIR 0.05–0.35% |
Reading. v2 kept v1's safety and made the model much better at ranking, but the fixed 0.9/0.8 gates were chosen before labels became expected costs and now block nearly every action. The next steps planned then:
- Thresholds: derive per-tier, per-trigger gate thresholds from held-out data (FIR ≤ 0.1% guard / 0.5% heal on clean real-app and sim traffic; fit on dev, verify on test) and ship them in the model's meta.json.
- Training: finish the 150M teacher (with real-app gold), distil, and run DAgger rounds.
What happened: the derived gates below; the teacher came out weaker than r17-v2b, so nothing was distilled; on-policy (DAgger) rounds a and b; then r17-v2c, r17-v2d and r17-v2dT (§7).
The thresholds are fitted per tier and trigger on held-out dev data, under these constraints: FIR ≤ 0.1% (guard)
or ≤ 0.5% (heal), and harm ≤ 0.2% or ≤ 1%. Each constraint must hold at the 95% upper confidence bound. The
thresholds ship in the model's meta.json gate. Verified on held-out test (95% bootstrap intervals):
| Policy | Fires | FIR sim | FIR real apps | Harm (sim / real) | Recall on clear cases | Gain captured |
|---|---|---|---|---|---|---|
| guard, derived gates (0.70–0.90) | 0.14% | 0.03% [0.00, 0.08] | 0.00% | 0.01% / 0.00% | 2.5% | 3.0% |
| guard, fixed 0.9 | – | 0.00% | 0.00% | – | 0.2% | 0.04% |
| heal, derived gates | 0.99% | 0.19% [0.13, 0.25] | 0.00% | 0.05% / 0.04% | 5.2% | 5.1% |
| heal, fixed 0.8 | – | 0.66% [0.56, 0.77] (over the limit) | – | – | 12.0% | 9.3% |
The derived gates keep both tiers inside their safety limits on data never used for fitting. Recall is now limited by the model's discrimination, which is what the 150M teacher, distillation and DAgger rounds target.
Model 0.1.0, superseded: model 0.2.0's detection rates are in the "Model 0.2.0" table below and in §11.
A detection is GenClass reporting a problem, such as "this response overwrote newer state", without changing
anything. This is the phase-1 product. Detections fire when the top diagnosis is not expected and its probability
is ≥ gate.report.
| Held-out test set | Report threshold | Detected | Precision | False detections on clean real-app traffic |
|---|---|---|---|---|
| Real apps never trained on (eval set) | 0.70 | duplicate submits 76%, stale overwrites 43%, broken state 27% | 0.96 | 0.62% |
| Real apps never trained on (eval set) | 0.85 (shipped in model 0.1.0) | duplicate ~76%, stale 28% | higher | 0.36% |
| Simulated apps (sim2e) | 0.70 | 50.9% of anomalies | 0.91 | gold-expected rows: 2.59% (over the 2% limit) |
| Simulated apps (sim2e) | 0.85 | – | – | 1.11% |
Model 0.1.0's card (MODEL_CARD.md at 1f0f617,
tag runtime-model-v0.1.0, "Observe mode: detections at report 0.85") also counts false detections on every held-out
test decision whose correct diagnosis was expected: 1.41% simulated (149 / 10,582: sim2e 1.11%, sim2f 1.72%)
and 3.78% real apps (305 / 8,079). On the real2e test slice, 240 of its 303 false detections came from one held-out
app, flagged as stale on its article.inProgress field; without that app the slice is 1.05% (63 / 6,006). The same
card: 65.0% (simulated) and 60.7% (real) of problem decisions were flagged, and 94.1% and 89.5% of flags had the right
diagnosis; that one app's 240 false detections were 25% of its 955 clean decisions; at report 0.9, 0.89% (simulated)
and 3.44% (real) of clean decisions were flagged, with 58% and 52% of problems flagged.
Per-class precision / recall at 0.70 on sim2e:
| Class | Precision | Recall |
|---|---|---|
| failing | 0.95 | 0.78 |
| slow | 0.94 | 0.91 |
| transient | 0.88 | 0.81 |
| stale | 0.87 | 0.57 |
| duplicate | 0.80 | 0.49 |
| inconsistent | 0.97 | 0.53 |
| conflict | 0.89 | 0.54 |
In r17-v2b's first fit, 0.85 was chosen after the dev-fitted 0.70 failed the simulated false-detection limit on
test, so that choice was test-informed. In the refit published as model 0.1.0, 0.85 is the dev fit itself; that
refit's one test-informed choice is the heal failure gate, 0.95 (training/EVAL.md). On the shipped model's own
traffic (on-policy round b test) the refit's 0.85 gave 3.1% false detections on simulated gold-expected rows, over
the 2% limit. The refit and the 0.2.0 head-to-head fitted at 0.8 × each limit on dev; model 0.2.0's final fit
used the limits themselves, with the real-app certification set added to dev ("Model 0.2.0" below).
| Gate | Acts | False interventions among acts | Harmful (≥ +1 cost) | Misses (clear action skipped) |
|---|---|---|---|---|
| shipping (0.9 / 0.8) | 2.6% of decisions | 12% | 366 | 43,662 (9.7%), mostly request and failure |
| exploratory 0.5 | 21.5% | 36% | 15,054 | 29,913 |
Where the false interventions come from:
coalesceof deliberate repeated actions;retryandserve_cachedduring real outages;discardwhere the response wasn't actually stale;rollback/resyncon coincidental relations.
The worst retry harm (POSTs that had already committed) is now prevented by the v2.2 rule: retry only when the
request is idempotent or carries an idempotency key. These rows train the next student, up-weighted.
| Gate | Acts | False among acts | Harmful | Misses |
|---|---|---|---|---|
| shipping (the model's own gates) | 1.9% | 11% | 201 (0.09% of rows) | 10.9% |
| exploratory 0.5 | 22% | 36% | 14,920 | – |
Compared with round a:
- failure is much cleaner: 6% of acts false, and 8 false retries vs 198, thanks to the v2.2 retry rule;
- delivery now acts: 99 acts, 18% false.
Remaining issues:
- the heal transition gate (0.55) is too loose on-policy: 41% of its acts are false. Model 0.2.0's gates were refitted with on-policy dev data, falling back to the tier default where evidence is thin ("Model 0.2.0" below);
- request acts are 19% false, mostly coalescing deliberate repeated clicks.
Shipped (dev-fitted) operating points, held-out test:
| v2c | v2d (mass gate) | v2dT (gain gate) | |
|---|---|---|---|
| guard FIR (sim) | 0.016% | 0.006% | 0.003% |
| heal FIR (sim) | 0.21% | 0.07% | 0.105% |
| heal gain captured | 5.2% | 4.4% | 4.4% |
| heal real-app action recall | 9.8% | 5.5% | 0.7% |
| FIR on real apps | 0.00% | 0.00% | 0.00% |
At equal safety (one global threshold swept on the same test records), v2dT captures the most:
| heal gain at FIR ≤ 0.05 / 0.1 / 0.2 / 0.5% | |
|---|---|
| v2dT | 2.1 / 3.4 / 6.2 / 12.3% |
| v2d | 1.6 / 2.5 / 5.3 / 10.0% |
| v2c | 1.6 / 2.8 / 4.6 / 8.3% |
v2dT also leads guard-mode real-app recall at FIR ≤ 0.5%: 14.1%, against 3.1–3.4%.
Why v2dT ships conservatively. There are too few clean real-app dev rows to certify per-trigger margins, so the
fit falls back to strict defaults. The same thin evidence pushes gate.report up to 0.97–0.99. The next step, a
large real-app certification dev set (dev apps only), a refit of all three and a choice of 0.2.0 by its shipped
operating point, is "Model 0.2.0" below.
Quantization. The q8 export was fixed by moving to MatMulNBits block 16: 223/223 decisions match PyTorch, the file is 10.16 MB, and it is fp16-free.
Gates are fitted on the shipped q8 outputs, using the real-app certification set (counted per trajectory) and shipping-gate on-policy rows. Results on held-out test:
| Profile | guard FIR | guard harm | guard recall (clear / real) | guard gain | heal FIR | heal harm | heal recall (clear / real) | heal gain | report | detected (duplicate / stale / broken) |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.1.0 (v2b) | 0.01% | 0.00% | 0.6% / 0% | 0.8% | 0.07% | 0.04% | – / 5.7% | 3.8% | 0.85 | 76% / 28% / – |
| 0.2.0 cautious | 0.005% | 0.007% | 2.4% / 1.4% | 3.1% | 0.26% | 0.06% | 3.7% / 3.8% | 3.2% | 0.95 | 72% / 28% / 0% |
| 0.2.0 balanced (default) | 0.13% | 0.03% | 7.8% / 7.5% | 6.7% | 0.59% | 0.11% | 7.1% / 10.4% | 5.7% | 0.90 | 75% / 30% / 6% |
| 0.2.0 eager | 0.54% | 0.11% | 14.2% / 21.0% | 12.5% | 1.84% | 0.29% | 17.1% / 24.2% | 14.2% | 0.70 | 77% / 45% / 13% |
The 0.1.0 row comes from the earlier, stricter fit without the certification set, so it is not exactly like-for-like. FIR on real apps is 0.00% for every profile.
Why v2dT: v2d is better in heal mode (gain 8.1% vs 5.7%, real recall 21.8% vs 10.4%), and v2dT is better in guard mode (6.7% vs 4.2%). v2dT ships because guard is the first mode that acts (observe only reports). When 0.2.0 was chosen, guard was the default of the branch it shipped from (runtime 0.1.0-beta.1); observe has been the default again since 0.1.0-beta.2 (CHANGELOG). Choosing the model per mode is an open item.
Targets: guard FIR ≤ 0.1% and heal FIR ≤ 0.5%. cautious meets both (0.005% / 0.26%); balanced, the default, is
slightly over both (0.13% / 0.59%); eager is well over (0.54% / 1.84%). FIR on held-out real apps is 0.00% for every
profile. Each profile was fitted to limits of its own, and all three meet them on held-out test: cautious guard
FIR ≤ 0.1%, heal ≤ 0.5%, report false ≤ 1% real / 2% simulated; balanced guard ≤ 0.3%, heal ≤ 1%, report ≤ 2% / 3%;
eager guard ≤ 1%, heal ≤ 3%, report ≤ 5% / 6% (training/EVAL.md). Calibration error ≤ 0.02 (not yet measured
for 0.2.0; r17-v2b: 0.021–0.023), diagnosis ≥ 95% (not yet), clear-case recall ≥ 80% (not yet).
The stage-1 row comes from the 200 sim sample rows (training/EVAL.md). On its own curriculum test set, stage 1
scores 98% / 98% with 0% FIR.
| Model | q8 size | WASM, 1 thread, ms at 500 / 780 / 1,170 tokens | Final round 1 diagnosis / action / guard FIR |
|---|---|---|---|
| R17 (ettin-17m, 16k pruned vocab) | 9.58 MB | 177 / 323 / 589 | 90.5% / 81.9% / 0.05% |
| R32 (GenClass 0.1 / ettin-32m, pruned) | 22.5 MB | 569 / 954 / 1,580 | 89.7% / 81.8% / 0.05% |
| GenClass 0.1 (original) | 57 MB | ≈ 900 at 780 | not trained for this |
Decision: R17 is the default on every device. It matches R32 at under half the size and about a third of the
latency. Both q8 exports are fp16-free, so they run on WebGPU without shader-f16, and agree with PyTorch on 223/223
decisions in onnxruntime-web.
Sources: the R17 row is the delivered round-1 export in training/EVAL.md. The R32 latencies are the delivered
round-1 R32 export, measured the same way and recorded in the training log (training/LOG.md at commit fcd1e68,
taken out of the public tree in 66c0f03); training/EVAL.md has only the earlier stage-1c R32 export, at
499 / 879 / 1,539 ms. Against either, R17 takes 31% to 38% of R32's time.
The shipped export (@genclass/runtime-model@0.2.0, r17-v2dT; packages/runtime-model/MODEL_CARD.md,
training/EVAL.md): q8 10,160,186 bytes (MatMulNBits 8-bit, block 16, which restored parity with PyTorch on 223/223
decisions) and fp16 13,569,879 bytes; 8.8M parameters (8.1M encoder + 0.7M heads after vocabulary pruning); about
182 / 330 / 599 ms at 500 / 780 / 1,170 tokens on one WASM thread (a least-squares fit over runs of onnxruntime-web
1.30 in Node on the training VM, not in a browser; training/ortweb/validate.mjs). The 9.58 MB size and the
latencies in the R17 row above are the round-1 export, obtained the same way, at MatMulNBits block 32, the q8
format training/README.md describes and still the default of training/export_runtime.py (--block 32); the 0.2.0
exports were made with block 16 (training/EVAL.md).
Other latencies. On WebGPU, on one Apple GPU with model 0.2.0, a warm-up pass on a situation of about 430 tokens
took 49–74 ms and the first real decision 137 ms (packages/runtime/CHANGELOG.md, 0.2.2; the model card); there is
no systematic WebGPU benchmark. Four WASM threads on a crossOriginIsolated page were about 3× faster than one with
GenClass 0.1's 32M-parameter q8 model, in headless Chromium on the shared train VM: 271–360 ms against 837–981 ms at
780 sequence tokens, and 464–556 ms against 1,452–1,791 ms at 1,174 (packages/runtime/src/model/README.md); threads
have not been re-measured with model 0.2.0.
From sim/SEPARABILITY.md:
- A linear model on the visible situation text reaches the same recall ceiling as R17, so the limit is information, not capacity.
- 62–82% of clear actionable rows had a benign twin with an identical fact set.
- 24% of clear rows were mislabelled
expected, which the gate never acts on. Fixed by S1. - Labels were hindsight-certain about things a runtime cannot observe: the user's next action, outage length, and whether a failed write committed. Fixed by S2, so labels are now expected cost given what is observable.
Format v2 adds measured facts (would put back a replaced value, overwrites newer typing, response changes nothing, known-stale values, cadence, failure scope, repeat-click evidence, …). Its effect on visible-text separability (100k rows, v1 → v2):
| Trigger | Benign look-alike rate | Linear recall at 1% FIR |
|---|---|---|
| failure | 40% → 17% | 6% → 11% |
| request | 48% → 38% | 11% → 15% |
| stall | 31% → 5% | – |
| mutation | 8% → 2% | – |
| inconsistency | – | 10% → 20% (AUC 0.73 → 0.81) |
GenClass in heal mode with an always-passive model, compared against observe mode (v1: realapps/README.md, Pilot,
"Interference: 4 of 198 clean runs changed"; v2: packages/runtime/CHANGELOG.md, 0.1.0-alpha.1):
| Runtime | Real apps (66), clean runs changed | Conduit (React/Redux RealWorld) | Demos (guard, zero actions) |
|---|---|---|---|
| v1 (store-write holds) | 4/198 (2 apps; 66 apps × 3 seeds) | never renders its home feed | board 11 card jump-backs, editor 37 overwritten keystrokes, checkout 24 reset quantities, search clean latency 6 → 389 ms |
| v2 (network-boundary decisions) | 0/396 (66 apps × 6 seeds) | identical, 0/30 | holds only at delivery; clean typeahead makes 0 model calls and holds for 0 ms |
Chaos runs under v2: 3/198 changed. A request held about 25 ms drew a different simulated network outcome (open
item). The v2 runs used the runtime as of 0.1.0-alpha.1 (2026-10-08) and were not re-run on later versions. They
compare final page text (inputs and alerts excluded) and server state, not request timing or store contents; the
baseline is observe mode, not running without GenClass. realapps' own sweeps on the same situation-v2 runtime (tag
6e5e86e), before its production run, found 0 of 132 and 0 of 256 clean runs changed (realapps/README.md,
"Production on situation-v2"). §16 has the later compatibility matrix, which compares every mode with no GenClass.
10 chaos + 5 clean seeds per demo and mode, Playwright, telemetry off, model served locally (WASM, 4 threads). Chaos bug rate over the six demos (60 trials per cell); FI = false interventions on 30 clean trials.
| configuration | off | observe | guard | heal | actions guard / heal (chaos) | FI guard / heal |
|---|---|---|---|---|---|---|
| beta.4, balanced (default) | 72% | 70% | 72% | 73% | 2 / 3 | 0 / 0 |
| beta.4, cautious | – | – | 72% | 77% | 0 / 0 | 0 / 0 |
| beta.4, eager | – | 72% | 73% | 75% | 6 / 12 | 2 / 0 |
| beta.4, margin 1 for every trigger | – | – | 70% | 77% | 20 / 39 | 3 / 3 |
| A/B, beta.4 arm (simultaneous) | 72% | 72% | 70% | 70% | 3 / 1 | 0 / 0 |
| A/B, heal/overnight arm (47f292c) | 73% | 70% | 73% | 72% | 1 / 1 | 0 / 0 |
Nothing is healed at the shipped gates: candidate gains are rarely above the margins (median ≈ 0; 3–8% of discard/retry candidates reach 4). Differences under ~5 points are noise (cautious, with no action at all, differs from balanced by 4 points in heal). A pilot app (a production Next.js ordering site, 9 fault scenarios): the only bug, a duplicate order after a lost-commit 502 and a re-tap, occurs in every mode; 0 actions, 0 false interventions.
Automatic state discovery (2026-10-10, local, bench/heal/, branch feat/one-line, model 0.2.0, balanced)
Demos, stores registered by hand vs found by discovery (GENCLASS_DISCOVER=1 build: useGenClassState is plain
useState, the Redux enhancer is the Redux DevTools one, the Zustand middleware is devtools(); 5 chaos + 3 clean
seeds per demo and mode; small samples):
| demo | stores, manual | stores, discovered | decisions observe / guard, manual | discovered | detections observe / guard, manual | discovered | chaos bugs (off / observe / guard) manual | discovered |
|---|---|---|---|---|---|---|---|---|
| checkout (React) | cart, orders atoms |
one observed React store (both states) | 23 / 20 | 24 / 21 | 9 / 10 | 7 / 8 | 4 / 4 / 4 of 5 | 4 / 4 / 4 of 5 |
| board (Zustand + React) | board adapter |
observed React useSyncExternalStore (Zustand devtools is off in a Vite production build) |
76 / 73 | 72 / 73 | 14 / 14 | 6 / 6 | 4 / 3 / 4 | 4 / 3 / 4 |
| editor (Redux) | notes adapter |
notes adapter (discovered, controllable) |
50 / 53 | 47 / 50 | 21 / 20 | 23 / 18 | 5 / 5 / 5 | 5 / 5 / 5 |
Delivery decisions with discovery: checkout 2 / 2, board 32 / 33, editor 19 / 13 (manual 2 / 1, 31 / 26, 17 / 17). No actions ran in either build (as in the night report at these gates); no clean-run bugs.
The pilot app (only the one line, React state): stores found orderPage, addToOrder (named after rendered elements:
production names are minified) and er (minified MenuBrowser). The nine existing scenarios: decisions unchanged
(12 per mode over 27 trials: failure 9, request 3; detections 9), since each of its page loads sees at most one response
per request kind that changes state. New scenario order-poll-reorder-2 (another device removes a line, the guest
removes the other while the next ticket poll is answered 3 s late): 3 / 3 delivery decisions with discovery, 0 / 3
with autoState: false, all expected (correct: the app drops stale polls; no bug in any mode). Overhead: §17
(+7 KB first-load JS, walk p95 under 2 ms at 4× throttling, +0.5 MB heap, no new long tasks).
The layer ships off by default in 0.3.0 (owner decision, 2026-10-13) until the gate below has run; its first-load cost with the lazy import is in §10.
The learned-flow expectations layer (docs/runtime/EXPECTATIONS.md; reporting only, no model, no action) has unit
tests on the fake clock but no demo or real-app measurement yet. Its acceptance gate before any public claim: zero
expectation reports across the six demos' clean trials (15 per mode per demo, bench/heal/), reported next to
how many of the demos' silent-flow bugs (stuck "Placing…", lost commit) it names, plus a two-session run on the
stored path (the trials start in a fresh browser context, so they never load stored flows). Measured so far:
first-load JavaScript, minified + gzip, /auto 106.5 KB and GenClass.init() 99.1 KB (test/bundle.test.ts, CI
run 38455195053 on branch wave1/expectations, 2026-10-13, after the review fixes; 105.8 / 98.3 KB in run
38448983470 before them), against about 99 KB and 91 KB before the layer (the README's figures at 0.2.2, where
test/bundle.test.ts kept them under 100 / 92 KB): about +7.5 KB per entry.
| Demo | Bug rate Off | Guard | Heal | False interventions on clean runs |
|---|---|---|---|---|
| search | 23% | 23% | 27% | 0 |
| editor | 83% | 90% | 87% | 0 |
| checkout | 70% | 77% | 83% | 0 |
| status | 100% | 100% | 100% | 0 |
| board | 57% | 63% | 77% | 0 |
| decisions | 83% | 97% | 97% | 0 |
Guard executed 0 actions. Heal took 156 actions, which fixed nothing. The trained model, 0.2.0, was measured on the same six demos on 2026-10-09 (the first table in this section).
| Dataset | Rows | Status |
|---|---|---|
| Curriculum (cur1–cur4) | ~1.9M | done |
| Sim v1 phase A / phase B | 600k / 1.4M | done (v1, superseded) |
| Sim v2 gold (S1+S2 labels, 46 features, 115 domains) | 10,423,855 (7.56M train / 318k dev / 2.55M test) | done, 20 nodes in ~31 min |
| Sim v2.3 gold top-up (relation-learner fixes) | 2,107,824 (inconsistency rows 5.5% → 1.6%) | done |
| Sim on-policy round a (model r17-v2a acting in the sim; DAgger) | 1,248,131 (448k at shipping gate, 672k at explore gate 0.5) | done |
| Sim on-policy round b (r17-v2b with its own gates, v2.3) | 1,094,953 | done |
| Sim v2 unlabeled (for teacher labelling) | 51,272,078 | done (~21k rows/s per node) |
Real-browser v2 gold (128 apps: 96 in v2c1..4, plus 32 in the v2d top-up, wave 4) |
616,437 (+523k unlabeled) | done; determinism 3,030/3,030 (96 apps × 30 seeds) and 640/640 (128 apps), interference 0/256 |
| Real-browser v2.3 top-up (relation-learner fixes) | 221,186 (real total 837,623) | done; real inconsistency triggers 18.3% → 2.6% of decisions, genuine share 10% → 40% |
| Real-app eval set (unambiguous) | 16,600 (clean-benign 4,000, benign-salient 4,000, duplicate 4,000, genuine break 3,085, stale 1,515) | done |
| When (UTC) | Event |
|---|---|
| 10-07 13:00–17:00 | Vocabulary pruned to 16k merges. q8 exports: R32 22 MB, R17 9.6 MB. Stage-1 curriculum built. |
| 10-07 17:58–19:00 | Stage 1 on 10 nodes, for both R17 and R32. |
| 10-07 19:00–20:05 | Stage 2 pilot on pre-freeze sim data: precise but timid, because the labels were mushy. |
| 10-07 23:20 | Runtime frozen as situation-v1. Sim phase A: 600k rows in 41 min. |
| 10-08 00:06–01:30 | Final round 1 on phase A (R32 on 6 nodes, R17 on 4). Results in §1. |
| 10-08 00:35 | Auto-shutdown disabled with the owner's OK. Quota raised to 2,048 vCPU; nodes c12–c23 added. |
| 10-08 02:00–04:00 | Separability analysis led to the S1/S2 label fixes and the F1–F9 facts. Runtime batches 4–5 (network-boundary decisions). |
| 10-08 ~04:00 | situation-v2 frozen. v2 generation launched: ~10M gold + ~50M unlabeled in the sim, ~495k real-browser rows. |
| 10-08 06:10–18:12 | v2 rounds (§1, training/EVAL.md): r17-v2a at 06:10; r17-v2b at 07:10, its gates refit on on-policy dev data at 10:30 and published as @genclass/runtime-model@0.1.0 at 13:38 (npm); the 150M teacher came out weaker than r17-v2b, so nothing was distilled; on-policy (DAgger) rounds a and b in the sim; r17-v2c; the head-to-head of v2c, v2d and v2dT at 11:45; at 16:30 the refit with the real-app certification set chose r17-v2dT, published as @genclass/runtime-model@0.2.0 at 18:12 (npm). |
Compute: Azure CPU nodes (F80 and D64 sizes); the day-by-day record is kept internally.
Measured by unit tests in GitHub CI (Node on ubuntu-latest), not in a browser and without the model. Branch
wave1/defaults-footprint, 2026-10-13; each value is the line the test prints in the "Unit tests" log of that run.
First-load JavaScript (packages/runtime/test/bundle.test.ts: esbuild bundles the built dist/ the way an app
would, minified, gzip level 9, first-load chunks only; the model worker, ONNX Runtime Web and the devtools are
separate chunks loaded on demand):
| entry | a7b60d7 (run 38442191071) |
e26c0dc (run 38442394416) |
ebbe912 (run 38445206305) |
limit |
|---|---|---|---|---|
@genclass/runtime/auto (with state discovery) |
100.61 KB | 100.61 KB | 100.70 KB | 101.5 KB |
main entry, GenClass.init({}) |
93.13 KB | 93.13 KB | 93.22 KB | 93.5 KB |
Telemetry volume (packages/runtime/test/telemetry.test.ts, "an hour at the built-in model's maximum rate"): one
page, 1,800 mutation decisions, one every 2 s (30 a minute, the model.maxDecisionsPerMinute default) for a simulated
hour on a fake clock. No model: a scripted decider answers expected (routine) or stale at p = 0.97 (every decision
a detection, above the default report gate). Observe mode, situation text off, every request body counted. Defaults
are decisionSample: 0.25, maxDecisions: 200 (packages/runtime/TELEMETRY.md, "Volume").
| commit (run) | routine, defaults | every one a detection, defaults | every one a detection, every record (decisionSample: 1, maxDecisions: Infinity) |
|---|---|---|---|
a7b60d7 (38442191071), e26c0dc (38442394416): summaries stopped after the cap (a bug) |
203.4 KB, 204.1 KB | 228.7 KB, 228.7 KB | 2,064.7 KB, 2,064.7 KB |
ebbe912 (38445206305): a summary about once a minute for the whole hour |
241.6 KB, 100 requests | 271.4 KB, 95 requests | 2,064.7 KB, 361 requests |
of which summary events, ebbe912 |
61, 39.4 KB | 61, 40.8 KB | 61, 40.8 KB |
The routine column depends on which decisions the session-id hash samples, so it moves between runs. The test fails if either default case reaches 300 KB. A real page sends far fewer decisions to the model (only salient situations), so these are upper bounds for one tab, not typical volumes. Superseded for 0.3.0 by §10: detections are no longer sampled or capped, so the "every one a detection, defaults" case now sends every record.
Hot-path timings (packages/runtime/test/review-perf.test.ts, run in CI, reported, not blocking; budgets: a
keystroke write under 1 ms, an async write under 2 ms, a Redux-style dispatch under 1 ms (2 ms from async work), a
settled point under 16 ms). The last recorded run is from a shared VM on 2026-10-07, on the runtime just before
0.1.0-alpha.1 (commit bac4409, packages/runtime/STATUS.md at that commit), and has not been re-recorded since: a
keystroke write to a store holding a 5,000-item array 0.22 ms, an async write 0.14 ms, a Redux-style dispatch on
5,000 entities 0.71 ms (0.70 ms from async work), a settled point 0.3 ms (+2.3 ms with an unchanged 5,000-item
adapter store).
packages/runtime/test/bundle.test.ts bundles the built dist/ the way an app's bundler would (esbuild, minified,
code splitting) and gzips (level 9) every file of the first load on its own; it prints the sizes and fails above the
limits. No model is involved, so no model, threshold or FIR applies. GitHub Actions ci.yml, Node 22, the same test
for both columns:
| entry | 0.2.2 | Unreleased (sink adapters, sinks by name; the Sentry, PostHog and Datadog adapters were later removed and never shipped, §10) | limit |
|---|---|---|---|
@genclass/runtime/auto (automatic state discovery; Unreleased also the sink names' proxy) |
99.17 KB in 3 files (96.25 + 2.74 + 0.19) | 99.97 KB in 3 files (97.04 + 2.74 + 0.19) | 100 KB |
main entry, GenClass.init({}) |
91.70 KB in 3 files (88.77 + 2.74 + 0.19) | 91.83 KB in 3 files (88.91 + 2.74 + 0.19) | 92 KB |
script tag, dist/genclass.global.min.js (one file; the worker, ONNX Runtime and the sink adapters load on demand) |
113.49 KB (333.8 KB minified) | 114.86 KB (337.2 KB minified) | none |
CI runs: 0.2.2 is run 38441488459 (packages/runtime as tagged v0.2.2, which is also main 87481d7, with only the
size-printing bundle test added); Unreleased is run 38443030760. The Unreleased changes add 0.80 KB to /auto,
0.13 KB to the main entry (SinkRecord.route and the sink-shape check) and 1.37 KB to the script tag; the sink
adapters, their name table and the fan-out are absent from both first loads (asserted in the same test). That leaves
0.03 KB under the /auto limit and 0.17 KB under the main entry's, so the next change that adds to the first load
needs a smaller first load or a new limit. The 0.3.0-beta.0 sizes and limits are in §10. The Sentry, PostHog and
Datadog adapters measured here were lazy chunks, outside both first loads; they are not shipped (§10, last rows).
Branch release/0.3.0 (wave2/accounts-base = main 88df703 + wave1/projects, then wave1/defaults-footprint,
wave1/dev-visible-reports, wave1/expectations made opt-in and lazy, and wave1/playwright; for the release candidate
then main f8c66f2 (collector limits, accounts, the CLI's login/connect/whoami/logout, the landing page),
wave2/detect-observe dd20233 (§11) and wave2/dash-detect ac2fc6f (§12)), 2026-10-13. GitHub
Actions ci.yml, Node 22; each value is the line the test prints in the "Unit tests" log of that run. No model is
involved (esbuild bundles and a scripted decider), so no recall, false-intervention or false-detection rate applies
to these numbers.
First-load JavaScript (packages/runtime/test/bundle.test.ts: esbuild bundles the built dist/ as an app
would, minified, code splitting, every first-load file gzipped on its own at level 9). From 53b04f2 on, the test
walks the first load from the app's own entry (<stdin>) and counts every file; the 738a05b row used the earlier
walk, which missed the two shared chunks (2.74 + 0.19 KB).
| step (commit, CI run) | /auto |
main entry GenClass.init({}) |
script tag (one file) |
|---|---|---|---|
| 0.2.2 (§9, run 38441488459) | 99.17 KB | 91.70 KB | 113.49 KB |
+ app tokens, protect() (wave1/projects 625a491, run 38436300545; earlier walk) |
100.68 KB | 93.19 KB | not printed |
+ device floor, telemetry sampling (738a05b, run 38461782967; earlier walk) |
102.16 KB | 94.66 KB | not printed |
+ sink adapters, sinks by name (53b04f2, run 38462380432; the Sentry, PostHog and Datadog adapters in it were removed before release and never shipped, last rows) |
102.94 KB in 3 files | 94.79 KB in 3 files | 117.96 KB |
+ flow expectations, src/learn/flows.ts lazy (fcefedb, run 38463736575) |
106.16 KB in 4 files | 98.13 KB in 4 files | 126.03 KB |
+ even sampling draw (c2f0b54, run 38464240236) |
106.23 KB | 98.20 KB | 126.11 KB |
version bump to 0.3.0-beta.0 (bff4a71, run 38465242176; not the published build, which is the visual-checks row below) |
106.24 KB (100.26 + 3.05 + 2.74 + 0.19) | 98.21 KB (92.24 + 3.05 + 2.74 + 0.19) | 126.12 KB (369.4 KB minified) |
+ telemetry 429 back-off, no sampling settings in session (7c7c6d6, run 38477010836) |
106.59 KB (100.62 + 3.05 + 2.74 + 0.19) | 98.57 KB (92.59 + 3.05 + 2.74 + 0.19) | 126.45 KB (370.5 KB minified) |
+ detection tiers, deferred decisions, review fixes (wave2/detect-observe dd20233, run 38482297879; §11) |
107.56 KB | 99.59 KB | 127.89 KB |
release candidate: + main (accounts and the CLI commands: no browser code) and wave2/dash-detect (the worker only) (18e89f0, run 38485063587) |
107.56 KB (101.59 + 3.05 + 2.74 + 0.19) | 99.59 KB (93.62 + 3.05 + 2.74 + 0.19) | 127.89 KB (374.9 KB minified) |
+ release review: telemetry masks path segments holding an @ and URLs in model error text (9d567a0, run 38488753971) |
107.61 KB (101.63 + 3.05 + 2.74 + 0.19) | 99.61 KB (93.64 + 3.05 + 2.74 + 0.19) | 127.92 KB (375.0 KB minified) |
+ no vendor sink adapters: the Sentry, PostHog and Datadog adapters removed (never shipped), one console warning for those names in a page config (wave2/no-vendor-sinks 31ddc98, run 38491077597) |
107.73 KB (101.75 + 3.05 + 2.74 + 0.19) | 99.61 KB (93.64 + 3.05 + 2.74 + 0.19) | 128.01 KB (375.2 KB minified) |
+ that warning reworded to claim no paid feature: "Sentry, PostHog and Datadog forwarding is not in the free runtime" (eea042b, run 38494904699) |
107.71 KB (101.74 + 3.05 + 2.74 + 0.19) | 99.61 KB (93.64 + 3.05 + 2.74 + 0.19) | 128.00 KB (375.1 KB minified) |
limit in bundle.test.ts (measured at bff4a71 + 1.5 KB, owner-approved for 0.3.0; not re-set: the release candidate was 0.13 / 0.10 KB under it, and with the vendor-name warning /auto is 0.03 KB under it) |
107.74 KB | 99.71 KB | none |
+ measured visual checks' first-load stub: wave2/visual 6ae209f merged (release/0.3.0-visual, merge 2c7014f, PR head be8cb7c, run 38676978903; §13) = the published 0.3.0-beta.0 (tag v0.3.0-beta.0, a86c928; the only later src/ change there is a comment) |
108.25 KB (102.28 + 3.05 + 2.74 + 0.19) | 100.13 KB (94.15 + 3.05 + 2.74 + 0.19) | 128.56 KB (376.5 KB minified) |
limit in bundle.test.ts for 0.3.0-beta.0, owner-approved 2026-10-15 (the measured sizes + 0.05 / 0.07 KB) |
108.30 KB | 100.20 KB | none |
| + interaction analytics' first-load stub (§15; limits owner-approved 2026-10-18) | 108.53 KB | 100.40 KB | not measured |
+ review of the mode-raise fix: normalizeRoute, the mode validation and its warning, the sample bucket's act@<sample> form, readWindowConfig's mode check (branch security/advisory-mode-raise, 0.3.0-beta.5; local, 2026-10-27, test/first-load-size.mjs and gzip -9 against main 49c8c03's dist: 108.36 / 100.28 / 128.55 KB) |
108.63 KB (+0.27) | 100.46 KB (+0.17) | 128.83 KB (387.4 KB minified, +0.28) |
limit in bundle.test.ts for 0.3.0-beta.5, owner OK 2026-10-28 (/auto: CI's Node 22 gzip measures 108.83 KB where Node 25 measures 108.63 KB, so the limit follows CI plus 0.07 KB; the main entry: CI measures 100.55 KB where Node 25 measures 100.46 KB, limit 100.60 KB) |
108.90 KB | 100.60 KB | none |
limit in bundle.test.ts for 0.3.0-beta.5 with the measured visual checks' counts in telemetry (VISUAL.md section 10), owner OK 2026-11-02: the session event's visual and fix fields grow both entries by 0.06 KB on Node 22 (/auto 108.89 KB, the main entry 100.61 KB); the summary counts live in the checks' own chunk |
109.00 KB | 100.70 KB | none |
Removing the vendor adapters changes no first load (they were lazy chunks of /auto and files the script tag loaded
from the CDN). The one warning for the names sentry, posthog and datadog (src/cdn/sinks.ts applySinks: the
name check and the text with the pricing link, as reworded in 2fea265) adds 0.10 KB to /auto and 0.08 KB to the
script tag; the main entry, which has no sink names, is unchanged. That leaves 0.03 KB under the /auto limit: the next change that adds to
/auto's first load needs a smaller first load or a new limit.
The measured visual checks (§13, VISUAL.md) were that change. Their first-load stub
(src/visual/start.ts: the development-host rule, the hidden-tab and prerender gates, the "check" listener set,
rt.checks() / rt.checkNow(), the "check" sink filter, the visual page-config key) adds 550 B to /auto and 524 B
to the main entry, measured by test/first-load-size.mjs against release/0.3.0 at 32cc476 in the same run's
visual-browser job (107.711 to 108.248 KB, 99.614 to 100.126 KB; the job fails at 0.6 KB of growth). The checks
themselves are one lazy file (dist/visual/index.js, about 35 KB gzip), outside every first load; the script tag
loads it by URL, and the script-tag file grows by 0.56 KB. Both entries are over the 107.74 / 99.71 KB limits, so
bundle.test.ts carried 108.30 / 100.20 KB for 0.3.0-beta.0; the owner approved these limits on 2026-10-15 (the
table above). The interaction-analytics stub later raised them to 108.6 / 100.48 KB (§15).
The flow expectations were +7.3 KB on /auto when src/learn/flows.ts was in the first load (106.5 KB vs 0.2.2's
99.17 KB on wave1/expectations, run 38455195053). Imported lazily, and only with learn.expectations: true, they
add +3.3 KB (102.94 to 106.24 KB): the runtime side (learned-state persistence, the console line, the hooks) and
the overhead of the util code the lazy chunk shares becoming a 3.05 KB chunk of its own. So the lazy import moves
about 4 KB (by difference, not measured on its own) out of every first load. The script-tag build has no code
splitting and carries the flow layer inline.
Telemetry volume (packages/runtime/test/telemetry.test.ts, "an hour at the built-in model's maximum rate"; the
setup of §8: one page, 1,800 decisions from a scripted decider at p = 0.97, observe mode, situation text off, every
request body counted). Since 738a05b detections, near-misses (a top diagnosis other than expected), actions and
vetoes are never sampled or capped; only routine decisions are (decisionSample: 0.25, maxDecisions: 200
routine records per page load):
| commit (run) | all routine, defaults | every one a detection, defaults | every record (decisionSample: 1, maxDecisions: Infinity) |
|---|---|---|---|
738a05b (38461782967) |
242.4 KB, 106 requests | 2,064.7 KB, 361 requests | 2,064.7 KB, 361 requests |
c2f0b54 (38464240236), the even draw below |
249.0 KB, 162 requests | 2,064.7 KB, 361 requests | 2,064.7 KB, 361 requests |
bff4a71 (38465242176), the version bump to 0.3.0-beta.0 (a longer version string in every envelope) |
250.3 KB, 163 requests | 2,067.1 KB, 361 requests | 2,067.1 KB, 361 requests |
7c7c6d6 (38477010836): 429 back-off; session without decisionSample / maxDecisions (about 40 bytes once per page load, below the 0.1 KB rounding) |
250.3 KB, 163 requests | 2,067.1 KB, 361 requests | 2,067.1 KB, 361 requests |
release candidate 18e89f0 (38485063587): with wave2/detect-observe (§11) |
251.9 KB, 165 requests | 2,068.5 KB, 361 requests | 2,068.5 KB, 361 requests |
release candidate f363e92 (38485289326), docs only |
250.9 KB, 157 requests | 2,068.5 KB, 361 requests | 2,068.5 KB, 361 requests |
release review 9d567a0 (38488753971): @ path segments and URLs in model error text masked |
251.9 KB, 165 requests | 2,068.5 KB, 361 requests | 2,068.5 KB, 361 requests |
Summaries: 61 per hour in every case, 39.3 KB (routine) and 40.8 KB (detections); 40.7 KB and 42.2 KB in the
release candidate, whose summaries also carry possibleDetections.
The routine-record draw (decisionDraw in src/telemetry/client.ts). The 0.25 share was drawn from fnv1a of
":", whose high bits hardly depend on the last characters, and decision ids (d1, d2,
...) differ only there: CI run 38463736575 failed on a page that sent 44 of 80 routine records. Measured with a Node
script over 200 fixed session ids, 80 decisions each: fnv1a alone sent 0 to 59 of 80 per page (0.2554 overall);
with murmur3's 32-bit finalizer after it (c2f0b54) 9 to 32 of 80 (0.2499 overall). The unit test "the
routine-record draw is even for every session" asserts the finalized draw on the same 200 sessions (5 to 36 per
page, 0.23 to 0.27 overall). Detections never go through this draw.
The 429 back-off (7c7c6d6, ported from wave2/detect-observe): test/telemetry.test.ts "a 429 or a
Retry-After is a back-off" asserts, with a scripted transport, that five 429s in a row (more than the three failures
that stop a page) never stop it, that nothing is sent during a back-off (30 s, 60 s, 120 s, 240 s, then 5 minutes
with no Retry-After; exactly 5 s with Retry-After: 5), and that the refused batch is sent in order once the
collector accepts it. These are behaviour checks, not rates: how many pages behind one address hit the collector's
limit in production is not measured.
Not measured here: browser timings, the install suite (test/install/run-all.sh, VM only), the demos'
clean-trial gate for the flow expectations (§5), and any number with the model.
Branch wave2/detect-observe (from release/0.3.0), 2026-10-13. Two kinds of change: a lower reporting tier in
observe (InitOptions.detect, CONTRACT §8), and observe situations that used to be dropped and are now decided later
(while the model loads or reloads, while the tab is hidden; OPTIONS-SPEC §5.1, §5.3), plus four fixes (idle unload,
the body wait, superseded writes in observe, priority aging). No model-visible text changed and no action gate
reads the tiers. The false-intervention rate is unchanged for every app without model.unloadAfterIdleMs (no
action path changed; deferred decisions are gated as observe; guard and heal build nothing while the model loads, as
before 0.3.0). With model.unloadAfterIdleMs, guard and heal act again after the model reloads, where 0.2.x stopped
for good after the first unload: those actions are new, and their FIR was not measured. Nothing in this section was
measured end to end on this branch; the numbers below are earlier measurements of the same thresholds, which is what
the "possible" tier uses.
Expected effect of the "possible" tier, from measurements already made. In observe at the default balanced
profile, likely + possible detections are exactly the detections at the eager profile's report gate (0.70; the
likely ones alone are those at balanced's 0.90). In observe the profile changes nothing but that gate, so the
eager measurements are the expected effect:
| measured | balanced (report 0.90): likely only, 0.2.x | eager (report 0.70): likely + possible, 0.3.0 observe default | source |
|---|---|---|---|
| detected with the right diagnosis, held-out real apps: duplicate submits | 75% | 77% | training/EVAL.md, @genclass/runtime-model@0.2.0 = r17-v2dT table (REAL eval test split, q8) |
| same: stale overwrites (situation v2 / v2.3) | 21% / 30% | 40% / 45% | same |
| same: genuine breaks | 6% | 13% | same |
| report false-detection limit the 0.2.0 fit was held to, per clean decision, met on held-out test (real / sim) | ≤ 2% / ≤ 3% | ≤ 5% / ≤ 6%, for likely and possible together; the share of possible detections that are false is not measured and is expected to be well above the likely tier's | same ("All profiles meet their limits on held-out test"); the exact rates are not published (the gate-fit curves are on a deallocated training node) |
| bench demos, observe, buggy chaos trials with at least one detection | 39 / 42 | 41 / 43 | bench/heal/results/demos/results-baseline-beta4-m020.json and results-baseline-beta4-m020-aggr-eager.json (model 0.2.0, runtime 0.1.0-beta.4, Playwright, model preloaded, visible tab), counted again for this section with a Python script over their raw trials |
| bench demos, observe, clean trials with a detection (a false-detection proxy) | 2 / 30 | 3 / 29 (search seeds 1502, 1503 and status 1503, all delivery:stale; one of the 30 eager clean trials showed a bug and is left out) |
same files |
Read with care: the bench counts are trial-level co-occurrence (a trial with any finding), not per-bug matches, and
n is small (one more clean trial with a finding in about 30). The compat matrix's false-finding baseline is at
balanced only (49 false findings in 880 correct-app runs, all in scenario g, polling and fan-out:
compat/RESULTS.md); the possible tier will add to it, by an amount not measured. Possible detections print with
console.debug only and reach rt.on("detect"), rt.on("report") and a report function only with
detect: "possible" set explicitly, so an app wired to those gets the likely ones by default, as before. Three
fixes below (superseded writes in observe, the body wait, the idle unload) can still add likely reports, false ones
included, at a rate not measured, and detect: "likely" does not turn them off; the deferred situations do not add
likely reports (they are capped at the possible tier, below).
Deferred situations and the fixes: expected direction, not measured.
- Model loading. Before 0.3.0 a page view lost every salient situation before the model was ready: those raised
while the model loaded (3.76 s on average on the one production site reporting telemetry, runtime 0.2.1, the
maintainers' admin view on 2026-10-13) were dropped before triage, uncounted; those raised before the idle preload
started (after the load event, up to 5 s, then up to 2 s of idle) were counted as failOpen
not-ready, and the first of them started the load. In observe both are now decided once the model is ready (at most 32 per page load), built at the smallest automatic situation budget while the device is not known yet. How many detections that adds is unmeasured: the bench and the compat matrix wait for the model before each trial (await gc.ready, preload eager), so they never exercised this path. - Hidden tabs. Unmeasured for the same reason (the trials run in a visible tab). Hidden-tab timers are throttled
and aligned (1 s, and once a minute after 5 minutes in Chrome), so background pollers can overlap: the polling and
fan-out pattern behind all 49 false findings in the compat matrix. A hidden tab never loads or reloads the model.
In production the loss counters (
summary.runtime.model.hiddenSkipped,summary.counts.failOpen) only reach the raw R2 objects; no aggregate reads them yet. - Reported at the possible tier at most. Because neither path has a measured false-report rate, a deferred
situation's detection (load window or hidden tab) is recorded as possible whatever its probability:
console.debug, the devtools,summary().possibleDetections, telemetry (tier: "possible"), never the likely channel, following the owner's precedent for the expectations layer (opt-in until measured).detect: "likely"turns deferral off. Lifting the cap needs the cold-start and hidden-tab clean run below, or the owner's explicit OK. - Idle unload, the 100 ms body wait, overwritten writes on slow devices, priority aging, the telemetry back-off:
regression tests on the fake clock (
test/deferred.test.ts,test/observe-delivery.test.ts,test/hold-late.test.ts,test/telemetry.test.ts), no recall or false-detection number. The back-off is 30 s doubling (or a longer Retry-After) up to 5 minutes, for a 429 or a 503 with a Retry-After only, and a refusal after 8 back-offs in a row (about 27 minutes) counts as a failure. - CPU on the user's device: unchanged ceiling, 30 model evaluations a minute (
model.maxDecisionsPerMinute); deferred decisions count against it. Raising it was not measured, so it stays.
What CI measured on this branch (GitHub Actions ci.yml, run 38471973596, commit 8501528 on
wave2/detect-observe; no model involved, so no recall or false-detection rate applies to these):
| 0.3.0-beta.0 version bump (§10, run 38465242176) | with this change (run 38471973596) | limit (unchanged) | |
|---|---|---|---|
first load, /auto |
106.24 KB | 107.66 KB (101.69 + 3.05 + 2.74 + 0.19) | 107.74 KB |
first load, main entry GenClass.init({}) |
98.21 KB | 99.69 KB (93.72 + 3.05 + 2.74 + 0.19) | 99.71 KB |
script tag genclass.global.min.js |
126.12 KB | 127.63 KB (374.5 KB minified) | none |
| telemetry hour, all routine (defaults) | 250.3 KB, 163 requests | 252.2 KB, 167 requests (the sampled share follows the random session id) | test fails at 300 KB |
| telemetry hour, every decision a (likely) detection | 2,067.1 KB, 361 requests | 2,068.5 KB, 361 requests | none |
of which 61 summary events |
39.3 / 40.8 KB | 40.7 / 42.2 KB (the possibleDetections count) |
none |
After release/0.3.0 took this branch's 429 back-off and dropped decisionSample / maxDecisions from the session
event (§10, e637c37), it was merged back here (fab159c, CI run 38477575339): first load 107.65 KB on /auto
(101.68 + 3.05 + 2.74 + 0.19) and 99.68 KB on the main entry (93.71 + 3.05 + 2.74 + 0.19), script tag 127.62 KB
(374.4 KB minified); telemetry hour 251.6 KB in 163 requests all routine (summaries 40.7 KB), 2,068.5 KB in 361
requests every decision a detection (summaries 42.2 KB). Against release/0.3.0's 106.59 / 98.57 KB this branch adds
1.06 / 1.11 KB (the back-off is in both). Unit tests 890 passed, 14 skipped.
The two first loads grew by 1.42 and 1.48 KB, inside the 1.5 KB the release left; the first CI run (38471360511) measured 107.77 / 99.80 KB, over the limits, and the code was trimmed (no behaviour change) rather than the limits raised. Unit tests: 889 passed, 14 skipped (the model-parity tests); the telemetry-worker job 24 passed.
After the review fixes (hidden tabs never load the model, guard and heal skip while it loads as before, possible
detections off the app's listeners by default, deferred decisions capped at the possible tier, trigger-time route,
bounded back-off; CI run 38482297879, commit 7c12cf5): first load 107.56 KB on /auto (101.59 + 3.05 + 2.74 +
0.19) and 99.59 KB on the main entry (93.62 + 3.05 + 2.74 + 0.19), 0.18 / 0.12 KB under the unchanged limits;
the fixes' own bytes were paid for by moving the expectation report's console text into the lazy flow chunk (the
first push with the fixes alone measured 107.92 / 99.93 KB, CI run 38481007044, over the limits). Script tag
127.89 KB (374.9 KB minified; it carries the flow chunk inline, so it grew by the fixes, +0.27 KB). Telemetry hour:
251.2 KB in 160 requests all routine (61 summaries, 40.7 KB), 2,068.5 KB in 361 requests every decision a (likely)
detection (summaries 42.2 KB). Unit tests 903 passed, 14 skipped; telemetry-worker job passed. No model involved, so
no recall or false-detection rate applies to these numbers.
Deploy order for the aggregates (no deploy from this branch): the telemetry-worker dashboard
(telemetry-worker/src/dashboard.ts, possible_detections) and the maintainers' admin aggregates
(site/src/admin.js, possible_detections and recent rows of kind possible; the live demo counter in
site/public/live.js skips them; since 2026-10-18 site/ is in the private repository genclass-dev/genclass-site)
both count tier: "possible" apart from detections. Both must be deployed before a
0.3.0 runtime reports to them; an older aggregate counts every possible detection, deferred ones included, as a
detection, and the tiers cannot be separated afterwards (the admin daily metric has no tier).
Not yet measured end to end (needs the owner's OK for the VM or a training node): the false-detection rate of the
possible tier on the 0.3.0 runtime (compat scenarios c to h and the bench clean trials, observe, default detect); the
exact 0.2.0 report curve (out/gates/r17-v2dT*-report.json); a cold-start run (no await gc.ready, default idle
preload, plus hidden-tab trials with background pollers) that would show how many detections the deferral adds and
how many false ones come with them, which is what lifting the possible-tier cap on deferred decisions needs; the
likely reports the three fixes add on correct apps (compat, observe, slow WASM and slow-body scenarios); and guard/heal
FIR with model.unloadAfterIdleMs.
Branch wave2/dash-detect (merged with release/0.3.0). The worker takes each page load's totals (page loads,
decisions, detections by diagnosis, possible detections, acted, triggers, diagnoses, action outcomes, vetoes, breaker
trips, fail-opens, model errors, action limits, hidden-tab skips, rate-limit drops, browser drops, sampled-out and
capped records) from its cumulative summary, through a per-page-load high-water mark, and from its events until the
first summary arrives (telemetry-worker/README.md, "Exact totals"). Routes, functions and recent rows still come
from the events.
| what | value | how |
|---|---|---|
| Counted totals vs the page load's own summary, unit fixtures | equal in every case, 0 double counts | telemetry-worker/test/dashboard.test.ts (CI run 38465992209, commit 8b0e949: 40 worker tests passed): a repeated batch, bigger / late older / partial summaries, a split flush in three orders including both parts processed at once, a client that sent 4 decision records for 500 decisions and 260 detections, state from the previous worker format, state swept after 2 days, a corrupt state row. Fixtures, not production traffic |
| The possible tier, unit fixtures | "Problems found" 0 and possible 1 for one possible detection sent before a summary that counts likely detections only; counted once with its decision record in the other batch, in both orders; a repeated batch adds nothing | same file, describe "possible issues: runtime 0.3.0's lower tier" (CI run 38479161101, commit 4ebc651: 45 worker tests passed). Fixtures, not production traffic |
| Local replay: 20 page loads, 1,557 events (839 decision records: every non-routine one, 1 in 4 routine), split into 276 batches of 1-12 events, 30% of them moved out of order, 30 sent twice | dashboard 1,520 decisions / 288 detections / 340 possible, equal to the page loads' own summaries; by diagnosis 288 / 340; 243 detections and 280 possible ones with a page (the rest arrived in another batch than their decision record); largest page-load state row 476 bytes | scratch replay through aggregate() and stats() on node:sqlite with the real migrations, seeded generator. The same replay through the worker before this fix (9cd0799, which counted every detect event as a detection and derived its own band from decision records): 504 detections and 60 possible |
Dashboard page HTML (/dashboard/<secret>) |
50,798 bytes on release/0.3.0 → 60,940 (gzip 15,555 → 17,526); 60,226 at 9cd0799 |
dashboardPage() output for one fixed project (name "Acme checkout", 22-char nonce); a worker page, not part of the runtime's first-load budget |
/start HTML; sample stats JSON (7 d, 2026-10-13T20:00Z) |
12,700 → 12,934 bytes; 8,580 → 9,698 bytes | same |
| Fig. 2 (possible issues) bar outline contrast | 5.72:1 light (--ink3 #5A6684 on #FFFFFF), 7.00:1 dark (#97A3C2 on #0B1736) |
dataviz validate_palette.js contrast(); one series, no new hue (ring keys reuse the validated type hues), so no CVD pair to check |
Possible issues. The tier is the runtime's own (§11, wave2/detect-observe): in observe, by default, a decision
whose top diagnosis is not expected with confidence under the report gate but at or over the eager profile's report
value is sent as a detect event with tier: "possible" and counted in summary.counts.possibleDetections. The
worker counts exactly that, in its own metrics, never in the problem counts, and derives nothing from decision records
(an earlier version of this branch counted a band [0.70, report gate) of decision records itself, in every mode; that
listed decisions the console never showed, and counted runtime-possible ones as detections). Reporting only: no action
reads it, so FIR 0 by construction. Its false-report rate is the runtime tier's: at balanced, likely + possible are
the detections at eager's report gate, which on held-out real apps detect 77% / 45% / 13% of duplicate / stale /
broken cases against 75% / 30% / 6% for likely alone, with report false-detection limits of ≤ 5% REAL / ≤ 6% SIM
against ≤ 2% / ≤ 3% (limits met on held-out test; the exact rates are unpublished; training/EVAL.md, §11). On the
bench demos, clean trials with a detection were 3 / 29 at eager against 2 / 30 at balanced (§11). Under
release/0.3.0's defaults the client never samples or caps detect events or the records of non-expected decisions,
so the tier's count is exact, not a sample.
Sampled metrics. Average decision time, decision mode, executed, routes and functions come from the decision records that arrived. Routine records are sampled (1 in 4, at most 200 per page load), so these lean towards non-routine decisions; the session event no longer carries the sampling settings (§10), so the worker cannot reweight, and the page now labels them as records ("average of N decision records received").
Not measured: anything on production data. When this was measured no released runtime sent a token or tier, so no
real page load reached a dashboard; 0.3.0-beta.0 (npm beta) is the first release that sends both.
These are deterministic measurements, not model results (VISUAL.md). The numbers in this section come
from fixture pages written by the same author as the checks (packages/runtime/test/browser/visual/), so they show the
checks work as designed. Section 14 has the bench on the demos, templates, compat apps and public sites.
| what | measured | source |
|---|---|---|
| planted faults found on the fixture pages | 35 of 35 (every planted fault, on its element, 23 kinds) | visual.spec.ts, 24 tests passed |
| findings on clean fixture pages (false reports) | 0 on the clean page at 1280 x 800 and at 390 x 844 (phone), 0 on the light variant of the dark-mode page, 0 on a 10,000-element page, 0 besides the planted fault on every other page | same |
| runtime fixes on fixture pages | text colour, pointer-events and scroll lock applied, verified, undone; 0 CSP violations from the owned rule under style-src 'self' |
same |
| first-load growth against main (minified + gzip) | first build, against an older main: /auto +486 B (99.649 KB), main entry +431 B (92.118 KB); limit 512 B. After the review, against main of 2026-10-13 (app tokens, protect(); CI run 38497342999): /auto +582 B (101.244 KB, over main's 101 KB ceiling), main entry +524 B (93.699 KB of 94 KB). The narrow development-host rule and the hidden-tab gate account for about 100 B. Ceilings raised on the branch to 101.5 KB and 0.6 KB growth; the owner approved the 0.6 KB growth gate with the 0.3.0-beta.0 limits (108.30 / 100.20 KB) on 2026-10-15. Merged into 0.3.0-beta.0 (release/0.3.0-visual, CI run 38676978903), against release/0.3.0: /auto +550 B (108.248 KB), main entry +524 B (100.126 KB); limits 108.30 / 100.20 KB for 0.3.0-beta.0, approved by the owner on 2026-10-15 (§10); raised to 108.6 / 100.48 KB on 2026-10-18 for interaction analytics (§15) |
test/first-load-size.mjs in the visual-browser job, test/bundle.test.ts |
lazy chunk dist/visual/index.js |
about 35 KB gzip (97 KB minified) | esbuild, minified |
| a 10,000-element page | 2 passes, 72.9 to 127.8 ms of check time in all (two runs), longest slice 8.5 ms (budget 8 ms per slice), no partial pass; after the review: 106.7 ms, longest slice 8.4 ms | big.html |
| the same page with hover, clicks and scrolling (after the review) | 137.1 ms of check time in all, longest slice 8.7 ms, no long task (50 ms or more) on the page once the checks ran | big.html, CI run 38497342999 |
5,000 stylesheet rules with var() (after the review) |
49 ms of check time, longest slice 10.9 ms, no long task | big-css.html, same run |
| correct pages a review found reported as broken (after the review) | 0 findings on each: dark mode from the color-scheme meta tag and a dark-only page (both schemes), white text over pointer-events: none overlay, canvas, video and a full-screen particles canvas, a dialog that scrolls inside, a web-component app, a full-page section scroller, a busy shield during a 6 s request; a fix undone when its background changed; a scroll lock held since load reported and not removed |
visual.spec.ts, 44 tests passed |
14. Visual bench: false reports, recall and cost (GitHub runners, Chromium 153, 2026-10-13, branch wave2/visual)
Measured checks, not model results. bench/visual runs the checks' engine
(packages/runtime/src/visual/engine.ts) through a harness host added with page.addInitScript, so nothing is added to
the pages (AGENTS.md rule 4), in .github/workflows/visual-measure.yml. Every finding on a page that is not broken is a
candidate false report until a person has looked at it; an unlabelled finding counts as false. Five runs on the same
day: the first measured v1 as built, each of runs 2 to 4 followed changes made because of what the previous one found,
and run 5 followed a review of the checks (VISUAL.md section 19, "Review of 2026-10-13"). Summaries:
bench/visual/results/2026-10-13-run{1,2,3,4,5}.json.
| run | GitHub run | commit | what changed before it |
|---|---|---|---|
| 1 | 38484391086 | 0e84cdc |
v1, plus a fix for passes that never finished on a page that is never idle |
| 2 | 38486735268 | 5612879 |
css-var-undefined, focus-obscured, layout-shift tightened |
| 3 | 38488461191 | ac5832f |
control-unreachable: controls in open shadow roots, covers without a box |
| 4 | 38489994866 | 5c47a71 |
scroll-lock-stuck and pointer-events-leak: an expanded popup trigger counts as open |
| 5 | 38497716085 | 47b94c1 |
the review: the canvas follows the page's colour scheme, layers that ignore the pointer, scroll attempts, busy shields, dialogs that scroll inside, web components, once-per-session events, the stylesheet walk in steps of 200 rules and every read in idle callbacks, re-measured fixes; main merged in |
Run 5 is at commit 47b94c1. The commits after it on the branch move two reads into idle callbacks (a scroll
attempt's height read, a new fix's 50 ms check) and change documents only; the Chromium suite (CI run 38498154990)
passed on them. From run 5 on, the public sites' run log and plain artifacts hold counts only (bench/visual/README.md),
so their findings are not looked at from crops unless the owner adds the VISUAL_BENCH_KEY secret.
Loads: desktop 1280 x 800 and phone 390 x 844 (an Android user agent), light and dark, scripted hover over 8 controls, 8 Tab presses, scrolling to the end, a pass at the top and one at the end. Local pages run in guard mode with fixes on, so a fix on a clean page would count as a false fix; public sites run report-only.
Clean local pages (runs 4 and 5, the same). The demos are the six demo pages and their index, served as demos/README.md says; the
templates run npm run dev; the compat apps are their production builds, scenarios c, d and h.
| pages | page views | findings | confirmed real | false reports | per page view | fixes applied |
|---|---|---|---|---|---|---|
| demos, 7 pages | 28 | 2 | 2 | 0 | 0 | 0 |
| templates, 4 | 16 | 0 | 0 | 0 | 0 | 0 |
| compat apps, 7 apps x 3 scenarios | 84 | 0 | 0 | 0 | 0 | 0 |
The two findings are one real bug in two colour schemes: on the demos index at desktop width, the lede, code block and
buttons under the hero headline move 59 px 0.2 s after navigation start (a layout-shift window of 0.34). It is labelled
real in bench/visual/labels.json and the demo is left alone. The same zero false reports held in all five runs.
Public sites. 40 home pages (bench/visual/sites.json): documentation, news, government, SaaS, shops, code hosting,
an encyclopedia, design sites. One load per site and configuration, no clicks, no forms. Dark loads count only where
the site has a dark design. Each finding was looked at from its element crop in the CI artifact; sites are not named
here.
| run | sites measured | loads counted | findings | per load | confirmed real | check errors | unconfirmed |
|---|---|---|---|---|---|---|---|
| 1 | 32 | 100 | 49 | 0.49 | 5 | 44 | 0 |
| 2 | 33 | 100 | 12 | 0.12 | 5 | 5 | 2 |
| 3 | 34 | 102 | 7 | 0.069 | 5 | 1 | 1 |
| 4 | 31 | 96 | 8 | 0.083 | 5 | 0 | 3 |
| 5 | 33 | 97 | 6 | 0.062 | not looked at | not looked at | 6 |
| kind | run 1 (real / check error) | run 4 (real / check error / unconfirmed) |
|---|---|---|
css-var-undefined |
0 / 35 | none |
focus-obscured |
1 / 6 | 1 / 0 / 0 |
layout-shift |
2 / 3 | 2 / 0 / 1 |
page-overflow-x |
2 / 0 | 2 / 0 / 2 |
control-unreachable (run 2) |
0 / 5 | none |
scroll-lock-stuck (run 3) |
0 / 1 | none |
-
The 5 confirmed real findings are 3 bugs on 2 of the 40 sites, each found in two colour schemes where it applies: on a phone, a statistic 16 px wider than the 390 px viewport, so the page scrolls sideways; on a phone, a layout shift of 0.27 1.2 s after navigation start that moves the content below by 782 px; on a phone, a keyboard-focused link entirely hidden under a fixed announcement banner.
-
The 3 unconfirmed findings in run 4: a sideways scroll of 20 px on a phone with no culprit element found (runs 2 and 4, not 3), and a layout shift scored 0.25, just above the threshold, 0.2 s after navigation start (run 4 only). Neither crop shows the cause, so neither is counted as real or as a check error.
-
Not measured: 3 or 4 bot challenges, 2 HTTP 403 answers, 1 HTTP/2 protocol error and 1 site whose loads exceeded the 240 s cap in runs 1, 2 and 4. In run 4 one site answered HTTP 429 (rate limited) for its own stylesheets and images on 2 loads; the checks reported exactly that (
stylesheet-failedandimage-brokenwith HTTP 429, and what follows from an unstyled page), and those loads are left out of the rate because visitors do not normally get them. -
Run 5: 6 findings on 4 loads of 3 sites,
layout-shift4 andpage-overflow-x2 (run 4:layout-shift3,page-overflow-x4,focus-obscured1, 5 of them confirmed real). They are the kinds of the real bugs and the unconfirmed findings of run 4, but with counts only in the artifact none was looked at, so all 6 count as unconfirmed. Notext-invisible,dialog-covered,scroll-lock-stuckorpointer-events-leakfinding in any run. Not measured in run 5: 5 bot challenges, 2 HTTP errors, 1 HTTP/2 protocol error, and 4 loads a site rate-limited (HTTP 429 on its own files).
Recall on planted faults (run 4; run 5 the same except as noted below the table). Faults are applied from the harness (bench/visual/faults.ts, seeded) to the 7
demo pages, the 4 templates and the 7 compat apps (scenario h), at desktop and phone width; each fault is checked to
have taken before it counts. The planted forms are the author's, so this is recall on those forms in real app
layouts, not on every way such a bug appears.
| kind | took | found | recall | kind | took | found | recall | |
|---|---|---|---|---|---|---|---|---|
text-invisible |
70 | 70 | 1.00 | page-overflow-x |
22 | 22 | 1.00 | |
image-broken |
36 | 36 | 1.00 | viewport-meta-missing |
11 | 11 | 1.00 | |
image-css-broken |
32 | 32 | 1.00 | layout-shift |
18 | 18 | 1.00 | |
media-failed |
36 | 36 | 1.00 | page-blank |
18 | 18 | 1.00 | |
stylesheet-failed |
32 | 32 | 1.00 | click-blocked |
18 | 12 | 0.67 | |
font-failed |
28 | 28 | 1.00 | pointer-events-leak |
18 | 18 | 1.00 | |
icon-font-failed |
36 | 36 | 1.00 | scroll-lock-stuck |
36 | 24 | 0.67 | |
csp-blocked |
36 | 36 | 1.00 | control-unreachable |
32 | 32 | 1.00 | |
text-template-leak |
54 | 54 | 1.00 | focus-obscured |
18 | 18 | 1.00 | |
text-placeholder |
28 | 28 | 1.00 | dialog-covered |
18 | 18 | 1.00 | |
text-encoding |
28 | 28 | 1.00 | css-var-undefined |
28 | 27 | 0.96 | |
image-distorted |
36 | 36 | 1.00 |
- Misses: every
click-blockedandscroll-lock-stuckmiss is on one of the six demo pages, which update their DOM all the time (a live server log). Both checks wait for a moment with no DOM change (between the two clicks, or within 300 ms of a scroll attempt), so they stay silent on such pages.css-var-undefinedmissed one phone load. - Faults that did not take are left out: sideways overflow on the demo pages (their root clips it, 14); the viewport meta tag, still in effect on six demo pages (they reload through a Service Worker) and the Next.js compat app (7); a missing file that a development server answers with a page and status 200 (8); the planted bar under the devtools overlay on the templates' development servers (4); text faults on pages with too few text elements in view.
- Runtime fixes on the planted faults (guard mode, fixes on): text colour 70 of 70 applied and still holding at the
end,
pointer-events18 of 18, scroll lock 24 of 24 found; none undone. Fixes applied on clean pages: 0 in all runs. - Run 5: the same recall for every kind except
pointer-events-leak, 17 of 18 (0.94): the miss is a compat app in scenario h, whose requests were in flight during the reads; since the review a lock is not judged while a request runs (a busy shield is intended), which costs recall on pages that are never quiet. Fixes in run 5: text colour 70 of 70 andpointer-events17 of 17 applied and holding; the planted scroll locks were reported (24 of 36) but not removed: the harness sets the lock before the first periodic read and no dialog ever opens, which the review's rule treats as a lock held since load (VISUAL.md 3.20). 0 fixes on clean pages. - Other findings on fault loads: 18, of which 7 were not on the clean load of the same page (a layout shift on phone loads caused by the planted box of image faults pushing the content down).
Cost (run 4, first automatic pass; budget 8 ms per slice, 100 ms per pass).
| loads | count | pass time p50 / p95 / max | pass within 100 ms | longest slice p50 / p95 / max | elements read p50 / max |
|---|---|---|---|---|---|
| local pages, 1x CPU | 128 | 6.1 / 19.0 / 27.9 ms | 100% | 6.0 / 8.9 / 18.7 ms | 138 / 2,178 |
| local pages, 4x CPU throttling | 32 | 28.3 / 68.8 / 90.9 ms | 100% (78.1% in runs 1, 2 and 3, p95 106.9, 104.5 and 106.1 ms) | 13.5 / 18.4 / 25.6 ms | 138 / 2,178 |
| public sites, 1x | 113 | 40.5 / 97.8 / 128.2 ms | 94% | 19.5 / 57.1 / 86.1 ms | 894 / 6,593 |
| run 5: local pages, 1x | 128 | 6.8 / 32.1 / 53.4 ms | 100% | 6.2 / 10.5 / 12.5 ms | 137 / 2,135 |
| run 5: local pages, 4x CPU throttling | 32 | 32.8 / 112.4 / 113.6 ms | 78.1% | 14.3 / 18.3 / 20.1 ms | 128 / 1,133 |
| run 5: public sites, 1x | 117 | 40.3 / 95.2 / 158.4 ms | 95.7% | 8.9 / 18.9 / 51.1 ms | 897 / 6,432 |
The longest slice on public sites is well above the 8 ms slice budget in run 4. One stylesheet walk, as
css-var-undefined did it in a single step that was exempt from the slow-read rule, took 12.1 ms at the median and
31.6 ms at p95 (54.6 ms at most) on the 82 public-site loads whose stylesheets were readable, against 0.1 ms and 2.3 ms
on the local pages. In run 5 the engine walks in steps of 200 rules (the whole walk, timed at once by the harness,
still takes 11.6 / 29.9 / 71.9 ms at p50 / p95 / max on 89 readable sites): the longest slice on public sites fell to
8.9 ms at the median and 18.9 ms at p95 (from 19.5 and 57.1 ms). One load still had a 51.1 ms slice, a single step
that waited for the browser's layout. Long tasks on the page after the checks started (any cause, the page's own work
included): 19 of 128 local loads (77 ms at most), 6 of 32 throttled ones and 41 of 117 public-site loads (1,297 ms at
most); the harness cannot tell whose task it was, and the Chromium suite's heavy pages show none from the checks.
Real integration (run 4, compat apps, window.GENCLASS_CONFIG = { visual: true }, no harness). On all 7 apps the
stub requested the chunk 1.5 s after load (median 1,500 ms) and the first pass finished 2.07 s after load
(median; 2.12 s at most). No findings.
Proposed gates (VISUAL.md 15) on run 5. Met: 0 false reports on the demos and templates and 0 per compat page; 0
false fixes; recall 1.00 for text-invisible and image-broken, and at least 0.7 for 19 of the other 21 kinds. Not
known: false reports per 10 public sites (6 findings, none looked at). Not met: p95 pass time on public sites 95.2 ms
(at most 50 ms; 32.1 ms on local pages); longest slice at most 12 ms: p95 10.5 ms on local pages, 18.9 ms on public
sites; pass 1 inside 100 ms at 4x throttling on 78.1% of clean local pages (at least 90%).
Proposed gates (VISUAL.md 15) on run 4. Met: 0 false reports on the demos and templates; 0 per compat page (at most
0.05); 0 confirmed false reports per 10 public sites for every kind (at most 1); recall 1.00 for text-invisible and
image-broken (at least 0.9); at least 0.7 for 19 of the other 21 kinds (click-blocked and scroll-lock-stuck
are at 0.67); 0 false fixes. Not met: p95 pass time on public sites 97.8 ms (at most 50 ms; 19.0 ms on local pages);
longest slice at most budget + 4 ms (12 ms): p95 8.9 ms on local pages, 57.1 ms on public sites. Not reliably met: pass
1 inside 100 ms at 4x throttling on at least 90% of clean local pages: 78.1% in runs 1, 2 and 3 and 100% in run 4; the
code change before run 4 (popup triggers for two interaction checks) did not touch cost, so run 4 is the outlier, not
an improvement.
15. Interaction analytics: first-load cost (train VM, 2026-10-18, branch feat/interaction-analytics)
test/first-load-size.mjs against the merge base with main (e22fc24), minified + gzip as an app's bundler ships
them, measured on the train VM (not CI):
| entry | merge base e22fc24 |
with the analytics stub | growth |
|---|---|---|---|
/auto |
108.248 KB | 108.529 KB | 288 B |
main entry (GenClass.init({})) |
100.126 KB | 100.395 KB | 275 B |
test/bundle.test.ts measures the same build at 108.53 / 100.39 KB (script tag 128.83 KB); the comment in that test
rounds the main entry's 100.395 KB to 100.40 KB. The merge base predates the WASM warm-up fix (042c813, which
changes src/runtime.ts and src/model/backend.ts) that reached main before this branch did, so current main is
not measured here. The recording itself is dist/analytics/index.js (12.5 KB minified, not gzipped), loaded only when an app turns
analytics on. No detection numbers: analytics never feed a decision, so recall and FIR do not change.
The branch raised the bundle.test.ts limits to 108.6 / 100.48 KB, approved by the owner on 2026-10-18, and was
merged into main the same day (pull request #53). Not released yet: it is in the CHANGELOG's unreleased
0.3.0-beta.1. Design: ANALYTICS.md.
Does adding GenClass break an app? This is a summary of compat/RESULTS.md, which a script
generates from a run on 2026-10-10 (5,850 trials, 0 incomplete) and which is not edited by hand; that page has every
cell, every finding and the method. No model accuracy is reported there: it counts bugs, actions and findings on
idiomatic apps.
Setup: @genclass/runtime packed from commit d04eff3 (version string 0.1.0-beta.4, never published: the
0.1.0-beta.4 code plus the "Model ready" log fix; it does not contain 0.2.0's audit trail, protect presets, hold-budget
ceilings and invariant fixes (e84ddc7, committed on another branch) and predates 0.2.1's first-request hold and
telemetry defaults (5a90f15), which were not run through the matrix, nor was any later release (0.2.2,
0.3.0-beta.0); the version note in compat/RESULTS.md calls it the code released as 0.2.0, which the
git history does not support: e84ddc7 is not an ancestor of d04eff3), model 0.2.0 (genclass-runtime-r17 2.0.0-rc4t) on
WASM at balanced, headless Chromium 153, telemetry off. 7 apps (React 19 + Vite 8, Next.js 16 App Router, Vue 3 +
Vite 8, SvelteKit 3, Angular 22, Solid 1.9 + Vite 8, plain HTML with the CDN script tag), 15 data layers, 8 scenarios
(a and b carry a latent app bug; c to h are correct apps), 10 seeds per cell, in observe, guard and heal, each run
paired with a run without GenClass on the same seed.
| measured | result |
|---|---|
| cells (app × data layer × scenario × mode) left as correct as without GenClass on every seed | 350 of 351 |
| bugs introduced | 0 in 3,510 mode runs |
| non-passive actions on correct apps (scenarios c to h, guard and heal) | 0 in 1,760 runs |
| observe findings on correct apps (scenarios c to h) | 319 in 880 runs: 270 report a failure the scenario injects on purpose (an HTTP 500, going offline), which the app handles; 49 are false (request:overload ×45 and request:duplicate ×4, all in scenario g, polling and fan-out) |
| latent app bugs (a: stale typeahead; b: double submit) | shown in 250 of 290 runs without GenClass; guard left 249 and heal 240 of the same runs buggy; the 10 fixed by an action were all heal block on one data layer (Angular with XMLHttpRequest, double submit) |
| runs where an app's bug did not show with GenClass although it took no action | 4 (GenClass's own work shifted the timing of the app's race) |
| determinism control (two runs without GenClass per seed) | 1,170 of 1,170 pairs identical |
| automatic state discovery | React state, the Redux store and Zustand with devtools enabled: true found in 240 of 240 runs each; Zustand's devtools middleware in a production build: 0 of 240 (it turns itself off there) |
The one ✗ is observe on Next.js 16 with SWR 2, scenario d (two overlapping optimistic updates, one rolled back), seed 7: without GenClass the screen ended wrong in both runs, with GenClass it ended right, and GenClass took no action. So observe mode is not byte-identical to running without GenClass on an app with its own race: it can change which side of the race wins. Limits (from that page): small apps, one machine, headless Chromium only, WASM inference, production builds, few seeds per cell.
What GenClass costs a real page: memory, download, main-thread time and the state-discovery walk, in one production app (the pilot app of §5, a Next.js 16 ordering site) with Playwright Chromium at a 390×844 phone viewport. These tables were first recorded in the package README's "Costs" section and are kept here as the record; no model accuracy is involved, so no recall or FIR applies. Not measured: other apps, real phones, Firefox and Safari.
Runtime 0.1.0-beta.3 with model 0.2.0 (2026-10-09). Production build, 4× CPU throttling, medians of 5 runs,
observe mode; the same build with and without ?genclass=off:
| with GenClass | |
|---|---|
| JavaScript on every page (first load, gzip) | +97 KB with 0.1.0-beta.3 (today's first load: §10) |
| Main thread: load event, LCP, long tasks | no measurable change; main JS heap +0.4 MB |
| Memory: renderer process (holds the model worker) | +190 to 280 MB RSS while the model is loaded (cold 448 vs 169 MB, warm 361 vs 171 MB) |
| First visit download, WASM | 12.7 MB after the load event, as recorded then: q8 model 9.45 MB, ONNX Runtime wasm 3.07 MB brotli, tokenizer 207 KB. The READMEs round this to about 13 MB; with the 0.2.0 q8 file at its full size, 10.16 MB (§2), the same sum is about 13.4 MB. WebGPU devices: the 13.6 MB fp16 model plus the larger ONNX Runtime build |
| Later visits | about 1 KB (model.json revalidated); the rest comes from Cache Storage, checked with sha256 |
| Model load | WASM (1 thread): 0.9 to 1.3 s cold, 0.4 s warm. WebGPU (Apple M-series): 12 s cold, 2 s warm |
| App build output | 0.1.0-beta.3: +41 MB of unused ONNX Runtime files. 0.2.0: none |
Automatic state discovery, runtime 0.1.0-beta.4 (never published; its changes shipped in 0.2.0), 2026-10-10. The pilot app with only the one
line, against the same build with autoState: false; Playwright Chromium, 390×844, observe mode, telemetry off:
| cost | |
|---|---|
| JavaScript (first load, bytes on the wire) | +7 KB on /menu (discovery's share of the /auto entry; GenClass.init() setups do not load it) |
| React commit walk, 4× CPU throttling | p50 0.2 ms, p95 1.8 ms, max 2.1 ms per commit including recording the change; total about 5 ms over the /order visit |
| React commit walk, no throttling, 54 bench trials (about 2,000 commits) | p50 below 0.1 ms (the timer's resolution), p95 0.8 to 0.9 ms, max 3.8 ms |
| Main-thread JS heap (after GC, 4× throttling) | 5.7 MB vs 5.2 MB without discovery |
| Long tasks (> 50 ms) | unchanged (1 per run, 65 to 70 ms, with and without) |
AGENT.md section 7. packages/mcp/test/e2e/race.e2e.mjs on the train VM: Node 22, headless Chromium
through Playwright 1.63.0, @genclass/runtime and @genclass/mcp built from the branch, GenClass.init({ model: false }) (no model: the agent loop does not need it). The page: a search box whose results are written by each response,
without a staleness guard (buggy) or keeping only the answer to the latest query (fixed). The backend answers every
query in 40 ms, in order; the steps type "react" at 80 ms a keystroke. Every step goes through the real MCP server
(tools/call) and the real bridge. Measured before and again after the review fixes, with the same results.
| measurement | buggy page | fixed page |
|---|---|---|
| natural runs (the two explore baselines, responses in request order) | 2 of 2 right (the race never shows on this backend) | 2 of 2 right |
genclass_explore, 6 seeded orders, slow_ms: 600 |
5 of 6 ended with results for an older query ("reac", "rea", "r"); the 6th ended right after an older response overwrote newer data along the way | 0 of 6 differ from the baseline (no false report) |
genclass_replay of the race finding |
reproduced | not reproduced |
| the generated Playwright regression test | fails (expected the results for "react", received "reac") | passes |
One page and one bug class: this proves the mechanism end to end, not a recall figure. Next: explore on the six demos and the real-apps sweep, races found next to false reports on their correct variants.
Review: an adversarial review (six reviewers, a skeptic per serious finding, 55 agents) confirmed 43 defects in the
new code; all are fixed and tested, and a second verification pass confirmed 6 more issues, also fixed (AGENT.md
section 7). It also found a pre-existing XHR delivery defect, fixed in
src/observe/xhr.ts (CHANGELOG).
First load: test/bundle.test.ts on the branch measures /auto 108.54 KB and the main entry 100.40 KB (minified +
gzip; limits 108.6 / 100.48 KB): the agent link is its own entry (dist/agent.js, no imports), never part of the first
load; the XHR fix adds about 10 bytes. Tests: 36 in test/agent.test.ts, 57 in packages/mcp; the runtime suite
passes (1,101 tests, 14 model-parity skips); review-perf 4 of 4.
19. Coding agents on a real library issue: redux-toolkit#5139 (local, 2026-10-25, Chromium through Playwright 1.63.0)
docs/adoption/hunts/rtk-5139.md. The agent link and @genclass/mcp (both built from
the repository) recording a standalone Redux Toolkit 2.13.0 reproduction: an optimistic updateQueryData patch
replaced by a polling or refetch-on-focus fetch while the mutation is pending. mode: "observe", model: false,
telemetry: false, Redux discovery only, connectAgent(rt, { hold: false }). 10 natural runs per variant.
| variant | bug visible in the UI | runs with the overwrote-newer witness on the overwritten data |
other witnesses |
|---|---|---|---|
| polling, the fetch starts after the patch (the issue) | 10/10 | 1/10 (the one run whose poll was already in flight at the click) | RTK bookkeeping fields (status, requestId, startedTimeStamp) in 4 runs |
| refetch on focus | 10/10 | 0/10 | status in 10 runs |
| polling, a fetch in flight at the click (#5103's shape) | 10/10 | 10/10 | none |
| control (no polling, no focus refetch) | 0/10 | 0/10 | none (0 findings) |
The witness fires only when the overwriting request started before the patch (by its definition, AGENT.md section 2);
in the issue's own shape the fetch is newer on the page and stale only against the server, so it is not witnessed.
MCP findings over the 40 runs: 4 stale-write, 1 the bug and 3 bookkeeping (true happens-before facts, not bugs);
none from the control. The Playwright test genclass_explain generated for the bug passed 5 of 5 on the buggy app
(the witness depends on the click's timing, which a delivery-order replay does not recreate). One case, not a recall
figure.
20. The hunt harness: explore and replay against apps it does not own (cloud session, 2026-10-25, branch claude/optimistic-feynman-8lbnyg)
bench/hunt runs the same loop against an app it does not own: an init script runs the
runtime's script-tag build inline before the page's first script (and removes the element), imports the agent link
from a blob URL and connects it to an in-process bridge; no request of the app is routed or rewritten. The MCP tools
are driven over tools/call. Headless Chromium 153 through Playwright 1.63.0, runtime 0.3.0-beta.1 and mcp 0.1.0,
harness commit bd943285, 6 seeded orders, slow 600. Reports: bench/hunt/results/.
The e2e pages, served by bench/hunt/scenarios/e2e-search-server.mjs with no GenClass in the page (state in a
closure, so no store is discovered: the verdicts rest on page text and request counts). "Reordered" counts the compared
orders that delivered at least one response after the answer to a later request.
| measurement | buggy search page (/) |
fixed (/fixed) |
double submit (/form) |
fixed (/form-fixed) |
double submit, then a navigation (/form-nav) |
|---|---|---|---|---|---|
genclass_explore, 6 seeded orders |
5 of 6 ended with an older query's results ("reac", "rea", "r"); 6 reordered | 0 of 6 differ; 6 reordered (no false report) | 0 of 6 differ; 1 reordered | no verdict: nothing reordered (one request per load) | no verdict: the form page leaves before its quiet point |
| other findings on the loads | none | none | one duplicate-request (POST /api/items twice in flight from two clicks) |
none | the same duplicate-request |
genclass_replay of the race finding |
reproduced | ||||
| the generated regression test | fails on the buggy page; passes 10 of 10 against the fix at the same path | ||||
plain.mjs, no GenClass, the "reac" request answered 1.5 s late |
2 of 2 end on "reac" | 0 of 2 differ | |||
| the harness's own DOM oracle | differs in 5 of 6 | same in 6 of 6 | same | same | same |
| wall time (8 or 9 page loads, the test run included) | 22 s | 17 s | 15 s | 15 s | 17 s |
The same counts as section 18, through the harness. One page and three planted bugs: a check that the harness and the injection do not change what GenClass finds, not a recall figure. False reports on the fixed variants: 0 of 6 (search); the fixed form gives no verdict rather than a false "no race".
One real app through the same command: Strapi 5.59.0 from npx create-strapi@latest --example (sqlite), logged in,
about 3 minutes per scenario; React and Redux state discovered (300 fields, the snapshot cap). Details in the README's
section "A real app".
| measurement | the author picker, generic typeahead.mjs (results/sample/) |
two searches on the article list (results/strapi-search/) |
|---|---|---|
| compared orders (reordered) | 4 (4) | 4 (4) |
| seeded loads not compared | 2: seeds 2 and 3 stayed on Strapi's 403 page ("You don't have the permissions to access that content") | 2: the same seeds, the same page |
| GenClass's verdict | along the way only: 4 of 4 overwrote newer page-load state, all ended right | along the way only: 2 of 4 overwrote RTK Query's cache tags, all ended right |
| what the user sees | no race: in seed 1 "da" was delivered after "dav" and the picker still showed "David Doe" | no race in the table |
| replay of the race finding | reproduced | reproduced |
| the generated test on the app | passed in the hunt's run; failed on its witness 10 of 10 when repeated | fails on its witness |
plain.mjs, no GenClass |
"da" answered 2 s late: 0 of 2 differ; permissions answered 2.5 s late: the 403 page in 2 of 2 | |
| other findings | 101 stale-write witnesses |
4 stale-write witnesses |
The 403 page is an order dependency of the admin's load that the harness surfaced and plain Playwright reproduces; it
is not reported upstream, and the mechanism was not traced. An earlier run of the sample showed a GenClass false
report: under an auth-like store name the runtime redacts a random id only when it contains a digit (util.ts
OPAQUE), so a 21-character id is raw in about 3% of runs and the compare reads that as a race; its replay said "not
reproduced". util.ts is frozen (AGENTS.md rule 3), so the fix belongs in packages/mcp/src/compare.ts.
Review: before these runs, six independent reviewers read the harness (55 raw findings, merged and checked against the
code); the confirmed ones were fixed in bd943285, among them a "no race" verdict printed when nothing was compared
or nothing was reordered, the document rewrite (redirect targets never rewritten, streamed HTML buffered, non-UTF-8
mangled), a generated-test precondition that failed on about 7 of 30 loads, findings lost on steps that navigate,
and session files outside .gitignore.
21. Coding agents on a real app: Gitea's markdown preview (cloud container, 2026-10-25, adoption hunt)
adoption/hunts/gitea.md. Gitea main at ff68b39 (same code as v28.2.0), built from
source with SQLite; GenClass from main at d77ffd8 (runtime 0.3.0-beta.1, mcp 0.1.0), injected from outside through
Playwright 1.63.0 / Chromium 1243 (the script-tag build plus connectAgent, model: false, telemetry off, one rt.atom
per watched pane because GenClass discovers no store on Gitea's stack). Each explore session: 2 fifo baselines + N
seeded loads of the same steps, at slow_ms 300, 600 and 900. "Fixed" is Gitea with the prepared patch (a request
counter in each preview handler).
| surface | seeded loads ending stale, unfixed | seeded loads differing from the baseline, fixed (false reports) |
|---|---|---|
| comment box Preview (N = 6) | 6 of 18 | 0 of 18 |
| file editor Preview (N = 6) | 5 of 18 | 0 of 18 |
| file editor Preview Changes (N = 12) | 12 of 36 (0 of 18 with N = 6) | 0 of 36 |
Every load where the newer response was delivered first ended stale on the unfixed build (23 of 23) and right on the fixed one (0 of 23); baselines were right on both. Confirmed without GenClass: a Gitea e2e test fails 20 of 20 before the fix and passes 20 of 20 after, and a person-paced run on a throttled network (no reordering) ends stale 5 of 5 before and right 5 of 5 after. Misses: with N = 6 the Preview Changes race was never tried in the harmful order (the seeded priorities of its two request keys favour the older one for seeds 1 to 8); and the Actions "Run workflow" branch change, a dropped update rather than a reordering, was reported as a race in 5 of 6 at 300 ms but not at 600 or 900 ms, where the baselines lost the update too. One app, three surfaces of one bug class: not a recall figure.
22. Coding agents: explore's false reports before and after (cloud session, 2026-10-25, branch claude/great-dijkstra-oa9emj)
AGENT.md sections 2 and 5.1. The Strapi race hunt (docs/adoption/hunts/strapi.md on the
claude/dazzling-cray-8tqzkn branch) found a real race on the Strapi admin and also showed genclass_explore
reporting races on Redux Toolkit Query's bookkeeping. This measures the same sessions with the comparison of main
(d77ffd8, "before", built in a git worktree) and with this branch ("after"), on the same machine: a cloud container
(4 cores), Node 22.22.0, Chromium 1243 headless shell through Playwright 1.63.0, Strapi 5.59.0 on SQLite set up as in
the hunt's section 1 (30 authors, 40 categories, 300 published articles; the database restored from the same seed
before each batch). Harness: the hunt's hunt.mjs unchanged, one genclass_explore session per scenario and latency
(2 fifo baselines and 6 seeded orders, slow_ms 300, 600 and 900, then a replay of the first race finding), each with
its own bridge state directory; scored by bench/explore/summarise.mjs, which compares explore's verdict per
reordered run with the harness's DOM oracle (the visible rows, URL and count; the picker's options; the edit view's
title, status and buttons): a run "differs" when its oracle equals neither baseline's. 24 sessions per batch, 192 page
loads; about 30 minutes a batch.
Headline. On the 21 interaction sessions (search, sort, filter, pagination, the relation picker typed slowly and fast, double Save and Publish; the DOM oracle: 0 of 126 reordered runs differ), the race-like verdicts went from 16 of 21 sessions to 0 of 21, and the reordered runs reported as races from 62 (all false) to 0. The sessions without a usable verdict went from 5 (3 split into scenario groups with nothing to compare, 2 that said "No race found" over 0 compared runs) to 0: the reordered runs compared with two baselines went from 82 to 126 of 126. On the real race (the Content Manager list opened by its URL, which ends on the 403 page when the permissions response lands last) the recall is unchanged: 6 of 6 reordered runs that ended on the 403 page reported, in 3 of 3 sessions (300, 600 and 900 ms), the replay of the finding reproduced each time; the false runs there went from 1 to 0 (the 900 ms session's third run, a redaction flip).
| before | after | |
|---|---|---|
| interaction sessions with a race-like verdict (all false by the DOM oracle) | 16 of 21 | 0 of 21 |
| interaction sessions without a usable verdict (no verdict, or "No race found" over 0 compared runs) | 5 of 21 | 0 of 21 |
| reordered interaction runs compared with two baselines | 82 of 126 | 126 of 126 |
| reordered interaction runs reported as a race (true, false) | 62 (0, 62) | 0 (0, 0) |
| race findings made by the interaction sessions | 16 | 0 |
| the 403 race: sessions reporting it, reordered runs reported (true, false), DOM oracle | 3 of 3, 7 (6, 1), 6 differ | 3 of 3, 6 (6, 0), 6 differ |
| the 403 race: replay of the finding | reproduced 3 of 3 | reproduced 3 of 3 |
stale-write findings in the interaction sessions' bridges (one per overwrote-newer witness shape) |
2,796 | 404 |
| store fields per snapshot (most) | 300 (the cap) | 1,000 (the budget; up to 1,221 left out, and said so) |
Per session (the DOM oracle column is the harness's own check; "tiers" are where the after comparison placed each reordered run: visible is a race, internal and transient are listed apart, same is no difference):
| session | DOM oracle: seeded runs differing (before / after) | before: verdict as reported | before: runs reported / compared (false) | after: verdict as reported | after: runs reported / compared (false) | after: tiers of the reordered runs | race findings (before / after) | stale-write findings (before / after) |
|---|---|---|---|---|---|---|---|---|
| double-save-300 | 0 / 0 | no verdict | 0 / 0 (0) | no race | 0 / 6 (0) | internal 6 | 0 / 0 | 372 / 90 |
| double-save-600 | 0 / 0 | no verdict | 0 / 0 (0) | no race | 0 / 6 (0) | internal 6 | 0 / 0 | 365 / 90 |
| double-save-900 | 0 / 0 | no verdict | 0 / 0 (0) | no race | 0 / 6 (0) | internal 6 | 0 / 0 | 319 / 88 |
| filter-300 | 0 / 0 | race | 3 / 6 (3) | no race | 0 / 6 (0) | same 3, internal 3 | 1 / 0 | 9 / 1 |
| filter-600 | 0 / 0 | race | 2 / 6 (2) | no race | 0 / 6 (0) | same 3, internal 3 | 1 / 0 | 7 / 1 |
| filter-900 | 0 / 0 | race | 4 / 6 (4) | no race | 0 / 6 (0) | same 2, internal 4 | 1 / 0 | 24 / 3 |
| load-list-300 | 2 / 2 | race | 2 / 6 (0) | race | 2 / 6 (0) | same 3, visible 2, internal 1 | 1 / 1 | 1 / 1 |
| load-list-600 | 2 / 2 | race | 2 / 6 (0) | race | 2 / 6 (0) | same 3, visible 2, internal 1 | 1 / 1 | 1 / 1 |
| load-list-900 | 2 / 2 | race | 3 / 6 (1) | race | 2 / 6 (0) | same 3, visible 2, internal 1 | 1 / 1 | 1 / 1 |
| pagination-300 | 0 / 0 | race | 6 / 6 (6) | no race | 0 / 6 (0) | same 2, internal 3, transient 1 | 1 / 0 | 92 / 3 |
| pagination-600 | 0 / 0 | race | 6 / 6 (6) | no race | 0 / 6 (0) | transient 3, internal 3 | 1 / 0 | 73 / 3 |
| pagination-900 | 0 / 0 | race | 6 / 6 (6) | no race | 0 / 6 (0) | transient 3, internal 3 | 1 / 0 | 69 / 3 |
| relation-search-300 | 0 / 0 | race | 5 / 5 (5) | no race | 0 / 6 (0) | transient 2, internal 3, same 1 | 1 / 0 | 179 / 15 |
| relation-search-600 | 0 / 0 | race | 3 / 3 (3) | no race | 0 / 6 (0) | transient 3, internal 3 | 1 / 0 | 259 / 18 |
| relation-search-900 | 0 / 0 | race | 5 / 5 (5) | no race | 0 / 6 (0) | transient 2, internal 3, same 1 | 1 / 0 | 206 / 19 |
| relation-search-fast-300 | 0 / 0 | race | 4 / 4 (4) | no race | 0 / 6 (0) | transient 2, internal 3, same 1 | 1 / 0 | 178 / 14 |
| relation-search-fast-600 | 0 / 0 | no race over 0 compared | 0 / 0 (0) | no race | 0 / 6 (0) | transient 3, same 1, internal 2 | 0 / 0 | 199 / 20 |
| relation-search-fast-900 | 0 / 0 | race | 6 / 6 (6) | no race | 0 / 6 (0) | transient 3, internal 3 | 1 / 0 | 168 / 17 |
| search-300 | 0 / 0 | race | 2 / 6 (2) | no race | 0 / 6 (0) | same 3, internal 3 | 1 / 0 | 4 / 1 |
| search-600 | 0 / 0 | race | 2 / 6 (2) | no race | 0 / 6 (0) | same 3, internal 3 | 1 / 0 | 4 / 1 |
| search-900 | 0 / 0 | race | 3 / 6 (3) | no race | 0 / 6 (0) | same 2, internal 3, transient 1 | 1 / 0 | 21 / 3 |
| sort-300 | 0 / 0 | race | 3 / 3 (3) | no race | 0 / 6 (0) | same 1, internal 4, transient 1 | 1 / 0 | 96 / 4 |
| sort-600 | 0 / 0 | race | 2 / 2 (2) | no race | 0 / 6 (0) | internal 4, same 1, transient 1 | 1 / 0 | 90 / 3 |
| sort-900 | 0 / 0 | no race over 0 compared | 0 / 0 (0) | no race | 0 / 6 (0) | transient 3, internal 3 | 0 / 0 | 62 / 7 |
What moved the verdicts, and why each rule. Scored on the before runs with the after comparison alone (no runtime change), the 62 false runs fall to 6, all in the 600 ms double-save session: the comparison rules account for the rest, the runtime changes for those 6.
- Where a difference shows. Every false run before was a store field or a witness with the page unchanged: request
ids,
fulfilledTimeStamp, subscriptions, the cache keys a tag provides, hook results. A difference counts as a race only when it reaches what the page shows, its form values, its unsafe requests or its errors; store-only differences are "internal state only" and overwrites a later response repaired are "along the way only", both listed without a finding. No library field list was needed (the task's fallback, anignoreFieldsdefault for Redux Toolkit Query, was not added: the generic rule left no false report, so it would have been code without a measured purpose). - Redactions.
[redacted]against any value is equal (the 900 ms load-list false run). - Scenario grouping. Digits blanked, parenthesised counts dropped (
author (1)), a typing burst split by a label change joined, then action kinds and roles (click divandclick spanon a sorted header, used for the 3 sort sessions) when names changed with the order. - Double-save (6 false runs left by the comparison alone): the "Saved document" toast had expired in the slower reordered runs, and the number of GET refetches of the document differed. The snapshot now lists the lines of ARIA live regions (status, alert) and the comparison takes those, and counts of safe requests (GET, HEAD, OPTIONS), as internal: 0 false runs after.
- Snapshots. 1,000 store fields in an order that does not depend on writes (registered, component state, then discovered stores, shallowest first) instead of the first 300 listed; a field missing from a snapshot that left fields out is not compared.
- Witnesses. A write that only reorders a field's elements, and a path the hub's layout re-cut in the same write (an
object past 32 keys or a store past 200 fields becomes one field), are no longer
overwrote-newer. Before, many witnesses were fields appearing or disappearing in such re-cuts (Redux Toolkit Query'sconfig.*"becoming undefined" when the store grew, though the app never deletes it): the stale-write findings of the interaction sessions fell from 2,796 to 404. The remaining ones are still mostly cache bookkeeping (subscriptions, statuses) and still listed bygenclass_findings: AGENT.md section 8.
Two small apps (bench/explore/apps.mjs): a search page on Redux Toolkit Query and one on TanStack Query, each in
a variant with a planted stale race (?bug: every fulfilled answer copied into what the page renders) and a correct
variant (the documented keyed query; the RTK variant keeps the copying slice, unrendered). Typed "react" at 80 ms a
key, backend 40 ms in order, 2 baselines and 6 seeded orders at 300, 600 and 900 ms, the same builds before and after.
| session | DOM oracle: seeded runs differing (before / after) | before: runs reported / compared (true, false) | after: runs reported / compared (true, false) | after: tiers | replay (before / after) |
|---|---|---|---|---|---|
| rtk-bug-300 | 5 / 5 | 6 / 6 (5, 1) | 5 / 6 (5, 0) | visible 5, transient 1 | reproduced / reproduced |
| rtk-bug-600 | 5 / 5 | 6 / 6 (5, 1) | 5 / 6 (5, 0) | visible 5, transient 1 | reproduced / reproduced |
| rtk-bug-900 | 6 / 6 | 6 / 6 (6, 0) | 6 / 6 (6, 0) | visible 6 | reproduced / reproduced |
| rtk-ok-300 | 0 / 0 | 6 / 6 (0, 6) | 0 / 6 (0, 0) | internal 5, transient 1 | reproduced / none |
| rtk-ok-600 | 0 / 0 | 6 / 6 (0, 6) | 0 / 6 (0, 0) | internal 5, transient 1 | reproduced / none |
| rtk-ok-900 | 0 / 0 | 6 / 6 (0, 6) | 0 / 6 (0, 0) | internal 6 | reproduced / none |
| tanstack-bug-300 | 5 / 5 | 6 / 6 (5, 1) | 5 / 6 (5, 0) | visible 5, transient 1 | reproduced / reproduced |
| tanstack-bug-600 | 5 / 5 | 6 / 6 (5, 1) | 5 / 6 (5, 0) | visible 5, transient 1 | reproduced / reproduced |
| tanstack-bug-900 | 6 / 6 | 6 / 6 (6, 0) | 6 / 6 (6, 0) | visible 6 | reproduced / reproduced |
| tanstack-ok-300 | 0 / 0 | 0 / 6 (0, 0) | 0 / 6 (0, 0) | same 6 | none / none |
| tanstack-ok-600 | 0 / 0 | 0 / 6 (0, 0) | 0 / 6 (0, 0) | same 6 | none / none |
| tanstack-ok-900 | 0 / 0 | 0 / 6 (0, 0) | 0 / 6 (0, 0) | same 6 | none / none |
Recall on the planted races: 32 of 32 reordered runs whose page ended on an older query reported, before and after
(12 sessions in all, each replay reproduced). False reports on the correct variants: 18 of 36 reordered runs before
(every Redux Toolkit Query session: "RACE" on the cache's bookkeeping and the unrendered slice), 0 of 36 after; on
the planted-race sessions the runs that ended right but had an overwrite along the way were reported before (4) and
are not after. The e2e page (packages/mcp/test/e2e/race.e2e.mjs, slow_ms 600): race in 5 of 6 reordered runs
before and after, 0 of 6 on the fixed page before and after; the replay reproduced on the buggy page and not on the
fixed one; the generated regression test failed on the buggy page and passed on the fixed one, before and after.
Caveats: the DOM oracle is the harness's (secondary, the same for both batches), not ground truth beyond what it reads; for double-save it leaves out the notifications and blanks the digits of the title the scenario makes unique. The two small apps were written by the author of the rules they measure (a mechanism check on the libraries' documented patterns, not an independent recall figure). One Strapi, one seed, one machine; seeds 1 to 6 only.
23. Issue 94: request-stalled on the demos, and the first-load cost of summary().requests (local, 2026-10-25, branch issue-94)
request-stalled (VISUAL.md 3.24), default threshold 15 s. bench/stall/run.ts, headless Chromium through
Playwright 1.63.0 on a Mac, the demos built against this branch's runtime (DEMOS_OUT_DIR=... node demos/scripts/build.mjs, no demo code changed), each run a fresh page in the demos' off mode (GenClass observes, no
model; the checks are on because 127.0.0.1:<port> is a development host). The demo's own synthetic user runs its
scripted steps (window.__trial.runSynthetic()), then the page stays open 20 s more so anything still pending
(EventSource streams, the status dashboard's polls, slow answers) has time to be reported. Findings are read from the
runtime's console lines. Clean trials: no chaos, correct apps, so every finding is a false report. Chaos trials: the
demos' chaos (hangs of 6 to 8 s, slowdown spikes, failures, outages). Planted: after a clean trial, the mock server
holds every response 60 s and the same steps run again.
| demo | clean runs | clean runs with a finding (false reports) | chaos runs | chaos runs with a finding | planted runs | planted runs found | planted findings naming a control |
|---|---|---|---|---|---|---|---|
| search | 10 | 0 | 10 | 0 | 3 | 3 | 0 of 26 |
| editor | 10 | 0 | 10 | 0 | 3 | 3 | 0 of 6 |
| checkout | 10 | 0 | 10 | 0 | 3 | 0 | 0 of 0 |
| status | 10 | 0 | 10 | 0 | 3 | 0 | 0 of 0 |
| board | 10 | 0 | 10 | 0 | 3 | 3 | 0 of 6 |
| decisions | 10 | 0 | 10 | 0 | 3 | 2 | 0 of 2 |
| all | 60 | 0 | 60 | 0 | 18 | 11 | 0 of 40 |
- False reports: 0 in 120 runs (60 clean, 60 chaos), with the board's EventSource stream and the status dashboard's polls open through the 20 s hold. The chaos runs' longest answers (8 s hangs) stay under the threshold, as intended.
- Recall on the planted hang: 11 of 18 runs. Status (0 of 3): its requests are polls on a schedule the cadence layer learned, which the check skips by design (the long-polling rule), so a poll that hangs is not reported. Checkout (0 of 3) was not diagnosed: the replayed steps may send no new request once the first order is placed. Decisions missed one of 3.
- No planted finding named a control: the demos that stalled (typeahead, autosave, kanban, photo grid) do not disable
a control or set
aria-busywhile they wait. The control path (the reporter's disabled button) is covered by the unit tests (test/visual-stall.test.ts), not measured on an app yet. - A typeahead that stalls reports one finding per stalled request (26 in 3 search runs), as specified (one check per request); the console folds them only by the per-pass line limit.
First-load cost. node packages/runtime/test/first-load-size.mjs against main's build on the same machine:
/auto 108.354 KB vs 108.320 KB (+34 B), the main entry 100.259 KB vs 100.225 KB (+35 B): the flag the fetch observer
sets when the app reads a non-JSON answer with response.json(). The request counting itself, the per-host
snapshot and request-stalled are in the measured checks' chunk. A first version that counted in the first load cost
+583 / +556 B and needed the owner-approved limits raised (test/bundle.test.ts), so it was moved; CI's build
measures about 0.2 KB more than this machine for both branches.
24. Measured visual checks ad-heavy and text-warped (development Mac, Chromium 153 through Playwright 1.63.0, 2026-10-25, branch strange-sites)
Measured checks, not model results (VISUAL.md sections 3.25 and 3.26). Run on a 16 GB development Mac, not on GitHub runners, so the cost figures are this machine's and only the comparison with main's engine on the same machine, the same day, means anything. The faults and the fixtures were written by the author of the checks, so the recall below is a mechanism check (each planted fault is found, nothing else is), not an independent recall figure.
Fixtures (test/browser/visual, Chromium; the whole suite, 49 tests, passed):
| fixture | planted | found | other findings |
|---|---|---|---|
ads.html (ad slots and ad-sized cross-site frames over 30% of the page) |
1 ad-heavy (density) |
1 | 0 |
ads-interstitial.html (a full-screen ad with a close button) |
1 ad-heavy (interstitial, high) |
1 | 0 |
warped.html (10 faults across the 7 reasons, 5 clean controls next to them) |
10 text-warped |
10, each with its reason | 0 |
clean-strange.html (two ad slots, video and map embeds, a sticky header, a cookie banner, an ellipsis, a line clamp, a masked "read more", a flip card, a carousel track, a rotated axis label, a 10° slant, tracking at 0.2 em and a logo at 0.6 em, a tight display heading, ruby, emoji, right-to-left text, vertical text, a backdrop word, a quarter-scale preview with pointer-events: none, a floating label), desktop and phone |
0 | 0 | 0 |
| the 44 existing fixture tests (their exact finding sets) | unchanged | unchanged | 0 new |
The first browser run had one false report on two clean elements (overlap on text with text-overflow: ellipsis,
VISUAL.md section 19); after the change, none.
Demos (bench/visual, group demos: the six demos and their index, desktop 1280 x 800 and phone 390 x 844, light
and dark, scripted hover, Tab and scrolling; faults T1 and ad-heavy only, --faults T1,ad-heavy):
and the four starter templates the same way (npm run dev, prepared with corpus.mjs prepare templates):
| clean pages, every kind | page views | findings | false reports | per page view | fixes applied |
|---|---|---|---|---|---|
| demos, 7 pages | 28 | 0 | 0 | 0 | 0 |
| templates, 4 | 16 | 0 | 0 | 0 | 0 |
| kind | group | planted | took | found | recall | desktop | phone |
|---|---|---|---|---|---|---|---|
text-warped (7 reasons per load) |
demos | 98 | 95 | 95 | 1.00 | 48 / 48 | 47 / 47 |
text-warped |
templates | 56 | 15 | 15 | 1.00 | 7 / 7 | 8 / 8 |
ad-heavy |
demos | 14 | 14 | 14 | 1.00 | 7 / 7 | 7 / 7 |
ad-heavy |
templates | 8 | 8 | 8 | 1.00 | 4 / 4 | 4 / 4 |
The templates are small pages: after the first two text faults no free text element was left in view for the others
(40 not applicable), so their text-warped figure covers mostly overlap and clipped. The demos' fault loads ran
twice, before and after the cost change below, with the same result.
Not applicable (left out of the denominator): on the index page at phone width no text element in view had two words
for spacing or was free for tiny; on the status page the planted half-emoji left the viewport; on one template the
clipped line did not take. Every ad-heavy
load also reported layout-shift on the content below the inserted slots: the harness inserts them after load, so
the shift is real, and it is the only finding outside the planted faults. No fix was applied anywhere (neither check
has one).
Cost on the demos, the same machine, main's engine (10f0c1d) against this branch (7 loads at 4x CPU throttling,
28 at 1x; the check time of the automatic first pass, the longest slice, the session's check time):
| first pass p50 / p95 | first pass inside the 100 ms budget | longest slice p95 / max | session p50 / p95 | |
|---|---|---|---|---|
| 1x, main | 15.1 / 26.3 ms | 100% | 9.6 / 9.6 ms | 39.1 / 62.4 ms |
| 1x, this branch | 21.4 / 36.3 ms | 100% | 10.5 / 10.9 ms | 47.9 / 75.4 ms |
| 4x, main | 70.5 / 87.2 ms | 100% | 15.3 / 15.3 ms | 94.8 / 127.8 ms |
| 4x, this branch | 78.8 / 112.3 ms | 86% (6 of 7) | 13.9 / 13.9 ms | 106.3 / 162.8 ms |
The first measurement of this branch was slower (4x first pass p50 98.1 ms); reading each element's line boxes and
transforms once per pass and building element descriptions only for findings brought it to the figures above (on the
10,000-element fixture big.html the session's check time went from 136 to 48 ms, longest slice 8.1 ms). Long
tasks during the 4x loads come from the throttled pages themselves (main: 6 of 7 loads, up to 258 ms; this branch: 6
of 7, up to 160 ms); the checks' own slices stay under 16 ms.
First load (test/first-load-size.mjs against main's dist built from 10f0c1d): /auto 108.324 KB against
108.320 KB, the main entry 100.227 KB against 100.225 KB. The first-load code is byte-identical (checked by diffing the
two bundles); the 4 B and 2 B are the content-hashed file names of the two lazy chunks it imports. The lazy chunks grow:
dist/visual/index.js 40.38 to 47.06 KB gzip (+6.69 KB), dist/analytics/index.js 3.69 to 4.26 KB gzip (+0.57 KB), each loaded only where its
feature is on.
Not run: the compat apps and the public sites for these two kinds (the bench workflow, visual-measure.yml, which
loads about 40 public sites and is the owner's to start); Firefox and WebKit.
Cost only. Observe mode took no action and these numbers are not detections, so no recall or false-intervention rate applies. Not GenClass's own bench and not GenClass's defaults: one Next.js 16.3 app (the pilot app of §5 and §17 and of the package README's "Costs"), measured by its owner on a desktop Mac. GenClass's own tests measure the first-load size (§10, §13, §15) but no load times or memory in a browser.
Method. Production build (Turbopack), served by next start on localhost; Chromium with a phone-sized viewport
and 4× CPU throttling; 5 cold and 5 warm runs per arm, medians. GenClass is behind a build-time switch (package
README, "Only in some builds (Next.js)"). With it on, @genclass/runtime/auto/observe loads first and synchronously
in instrumentation-client.ts, before hydration, with options that are not the defaults: telemetry with
decisionSample: 1 and maxDecisions: Infinity (an options object, so it also sends from localhost) and
include.situation: false, detect: "possible", visual: true, learn: { expectations: true }, debug on
localhost, and the model self-hosted (WASM q8, device: "wasm", 1 thread). Being on localhost,
development-host defaults applied too. The app wraps GenClass in a small module of its own (it keeps the last 500
events for its tests), which is in the JavaScript numbers. Control: the same commit with the switch off, which the
first table shows ships no GenClass code.
Builds (next build, gzip -9 of every chunk; deltas against the commit before GenClass was added):
| before GenClass was added | GenClass added, switch off | GenClass added, switch on |
|---|---|---|
| 52 JS chunks, 773,548 B gzip | 52 JS chunks, 773,612 B gzip (+64 B; +161 B raw), the only addition the empty instrumentation-client hook; 0 files that mention GenClass (grep -ril genclass .next/static) |
67 files, 1,048,345 B gzip (+274,797 B; +806,550 B raw) with the lazy worker, ONNX Runtime and visual chunks; 0 .wasm files in .next |
Pages (switch off against switch on, the same server setup):
| off | on | how | |
|---|---|---|---|
| JavaScript before the load event (on the wire) | / 346.0 KB, /menu 285.7 KB |
/ 457.4 KB, /menu 397.1 KB (+111.4 KB each) |
resource timing, cold, median of 5 |
| load event | / 222 ms, /menu 213 ms |
/ 295 ms, /menu 369 ms (+73 to +156 ms) |
medians of 5 |
| first long task | / 57-71 ms |
/ 116-132 ms (about +55 ms); warm loads: none off, 1 or 2 of 50-62 ms on |
PerformanceObserver longtask |
| LCP | / 156 ms, /menu 136 ms |
/ 116 ms, /menu 148 ms (no regression above the noise) |
medians of 5 |
| main-thread JS heap | +1.5 to 1.8 MB | ||
| memory per renderer process (renderer and model worker) | / 77 MB cold, 86 MB warm; /menu 49 MB cold, 58 MB warm |
/ 249 / 208 MB; /menu 224 / 175 MB (+117 to +175 MB) |
macOS top -stats mem 12 s after load, median of 5 (RSS was too noisy to use) |
| first visit download | 13,708 KiB (about 14.0 MB) from the app's own origin, of which the q8 model is 9,922 KiB (10.2 MB, sent uncompressed); about 1 KB on later visits | Playwright requestfinished sizes, worker requests included |
|
| model load | / 0.85 s, /menu 1.01 s cold (runs 0.78 to 1.24 s); / 0.37 s, /menu 0.39 s warm (runs 0.36 to 0.57 s) (WASM q8, 1 thread) |
status.loadMs: from the start of the model load (its card) to ready, on the runtime's clock; medians of 5 |
|
| model ready | 1.25 to 1.5 s cold, 0.75 to 0.85 s warm (medians per page; warm runs 0.72 to 1.03 s; WASM q8, 1 thread) | the status ready event's time after navigation (the page's time origin, page startup included): rt.summary().startedAt plus the event's offset from the start of the app's wrapper, which starts just after the runtime, so about performance.now() at ready (not ready minus startedAt) |
|
| model latency per decision | 97 to 127 ms | Decision.latencyMs: from the call to the worker to its answer, on the main thread's clock (not the worker's compute time alone) |
Units are binary, as the app's harness computed them, though the Pages table writes KB and MB: the JavaScript sizes
are KiB (resource timing transferSize / 1,024), the heap and the memory per process are MiB (JSHeapUsedSize /
1,048,576; macOS top counts in binary units), and the download is in KiB as written, with decimal MB in parentheses
(the app's own report labelled those KiB values KB). The Builds table counts bytes. Not measured: what made the first long task longer (evaluating the entry before hydration, or the runtime's install
work), GenClass's defaults on a deployed host, and phones. The 0.1.0-beta.3 numbers in the package README (same app,
2026-10-09) used the same build loaded with ?genclass=off as the control, which still loads GenClass's code, and
renderer RSS for memory; they are not comparable with these.
26. What a production build keeps of the lines genclass_setup prints (development Mac, 2026-11-05, @genclass/runtime 0.3.0-beta.5)
GenClass.init() inside the development check against the older form, GenClass.init() outside it with only
connectAgent inside. "Runtime strings" searches the output for __GENCLASS_AGENT__, the diagnostics notice and the
telemetry opt-out key. No false-intervention rate applies: nothing here is a detection.
| build | printed lines | JavaScript written | runtime strings |
|---|---|---|---|
Vite (vitest's copy), import.meta.env.DEV |
inside the check | 702 bytes | none |
| Vite, same app | outside the check | 918,333 bytes | yes |
esbuild, process.env.NODE_ENV set to "production" |
inside the check | 33 bytes, the app's own line | none |
| Next.js 15.5.29 (webpack), a client component in the root layout | inside the check | First Load JS 103 kB, what the framework ships | none, in .next/static and .next/server/app |
Not measured: Turbopack, Parcel, and a plain webpack project. packages/mcp/test/mcp.test.ts repeats the esbuild row on
the exact lines setupText prints for Vite and Next.js, whenever the runtime is built.
27. Explore's verdict on real apps: false reports and recall before and after (development Mac, Chromium through Playwright 1.63.0, 2026-11-05, branch fix/explore-verdict-noise)
AGENT.md section 5.1. On 2026-11-04 four hunts ran genclass_explore next to their own check of the page, on released apps and on fixed builds. This measures the comparison of origin/main (91139b92, "before") against this branch ("after") on those reports, on OpenCode's own recorded runs and on small pages made of what they met. Two independent reviews of earlier versions of this branch found rules that hid real races main reports (a wrong value that reads like a time, a lost POST, a stale write from a hosted API, a swap beside a new row) and one that reported from a single sample; every number here is measured after those fixes unless it says otherwise.
Headline. Reordered runs reported as a race on fixed builds, where the hunts' own check said the page was right: 85 of 120 before, 0 of 120 after, of which only 36 rest on evidence the reports list (below). On every session of the four apps whose check was right in all its loads: 211 of 466 before, 24 of 466 after. Recall on the bugs the reports could show: sessions with a wrong run and a race verdict go from 35 of 37 to 25 of 37; the 10 are 9 OpenCode sessions whose "RACE" was a read-only POST count (the same verdict on the fixed build, so never evidence) and Notes at 600 ms, which needs the new load. On OpenCode's recorded runs the correct scenarios went from race to no race (0 of 6 where main said 3, 6, 6 and 5 of 6) and the real file bug from an indistinct count to a slow response's race. On planted pages: false reports on correct variants 49 of 360 runs (12 of 66 sessions) before, 18 of 396 (6 of 66) after, sessions with a planted bug given a race verdict 38 of 54 before, 48 of 54 after, with 0 false runs among them after. The 18 false runs left are a save queue's PUT count and clocks moving, both false on main too.
Stored reports. bench/explore/stored.mjs parses the 136 explore reports kept as text back into runs (what each listed, over one synthetic baseline) and gives them to compareRuns; the page check is each file's load N: lines. "Listed cap" counts cleared runs whose list hit its cap of 4 page lines a side.
| reports | sessions | reordered runs compared | reported before (all false by the page check) | reported after | listed cap |
|---|---|---|---|---|---|
| OpenCode, fixed file refresh | 3 | 12 | 9 | 0 | 0 |
| OpenCode, fixed file reopen | 3 | 12 | 9 | 0 | 0 |
| n8n, fixed Execute Sub-workflow picker | 3 | 18 | 18 | 0 | 0 |
| Open WebUI, fixed searches | 9 | 54 | 43 | 0 | 43 |
| Langflow, fixed New Flow and paging | 4 | 24 | 6 | 0 | 6 |
| fixed builds | 22 | 120 | 85 | 0 | 49 |
| OpenCode, every session whose check was right in all loads | 16 | 68 | 52 | 8 | 0 |
| n8n, the same | 28 | 98 | 48 | 0 | 5 |
| Open WebUI, the same | 27 | 162 | 60 | 13 | 43 |
| Langflow, the same (the 51 of 138 of the hunt) | 23 | 138 | 51 | 3 | 32 |
| all sessions whose check was right in all loads | 94, of which 7 compared nothing (2 OpenCode, 5 n8n) | 466 | 211 | 24 | 80 |
What this rescoring can and cannot say:
- 49 of the 85 cleared runs had capped lists (Open WebUI 43 of 43, Langflow 6 of 6). A line behind the cap is unknown, so what the listed evidence clears is 36 of 85. The other 49 listed only ages, and their page check said the page was right.
- The "sent directly" evidence for a POST cannot fire here: a report has no send steps. So a double submit with a different non-zero count reads as internal in this rescoring even when a live run would keep it. No stored run depended on it.
- No earlier row measured a time-shaped wrong value or a lost POST: none of the stored reports or earlier benches has one, which is how the first version of this branch passed them. They are measured below, on planted pages and the review's inputs.
- A field with a generated name listed as absent on one side is taken to be there with the same value (a reading of the text), checked against the only raw session of that hunt kept. Runs here have no times, so two ages are one line within one minute, the least a real session gets. It has no baseline with no added delay.
The 24 left, by what they list: OpenCode new-session (4) and double-enter (4), a stub's reply caught mid-stream at the quiet point and session labels that came with it; Open WebUI rename-two (5: "Rename two" against "Renamed second", which may be a real bug its check does not read), settings-toggle (3: "Title Auto-Generation") and note-autosave (5: "Today at 3:27 PM" against 3:28, a clock moving on a correct page, which the first version hid along with real wrong values and is now reported: the cost of not taking clocks for ages); Langflow "New Project (1)" (2), "Delta" (1). Causes that fell away: ages, read-only POST counts, generated names, the star count.
| sessions with a run the page check called wrong | sessions | wrong runs | race verdict before, after | wrong runs reported before, after | right runs reported before, after |
|---|---|---|---|---|---|
| OpenCode | 15 | 55 | 15, 6 | 39, 12 | 13, 5 |
| n8n | 10 | 27 | 10, 10 | 27, 27 | 6, 0 |
| Open WebUI | 4 | 9 | 4, 3 | 9, 8 | 8, 3 |
| Langflow | 8 | 35 | 6, 6 | 20, 15 | 2, 0 |
| all | 37 | 126 | 35, 25 | 95, 62 | 29, 8 |
What the drop is. OpenCode, 27 runs in 10 sessions (9 lost their verdict): each rested on a count such as POST /api/worktree/refresh, and the fixed builds got the same verdict from the same line, so these were coincidence, not recall. Four of the sessions are wrong the same way in all their loads (file reopen, file open), which depends on no timing and no order comparison can see; the other six are the dropped refresh, the new load's case, measured below. Open WebUI, 1 run: Notes at 600 ms. Langflow, 5 runs: their only listed difference was the star count 156k; those runs' real difference was a second flow or variable on the server, not in the list. On the same reports the real lines stay reported: n8n's "helper task 29", "row-040", "Total 41"; Langflow's "New Flow (1)"; OpenCode's "MARKER running-2 question".
What each rule costs, on the stored reports.
- POST. A different non-zero count, with the page the same and nothing that makes it a write, is demoted; a POST one side never sent stays visible. Of the 113 POST lines in the reports, 106 stay demoted, all of them non-zero:
POST /api/worktree/refresh89,opencode.goal/get9,authz/me/permissions4,workflow-dependencies/counts4. 7 stay visible, one-sided:POST /api/session/:id/view, 3 runs innew-sessionand 4 insend-two, runs already reported for page lines, so no verdict moves. Keeping lost POSTs visible costs 0 false runs here. On the live pages below it is what bringslost-postback. - Ages only. Taking clocks, days and bare forms as values adds 5 false runs on correct sessions (
note-autosave) and 3 on a session with a real bug (Notes at 900 ms: "2m" against "3m", on runs its check called right), and brings back 2 true ones there. Against that, all 17 hidden inputs below report. - Both slow baselines. The report does not have the load, so this has no cost here; the planted pages and OpenCode's runs below have it.
OpenCode 2.2.5, recorded runs. The review ran OpenCode itself (slow_ms 600, 6 seeds, runtime 0.3.0-beta.5, its page checks), on the commit before these fixes; the "now" column is this comparison on the same recorded runs, no new load.
| scenario | sessions | page check | main |
on the commit before these fixes | now |
|---|---|---|---|---|---|
file-refresh (a real bug) |
2 | stale in 2 of 2 slow baselines, 0 of 1 no-delay, 5 of 6 seeded | 5 of 6 in both, on request counts | RACE, a slow response alone | RACE, a slow response alone, "draft-one", 2 of 2; seeded 0 of 6 |
switch |
3 | right in 27 of 27 loads | 3, 6 and 6 of 6 | no race in 3 of 3 | no race in 3 of 3 |
palette |
1 | right in 9 of 9 | 5 of 6 | no race | no race |
new-session |
1 | right in 9 of 9 by a narrow check | 6 of 6 | RACE, a slow response alone, plus 4 of 6 | RACE, a slow response alone, plus 4 of 6: the stub's reply and the composer's text; it may be a false report, and the check reads only that one session was created |
mention |
1 | wrong in 9 of 9 by its check, the same way in all | 5 of 6 | no race | no race: nothing depends on timing |
"Worth a look" prints on 6 of the 8 sessions (POST /api/session/:id/view absent without the delay, or opencode.goal/get sent 4 times against 2 or 3): read-only POSTs still make noise there, under that line.
The review's inputs (scratchpad/review-explore/adv.mjs, adv2.mjs, hidden.mjs; each is now a test in packages/mcp/test/compare.test.ts). compareRuns on the same synthetic runs:
| inputs | main reports a race on |
the commit before these fixes | now |
|---|---|---|---|
| a wrong value that reads like a time (17: "Your appointment: 3:30 PM" against 4:00 PM, "Arrives today", "Ships in 2 days", "Last synced 3 days ago" against "just now", "3d" against "5d", ...) | 17 | 1 | 17 |
a wrong value in a field with a generated-looking name (6: qty-100 and qty-200 swapped, :r1: and :r2: swapped, ...) |
6 | 3 | 6 |
| noise beside a real difference (6) | 1 | 4 | 4 (all of main's, and "version-2", "Order 125 total 90", "Build 123 status 9") |
| POST counts (9) | 9 | 4 | 5: a lost POST by either route, direct double submit, in flight, PUT. The 4 not reported are a different non-zero count behind a request or a timer, or with no steps recorded |
correct pages with slow_ms (12), races given |
3 on seeded runs | 4 on seeded runs, plus 8 "slow response alone" | 2 on seeded runs, both also on main, plus 1 "slow response alone" (a load time equal by chance in both slow baselines) |
| another site from localhost (3) | 3 | 1 | 2: the stale count and its labelled form. The POST sent twice stays in the counts, as internal, under the POST rule |
Decided for the localhost page with a hosted API: only safe requests leave the counts, since from localhost a hosted API of the page's own and someone else's look alike, and the page's own network never counts as another site (host.docker.internal, a private address, a name with no dot, .local, .internal). A bare count is internal only when it stands for a store field exactly one request wrote, the same one in both runs, to another site, requested once, with no overwrite recorded on the field, and its digits are that field's: so a stale cart count stays a race, and Langflow's star count (one request, a rate limit's null) does not. In Chromium (hidden.mjs, slow_ms 300, the page wrong in 4 of 6 loads): slot-clock, slot-day and slot-days went from no race to RACE in 4 of 6, as main; lost-post from no race to RACE, 2 of 6 with slow_ms and 2 of 4 without, as main on the same runs; the random tip, on the first version a false "slow response alone" with 2 of 6, here read 3 of 6 on seeded runs, which main gives on the same runs: all three baselines landed on one value, and nothing can tell that from a race; queue-put still RACE in 4 of 6, as main.
The second review's inputs (scratchpad/review2/a1.mjs, a2.mjs, a6.mjs, live.mjs; each is now a test). Races reported on the same synthetic runs:
| inputs | main |
the commit before these fixes | now | what stays unreported |
|---|---|---|---|---|
a load's GET over a POST's answer from a hosted API, two GETs writing one field, a price, a backend on localhost (4) |
3 of the 3 it can read | 1 | 4 | nothing |
a swap or a wrong value beside a new row (qty-100, qty-200, qty-9001) (3) |
3 | 0 | 3 | nothing |
| ages, both directions and rounded phrases (19) | 19 | 4 | 16 | "2 minutes" then "3 minutes" in 10 s, "a few minutes" against "just now", "over a year" against "a year": each an age that can have moved |
| parsing ("2 hours and 5 minutes", "1,000 days") (9) | 9 | 4 | 7 | "1,000 minutes" then "1,001", "1,000 seconds" then "1,030": the same |
| field names (10) | 10 | 5 | 7 | names made for each load swapping values, a noisy one beside a wrong one, or holding a value (total-9999): not told from nothing |
| POST rule (6) | 5 | 3 | 3 | a different non-zero count behind a timer or with no action recorded, named by "No race found" |
Also: five adjacent [redacted] in a state value against 199 characters took over 15 s and now about 4 ms; the plain load alone on an error page gives no race and no finding when nothing from the slow side differs, and RACE, on one load's word when one line stands in for another and no other load sides with it; a direct double submit no longer prints under both RACE and "worth a look". None of the numbers in this section's other tables moved with these fixes.
Planted pages. bench/explore/planted.mjs: small pages in headless Chromium with the real runtime and MCP server, each scenario with a correct variant and, where it has one, a planted bug; slow_ms 300, 600 and 900, 2 slow baselines, 1 with no added delay, 6 seeded orders (9 loads a session, 120 sessions, 2 browsers). Each load has its own check of the page or of what the server received. Before and after are scored on the same recorded runs (before without the third baseline, which its plan did not have). "Reported" counts seeded runs: a run that ends like the slow baselines is not reported after, because the session's headline is that difference, so read the sessions column first. Written by the author of the rules they measure: they show the mechanism and what each rule costs, not an independent recall figure.
| scenario | sessions with a race verdict, before | after | runs reported of compared, before (true, false) | after (true, false) | after: names a slow response alone | after: worth a look |
|---|---|---|---|---|---|---|
times: ages crossing a "minute" every 30 s, correct |
0 of 3 | 0 of 3 | 0 of 18 (0, 0) | 0 of 18 (0, 0) | 0 | 0 |
times: ages crossing a "minute" every 30 s, planted bug |
3 of 3 | 3 of 3 | 16 of 18 (16, 0) | 16 of 18 (16, 0) | 0 | 0 |
clocks: a clock, "today", a bare "7m" moving on that clock, correct |
2 of 3 | 2 of 3 | 6 of 18 (0, 6) | 6 of 18 (0, 6) | 0 | 1 |
counts: read-only POSTs whose number depends on order, correct |
3 of 3 | 0 of 3 | 9 of 18 (0, 9) | 0 of 18 (0, 0) | 0 | 0 |
fields: generated names, correct |
3 of 3 | 0 of 3 | 18 of 18 (0, 18) | 0 of 18 (0, 0) | 0 | 0 |
fields: generated names, planted bug |
3 of 3 | 3 of 3 | 18 of 18 (9, 9) | 9 of 18 (9, 0) | 0 | 0 |
slowonly: needs only a slow response (Notes), planted bug |
3 of 3 | 3 of 3 | 18 of 18 (18, 0) | 17 of 18 (17, 0) | 3 | 0 |
fileslow: a refresh dropped during a slow read (OpenCode), a bare counter beside it, planted bug |
0 of 3 | 3 of 3 | 0 of 18 (0, 0) | 4 of 18 (4, 0) | 1 | 0 |
counter: "version-1" against "version-2" beside a bare counter, planted bug |
0 of 3 | 3 of 3 | 0 of 18 (0, 0) | 3 of 18 (3, 0) | 0 | 0 |
route: a new path on every load, planted bug |
0 of 3 | 3 of 3 | 0 of 0 (0, 0) | 16 of 18 (16, 0) | 0 | 0 |
optional: a click only some loads did, planted bug |
0 of 3 | 3 of 3 | 0 of 0 (0, 0) | 16 of 18 (16, 0) | 0 | 0 |
dbl-shown: a double submit the page lists, planted bug |
3 of 3 | 3 of 3 | 11 of 18 (6, 5) | 0 of 18 (0, 0) | 3 | 3 |
dbl-silent: the page does not show it, planted bug |
3 of 3 | 3 of 3 | 5 of 18 (0, 5) | 0 of 18 (0, 0) | 3 | 3 |
dbl-body: the same with a body that differs, planted bug |
3 of 3 | 3 of 3 | 5 of 18 (0, 5) | 0 of 18 (0, 0) | 3 | 3 |
dbl-chain: sent behind another request, planted bug |
0 of 3 | 0 of 3 | 0 of 18 (0, 0) | 0 of 18 (0, 0) | 0 | 3 |
dbl-timer: sent behind a 30 ms timer, planted bug |
2 of 3 | 0 of 3 | 4 of 18 (0, 4) | 0 of 18 (0, 0) | 0 | 3 |
lost-post: a save sent only when one answer beat another, planted bug |
3 of 3 | 3 of 3 | 9 of 18 (9, 0) | 9 of 18 (9, 0) | 0 | 0 |
slot-clock: "Your appointment: 3:30 PM" against 4:00 PM, planted bug |
3 of 3 | 3 of 3 | 12 of 18 (12, 0) | 12 of 18 (12, 0) | 0 | 0 |
slot-day: "Arrives today" against tomorrow, planted bug |
3 of 3 | 3 of 3 | 12 of 18 (12, 0) | 12 of 18 (12, 0) | 0 | 0 |
slot-in: "Ships in 2 days" against 5, planted bug |
3 of 3 | 3 of 3 | 12 of 18 (12, 0) | 12 of 18 (12, 0) | 0 | 0 |
slot-ago: "Last synced 3 days ago" against just now, planted bug |
3 of 3 | 3 of 3 | 12 of 18 (12, 0) | 12 of 18 (12, 0) | 0 | 0 |
slot-bare: "3d" against "5d", planted bug |
3 of 3 | 3 of 3 | 12 of 18 (12, 0) | 12 of 18 (12, 0) | 0 | 0 |
tip: a random two-value line, correct |
1 of 3 | 1 of 3 | 4 of 18 (0, 4) | 0 of 18 (0, 0) | 1 | 2 |
queue: a save queue, fewer PUTs when answers are slow, correct |
3 of 3 | 3 of 3 | 12 of 18 (0, 12) | 12 of 18 (0, 12) | 0 | 1 |
the other 16 correct variants (slowonly, fileslow, counter, route, optional, five dbl-*, lost-post, five slot-*) |
0 of 48 | 0 of 48 | none (0, 0); route and optional compared 0 |
0 of 288 (0, 0) | 0 | 0 |
| correct variants | 12 of 66 | 6 of 66 | 49 of 360 (0, 49) | 18 of 396 (0, 18) | 1 | 4 |
| planted bugs | 38 of 54 | 48 of 54 | 146 of 288 (118, 28) | 150 of 324 (150, 0) | 13 | 15 |
What the POST rule costs, measured. "No race found" now says when a POST count differed, that a duplicate write may have happened with the page unchanged, and to check what the server holds. On the planted pages a lost POST is back (lost-post 3 of 3 sessions, 9 of 18 runs, 0 false). The direct rule keeps the double submits sent from a click handler (dbl-shown, dbl-silent, dbl-body: 9 of 9 sessions, each as a slow response's race, 2 of 2 baselines; before, the same sessions got their verdict with 15 of 21 reported runs false, correct runs listed against wrong baselines). It loses the double submit behind another request or a timer that never overlaps with the page unchanged: dbl-timer 2 of 3 sessions before, 0 of 3 after (the 2 before rested on 4 correct runs reported as racing; dbl-chain was 0 of 3 both ways), all 6 named under "worth a look", none a finding. identical-in-flight rarely helped on these pages (3 of 101 runs that sent two POSTs): the witness counts a request's own start and end, the answers came back in 40 ms, and two clicks were 120 ms apart; only the schedule held them longer. On the hunts: of the recorded sends in the stored explanations, every read-only POST that made a false report was sent via another request (/api/worktree/refresh 60 of 60, opencode.goal/get 29, workflow-dependencies/counts 38, authz/me/permissions 40), and Langflow's real POST /api/v1/variables/ double submit directly from two actions (4 of 4). Langflow's other one, POST /api/v1/flows/, was seen both ways (3 direct, 5 via GET /api/v1/variables/ from one action): when the page shows no second flow it is internal now.
What requiring both slow baselines costs. A bug that shows in one slow baseline and not the other is noise, as on main, and named under "worth a look" with how many reordered runs show it, not a race: OpenCode's file refresh when its baselines split (the first hunt's file-refresh-600: stale, right), and Langflow's New Flow at 600 ms. Both slow baselines landing on one value of a random line, and the plain load on another, is still a false "slow response alone" (tip: 1 of 3 sessions; by chance 1 in 4).
Small apps and the end-to-end checks, for regressions. bench/explore/apps.mjs (Redux Toolkit Query, TanStack Query; section 22's bench, 12 sessions, now 9 loads each): recall on the planted stale race 32 of 32 runs before and after, with 0 false both; false reports on the correct variants 0 of 36 both; every replay reproduced. packages/mcp/test/e2e/race.e2e.mjs: race in 5 of 6 reordered runs on the buggy page, 0 of 6 on the fixed one, replay reproduced and not reproduced, the generated test failed and passed. bench/hunt e2e pages: race on the buggy page, no race on the fixed one, and the form that sends one POST per click a duplicate-request finding with no race, as before.
Report text. On the hunts' Notes at 900 ms the sample before named "2m", "8", "3m", "Loading..." and never the differing "Maple note" row; the lists are now in reading order and the headline names the row. On planted pages the headline names the planted line in 48 of 48 sessions with a verdict (that it shows "Last synced 3 days ago", which the baseline does not; that form field "el-id-*-33" is "default one"; that requests sent to "POST /api/save" are "(absent)", "1" in the baseline). The Notes report itself cannot be shown again without its app.
Caveats: one machine, seeds 1 to 6 only, other agents running on it (load 4 to 11). The stored rescoring reads text, not the runs: the limits above apply, and it says nothing about the new baseline. A page check is the hunt's own and narrow: a run it calls right can differ for a real reason (rename-two, new-session), and "false" means by that check. Real apps with the new baseline: OpenCode only, one build, the review's four scenarios. Not run: n8n, Open WebUI and Langflow with the new load (no installs, no room), the demos (each has its own runtime and shell on the page, so the harness's injected one would make two), the model suite, realapps/.