Explore: a verdict that holds on real apps

bench/explore/README.md

@@ -2,14 +2,17 @@
 
 Measures `genclass_explore` (docs/runtime/AGENT.md section 5) the way an agent uses it: two fifo baselines and
 seeded reorderings of the same user steps, compared by the MCP server. Every number goes to
-[RESULTS.md](../../docs/runtime/RESULTS.md) section 22 with its false-report rate next to it.
+[RESULTS.md](../../docs/runtime/RESULTS.md) (sections 22 and "explore's verdict on real apps") with its false-report rate
+next to it.
 
 ## What is here
 
 | file | what |
 |---|---|
 | `summarise.mjs` | scores a directory of explore sessions: per reordered run, whether explore reported it as a race, against a DOM oracle the harness recorded for the same load (a run "differs" when its oracle equals neither baseline's). With `--old <packages/mcp/dist of another checkout>` it scores the older comparison on the same recorded runs. |
 | `apps.mjs` | the two small apps below in headless Chromium with the real runtime and MCP server; writes sessions `summarise.mjs` reads |
+| `planted.mjs` | small pages made of what the 2026-11-04 hunts and the review of the fix met (ages, read-only POSTs, double submits, a lost POST, generated field names, bugs that need only a slow response, a bare number beside a version, a new path per load, an optional click, wrong values that read like a time, clocks moving, a random tip, a save queue), each with a correct variant and, where it has one, a planted bug; scores each session itself against each load's own check, with `--old` the older comparison on the same runs |
+| `stored.mjs` | rescoring reports a hunt kept only as text: parses the listed differences back into runs and gives them to `compareRuns`; its header says what the text cannot show. `--raw <state.json>` compares a bridge's recorded runs directly |
 | `apps/` | a search page on Redux Toolkit Query and one on TanStack Query (`src/rtk.jsx`, `src/tanstack.jsx`), each with a planted stale race (`?bug`) and a correct variant; installed on its own (`npm install` in `apps/`, not a workspace) |
 
 The Strapi sessions come from the hunt harness on the `claude/dazzling-cray-8tqzkn` branch
@@ -28,6 +31,15 @@ node bench/explore/summarise.mjs /tmp/explore-apps
 # the same with the comparison of another checkout (a git worktree of main, built)
 node bench/explore/apps.mjs /tmp/explore-apps-before --rt ../before/packages/runtime/dist --mcp ../before/packages/mcp/dist --slow 300,600,900
 node bench/explore/summarise.mjs /tmp/explore-apps-before --old ../before/packages/mcp/dist
+
+# the pages made of what the hunts met; shards run side by side, each with its own browser, and write into one directory
+node bench/explore/planted.mjs /tmp/planted --shard 0/2 --port 41700 &
+node bench/explore/planted.mjs /tmp/planted --shard 1/2 --port 41700 &
+wait
+node bench/explore/planted.mjs /tmp/planted --score --old ../before/packages/mcp/dist
+
+# reports a hunt kept as text
+node bench/explore/stored.mjs <hunt results directory>
 ```
 
 Not in CI: it needs Chromium, the apps' dependencies and (for Strapi) a running Strapi.

bench/explore/apps.mjs

@@ -125,12 +125,14 @@ async function session(browser, name, bug, slow) {
   };
   const result = { scenario: `${name}-${bug ? "bug" : "ok"}`, slow_ms: slow, runs: RUNS, loads: [] };
   result.start = await tool("genclass_explore", { action: "start", runs: RUNS, slow_ms: slow });
-  for (let i = 0; i < RUNS + 2; i++) result.loads.push(await load(i));
+  // the plan says how many loads: 2 baselines, with slow_ms a third with no added delay, and the seeded ones
+  const planned = Number(/next (\d+) page loads/.exec(result.start)?.[1] ?? RUNS + 2);
+  for (let i = 0; i < planned; i++) result.loads.push(await load(i));
   result.report = await tool("genclass_explore", { action: "report" });
   const fid = /Race finding\(s\): (f\d+)/.exec(result.report)?.[1];
   if (fid) {
     result.replay = { id: fid, arm: await tool("genclass_replay", { action: "arm", id: fid }) };
-    result.replay.load = await load(RUNS + 2);
+    result.replay.load = await load(planned);
     result.replay.result = await tool("genclass_replay", { action: "result", id: fid });
   }
   writeFileSync(join(OUT, `${tag}.json`), JSON.stringify(result, null, 2));

bench/explore/stored.mjs

@@ -0,0 +1,205 @@
+// Scores this checkout's comparison on explore reports a hunt kept only as text (docs/runtime/RESULTS.md, "explore's verdict on real apps"):
+// each report's groups are parsed back into recorded runs (what the report listed per run, over one synthetic baseline)
+// and given to compareRuns() from packages/mcp/dist. The harness's own page check is read from the same file's lines
+// `load N: DIFFER|same|ERROR ...`.
+//
+//   node bench/explore/stored.mjs <dir> [--json <file>]
+//
+// It can only move what a report listed: a run's first 4 page lines per side and first 6 fields. So
+//   - a line hidden by those caps stays hidden: runs whose list hit the cap and came out clean are counted `capped`
+//     (their verdict is not known from the text), not as cleared;
+//   - a baseline that shared a bug, or a difference filed as noise by shape, cannot come back from the text: fixes that
+//     need a new load (the baseline with no added delay) or the raw lines are measured on real pages instead;
+//   - a report has no send steps, so a POST count here never has "sent directly by a user action" as evidence;
+//   - a run's list was cut at 6 fields and 4 lines a side, and a partner filed as noise by the old rule is not listed: so a
+//     field with a generated name listed as absent on one side (its partner was noise, or cut) is taken to be there with
+//     the same value, and a line that holds an age is dropped from a list that hit its cap or a report whose noise holds
+//     an age. Both are this script's reading of the text, not the comparison: the raw runs a hunt kept check the first
+//     (`--raw <state.json>` compares a bridge's recorded runs directly);
+//   - the runs here have no times, so two ages are one line within one minute, the least a real session gets.
+
+import { readdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
+import { join, relative, resolve } from "node:path";
+import { pathToFileURL } from "node:url";
+
+const args = process.argv.slice(2);
+const dir = resolve(args[0] ?? ".");
+const opt = (name) => (args.includes(name) ? args[args.indexOf(name) + 1] : undefined);
+const here = new URL(".", import.meta.url);
+const { compareRuns, shownField, fieldShape, agesIn } = await import(pathToFileURL(resolve(opt("--dist") ?? new URL("../../packages/mcp/dist", here).pathname, "index.js")).href);
+
+const walk = (d) => readdirSync(d).flatMap((f) => (statSync(join(d, f)).isDirectory() ? walk(join(d, f)) : [join(d, f)]));
+const strings = (s) => [...s.matchAll(/"((?:[^"\\]|\\.)*)"/g)].map((m) => JSON.parse(`"${m[1]}"`));
+
+/** One report's groups as written: its scenario, its noise line and, per listed run, what it listed. */
+function parseReport(text) {
+  const groups = [];
+  let g = null;
+  let section = null;
+  let run = null;
+  for (const line of text.split("\n")) {
+    let m;
+    if ((m = /^## (.*?) · (\d+) run\(s\) · scenario/.exec(line))) {
+      g = { route: m[1], runs: Number(m[2]), scenario: [], races: 0, compared: 0, listed: [], noVerdict: false };
+      groups.push(g);
+      section = "scenario";
+      run = null;
+      continue;
+    }
+    if (!g) continue;
+    if (section === "scenario" && line.startsWith("  ")) {
+      g.scenario.push(line.trim());
+      continue;
+    }
+    if ((m = /^RACE: (\d+) of (\d+)/.exec(line))) {
+      section = "visible";
+      g.races = Number(m[1]);
+      g.compared = Number(m[2]);
+      continue;
+    }
+    if ((m = /^No race found: none of the (\d+)/.exec(line))) {
+      section = "none";
+      g.compared = Number(m[1]);
+      continue;
+    }
+    if (line.startsWith("Ignored as noise") && /\bminutes? ago|\bhours? ago|an hour ago|a minute ago/.test(line)) g.timeNoise = true;
+    if (/^(No verdict|Only one baseline|No baseline run)/.test(line)) {
+      g.noVerdict = true;
+      section = "none";
+      continue;
+    }
+    if (line.startsWith("Internal state only")) {
+      section = "internal";
+      continue;
+    }
+    if (line.startsWith("Along the way only")) {
+      section = "transient";
+      continue;
+    }
+    if ((m = /^- run (\S+) \((.*?)\)/.exec(line))) {
+      run = { id: m[1], label: m[2], tier: section, onlyBase: [], onlyRun: [], fields: [], errors: [], witnesses: [] };
+      g.listed.push(run);
+      continue;
+    }
+    if (!run || !line.startsWith("    ")) continue;
+    if ((m = /^ {4}page text only in the baseline: (.*)$/.exec(line))) run.onlyBase = strings(m[1]);
+    else if ((m = /^ {4}page text only in this run: (.*)$/.exec(line))) run.onlyRun = strings(m[1]);
+    else if ((m = /^ {4}(field|state|requests sent to) ("(?:[^"\\]|\\.)*"): baseline ("(?:[^"\\]|\\.)*"), this run ("(?:[^"\\]|\\.)*")/.exec(line))) run.fields.push({ where: m[1] === "requests sent to" ? "requests" : m[1], name: JSON.parse(m[2]), baseline: JSON.parse(m[3]), run: JSON.parse(m[4]) });
+    else if ((m = /^ {4}error only here: (.*)$/.exec(line))) run.errors.push(JSON.parse(m[1]));
+    else if ((m = /^ {4}witness only here: (.*)$/.exec(line))) run.witnesses.push(m[1]);
+  }
+  return groups;
+}
+
+/** A group's listed runs back into recorded runs: two equal baselines and one run per seed, their lines and fields as listed. */
+function runsOf(g, plan) {
+  const page = { url: `http://localhost:1${g.route}`, route: g.route };
+  const steps = [{ k: "act", i: 0, t: 0, kind: "click", target: 'button "x"' }, ...writes];
+  // what the listing's caps and the old noise rule hid: see the header
+  for (const r of g.listed) {
+    r.capped = r.onlyBase.length >= 4 || r.onlyRun.length >= 4;
+    if (r.capped || g.timeNoise) {
+      const partnerHidden = (l) => !!agesIn(l);
+      r.onlyBase = r.onlyBase.filter((l) => !partnerHidden(l));
+      r.onlyRun = r.onlyRun.filter((l) => !partnerHidden(l));
+    }
+    r.fields = r.fields.filter((f) => !(f.where === "field" && fieldShape(f.name) !== f.name && (f.baseline === "(absent)" || f.run === "(absent)")));
+  }
+  const count = new Map();
+  const fields = new Map();
+  for (const r of g.listed) {
+    const n = new Map();
+    for (const l of r.onlyBase) n.set(l, (n.get(l) ?? 0) + 1);
+    for (const [l, c] of n) count.set(l, Math.max(count.get(l) ?? 0, c));
+    for (const f of r.fields) if (f.baseline !== "(absent)") fields.set(`${f.where}|${f.name}`, f.baseline);
+  }
+  const baseText = [...count].flatMap(([l, c]) => Array(c).fill(l));
+  const snapshotOf = (r) => {
+    const text = [...baseText];
+    for (const l of r?.onlyBase ?? []) text.splice(text.indexOf(l), 1);
+    text.push(...(r?.onlyRun ?? []));
+    const s = { t: 1000, state: {}, text, fields: {}, requests: Object.fromEntries(writes.map((w) => [w.by, 1])), witnesses: 0, errors: 0, forced: 0 };
+    const put = (where, name, v) => {
+      if (where === "state") s.state[name] = v;
+      else if (where === "field") s.fields[name] = v;
+      else s.requests[name] = Number(v);
+    };
+    for (const [k, v] of fields) put(k.slice(0, k.indexOf("|")), k.slice(k.indexOf("|") + 1), v);
+    for (const f of r?.fields ?? []) {
+      if (f.run !== "(absent)") put(f.where, f.name, f.run);
+      else if (f.where === "state") delete s.state[f.name];
+      else if (f.where === "field") delete s.fields[f.name];
+      else delete s.requests[f.name];
+    }
+    return s;
+  };
+  const run = (id, order, seed, r) => ({ id, started: 0, lastSeen: 0, page, schedule: { order, ...(seed ? { seed } : {}), label: r?.label }, steps, lost: 0, order: [], forced: 0, witnesses: [], errors: (r?.errors ?? []).map((message) => ({ t: 1, name: "Error", message, chain: [] })), detections: [], interventions: [], checks: [], expectations: [], snapshot: snapshotOf(r), snapshots: 1, ended: true });
+  const seeds = plan.map((label, i) => {
+    const listed = g.listed.find((r) => r.label === label);
+    return run(listed?.id ?? `s${i}`, "seeded", i + 1, listed ?? { label });
+  });
+  return [run("b1", "fifo"), run("b2", "fifo"), ...seeds];
+}
+
+const rows = [];
+const files = walk(dir).filter((f) => f.endsWith(".txt"));
+// the writes other sites' responses made (a finding's explanation lists them, in some reports of a hunt): the same fields in all
+const writes = [...new Map(files.flatMap((f) => [...readFileSync(f, "utf8").matchAll(/ write (\S+) by ([A-Z]+ [^\s/]+\.[^\s/]+\/[^\s:#]*)/g)]).map((m) => [m[1], { k: "write", t: 1, store: "x", paths: [m[1]], by: m[2] }])).values()];
+if (opt("--raw")) {
+  const st = JSON.parse(readFileSync(opt("--raw"), "utf8"));
+  const byId = new Map(st.runs.map((r) => [r.id, r]));
+  for (const x of st.explores) {
+    const runs = x.runs.map((id) => byId.get(id)).filter(Boolean);
+    for (const g of compareRuns(runs)) console.log(`${x.id} ${g.route}: compared ${g.compared}, ${g.diffs.map((d) => `${d.label}: ${d.tier}`).join(", ") || "no differences"}; noise ${g.noise.filter((n) => !n.startsWith("state")).slice(0, 6).join(", ")}`);
+  }
+  process.exit(0);
+}
+for (const file of files) {
+  const text = readFileSync(file, "utf8");
+  if (!/^Explore x\d+: /m.test(text)) continue;
+  const planned = [...text.matchAll(/^ {2}\d+\. (seed \d+)$/gm)].map((m) => m[1]);
+  const loads = [...text.matchAll(/^load (\d+): (DIFFER|same|ERROR)/gm)].map((m) => m[2]);
+  const oracleOf = (label) => (/^seed (\d+)$/.test(label) ? loads[Number(label.slice(5)) + 1] : undefined);
+  const baselineWrong = loads.slice(0, 2).filter((l) => l === "DIFFER").length;
+  const groups = parseReport(text);
+  const row = { file: relative(dir, file), oracle: loads.map((l) => (l === "DIFFER" ? "D" : l === "ERROR" ? "E" : "s")).join(""), compared: 0, seeds: 0, wrongSeeds: 0, baselineWrong, before: { races: 0, tp: 0, fp: 0, fn: 0 }, after: { races: 0, tp: 0, fp: 0, fn: 0 }, capped: 0, cleared: [], left: [] };
+  for (const g of groups) {
+    if (g.noVerdict || !g.compared) continue;
+    row.compared += g.compared;
+    const plan = planned.slice(0, g.compared);
+    const report = compareRuns(runsOf(g, plan))[0];
+    const raced = new Map((report?.diffs ?? []).filter((d) => d.tier === "visible").map((d) => [d.label, d]));
+    for (const label of plan) {
+      const o = oracleOf(label);
+      const before = g.listed.find((r) => r.label === label);
+      const wasRace = before?.tier === "visible";
+      const isRace = wasRace && raced.has(label);
+      row.seeds++;
+      if (o === "DIFFER") row.wrongSeeds++;
+      const tally = (t, hit) => {
+        if (hit) t.races++;
+        if (hit && o === "DIFFER") t.tp++;
+        if (hit && o === "same") t.fp++;
+        if (!hit && o === "DIFFER") t.fn++;
+      };
+      tally(row.before, wasRace);
+      tally(row.after, isRace);
+      if (wasRace && !isRace) {
+        if (before.capped) row.capped++;
+        row.cleared.push(label);
+      }
+      if (isRace) {
+        const d = raced.get(label);
+        row.left.push(...d.textOnlyRun, ...d.textOnlyBaseline, ...d.fields.filter(shownField).map((f) => f.name));
+      }
+    }
+  }
+  row.left = [...new Set(row.left)].slice(0, 8);
+  rows.push(row);
+}
+rows.sort((a, b) => a.file.localeCompare(b.file));
+if (opt("--json")) writeFileSync(opt("--json"), JSON.stringify(rows, null, 2));
+console.log("| report | page check per load | compared | before: races (true, false, missed) | after: races (true, false, missed) | cleared with a capped list | still listed |");
+console.log("|---|---|---|---|---|---|---|");
+const cell = (t) => `${t.races} (${t.tp}, ${t.fp}, ${t.fn})`;
+for (const r of rows) console.log(`| ${r.file} | ${r.oracle} | ${r.compared} | ${cell(r.before)} | ${cell(r.after)} | ${r.capped} | ${r.left.map((l) => JSON.stringify(l.slice(0, 40))).join(", ")} |`);

bench/explore/summarise.mjs

@@ -10,7 +10,7 @@
 //
 // A reordered run "differs" when its DOM oracle equals neither baseline's (a title the scenario makes unique per run
 // has its digits blanked). A run explore "reports" is one in a group with two baselines whose difference reaches the
-// race verdict: any difference for the old comparison (it reported every one as "RACE"), the visible tier for the new.
+// race verdict: the visible tier (any difference for a comparison older than tiers, which reported every one as "RACE").
 
 import { existsSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
 import { join, resolve } from "node:path";
@@ -41,8 +41,8 @@ function groupsOf(impl, runs) {
   return [...m.values()];
 }
 
-/** Per reordered run: compared with two baselines, reported as a race, and its tier. `isNew`: only the visible tier is a race. */
-function score(impl, runs, isNew) {
+/** Per reordered run: compared with two baselines, reported as a race, and its tier. Only the visible tier is a race (a comparison older than tiers: any difference). */
+function score(impl, runs) {
   const out = new Map();
   const reports = impl.compareRuns(runs);
   const groups = groupsOf(impl, runs);
@@ -53,7 +53,7 @@ function score(impl, runs, isNew) {
       if (r.schedule?.order === "fifo") continue;
       const d = diffs.get(r.id);
       const tier = !d ? "same" : d.tier ?? (d.fields.length || d.textOnlyBaseline.length || d.textOnlyRun.length || d.errors.length ? "ended differently" : "witness only");
-      out.set(r.id, { compared: two, reported: two && !!d && (!isNew || d.tier === "visible"), tier, grouping: g.grouping });
+      out.set(r.id, { compared: two, reported: two && !!d && (d.tier === undefined || d.tier === "visible"), tier, grouping: g.grouping });
     }
   });
   return out;
@@ -71,11 +71,13 @@ for (const f of readdirSync(dir).filter((x) => /^[a-z-]+-\d+(-[a-z]+)?\.json$/.t
   const runs = x.runs.map((id) => byId.get(id)).filter(Boolean);
   // the DOM oracle per run (the harness's own check, secondary)
   const oracle = new Map((d.loads ?? []).filter((l) => l.run).map((l) => [l.run, l]));
+  // the slow baselines and the one with no added delay: a run like any of them ended in a state a baseline did
   const baseKeys = new Set((d.loads ?? []).filter((l) => l.schedule?.order === "fifo").map((l) => key(l.oracle)));
   const seeded = runs.filter((r) => r.schedule?.order !== "fifo");
   const truth = new Map(seeded.map((r) => [r.id, oracle.has(r.id) ? !baseKeys.has(key(oracle.get(r.id).oracle)) : null]));
-  const sNew = score(cur, runs, true);
-  const sOld = old ? score(old, runs, false) : null;
+  const sNew = score(cur, runs);
+  // the older comparison, on what its plan recorded: without the baseline with no added delay (it would read as a baseline)
+  const sOld = old ? score(old, runs.filter((r) => !(r.schedule?.order === "fifo" && !r.schedule?.delay) || !runs.some((o) => o.schedule?.order === "fifo" && o.schedule?.delay))) : null;
   const tally = (s) => {
     if (!s) return null;
     let compared = 0, reported = 0, tp = 0, fp = 0, fn = 0;

bench/hunt/README.md

@@ -10,7 +10,7 @@ What one run does:
 1. starts the GenClass MCP server in-process (`@genclass/mcp`, its bridge on a free loopback port) and headless
    Chromium;
 2. arms `genclass_explore` (two baselines with responses in request order, then N seeded orders, optionally slower,
-   like a bad network) and, for each planned load, opens a fresh browser context, puts `@genclass/runtime` on the page
+   like a bad network, which adds a third baseline with no added delay) and, for each planned load, opens a fresh browser context, puts `@genclass/runtime` on the page
    (see "How the injection works"), connects the agent link to the bridge, waits for the page's first quiet point (as
    the generated regression test does), runs the scenario's `steps(page)`, waits for the next quiet point
    (`whenSettled`), and records the scenario's `oracle(page)`;