Skip to content

Explore: a verdict that holds on real apps - #168

Merged
Karanvir1729 merged 11 commits into
mainfrom
fix/explore-verdict-noise
Nov 5, 2026
Merged

Karanvir1729 merged 11 commits into
mainfrom
fix/explore-verdict-noise

Conversation

@Karanvir1729

@Karanvir1729 Karanvir1729 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Four hunts on 2026-11-04 ran genclass_explore next to their own page checks. On fixed builds where the page was right it said RACE in 85 of 120 reordered runs, and it missed real bugs. Two independent reviews of this branch then found rules that hid real races main reports, and one that reported from a single sample. All are fixed here; the numbers are after those fixes.

  • An age ("13 minutes ago", "an hour ago", "just now", "10m ago") is one line when the later load is older by about the time between the two, plus a minute. "26 minutes ago" then "27" is one line; "just now" against "3 days ago", and "12 minutes ago" then "2" ten minutes on, are not. A clock, "today", "in 2 days" and a bare "3d" are values a page can get wrong, compared as they are unless the baselines show them varying.
  • A POST one side never sent stays a race, however it was sent. Only a different non-zero count, with the page the same, needs evidence: two identical requests in flight, or a user action sending it directly a different number of times. No URL lists.
  • A field name some baseline lacks compares with itself by shape (el-id-*-32, *-input). A name every baseline has is a name, whatever row beside it is new, so qty-100 and qty-200 swapping values is a race.
  • With slow_ms the plan adds a baseline with no added delay (runs + 3 loads). What both slow baselines show the same and it does not is reported as what a slow response alone does, and says "on one load's word" when no other load sides with it. What one slow baseline alone does stays noise, as on main, and is named under "worth a look", as is a line only that load shows.
  • Noise blanks only the tokens that varied. "version-1" against "version-2" stays a difference, and another number on that line may drift a little and no further.
  • The report orders page content first, and the headline names that line. A report that compared nothing says NO VERDICT, never beside a race finding.
  • Routes group by shape, and an action only some loads did no longer splits a session.
  • Safe requests to other sites leave the counts. Writes stay: from localhost a hosted API of the page's own looks like anyone's. A bare count is internal only when it stands for a store field one such response alone wrote, with no overwrite on it, so a stale cart count from a hosted API stays a race. They still join the schedule, which needs the runtime.
  • Matching redactions no longer backtracks: a page could hang the bridge for over 15 s with one value.
  • The review's inputs are tests, and bench/explore/planted.mjs has pages for wrong values that read like a time and for a lost POST.

Measured (RESULTS section 27):

  • Fixed builds, rescored from stored reports: false reports 85 of 120 to 0 of 120. 49 of the 85 had capped lists (Open WebUI 43 of 43), so listed evidence clears 36.
  • Sessions whose page check was right: 211 of 466 runs to 24 of 466, over 94 sessions, 7 of which compared nothing.
  • Sessions with a wrong run that get a RACE verdict: 35 of 37 to 25 of 37. Nine are OpenCode verdicts that rested on a read-only POST count, the same on the fixed build, and one needs the new load.
  • OpenCode 2.2.5, recorded runs (slow_ms 600): file-refresh RACE as a slow response's where main said 5 of 6 on counts; switch and palette no race where main said 3, 6, 6 and 5 of 6; new-session still RACE, possibly false.
  • Planted pages, false runs on correct variants: 49 of 360 to 18 of 396 (a save queue's PUT count and clocks moving, both false on main too).
  • Planted bugs given a verdict: 38 of 54 to 48 of 54, with 28 false runs to 0.
  • The first review's inputs: the 17 wrong values that read like a time are reported in 17 of 17, as main, up from 1 of 17 on this branch's first version; hidden.mjs reports all of them in Chromium, and the lost POST.
  • The second review's inputs: the stale write from a hosted API and the swap beside a new row are reported, 7 of 7, up from 1 of 7. What stays unreported is listed in RESULTS: ages that can have moved, and names made for each load swapping values.
  • RTK and TanStack: 32 of 32 found, 0 of 36 false, unchanged.

Cost:

  • A duplicate POST behind another request or a timer, with no overlap and the page unchanged, is internal: dbl-timer 2 of 3 sessions to 0 of 3, dbl-chain 0 of 3 both ways, all 6 named under "worth a look". "No race found" says when a POST count differed, that a duplicate write may have happened, and to check the server. Of the stored reports' 113 POST lines, 106 stay demoted, which is what moved the nine OpenCode verdicts above, and 7 stay visible, which moved no verdict.
  • Clocks and bare forms are compared as values: 5 more false runs on correct stored sessions (note-autosave, "Today at 3:27 PM"), 3 on a session with a real bug.
  • A bug that shows in only one slow baseline is noise, as on main, and named under "worth a look": OpenCode's refresh when its baselines split, Langflow's New Flow at 600 ms.

Limits: the no-delay baseline has run on one real app only (OpenCode). Both slow baselines landing on one value of a random line can read as a slow response's race (tip: 1 of 3 sessions). Field names made for each load pool their values, so two of them swapping is not seen. The rescoring reads text and has no send steps, so "sent directly" cannot fire in it. No earlier row measured a time-shaped wrong value or a lost POST.

Skill text changes (--repeat-each 9, "worth a look"), so the plugin re-syncs. No runtime change, and no model-visible text changed.

🤖 Generated with Claude Code

Karanvir1729 and others added 11 commits November 5, 2026 01:47
…d field names, routes, how much a line says)

timeless() puts one token in place of "13 minutes ago", "an hour ago", "just now", "yesterday", "10m ago", a line that is
only "7m", a clock. fieldShape() blanks a name's generated part ("el-id-*-32", "*-input"); routeShape() blanks id-like
segments; lineWeight() tells real content from short labels from counts. Nothing uses them yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… POSTs, generated names, a baseline with no added delay)

Four hunts ran explore next to their own page checks: on fixed builds where the page was right it said RACE in 85 of 120
reordered runs, and it missed real bugs. compare.ts, by cause:
- Relative times compare by shape whatever the baselines did.
- A POST count alone is no race: only with two identical requests in flight, or a user action sending it directly a
  different number of times (from the run's steps and witnesses, no URL list). PUT, PATCH, DELETE as before.
- Generated field names compare with themselves; fields of one shape pool their values.
- With slow_ms the plan gets a baseline with no added delay, after the two slow ones (runs + 3 loads). What it differs in
  from them is reported first as what a slow response alone does (a race finding replaying the slow schedule); a seeded
  run like it is not listed; it settles what the slow baselines disagree on (two values, a line of real content, an
  error, a write with evidence). A bare number and a short label only one shows stay noise.
- Noise blanks only the tokens that varied, at their position, each standing for its own shape; a line both baselines
  show is never noise.
- Report: page content before labels before counts, in a run's list and in which runs come first; the headline and the
  title name that line; store values capped with a count of the rest.
- Routes group by shape, also when armed and replayed; an action only some loads did no longer splits a session; a report
  that compared nothing says NO VERDICT.
- Requests to other sites leave the counts unless a user action or the page's own origin led to them; a bare count on the
  page beside the field such a response wrote is internal. They still join the schedule (that needs the runtime).
- A race finding's report says to fix only if the user asked for a fix.

The header says what is matched now: text shapes, none of them a bug pattern. Tools: only the report's text changes in
tools.ts. The replay and the generated test read relative times and generated names by shape too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he hunts met, and the plan's third baseline

stored.mjs parses reports a hunt kept only as text back into runs for compareRuns (its header says what the text cannot
show). planted.mjs runs small pages, each with a correct variant and a planted bug, and scores before and after on the
same recorded runs. apps.mjs and hunt.mjs follow the plan's length; summarise.mjs scores the older comparison without the
baseline its plan did not have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….md 5.1 and 8, the skill, README, changelog)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…false reports on fixed builds to 0, the recall next to it)

Rescored from the 136 reports the 2026-11-04 hunts kept, and measured on planted pages in Chromium (39 of 198 false runs to
0 of 234; 20 of 36 planted bugs to 30 of 36). Says what the rescoring cannot see, what the POST rule costs, and that the
baseline with no added delay has not been run on a real app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline, the tool description, RESULTS section 27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n length, lost POSTs, both slow baselines for a slow response's race)

Independent review of the verdict work: two rules hid real races main reports, one reported from a single sample.
- Only an age ("N unit ago", "just now") moves on its own, and only by how long the session ran plus a minute: "26 minutes
  ago" against "27" is one line, "just now" against "3 days ago" is not. A clock, today and tomorrow, a future form and a
  bare "3d" are values, compared as they are unless the baselines show them varying.
- A POST one side never sent is a lost or an extra write and stays visible. Only a different non-zero count, with the page
  the same, needs evidence.
- A slow response's race needs both slow baselines ending the same way, unlike the load with no added delay. One of them
  against it is one load against one load: noise as on main, named under "worth a look" with how many reordered runs show
  it. The load with no added delay no longer decides what is noise. A count of writes that only changed with the delay,
  and a number a little off, go under "worth a look" too. A seeded run is dropped only for what the headline already says.
- A race finding and NO VERDICT never stand together, and with no run at a quiet point NO VERDICT prints.
- A field name is generated only when it differs between the baselines: qty-100 and qty-200 swapping values is a race, and
  a replay reads such a field by its name. A field with nothing but a generated name gets a note in the generated test.
- Another number on a line that varied may drift a little and no further ("CPU 13% memory 41%", not "Order 125 total 90").
- Safe requests alone leave the counts: from a localhost page a hosted API of its own looks like another site. A bare count
  beside a field another site's one response wrote stays internal; two requests can answer out of order, so that stays a race.
- bench/hunt no longer counts internal differences as a race, and counts a slow response's.

Each of the review's inputs is a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd a lost POST, plus a tip, a save queue and clocks

Each slot page keeps a value that reads like a time (a clock, a day, "in 2 days", an age, a bare "3d") and shows the older
answer when the bug is planted; lost-post sends its save only when one answer beat another. tip, queue and clocks are
correct pages that vary by themselves, what those rules cost in false reports. The ages page moves no further than a session
could. stored.mjs reads the new rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lost POSTs, both slow baselines, "worth a look")

AGENT.md 5.1 and 8, the skill, the README and the changelog say the corrected rules. RESULTS 27 is redone: rescored stored
reports (85 of 120 to 0 of 120 on fixed builds, of which 36 by listed evidence; 211 of 466 to 24 of 466), what each rule
costs there, OpenCode 2.2.5's recorded runs, the review's inputs, and planted pages for wrong values that read like a time
and a lost POST (false runs 49 of 360 to 18 of 396; planted bugs 38 of 54 to 48 of 54).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I, a swap beside a new row), ages by direction

- A bare count is internal only when it stands for a store field exactly one request wrote, the same in both runs, to
  another site, requested once, with no overwrite recorded on the field, and its digits are that field's. The load's GET
  landing over a click's POST from a hosted API, and two GETs writing one field, are races. "$45.00" is not bare, and a
  backend on the page's own network (host.docker.internal, a private address, a name with no dot) is not another site.
- Only names some baseline lacks pool: qty-100 and qty-200 swapping values is a race beside a new row, whatever its value.
- An age is compared by direction, from the runs' own load and snapshot times: a later load can only be older by about the
  time between them. lastSeen no longer decides, so reporting an hour later changes nothing. "2 hours and 5 minutes" and
  "1,000 days" parse; rounded phrases stay near what a library shows ("a month" is not 40 days, "a few years" not 9).
- A line only the load with no added delay shows is no race (an error page it landed on): named under "worth a look". One
  line in place of another with no other load on its side says "RACE, on one load's word", and to load it once more.
- A direct double submit prints once. A price one slow baseline alone shows is named. "$1,000" is one number.
- Matching redactions walks positions instead of backtracking: five in a row against 199 characters went from over 15 s to
  about 4 ms, and a page can send that.
- "No race found" says what it covers, and when a POST count differed that a duplicate write may have happened and to check
  the server.

Each input is a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he one-load wording, sessions that compared nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Karanvir1729
Karanvir1729 merged commit cc78bfc into main Nov 5, 2026
5 checks passed
@Karanvir1729
Karanvir1729 deleted the fix/explore-verdict-noise branch November 5, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant