Repository navigation
Explore: a verdict that holds on real apps - #168
Merged
Merged
Conversation
…d field names, routes, how much a line says)
timeless() puts one token in place of "13 minutes ago", "an hour ago", "just now", "yesterday", "10m ago", a line that is
only "7m", a clock. fieldShape() blanks a name's generated part ("el-id-*-32", "*-input"); routeShape() blanks id-like
segments; lineWeight() tells real content from short labels from counts. Nothing uses them yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… POSTs, generated names, a baseline with no added delay) Four hunts ran explore next to their own page checks: on fixed builds where the page was right it said RACE in 85 of 120 reordered runs, and it missed real bugs. compare.ts, by cause: - Relative times compare by shape whatever the baselines did. - A POST count alone is no race: only with two identical requests in flight, or a user action sending it directly a different number of times (from the run's steps and witnesses, no URL list). PUT, PATCH, DELETE as before. - Generated field names compare with themselves; fields of one shape pool their values. - With slow_ms the plan gets a baseline with no added delay, after the two slow ones (runs + 3 loads). What it differs in from them is reported first as what a slow response alone does (a race finding replaying the slow schedule); a seeded run like it is not listed; it settles what the slow baselines disagree on (two values, a line of real content, an error, a write with evidence). A bare number and a short label only one shows stay noise. - Noise blanks only the tokens that varied, at their position, each standing for its own shape; a line both baselines show is never noise. - Report: page content before labels before counts, in a run's list and in which runs come first; the headline and the title name that line; store values capped with a count of the rest. - Routes group by shape, also when armed and replayed; an action only some loads did no longer splits a session; a report that compared nothing says NO VERDICT. - Requests to other sites leave the counts unless a user action or the page's own origin led to them; a bare count on the page beside the field such a response wrote is internal. They still join the schedule (that needs the runtime). - A race finding's report says to fix only if the user asked for a fix. The header says what is matched now: text shapes, none of them a bug pattern. Tools: only the report's text changes in tools.ts. The replay and the generated test read relative times and generated names by shape too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he hunts met, and the plan's third baseline stored.mjs parses reports a hunt kept only as text back into runs for compareRuns (its header says what the text cannot show). planted.mjs runs small pages, each with a correct variant and a planted bug, and scores before and after on the same recorded runs. apps.mjs and hunt.mjs follow the plan's length; summarise.mjs scores the older comparison without the baseline its plan did not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….md 5.1 and 8, the skill, README, changelog) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…false reports on fixed builds to 0, the recall next to it) Rescored from the 136 reports the 2026-11-04 hunts kept, and measured on planted pages in Chromium (39 of 198 false runs to 0 of 234; 20 of 36 planted bugs to 30 of 36). Says what the rescoring cannot see, what the POST rule costs, and that the baseline with no added delay has not been run on a real app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline, the tool description, RESULTS section 27 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n length, lost POSTs, both slow baselines for a slow response's race)
Independent review of the verdict work: two rules hid real races main reports, one reported from a single sample.
- Only an age ("N unit ago", "just now") moves on its own, and only by how long the session ran plus a minute: "26 minutes
ago" against "27" is one line, "just now" against "3 days ago" is not. A clock, today and tomorrow, a future form and a
bare "3d" are values, compared as they are unless the baselines show them varying.
- A POST one side never sent is a lost or an extra write and stays visible. Only a different non-zero count, with the page
the same, needs evidence.
- A slow response's race needs both slow baselines ending the same way, unlike the load with no added delay. One of them
against it is one load against one load: noise as on main, named under "worth a look" with how many reordered runs show
it. The load with no added delay no longer decides what is noise. A count of writes that only changed with the delay,
and a number a little off, go under "worth a look" too. A seeded run is dropped only for what the headline already says.
- A race finding and NO VERDICT never stand together, and with no run at a quiet point NO VERDICT prints.
- A field name is generated only when it differs between the baselines: qty-100 and qty-200 swapping values is a race, and
a replay reads such a field by its name. A field with nothing but a generated name gets a note in the generated test.
- Another number on a line that varied may drift a little and no further ("CPU 13% memory 41%", not "Order 125 total 90").
- Safe requests alone leave the counts: from a localhost page a hosted API of its own looks like another site. A bare count
beside a field another site's one response wrote stays internal; two requests can answer out of order, so that stays a race.
- bench/hunt no longer counts internal differences as a race, and counts a slow response's.
Each of the review's inputs is a test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd a lost POST, plus a tip, a save queue and clocks Each slot page keeps a value that reads like a time (a clock, a day, "in 2 days", an age, a bare "3d") and shows the older answer when the bug is planted; lost-post sends its save only when one answer beat another. tip, queue and clocks are correct pages that vary by themselves, what those rules cost in false reports. The ages page moves no further than a session could. stored.mjs reads the new rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lost POSTs, both slow baselines, "worth a look") AGENT.md 5.1 and 8, the skill, the README and the changelog say the corrected rules. RESULTS 27 is redone: rescored stored reports (85 of 120 to 0 of 120 on fixed builds, of which 36 by listed evidence; 211 of 466 to 24 of 466), what each rule costs there, OpenCode 2.2.5's recorded runs, the review's inputs, and planted pages for wrong values that read like a time and a lost POST (false runs 49 of 360 to 18 of 396; planted bugs 38 of 54 to 48 of 54). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I, a swap beside a new row), ages by direction
- A bare count is internal only when it stands for a store field exactly one request wrote, the same in both runs, to
another site, requested once, with no overwrite recorded on the field, and its digits are that field's. The load's GET
landing over a click's POST from a hosted API, and two GETs writing one field, are races. "$45.00" is not bare, and a
backend on the page's own network (host.docker.internal, a private address, a name with no dot) is not another site.
- Only names some baseline lacks pool: qty-100 and qty-200 swapping values is a race beside a new row, whatever its value.
- An age is compared by direction, from the runs' own load and snapshot times: a later load can only be older by about the
time between them. lastSeen no longer decides, so reporting an hour later changes nothing. "2 hours and 5 minutes" and
"1,000 days" parse; rounded phrases stay near what a library shows ("a month" is not 40 days, "a few years" not 9).
- A line only the load with no added delay shows is no race (an error page it landed on): named under "worth a look". One
line in place of another with no other load on its side says "RACE, on one load's word", and to load it once more.
- A direct double submit prints once. A price one slow baseline alone shows is named. "$1,000" is one number.
- Matching redactions walks positions instead of backtracking: five in a row against 199 characters went from over 15 s to
about 4 ms, and a page can send that.
- "No race found" says what it covers, and when a POST count differed that a duplicate write may have happened and to check
the server.
Each input is a test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he one-load wording, sessions that compared nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four hunts on 2026-11-04 ran
genclass_explorenext to their own page checks. On fixed builds where the page was right it said RACE in 85 of 120 reordered runs, and it missed real bugs. Two independent reviews of this branch then found rules that hid real racesmainreports, and one that reported from a single sample. All are fixed here; the numbers are after those fixes.el-id-*-32,*-input). A name every baseline has is a name, whatever row beside it is new, soqty-100andqty-200swapping values is a race.slow_msthe plan adds a baseline with no added delay (runs + 3loads). What both slow baselines show the same and it does not is reported as what a slow response alone does, and says "on one load's word" when no other load sides with it. What one slow baseline alone does stays noise, as onmain, and is named under "worth a look", as is a line only that load shows.NO VERDICT, never beside a race finding.bench/explore/planted.mjshas pages for wrong values that read like a time and for a lost POST.Measured (RESULTS section 27):
slow_ms600):file-refreshRACE as a slow response's wheremainsaid 5 of 6 on counts;switchandpaletteno race wheremainsaid 3, 6, 6 and 5 of 6;new-sessionstill RACE, possibly false.maintoo).main, up from 1 of 17 on this branch's first version;hidden.mjsreports all of them in Chromium, and the lost POST.Cost:
dbl-timer2 of 3 sessions to 0 of 3,dbl-chain0 of 3 both ways, all 6 named under "worth a look". "No race found" says when a POST count differed, that a duplicate write may have happened, and to check the server. Of the stored reports' 113 POST lines, 106 stay demoted, which is what moved the nine OpenCode verdicts above, and 7 stay visible, which moved no verdict.note-autosave, "Today at 3:27 PM"), 3 on a session with a real bug.main, and named under "worth a look": OpenCode's refresh when its baselines split, Langflow's New Flow at 600 ms.Limits: the no-delay baseline has run on one real app only (OpenCode). Both slow baselines landing on one value of a random line can read as a slow response's race (
tip: 1 of 3 sessions). Field names made for each load pool their values, so two of them swapping is not seen. The rescoring reads text and has no send steps, so "sent directly" cannot fire in it. No earlier row measured a time-shaped wrong value or a lost POST.Skill text changes (
--repeat-each 9, "worth a look"), so the plugin re-syncs. No runtime change, and no model-visible text changed.🤖 Generated with Claude Code