FE-1230: Compare greenfield execution#345
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
kostandinang
changed the title
|
Semgrep found 6
RegExp() called with a View Dataflow Graphflowchart LR
classDef invis fill:white, stroke: none
classDef default fill:#e7f5ff, color:#1c7fd6, stroke: none
subgraph File0["<b>src/dev/execution-comparison/case-contract.ts</b>"]
direction LR
%% Source
subgraph Source
direction LR
v0["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L93 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 93] value</a>"]
end
%% Intermediate
subgraph Traces0[Traces]
direction TB
v2["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L93 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 93] value</a>"]
v3["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L99 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 99] requiredRecord</a>"]
v4["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L164 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 164] value</a>"]
v5["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L165 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 165] selected</a>"]
v6["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L99 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 99] accessibility</a>"]
v7["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L100 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 100] requiredRecord</a>"]
v8["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L164 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 164] value</a>"]
v9["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L165 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 165] selected</a>"]
v10["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L100 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 100] dynamic</a>"]
v11["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L121 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 121] accessiblePattern</a>"]
v12["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L188 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 188] value</a>"]
end
v2 --> v3
v3 --> v4
v4 --> v5
v5 --> v6
v6 --> v7
v7 --> v8
v8 --> v9
v9 --> v10
v10 --> v11
v11 --> v12
%% Sink
subgraph Sink
direction LR
v1["<a href=https://github.com/hashintel/brunch/blob/aa7a4ff9cc0191de086008ba3bf9ae59222daddf/src/dev/execution-comparison/case-contract.ts#L192 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 192] new RegExp(value['namePattern'], 'u')</a>"]
end
end
%% Class Assignment
Source:::invis
Sink:::invis
Traces0:::invis
File0:::invis
%% Connections
Source --> Traces0
Traces0 --> Sink
|
PR SummaryMedium Risk Overview Case and contracts: Dev harness ( Brunch entry: Docs and release: Reviewed by Cursor Bugbot for commit 0180267. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Record the approved Petri editor input and bounded oracle-smoke scope so implementation proceeds against a durable comparison boundary. Co-authored-by: Cursor <cursoragent@cursor.com>
Add the public and controller-only contracts, browser/reference oracles, immutable evidence packets, and a pinned Opus Brunch lane so matched execution attempts can be judged reproducibly. Co-authored-by: Cursor <cursoragent@cursor.com>
Treat the browser's implicit favicon request as harness noise so execution outputs are judged only on public application behavior. Co-authored-by: Cursor <cursoragent@cursor.com>
Accept either public feedback role, reject malformed CLI option values, and compile only the frozen accessible-name patterns so the oracle stays portable and bounded. Co-authored-by: Cursor <cursoragent@cursor.com>
aa7a4ff to
9c014bf
Compare
Record that the controller-only execution comparison tooling does not require a package version bump so the release-intent gate can distinguish it from shipped changes. Co-authored-by: Cursor <cursoragent@cursor.com>
kostandinang
changed the title
Use canonical module identity for the lane entrypoint and platform-aware path containment for static assets so compiled and Windows runs preserve the same harness behavior. Co-authored-by: Cursor <cursoragent@cursor.com>
Declare the greenfield execution comparison framework for the next alpha release. Co-authored-by: Cursor <cursoragent@cursor.com>
Strip leading section metadata before applying title fallback so body-less decisions seed meaningful chosen options, and remove the redundant empty changeset now that release intent is explicit. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0180267. Configure here.
| const input = page.getByRole('button', { name: 'Import JSON', exact: true }); | ||
| await input.setInputFiles(path); | ||
| await feedback(page, /Import rejected/iu).waitFor(); | ||
| } |
There was a problem hiding this comment.
Stale import feedback race
Medium Severity
importFile waits for /Import rejected/ feedback that may already be on screen from a prior failed import. On the second invalid import in invalid-and-cascade, that wait can resolve before the new async import finishes, so the net-unchanged assertion can run too early and miss a lane that incorrectly accepts schema-invalid.json.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 0180267. Configure here.
| headless: true, | ||
| }); | ||
| const context = await browser.newContext({ acceptDownloads: true }); | ||
| const page = await context.newPage(); |
There was a problem hiding this comment.
Server leak on browser launch failure
Medium Severity
The static server is started before Chrome resolution and browser/context creation, but cleanup only runs inside the later try/finally. If resolveChromeExecutable or chromium.launch fails, the listening server is never closed, leaking ports across slow oracle runs.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 0180267. Configure here.
1 participant
Why
Greenfield execution outputs need comparable evidence from the same frozen specification without exposing hidden evaluation criteria or landing generated code.
What
Add the Petri-net editor comparison case, controller-only oracles, immutable attempt evidence, and an isolated Brunch lane that stops at
promotion_prepared. The tracer retains valid and invalid attempts and separates masked outcome evidence from unblinded process evidence.How to test
npm run checknpm test -- src/dev/execution-comparisonnpm run test:slow -- src/dev/execution-comparison/__tests__/browser-oracle.slow.test.tsnpm run verify:full