Repository navigation
test: reproduce CF-8 reference captures and candidate performance - #514
Conversation
| const dir = fs.mkdtempSync(path.join(probes, "cf8-retain-worker-control-")); t.after(() => fs.rmSync(dir, { recursive: true, force: true })); | ||
| const inputs = path.join(dir, "retained-inputs"), locator = path.join(dir, "fixture.json"), runtimeTemp = path.join(dir, "runtime-temp"); fs.mkdirSync(inputs, { mode: 0o700 }); fs.mkdirSync(runtimeTemp, { mode: 0o700 }); | ||
| const generated = retention.createWorkloadDrivers({ templateDir: path.join(root, "apps/desktop/bench/retention"), destination: path.join(dir, "generated-driver"), targetAppDir: path.join(root, "apps/desktop"), libraryFile: path.join(root, "scripts/lib/cf8-retention.mjs") }); | ||
| fs.writeFileSync(path.join(dir, "fixture.spec.ts"), `import { test } from "@playwright/test"; import fs from "node:fs"; import os from "node:os"; import { seedHome } from ${JSON.stringify(path.join(generated.directory, "target-support.ts"))}; test("intentional early fixture failure", () => { const before = os.tmpdir(); const fixture = seedHome(); fs.writeFileSync(${JSON.stringify(locator)}, JSON.stringify({ ...fixture, runtimeBefore: before, runtimeAfter: os.tmpdir() })); throw new Error("intentional benchmark fixture failure before app launch"); });\n`); |
There was a problem hiding this comment.
Reviewed at 8771c3c: this value is an owned temporary fixture path serialized as a JavaScript string literal in a standalone .spec.ts module consumed directly by Playwright/Node. The generated source is never embedded in HTML or a <script> element, so an HTML script terminator does not change the parsing context. This is also not shell-command escaping.
A separate Node 24.22.0 control exercised ordinary paths, HTML script terminators, quote/statement injection and backtick/interpolation/newline/Unicode-separator strings. All four round-tripped exactly as data; no injected expression ran and the marker remained absent. The existing intentional failed-worker regression also passes. I am retaining this as a contextual false positive, without suppressing the rule or changing the alert state; future HTML embedding would require a different escaping boundary.
Provides the portable tooling and evidence package for #281’s combined reproducible reference/candidate capture and performance criterion. The corresponding issue box remains unchecked until protected on-main delivery and a literal clause audit; full #281 acceptance stays open.
0e674e7and capture source18aa43fkeep their actual identities; only three docs differ between18aa43fand qualified candidate8771c3c, and all 792 rebuilt desktop files match.8771c3cpassed twelve fresh-app retention cases against baseline89c98f2, then both canonical performance cases. Five first transcript opens after fresh-app window readiness had a 59.8 ms median against the existing 500 ms gate; sampled log output stayed at 262,144 bytes. This is not whole-app startup or provider latency. The earlier accepted S589/44series remains unchanged.Qualification reused for unchanged runtime: signed
91c033epassed local build/typechecks and 2,566 units with 36 skips, including all 36 benchmark guards. Earlier source8771c3cpassed all twelve observed checks. Current signed source0df7accnow also passed all twelve observed checks in run43248948819: actual previewa3a3c265has parents1d3d2f43and0df7accand matches the independent merge projection. Its raw logs show 2,570 unit passes/36 skips (including four newly landed worktree tests), 523 desktop passes/eight skips, eight site passes and four web passes, without failures/cancellations or observed retries. The source iPhone workflow passed separately. All four source artifacts are retained with matching GitHub SHA256 digests and valid ZIP integrity. These checks qualify those exact sources and bases; protected-queue delivery remains pending.Independent review of the new 41-file evidence slice checked hashes, numeric measurements, source/driver/CI attribution, reproduction safety and acceptance boundaries. One coalesced correction removes remaining renderer identifiers and makes the composer comparison neutral; the publication check preserves all 1,419 canonical scalar values, raw-receipt digests and 33 PNG hashes. The reviewed unpublished original is retained privately. CodeQL’s standalone test-module string-literal finding has a reasoned contextual disposition, without rule suppression or alert dismissal.
The receipts preserve original setup failures and unit deadline cancellations without assertion/deadline relaxation. Two archived fixture files deliberately retain their extra EOF blank lines, so full-branch diff-check reports those two cosmetic warnings. New Markdown paragraphs, links, shell/Node recipe syntax, JSON metadata and image/control digests pass publication preflight.
All captures and measurements use synthetic fixtures and common current dependencies on the named host. Fixed order, dynamic metadata, forced-GC endpoints and natural snapshots are disclosed; no pixel parity, causal improvement, arbitrary memory ceiling or leak-freedom claim is made. This changes no application behavior, ordinary E2E cleanup, feature default/admission, provider budget, model downloads or release state. Native/provider/device/older-client/human/child and rollout gates remain open.
Related: #281, #273. The separate CF-8/Local Models acceptance reconciliation remains tracked in #507.
The earlier protected-queue request
4e2ecf54-4135-4c5c-8015-6e0392efa957ended in a hosted-runner acquisition failure: candidate2612169fran no test steps, and the queue bot removed it at 18:34:21 UTC (event35948050208). Original capacity annotations and the empty valid log ZIP are retained. After a new macOS runner assignment and fresh signature/check/review/projection validation, one distinct asynchronous queue request was accepted for exact source0df7acc, withbypass_rules=falseand UUID79895230-514b-4f1d-a7a5-74169c9e2241; read-back confirms enqueued. Signed candidate30a56557matches the independent merge projection onto queued parent6de70967; its required macOS check is running. The merge-group iPhone job is skipped, not passed. A synchronous-endpoint mistake before the corrected request was rejected with HTTP 405 and created no merge or queue entry. The current request will be observed without automatic resending. Protected delivery and the combined #281 criterion remain incomplete.