Skip to content

test: reproduce CF-8 reference captures and candidate performance - #514

Merged
BunsDev merged 6 commits into
mainfrom
cf8-reproducible-series
Nov 12, 2026
Merged

BunsDev merged 6 commits into
mainfrom
cf8-reproducible-series

Conversation

@BunsDev

@BunsDev BunsDev commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Provides the portable tooling and evidence package for #281’s combined reproducible reference/candidate capture and performance criterion. The corresponding issue box remains unchecked until protected on-main delivery and a literal clause audit; full #281 acceptance stays open.

  • The opt-in retention CLI rebuilds explicit full revisions with common pinned dependencies, verifies source/build/core/driver identities, retains seeded inputs and first logs, and stops at the first failure without automatic retries.
  • The visual package contains 33 candidate PNGs, exact reference/candidate hashes, both source-export reproduction controls, a branch-preserving recipe and an explicit inventory of remaining visual differences. Each source’s fourteen unchanged capture cases passed with 33 images, zero skips/retries. Historical reference 0e674e7 and capture source 18aa43f keep their actual identities; only three docs differ between 18aa43f and qualified candidate 8771c3c, and all 792 rebuilt desktop files match.
  • Exact candidate 8771c3c passed twelve fresh-app retention cases against baseline 89c98f2, then both canonical performance cases. Five first transcript opens after fresh-app window readiness had a 59.8 ms median against the existing 500 ms gate; sampled log output stayed at 262,144 bytes. This is not whole-app startup or provider latency. The earlier accepted S5 89/44 series remains unchanged.

Qualification reused for unchanged runtime: signed 91c033e passed local build/typechecks and 2,566 units with 36 skips, including all 36 benchmark guards. Earlier source 8771c3c passed all twelve observed checks. Current signed source 0df7acc now also passed all twelve observed checks in run 43248948819: actual preview a3a3c265 has parents 1d3d2f43 and 0df7acc and matches the independent merge projection. Its raw logs show 2,570 unit passes/36 skips (including four newly landed worktree tests), 523 desktop passes/eight skips, eight site passes and four web passes, without failures/cancellations or observed retries. The source iPhone workflow passed separately. All four source artifacts are retained with matching GitHub SHA256 digests and valid ZIP integrity. These checks qualify those exact sources and bases; protected-queue delivery remains pending.

Independent review of the new 41-file evidence slice checked hashes, numeric measurements, source/driver/CI attribution, reproduction safety and acceptance boundaries. One coalesced correction removes remaining renderer identifiers and makes the composer comparison neutral; the publication check preserves all 1,419 canonical scalar values, raw-receipt digests and 33 PNG hashes. The reviewed unpublished original is retained privately. CodeQL’s standalone test-module string-literal finding has a reasoned contextual disposition, without rule suppression or alert dismissal.

The receipts preserve original setup failures and unit deadline cancellations without assertion/deadline relaxation. Two archived fixture files deliberately retain their extra EOF blank lines, so full-branch diff-check reports those two cosmetic warnings. New Markdown paragraphs, links, shell/Node recipe syntax, JSON metadata and image/control digests pass publication preflight.

All captures and measurements use synthetic fixtures and common current dependencies on the named host. Fixed order, dynamic metadata, forced-GC endpoints and natural snapshots are disclosed; no pixel parity, causal improvement, arbitrary memory ceiling or leak-freedom claim is made. This changes no application behavior, ordinary E2E cleanup, feature default/admission, provider budget, model downloads or release state. Native/provider/device/older-client/human/child and rollout gates remain open.

Related: #281, #273. The separate CF-8/Local Models acceptance reconciliation remains tracked in #507.

The earlier protected-queue request 4e2ecf54-4135-4c5c-8015-6e0392efa957 ended in a hosted-runner acquisition failure: candidate 2612169f ran no test steps, and the queue bot removed it at 18:34:21 UTC (event 35948050208). Original capacity annotations and the empty valid log ZIP are retained. After a new macOS runner assignment and fresh signature/check/review/projection validation, one distinct asynchronous queue request was accepted for exact source 0df7acc, with bypass_rules=false and UUID 79895230-514b-4f1d-a7a5-74169c9e2241; read-back confirms enqueued. Signed candidate 30a56557 matches the independent merge projection onto queued parent 6de70967; its required macOS check is running. The merge-group iPhone job is skipped, not passed. A synchronous-endpoint mistake before the corrected request was rejected with HTTP 405 and created no merge or queue entry. The current request will be observed without automatic resending. Protected delivery and the combined #281 criterion remain incomplete.

@vercel

vercel Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
modex Ready Ready Preview Nov 12, 2026 3:32pm UTC
modex-app Ready Ready Preview Nov 12, 2026 3:32pm UTC

Request Review

const dir = fs.mkdtempSync(path.join(probes, "cf8-retain-worker-control-")); t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
const inputs = path.join(dir, "retained-inputs"), locator = path.join(dir, "fixture.json"), runtimeTemp = path.join(dir, "runtime-temp"); fs.mkdirSync(inputs, { mode: 0o700 }); fs.mkdirSync(runtimeTemp, { mode: 0o700 });
const generated = retention.createWorkloadDrivers({ templateDir: path.join(root, "apps/desktop/bench/retention"), destination: path.join(dir, "generated-driver"), targetAppDir: path.join(root, "apps/desktop"), libraryFile: path.join(root, "scripts/lib/cf8-retention.mjs") });
fs.writeFileSync(path.join(dir, "fixture.spec.ts"), `import { test } from "@playwright/test"; import fs from "node:fs"; import os from "node:os"; import { seedHome } from ${JSON.stringify(path.join(generated.directory, "target-support.ts"))}; test("intentional early fixture failure", () => { const before = os.tmpdir(); const fixture = seedHome(); fs.writeFileSync(${JSON.stringify(locator)}, JSON.stringify({ ...fixture, runtimeBefore: before, runtimeAfter: os.tmpdir() })); throw new Error("intentional benchmark fixture failure before app launch"); });\n`);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8771c3c: this value is an owned temporary fixture path serialized as a JavaScript string literal in a standalone .spec.ts module consumed directly by Playwright/Node. The generated source is never embedded in HTML or a <script> element, so an HTML script terminator does not change the parsing context. This is also not shell-command escaping.

A separate Node 24.22.0 control exercised ordinary paths, HTML script terminators, quote/statement injection and backtick/interpolation/newline/Unicode-separator strings. All four round-tripped exactly as data; no injected expression ran and the marker remained absent. The existing intentional failed-worker regression also passes. I am retaining this as a contextual false positive, without suppressing the rule or changing the alert state; future HTML embedding would require a different escaping boundary.

@github-code-quality

github-code-quality Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code coverage overview

Languages: TypeScript

TypeScript / code-coverage-agent

main 5a580d4 cf8-reproducible-ser... 0df7acc +/-
66% 66% 0%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated November 12, 2026 16:05 UTC

@BunsDev BunsDev changed the title test: make CF-8 retention measurements reproducible test: reproduce CF-8 reference captures and candidate performance Nov 12, 2026
@BunsDev
BunsDev added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@BunsDev
BunsDev added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 30a5655 Nov 12, 2026
13 checks passed
@BunsDev
BunsDev deleted the cf8-reproducible-series branch November 12, 2026 19:43

This branch was successfully deployed

1 active deployment
Preview – modex — 0df7acc3 Deployed Nov 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants