Skip to content

ci: tsc differential parse for the TypeScript grammar - #12

Merged
inth3shadows merged 3 commits into
mainfrom
ci/ts-differential
Nov 6, 2026
Merged

inth3shadows merged 3 commits into
mainfrom
ci/ts-differential

Conversation

@inth3shadows

@inth3shadows inth3shadows commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

What

This adds a CI job, TS grammar differential, that exists only in the fork. It treats tsc as the ground truth for TypeScript syntax and checks the vendored tree-sitter-typescript and tsx wasm grammars against it.

  • Oracle. tsc's verdict comes from ts.createSourceFile(...).parseDiagnostics, syntax only. Tree-sitter's verdict is rootNode.hasError. The job runs under --liftoff-only, the same flag as production (a self re-exec, checked against src/extraction/wasm-runtime-flags.ts). Canaries check that the tsc oracle (both TS and TSX) and both grammars can still tell good input from bad. If either can't, the job fails before it does any corpus work.
  • Corpora. Effect, Zod, Excalidraw, VS Code src/, and the in-repo __tests__/fixtures/ts-syntax/. The external ones are pinned by SHA and cached under $RUNNER_TEMP.
  • Gate (scripts/ts-differential.baseline.json records each divergent file's first error row):
    • exit 1 if the run finds any of these:
      • a file that tsc accepts, tree-sitter rejects, and the baseline doesn't list
      • in a pinned corpus, a baselined file whose first error moved to an earlier line
      • a fixture that tsc rejects
      • a non-gap fixture that diverges
    • exit 2 if the job can't trust its inputs:
      • the tsc or web-tree-sitter version doesn't match the baseline
      • a canary fails
      • a corpus is empty
      • a baselined path, or a whole baselined corpus, has vanished
      • a baseline row is malformed
      • any other setup error
    • Reported but not failing: baselined files that now agree with tsc, and errors that moved later. Both suggest a re-baseline with --update.
  • Known gaps. Each of these fixtures holds one construct: variance.ts (tree-sitter-typescript#370), static accessor, import attributes on export {..} from and on export * from, using in a for-of head, and await using in a for-await head. A grammar that fixes one reports it as now-agree, which is how a candidate fork is compared with ours.

Scope: wasm only. The native kernel's grammar is not exercised here. Wasm/kernel parity is scripts/kernel-parity.mjs's job.

Background: tree-sitter-typescript has had no commit since 2025-01-30 (tree-sitter/tree-sitter#6091), and we carry grammar patches (colbymchenry#3226).

Verified

Full local run against current main's wasm (tsc 5.9.3, web-tree-sitter 0.25.10). It exits 0:

corpus      files  DIVERGENT  new  worse
effect       2056        239    0      0
zod           515          4    0      0
excalidraw    695          5    0      0
vscode      11267         11    0      0
local          25          6    0      0

Two review rounds (/code-review plus a code-reviewer agent) found 21 issues in total. All are fixed except one accepted low-severity item: SIGTERM sent to the parent orphans the relaunched child. Each fix has a before/after check: a new fixture missing from the baseline exits 1; a version mismatch, a renamed fixture, a 0-file corpus, a blind oracle, flag drift and a malformed row each exit 2.

tsc is the ground truth for whether a file is valid TypeScript syntax.
The new job parses pinned corpora (Effect, Zod, Excalidraw, VS Code src/)
and the in-repo ts-syntax fixtures with both tsc and the vendored
tree-sitter-typescript/tsx wasm. It fails on any file that tsc accepts and
tree-sitter rejects when that file is not in the checked-in baseline.
Baseline files that now parse are reported as "now agree".

The gap-*.ts and variance.ts fixtures record known grammar gaps
(tree-sitter-typescript#370 and others), so a patched grammar or a
candidate fork shows its fixes as "now agree".

Fork-only. It runs on PRs and main pushes that touch the TS wasm, the
grammar sources or patch, the lockfile, or the job's own files.
- run under --liftoff-only via self re-exec (V8 Zone OOM guard, colbymchenry#293/colbymchenry#298)
- strict fixtures: tsc-invalid or non-gap divergent fixtures fail
- 0-file corpus is a setup error; vanished baseline paths are reported, never now-agree
- baseline stores each file's first error row; an earlier row fails as a new error
- one construct per gap fixture
- upload the --json report; restore/save cache split so failing runs still cache
- trigger paths narrowed to the TS grammar sources; docs say wasm only
- step summary on exit 2; rename race tolerated; oracle and grammar canaries
- cache dir and marker keyed on subdir too
- cache path under $RUNNER_TEMP via GITHUB_ENV (actions/cache rejects '..'); key v3
- every exit-2 setup error writes the step summary
- relaunch guard without --liftoff-only fails closed
- wasm flags checked against src/extraction/wasm-runtime-flags.ts (and that file triggers the job)
- first-error-row gate skips the editable local fixtures; pinned corpora hint at --update
- baseline corpora missing from corpora.json are reported as vanished
- TSX oracle canary; malformed baseline rows are a setup error
@inth3shadows
inth3shadows merged commit 954479d into main Nov 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant