Skip to content

fix(typescript): parse variance annotations (<in T>, <out T>, <in out T>) - #3226

Open
inth3shadows wants to merge 2 commits into
colbymchenry:mainfrom
inth3shadows:fix/ts-variance
Open

inth3shadows wants to merge 2 commits into
colbymchenry:mainfrom
inth3shadows:fix/ts-variance

Conversation

@inth3shadows

Copy link
Copy Markdown
Contributor

What

TypeScript 4.7 variance annotations, such as interface Stream<out A, in E>, <in out T> and <const in T>, are parse errors in tree-sitter-typescript v0.23.2. That is upstream tree-sitter/tree-sitter-typescript#370. Two upstream fix PRs are open, #361 and #364, and both break type F<out> = out[]. The repo has had no commit since 2025-01-30, so this carries a patch, following the existing docs/grammars/*.patch pattern.

How

  • docs/grammars/tree-sitter-typescript.patch applies to v0.23.2 (f975a62) and changes one rule. type_parameter takes an optional in / out / in out, with const on either side. out stays a valid parameter name through brajevicm/tree-sitter-typescript@9fcaa13's alias. The external scanner is untouched.
  • The patch changes a grammar rule, so unlike kotlin and swift the parser is regenerated with tree-sitter-cli 0.25.10 (ABI 15, same as the nix wasm).
  • The kernel stops using the tree-sitter-typescript crate. It compiles the same generated C from codegraph-kernel/grammars/typescript/, which is laid out like upstream so scanner.c's ../../common/scanner.h resolves without edits. The wasm is built from the same sources, so kernel-grammar-parity holds.
  • Provenance, hashes, rebuild steps and known limits are in docs/grammars/tree-sitter-typescript.md.

Measured

These figures compare the previous wasm with the patched one (web-tree-sitter 0.25.10). "Trees changed" compares node type, field and byte range for every node.

corpus files files with an error trees changed
Effect-TS/effect 7a87ff9 2,054 239 → 185 121 (54 now error-free, none newly broken)
colinhacks/zod 2554b70 481 4 → 0 4
microsoft/vscode ca43fc94 src/ (.ts / .tsx) 11,265 / 257 unchanged 0 / 0
excalidraw/excalidraw dd2f8fc (.ts / .tsx) 372 / 323 unchanged 0 / 0

A review pass also ran 1,287 more .ts/.tsx files that use in / out after <, , or (. None broke and 24 were fixed.

Regenerating the unpatched grammar with 0.25.10 gives the same trees as the previous wasm, so regeneration on its own changes nothing. The upstream corpus passes before and after (112/112).

Limits, stated plainly:

  • Many of Effect's largest files, such as Schema.ts and Stream.ts, still parse as one whole-file ERROR. Their next error is a different upstream bug, bug: anonymous generic call signatures fail to parse when separated only by line breaks tree-sitter/tree-sitter-typescript#335: a generic call signature on its own line in an object type. A fix for that comes in a separate PR.
  • in / out now lex as keywords where an expression < could open type parameters. So a type or JSX tag literally named out / in in that spot no longer parses: <out>x, and in tsx <out>…</out>, <in>…</in> and <out.Provider>. I tried two ways to keep them. Accepting out as a type name everywhere broke [out: T] tuple labels, which Effect uses. Aliasing it in JSX gave GLR conflicts through every attribute. Neither shipped.

Tests

  • __tests__/typescript-variance-parse.test.ts checks tree shape, not just hasError. It checks the modifier tokens and name of each type parameter in typescript and tsx, out as a parameter name, tuple label, mapped-type key, index-signature key, type-predicate subject and type argument, and an indexed file's declarations. 5 of its 7 tests fail on the old grammar, and all pass with the patch.
  • With the kernel built (npm run build:kernel), kernel-grammar-parity, kernel-tsjs-parity, kernel-scaffold, grammar-wasm-bytes and the new test give 144/144. tsc --noEmit exits 0.
  • Full suite on Linux: 885 test files and 11076 tests passed (vitest exit 0); the ui-server-api "stays fast" timing test went over its 100 ms budget once under full-suite load in an earlier run and passed 3/3 alone.

EXTRACTION_VERSION and KERNEL_ABI_VERSION are not bumped, matching the kotlin (#2173) and swift (#2683) grammar patches. Your call if a re-index hint is wanted.

Re-index TypeScript projects after upgrading.

…n out T>`)

Vendor tree-sitter-typescript v0.23.2 with a grammar patch, regenerated
with tree-sitter-cli 0.25.10, for both the wasm and the kernel (which now
compiles the same C instead of the crate). `type_parameter` takes an
optional `in` / `out` / `in out`, with `const` on either side; `out` stays
usable as a parameter name (upstream colbymchenry#364's rule + brajevicm@9fcaa13).

Known limit: a type or JSX tag literally named `out`/`in` right after an
expression `<` (`<out>x`, `<out>…</out>`) no longer parses; accepting `out`
as a type name everywhere instead broke `[out: T]` tuple labels.

Effect: 239 -> 185 files with a parse error, 54 newly clean, none newly
broken; Zod 4 -> 0. VS Code src/ and Excalidraw (.ts/.tsx) parse to
identical trees. Provenance in docs/grammars/tree-sitter-typescript.md.
…g claims

Review follow-ups on the variance patch: assert tree shape (call_expression
type arguments, mapped-type and type-predicate names, tuple labels) rather
than only hasError; make the extraction test fail without the patch; scope
the indexing setup to the test that uses it; drop the next-line-method guard
(it belongs with the colbymchenry#335 scanner change). The changelog no longer names
Effect files that colbymchenry#335 still blocks, and the docs list the `<out.Provider>`
tsx limit, the real over-acceptance (`<out in>`), and that the second
`const` is ours rather than colbymchenry#364's.
@inth3shadows

Copy link
Copy Markdown
Contributor Author

I wrote a CI check for this PR on my fork, plus two small follow-ups on the PR itself.

Differential parse check. tsc is treated as ground truth: ts.createSourceFile(...).parseDiagnostics, syntax only. The check flags every file that tsc accepts but the shipped TS/TSX wasm parses with an error, across pinned corpora and a folder of one-construct fixtures. It fails on any such file that isn't in a checked-in baseline, and it reports baselined files that now parse. On this PR's wasm, against v0.23.2 (verified, run locally and in CI):

corpus files divergent before → after
Effect-TS/effect 7a87ff9 2,056 239 → 185 (54 fixed, none new)
colinhacks/zod 2554b70 515 4 → 0
excalidraw dd2f8fc 695 5 → 5
vscode ca43fc94 src/ 11,267 11 → 11

The fixtures also record grammar gaps still open on the vendored grammar: static accessor, import attributes on export … from, and using / await using in for-of heads. When the grammar is bumped or replaced by a fork, any of these it fixes show up as "now agree". The script is scripts/ts-differential.mjs; the workflow and docs are in inth3shadows#12. It needs no build, and a cached run takes about a minute on GitHub's runners. It's fork-only for now, and I'm happy to open a PR here if you want it.

Two things on this PR:

  1. EXTRACTION_VERSION stays at 28. An existing TypeScript index therefore isn't flagged as stale, and a file that used to parse as one whole-file ERROR keeps indexing as almost nothing until the user re-indexes by hand. A bump would let codegraph status / upgrade prompt for the re-index the changelog entry asks for.
  2. That changelog entry has no (#…) reference, unlike its neighbours (AGENTS.md, changelog rule 4).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant