fix(extract): mask bare & in TSX JSX text so partial extraction stops (#2922) - #2934
fix(extract): mask bare & in TSX JSX text so partial extraction stops (#2922)#2934santhiprakash wants to merge 1 commit into
Conversation
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds a _mask_tsx_ampersands pre-parse pass in graphify/extract.py that escapes bare & to & in JSX text content of TSX sources, using a context-stack walker with <-vs-generic disambiguation to leave & in code, strings, comments, and expression containers untouched. Wires the mask into the TSX extraction path so bare-ampersand JSX text no longer produces ERROR nodes and partial-extraction warnings. Adds fixture and unit tests covering ampersand masking and the silent-JSX-text behavior.
Worth a look
- Template literal treated as plain string ignores ${} interpolation and nested JSX —
graphify/extract.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- TSX ampersand mask stays in JSX text after closed tags —
graphify/extract.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1569 functions depend on the 238 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 477 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_js()— 80 callers, 4 callees - new:
dispatch_command()— 2 callers, 119 callees - new:
_get_extractor()— 26 callers, 6 callees - new:
run_pipeline()— 8 callers, 13 callees - new:
collect_files()— 17 callers, 6 callees - …and 23 more — each is listed as a finding
Verification — 1569 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1421 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify extract\_js.
The verifier did not have enough to check extract\_js, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
· 1 grounded finding(s) anchored inline below; 30 more finding(s) on lines outside this diff (see the check run).
| return ''.join(out) | ||
|
|
||
|
|
||
| def extract_js(path: Path) -> dict: |
There was a problem hiding this comment.
extract_js()
80 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
d1d687b to
22ae0e9
Compare
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds _mask_tsx_ampersands and its _TSX_ENTITY_RE/_TSX_LT_EXPR_PREV helpers to escape bare & in TSX JSX text to & so tree-sitter parses cleanly instead of emitting ERROR nodes and dropping symbols (#2922), tracking string/comment/tag/expr/jsx_text contexts and disambiguating < as JSX tag vs. generic/comparison. Wires the masking into the TSX extraction path and adds a fixture plus tests covering entity preservation and &-masking elsewhere. Renames/renumbers several extractor engine internal helpers and rationale symbols.
Worth a look
- Closing JSX tags never exit jsx_text state —
graphify/extract.py:980· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- TSX ampersand mask treats code after a JSX element as JSX text —
graphify/extract.py:986· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Nested JSX in expression containers is not masked —
graphify/extract.py:995· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1836 functions depend on the 452 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 477 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
_extract_generic()— 18 callers, 24 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_objc()— 27 callers, 9 callees - new:
extract_js()— 80 callers, 3 callees - new:
dispatch_command()— 2 callers, 119 callees - new:
_get_extractor()— 26 callers, 6 callees - …and 34 more — each is listed as a finding
Verification — 1836 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1688 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_extract\_generic.
The verifier did not have enough to check \_extract\_generic, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
· 42 more finding(s) on lines outside this diff (see the check run).
22ae0e9 to
9a8827a
Compare
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 4 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds a _mask_tsx_ampersands pre-parse pass in graphify/extract.py that rewrites bare & in JSX text to & so tree-sitter's TSX grammar parses cleanly instead of emitting ERROR nodes and dropping symbols (#2922). Uses a context-stack walker that distinguishes JSX text from tags, expression containers, strings, comments, and TS code (bitwise &, generic <T> vs JSX tag disambiguation), and short-circuits when the source has no &. Adds tests/test_tsx_jsx_text_ampersand covering entity pass-through and related cases.
Worth a look
- New test module ends with incomplete function definition —
tests/test_tsx_jsx_text_ampersand.py:293· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- prev_code_keyword substring check via 'return' etc. can false-match identifier suffixes —
graphify/extract.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- TSX ampersand mask rewrites regex literals —
graphify/extract.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Fixture test mutates process-wide cwd during extraction —
tests/test_tsx_jsx_text_ampersand.py:69· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1843 functions depend on the 459 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 477 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
_extract_generic()— 18 callers, 24 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_objc()— 27 callers, 9 callees - new:
extract_js()— 80 callers, 3 callees - new:
dispatch_command()— 2 callers, 119 callees - new:
_get_extractor()— 26 callers, 6 callees - …and 35 more — each is listed as a finding
Verification — 1843 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1695 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_extract\_generic.
The verifier did not have enough to check \_extract\_generic, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
· 1 grounded finding(s) anchored inline below; 42 more finding(s) on lines outside this diff (see the check run).
| ) | ||
|
|
||
|
|
||
| def _mask_tsx_ampersands(src: str) -> str: |
There was a problem hiding this comment.
_mask_tsx_ampersands()
7 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
…Graphify-Labs#2922) tree-sitter-typescript requires & in JSX text (the run between > and < inside a JSX element) to begin an HTML entity reference (&, &#NN;, <, ...). A bare & produces an ERROR node and the partial-extraction path surfaces a parse_errors warning (Graphify-Labs#2551, Graphify-Labs#2788) — even though esbuild, tsc, and React all accept the file. On the reporter's 3000-file TSX codebase, 31 files (~1 %) were silently dropping every function, class, and import — UI labels like "Conexões & Integrações" tripped the gate. The fix is a context-tracking walker that masks bare & to & ONLY in JSX text. & inside JSX tag attribute values, { ... } expression containers, string literals, comments, and TypeScript code (where it is bitwise AND or an intersection type) is left untouched. The walker disambiguates JSX tags from TypeScript generic type-parameter openers (function f<T>, type Bar<T>, <T extends X>, const pick = <T,>(x: T) => x) by combining a previous-non-whitespace character set (operator/punctuation ⇒ JSX, alphanumeric ⇒ code) with a short keyword list (return/yield/new/as/typeof/ void/delete) that flips < after an identifier into JSX context. Already-formed entities (&, &#NN;, <, …) are passed through; multi-ampersand JSX text runs are masked independently. Tag lifecycle is tracked on the context stack: a closing tag pops the element's jsx_text context (returning to code, an expression container, or the parent element's text) and a self-closing tag never opens one, so code following an element — bitwise & included — is never masked (the first cut left jsx_text on the stack after </tag>, corrupting a later "a & b" into "a & b" and reintroducing a parse error). The same tag/generic shape disambiguation runs inside JSX expression containers with expression context forced on, so nested JSX ({ok ? <span>a & b</span> : null}) is masked like top-level JSX. Adds tests/test_tsx_jsx_text_ampersand.py (12 tests, including regression canaries for bitwise AND, JSX expression &&, existing &, arrow generics, as cast, JSX attribute &, code after a closed element, self-closing and fragment lifecycle, nested JSX in expression containers, and the fixture) and tests/fixtures/tsx_jsx_text_ampersand.tsx (the real-world shape with mixed JSX-text, JSX-attribute, expression-container, and TS-code & in a single file). The mask is wired through a new optional LanguageConfig.source_transform bytes hook applied in _extract_generic's read path, so extract_js() stays a pure suffix→config dispatch with a single file read: the TSX-specific masking lives with _TSX_CONFIG, the walker is unchanged, and non-TSX languages (source_transform unset) parse exactly as before. Vue SFCs with lang="tsx" scripts share _TSX_CONFIG and now get the same mask.
9a8827a to
4486f5f
Compare
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds a TSX-specific _mask_tsx_ampersands preprocessor (plus _TSX_ENTITY_RE/_TSX_LT_EXPR_PREV) in graphify/extract.py that escapes bare & to & in JSX text content so tree-sitter parses cleanly instead of returning a partial tree (#2922). The masking walks a context stack (tags, strings, comments, expression containers, JSX text) and disambiguates < between JSX tags and generic type parameters to avoid touching bitwise & in code. Includes a new test_tsx_jsx_text_ampersand test.
Worth a look
- TSX ampersand transform violates source_transform byte contract —
graphify/extract.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Single-letter uppercase JSX tags are misclassified as generics —
graphify/extract.py:965· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1835 functions depend on the 451 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 477 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
_extract_generic()— 18 callers, 24 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_objc()— 27 callers, 9 callees - new:
extract_js()— 80 callers, 3 callees - new:
dispatch_command()— 2 callers, 119 callees - new:
_get_extractor()— 26 callers, 6 callees - …and 34 more — each is listed as a finding
Verification — 1835 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1687 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_extract\_generic.
The verifier did not have enough to check \_extract\_generic, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
· 42 more finding(s) on lines outside this diff (see the check run).
1 participant
Summary
A bare
&inside TSX JSX text (e.g.<div>VoIP & Chamadas</div>) is valid TSX — esbuild, tsc, and React all accept it — but tree-sitter-typescript's grammar requires&in JSX text to begin an HTML entity reference. The resulting ERROR node trips the partial-extraction path (#2551 / #2788) and, on the reporter's 3000-file TSX codebase, silently drops every function, class, and import from 31 files (1 %), all because UI labels use&as a natural-language connector (Conexões & Integrações,Configurações & Perfil,Privacidade & LGPD).Fix
A small, context-tracking walker that masks bare
&to&only in JSX text content.&is left untouched everywhere the grammar already accepts it:<a href="/search?q=a&b=c">),{ ... }expression containers ({flag && <span/>}),&is bitwise AND (0xff & 0x0f) or an intersection type (type X = A & B).Already-formed entities (
&,&#NN;,<, …) are passed through without double-masking.The walker disambiguates JSX tag starts from TypeScript generic type-parameter openers by combining a previous-non-whitespace character set (operator/punctuation ⇒ JSX, alphanumeric ⇒ code) with a short keyword list (
return/yield/new/as/typeof/void/delete) that flips<after an identifier into JSX context.<T>,<T,>,<T extends X>,<T = X>, and<T>(...) => ...are all treated as code (generic shape) so subsequent bitwise&in code is not masked.Verification
tests/test_tsx_jsx_text_ampersand.py— 9 tests, all green. Covers the fixture (mixed JSX-text / JSX-attribute / expression-container / TS-code&in a single file), bare&in JSX text is silent, bitwise AND in TS code is preserved,&&in JSX expression is preserved, existing&passes through, mask is byte-neutral on JSX text, every non-JSX-text&location is left intact, fast-path for empty/no-ampersand sources, multi-bare-ampersand JSX text runs.tests/test_ts_parse_warning.py,tests/test_partial_extraction_warning.py,tests/test_ts_*.py,tests/test_js_*.py,tests/test_typescript_module_extensions.py,tests/test_extract*.py,tests/test_extraction_spec_ids.py,tests/test_extractors_registry.py,tests/test_file_node_id_spec.py,tests/test_node_id_canonical.py,tests/test_no_dedup_flag.py,tests/test_partial_cache.py— 495 passed, 4 skipped. No regressions.Reproduction (before fix)
After the fix,
A()extracts cleanly with no parse_errors.Closes #2922.