Skip to content

fix(extract): populate body_tokens before MinHash gate; add OS/TS identifier node types#1060

Open
isc-tdyar wants to merge 1 commit into
DeusData:mainfrom
isc-tdyar:fix/semantic-body-tokens
Open

fix(extract): populate body_tokens before MinHash gate; add OS/TS identifier node types#1060
isc-tdyar wants to merge 1 commit into
DeusData:mainfrom
isc-tdyar:fix/semantic-body-tokens

Conversation

@isc-tdyar

Copy link
Copy Markdown
Contributor

body_tokens was not populated for short functions; type-identifier node kinds were missing

Two independent issues in extract_body_ident_tokens / compute_fingerprint that reduce semantic search quality:

1. body_tokens gated behind MinHash — short functions get no tokens

def->body_tokens was assigned after the cbm_minhash_compute early-return guard in compute_fingerprint. Any function below the MinHash length threshold (common for short methods, property accessors, constructors) had body_tokens = NULL even when the function body contained meaningful type identifiers. This degrades semantic search for the most common class of methods.

Fix: move def->body_tokens = extract_body_ident_tokens(ctx, body) to before the MinHash gate. Body tokens are now always populated when a body node exists, regardless of function length.

2. Type-identifier node kinds excluded from body-token extraction

The inclusion check in extract_body_ident_tokens matched identifier and property_identifier but not TypeScript/ObjectScript type annotation nodes. This meant type names in signatures and return types (e.g. SerializedResult, MyModel) were silently dropped from body_tokens, reducing semantic signal for typed languages.

Fix: add the following node kinds to the inclusion check:

  • type_identifier (TypeScript, C, C++, Go, …)
  • class_name, method_name, routine_name (ObjectScript UDL/routine)
  • objectscript_identifier_special, identifier_segment_immediate_special, quote_permitting_identifier (ObjectScript)

3. Raise body-token buffer limits

BT_MAX_IDENTS (40→128), BT_BUF (512→2048), BT_SEEN (128→256), BT_SEEN_MASK (127→255) — the old limits silently truncated token lists for large function bodies.

Regression test

body_tokens_type_identifier in tests/test_extraction.c — indexes a TypeScript function with MyModel and SerializedResult type annotations and asserts both appear in body_tokens.

Impact

Observed on a 1,162-class ObjectScript codebase: before this fix, semantic similarity scores for structurally similar classes were negative or near-zero (body_tokens NULL for most methods). After: scores in the 0.86–0.87 range for known-similar class pairs.

…ntifier node types

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@isc-tdyar isc-tdyar requested a review from DeusData as a code owner July 12, 2026 19:31
@DeusData DeusData added this to the 0.9.1-rc milestone Jul 14, 2026
@DeusData DeusData added bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/normal Standard review queue; useful PR with ordinary maintainer urgency. labels Jul 14, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for the focused extractor fix and test. This is a normal-priority 0.9.1-rc parsing PR. Review will verify that short functions now retain body tokens before the MinHash gate, the larger token limits remain bounded, and the additional ObjectScript and TypeScript identifier node kinds do not introduce duplicate or noisy tokens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/normal Standard review queue; useful PR with ordinary maintainer urgency.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants