fix(extract): populate body_tokens before MinHash gate; add OS/TS identifier node types#1060
Open
isc-tdyar wants to merge 1 commit into
Open
fix(extract): populate body_tokens before MinHash gate; add OS/TS identifier node types#1060isc-tdyar wants to merge 1 commit into
isc-tdyar wants to merge 1 commit into
Conversation
…ntifier node types Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Owner
|
Thanks for the focused extractor fix and test. This is a normal-priority 0.9.1-rc parsing PR. Review will verify that short functions now retain body tokens before the MinHash gate, the larger token limits remain bounded, and the additional ObjectScript and TypeScript identifier node kinds do not introduce duplicate or noisy tokens. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
body_tokenswas not populated for short functions; type-identifier node kinds were missingTwo independent issues in
extract_body_ident_tokens/compute_fingerprintthat reduce semantic search quality:1.
body_tokensgated behind MinHash — short functions get no tokensdef->body_tokenswas assigned after thecbm_minhash_computeearly-return guard incompute_fingerprint. Any function below the MinHash length threshold (common for short methods, property accessors, constructors) hadbody_tokens = NULLeven when the function body contained meaningful type identifiers. This degrades semantic search for the most common class of methods.Fix: move
def->body_tokens = extract_body_ident_tokens(ctx, body)to before the MinHash gate. Body tokens are now always populated when a body node exists, regardless of function length.2. Type-identifier node kinds excluded from body-token extraction
The inclusion check in
extract_body_ident_tokensmatchedidentifierandproperty_identifierbut not TypeScript/ObjectScript type annotation nodes. This meant type names in signatures and return types (e.g.SerializedResult,MyModel) were silently dropped frombody_tokens, reducing semantic signal for typed languages.Fix: add the following node kinds to the inclusion check:
type_identifier(TypeScript, C, C++, Go, …)class_name,method_name,routine_name(ObjectScript UDL/routine)objectscript_identifier_special,identifier_segment_immediate_special,quote_permitting_identifier(ObjectScript)3. Raise body-token buffer limits
BT_MAX_IDENTS(40→128),BT_BUF(512→2048),BT_SEEN(128→256),BT_SEEN_MASK(127→255) — the old limits silently truncated token lists for large function bodies.Regression test
body_tokens_type_identifierintests/test_extraction.c— indexes a TypeScript function withMyModelandSerializedResulttype annotations and asserts both appear inbody_tokens.Impact
Observed on a 1,162-class ObjectScript codebase: before this fix, semantic similarity scores for structurally similar classes were negative or near-zero (
body_tokensNULL for most methods). After: scores in the 0.86–0.87 range for known-similar class pairs.