Repository navigation
Conversation
lang_thrift.py reads structs, unions, exceptions, enums, typedefs, services and service functions with a token scanner in the shape of lang_proto.py. Field, argument, return and throws types (through list/set/map) and a service's extends are reference sites bound by lane_units.bind_sites; base types bind to nothing. Includes resolve against the including file's directory and its ancestors. .thrift leaves the plain-text list. Closes ClickHouse#624 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
zlareb1
left a comment
There was a problem hiding this comment.
Commented-out Thrift includes become real dependency edges. The inline example reproduces the error on this head. All 212 focused lane, extractor-stamp, manifest and inventory tests pass, but they do not cover includes inside block comments.
|
This head has merge conflicts with |
# Conflicts: # docs/development/adding-language-support.md # server/code/app/index/ingest_parse.py # tests/code_lanes/corpus.py # tests/fixtures/code_lanes/NOTICE.md
An include inside a block comment was read by the raw-source regex and became a dependency edge. Includes now come from the token stream, which drops comments.
|
Thanks @zlareb1. Both are fixed:
In a Linux container, the Thrift, Lua, extractor-stamp, manifest, corpus, known-gaps and regex-cost tests give 991 passed and 1 failed. The failure is |
zlareb1
left a comment
There was a problem hiding this comment.
The commented-include finding is fixed. A separate valid field form still loses dependency edges: the parser reads field types only after a numeric ID. The focused parser, corpus, stamp, manifest and inventory checks pass; the inline reproduction is missing from that coverage.
zlareb1
left a comment
There was a problem hiding this comment.
The optional-field-ID finding remains on this head. The parser is unchanged: struct User { 1: string id }\nstruct Envelope { optional User user } still yields no type sites, while adding 1: before optional yields the expected User site. All 215 focused lane, stamp, manifest and inventory tests pass; they still omit this valid field form.
The IDL grammar makes FieldID optional in struct fields, argument lists and throws lists. _fields now reads each field as [ID:] [req] Type [&] name [= value] [xsd options] [(annotations)], so an ID-less field's type is a reference site, and default values and annotations are skipped. Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com> # Conflicts: # docs/development/adding-language-support.md # server/code/app/index/ingest_parse.py # server/code/domain/extract/lang_detect.py # server/code/domain/extract/lang_text.py # tests/code_lanes/corpus.py
…ric ID Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
zlareb1
left a comment
There was a problem hiding this comment.
The optional-field-ID finding is fixed, including argument and throws types, defaults and annotations. No further blocking findings after checking parser registration, include resolution, binding, corpus output and extractor-stamp isolation. All 231 focused lane, corpus, known-gap, stamp, manifest and Thrift cost checks pass on Linux.
zlareb1
left a comment
There was a problem hiding this comment.
A remaining plain-text classification assertion fails in CI and locally on this head. The new lane needs the corresponding test inventory update before merge; the earlier field-ID fix remains verified.
| ".rego": "config", | ||
| ".mdc": "doc", | ||
| ".proto": "proto", | ||
| ".thrift": "thrift", |
There was a problem hiding this comment.
P2 — Update the plain-text classification test when registering the Thrift parser. tests/unit/test_lang_text.py::test_code_without_a_parser_and_plain_text_are_kept_as_text still includes api/service.thrift and requires both text_lang_for_path(...) == "text" and parse_for_ingest(...).lang == "text". Removing .thrift from the text extension set makes the first assertion fail at line 35; the same test fails in unit part 2 and in a focused Linux run on this head. Move that fixture out of the unclaimed-text cases and assert that the production dispatcher selects the Thrift lane and its service symbol. Keep the assertions for the other text-only paths.
Closes: #624
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
The code index reads Thrift IDL files: structs, unions, exceptions, enums, typedefs, services and their functions become symbols, field and argument types become type-use edges, a service's
extendsbecomes an inherits edge, andinclude "x.thrift"becomes an include.Why
.thriftwas in_CODE_TEXT_EXTENSIONS, so Thrift files were chunked and searchable butcode.symbol,code.callersandcode.outlineknew nothing in them.Scope and risk
A tree-less lane, following
docs/development/adding-language-support.mdand shaped likelang_proto.py:server/code/domain/extract/lang_thrift.py:parse_thrift(path, source) -> ParsedFile,reference_sites,SITE_RULES,CHUNK_OPTIONS,wanted_paths/resolve_imports.struct/union/exception→struct;enum→enum;typedef T X→type;service→interface; each service function →function<mod>::Service.fwhose signature is the function as written (Machine get(1: MachineId id) throws(1: shared.NotFound not_found)). A token scanner with one-pass brace matching; comments (//,#,/* */) and strings are skipped, so a definition inside a comment is not a symbol.throwsentry's named type (field_type→uses_type, to struct / enum / type), and a service'sextends(inherits, to an interface), bound bylane_units.bind_sites(same file 1.0, repository-unique 0.6, ambiguous → nothing).list<>/set<>/map<>are read through; base types (i32,string,binary, …) name nothing; an include-qualifiedshared.Userbinds by its last part.include "shared.thrift"→ the including file's directory and its nearest ancestors, then the root (lane_units.ancestor_candidates), settled against the namespace's live paths.".thrift": "thrift"inLANG_BY_EXTENSION, removed from_CODE_TEXT_EXTENSIONS;_parse_thriftin_PARSERS,thriftinLANE_UNIT_LANGSand_lane_module;"thrift": SCRIPTinname_scope;thriftinsearch_scope.lang_thrift.pyis listed inNON_EXTRACTOR_MODULESand no stamped module changes, so existing deployments re-chunk nothing. A namespace that already indexed Thrift as text gains the rows withloom code backfill --paths-fileof its.thriftfiles.{(orxsd_all {) after its name, and an unfinishedtypedefskips what it read, so repeated unclosed headers cost their own length.Compatibility, rollout and rollback
.thriftfiles only)..thriftfiles gives an already-indexed namespace its Thrift symbols and edges..thriftto plain text; the Thrift rows written meanwhile stay until the files are re-indexed.Tests
tests/unit/test_code_lang_thrift.py(9 tests): kinds and qualified names; spans and signatures; sites through containers and include-qualified names, no base types, nothing from a comment; includes deduplicated and resolved nearest first; unterminated input; registration (detection,LANE_UNIT_LANGS,_lane_module); binding (same file, repository-unique,extends→ inherits,throws, builtins bind to nothing); two linear-time cases (assert_linear).tests/fixtures/code_lanes/thrift/acme/(synthetic, listed inNOTICE.md):machines.thriftincludesshared.thriftand its service extendsBaseService. Type names don't reuse the proto fixture's (Instance), which would make proto's repository-wide binding ambiguous.expected.jsonregenerated withpython -m tests.code_lanes.corpus --write; its diff is only the two Thrift files (459 → 462 edge keys:machines.thriftgains 1 include, 1 inherits and 15 uses_type edges).SAMPLESintests/unit/lane_walks.pygainsthrift, plus athrift-includeswalk, fortest_code_regex_cost.py.Run in Linux (
python:3.12-bookworm,uv sync --frozen --all-extras, as CI):the lane on the fixture: symbols with spans and signatures, sites, include resolution · test logs and the expected.json diff
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code. I checked every registration point against the adding-language-support guide, ran the lane on the fixture, ran the lane, corpus, stamp, manifest and cost tests in Linux and reviewed the regenerated
expected.json.I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed. (ruff format/check, mypy on the module, and the code-lane tests above were run; the fullmake test-unitwas not)expected.jsonregenerated; no other generated file changes)main; no merged migration is edited. (n/a)frontend/changed. (n/a)docs/describe the system as it now is. (code-index overview language table; code intelligence by language; adding-language-support lane list)git commit -s), as CONTRIBUTING.md asks.