Skip to content

A tree-less Thrift IDL lane in the code index - #760

Open
rahulsjha wants to merge 9 commits into
ClickHouse:mainfrom
rahulsjha:feat/thrift-lane
Open

rahulsjha wants to merge 9 commits into
ClickHouse:mainfrom
rahulsjha:feat/thrift-lane

Conversation

@rahulsjha

Copy link
Copy Markdown
Contributor

Closes: #624

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

The code index reads Thrift IDL files: structs, unions, exceptions, enums, typedefs, services and their functions become symbols, field and argument types become type-use edges, a service's extends becomes an inherits edge, and include "x.thrift" becomes an include.

Why

.thrift was in _CODE_TEXT_EXTENSIONS, so Thrift files were chunked and searchable but code.symbol, code.callers and code.outline knew nothing in them.

Scope and risk

A tree-less lane, following docs/development/adding-language-support.md and shaped like lang_proto.py:

  • server/code/domain/extract/lang_thrift.py: parse_thrift(path, source) -> ParsedFile, reference_sites, SITE_RULES, CHUNK_OPTIONS, wanted_paths / resolve_imports.
    • Symbols: struct / union / exception → struct; enum → enum; typedef T X → type; service → interface; each service function → function <mod>::Service.f whose signature is the function as written (Machine get(1: MachineId id) throws(1: shared.NotFound not_found)). A token scanner with one-pass brace matching; comments (//, #, /* */) and strings are skipped, so a definition inside a comment is not a symbol.
    • Sites: a field's, argument's, return's or throws entry's named type (field_type → uses_type, to struct / enum / type), and a service's extends (inherits, to an interface), bound by lane_units.bind_sites (same file 1.0, repository-unique 0.6, ambiguous → nothing). list<> / set<> / map<> are read through; base types (i32, string, binary, …) name nothing; an include-qualified shared.User binds by its last part.
    • Includes: include "shared.thrift" → the including file's directory and its nearest ancestors, then the root (lane_units.ancestor_candidates), settled against the namespace's live paths.
  • Registration: ".thrift": "thrift" in LANG_BY_EXTENSION, removed from _CODE_TEXT_EXTENSIONS; _parse_thrift in _PARSERS, thrift in LANE_UNIT_LANGS and _lane_module; "thrift": SCRIPT in name_scope; thrift in search_scope.
  • Extractor stamp: lang_thrift.py is listed in NON_EXTRACTOR_MODULES and no stamped module changes, so existing deployments re-chunk nothing. A namespace that already indexed Thrift as text gains the rows with loom code backfill --paths-file of its .thrift files.
  • Cost: an unclosed string or block comment runs to the end once; a struct header only accepts { (or xsd_all {) after its name, and an unfinished typedef skips what it read, so repeated unclosed headers cost their own length.

Compatibility, rollout and rollback

  • Wire: none (new rows for .thrift files only).
  • Operator steps: none required; a backfill of .thrift files gives an already-indexed namespace its Thrift symbols and edges.
  • Rollback: reverting returns .thrift to plain text; the Thrift rows written meanwhile stay until the files are re-indexed.

Tests

  • tests/unit/test_code_lang_thrift.py (9 tests): kinds and qualified names; spans and signatures; sites through containers and include-qualified names, no base types, nothing from a comment; includes deduplicated and resolved nearest first; unterminated input; registration (detection, LANE_UNIT_LANGS, _lane_module); binding (same file, repository-unique, extends → inherits, throws, builtins bind to nothing); two linear-time cases (assert_linear).
  • tests/fixtures/code_lanes/thrift/acme/ (synthetic, listed in NOTICE.md): machines.thrift includes shared.thrift and its service extends BaseService. Type names don't reuse the proto fixture's (Instance), which would make proto's repository-wide binding ambiguous. expected.json regenerated with python -m tests.code_lanes.corpus --write; its diff is only the two Thrift files (459 → 462 edge keys: machines.thrift gains 1 include, 1 inherits and 15 uses_type edges).
  • SAMPLES in tests/unit/lane_walks.py gains thrift, plus a thrift-includes walk, for test_code_regex_cost.py.

Run in Linux (python:3.12-bookworm, uv sync --frozen --all-extras, as CI):

pytest tests/unit/test_code_lang_thrift.py test_code_extractor_stamp.py test_code_lanes_corpus.py
       test_code_lanes_manifest.py test_code_regex_cost.py                                   967 passed
all tests/unit/test_code_*.py + test_lang_*.py, main vs branch back to back in one container:
  main:   1 failed, 5152 passed (cpp-deep-includes, timing)
  branch: no new failure from this change (timing flakes only, e.g. lang_hcl._REF_RE)
ruff format / ruff check / mypy lang_thrift.py                                               clean

the lane on the fixture: symbols with spans and signatures, sites, include resolution · test logs and the expected.json diff

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Claude Code. I checked every registration point against the adding-language-support guide, ran the lane on the fixture, ran the lane, corpus, stamp, manifest and cost tests in Linux and reviewed the regenerated expected.json.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed. (ruff format/check, mypy on the module, and the code-lane tests above were run; the full make test-unit was not)
  • Generated files are regenerated in this PR, not by hand. (expected.json regenerated; no other generated file changes)
  • A new migration is additive and numbered after the latest on main; no merged migration is edited. (n/a)
  • A behaviour change to any file a plugin ships bumps both plugin manifests' version. (n/a)
  • The frontend builds and its tests pass, if frontend/ changed. (n/a)
  • Docs under docs/ describe the system as it now is. (code-index overview language table; code intelligence by language; adding-language-support lane list)
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers.
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks.

lang_thrift.py reads structs, unions, exceptions, enums, typedefs, services
and service functions with a token scanner in the shape of lang_proto.py.
Field, argument, return and throws types (through list/set/map) and a
service's extends are reference sites bound by lane_units.bind_sites; base
types bind to nothing. Includes resolve against the including file's
directory and its ancestors. .thrift leaves the plain-text list.

Closes ClickHouse#624

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 18:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

rahulsjha and others added 2 commits November 18, 2026 05:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
@zlareb1 zlareb1 self-assigned this Nov 18, 2026

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented-out Thrift includes become real dependency edges. The inline example reproduces the error on this head. All 212 focused lane, extractor-stamp, manifest and inventory tests pass, but they do not cover includes inside block comments.

@zlareb1

zlareb1 commented Nov 19, 2026

Copy link
Copy Markdown
Member

This head has merge conflicts with main in docs/development/adding-language-support.md, server/code/app/index/ingest_parse.py, tests/code_lanes/corpus.py and tests/fixtures/code_lanes/NOTICE.md. Please rebase and resolve those conflicts so CI can validate the combined changes. GitHub reports mergeable: CONFLICTING; a local merge-tree check confirms these four files.

# Conflicts:
#	docs/development/adding-language-support.md
#	server/code/app/index/ingest_parse.py
#	tests/code_lanes/corpus.py
#	tests/fixtures/code_lanes/NOTICE.md
An include inside a block comment was read by the raw-source regex and became a
dependency edge. Includes now come from the token stream, which drops comments.
@rahulsjha

Copy link
Copy Markdown
Contributor Author

Thanks @zlareb1. Both are fixed:

  • Includes in comments (e78d117). parse() now reads an include from the token stream: an include token followed by a closed string literal. The comment-aware tokenizer has already dropped comments by then. Your example now returns includes == []. The raw-source _INCLUDE_RE is gone. test_includes_inside_comments_are_not_includes covers a block comment, //, #, an include followed by a comment on the same line, and an unclosed literal.
  • Conflicts (26ab218). I merged main (4cee575) into the branch. All four conflicts came from the Lua lane landing in the same lists, and I kept both entries in each: LANE_UNIT_LANGS and _lane_module in ingest_parse.py, the three lane lists and _lane_module in tests/code_lanes/corpus.py, the NOTICE row, and the docs table, which I re-formatted with prettier 3.9.9.

In a Linux container, the Thrift, Lua, extractor-stamp, manifest, corpus, known-gaps and regex-cost tests give 991 passed and 1 failed. The failure is test_a_lane_walk_costs_linear_time[cpp-deep-includes], the C++ lane, which this PR doesn't touch. My machine was running another container at the time, and the walk hit the test's 60 s deadline. Timed back to back, the 4N walk is the same on both trees: 2.97 s on main and 2.92 s on this branch. A cold run on main also took 81 s, so the failure comes from the load, not from this change.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commented-include finding is fixed. A separate valid field form still loses dependency edges: the parser reads field types only after a numeric ID. The focused parser, corpus, stamp, manifest and inventory checks pass; the inline reproduction is missing from that coverage.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-field-ID finding remains on this head. The parser is unchanged: struct User { 1: string id }\nstruct Envelope { optional User user } still yields no type sites, while adding 1: before optional yields the expected User site. All 215 focused lane, stamp, manifest and inventory tests pass; they still omit this valid field form.

The IDL grammar makes FieldID optional in struct fields, argument lists and
throws lists. _fields now reads each field as [ID:] [req] Type [&] name
[= value] [xsd options] [(annotations)], so an ID-less field's type is a
reference site, and default values and annotations are skipped.

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>

# Conflicts:
#	docs/development/adding-language-support.md
#	server/code/app/index/ingest_parse.py
#	server/code/domain/extract/lang_detect.py
#	server/code/domain/extract/lang_text.py
#	tests/code_lanes/corpus.py
…ric ID

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-field-ID finding is fixed, including argument and throws types, defaults and annotations. No further blocking findings after checking parser registration, include resolution, binding, corpus output and extractor-stamp isolation. All 231 focused lane, corpus, known-gap, stamp, manifest and Thrift cost checks pass on Linux.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A remaining plain-text classification assertion fails in CI and locally on this head. The new lane needs the corresponding test inventory update before merge; the earlier field-ID fix remains verified.

".rego": "config",
".mdc": "doc",
".proto": "proto",
".thrift": "thrift",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — Update the plain-text classification test when registering the Thrift parser. tests/unit/test_lang_text.py::test_code_without_a_parser_and_plain_text_are_kept_as_text still includes api/service.thrift and requires both text_lang_for_path(...) == "text" and parse_for_ingest(...).lang == "text". Removing .thrift from the text extension set makes the first assertion fail at line 35; the same test fails in unit part 2 and in a focused Linux run on this head. Move that fixture out of the unclaimed-text cases and assert that the production dispatcher selects the Thrift lane and its service symbol. Keep the assertions for the other text-only paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a tree-less Thrift IDL lane: the code index reads .thrift files as text only

3 participants