Skip to content

Native kernel emits no markdown path references: code → doc edges are silently absent for every default-routed language #1716

Description

@bompus

extractMarkdownPathReferencesFrom* exists only in the wasm TreeSitterExtractor. The native kernel emits no markdown path references, so code → documentation edges are silently absent for every default-routed language — which is all of TypeScript, TSX, JavaScript, JSX, Java, Python, Go, C, C++, Rust, C#, Ruby, PHP, Swift, Kotlin, R, Lua, Luau, Scala and Dart.

Since 1.5.0 ships kernel prebuilds and DEFAULT_ROUTED covers the JS/TS family, this means the feature does not run in a released build on the languages it was written for.

Reproduction

Two tests already in the tree fail whenever the kernel is present, and pass without it:

__tests__/extraction.test.ts
  > Code to Markdown Reference Extraction > should extract Markdown path references from code string literals
__tests__/integration/full-pipeline.test.ts
  > Integration: full pipeline > resolves code string references to Markdown headings
# kernel prebuilt in codegraph-kernel/prebuilds/<platform>/
node node_modules/vitest/vitest.mjs run __tests__/extraction.test.ts -t "Markdown path references"
#   Tests  1 failed

CODEGRAPH_KERNEL=0 node node_modules/vitest/vitest.mjs run __tests__/extraction.test.ts -t "Markdown path references"
#   Tests  1 passed

The received value shows the reference is not merely mis-shaped, it is absent — only the ordinary call reference survives:

+ Array [
+   Object {
+     "fromNodeId": "function:da6cf4de2954c0915a85d4b2909a9aaa",
+     "referenceKind": "calls",
+     "referenceName": "fs.readFileSync",
+   },
  ]

Where the gap is

Every call site of the extractor is inside src/extraction/tree-sitter.ts — 14 of them (:972, :2169, :2234, :2275, :2290, :2773, :2890, :2914, :2955, :2977, :3053, :5489, plus the two private methods at :1512 and :1532). There is no counterpart on the kernel side, and tryKernelExtract returns before the wasm extractor is constructed (tree-sitter.ts:7018), so a routed language never reaches any of them.

Why CI does not catch it

Two things have to line up, and they do:

  1. The kernel prebuilds are not tracked in git (git ls-files codegraph-kernel/prebuilds → empty), so an ordinary checkout runs the wasm path and both tests pass.
  2. The only workflow that runs tests is release.yml, and the one step that runs with prebuilds downloaded (Kernel contract + full walker-parity gate, :179) runs CODEGRAPH_KERNEL_EXPECT=1 npx vitest run __tests__/kernel-*.test.ts — the kernel tests only. The full suite never executes with a kernel loaded.

So the divergence is invisible unless someone builds the kernel locally and runs the whole suite, which is how I hit it.

The parity harness would catch it, with one line

__tests__/kernel-tsjs-parity.test.ts already compares kernel against wasm and already asserts on references:

expect(k.refs, `${filePath}: refs`).toEqual(w.refs);

It does not fire because no fixture exercises the path. The only .md strings under __tests__/fixtures/kernel-parity/ are in comments (torture.R:2, torture.rs:2), which are not string literals and cannot reach isMarkdownPathStringNode.

Adding a single markdown path string literal to torture.tsx — e.g. const GUIDE = '../docs/guide.md#install'; — would turn this from an invisible gap into a failing parity assertion, and would keep it caught. That seems worth doing regardless of when the kernel side is implemented, since it is the difference between "known gap" and "silently wrong."

Scope note

Markdown files are unaffected: markdown is not in DEFAULT_ROUTED, so .md files are still indexed by their own extractor and the doc tier in codegraph_explore works normally. What is lost is only the edge from a code file's string literal to the document it names.

I have not touched the kernel side — implementing it there is a Rust change I did not want to guess at, and the choice between implementing it in the kernel versus running the capture as a TS post-pass over kernel output is a design call for you. Happy to do either if you say which.

Found while running the full suite on Node v24.16.0 against #1699.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions