plugin: enforce index-first search with plugin hooks

[ekechinwokah]

Installing the plugin now makes the index the way agents search code, and the
index has exactly one query surface: read-only SQL with ranked search
table-valued functions.

The MCP surface (breaking)

• The search tool is gone. Two tools remain: sql and reindex.
• sql is the search surface: vector search for semantic, hybrid_search
  for the fused keyword + semantic pass, bm25_search(...) for the keyword-only
  window while vectors backfill, GROUP BY over either for counts/rankings,
  regexp_like in WHERE. The instructions and tool description lead with the
  retrieval query shape instead of presenting sql as an analytics side-tool.
• The index builds eagerly at server startup instead of on the first query;
  a query that beats the build still triggers it inline (CX_AUTO_INDEX=0
  restores the strict error).

The search path is removed, not deprecated

Core search() and its result shapes, the cx search CLI command,
CX_SEARCH_K, and the search-flavored usage receipt are deleted. Tests are
ported, not dropped: every retrieval assertion now goes through runSql with
the TVFs; truncation asserts partialIndex() directly; the bench recall lane
ranks through hybrid_search SQL (its vector-only lane keeps the raw engine
call — it isolates the embedder, not the agent surface). README, AGENTS.md,
and llms.txt describe the two-tool surface.

Enforcement hooks (ship with the plugin)

hooks/hooks.json + hooks/deny-grep.mjs, auto-loaded on install:

• Deny the Grep tool and standalone grep/rg/git grep Bash commands with
  a reason redirecting to the sql TVFs — but only once the index fully covers
  the repo (manifest present, vectors ready, nothing truncated by the file
  cap). Until then grep passes through: agents are never forced onto an index
  that can't answer yet.
• grep as a pipe filter on other command output always passes — filtering
  logs or test output is a job the index does not do.
• Deny the legacy search tool (older server builds) and vector_search
  inside sql statements, same redirects.
• A SessionStart note announces the policy, adapting to index state.
• The Bash hook uses Node (already the package's floor) rather than jq, which
  isn't guaranteed on user machines.

The denials are self-correcting: Claude Code feeds the reason back to the
agent, so a reflexive grep becomes a sql query on the next step.

Verification

• tsc --noEmit clean; vitest 100/100 across 13 files.
• Hooks pipe-tested against synthetic payloads and live-tested in a Claude
  Code session: standalone rg denied with the redirect, pipeline grep
  untouched, and the redirected query auto-built the index and answered in
  one step.

Ships to users as the next npm release (needs a version bump + publish);
the hooks take effect on plugin update as soon as this merges.

[ashishmishra26]

The hooks only exist for Claude Code plugin users, but the search removal hits every MCP client. Cursor, Windsurf, and anyone on plain claude mcp add-json get no enforcement and no SessionStart nudge; for them this release is just a harder query surface. The most common agent action goes from search(query) to a hand-composed SELECT with a TVF, an embed map, and SQL escaping (the bench port in this PR needed replaceAll("'", "''") to survive quotes, and agents will hit the same thing mid-session).

Since the legacy-search deny in the hook already proves the hook can steer agents to sql without the tool being deleted, could we split this? Land hooks + eager indexing now, and hold the search deletion for a follow-up. That also keeps the blast radius of each change separate if something regresses.

[ashishmishra26]

The README today argues three tools is deliberate design ("one way to find, one way to count"); this PR rewrites that paragraph to argue two tools is deliberate design. Both can't be self-evident, so the deciding factor should be data about agent behavior, and I don't see any in the verification section. tsc, vitest, and the recall bench all pass, but recall measures ranking quality, not whether an agent querying through SQL finds code as reliably and cheaply as one with a dedicated search tool. That's the actual question this PR bets on.

Before we delete the path, can we run even a small A/B: N fixed questions through a live session against the same repo, search-tool surface vs sql-only surface, comparing turns-to-answer, token spend, and failed/retried queries? If sql-only holds up, the deletion ships with confidence and the number goes in the PR description. If it doesn't, we just saved every non-Claude client from a regression.

[ashishmishra26]

indexState walks up from the session cwd, but the deny fires on what the command targets. Two false-positive classes:

1. Multi-repo workspaces: if the cwd's repo has a ready index, rg foo ../other-repo gets denied even though other-repo was never indexed. The hook should probably inspect the grep target path (or the Grep tool's path input) rather than assuming cwd.
2. Things the index excludes by design: gitignored files, files over the size cap, and edits newer than the last sync. "ready" means the index covers what it indexes, not everything grep can see. An agent grepping for a symbol it added thirty seconds ago can get denied while sql also misses it, which is the worst version of the redirect.

Related: is there an off switch? A search plugin silently disabling a built-in tool needs a documented escape hatch (CX_NO_ENFORCE=1 or similar) or the first confused user files an angry issue instead of an env var. Happy with deny as the default, just want the exit to exist.

[ekechinwokah]

thanks — all three addressed in a001958:

1. target, not cwd. the hook reads the command's explicit path targets (and the Grep tool's path and checks each against the discovered index root — a target outside the repo is allowed silently, so passes whatever this repo's index says.
2. index-excluded classes. gitignored (), over , dot-paths, nonexistent → allow; if sql has no rows for it, grep stays open. the edit from thirty seconds ago still reads as covered (it exists, it isn't excluded), so that case is handled by the escape below rather than the coverage check.
3. off switch: . the deny message now names it, plus the per-need exit — re-run the command prefixed with and the decision becomes instead of . both in the README config table.

sixteen decision cases pinned against a fixture repo: deny/allow/ask, orphaned-manifest fail-open, grep (BSD grep, GNU compatible) 2.6.0-FreeBSD.
BODY
)

[muralikpbhat]

Reviewed across the two-tool surface, the eager index, the grep-deny hooks, the ported tests, and the docs. The core direction is good and a lot of it is executed carefully; the blocking items are the grep-deny trapping the agent, a lost readiness signal that compounds with eager indexing, two weakened tests, and two stale doc surfaces.

What's solid (no change needed)

• search → sql port is faithful. The TVFs preserve every field search() returned — path, line range, score (via TVF ordering), content — and k; content is now un-capped rather than truncated at 4000 chars, a gain. No ranking/field capability lost in the port itself.
• Eager index at startup is correct. buildIndex goes through the exclusive() mutation lock, so a concurrent first query's ensureIndexed sees the build in flight and awaits ctx.mutation (no double-build); it's setImmediate-deferred so it does not block the MCP handshake; and CX_AUTO_INDEX=0 correctly gates the eager block and restores the strict noIndex error.
• Truncation + usage ports are equivalent (partialIndex() asserts the same filesSkipped/file-cap condition; the sql receipt is asserted, not just non-empty), and CX_SEARCH_K / DEFAULT_SEARCH_K / searchCmd / cx search are fully removed with no dangling refs. README/AGENTS.md/llms.txt are updated to two tools.
• The hook is built in the safe direction — it fails open (unparseable payload, probe throw, missing manifest → allow), only denies when vectors === "ready", and matches an anchored regex (/^\s*(rg|grep|git\s+grep)\s/) so foogrep, ps | grep, cmd && grep don't over-match. Hygiene is clean (no secrets in the hooks).

Findings to address

1. The grep-deny traps the agent (blocking). Three problems, design fix below:

• Hard deny with no per-need fallback (deny-grep.mjs:95). Once vectors are ready it returns permissionDecision: "deny"; the only escape is CX_NO_EMBED + a server restart (global, not something the agent can do mid-task). When SQL search genuinely comes up short, the fallback grep is walled off and the agent loops search→deny→search.
• False-complete gate (deny-grep.mjs:38 reading indexer.ts:152). Completeness = truncatedFiles == 0, but truncatedFiles only counts the file-count cap. Files excluded by shouldIndexFile() and the maxFileBytes byte cap (chunker.ts:110, config.ts:38-39) are silently unindexed and unrecorded — yet the hook claims "fully covers this repo." So grep API_KEY .env, grep <tok> Cargo.lock, a >1 MB generated JSON, a .csv → denied, and hybrid_search/bm25 have no rows for them → findable by no tool.
• Over-broad Bash denial (deny-grep.mjs:43) — scopes on command prefix only, never on the target, so grep --version, grep pattern /var/log/system.log, grep foo ./build/output.txt, git grep in another tree are all blocked though none are repo code-search. (Also note the flip side: true && grep, LANG=C grep, cat f | grep slip the anchored regex — so enforcement is trivially bypassable for callers who don't need it, while the honest bare command is walled. Neither a reliable guardrail nor a usable fallback.)

2. The vector-readiness signal is gone, and eager indexing makes it bite (blocking). The deleted search() returned note: "vectors not ready yet — keyword-ranked only" on every call and auto-fell-back to bm25. The SQL surface surfaces readiness nowhere (except the one-time auto_indexed note when the query itself built the index). Because the index now builds eagerly at startup, an agent querying in the first seconds calls hybrid_search while vectors backfill and gets a keyword-only/partial result with no signal it's degraded — treating it as full hybrid recall. The old surface was self-correcting here; the new one is silent (searcher.ts runSql 94-114 / server.ts sql handler 226-260 pass straight to the engine regardless of manifest.vectors).

3. Two ports were weakened, so the regressions they guarded are now untested.

• integration.test.ts:111-118 — old "keyword ranking while vectors not ready" forced a vectors:'building' handle and asserted ranking==='keyword' and note ~ /vectors not ready/. The port runs bm25 against a fully-ready handle and only asserts rows > 0 — the not-ready path (finding #2) is no longer exercised.
• sync.test.ts:124-131 — old asserted ranking==='hybrid' proving sync re-embedded; dropped. The query term is bm25-satisfiable, so the test passes even if the vector stage never re-embeds the synced rows (the exact regression it was written to catch).

4. Two doc surfaces still advertise the removed search tool. skills/code-context/SKILL.md still says "three MCP tools (search, sql, reindex)", has a full ## search section, routes "how does X work / where is Y handled" to search, and tells agents to ToolSearch "search sql reindex" — it points agents at a tool that no longer exists and is actively denied by the hook. docs/faq.md still answers "Three, by design: search…" and references cx search. (README/AGENTS/llms.txt were updated; these two were missed.) Minor: a couple of stale search/sql comments (server.ts:67, the deny-grep header).

The compounding case worth calling out: on a freshly-started server an agent gets keyword-only results (vectors backfilling) → no signal it's degraded (#2) → reaches for grep to sanity-check → hard-denied with no fallback (#1) → and if it consults the skill, it's told to use search, which is gone (#4). The trap arrives from three directions at once.

Design: a grep-deny that nudges without trapping

Make the decision three-way, keyed on the target and an explicit fallback intent, so the ask fires only on a genuine fallback (not on every grep, which would just trade the trap for prompt-fatigue):

1. Target the index doesn't cover → allow silently. A grep whose target is an unindexed file/path (excluded by shouldIndexFile/byte-cap, a log, ./build/…, a path outside the repo) is auto-allowed — SQL can't answer it anyway. This alone removes most of the trap and fixes the over-broad denial. It requires the completeness/coverage check to account for the real excluded set, not just the count cap.
2. Reflexive grep on indexed source, no fallback signal → deny + redirect (as today), but the redirect names the escape. Keep nudging the first reflexive grep to hybrid_search — but the message must say how to fall back (it currently names none), e.g. "if the index search didn't find it, re-run with CX_GREP_FALLBACK=1 and it will be allowed."
3. Grep carrying the fallback marker → ask. The agent reaches for this only after a search came up short, so the human is prompted exactly when they should weigh in ("the agent tried search, it wasn't enough — allow this grep?") and nowhere else.

Notes: allow/deny/ask are all valid permissionDecision values and the hook already has the command string + the manifest, so this is a modest change to deny-grep.mjs, not a redesign. The marker relies on the agent's honesty — that's fine; the goal is making SQL the default, not hard-forbidding grep, and the ask is where a real human gate lives. Separately, guard the corrupted/stale-manifest case (a table deleted after the manifest went ready currently denies grep and errors sql — losing both paths); confirm the index is queryable, or fail open, before denying.

Net

The two-tool surface, SQL-as-search, and the eager-index mechanics are a good foundation and cleanly done. I'd hold on: the grep-deny redesign above, restoring the vectors-not-ready note on the SQL path, restoring the two tests' state assertions, and updating SKILL.md + faq.md to the two-tool surface.

[ekechinwokah]

keeping the deletion, running the A/B.

why not split. sql-as-the-only-door is the position, not a side effect, and a compat window recreates what this PR removes — two doors onto one index, choice per call. AGENTS.md's "near-duplicate retrieval tools worsen selection" bites hardest during a transition, not least.

non-Claude clients. they lose the hooks, not the steering. for them the steering is the sql tool description, which now leads with the retrieval shape (SELECT ... FROM hybrid_search(...) plus the embed map) instead of framing sql as an analytics side-tool. on escaping: the semantic text travels through the embed map and never sits inside SQL quoting, so ''-doubling only bites a keyword terms string that itself contains a quote — the bench port hit it because it interpolates gold query text.

the A/B — agreed, and it gates the release. same bench/ lanes and question set, search-surface (0.2.0) vs sql-only, reporting turns-to-answer, tokens, failed/retried queries; table in the release notes before I tag. if comprehension regresses materially the release holds. calibration: the aggregation lane (-43% tokens, -71% calls) was already sql, and the deleted tool's job maps 1:1 onto hybrid_search in FROM — but that's the bet, not the evidence.

[ekechinwokah]

thanks for the depth here — the coverage gap and the lost readiness signal were both real. all four on a001958 (fa43f77 merges current main; clean against v0.2.0):

1. grep-deny: three-way and target-scoped, as sketched. uncovered target (outside repo, gitignored, over the byte cap, dot-path, nonexistent) → silent allow; covered source → deny with a message that names its own exits; CX_GREP_FALLBACK=1 prefix → ask, so the human gate fires on the fallback and nowhere else. CX_NO_ENFORCE=1 disables outright, grep --version passes, and a ready manifest whose table is gone fails open rather than denying grep while sql also errors. one divergence: no global excluded-set completeness signal — computing it is a re-walk — coverage is checked per-target at decision time, which is exactly where shouldIndexFile / the byte cap / gitignore bite.
2. readiness note restored on the sql path. vectorsNote() rides every sql result until the backfill lands. you're right that eager indexing is what made the silent window bite: it moved the degraded moment to a session's first queries.
3. both tests have their state assertions back. integration asserts the not-ready note against a vectors:'building' manifest; sync proves re-embedding through vector_search, which can only return rows that have vectors — strictly stronger than the old ranking==='hybrid', which only proved the manifest flag.
4. SKILL.md + docs/faq.md on the two-tool surface. both still advertised search while the hook denied it — the worst leg of the compounding case. the stale search/sql comments in server.ts and the hook header went with the rewrites.

compounding case end to end now: a first-seconds agent gets bm25 rows carrying the backfilling note, its sanity-check grep passes (deny gates on ready), and the skill it consults describes sql.

gates: tsc --noEmit clean, vitest 100/100 across 13 files.

[ekechinwokah]

Correcting my two earlier replies before anything else: both overstated the coverage fix, and adversarial re-verification against real indexes proved it.

What I claimed vs what was true. I said "if sql has no rows for it, grep stays open" and that coverage was checked "exactly where shouldIndexFile / the byte cap / gitignore bite." Executed counter-examples: on a fixture whose table held two files, the hook denied grep commander package-lock.json, grep 1 data.csv, grep Apache LICENSE, and grep -rn Foo vendor/ — sql had zero rows for all of them, so the agent lost both paths, which is precisely the failure you both flagged. The re-derived rules also allowed indexed dotfiles and leaked on rg pat src | head (pipeline words read as targets). My "sixteen decision cases" tested the hook against a model of the index, never against the index — that was the methodological hole.

What replaced it (65b8093). Coverage is no longer modelled; it is read from the index's own record. Filestate entries now carry the chunk count each file produced (stamped by build and sync, 0 included), and the hook covers exactly the files with rows. Ground truth is the decisive gate now: on four fixtures built by the real indexer, the hook's deny set equals SELECT DISTINCT path FROM chunks for every file, directory, and glob form — including empty __init__.py, NUL-carrying .py, symlinks, vendor/, lockfiles, and indexed dotfiles. A wholly pre-upgrade filestate (no counts anywhere) fails open until a rebuild stamps it. manifest.root (new, optional, format version untouched) keeps a CX_INDEX_DIR-relocated index scoped correctly, and is only trusted when it agrees with where the index was found, so a copied repo's stale root cannot switch enforcement off silently.

Parser, same commit. Operands end at an unquoted |/# (closes rg pat src | head), heredoc bodies are data, if/while/for/{/(/! and backgrounding & are stepped over, launchers match by basename (/usr/bin/rg), git's global flags are walked (-C also moves the scope), wrapper value-flags are per-wrapper (env -i was eating the launcher), and glob star-runs collapse before compiling — a 24-star token used to wedge the regex past the hook timeout. Still failing open by declared design: $(...), sh -c bodies, separate-value flags (-m 5), find -exec.

The distribution gap (088ab81). The hooks only ever reached plugin installs; a hand-wired server (npx, claude mcp add-json) got tools with no steering — restarting changes nothing, which is how this thread started. cx install is back for the two-tool surface: copies the hook to ~/.claude/hooks, merges the two settings entries, embeds the absolute node it ran under (a client process often has no node on PATH; a hook that can't find node fails silently). The settings edit is conservative — ownership per hook command on the resolved path, symlink-following atomic rewrite, hardlinks written in place, sibling settings files consulted before the shared script is deleted, project-scoped .claude refused without --force, byte-identical uninstall. Claude Code only, and says so.

Gates: tsc clean, vitest 199/199 across 15 files (73 enforcement, 39 install), ground-truth equality on fx1/fx3/fx5/fxpy, wildcard-flood answers in ~125ms, install round-trips byte-identical through symlinked/hardlinked/foreign-hook settings. The verification harness runs the real indexer and compares against the table — the standard any future change to the hook should be held to.