@imqueue search-ranker

One file — search.js — holding the search ranker shared by everything that searches the @imqueue documentation:

Consumer	How it uses this file
imqueue.org / imqueue.com	Submodule at vendor/search-ranker, content-hashed and served as /js/search.<hash>.js. Runs in the browser: the ⌘K dialog, the /search/ page and the blog sidebar.
@imqueue/mcp	Submodule, copied into dist/. Runs under Node and inside a Cloudflare Worker to answer the search_docs MCP tool.

It exists as its own repo so that those two never drift. Before the split the MCP server carried its own ranker, and the two answered the same question differently — measurably so: on a 1,000-query corpus the site ranker placed a correct result in the top 6 for 99.5% of queries against the MCP ranker's 97.2%, and the gap was invisible because nothing compared them.

Two environments, one file

The file is a plain IIFE, no build step, ES5-compatible syntax throughout — because it is served to browsers directly and executed in a Worker where eval and new Function are both forbidden.

Under Node it detects the absence of a DOM and exports the ranker instead of wiring up any UI:

if (typeof document === "undefined") {
  module.exports = { parseQuery, prepare, prepareSections, search, groupKey, state, FEED_V };
  return;
}

Everything below that line is DOM: the dialog, the results page, keyboard handling, analytics. A Node consumer never reaches it. That early return is load-bearing — do not turn the export block into an else, and do not move it, or a Worker starts evaluating document.addEventListener.

The feed contract is versioned, and that is not decoration

The ranker does not carry a corpus. It reads four JSON feeds built by scripts/lib/search-corpus.js in the website repo:

/search-index.json        every page, API symbol and question-shaped section — no bodies
/search-text.json         the prose corpus at heading-section granularity
/search-peer-*.json       the same two shapes for the other edition

Records are positional arrays, not objects, because the index is downloaded on every first search. So a field appended in the middle of a tuple does not throw and does not return nothing — it silently scores the wrong text.

Two independent declarations guard that:

• FEED_V here says which shape this ranker reads;
• FEED_V in the website's corpus generator says which shape it writes;
• scripts/check-search-index.js fails the build when they disagree, or when a built feed carries a third value.

Bump FEED_V in the same change that alters a tuple, in both repos. It is deliberately not one shared constant — a shared constant would agree with itself and assert nothing.

The MCP server fetches the feeds from the live site at runtime while its ranker is pinned to a commit here. A pinned-stale ranker reading today's feeds is exactly the failure this version number exists to make loud.

Working on it

There is no test suite in this repo, on purpose: the ranker cannot be judged without a corpus, and the corpus belongs to the website. The measurement harness lives in imqueue.com/scripts/search-kpi/:

git clone --recurse-submodules https://github.com/imqueue/imqueue.com.git
cd imqueue.com && npm ci && npm run build:all
npm run kpi:search     # relevance against a natural + an artificial query set
npm run kpi:recall     # recall@6 — the metric that matters to an agent
npm run kpi:compare    # this working copy against any git ref, query by query

Edit vendor/search-ranker/search.js inside that clone, measure, then commit in the submodule and update the pointer in both consumers.

Two things the harness has already established, worth knowing before tuning:

• A flat average hides mass churn. Read the per-query deltas kpi:compare prints, not the summary line. A change that moves the macro average by +0.1 while moving 300 queries is not an improvement, it is a different ranker.
• The artificial query set prefers a broken ranker on some signals. When the two sets disagree, the natural set — real queries, human-judged — wins.

Licence

GPL-3.0, matching every other repo in the organisation.