Retrieval eval suite for help desk and chatbot answers #198

Follow-up to #194 / #196 (hybrid search over docs, pages, blog, changelog and API reference).

Problem

We've confirmed the search system works end to end, but we haven't measured whether it surfaces the right information. Current coverage:

• Unit tests: a handful of hand-picked cases, using a toy embedding space for paraphrases and mocked model replies for rerank.
• Browser smoke test: content is indexed, site search returns hits, and the chat widget shows citations. That proves it runs, not that it's accurate.
• Never run: bun run search:eval (accuracy over a question set), embeddings with a real key, and a full tool-calling chatbot answer from a real model.

So any change to snippet size, scoring, the similarity floor or the rerank prompt is a guess, and the search can get worse without anyone noticing.

Proposal

Keep it small:

1. Sample site fixture, committed to the repo. One made-up product with about 15 docs pages, pricing and FAQ pages, a few blog posts and changelog entries, and a short API reference. Include deliberate tricky cases: two pages that say similar things, a fact that appears only in an FAQ, and an old changelog fix that has since changed.
2. About 50 questions with the expected source for each (<kind>:<source>, the format search:eval already uses), in four groups:
   • worded like the content
   • reworded ("money back" vs "refund")
   • typos and loose phrasing
   • questions the content doesn't answer (expected: nothing relevant)
3. A check in the test suite. Index the fixture into PGlite, run keyword search, and assert recall@1 and recall@5 stay above a set line, starting a bit under the first measured value. No API keys and no cost, so regressions are caught before merge.
4. A manual mode with real keys. Point search:eval at the fixture and report keyword only, keyword plus embeddings, and both plus rerank side by side. Run it when tuning search, not in CI.

Out of scope for now

• Automated grading of full chatbot answers by another LLM. Read about 15 answers by hand the first time: correct citation, no invented facts, says "I don't know" when it should.
• Dashboards or stored score history.

Needs

• An OpenAI key for embeddings (a few cents at this size).
• A chat model that supports tool calls, for rerank and full answers. The OpenCode provider ignores custom tools.

Done when

• The fixture and questions are committed.
• The keyword-only threshold check runs in the test suite.
• The first numbers for all three modes are recorded on this issue.
• Any obvious weak spots found are fixed or filed.