[s3rot0n1n]

Implements #194: industry-standard RAG for the site chatbot and help desk AI, grounded in docs, builder pages, blog, changelog and the API reference.

What changes

One search index (search_chunks) replaces document_chunks, document_index_jobs and the docs-only documentation_embeddings / documentation_embedding_configs.

• Every team gets Postgres full-text search (generated tsvector, GIN index, stemming, prefix match).
• With an embedding key, vector search (pgvector HNSW) runs too, and the two lists are merged with reciprocal rank fusion.
• No key means keyword-only. The fake hash-vector fallback is gone.

Indexing is a per-team reconcile worker (search/indexer.ts):

• Publishing anything queues the team: docs activation/purge/API reference (via the docs delivery outbox), page releases, blog/changelog publish/edit/delete, and AI setting changes.
• The worker rebuilds only sources whose hash changed, drops unpublished ones, and embeds chunks lacking a vector for the current embedding model.
• Search also filters to what's live at query time (active docs publication, active site release, published articles), so a lagging index never returns unpublished content.

Builder pages are indexed one chunk per block, covering titles, descriptions, FAQ questions and answers, plans and rich text. Chrome, forms, embeds and noindex pages are skipped.

Contextual chunks: each chunk is indexed and embedded with a kind › title › section line plus the page summary.

Reranking: the chatbot and help desk AI fetch 30 candidates and rerank them with the team's own chatbot model. Any failure falls back to the fused order. The agent prompt asks for standalone queries and a retry with other wording. Site search doesn't rerank.

Feedback (M7):

• Thumbs up/down on bot answers in the widget, via POST /public/:teamId/chat/feedback, stored on ticket_messages.feedback.
• A thumbs down files the question under the existing documentation gaps report (negative_feedback).
• The inbox shows the visitor's rating on bot answers.

Dashboard:

• The docs "Keyword / Smart" switch is replaced by a read-only "Search quality" summary showing what the team's keys unlock: keyword always, meaning-based with an embedding key, reranking with a chat key.
• The same summary is added to Settings → AI.

Also:

• Site search links pages, blog and changelog correctly.
• Changelog entries get #slug anchors.
• bun run search:eval <team> <questions.json> [--rerank] reports recall@1 and recall@5.
• Removed the unused getChatAnswer / streamChatAnswer.
• docs/architecture.md §6–7 rewritten.

Migration 0015_search_index creates the new tables, adds ticket_messages.feedback, drops the old tables, and queues every existing team for indexing.

Testing

• tsc is clean for api, web and sites. Tests pass: api 771, web 97, sites 82.
• New tests cover:
  • page extraction, live-scope filtering and team scoping
  • reconcile, including no rework on unchanged content
  • a paraphrase found via vectors (mocked embeddings)
  • rerank ordering and fallbacks
  • feedback
• Browser smoke on a local stack:
  • A doc published in the editor became searchable within seconds.
  • A page and a blog post were indexed and linked correctly.
  • Settings show the right search tiers.
  • The widget showed an answer citing /pricing and /docs/refunds#refund-policy. Thumbs down was stored, shown in the inbox and listed under "Review gaps".
  • Site search returned both the doc and the page.
• Reranking ran live against a free OpenCode model and kept the 2 relevant passages out of 3.

QA smoke with a real OpenAI key

Deployed to QA (docs-pages-issues-356d889) and tested on a seeded team: 4 docs, a pricing FAQ page, a blog post and 2 changelog entries.

• Embeddings are built, and meaning-based search finds paraphrases ("reimbursement" finds Refunds).
• The chatbot answers questions about pricing, refunds, QuickBooks and payouts correctly, citing the right source. The help desk auto-replies to tickets from the right doc.
• Ratings show in the inbox, and the gaps report records negative feedback and unanswered questions.
• Removed content stops appearing in search within 2 seconds. Tested 7 ways: unpublishing a doc, turning off indexing for one doc, turning off docs search for the team, deleting a blog post, turning off indexing for one changelog entry, turning off changelog indexing for the team, and hiding a page from search engines. The chatbot no longer retrieves or cites removed content.

Follow-up fixes from that test:

• Uncited answers: the model sometimes answered correctly but left out the [n] marker, so the answer was replaced with "couldn't verify". The model is now asked once more to cite the passages or reply NO_ANSWER. A question is recorded as a documentation gap only when search found nothing or the model says the passages don't answer it.
• Repeated markers: two passages from one source now produce one marker ([1], not [1][1]).

Retrieval noise and relevance scores are tracked in #199. Building a retrieval eval suite is tracked in #198.

Not verified live: the citation retry itself (the model couldn't be made to skip a citation on demand), and Azure embeddings.

Found while testing, not changed here

• With the OpenCode provider, a chat timeout leaves an unhandled TimeoutError that crashes the API process.
• Docs publishing mixes Postgres now() with JS UTC dates on timezone-less columns, so it fails on non-UTC databases. The new index queue avoids this.
• A plain email in a doc still breaks saving (Page builder: saving rich text with a plain email address fails with a 500 ("Couldn't save changes") #150).
• sitesOrigin hardcodes port 3002 for localhost URLs.

Closes #194

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-07T10:11:51.276291Z	f9538eb	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

frontlit/apps/api/src/documentation/index-status.ts

Lines 45 to 49 in f9538eb

	(item) =>
	item.documentId === (doc as any).documentId &&
	isDocIndexable(item.meta),
	)
	? ("indexed" as const)

Report docs as pending until indexed

When a doc has just been published or the search-index worker fails, this branch reports it as indexed solely because it appears in the active publication; it never checks whether chunks were created or whether the team's index request failed. As a result, document and MCP responses can report success while the new content is not searchable and the error is hidden. Use the worker and chunk state to distinguish indexed, pending, and failed.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".