Skip to content

Example: search two Markdown documents with citations, and abstain on an off-topic query - #988

Open
rahulsjha wants to merge 4 commits into
ClickHouse:mainfrom
rahulsjha:feat/example-document-search
Open

rahulsjha wants to merge 4 commits into
ClickHouse:mainfrom
rahulsjha:feat/example-document-search

Conversation

@rahulsjha

Copy link
Copy Markdown
Contributor

Closes #720.

sdk/python/examples/document_search.py, on the agent_memory.py layout: add two Markdown documents, search them and print the hits as citations, and get nothing back for a question they don't answer.

  1. Add. knowledge.add takes two short Markdown documents, each with its own source_uri and title.
  2. Cite. knowledge.search runs once per subject. Each hit prints as a citation: source_uri#chunk_idx, the section, the passage's first line and its similarity. The top hit has to be the document the question is about.
  3. Abstain. An off-topic question goes through bot.client.search.query, with a comment saying why: knowledge.search alone still returns its nearest chunks, and search.query drops knowledge hits below LOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITY. No passage may come back.
  4. Scope. Every knowledge.search and search.query passes namespace_scope: "this".
  5. Cleanup. A finally runs bot.knowledge.delete(source_uri) on both documents.

It exits 0 only when every check passes, and otherwise prints each failure and exits 1.

The default floor and the local embedder

The default LOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITY is 0.30, and that is too low for the local embedder:

Similarity (local bge-small) With the 0.30 floor
On-topic: the right document's passage 0.88 and 0.89 kept
Off-topic ("What is the boiling point of water at the top of Mount Everest?"), all four passages 0.34, 0.35, 0.40, 0.43 all kept

With the default, search.query returned all four passages and the script exited 1. With the floor at 0.5 on the server, it abstained and every check passed (below). The script and the README say the floor has to sit above what the embedder scores unrelated text.

@zlareb1, is 0.30 meant for the provider embedders? If the local embedder is the released stack's default, a default floor per embedding model might be worth an issue. I'm glad to open it.

Real run

Local Loom from source (ClickHouse 26.10 with the repo's deploy/clickhouse-config, auth on, LOOM_EMBEDDING_MODEL=local (bge-small, fetched with python -m server.embedding.local_models fetch bge-small), LOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITY=0.5, a minted writer token):

added docs://example/refunds.md (Refund policy)
added docs://example/shipping.md (Shipping times)

Q: How many days do customers have to return an item for a refund?
  [docs://example/refunds.md#0] Refund policy > Eligibility: Customers can return an unopened item within 30 days of delivery for a full refund.  (0.88)
  [docs://example/refunds.md#1] Refund policy > How refunds are paid: Refunds go back to the original payment method within 5 business days.  (0.80)
  [docs://example/shipping.md#0] Shipping times > Domestic: Standard domestic orders arrive in 3 to 5 business days.  (0.61)

Q: How long does international shipping take?
  [docs://example/shipping.md#1] Shipping times > International: International orders ship by air and arrive in 7 to 14 business days.  (0.89)
  [docs://example/shipping.md#0] Shipping times > Domestic: Standard domestic orders arrive in 3 to 5 business days.  (0.80)
  [docs://example/refunds.md#1] Refund policy > How refunds are paid: Refunds go back to the original payment method within 5 business days.  (0.57)

Q: What is the boiling point of water at the top of Mount Everest?
  no knowledge passage came back: the documents do not answer it
all checks passed

Docs

  • sdk/python/examples/README.md: a document_search.py section with the command, the steps, the embedder and floor note, this run, and a link to Lab 2 of the teaching guide as the exercise version.
  • docs/use-cases/document-search.md: a link to the script from "Start".
  • docs/reference/python-sdk.md: the script listed under "Examples".

Tests

tests/unit/test_sdk_example_document_search.py loads the script by path and runs it against an httpx.MockTransport server:

  • Every check passes, so it exits 0. Both documents are deleted, every search sends namespace_scope: "this", and the citation line prints.
  • knowledge.search answers 422. The ValidationError propagates and both documents are still deleted.
  • The off-topic query returns a knowledge hit, so it exits 1 with both documents deleted.

Checks (Linux container, Python 3.12)

  • pytest tests/unit/test_doc_links.py tests/unit/test_public_copy.py tests/unit/test_sdk_example_*.py: 298 passed. That includes the new test (3) and test_sdk_example_agent_memory.py.
  • python -m scripts.codegen.docs_reference --check: current.
  • ruff check and ruff format --check on the example and its test: clean.

… and abstains off topic

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
…or it needs

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 17:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@zlareb1 zlareb1 self-assigned this Nov 19, 2026

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. Checked knowledge citations, namespace scope, similarity-floor abstention and cleanup against the SDK and server contracts. All five document-search and existing agent-memory example tests pass; the embedder and floor requirements are documented.

@zlareb1

zlareb1 commented Nov 19, 2026

Copy link
Copy Markdown
Member

rebase

@zlareb1

zlareb1 commented Nov 19, 2026

Copy link
Copy Markdown
Member

GitHub reports mergeable: CONFLICTING for this head. A local merge check confirms conflicts in docs/reference/python-sdk.md and sdk/python/examples/README.md after the support example merged. Please rebase and retain both examples in those files so the combined documentation can be validated.

…-search

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>

# Conflicts:
#	docs/reference/python-sdk.md
#	sdk/python/examples/README.md

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. Checked the rebased example against the SDK and search similarity-floor contract, including namespace scope, citations and cleanup. All five document-search and agent-memory example tests and 270 documentation checks pass; both existing examples remain in the merged documentation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example: search two Markdown documents with citations, and abstain on an off-topic query

3 participants