Repository navigation
Conversation
… and abstains off topic Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
…or it needs Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
zlareb1
left a comment
There was a problem hiding this comment.
No blocking findings. Checked knowledge citations, namespace scope, similarity-floor abstention and cleanup against the SDK and server contracts. All five document-search and existing agent-memory example tests pass; the embedder and floor requirements are documented.
|
rebase |
|
GitHub reports |
…-search Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com> # Conflicts: # docs/reference/python-sdk.md # sdk/python/examples/README.md
zlareb1
left a comment
There was a problem hiding this comment.
No blocking findings. Checked the rebased example against the SDK and search similarity-floor contract, including namespace scope, citations and cleanup. All five document-search and agent-memory example tests and 270 documentation checks pass; both existing examples remain in the merged documentation.
Closes #720.
sdk/python/examples/document_search.py, on theagent_memory.pylayout: add two Markdown documents, search them and print the hits as citations, and get nothing back for a question they don't answer.knowledge.addtakes two short Markdown documents, each with its ownsource_uriandtitle.knowledge.searchruns once per subject. Each hit prints as a citation:source_uri#chunk_idx, the section, the passage's first line and its similarity. The top hit has to be the document the question is about.bot.client.search.query, with a comment saying why:knowledge.searchalone still returns its nearest chunks, andsearch.querydrops knowledge hits belowLOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITY. No passage may come back.knowledge.searchandsearch.querypassesnamespace_scope: "this".finallyrunsbot.knowledge.delete(source_uri)on both documents.It exits 0 only when every check passes, and otherwise prints each failure and exits 1.
The default floor and the
localembedderThe default
LOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITYis 0.30, and that is too low for thelocalembedder:localbge-small)With the default,
search.queryreturned all four passages and the script exited 1. With the floor at 0.5 on the server, it abstained and every check passed (below). The script and the README say the floor has to sit above what the embedder scores unrelated text.@zlareb1, is 0.30 meant for the provider embedders? If the
localembedder is the released stack's default, a default floor per embedding model might be worth an issue. I'm glad to open it.Real run
Local Loom from source (ClickHouse 26.10 with the repo's
deploy/clickhouse-config, auth on,LOOM_EMBEDDING_MODEL=local(bge-small, fetched withpython -m server.embedding.local_models fetch bge-small),LOOM_MCP_LOOM_SEARCH_KNOWLEDGE_MIN_SIMILARITY=0.5, a minted writer token):Docs
sdk/python/examples/README.md: adocument_search.pysection with the command, the steps, the embedder and floor note, this run, and a link to Lab 2 of the teaching guide as the exercise version.docs/use-cases/document-search.md: a link to the script from "Start".docs/reference/python-sdk.md: the script listed under "Examples".Tests
tests/unit/test_sdk_example_document_search.pyloads the script by path and runs it against anhttpx.MockTransportserver:namespace_scope: "this", and the citation line prints.knowledge.searchanswers 422. TheValidationErrorpropagates and both documents are still deleted.Checks (Linux container, Python 3.12)
pytest tests/unit/test_doc_links.py tests/unit/test_public_copy.py tests/unit/test_sdk_example_*.py: 298 passed. That includes the new test (3) andtest_sdk_example_agent_memory.py.python -m scripts.codegen.docs_reference --check: current.ruff checkandruff format --checkon the example and its test: clean.