Description
CosmosCollection leaves out records whose field is null when an eq filter is negated. InMemoryCollection includes them. This affects both get(filter=...) and search(filter=...), which share the filter translation.
With records a (category="x"), b (category=None) and c (category="y"), FilterGroup("not", [Filter("category", "eq", "x")]) returns b and c from InMemoryCollection. Going by the Cosmos operator tables, Cosmos returns only c. I couldn't run it against a live account; the WHERE clause below is what the connector sends.
_translate_filter builds eq as (IS_DEFINED(x) AND x = @p) (_vector_store.py L1001-L1008), so the WHERE clause is:
(NOT (IS_DEFINED(c["category"]) AND c["category"] = @filter_0))
In Cosmos SQL, comparing null with a string is undefined, true AND undefined is undefined, NOT undefined is undefined, and an item whose filter is undefined isn't returned (operators). So row b drops out.
The other leaves already evaluate to true/false on null: ne has IS_NULL(x) OR ..., in/not_in and the ordered operators have NOT IS_NULL(x), contains* has IS_ARRAY, and the text operators have IS_STRING. Only eq is missing a guard. The SQL Server connector guards eq the same way ({column} IS NOT NULL AND {column} = ..., _sql.py L289).
Expected: the same rows as InMemoryCollection, b and c.
Proposed fix: build eq as (IS_DEFINED(x) AND NOT IS_NULL(x) AND x = @p). Top-level eq results don't change, because Filter rejects eq with None, so null rows never matched anyway. Under NOT the leaf now evaluates to false for null rows, so they're included. I have the fix with tests and will open a PR.
Code Sample
import asyncio
from unittest.mock import MagicMock
from agent_framework import Filter, FilterGroup, InMemoryCollection, VectorStoreCollectionDefinition, VectorStoreField
from agent_framework_azure_cosmos import CosmosCollection
definition = VectorStoreCollectionDefinition(
[
VectorStoreField("key", name="id", type_="str"),
VectorStoreField("data", name="category", type_="str", is_indexed=True),
VectorStoreField("vector", name="embedding", type_="float32", dimensions=3, distance_function="cosine_similarity"),
],
collection_name="items",
)
records = [
{"id": "a", "category": "x", "embedding": [1.0, 0.0, 0.0]},
{"id": "b", "category": None, "embedding": [0.0, 1.0, 0.0]},
{"id": "c", "category": "y", "embedding": [0.0, 0.0, 1.0]},
]
not_eq = FilterGroup("not", [Filter("category", "eq", "x")])
cosmos = CosmosCollection(record_type=dict, definition=definition, collection_name="items", container_client=MagicMock())
print(cosmos._prepare_filter(not_eq)[0])
async def main() -> None:
reference = InMemoryCollection(record_type=dict, definition=definition, collection_name="items")
await reference.ensure_collection_exists()
await reference.upsert(records, generate_vectors=False)
print(sorted(r["id"] for r in await reference.get(filter=not_eq)))
asyncio.run(main())Output:
(NOT (IS_DEFINED(c["category"]) AND c["category"] = @filter_0))
['b', 'c']
Package Versions
agent-framework-core: 1.19.0, agent-framework-azure-cosmos: 1.0.0b260918 (released package; also main @ 2c46deb)
Python Version
Python 3.13.9 (macOS, arm64)
Additional Context
With the integration test records one/two/three (optional = None/"value"/"other"), NOT(optional eq "value") should return {one, three}. By the operator tables above, main returns only three.
Same family as #8653 (DocumentDB membership with null), which was fixed in #8654.
Description
CosmosCollectionleaves out records whose field isnullwhen aneqfilter is negated.InMemoryCollectionincludes them. This affects bothget(filter=...)andsearch(filter=...), which share the filter translation.With records
a(category="x"),b(category=None) andc(category="y"),FilterGroup("not", [Filter("category", "eq", "x")])returnsbandcfromInMemoryCollection. Going by the Cosmos operator tables, Cosmos returns onlyc. I couldn't run it against a live account; the WHERE clause below is what the connector sends._translate_filterbuildseqas(IS_DEFINED(x) AND x = @p)(_vector_store.py L1001-L1008), so the WHERE clause is:In Cosmos SQL, comparing null with a string is
undefined,true AND undefinedisundefined,NOT undefinedisundefined, and an item whose filter isundefinedisn't returned (operators). So rowbdrops out.The other leaves already evaluate to true/false on null:
nehasIS_NULL(x) OR ...,in/not_inand the ordered operators haveNOT IS_NULL(x),contains*hasIS_ARRAY, and the text operators haveIS_STRING. Onlyeqis missing a guard. The SQL Server connector guardseqthe same way ({column} IS NOT NULL AND {column} = ..., _sql.py L289).Expected: the same rows as
InMemoryCollection,bandc.Proposed fix: build
eqas(IS_DEFINED(x) AND NOT IS_NULL(x) AND x = @p). Top-leveleqresults don't change, becauseFilterrejectseqwithNone, so null rows never matched anyway. UnderNOTthe leaf now evaluates tofalsefor null rows, so they're included. I have the fix with tests and will open a PR.Code Sample
Output:
Package Versions
agent-framework-core: 1.19.0, agent-framework-azure-cosmos: 1.0.0b260918 (released package; also main @ 2c46deb)
Python Version
Python 3.13.9 (macOS, arm64)
Additional Context
With the integration test records
one/two/three(optional=None/"value"/"other"),NOT(optional eq "value")should return{one, three}. By the operator tables above, main returns onlythree.Same family as #8653 (DocumentDB membership with null), which was fixed in #8654.