Repository navigation
Description
Description
QdrantCollection builds its null and presence checks on two Qdrant server behaviors that changed in Qdrant 1.19:
- 1.19.0 (fix: treat missing field as count 0 for values_count filter qdrant/qdrant#9607, fix for .NET: Rename and restructure MCP skills cache #9586):
values_countnow treats a missing payload key as 0 values, soFieldCondition(values_count=ValuesCount(gte=0))also matches points that lack the key. - 1.19.1 (fix(segment): make IsNullCondition consistent for arrays qdrant/qdrant#10101):
is_nullnow also matches arrays that containnull([null],["a", null]).
The connector uses values_count >= 0 as "field is present" (_prepare_presence_condition) and bare is_null as "field is null" (_prepare_null_condition). On a 1.19 server this causes:
- Wrong records, no error (data written by the connector itself). For a list field whose value contains
None:Filter("tags", "is_null")also returns["a", None].Filter("tags", "is_not_null"),Filter("tags", "contains_all", ["a"])andNOT(is_null)drop["a", None].
- Missing keys are no longer excluded.
exists,is_not_null,neandnot_inmatch points that lack the key (points written before a field was added to the model, or by another ingestion job). The core contract says a missing field matches nothing exceptexists. Throughget()/search()the extra point then fails deserialization, so a filter that works on 1.18 raisesIntegrationInvalidResponseExceptionon 1.19.
CI does not catch this because the integration workflows pin qdrant/qdrant:v1.16.2. Running the existing tests/qdrant/test_server_filters.py against other server versions gives:
| Server | test_server_native_portable_semantics |
|---|---|
| 1.15.5, 1.16.2, 1.17.1, 1.18.3 | 39 passed |
| 1.19.0 | 9 of 39 failed |
| 1.19.1 | 10 of 39 failed |
Code Sample
import asyncio, uuid
from agent_framework import Filter, VectorStoreCollectionDefinition, VectorStoreField
from agent_framework_qdrant import QdrantCollection
from qdrant_client import AsyncQdrantClient, models
definition = VectorStoreCollectionDefinition([
VectorStoreField("key", name="id", type_="int"),
VectorStoreField("data", name="status", type_="str"),
VectorStoreField("data", name="tags", type_="list"),
VectorStoreField("vector", name="embedding", dimensions=2),
])
async def main() -> None:
client = AsyncQdrantClient(url="http://localhost:6333") # qdrant/qdrant:v1.19.1
c = QdrantCollection(dict, definition=definition, collection_name=f"repro_{uuid.uuid4().hex[:8]}", async_client=client)
await c.ensure_collection_exists()
await c.upsert([
{"id": 1, "status": "active", "tags": ["a", None], "embedding": [1.0, 0.0]},
{"id": 2, "status": "deleted", "tags": ["a"], "embedding": [0.0, 1.0]},
{"id": 3, "status": None, "tags": None, "embedding": [1.0, 1.0]},
], generate_vectors=False)
for f in [Filter("tags", "is_null"), Filter("tags", "is_not_null"), Filter("tags", "contains_all", ["a"])]:
print(f.operator, sorted(r["id"] for r in await c.get(filter=f)))
# point written before "status" existed in the model
await client.upsert(c.collection_name, points=[models.PointStruct(id=4, vector={}, payload={"tags": ["x"]})], wait=True)
print("ne", sorted(r["id"] for r in await c.get(filter=Filter("status", "ne", "deleted"))))
await c.ensure_collection_deleted()
asyncio.run(main())Expected (and actual on Qdrant 1.18.3):
is_null [3]
is_not_null [1, 2]
contains_all [1, 2]
ne [1, 3]
Actual on Qdrant 1.19.1:
is_null [1, 3]
is_not_null [2]
contains_all [2]
agent_framework.exceptions.IntegrationInvalidResponseException: Vector store response is missing required field 'status'.
Root cause
python/packages/qdrant/agent_framework_qdrant/_vector_store.py:
- L224-226
_prepare_presence_condition:values_count(gte=0)is used as "key present". The comment there ("values_count distinguishes missing from null and []") no longer holds from 1.19.0. - L229-230
_prepare_null_condition: bareis_null, which from 1.19.1 also matches arrays containing null. - L233-234
_prepare_non_null_conditionbuilds on both. It is used byis_not_null,in,not_inandcontains_all(L617, L641-L642, L649).neuses the presence condition directly (L621).
Proposed fix
Build both checks from conditions whose meaning is the same on 1.15 through 1.19.1:
- null:
is_null AND is_empty. This matches only an explicitnull, becauseis_emptyexcludes non-empty arrays. - present, for declared scalar fields (
str/int/float/bool):NOT is_empty OR null. A scalar field cannot hold[], sovalues_countis not needed. Non-null scalar:NOT is_empty. - Collection fields keep
values_countfor presence. From 1.19.0 Qdrant has no condition that tells[]apart from a missing key (checkedvalues_count,is_empty,is_null,exceptandf[]paths). So on 1.19+,exists,is_not_nulland emptycontains_allon a collection field still match points without the key. This should be documented.
I checked these conditions on 1.15.5, 1.16.2, 1.17.1, 1.18.3, 1.19.0 and 1.19.1, with and without payload indexes. A PR with the fix and regression tests is ready.
Code Sample
Error Messages / Stack Traces
Package Versions
agent-framework-qdrant: 1.0.0a260918, agent-framework-core: 1.20.0 (main @ e9857bd), qdrant-client: 1.19.1, Qdrant server: 1.19.0 and 1.19.1
Python Version
Python 3.12
Additional Context
No response
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Oct 2, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
The defect is in
python/packages/qdrant/agent_framework_qdrant/_vector_store.py::_prepare_presence_conditionand::_prepare_null_condition: on Qdrant 1.19.1, missing keys satisfyvalues_count >= 0, while bareis_nullmatches arrays containing null. Minimal repro: startqdrant/qdrant:v1.19.1, setQDRANT_TEST_URL, and runtest_server_native_portable_semantics; 10 of 39 cases fail versus 39 passing on v1.18.3.python/packages/qdrant/tests/qdrant/test_server_filters.py::test_server_native_portable_semantics1.0.0a2609181.0a260918eavanvalkenburg commentedon Nov 3, 2026
/plan Scope the Qdrant portable-filter compatibility repair through existing #8982 and its latest tests/review, not competing implementation. Main presence uses values_count(gte=0) and scalar-null uses bare is_null; reported 1.19 server changes break the maintained missing/null/list semantics, including connector-written lists containing None and missing-field deserialization. Verify the current version matrix before accepting compatibility claims. Separate stable null/non-array repairs from unresolved collection presence: do not silently weaken the core contract or treat documentation of false matches as a complete fix. Compare faithful server-side composition, a clearly rejected unsupported combination/version boundary, and any justified ingestion/migration approach; account for already-existing and externally ingested payloads and get/search filtering/limits rather than assuming a new marker solves old records. The author reports dict-field handling fixed in 7489ed6 with server regression in f02d652; verify current head instead of repeating the old dict omission as unfixed. Cover scalar and dict values, explicit null, missing keys, empty/nonempty lists and lists containing null, nested NOT/empty contains_all, indexed/unindexed payloads and get/search deserialization. Use the existing server tests across an older baseline and 1.19.0/1.19.1/current patch, and make any remaining unsupported semantics explicit before readiness or owner acceptance. This triage checked source/specification and current review, not a new live-server matrix. No ready label, new assignment or contributor PR closure is implied.
eavanvalkenburg commentedon Nov 12, 2026
/plan Scope the Qdrant portable-filter compatibility repair through existing #8982 and its latest tests/review, not competing implementation. Main presence uses values_count(gte=0) and scalar-null uses bare is_null; reported 1.19 server changes break the maintained missing/null/list semantics, including connector-written lists containing None and missing-field deserialization. Verify the current version matrix before accepting compatibility claims. Separate stable null/non-array repairs from unresolved collection presence: do not silently weaken the core contract or treat documentation of false matches as a complete fix. Compare faithful server-side composition, a clearly rejected unsupported combination/version boundary, and any justified ingestion/migration approach; account for already-existing and externally ingested payloads and get/search filtering/limits rather than assuming a new marker solves old records. The author reports dict-field handling fixed in 7489ed6 with server regression in f02d652; verify current head instead of repeating the old dict omission as unfixed. Cover scalar and dict values, explicit null, missing keys, empty/nonempty lists and lists containing null, nested NOT/empty contains_all, indexed/unindexed payloads and get/search deserialization. Use the existing server tests across an older baseline and 1.19.0/1.19.1/current patch, and make any remaining unsupported semantics explicit before readiness or owner acceptance. This triage checked source/specification and current review, not a new live-server matrix. No ready label, new assignment or contributor PR closure is implied.
github-actions commentedon Nov 12, 2026
Summary
The Qdrant connector builds "field is present" as
values_count >= 0and "field is null" as a bareis_null. The issue reports that Qdrant 1.19.0 and 1.19.1 changed what these match: a missing key now counts as 0 values, andis_nullalso matches arrays containingnull. That would break the connector's documented rule that a missing field matches nothing exceptexists. This comment is based on the source on main and the existing tests. I did not review the open PR #8982's code or reviews, and I did not run any server tests. The version table in the issue is the reporter's claim and is unverified here.What the source confirms
_prepare_presence_conditionand_prepare_null_conditionstill usevalues_count(gte=0)and bareis_null. The code comment claimingvalues_counttells missing fromnull/[]is the assumption the issue says 1.19 broke.exists,is_null,is_not_null,ne,in/not_inandcontains_allall depend on these two helpers. NestedNOTis plainmust_not, so it inherits any wrong leaf result.get()/search()returns whatever the server matches. A wrongly matched point with nostatuskey then fails deserialization withIntegrationInvalidResponseException. This explains the reported exception.qdrant/qdrant:v1.16.2inpython-integration-tests.ymlandpython-merge-tests.yml. Nothing in the repo runs the server tests against 1.17 or later, so a regression of this kind would not be caught.test_server_filters.pyalready covers a good part of the matrix: an empty payload, a missing key,null,[],[None]and nestedNOT. It does not cover["a", None], dict fields, indexed versus unindexed payloads, orget/searchwith a missing declared field. I could not check whether Python: Fix Qdrant null/presence filters on Qdrant server 1.19 #8982 adds these, so I list them as gaps to verify.Recommendation
Code changes are warranted. This is a real, high-priority correctness gap. Silently wrong filter results are worse than errors. It should be split into two parts with different outcomes:
is_null AND is_empty, and a non-null scalar becomesNOT is_empty. Present-scalar becomesNOT is_empty OR null. A declared scalar field cannot be[], so it does not needvalues_count. This is the reporter's design, and it is plausible from the Qdrant semantics described in the issue. It still needs to be proven by the server matrix.[]from a missing key. If that holds, presence for list fields cannot be exact in a single server-side filter. Documenting "may match missing points" and stopping would not be a complete fix. It would weaken the core contract only for list fields on 1.19 and later, andget/searchwould then raise on those points.Options for collection presence
NotImplementedErrorforexists,is_not_null,neand emptycontains_allon list fields when the server is 1.19.0 or later.contains_allwith a non-empty value,containsandeqnever needed presence, so only the vacuous forms are affected.existsoris_not_nullon lists.__presentflag or a normalised empty-list sentinel.I lean toward fixing the stable parts and choosing A (reject on 1.19+) or a clearly documented limitation for the collection-presence parts. I would not choose C unless a maintainer decides the migration cost is justified. Because the effect on stored data is unclear, that is an owner decision.
Proposed plan (not implemented)
Re-run
test_server_native_portable_semanticsagainst an older baseline (such as 1.16.2 or 1.18.3), 1.19.0, 1.19.1 and the latest patch release. Record exact results before accepting the compatibility claims.Replace the null, non-null and scalar-presence helpers with the composed conditions above. Keep collection presence on
values_countor on the chosen rejection path.Extend the server tests:
nulland missing key;[],["a"],[None]and["a", None];NOT is_nullon a list;contains_all;get/searchwith a missing declared field, to confirm there is no deserialization error;limitbehavior, so a wrongly matched point cannot push out correct results.Add a server-version matrix to CI (at least baseline, 1.19.x and latest), or a documented scheduled job.
Document any remaining unsupported semantics, including what happens to existing and externally ingested points, in the package docs.
Success means every case in the matrix passes on all tested server versions, or the unsupported combination fails loudly instead of returning wrong data.
Questions for the owner
existsstay supported there?No ready label, assignment or PR decision is implied by this comment.