v5.6.0 — Atlas embeddings endpoint, streamed media, and retrieval correctness fixes

[AdrianCurtin]

Adds MongoDB's Atlas Embedding and Reranking API as a Voyage endpoint, video embedding, and memory-safe streaming for local media — and corrects three retrieval defects found while validating that work against the live API.

The correctness fixes are the part worth reviewing closely. All three were invisible to a fully green mocked suite, which is also why this PR adds a live contract-test harness.

Embeddings

• Atlas endpoint. Voyage models are served by both Voyage's own API and MongoDB's Atlas endpoint. The wire contract is identical but the credentials are not interchangeable, so an Atlas model API key is detected by prefix and routed to ai.mongodb.com. endpoint: names it explicitly, and a named endpoint that contradicts an explicit base_url is refused rather than silently reconciled.
• New models: voyage-3.5, voyage-3.5-lite, voyage-code-2, voyage-multimodal-3.5. Models Atlas does not serve are refused at construction with a replacement named, and stay valid against Voyage's own API.
• Default model moves to voyage-3.5 (breaking — see below). voyage-3 is retired from Atlas, so the old default made an Atlas key fail at construction whenever no model was named.
• Self-hosted-only models are tracked separately. voyage-4-nano turned out to be served by neither hosted endpoint — it is open-weight and meant to be self-hosted. Lumping it in with the Atlas-retired models told an Atlas user to "target Voyage's own API" for a model Voyage does not serve either. It now lives in SELF_HOSTED_ONLY_MODELS, is refused against both hosted hosts with a pointer to a self-hosted base_url: or LocalHTTP, and remains usable through either.
• Video. embed_video on voyage-multimodal-3.5, plus embed_video and supports_modality? on the base provider protocol so video is a declared capability rather than one adapter's method.
• Streaming. Serializing media with to_json costs roughly 2.4x the file size resident — raw bytes, the base64 copy, and the serialized document — enough to exhaust a small dyno on one moderate video. MediaFile reads only a 16-byte header; StreamingBody base64-encodes into the socket in fixed chunks, with an exact Content-Length and a retry-safe rewind. Passing a URL keeps the SDK out of the transfer entirely.
• Dimension table corrected. The whole v4 family defaults to 1024; the wider and narrower widths are Matryoshka options, not native defaults. Since the provider validates returned width against the declared one, voyage-4-large and voyage-4-lite raised on every call. Per-model supported widths replace the coarse Matryoshka gate.
• Media validation. Video restricted to MP4, the only container Voyage accepts. ISO-BMFF brands are matched explicitly so an ftyp box no longer implies MP4 on its own, and Apple's audio-only M4A brand is excluded — it previously let an audio file pass as video. Media is capped at the documented 20 MB, enforced by the adapter independently of the global limit.

Vector search

• Results no longer fall short of k under ACL. $vectorSearch applied its limit before the ACL $match, protectedFields redaction, pointer-field filtering, and any caller filter. A scoped caller who could read 2 of the top 10 asked for 10 and received 2, while readable matches sat further down the ranking. The search now requests a wider candidate window, enforces, then trims to k. The window is raised only when something can actually drop rows, and HNSW width stays anchored to k so the wider window does not widen the ANN search.
• Hybrid search requested far more than intended. The branch limit was passed as the vector branch's k and then multiplied again by the plain search's own window derivation: k: 10 asked Atlas for 1,000 rows instead of 100, an explicit candidate_limit diverged between the two fusion paths, and any k above 100 produced a branch k beyond the plain search's ceiling and failed outright. Fusion depth and candidate window are now separate values, bounded by MAX_K and Atlas's 10,000 ceiling respectively.
• Native $rankFusion trimmed to k in a $limit that runs after its ACL $match, reintroducing the same underfill. It now limits to the candidate window and trims once enforcement has run.
• Telemetry. parse.vector_search.search and parse.vector_search.hybrid carry counts named for where they are measured, so an underfill is observable rather than silent. There is no cheap way to learn how many rows $vectorSearch emitted before the server-side $match, so no field claims to be that number.

Model declarations

• model: is enforced. It was recorded on :vector properties and never checked. Because models in a family usually share a width (voyage-3 and voyage-3.5 are both 1024), swapping the registered provider's model silently mixed incomparable vectors into one index — no error, just degraded recall, repairable only by re-embedding. model: and dimensions: are now verified before any request, fail closed when a provider cannot report them, and are exposed as an explicit audit for boot or CI.
• Declaration-time validation. similarity: is checked against the functions Atlas accepts. A property wider than an Atlas index allows must declare searchable: false, which now makes it genuinely storage-only: excluded from field resolution, refused when named directly, and rejected by agent_searchable at class load.

Breaking

The Voyage default model moves from voyage-3 to voyage-3.5. Vectors from the two are not comparable. Code relying on the default must pin model: "voyage-3" explicitly to keep existing embeddings valid, or re-embed. A :vector property that declares model: is unaffected — the new binding audit catches the mismatch before any request rather than letting the two mix silently.

Other behavior changes to note

• A :vector property above the Atlas index cap without searchable: false now fails at declaration, and a storage-only field fails at resolution rather than at Atlas. Intended, but not purely additive.
• Scoped hybrid calls fetch a wider per-branch window than before. Correct given the underfill, but a real cost increase — it is a measured line item in the v5.7 retrieval baseline rather than something this PR claims to have quantified.
• A provider that cannot report #model_name or #dimensions against a property that declares them is now refused rather than skipped.
• voyage-4-nano can no longer be constructed against a hosted endpoint. It never worked there; the failure has moved from an opaque provider 400 to a constructor error naming the fix.

Testing

rake test:unit passes. rake test:contract passes against both an Atlas model API key and a Voyage key.

rake test:contract is new: live provider contract tests pinning request routing, native dimensions, the Matryoshka ladder, accepted media, size limits, model availability, and response shape. It is excluded from both rake test and rake test:unit, so no ordinary run becomes billable because a key happens to be exported.

Refusal probes issue raw requests instead of going through the SDK — a local guard asserted against itself proves nothing about the contract it encodes. Infrastructure conditions are classified rather than guessed at: a bad credential fails (the operator must fix it), while rate limiting and 5xx skip as "contract not determined", since failing there would cry wolf on a low-quota key and passing would hide real drift.

The suite earned its keep immediately. The live runs are what established that the v4 family defaults to 1024, that WebM and QuickTime are rejected, that voyage-3/voyage-3-lite match their declared widths, and that voyage-4-nano is absent from both hosted endpoints. Every one of those was wrong or unverified in the SDK while the mocked suite was fully green.

For low-quota keys, VOYAGE_CONTRACT_RPM paces the suite per model and disables the provider's own retries, which otherwise spend the rate budget invisibly during backoff.

Still outstanding: on a 3-requests-per-minute key, the Matryoshka-ladder, video, and WebM-rejection probes exhaust the budget and skip. All three facts are separately confirmed against Atlas; only the Voyage-side assertion needs more quota. Wiring the contract suite into a scheduled job still needs a CI secret and a schedule, and should be two jobs — Atlas credentials in one, Voyage in the other — kept out of PR CI.