perf: lazy + projected payload reads (#116)#118
Merged
Conversation
4 tasks
Reads previously hard-loaded `binary_payload` (and `text_payload` /
`embedding`) for every row on every `list` / `search` / `get`
(`batch_to_records` required the columns; the scanners didn't project). For
multi-modal records that means a metadata `list` or a vector `search` pulled
every image's bytes into memory.
This adds column projection on the read path plus an on-demand blob fetch:
- `ReadProjection { text, binary, embedding }` (default = load all, backward
compatible) with `metadata_only()` / `without_binary()` helpers.
- `batch_to_records` now tolerates those columns being projected out
(omitted fields decode as `None`).
- `ContextStore::list_filtered_projected` and `search_filtered_projected`
read only the requested columns; the existing `*_with_options` methods
delegate with the default (full) projection, so behavior is unchanged.
Search always reads the embedding internally to score, then drops it from
results if `projection.embedding` is false.
- `ContextStore::get_blob(id)` fetches a single record's `binary_payload` on
demand via a projected, id-filtered scan.
Python:
- `Context.list(..., include_binary=True, include_embedding=True)` and
`Context.search(..., include_binary=..., include_embedding=...)` — set
`include_binary=False` to query metadata/search without materializing media
bytes (omitted payloads come back as `None`).
- `Context.get_blob(id) -> bytes | None` for on-demand fetch.
- The new pyo3 args default to `true`, so existing callers are unaffected.
Tests: core (projection drops binary/embedding but keeps metadata; `get_blob`
fetches on demand; search projection keeps ranking) and Python
(`test_lazy_payload.py`). README gains a multi-modal read example.
Independent of lance-format#115 (operates on the existing inline payload columns).
Follow-up (noted, deferred): REST `include_binary` params + a blob GET route,
and Lance `take_blob`/`BlobFile` zero-copy lazy reads for blob-encoded
columns.
Closes lance-format#116
d10aa4b to
e40e8d1
Compare
dcfocus
added a commit
that referenced
this pull request
Jun 28, 2026
## Summary Closes #117. Adds multi-modal embeddings + **cross-modal retrieval** (text query → image results). lance-context bundles no models — the encoder is user-supplied. > **Stacked on #118 (#116).** Review/merge #116 first; until then this PR shows both commits (the multi-modal commit is last). ## Motivation Embedding providers were **text-only** (`embed_texts`), so non-text payloads got no embedding unless the caller computed one, and an image could never be retrieved by a text query (`embeddings.py`; `api.py` auto-embed gated on `isinstance(str)`). ## What changed (Python embedding layer only — no core/Rust change) - **`MultiModalEmbeddingProvider`** protocol — extends `EmbeddingProvider` with `embed_media(items: list[tuple[bytes, str]]) -> list[list[float]]`, embedding `(payload_bytes, content_type)` into the **same** space as `embed_texts`. Plus a `supports_media(provider)` helper. Both exported from the package. - **Auto-embed media** — `add` / `upsert` / `add_many` / `upsert_many` embed `bytes` payloads via `embed_media` when the provider supports it (text still uses `embed_texts`). - **Cross-modal retrieval for free** — because the two encoders share one space, `ctx.search("a photo of a cat")` embeds the text with `embed_texts` and matches image embeddings through the existing vector search. `search` also accepts a `bytes` query (embedded via `embed_media`). ```python clip_ctx = Context.create("multimodal.lance", embedding_dim=512, embedding_provider=my_clip_provider) clip_ctx.add("user", image_bytes, content_type="image/png", external_id="img-1") results = clip_ctx.search("a photo of a cat") # text -> image ``` ## Tests (`test_multimodal_embeddings.py`, deterministic CLIP-style stub) - protocol / `supports_media` runtime check - image auto-embedded via `embed_media` - **cross-modal** text→image retrieval - batch auto-embed of images - text-only provider does **not** embed images ## Checks - `ruff format --check`, `ruff check`, `pyright` — clean - new tests + `test_lazy_payload.py` (from #116) pass ## Acceptance criteria (#117) - [x] Pluggable multi-modal embedder (`embed_media`), no bundled models - [x] Auto-embed image/bytes payloads - [x] Cross-modal text→image retrieval via the shared space - [x] Existing text retrieval unchanged; tests with a stub provider ## Note on pre-existing failures The 8 failures in `test_embeddings.py` are **pre-existing and unrelated** to this change: its hand-written `_DummyInner.add()` mock accepts 10–18 positional args while the real binding now passes 22 (`TypeError` at `api.py` `inner.add`, predating this PR; not caught by the path-filtered CI job). The new tests here use a real `Context`, not that mock. Fixing the stale mock is a separate cleanup. Closes #117
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #116. Reads previously fully materialized
binary_payload(andtext_payload/embedding) for every row on everylist/search/get— so a metadata listing or a vector search over multi-modal records pulled every image's bytes into memory. This adds column projection on the read path plus on-demand blob fetch.Independent of #115 — it operates on the existing inline payload columns, so it can land on its own.
What changed
Core (
lance-context-core)ReadProjection { text, binary, embedding }— default loads everything (backward compatible), withmetadata_only()/without_binary()helpers.batch_to_recordsnow tolerates those columns being projected out (omitted fields decode asNone).ContextStore::list_filtered_projectedandsearch_filtered_projectedread only the requested columns; the existing*_with_optionsmethods delegate with the full projection (no behavior change). Search always reads the embedding internally to score, then drops it from results ifprojection.embeddingisfalse.ContextStore::get_blob(id) -> Option<Vec<u8>>fetches one record's bytes via a projected, id-filtered scan.Python
Context.list(..., include_binary=True, include_embedding=True)andContext.search(..., include_binary=..., include_embedding=...).Context.get_blob(id) -> bytes | None.true, so existing callers are unaffected.Tests
get_blobfetches on demand (and returnsNonefor missing); search projection still ranks correctly.test_lazy_payload.py— list/search exclude binary,get_blobround-trip.Checks
cargo test -p lance-context-core --lib— 98 passedcargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings— cleanruff format --check,ruff check,pyright— cleanAcceptance criteria (#116)
list/searchcan avoid materializingbinary_payload(andembedding)include_binaryflag controls inclusion (default preserves current behavior)get_blob(id)fetches a single record's bytes on demandinclude_binaryparams + blob GET route — deferred follow-uptake_blob/BlobFilezero-copy lazy reads for blob-encoded columns — deferred follow-upNotes
list/getresults would be a breaking change. Flipping the default can be revisited as a separate, deliberate change.Closes #116