Skip to content

perf: lazy + projected payload reads (#116)#118

Merged
dcfocus merged 1 commit into
lance-format:mainfrom
dcfocus:feat/issue-116-lazy-reads
Jun 28, 2026
Merged

perf: lazy + projected payload reads (#116)#118
dcfocus merged 1 commit into
lance-format:mainfrom
dcfocus:feat/issue-116-lazy-reads

Conversation

@dcfocus

@dcfocus dcfocus commented Jun 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #116. Reads previously fully materialized binary_payload (and text_payload / embedding) for every row on every list / search / get — so a metadata listing or a vector search over multi-modal records pulled every image's bytes into memory. This adds column projection on the read path plus on-demand blob fetch.

Independent of #115 — it operates on the existing inline payload columns, so it can land on its own.

What changed

Core (lance-context-core)

  • ReadProjection { text, binary, embedding } — default loads everything (backward compatible), with metadata_only() / without_binary() helpers.
  • batch_to_records now tolerates those columns being projected out (omitted fields decode as None).
  • ContextStore::list_filtered_projected and search_filtered_projected read only the requested columns; the existing *_with_options methods delegate with the full projection (no behavior change). Search always reads the embedding internally to score, then drops it from results if projection.embedding is false.
  • ContextStore::get_blob(id) -> Option<Vec<u8>> fetches one record's bytes via a projected, id-filtered scan.

Python

  • Context.list(..., include_binary=True, include_embedding=True) and Context.search(..., include_binary=..., include_embedding=...).
  • Context.get_blob(id) -> bytes | None.
  • New pyo3 args default to true, so existing callers are unaffected.
lean = ctx.list(filters={"content_type": "image/png"}, include_binary=False)  # no bytes
img  = ctx.get_blob(lean[0]["id"])                                            # fetch on demand
hits = ctx.search(query_embedding, include_binary=False)                      # ranked, no bytes

Tests

  • core: projection drops binary/embedding but keeps metadata; get_blob fetches on demand (and returns None for missing); search projection still ranks correctly.
  • python: test_lazy_payload.py — list/search exclude binary, get_blob round-trip.

Checks

  • cargo test -p lance-context-core --lib — 98 passed
  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings — clean
  • ruff format --check, ruff check, pyright — clean

Acceptance criteria (#116)

  • list / search can avoid materializing binary_payload (and embedding)
  • Projection / include_binary flag controls inclusion (default preserves current behavior)
  • get_blob(id) fetches a single record's bytes on demand
  • Tests cover projection + on-demand fetch
  • REST include_binary params + blob GET route — deferred follow-up
  • Lance take_blob/BlobFile zero-copy lazy reads for blob-encoded columns — deferred follow-up

Notes

  • I defaulted projection to load-all (non-breaking) rather than flipping the default to exclude binary, since silently dropping bytes from existing list/get results would be a breaking change. Flipping the default can be revisited as a separate, deliberate change.

Closes #116

Reads previously hard-loaded `binary_payload` (and `text_payload` /
`embedding`) for every row on every `list` / `search` / `get`
(`batch_to_records` required the columns; the scanners didn't project). For
multi-modal records that means a metadata `list` or a vector `search` pulled
every image's bytes into memory.

This adds column projection on the read path plus an on-demand blob fetch:

- `ReadProjection { text, binary, embedding }` (default = load all, backward
  compatible) with `metadata_only()` / `without_binary()` helpers.
- `batch_to_records` now tolerates those columns being projected out
  (omitted fields decode as `None`).
- `ContextStore::list_filtered_projected` and `search_filtered_projected`
  read only the requested columns; the existing `*_with_options` methods
  delegate with the default (full) projection, so behavior is unchanged.
  Search always reads the embedding internally to score, then drops it from
  results if `projection.embedding` is false.
- `ContextStore::get_blob(id)` fetches a single record's `binary_payload` on
  demand via a projected, id-filtered scan.

Python:
- `Context.list(..., include_binary=True, include_embedding=True)` and
  `Context.search(..., include_binary=..., include_embedding=...)` — set
  `include_binary=False` to query metadata/search without materializing media
  bytes (omitted payloads come back as `None`).
- `Context.get_blob(id) -> bytes | None` for on-demand fetch.
- The new pyo3 args default to `true`, so existing callers are unaffected.

Tests: core (projection drops binary/embedding but keeps metadata; `get_blob`
fetches on demand; search projection keeps ranking) and Python
(`test_lazy_payload.py`). README gains a multi-modal read example.

Independent of lance-format#115 (operates on the existing inline payload columns).

Follow-up (noted, deferred): REST `include_binary` params + a blob GET route,
and Lance `take_blob`/`BlobFile` zero-copy lazy reads for blob-encoded
columns.

Closes lance-format#116
@dcfocus dcfocus force-pushed the feat/issue-116-lazy-reads branch from d10aa4b to e40e8d1 Compare June 28, 2026 01:58
@dcfocus dcfocus merged commit dbc579c into lance-format:main Jun 28, 2026
9 checks passed
dcfocus added a commit that referenced this pull request Jun 28, 2026
## Summary

Closes #117. Adds multi-modal embeddings + **cross-modal retrieval**
(text query → image results). lance-context bundles no models — the
encoder is user-supplied.

> **Stacked on #118 (#116).** Review/merge #116 first; until then this
PR shows both commits (the multi-modal commit is last).

## Motivation

Embedding providers were **text-only** (`embed_texts`), so non-text
payloads got no embedding unless the caller computed one, and an image
could never be retrieved by a text query (`embeddings.py`; `api.py`
auto-embed gated on `isinstance(str)`).

## What changed (Python embedding layer only — no core/Rust change)

- **`MultiModalEmbeddingProvider`** protocol — extends
`EmbeddingProvider` with `embed_media(items: list[tuple[bytes, str]]) ->
list[list[float]]`, embedding `(payload_bytes, content_type)` into the
**same** space as `embed_texts`. Plus a `supports_media(provider)`
helper. Both exported from the package.
- **Auto-embed media** — `add` / `upsert` / `add_many` / `upsert_many`
embed `bytes` payloads via `embed_media` when the provider supports it
(text still uses `embed_texts`).
- **Cross-modal retrieval for free** — because the two encoders share
one space, `ctx.search("a photo of a cat")` embeds the text with
`embed_texts` and matches image embeddings through the existing vector
search. `search` also accepts a `bytes` query (embedded via
`embed_media`).

```python
clip_ctx = Context.create("multimodal.lance", embedding_dim=512,
                          embedding_provider=my_clip_provider)
clip_ctx.add("user", image_bytes, content_type="image/png", external_id="img-1")
results = clip_ctx.search("a photo of a cat")   # text -> image
```

## Tests (`test_multimodal_embeddings.py`, deterministic CLIP-style
stub)
- protocol / `supports_media` runtime check
- image auto-embedded via `embed_media`
- **cross-modal** text→image retrieval
- batch auto-embed of images
- text-only provider does **not** embed images

## Checks
- `ruff format --check`, `ruff check`, `pyright` — clean
- new tests + `test_lazy_payload.py` (from #116) pass

## Acceptance criteria (#117)
- [x] Pluggable multi-modal embedder (`embed_media`), no bundled models
- [x] Auto-embed image/bytes payloads
- [x] Cross-modal text→image retrieval via the shared space
- [x] Existing text retrieval unchanged; tests with a stub provider

## Note on pre-existing failures
The 8 failures in `test_embeddings.py` are **pre-existing and
unrelated** to this change: its hand-written `_DummyInner.add()` mock
accepts 10–18 positional args while the real binding now passes 22
(`TypeError` at `api.py` `inner.add`, predating this PR; not caught by
the path-filtered CI job). The new tests here use a real `Context`, not
that mock. Fixing the stale mock is a separate cleanup.

Closes #117
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-modal: lazy + projected payload reads — stop materializing binary/text/embedding on every read

1 participant