Skip to content

feat: add cell flags - #8655

Draft
Xuanwo wants to merge 45 commits into
mainfrom
xuanwo/cell-flags
Draft

feat: add cell flags#8655
Xuanwo wants to merge 45 commits into
mainfrom
xuanwo/cell-flags

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Why

Lance needs snapshot-versioned, field-scoped Boolean cell state that is independent of Arrow validity and ordinary column values. This PR introduces Cell Flags as a native dataset capability with stable identities, sparse per-fragment state, query expressions, explicit mutation APIs, rewrite preservation, transaction binding, and Rust/Python/Java bindings.

Cell Flags are false by absence, change only through explicit Cell Flag mutations, and commit atomically with the dataset operations that carry them.

Performance status

The complete 10M-row Local FS and S3 matrix at 1f6af401 passed all integrity checks and showed substantial end-to-end, storage, request, and write-amplification improvements over an equivalent application-managed Boolean-column-plus-bitmap design. Under the accepted evaluation criteria, RSS is informational, shared paths must avoid material regression rather than win every microcase, and full-manifest metadata growth is not a strict blocker.

Follow-up profiling found that cold historical checkout eagerly materialized compact Cell Flag transaction row selections. The current exact head (488aecce) caches those inline protobuf bytes and materializes them only when read_transaction() is requested. On the same 10M-row S3 dataset, seven exact-head samples produced:

  • cold time travel: 37.44 ms median, down from 361.96 ms before the fix; Boolean baseline was 101.92 ms;
  • warm time travel: 11.17 ms median, down from 14.76 ms; Boolean baseline was 16.17 ms;
  • unchanged I/O: 2 reads / 46,184 bytes cold and 1 read / 23,092 bytes warm;
  • 7/7 integrity matches for 10,000,000 rows and 1,000,000 selected rows.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings
  • Cell Flag Rust tests: 80/80
  • scalar expression tests: 45/45
  • count pushdown tests: 10/10
  • Python lint and binding checks passed before the final Rust-only performance commits

The PR remains a draft for review; the implementation and benchmark evidence are available for citation.

Xuanwo added 30 commits August 15, 2026 07:43
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-deps Dependency updates A-format On-disk format: protos and format spec docs enhancement New feature or request labels Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant