feat: add snapshot-level field assignment tracking

[Xuanwo]

Why

Lance needs a snapshot-level primitive that records whether each logical field has been successfully written for each row. Applications currently have to materialize and maintain a separate Boolean state column, which couples assignment semantics to user data and requires independent index maintenance.

This implements the contract accepted in Discussion #8520. Assignment is bound to stable field IDs; explicit NULL writes are assigned, omitted fields are unassigned, explicit invalidation clears assignment atomically, and physical rewrites preserve rather than create assignment. Ordinary projection continues to return stored Arrow values, while is_assigned(field) is a non-null Boolean expression across projection, filtering, aggregation, ordering, DataFusion, and the Lance Substrait extension.

The implementation keeps this as a Lance storage/query/mutation primitive. It does not add computed-column dependencies, provenance, jobs, or scheduling semantics.

Compatibility

Tracked snapshots require FieldAssignment reader and writer feature support, so released implementations that do not understand the format fail closed. On the final PR head, the current writer opened the measured S3 snapshot at version 7 before and after the compatibility probe, while Lance 10.0.0 rejected the same snapshot with unsupported feature flag 256. Queries that do not reference is_assigned bypass assignment binding and do not load assignment state.

Validation

The final head passes the 20-test FieldAssignment integration suite, the ordinary-scan binding fast-path regression, the exact all-feature/all-target Clippy command used by CI, and the Java VectorIndexTest suite with the default 1 MiB JVM native stack. The PR's Java 11/17/21/25, Python, Rust coverage, Windows, rustdoc, MSRV, format, and Clippy jobs have also passed.

Performance evidence

This is not a before/after comparison with an existing equivalent feature. It compares the new FieldAssignment primitive with the closest application-managed representation: a materialized assigned_state: bool column plus a Lance BITMAP scalar index. Boolean mutation timings include the required index catch-up.

Both variants were measured at exact PR head ee730236cf7896e19e972cd5497787628fbd213e on the same retained c7i.4xlarge EC2 instance, using 10,000,000 identical rows, 100 initial fragments, concurrency 8, seven query repeats, and three mutation/compaction repeats. Local FS and S3 used the same parameters; Local cold runs dropped the Linux page cache.

Representative medians (seconds):

Storage	Operation	FieldAssignment	Boolean + BITMAP
Local FS	sparse assign	0.1120	0.1227
Local FS	sparse invalidation	0.1809	0.1954
Local FS	compaction	0.6628	4.1942
Local FS	cold / warm filter	0.0825 / 0.00999	0.0786 / 0.00995
Local FS	cold / warm planning	0.00146 / 0.000877	0.00180 / 0.000503
S3	sparse assign	2.0776	2.8650
S3	sparse invalidation	1.4317	2.2483
S3	compaction	1.6654	5.3000
S3	cold / warm filter	0.9740 / 0.3302	1.0375 / 0.3293
S3	cold / warm planning	0.00124 / 0.000875	0.000973 / 0.000557

Each full storage run contains 108 measurements and 27 integrity records with no mismatches. Every FieldAssignment filter plan used exact row-selection pushdown, and both variants scanned 10,000 of 10,000,000 rows. All 14 S3 FieldAssignment planner samples recorded zero assignment I/O and zero host-network delta. S3 warm-filter p95 was 0.411 seconds for FieldAssignment versus 0.398 seconds for the Boolean baseline.

The trade-off is metadata and write amplification. FieldAssignment-managed objects occupy 24,596 bytes versus 1,714 bytes for the Boolean index, while total incremental setup storage is 3,349,314 bytes versus 1,309,678 bytes on S3. Sparse assignment write amplification is 9.44x versus 6.29x; invalidation is 118.66x versus 86.80x. In this workload, avoiding separate index maintenance still yields lower end-to-end mutation and compaction latency.

The reproducible harness, checksums, exact commands, machine inventory, and machine-readable Local FS/S3 results are retained outside this project repository in the task-specific private personal repository Xuanwo/lance-field-assignment-benchmark at signed commit 7d219949e6aa54270c7932e302096741a0e0fc1b.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Xuanwo]

This implementation is superseded by the generalized Cell Flags contract in #8655, with stable field identity handled by #8658. Tracking continues in #8547.