perf: run-optimize roaring bitmaps before persisting

[zhangyue19921010]

Bitmaps holding highly contiguous id sets were serialized without run containers, storing O(elements) bytes where the information content is O(runs):

• Index fragment bitmaps (fragment ids are allocated monotonically) — inlined in the manifest and rewritten on every commit.
• Overlay coverage bitmaps (physical row offsets; dense overlays cover contiguous ranges) — also inlined in the manifest.
• Compaction-captured row-addr treemaps persisted for deferred index remap — compaction reads whole fragments, so addresses are dense per-fragment ranges.

Change

Call optimize() before serializing at each of the three write points:

Write point	File
From<&IndexMetadata> for pb::IndexMetadata	rust/lance-table/src/format/index.rs
serialize_roaring (covers both the protobuf and serde paths)	rust/lance-table/src/format/overlay.rs
Row-addr capture in rewrite_files	rust/lance/src/dataset/optimize.rs

Compatibility

Run containers are part of the standard roaring serialization format, so existing readers decode the new bytes unchanged. No proto or manifest envelope changes.

Impact

A dense 1M-id bitmap shrinks from ~128 KiB to ~230 bytes (~570x); incompressible bitmaps are written as before (optimize() only converts a container when the run form is smaller).

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change applies run optimization at the three serialization boundaries that own these persisted bitmaps, while preserving caller-visible values and the standard Roaring format. Focused round-trip and size tests cover index metadata, overlay coverage, and compaction capture.