feat: add segmented BTree index planning and merge support

[zhangyue19921010]

Summary

This PR adds segmented index plan, merge, and optimize support for BTree scalar indexes.

What Changed

• Adds a BTree implementation for the segmented index builder.

  • Supports planning staged BTree segments.
  • Supports 1:1 passthrough when segments do not need consolidation.
  • Supports N:1 consolidation when target_segment_bytes is set.

• Adds BTree support to merge_existing_index_segments.

  • Merges multiple existing BTree segments into one physical segment.

• Adds a lower-level BTreeIndex::merge_segments API.

  • Merges multiple BTree indexes plus optional new training data.

• Extends scalar index optimize support for multi-segment BTree indexes.

  • Can merge multiple existing BTree segments with newly appended fragments.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 80.14706% with 189 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/scalar/btree.rs	57.82%	76 Missing and 21 partials ⚠️
rust/lance/src/index/append.rs	85.02%	47 Missing and 18 partials ⚠️
rust/lance-index/src/scalar/btree.rs	63.63%	18 Missing and 6 partials ⚠️
rust/lance/src/index/create.rs	98.61%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!