feat: stabilize field IDs across schema evolution

[Xuanwo]

Field IDs currently act as physical bindings throughout Lance, but their allocator is derived from fields still referenced by the current snapshot. Once a dropped identity disappears from both the schema and data files, that integer can be reused for a different logical field. This makes a bare field ID ambiguous across schema evolution and prevents durable metadata from safely following a field through rename, drop, replacement, and restore.

This PR defines stable field identity as a one-way, branch-ancestry-scoped format contract. Activated manifests persist a monotonically increasing field-ID high-water mark; rename and metadata-only changes preserve identity, while new fields, type replacements, and overwrite allocate fresh identities. New datasets activate the contract in their initial manifest with fail-closed reader and writer requirements. Existing legacy datasets retain their current behavior until explicitly migrated, including a controlled writer-only rollout mode for deployments that have already retired pre-gate writers.

The same contract is enforced at transaction, clone, restore, Arrow conversion, Java/Python binding, and Blob representation boundaries so external schema metadata and file-local synthetic fields cannot manufacture dataset identities or silently downgrade required writer capabilities.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[westonpace]

Do you think this will cause transactions to conflict with each other that did not before? I don't think it does today since we treat any schema change as conflicting with any other schema change.

Do we know why we reused field ids in the first place?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The final schema-only Project gap is fixed: Java now preserves explicit existing identities, retains the existing same-name/type binding for metadata-free drop and reorder operations, and rejects unmatched fields before commit. This keeps released legacy Project behavior while preventing stable identities without backing data.