feat(transaction): let two writers extend one index at once - #8641
Closed
wjones127 wants to merge 41 commits into
Closed
feat(transaction): let two writers extend one index at once#8641wjones127 wants to merge 41 commits into
wjones127 wants to merge 41 commits into
Conversation
Introduces the Rust side of the action-based transaction draft: `Ref`, `UserOperation`, `UserAction`, and the eight `Action` variants this build implements (AddFragment, AddDataFile, AddField, AddBase, TombstoneFieldData, RemoveFragment, SetDeletionFile, AlterField). Types only -- no wire conversion, apply, or conflict handling yet, so nothing reaches these from the commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `Operation::UserOperation`, the Transaction V2 arm of the transaction oneof, and its protobuf conversions in both directions. Loading a V2 transaction now yields an action set instead of an outright rejection; an action the build does not implement is still rejected rather than skipped, so a concurrent V2 commit can never be silently treated as a no-op. The rest of the transaction machinery gains the variant but no behavior: build_manifest returns NotSupported, and conflict checks route through a single `check_action_txn` that conservatively demands a retry. Apply and conflict rules follow in later commits. Also extracts `From<&DeletionFile> for pb::DeletionFile`, previously inlined in the fragment conversion and now needed by SetDeletionFile too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pulls the two operation-independent stages out of `build_manifest`: `normalize_fragments` (order, drop fully-tombstoned files, check overlay order) and `assemble_manifest` (construct the manifest and apply the tag, feature flags, timestamp, and fragment id watermark). The overwrite-only storage format override becomes an explicit parameter rather than a match on the operation inside the assembly step. No behavior change; the action-based apply path needs the same two stages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the action apply path: `build_manifest_from_actions` walks the action set in order against a working copy of the read-version state, and the four minting actions (AddFragment, AddDataFile, AddField, AddBase) allocate ids from the target's counters as they are reached. Each mint records its id against the action's local token, so a later action in the same operation resolves that token to the id this apply chose. The same action set replayed against a different version therefore lands on different ids without any action changing -- the property branch merge needs. An action set requires an existing dataset: it is a delta, so there is nothing for it to be a delta against at creation time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applies TombstoneFieldData, RemoveFragment, SetDeletionFile, and AlterField, completing the eight-action apply path. Tombstoning a field's data and retyping a field both leave any index over that field describing values the fragment no longer holds, so the affected fragments are dropped from the index bitmap rather than the index being discarded outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ions Adds `TryFrom<&Operation> for Vec<UserAction>`, the recipe that turns a named operation into the granular actions it decomposes into. Squashing several operations into one commit is concatenation once both sides speak actions. Translation is fail-closed: an operation with no recipe yet, or one carrying a detail the actions cannot express, is rejected. Each case is covered by a parity test that builds the manifest twice -- once down the legacy path, once through the translation -- and asserts the two agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replacing a field's data becomes TombstoneFieldData followed by AddDataFile. The legacy path swaps the path on the existing file in place, so the two produce the same set of data files per fragment but not necessarily the same order; files are addressed by field, so the order carries no meaning. Also documents why Merge and Project are not translated: both hand over a whole new schema rather than a delta, and Project needs a field-removal action this draft does not define. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A footprint is the set of coordinates an action set writes. Two concurrent sets can both commit when neither writes what the other writes -- one structural rule instead of a matrix over operation pairs, so adding an action means saying which coordinates it writes rather than extending an N-by-N table. Only committed coordinates appear. A minted fragment, field, or base has no id in the read version, so two writers minting at the same time never collide. Footprints are derived from the actions at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict and the rule can be tightened without a format change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the conservative always-retry with a footprint comparison. A legacy operation on the other side gets a footprint through its action translation, so an action set can be checked against a concurrent named operation without either needing an entry in the operation-pair matrix. An operation with no translation still falls back to retry. Rebasing an action set onto a newer version is a no-op: its minted ids are allocated when the actions are applied, against whichever manifest they land on, and its committed references name coordinates that do not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaying the same actions against the manifest a previous run produced re-resolves their local tokens against the newer counters, so the second run mints different fragment and field ids without any action changing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each test is one commit doing work that would otherwise have taken several: a fragment added and then modified, a field added and then filled, both inside a single version. Also covers row id assignment for minted fragments and both sides of the footprint conflict rule through the real commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes a field from the schema, taking its descendants with it. The slots that backed the dropped fields in each data file are tombstoned rather than removed, so a file's remaining columns stay at the positions they were written at; a file left backing nothing live is pruned during normalization. For conflicts, a drop writes every coordinate belonging to the field -- its definition and its data in every fragment -- so it collides with a concurrent alter or data rewrite of the same field. Field ids come from a monotonic counter, so a dropped id is never reused and a stale data file naming it cannot be mistaken for a later field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The action code was split by phase -- apply, footprint, proto -- so understanding one action meant reading four files and adding one meant touching four match statements. Each action now owns a module holding its definition, `apply`, `footprint`, wire conversions, and tests. `Action` dispatches to them. The phase modules keep what is genuinely shared: `apply` holds the `ApplyState` the actions program against, `footprint` the coordinate space and the conflict comparison, `proto` the envelope and dispatch. No behavior change. The one difference is that `AddField` and `AddBase` now check for a duplicate local token before bumping the id counter, matching `AddFragment`; a duplicate token aborts the whole apply either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes a contiguous range of fragment ids off the counter without minting fragments for them, so a later (possibly distributed) writer can populate the range and name the ids as committed. The counterpart of the legacy ReserveFragments operation. Nothing backs a reserved id, so the manifest assembly cannot infer it from the fragment list; apply raises the manifest's high-water mark to cover the range instead. The range starts wherever the counter stands, which unlike the legacy operation does not waste an id on an empty table. The action writes no coordinates: ids come off a monotonic counter, so two operations reserving at once get disjoint ranges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Empties the table -- schema, schema metadata, fragments and indices -- leaving the config, table metadata and base paths alone. This is how a full Overwrite / CREATE OR REPLACE decomposes: reset, then write the fresh schema and data as later actions in the same operation. The id counters keep going across the reset, so a field or fragment added afterwards never reuses an id a stale file might still name. Conflict detection gains its first non-enumerable footprint. A reset writes every coordinate there is, including ones a concurrent set would only mint, so it takes the table exclusively: any concurrent action set is preempted, including a pure append that writes no committed coordinate at all. Two proto tests used ResetTable as their example of a drafted-but- unimplemented action; they now use RefreshRowVersionMetadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edits the four string maps a manifest carries -- dataset config, table metadata, schema metadata and per-field metadata -- with the same UpdateMap the legacy UpdateConfig operation takes. Per-field updates are keyed by Ref, so a field minted earlier in the same operation can be given metadata before it has a committed id. The unenforced primary and clustering keys keep the immutability rules the legacy path enforces: each is rejected if changed once set, or if a reserved key is written with a value that installs no valid key. The check runs on every apply including a conflict rebase, so it catches the concurrent-writer race too. Conflicts are per key: two operations editing different keys of the same map commute. A replacement writes keys it does not name, so like a fragment removal it is matched by map rather than by key, and two replacements of one map collide even when both are clears. A field's metadata belongs to the field, so dropping the field collides with a concurrent update to its metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`UserOperation` read as a category next to `Operation::Append` rather than naming what distinguishes it. Rename it to `CompositeOperation`, which says the thing: a composite of granular actions committed atomically. `UserAction` keeps its name -- it is the user-facing grouping of actions, the level a user recognizes, as distinct from the granular actions inside it. Also drops `CompositeOperation::description`. Nothing reads it, and every construction site set it to something the step descriptions already imply. The per-step `UserAction::description` is what keeps history readable; an operation-level name can come back if squashing turns out to need one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s differ `AddFragment`'s version metadata fields said only "stamp at apply" without saying who would ever set them. Name the two producers that must: an update carries each row's `created_at` forward from its source fragment, and a compaction rechunks both sequences off the fragments it merged. `TombstoneFieldData::data_change` deferred to `AddFragment`'s doc, which does not cover it. State the case directly: a tombstone paired with a re-add in the same operation moves bytes without changing values. Strikes an over-long comment in `conflicts.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding an action meant editing five per-variant matches -- the enum, `name`, `is_data_change`, `apply`, `footprint` -- plus both directions of the wire encoding, in two files. Nothing but review caught a variant handled inconsistently across them. `for_each_action!` now holds the vocabulary as a single list of variant names, and the enum, its four forwarding methods, and both proto conversions expand from it. Adding an action is a module plus one line. The variant name is also the protobuf oneof variant and the name in errors, so those cannot drift. `is_data_change` moves into the action modules alongside `apply` and `footprint`, so an action's module answers every question about it. The grouped comments that used to sit in the central match move with it, one to each action. This is the `enum_dispatch` pattern done locally: it needs no new dependency, and it also covers the decode direction, which dispatches on the protobuf oneof tag rather than on `self` and so is out of `enum_dispatch`'s reach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dedicated integration test gets its own binary, which links every dependency of the crate. Fold the six tests into the existing `dataset::tests::dataset_transactions` module as a `composite` submodule and delete the standalone target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The field claimed a minted fragment "has no committed rows to delete", which reads as a semantic restriction and is at best ambiguous. The actual constraint is mechanical: a deletion file's path embeds the fragment id, so the writer must know the committed id before it can write the file, and a minted id does not exist until apply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ifest `build_manifest_from_actions` ran the actions and then assembled the manifest in one 80-line body. Move the assembly to `ApplyState::into_manifest`, so the entry point reads as the three steps it is: make the state, run the actions, turn it into a manifest. `ApplyState` now borrows the manifest it was built from rather than copying everything it needs out of it, which is also what lets `into_manifest` reach the read version without being handed it back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`AlterField`, `DropField`, and `TombstoneFieldData` took raw committed field ids, so none of them could name a field minted earlier in the same operation. Squashing needs exactly that: collapsing "add column c" and "rename c" into one operation leaves an `AddField` whose field has no committed id, and a squash rewrites already-committed transactions, so it cannot re-plan the data to avoid the reference. All three now take a `Ref`, matching the fragment actions. Footprints follow the rule already used there: a local reference records no coordinate, since a field this operation mints is invisible to a concurrent writer. `AlterField` loses its `Default` impl -- `Ref` has no meaningful zero, and a default field reference would silently mean field 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebasing onto the merged transaction module split brings three API changes the action code predates: `RowIdMeta::Inline` wraps `InlineRowIds` rather than a byte vector, `DataFile::new`/`new_unstarted` take a `ConcreteFileVersion` instead of a major/minor pair, and `Operation::Project` carries `preserves_nullability`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The index actions edit the index list as they are reached, so the list has to be part of the state actions are applied against rather than an argument the manifest assembly receives separately. ResetTable clears it where it stands instead of setting a flag the assembly reads back.
The format has no first-class index apart from its segments, so one action covers both creating an index and extending one: a logical index is the set of segments sharing a name. Its fields, coverage, and base path are Refs, so a segment can index what the same operation just wrote. Three format changes fall out of implementing it: - `covered_fragments` becomes an optional wrapper message. A bare `repeated` cannot tell "no coverage recorded" -- what the system indices carry, and what the query path treats as "serve this segment" -- from "covers no fragment", which it treats as "skip it". - Added `base`, without which a segment imported from another dataset cannot be expressed. - Added `created_at` and `dataset_version`, both describing the build rather than where it lands. `dataset_version` in particular is a correctness gate (an overlay committed at or before it counts as folded into the index) and a merged segment reflects only as much as its oldest input, so it is genuinely below the read version and cannot be derived. It defaults to the read version and may not exceed it.
Dropping a logical index is one of these per segment carrying its name, since the format knows only segments. Removing a segment the dataset does not have is rejected rather than treated as a no-op: it means the operation was planned against a different set of segments. Segments are named by uuid, which the writer picks, so the footprint coordinate is the segment itself -- a concurrent writer extending the same logical index adds a segment of its own and does not collide.
Moves fragments in and out of a segment's coverage without rewriting the segment, which is what lets an append and the coverage extension over what it appended commit as one operation. A segment recording no coverage is rejected rather than treated as an empty set to add to: "unknown coverage" is what the query path serves everything for, so turning it into a concrete set would silently narrow the segment.
The legacy operation already carries its removals and additions as two lists, so the recipe is one RemoveIndexSegment per removal followed by one AddIndexSegment per addition. Parity tests build the same manifest down both paths and assert the resulting index metadata is identical. Two edges the legacy path tolerates are rejected here: removing a segment the manifest does not have, and adding one whose uuid an existing segment already uses. Either means the operation was planned against a different set of segments than it is landing on.
Covers the three index actions through the real commit path: one commit that appends a fragment and adds a segment covering it by local token, one that swaps a segment out, and one that moves coverage around.
…itions `AddIndexSegment`'s DeepSizeOf skipped `index_details` on the grounds that it is opaque. It is only a type url and a byte string, so both are now measured. `AdjustIndexCoverage` did not say when adding a fragment to a segment's coverage is legitimate. It is one case -- a rewrite moved rows the segment already covered into a new fragment, which the segment reaches through the fragment-reuse remapping. Adding a fragment of new rows is a writer error that nothing here can detect, so it is called out.
Appends overlay files to a fragment, supplying new values for a subset of its (row offset, field) cells without rewriting its base data files. Each overlay's `committed_version` is stamped with the version the commit produces, so a retry against a newer manifest re-stamps rather than backdates. Overlays are appended, never replaced, so the action writes no coordinate of its own -- two concurrent overlays over the same cells both land and the newer version wins. It does record that the fragment must still be there, which is a new kind of entry in the footprint: a dependency rather than a write.
Restamps the per-row `last_updated_at_version` sequence of fragments whose columns were rewritten in place, which is what a legacy Merge does implicitly. Nothing else in an operation restates when those rows last changed, because rewriting columns in place leaves the rows where they are. `created_at_version` is left alone: the rows are the same rows, and a row this operation mints gets both stamps from the AddFragment that minted it. Naming a fragment on a dataset without stable row ids is rejected rather than fabricating sequences that have nowhere to live.
Records which MemWAL SSTables have been compacted into the base table, in the MemWAL system index. Per shard the highest generation wins, so replaying an older commit over a newer one cannot walk the progress backwards. The rows were already readable through the WAL, so this is bookkeeping about where they live rather than a change to them. The drafted `update_compacted_sstables` oneof field is renamed to `update_compacted_ss_tables` so the generated variant name matches the message name, which is what the action vocabulary keys the wire encoding off. The tag is unchanged.
A precondition rather than a delta: the keys this operation inserts must not collide with keys a concurrent commit inserted. The key columns are an unenforced primary key, so nothing in the manifest records which keys exist -- the filter of inserted key hashes has to travel with the operation because it cannot be recovered from any post-image. This is the first thing two footprints compare that is not a coordinate, so the footprint grows a row-insertion marker (set by an AddFragment that is a data change) and the assertions themselves. Two sets are compatible when both say which keys they insert, over the same columns, and the filters provably do not intersect; an unqualified insert, different key columns, or filters built with incomparable parameters all leave the assertion unverifiable, which counts as a conflict. With this the implemented vocabulary covers the whole draft, so the "drafted but not implemented" rejection has nothing left to reject and is replaced by one for an action written by a newer Lance -- which protobuf decodes as no variant at all.
…ctions Both lower onto a single new action each, so the parity tests build the same manifest twice -- once down the legacy path, once through the actions -- and assert they agree. They agree except in one place: the legacy UpdateMemWalState arm never carries the read version's fragments into the manifest it builds, so it empties the table. The translated path leaves the data alone, which is what the operation means. The test asserts both, so the difference is recorded rather than hidden.
Four commits through the real commit path: appending a fragment and overlaying an existing one in the same version, restamping row versions for a fragment whose column was rewritten in place, recording MemWAL compaction progress, and carrying a key assertion alongside the insert it guards.
`Operation::UpdateMemWalState` now carries `require_index_catchup`. Requiring catch-up is a one-way feature-flag migration with no action of its own, so an operation asking for it does not translate; ordinary progress updates translate as before. `update_mem_wal_index_compacted_sstables` also stopped creating the index and stopped tolerating a stale generation, so `UpdateCompactedSsTables` now inherits both rejections. Its docs and tests say so, and the tests seed the index the way a real table would have it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-format
The comment claimed rows are not a coordinate because a row a concurrent writer inserts has no id anyone could name. Both halves are wrong: rows do have ids, and a new data file can carry rows that already existed. The actual reason is that two writers inserting the same user-supplied key write it into fragments of their own, so their coordinates stay disjoint however badly the keys collide. Also records that the flag over-approximates -- the rows a merge insert updates arrive in a new fragment too -- and why `Update` has no translation yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wjones127
force-pushed
the
will/transaction-v2-index-claims
branch
from
August 19, 2026 21:02
6023622 to
a66cb26
Compare
An update that moves rows -- out of the fragments they were in, by deletion file or by the fragment going away, and into fragments it mints -- is a `Delete` and an `Append` in one step, plus the inserted-key assertion and the SSTable progress it carries. It now translates, reusing the recipes those two operations already have. Its other forms are rejected, for the same reason `Merge` and `Project` are: they turn on what the read version holds, which a conversion taking only the operation cannot see. A row rewrite decides which indices still cover the rows it moved by reading the current indices, schema and overlays; a column rewrite tombstones overlaid fields by reading the overlays a fragment carries now; in-place field modification would have to diff against the read version to tell the data files it wrote from the ones already there; and a partial restamp has no action, since RefreshRowVersionMetadata restamps a whole fragment. The conflict resolver gets this for free: a legacy vertical update concurrent with an action set is now compared by footprint instead of being conservatively rejected, so a deletion and a column rewrite of one fragment both commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflict detection treated a logical index as a single coordinate, so two writers adding segments to the same index always collided. That was a port of the legacy `CreateIndex` rule, which conflicts on index name alone. Since the query path unions the segments of an index, two segments over disjoint fragments are both valid and should both commit. Replaces `Coordinate::IndexName` with an index claim, the same kind of pairwise entry `AssertUniqueKeys` uses. Two claims on one index conflict when they describe overlapping committed fragments, when either does not state its reach, or when they disagree about what the index is (fields, details, version). Fragments minted in the same operation are left out of the comparison: they have no id a concurrent writer could be covering. `RemoveIndexSegment` and `AdjustIndexCoverage` gain a `name`. The coverage adjustment needs it to make its claim; both use it at apply to reject an action whose segment belongs to a different index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wjones127
force-pushed
the
will/transaction-v2-index-claims
branch
from
August 19, 2026 21:36
a66cb26 to
70d1d07
Compare
wjones127
force-pushed
the
will/transaction-v2-remaining-actions
branch
from
August 19, 2026 22:49
cf2a871 to
adc53c8
Compare
This was referenced Aug 19, 2026
Contributor
Author
|
Folded into #8645, alongside the index management actions it builds on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Conflict detection treated a logical index as a single thing that only one writer could touch at a time: any two commits adding a segment to the same index collided. That was a faithful port of the legacy
CreateIndexrule, which conflicts on index name alone.But an index is the set of segments sharing a name, and the query path unions them. Two writers indexing different fragments are both right, and both should be able to commit. This PR replaces the name-level rule with a claim on the index, the same kind of pairwise footprint entry that key assertions use. Two claims on one index conflict when they cover overlapping committed fragments, when either does not state what it covers (the system indices), or when they disagree about what the index is — its fields, its config, or its version. Fragments minted by the operation itself are left out of the comparison: they have no id yet, so no concurrent writer can be covering one.
RemoveIndexSegmentandAdjustIndexCoveragegain anamefield. The coverage adjustment needs it to say which index it is widening; both use it when applying to reject an action whose segment turns out to belong to a different index than the one it was planned against.Example
Two writers each append a fragment and build an index segment over what they appended. Previously the second commit was rejected as a conflict; now both land, and the index is the union of the two segments.
Breaking changes
Adds two fields to the unstable Transaction V2 action protos. The format is unstable, so no compatibility path is provided.