feat(transaction): implement the Transaction V2 action vocabulary - #8644
Draft
wjones127 wants to merge 24 commits into
Draft
feat(transaction): implement the Transaction V2 action vocabulary#8644wjones127 wants to merge 24 commits into
wjones127 wants to merge 24 commits into
Conversation
Introduces the Rust side of the action-based transaction draft: `Ref`, `UserOperation`, `UserAction`, and the eight `Action` variants this build implements (AddFragment, AddDataFile, AddField, AddBase, TombstoneFieldData, RemoveFragment, SetDeletionFile, AlterField). Types only -- no wire conversion, apply, or conflict handling yet, so nothing reaches these from the commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `Operation::UserOperation`, the Transaction V2 arm of the transaction oneof, and its protobuf conversions in both directions. Loading a V2 transaction now yields an action set instead of an outright rejection; an action the build does not implement is still rejected rather than skipped, so a concurrent V2 commit can never be silently treated as a no-op. The rest of the transaction machinery gains the variant but no behavior: build_manifest returns NotSupported, and conflict checks route through a single `check_action_txn` that conservatively demands a retry. Apply and conflict rules follow in later commits. Also extracts `From<&DeletionFile> for pb::DeletionFile`, previously inlined in the fragment conversion and now needed by SetDeletionFile too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pulls the two operation-independent stages out of `build_manifest`: `normalize_fragments` (order, drop fully-tombstoned files, check overlay order) and `assemble_manifest` (construct the manifest and apply the tag, feature flags, timestamp, and fragment id watermark). The overwrite-only storage format override becomes an explicit parameter rather than a match on the operation inside the assembly step. No behavior change; the action-based apply path needs the same two stages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the action apply path: `build_manifest_from_actions` walks the action set in order against a working copy of the read-version state, and the four minting actions (AddFragment, AddDataFile, AddField, AddBase) allocate ids from the target's counters as they are reached. Each mint records its id against the action's local token, so a later action in the same operation resolves that token to the id this apply chose. The same action set replayed against a different version therefore lands on different ids without any action changing -- the property branch merge needs. An action set requires an existing dataset: it is a delta, so there is nothing for it to be a delta against at creation time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applies TombstoneFieldData, RemoveFragment, SetDeletionFile, and AlterField, completing the eight-action apply path. Tombstoning a field's data and retyping a field both leave any index over that field describing values the fragment no longer holds, so the affected fragments are dropped from the index bitmap rather than the index being discarded outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ions Adds `TryFrom<&Operation> for Vec<UserAction>`, the recipe that turns a named operation into the granular actions it decomposes into. Squashing several operations into one commit is concatenation once both sides speak actions. Translation is fail-closed: an operation with no recipe yet, or one carrying a detail the actions cannot express, is rejected. Each case is covered by a parity test that builds the manifest twice -- once down the legacy path, once through the translation -- and asserts the two agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replacing a field's data becomes TombstoneFieldData followed by AddDataFile. The legacy path swaps the path on the existing file in place, so the two produce the same set of data files per fragment but not necessarily the same order; files are addressed by field, so the order carries no meaning. Also documents why Merge and Project are not translated: both hand over a whole new schema rather than a delta, and Project needs a field-removal action this draft does not define. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A footprint is the set of coordinates an action set writes. Two concurrent sets can both commit when neither writes what the other writes -- one structural rule instead of a matrix over operation pairs, so adding an action means saying which coordinates it writes rather than extending an N-by-N table. Only committed coordinates appear. A minted fragment, field, or base has no id in the read version, so two writers minting at the same time never collide. Footprints are derived from the actions at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict and the rule can be tightened without a format change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the conservative always-retry with a footprint comparison. A legacy operation on the other side gets a footprint through its action translation, so an action set can be checked against a concurrent named operation without either needing an entry in the operation-pair matrix. An operation with no translation still falls back to retry. Rebasing an action set onto a newer version is a no-op: its minted ids are allocated when the actions are applied, against whichever manifest they land on, and its committed references name coordinates that do not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaying the same actions against the manifest a previous run produced re-resolves their local tokens against the newer counters, so the second run mints different fragment and field ids without any action changing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each test is one commit doing work that would otherwise have taken several: a fragment added and then modified, a field added and then filled, both inside a single version. Also covers row id assignment for minted fragments and both sides of the footprint conflict rule through the real commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes a field from the schema, taking its descendants with it. The slots that backed the dropped fields in each data file are tombstoned rather than removed, so a file's remaining columns stay at the positions they were written at; a file left backing nothing live is pruned during normalization. For conflicts, a drop writes every coordinate belonging to the field -- its definition and its data in every fragment -- so it collides with a concurrent alter or data rewrite of the same field. Field ids come from a monotonic counter, so a dropped id is never reused and a stale data file naming it cannot be mistaken for a later field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The action code was split by phase -- apply, footprint, proto -- so understanding one action meant reading four files and adding one meant touching four match statements. Each action now owns a module holding its definition, `apply`, `footprint`, wire conversions, and tests. `Action` dispatches to them. The phase modules keep what is genuinely shared: `apply` holds the `ApplyState` the actions program against, `footprint` the coordinate space and the conflict comparison, `proto` the envelope and dispatch. No behavior change. The one difference is that `AddField` and `AddBase` now check for a duplicate local token before bumping the id counter, matching `AddFragment`; a duplicate token aborts the whole apply either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes a contiguous range of fragment ids off the counter without minting fragments for them, so a later (possibly distributed) writer can populate the range and name the ids as committed. The counterpart of the legacy ReserveFragments operation. Nothing backs a reserved id, so the manifest assembly cannot infer it from the fragment list; apply raises the manifest's high-water mark to cover the range instead. The range starts wherever the counter stands, which unlike the legacy operation does not waste an id on an empty table. The action writes no coordinates: ids come off a monotonic counter, so two operations reserving at once get disjoint ranges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Empties the table -- schema, schema metadata, fragments and indices -- leaving the config, table metadata and base paths alone. This is how a full Overwrite / CREATE OR REPLACE decomposes: reset, then write the fresh schema and data as later actions in the same operation. The id counters keep going across the reset, so a field or fragment added afterwards never reuses an id a stale file might still name. Conflict detection gains its first non-enumerable footprint. A reset writes every coordinate there is, including ones a concurrent set would only mint, so it takes the table exclusively: any concurrent action set is preempted, including a pure append that writes no committed coordinate at all. Two proto tests used ResetTable as their example of a drafted-but- unimplemented action; they now use RefreshRowVersionMetadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edits the four string maps a manifest carries -- dataset config, table metadata, schema metadata and per-field metadata -- with the same UpdateMap the legacy UpdateConfig operation takes. Per-field updates are keyed by Ref, so a field minted earlier in the same operation can be given metadata before it has a committed id. The unenforced primary and clustering keys keep the immutability rules the legacy path enforces: each is rejected if changed once set, or if a reserved key is written with a value that installs no valid key. The check runs on every apply including a conflict rebase, so it catches the concurrent-writer race too. Conflicts are per key: two operations editing different keys of the same map commute. A replacement writes keys it does not name, so like a fragment removal it is matched by map rather than by key, and two replacements of one map collide even when both are clears. A field's metadata belongs to the field, so dropping the field collides with a concurrent update to its metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`UserOperation` read as a category next to `Operation::Append` rather than naming what distinguishes it. Rename it to `CompositeOperation`, which says the thing: a composite of granular actions committed atomically. `UserAction` keeps its name -- it is the user-facing grouping of actions, the level a user recognizes, as distinct from the granular actions inside it. Also drops `CompositeOperation::description`. Nothing reads it, and every construction site set it to something the step descriptions already imply. The per-step `UserAction::description` is what keeps history readable; an operation-level name can come back if squashing turns out to need one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s differ `AddFragment`'s version metadata fields said only "stamp at apply" without saying who would ever set them. Name the two producers that must: an update carries each row's `created_at` forward from its source fragment, and a compaction rechunks both sequences off the fragments it merged. `TombstoneFieldData::data_change` deferred to `AddFragment`'s doc, which does not cover it. State the case directly: a tombstone paired with a re-add in the same operation moves bytes without changing values. Strikes an over-long comment in `conflicts.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding an action meant editing five per-variant matches -- the enum, `name`, `is_data_change`, `apply`, `footprint` -- plus both directions of the wire encoding, in two files. Nothing but review caught a variant handled inconsistently across them. `for_each_action!` now holds the vocabulary as a single list of variant names, and the enum, its four forwarding methods, and both proto conversions expand from it. Adding an action is a module plus one line. The variant name is also the protobuf oneof variant and the name in errors, so those cannot drift. `is_data_change` moves into the action modules alongside `apply` and `footprint`, so an action's module answers every question about it. The grouped comments that used to sit in the central match move with it, one to each action. This is the `enum_dispatch` pattern done locally: it needs no new dependency, and it also covers the decode direction, which dispatches on the protobuf oneof tag rather than on `self` and so is out of `enum_dispatch`'s reach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dedicated integration test gets its own binary, which links every dependency of the crate. Fold the six tests into the existing `dataset::tests::dataset_transactions` module as a `composite` submodule and delete the standalone target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The field claimed a minted fragment "has no committed rows to delete", which reads as a semantic restriction and is at best ambiguous. The actual constraint is mechanical: a deletion file's path embeds the fragment id, so the writer must know the committed id before it can write the file, and a minted id does not exist until apply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ifest `build_manifest_from_actions` ran the actions and then assembled the manifest in one 80-line body. Move the assembly to `ApplyState::into_manifest`, so the entry point reads as the three steps it is: make the state, run the actions, turn it into a manifest. `ApplyState` now borrows the manifest it was built from rather than copying everything it needs out of it, which is also what lets `into_manifest` reach the read version without being handed it back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`AlterField`, `DropField`, and `TombstoneFieldData` took raw committed field ids, so none of them could name a field minted earlier in the same operation. Squashing needs exactly that: collapsing "add column c" and "rename c" into one operation leaves an `AddField` whose field has no committed id, and a squash rewrites already-committed transactions, so it cannot re-plan the data to avoid the reference. All three now take a `Ref`, matching the fragment actions. Footprints follow the rule already used there: a local reference records no coordinate, since a field this operation mints is invisible to a concurrent writer. `AlterField` loses its `Default` impl -- `Ref` has no meaningful zero, and a default field reference would silently mean field 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebasing onto the merged transaction module split brings three API changes the action code predates: `RowIdMeta::Inline` wraps `InlineRowIds` rather than a byte vector, `DataFile::new`/`new_unstarted` take a `ConcreteFileVersion` instead of a major/minor pair, and `Operation::Project` carries `preserves_nullability`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7954, which defines the Transaction V2 wire format. That PR defines the actions on the wire; this one implements them in Rust. It carries no format change of its own — every proto edit this stack needs now lands in #7954, so the format is a single vote.
Supersedes #8554, which had its head on a fork and so could not be part of a GitHub stack.
Today a transaction names one operation from a fixed list —
Append,Delete,Merge, and so on — and each one is a post-image: it says what the dataset should look like afterwards. Two consequences follow. Work that spans several kinds of change needs several commits, so there is no way to add a fragment and then modify it atomically. And every new operation has to be taught how it conflicts with every existing one, which is an N-by-N table that grows quadratically.This PR adds the delta-based alternative. A transaction can now carry a
CompositeOperation: an ordered list of granular actions that commit together as one manifest change. Twelve actions are implemented —AddFragment,AddField,AddDataFile,AddBase,TombstoneFieldData,RemoveFragment,SetDeletionFile,AlterField,DropField,ConfigUpdate,ReserveFragmentIds, andResetTable. Between them these cover the whole fragment, data-file, schema, and config surface; the drafted actions left for later are the index-segment, overlay, MemWAL, and assertion families.Each action gets its own module holding its definition, how it is applied, which coordinates it writes, and its wire encoding, so one action can be reviewed — or added — without reading the other eleven.
Two things fall out of the delta model:
Steps can reference ids the same commit is about to create. An action that allocates a fragment, field, or base id names it with a local token; later actions in the same operation refer back to that token. Ids are minted when the actions are applied, against whichever manifest they land on, so replaying the same action set against a newer version simply produces different ids. Moving an action set to a newer version needs no rewriting at all.
Conflicts are decided by comparing write sets. Each action says which coordinates it writes — a field's data in a fragment, a fragment's deletion file, a field's definition, a base path's name, a config key. Two concurrent transactions can both commit when neither writes what the other writes. Adding an action means saying which coordinates it writes, not adding a row and a column to the conflict table. Footprints are computed at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict.
Some writes cannot be enumerated, and those are tracked as such rather than approximated: removing a fragment writes everything inside it, replacing a config map writes keys it never names, and
ResetTablewrites the whole table, so it conflicts with any concurrent action set at all — including a pure append that writes no committed coordinate.The two models interoperate.
Append,Delete,UpdateBases, andDataReplacementcan be translated into the actions they decompose into, which is both how an action set is checked against a concurrent named operation and the groundwork for squashing several operations into one commit. Each translation has a parity test that builds the manifest twice — once down the legacy path, once through the translation — and asserts the two agree.Anything not yet expressible is rejected rather than approximated: an unimplemented action fails to parse, an untranslatable operation falls back to the conservative always-retry, and an action naming something that does not exist is an error.
Not included
MergeandProjectare not translated. Both hand over a whole new schema instead of a description of what changed, so recovering the delta needs the read version's schema to diff against, whichTryFrom<&Operation>does not have. The actions themselves are sufficient —Projectis a set ofDropFields andMergea set ofAddFields plus their data files — so this is a plumbing gap, not a vocabulary one.Seven of the nineteen drafted actions are still unimplemented:
AddOverlays,RefreshRowVersionMetadata,UpdateCompactedSsTables,AddIndexSegment,RemoveIndexSegment,AdjustIndexCoverage, andAssertUniqueKeys. Three of those need the index-segment model settled and three need the MemWAL and overlay subsystems;AssertUniqueKeysis not a delta at all but a precondition, so it needs a validation hook rather than an apply. Parsing any of them is an error rather than a skip, so an old reader cannot apply a partial transaction.There are no Python or Java bindings. The Rust API is public but documented as a pre-vote draft, matching #7954's stability caveat, so it can change without a deprecation cycle.
Squashing several already-committed operations into one action set is left for later. The composite behavior it depends on is proven end-to-end here (
rust/lance/tests/composite_transaction.rs), but rewiring one operation's committed references onto another's freshly minted ids is a separate problem.