Skip to content

feat(transaction): implement the Transaction V2 action vocabulary - #8644

Draft
wjones127 wants to merge 24 commits into
will/oss-1530-draft-action-based-transactionfrom
will/transaction-v2-actions
Draft

feat(transaction): implement the Transaction V2 action vocabulary#8644
wjones127 wants to merge 24 commits into
will/oss-1530-draft-action-based-transactionfrom
will/transaction-v2-actions

Conversation

@wjones127

Copy link
Copy Markdown
Contributor

Stacked on #7954, which defines the Transaction V2 wire format. That PR defines the actions on the wire; this one implements them in Rust. It carries no format change of its own — every proto edit this stack needs now lands in #7954, so the format is a single vote.

Supersedes #8554, which had its head on a fork and so could not be part of a GitHub stack.

Today a transaction names one operation from a fixed list — Append, Delete, Merge, and so on — and each one is a post-image: it says what the dataset should look like afterwards. Two consequences follow. Work that spans several kinds of change needs several commits, so there is no way to add a fragment and then modify it atomically. And every new operation has to be taught how it conflicts with every existing one, which is an N-by-N table that grows quadratically.

This PR adds the delta-based alternative. A transaction can now carry a CompositeOperation: an ordered list of granular actions that commit together as one manifest change. Twelve actions are implemented — AddFragment, AddField, AddDataFile, AddBase, TombstoneFieldData, RemoveFragment, SetDeletionFile, AlterField, DropField, ConfigUpdate, ReserveFragmentIds, and ResetTable. Between them these cover the whole fragment, data-file, schema, and config surface; the drafted actions left for later are the index-segment, overlay, MemWAL, and assertion families.

Each action gets its own module holding its definition, how it is applied, which coordinates it writes, and its wire encoding, so one action can be reviewed — or added — without reading the other eleven.

Two things fall out of the delta model:

Steps can reference ids the same commit is about to create. An action that allocates a fragment, field, or base id names it with a local token; later actions in the same operation refer back to that token. Ids are minted when the actions are applied, against whichever manifest they land on, so replaying the same action set against a newer version simply produces different ids. Moving an action set to a newer version needs no rewriting at all.

Conflicts are decided by comparing write sets. Each action says which coordinates it writes — a field's data in a fragment, a fragment's deletion file, a field's definition, a base path's name, a config key. Two concurrent transactions can both commit when neither writes what the other writes. Adding an action means saying which coordinates it writes, not adding a row and a column to the conflict table. Footprints are computed at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict.

Some writes cannot be enumerated, and those are tracked as such rather than approximated: removing a fragment writes everything inside it, replacing a config map writes keys it never names, and ResetTable writes the whole table, so it conflicts with any concurrent action set at all — including a pure append that writes no committed coordinate.

The two models interoperate. Append, Delete, UpdateBases, and DataReplacement can be translated into the actions they decompose into, which is both how an action set is checked against a concurrent named operation and the groundwork for squashing several operations into one commit. Each translation has a parity test that builds the manifest twice — once down the legacy path, once through the translation — and asserts the two agree.

Anything not yet expressible is rejected rather than approximated: an unimplemented action fails to parse, an untranslatable operation falls back to the conservative always-retry, and an action naming something that does not exist is an error.

Not included

Merge and Project are not translated. Both hand over a whole new schema instead of a description of what changed, so recovering the delta needs the read version's schema to diff against, which TryFrom<&Operation> does not have. The actions themselves are sufficient — Project is a set of DropFields and Merge a set of AddFields plus their data files — so this is a plumbing gap, not a vocabulary one.

Seven of the nineteen drafted actions are still unimplemented: AddOverlays, RefreshRowVersionMetadata, UpdateCompactedSsTables, AddIndexSegment, RemoveIndexSegment, AdjustIndexCoverage, and AssertUniqueKeys. Three of those need the index-segment model settled and three need the MemWAL and overlay subsystems; AssertUniqueKeys is not a delta at all but a precondition, so it needs a validation hook rather than an apply. Parsing any of them is an error rather than a skip, so an old reader cannot apply a partial transaction.

There are no Python or Java bindings. The Rust API is public but documented as a pre-vote draft, matching #7954's stability caveat, so it can change without a deprecation cycle.

Squashing several already-committed operations into one action set is left for later. The composite behavior it depends on is proven end-to-end here (rust/lance/tests/composite_transaction.rs), but rewiring one operation's committed references onto another's freshly minted ids is a separate problem.

wjones127 and others added 24 commits August 19, 2026 15:12
Introduces the Rust side of the action-based transaction draft: `Ref`,
`UserOperation`, `UserAction`, and the eight `Action` variants this build
implements (AddFragment, AddDataFile, AddField, AddBase, TombstoneFieldData,
RemoveFragment, SetDeletionFile, AlterField).

Types only -- no wire conversion, apply, or conflict handling yet, so nothing
reaches these from the commit path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `Operation::UserOperation`, the Transaction V2 arm of the transaction
oneof, and its protobuf conversions in both directions. Loading a V2
transaction now yields an action set instead of an outright rejection; an
action the build does not implement is still rejected rather than skipped,
so a concurrent V2 commit can never be silently treated as a no-op.

The rest of the transaction machinery gains the variant but no behavior:
build_manifest returns NotSupported, and conflict checks route through a
single `check_action_txn` that conservatively demands a retry. Apply and
conflict rules follow in later commits.

Also extracts `From<&DeletionFile> for pb::DeletionFile`, previously inlined
in the fragment conversion and now needed by SetDeletionFile too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pulls the two operation-independent stages out of `build_manifest`:
`normalize_fragments` (order, drop fully-tombstoned files, check overlay
order) and `assemble_manifest` (construct the manifest and apply the tag,
feature flags, timestamp, and fragment id watermark).

The overwrite-only storage format override becomes an explicit parameter
rather than a match on the operation inside the assembly step. No behavior
change; the action-based apply path needs the same two stages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the action apply path: `build_manifest_from_actions` walks the action
set in order against a working copy of the read-version state, and the four
minting actions (AddFragment, AddDataFile, AddField, AddBase) allocate ids
from the target's counters as they are reached.

Each mint records its id against the action's local token, so a later action
in the same operation resolves that token to the id this apply chose. The
same action set replayed against a different version therefore lands on
different ids without any action changing -- the property branch merge needs.

An action set requires an existing dataset: it is a delta, so there is
nothing for it to be a delta against at creation time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applies TombstoneFieldData, RemoveFragment, SetDeletionFile, and
AlterField, completing the eight-action apply path.

Tombstoning a field's data and retyping a field both leave any index
over that field describing values the fragment no longer holds, so the
affected fragments are dropped from the index bitmap rather than the
index being discarded outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ions

Adds `TryFrom<&Operation> for Vec<UserAction>`, the recipe that turns a
named operation into the granular actions it decomposes into. Squashing
several operations into one commit is concatenation once both sides speak
actions.

Translation is fail-closed: an operation with no recipe yet, or one
carrying a detail the actions cannot express, is rejected.

Each case is covered by a parity test that builds the manifest twice --
once down the legacy path, once through the translation -- and asserts
the two agree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replacing a field's data becomes TombstoneFieldData followed by
AddDataFile. The legacy path swaps the path on the existing file in
place, so the two produce the same set of data files per fragment but
not necessarily the same order; files are addressed by field, so the
order carries no meaning.

Also documents why Merge and Project are not translated: both hand over
a whole new schema rather than a delta, and Project needs a
field-removal action this draft does not define.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A footprint is the set of coordinates an action set writes. Two
concurrent sets can both commit when neither writes what the other
writes -- one structural rule instead of a matrix over operation pairs,
so adding an action means saying which coordinates it writes rather than
extending an N-by-N table.

Only committed coordinates appear. A minted fragment, field, or base has
no id in the read version, so two writers minting at the same time never
collide.

Footprints are derived from the actions at conflict time and never
serialized, so a writer cannot pin down what a reader treats as a
conflict and the rule can be tightened without a format change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the conservative always-retry with a footprint comparison. A
legacy operation on the other side gets a footprint through its action
translation, so an action set can be checked against a concurrent named
operation without either needing an entry in the operation-pair matrix.
An operation with no translation still falls back to retry.

Rebasing an action set onto a newer version is a no-op: its minted ids
are allocated when the actions are applied, against whichever manifest
they land on, and its committed references name coordinates that do not
move.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaying the same actions against the manifest a previous run produced
re-resolves their local tokens against the newer counters, so the second
run mints different fragment and field ids without any action changing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each test is one commit doing work that would otherwise have taken
several: a fragment added and then modified, a field added and then
filled, both inside a single version. Also covers row id assignment for
minted fragments and both sides of the footprint conflict rule through
the real commit path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes a field from the schema, taking its descendants with it. The
slots that backed the dropped fields in each data file are tombstoned
rather than removed, so a file's remaining columns stay at the positions
they were written at; a file left backing nothing live is pruned during
normalization.

For conflicts, a drop writes every coordinate belonging to the field --
its definition and its data in every fragment -- so it collides with a
concurrent alter or data rewrite of the same field.

Field ids come from a monotonic counter, so a dropped id is never reused
and a stale data file naming it cannot be mistaken for a later field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The action code was split by phase -- apply, footprint, proto -- so
understanding one action meant reading four files and adding one meant
touching four match statements.

Each action now owns a module holding its definition, `apply`,
`footprint`, wire conversions, and tests. `Action` dispatches to them.
The phase modules keep what is genuinely shared: `apply` holds the
`ApplyState` the actions program against, `footprint` the coordinate
space and the conflict comparison, `proto` the envelope and dispatch.

No behavior change. The one difference is that `AddField` and `AddBase`
now check for a duplicate local token before bumping the id counter,
matching `AddFragment`; a duplicate token aborts the whole apply either
way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes a contiguous range of fragment ids off the counter without minting
fragments for them, so a later (possibly distributed) writer can populate
the range and name the ids as committed. The counterpart of the legacy
ReserveFragments operation.

Nothing backs a reserved id, so the manifest assembly cannot infer it
from the fragment list; apply raises the manifest's high-water mark to
cover the range instead. The range starts wherever the counter stands,
which unlike the legacy operation does not waste an id on an empty table.

The action writes no coordinates: ids come off a monotonic counter, so
two operations reserving at once get disjoint ranges.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Empties the table -- schema, schema metadata, fragments and indices --
leaving the config, table metadata and base paths alone. This is how a
full Overwrite / CREATE OR REPLACE decomposes: reset, then write the
fresh schema and data as later actions in the same operation.

The id counters keep going across the reset, so a field or fragment
added afterwards never reuses an id a stale file might still name.

Conflict detection gains its first non-enumerable footprint. A reset
writes every coordinate there is, including ones a concurrent set would
only mint, so it takes the table exclusively: any concurrent action set
is preempted, including a pure append that writes no committed
coordinate at all.

Two proto tests used ResetTable as their example of a drafted-but-
unimplemented action; they now use RefreshRowVersionMetadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edits the four string maps a manifest carries -- dataset config, table
metadata, schema metadata and per-field metadata -- with the same
UpdateMap the legacy UpdateConfig operation takes. Per-field updates are
keyed by Ref, so a field minted earlier in the same operation can be
given metadata before it has a committed id.

The unenforced primary and clustering keys keep the immutability rules
the legacy path enforces: each is rejected if changed once set, or if a
reserved key is written with a value that installs no valid key. The
check runs on every apply including a conflict rebase, so it catches the
concurrent-writer race too.

Conflicts are per key: two operations editing different keys of the same
map commute. A replacement writes keys it does not name, so like a
fragment removal it is matched by map rather than by key, and two
replacements of one map collide even when both are clears. A field's
metadata belongs to the field, so dropping the field collides with a
concurrent update to its metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`UserOperation` read as a category next to `Operation::Append` rather than
naming what distinguishes it. Rename it to `CompositeOperation`, which says
the thing: a composite of granular actions committed atomically.

`UserAction` keeps its name -- it is the user-facing grouping of actions,
the level a user recognizes, as distinct from the granular actions inside it.

Also drops `CompositeOperation::description`. Nothing reads it, and every
construction site set it to something the step descriptions already imply.
The per-step `UserAction::description` is what keeps history readable; an
operation-level name can come back if squashing turns out to need one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s differ

`AddFragment`'s version metadata fields said only "stamp at apply" without
saying who would ever set them. Name the two producers that must: an update
carries each row's `created_at` forward from its source fragment, and a
compaction rechunks both sequences off the fragments it merged.

`TombstoneFieldData::data_change` deferred to `AddFragment`'s doc, which
does not cover it. State the case directly: a tombstone paired with a re-add
in the same operation moves bytes without changing values.

Strikes an over-long comment in `conflicts.rs`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding an action meant editing five per-variant matches -- the enum, `name`,
`is_data_change`, `apply`, `footprint` -- plus both directions of the wire
encoding, in two files. Nothing but review caught a variant handled
inconsistently across them.

`for_each_action!` now holds the vocabulary as a single list of variant names,
and the enum, its four forwarding methods, and both proto conversions expand
from it. Adding an action is a module plus one line. The variant name is also
the protobuf oneof variant and the name in errors, so those cannot drift.

`is_data_change` moves into the action modules alongside `apply` and
`footprint`, so an action's module answers every question about it. The
grouped comments that used to sit in the central match move with it, one to
each action.

This is the `enum_dispatch` pattern done locally: it needs no new dependency,
and it also covers the decode direction, which dispatches on the protobuf
oneof tag rather than on `self` and so is out of `enum_dispatch`'s reach.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dedicated integration test gets its own binary, which links every
dependency of the crate. Fold the six tests into the existing
`dataset::tests::dataset_transactions` module as a `composite` submodule and
delete the standalone target.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The field claimed a minted fragment "has no committed rows to delete", which
reads as a semantic restriction and is at best ambiguous. The actual
constraint is mechanical: a deletion file's path embeds the fragment id, so
the writer must know the committed id before it can write the file, and a
minted id does not exist until apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ifest

`build_manifest_from_actions` ran the actions and then assembled the manifest
in one 80-line body. Move the assembly to `ApplyState::into_manifest`, so the
entry point reads as the three steps it is: make the state, run the actions,
turn it into a manifest.

`ApplyState` now borrows the manifest it was built from rather than copying
everything it needs out of it, which is also what lets `into_manifest` reach
the read version without being handed it back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`AlterField`, `DropField`, and `TombstoneFieldData` took raw committed field
ids, so none of them could name a field minted earlier in the same operation.
Squashing needs exactly that: collapsing "add column c" and "rename c" into
one operation leaves an `AddField` whose field has no committed id, and a
squash rewrites already-committed transactions, so it cannot re-plan the data
to avoid the reference.

All three now take a `Ref`, matching the fragment actions. Footprints follow
the rule already used there: a local reference records no coordinate, since a
field this operation mints is invisible to a concurrent writer.

`AlterField` loses its `Default` impl -- `Ref` has no meaningful zero, and a
default field reference would silently mean field 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebasing onto the merged transaction module split brings three API changes the
action code predates: `RowIdMeta::Inline` wraps `InlineRowIds` rather than a
byte vector, `DataFile::new`/`new_unstarted` take a `ConcreteFileVersion`
instead of a major/minor pair, and `Operation::Project` carries
`preserves_nullability`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant