feat: add CommitStrategy enum with Pessimistic and Optimistic modes

[Jay-ju]

Add CommitStrategy: Optimistic vs Pessimistic Conflict Resolution

Summary

This PR introduces a CommitStrategy enum that allows users to choose between Pessimistic (default, current behavior) and Optimistic commit strategies for concurrent write conflict resolution.

Background

Currently, Lance always performs a "rebase" before every commit attempt — loading new transactions since the last read and running TransactionRebase::try_new() to reconcile conflicts. This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

Changes

Core Logic (rust/lance/src/io/commit.rs)

The commit loop now conditionally skips rebase based on the configured strategy:

• Pessimistic (default): Always rebase before commit. This is the existing behavior — safe for all scenarios but incurs I/O on every attempt.
• Optimistic: Attempt to commit first without rebase. Only rebase on conflict (i.e., when backoff.attempt() > 0). This skips the rebase I/O on the fast path (no conflict), and even when a conflict occurs, the cost of a failed commit attempt is typically lower than the rebase I/O overhead.

let needs_rebase = !strict_overwrite
    && match commit_config.commit_strategy {
        CommitStrategy::Pessimistic => true,
        CommitStrategy::Optimistic => backoff.attempt() > 0,
    };

New Types (rust/lance-table/src/io/commit.rs)

• CommitStrategy enum with Pessimistic (default) and Optimistic variants
• CommitConfig.commit_strategy field

Public API (rust/lance/src/dataset/write/commit.rs)

• CommitBuilder::with_commit_strategy(strategy) — allows users to configure the commit strategy

Benchmark (python/python/ci_benchmarks/benchmarks/test_concurrent_write.py)

Added a pytest-benchmark based concurrent write benchmark that measures throughput and latency under different concurrency levels and operation mixes (append, delete, update).

---

Benchmark Test Report

How to Reproduce

# 1. Build with Release mode
cd python && maturin develop --release

# 2. Run with Pessimistic strategy (default)
pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only --benchmark-json pessimistic.json

# 3. Run with Optimistic strategy
LANCE_COMMIT_STRATEGY=optimistic \
    pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only --benchmark-json optimistic.json

# 4. (Optional) Run against S3/TOS remote storage
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_ENDPOINT=https://your-endpoint
export AWS_REGION=your-region
export LANCE_BENCH_DATASET_URI=s3://bucket/path
pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only

Test Environment

• Build: maturin develop --release (Release mode)
• Storage: TOS (S3-compatible, remote object store)
• Concurrency: 20 appenders + 10 deleters + 10 updaters (mixed), 30 appenders + 15 deleters + 15 updaters (high concurrency)

Results — Mixed Workload (20 appenders, 10 deleters, 10 updaters)

Metric	Pessimistic	Optimistic	Delta
Total time	27.1s	24.6s	-9.2%
Throughput	2.6 ops/s	2.9 ops/s	+12.5%
Delete avg latency	1.34s	0.94s	-29.9%
Update avg latency	1.22s	0.97s	-20.5%
Append avg latency	0.68s	0.63s	-7.4%

Results — High Concurrency (30 appenders, 15 deleters, 15 updaters)

Metric	Pessimistic	Optimistic	Delta
Total time	42.3s	38.1s	-9.9%
Throughput	2.1 ops/s	2.4 ops/s	+14.3%
Delete avg latency	2.01s	1.38s	-31.3%
Update avg latency	1.85s	1.42s	-23.2%
Append avg latency	0.95s	0.87s	-8.4%

Key Findings

1. Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels.
2. Delete and Update benefit the most (~20-30% latency reduction) because these operations trigger TransactionRebase::try_new() which loads initial_fragments — an expensive I/O operation that Optimistic skips on the fast path.
3. Append also benefits (~7-8% improvement) from skipping the load_and_sort_new_transactions call.
4. Higher concurrency amplifies the gain — the Optimistic advantage grows as contention increases.

---

Files Changed

File	Change
rust/lance-table/src/io/commit.rs	Add CommitStrategy enum and commit_strategy field in CommitConfig
rust/lance/src/io/commit.rs	Conditional rebase logic based on CommitStrategy
rust/lance/src/dataset/write/commit.rs	Add CommitBuilder::with_commit_strategy() API
python/python/ci_benchmarks/benchmarks/test_concurrent_write.py	Concurrent write benchmark using pytest-benchmark

Backward Compatibility

• Fully backward compatible: Default strategy is Pessimistic, which preserves the existing behavior.
• No changes to public API signatures; only an additive new method with_commit_strategy().

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 78.84615% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/cleanup.rs	69.23%	6 Missing and 2 partials ⚠️
rust/lance/src/io/commit.rs	81.25%	0 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[wjones127]

This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

This is a helpful finding. I originally wrote the initial behavior here: #3117

Back then, I would have thought the pessimistic strategy would be better for high contention, at the cost of being slower when there is low contention. But looks like empirically I was wrong. Thanks for running this benchmark.

TBH, I'm not sure we need to add a setting for this. I think it would be totally acceptable just change the default to optimistic, especially given "Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels." If the results were mixed, I could see the case for having a setting. But if it's better in all cases, I think it's fine to change the behavior. Plus this isn't a behavior that users rely upon, it's most of an optimization. So it's not a breaking change in that sense.

What do you think of just making Optimistic the default?

[Jay-ju]

This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

This is a helpful finding. I originally wrote the initial behavior here: #3117

Back then, I would have thought the pessimistic strategy would be better for high contention, at the cost of being slower when there is low contention. But looks like empirically I was wrong. Thanks for running this benchmark.

TBH, I'm not sure we need to add a setting for this. I think it would be totally acceptable just change the default to optimistic, especially given "Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels." If the results were mixed, I could see the case for having a setting. But if it's better in all cases, I think it's fine to change the behavior. Plus this isn't a behavior that users rely upon, it's most of an optimization. So it's not a breaking change in that sense.

What do you think of just making Optimistic the default?

@wjones127 Thanks for the context! I agree — since Optimistic outperforms across all scenarios, making it the default is the right call. I have updated the PR already.

For context, the motivation came from LLM agent workflows where LanceDB serves as the memory module: each conversation turn triggers an individual write, creating high write concurrency that made the rebase overhead very visible.

One question while on this topic — under extreme write concurrency (e.g. many agents writing simultaneously), even the optimistic strategy still faces repeated conflict retries. I've been thinking about whether a WAL/buffer layer on top of Lance could help absorb writes and batch-flush them, similar to LSM-tree approaches. Do you have any thoughts on whether this is a direction worth exploring, or if there are other approaches you'd recommend for high-concurrency scenarios?

[wjones127]

One question while on this topic — under extreme write concurrency (e.g. many agents writing simultaneously), even the optimistic strategy still faces repeated conflict retries. I've been thinking about whether a WAL/buffer layer on top of Lance could help absorb writes and batch-flush them, similar to LSM-tree approaches. Do you have any thoughts on whether this is a direction worth exploring, or if there are other approaches you'd recommend for high-concurrency scenarios?

Yeah, that's essentially the work that's going on in #3985

[wjones127]

since Optimistic outperforms across all scenarios, making it the default is the right call. I have updated the PR already.

Actually, what do you think about going a step further, and just eliminating the option? Just change the behavior directly to use optimistic always.

[wjones127]

issue(blocking): this seems like a drive-by bug fix. Could we save this for a different PR and make sure we have a issue describing it?

[wjones127]

issue(blocking): this also seems like a drive-by change that could use it's own issue.

[Jay-ju]

Thanks for the thorough review! Let me address the two blocking items and the suggestion to remove the CommitStrategy option.

Regarding the two "drive-by fixes":

These are not drive-by fixes — they are direct consequences of the Optimistic strategy and are required for correctness. When the first commit attempt skips rebase, the manifest inherits stale writer_version and config from an outdated snapshot via Manifest::new_from_previous. Both fixes address correctness issues that only manifest under Optimistic commit:

1. disk_indices check: Without rebase, must_recalculate_fragment_bitmap() returns true due to stale writer_version, causing migrate_indices to attempt opening an index with a remapped uuid that doesn't exist on disk yet. This was caught by test_v0_8_14_invalid_index_fragment_bitmap failing under Optimistic mode.

2. auto_cleanup_hook validation: Without rebase, the committed manifest can inherit stale auto_cleanup config from an outdated snapshot, triggering cleanup that should have been disabled by a concurrent commit. This was caught by test_enable_disable_auto_cleanup failing under Optimistic mode.

Both fixes are prerequisites — without them, the Optimistic strategy breaks existing tests. That said, I'm happy to create separate issues to document these edge cases if you'd like, while keeping the fixes in this PR.

[Jay-ju]

Regarding removing the CommitStrategy option:

I agree — since Optimistic outperforms Pessimistic across all scenarios, there's no practical reason to keep the option.

However, in my local tests, pessimistic locks deliver better performance during the first execution of preheating some S3 links, while optimistic locks generally yield superior results after preheating is completed.

I'll remove the CommitStrategy enum entirely and inline the Optimistic logic directly.

[wjones127]

I'm still not sure of these changes. It seems like we are changing more code than we should.

Also, make sure to update the PR description to reflect the nature of the changes now.

[wjones127]

issue(blocking): It doesn't make sense to me why this should depend on operation. If it's safe to skip rebase for some operations, but not others, that seems a like a dangerous API that will have lots of bugs. I'd rather we get it so there's no need to gate on operation at all, and just make let needs_rebase = backoff.attempt() > 0.

[wjones127]

suggestion: could we replace this comment with one that gives a justification for the optimistic by default behavior. It could read similar to the old one, but say that benchmarks showed that optimistic was better in all cases. Any maybe explain a little why.