Skip to content

Batch standalone multi-updates by write-set capacity - #500

Draft
Meowooh wants to merge 1 commit into
eloqdata:mainfrom
Meowooh:codex/update-many-write-set-batching
Draft

Meowooh wants to merge 1 commit into
eloqdata:mainfrom
Meowooh:codex/update-many-write-set-batching

Conversation

@Meowooh

@Meowooh Meowooh commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

A standalone updateMany currently fails with TransactionTooLarge when its accumulated document and index writes exceed the 62 MiB transaction write-set limit. This change executes ordinary multi-updates in capacity-bounded transactions. A batch commits when the next complete document plus its index writes cannot fit, or when the scan ends; crossing a cursor prefetch window does not trigger a commit.

The executor owns transaction boundaries, saves and restores scan state, and re-reads pending documents after a transaction change. It confirms candidate IDs and statistics only after a successful commit, and retries only the uncommitted batch after a definite write-conflict abort. Index admission and execution share the same encoded-byte accounting, including multikey, partial unique, and creating-index writes.

Behavior to review

  • A later error rolls back the current batch while preserving earlier committed batches. Error responses retain the existing write-error representation; a failed entry's count is not a partial-progress API.
  • Explicit transactions, retryable writes, and existing outer atomic operations keep the original transaction path. Ordered/unordered update entries and legacy OP_UPDATE use the same batching rules where eligible.
  • Unknown commit outcomes are surfaced without replaying a possibly committed non-idempotent update. A single document plus its index changes that exceeds the hard limit still fails.
  • Cursor restore rejects external schema changes while accepting this operation's own committed multikey metadata.

The change is based directly on upstream main 6959f0c79abf5eda5347f7301bced8c7bff716e2. It uses the existing modifier-update interfaces and adds no aggregation pipeline-update support. Pipeline-dependent regression scenarios have been rewritten with ordinary modifiers while retaining schema and batch-failure coverage.

Dependency

Depends on eloqdata/tx_service#574 for the byte-accounting APIs. The submodule pin contains only that change on top of the current EloqDoc main pin; .gitmodules continues to use the upstream GitHub URL. If the dependency is squash-merged, update the pin to the accepted upstream commit before merging this PR.

Validation

  • Six JS regression scripts pass syntax checks; clang-format 18.1.8 passes for the complete Eloq module; shell syntax and whitespace checks pass.
  • The three adapted regression scripts (capacity/index movement, schema changes, and batch failures) pass against the existing downstream batching runtime. This is a test-adaptation check, not proof of the upstream build.
  • Six new JS scripts are collected by eloq_basic / eloq_core. Two C++ accounting tests are explicitly built and executed by the existing CI build phase, before build-output cleanup.
  • Full upstream-workflow validation is running at https://github.com/Meowooh/eloqdoc/actions/runs/34807034571; exact upstream build and runtime results are pending.

Reviewer focus: transaction ownership and partial completion, retained-ID replay and deduplication, exact document/index byte accounting, and schema lifetime across commits. Reverting this PR restores command-level transaction handling for standalone multi-updates.

@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant