transaction: refuse to resolve shared locks instead of mis-handling them#556
Open
eduralph wants to merge 3 commits into
Open
transaction: refuse to resolve shared locks instead of mis-handling them#556eduralph wants to merge 3 commits into
eduralph wants to merge 3 commits into
Conversation
…revision The vendored protos were an unrecorded late-2023/early-2024 vintage (errorpb and pdpb untouched since tikv#324, one hand-patched addition from tikv#519). Nothing recorded which kvproto revision they came from, and the staleness now blocks real features. Re-vendor everything at pingcap/kvproto b41e86365ce0 — the revision client-go currently pins — and add proto/VERSION so the vintage is never again guesswork. Regenerated with the existing tonic-build pipeline (21 files). One mechanical source fix: pdpb.RequestHeader gained caller_component/caller_id, so the TSO stream's header construction now defaults the new fields. This commit is the re-vendor and nothing else. Newly available surface includes KvFlush / KvBufferBatchGet (pipelined DML), HealthFeedback, and pdpb.BatchScanRegions. Two new protocol semantics become VISIBLE with the refresh — errorpb.UndeterminedResult (an unknown raft apply outcome) and shared locks (SharedLock / SharedPessimisticLock, whose real holders live in shared_lock_infos). Handling them is deliberately left to follow-up PRs so this one stays a pure re-vendor. Neither is a regression introduced here: - UndeterminedResult falls through to the generic region-error arm, which backs off and retries — the same default the follow-up keeps for every plan that is not a commit point. - A shared-lock wrapper carries no key bytes on the wire regardless of which proto vintage parses it, so the keyspace truncation hazard predates this commit; the refresh only makes the wrapper identifiable. Tests: make check green; 67 lib tests; txn/raw/failpoint integration suites green against a local api-v2 cluster. Signed-off-by: Eduard R. <eduard@ralphovi.net>
ti-chi-bot
Bot
added
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (21)
📒 Files selected for processing (42)
📝 WalkthroughWalkthroughThe PR re-vendors kvproto definitions, adds protobuf code-generation options, expands multiple service and message contracts, and updates Rust transaction handling for shared-lock metadata and unset keys. ChangesProtocol foundation
Backup, encryption, and imports
Compute and routing
PD, raft, and transaction APIs
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
contribution
The re-vendored kvproto exposes shared locks (Op::SharedLock / Op::SharedPessimisticLock). Their contract is unusual: a shared lock's real holders live ONLY in kvrpcpb.LockInfo.shared_lock_infos — the proto comment is explicit that you must "DO NOT read from the wrapper LockInfo", whose own key and lock_version are unset. This client does not implement shared-lock resolution, and every partial handling of one is worse than none: resolving the wrapper checks transaction 0; filtering on the wrapper's fields (use_async_commit, lock_type) silently drops the real members; and the pessimistic-lock special cases in the resolver do not know SharedPessimisticLock. So resolution REFUSES them with an explicit error — in resolve_locks, in LockResolver's crate-public entry point, and in CleanupLocks::execute before any filter runs. Until real support lands, an explicit error is the only answer that cannot roll back a live transaction or skip a dead one. Servers that predate shared locks never produce them, so this is a no-op there. The API-v2 keyspace codecs are made shared-lock-aware separately, because they run on scan results that never reach the resolver. They now take the shape of client-go's codecV2.decodeLockInfo (internal/apicodec/codec_v2.go), which decodes a LockInfo's own Key/PrimaryLock/Secondaries and then recurses into SharedLockInfos — it has no wrapper special case. This client now does the same: convert the lock's OWN key fields, then recurse into the members. An earlier revision skipped a wrapper's own fields entirely, reading the proto's "DO NOT read from the wrapper LockInfo" as covering them. Checking the writer (TiKV's SharedLocks::into_lock_info in components/txn_types/src/lock.rs, identical at v8.5.7 and on master) shows that is only half right, and the half it gets wrong matters: info.set_shared_lock_infos(shared_locks.into()); info.set_key(raw_key); A wrapper sets lock_type, shared_lock_infos and key — the key it locks — and leaves primary_lock and lock_version at their defaults; each member is built from that SAME raw key plus its own primary and version. So the caveat scopes the per-transaction fields, not the locked key. Skipping the wrapper wholesale would have handed scan_locks a physical key beside decoded member keys, while converting it wholesale would have hit the length assertion on the unset primary. Both fields are exercised by tests. The two directions guard differently, on purpose. Truncating, empty bytes can only mean "unset" — an encoded key always carries its 4-byte prefix — so unset fields are skipped rather than run into pretruncate_bytes' length assertion. Encoding, the input is a LOGICAL key and the empty logical key is valid in API v2, so nothing is skipped: scan_locks -> resolve_locks round-trips locks through truncate-then-encode, and skipping empties there would strand a lock on the empty key with no prefix and send resolution after an empty physical key. Wrappers need no encode-side guard because resolution refuses them first. That panic hazard predates the re-vendor: a wrapper arrives the same way whichever proto vintage parses it. Full shared-lock support is deliberately follow-up work. Split out of the kvproto re-vendor at pingyu's suggestion (tikv#550). Tests: 4 new — the resolver refusal (a plain lock passes; a wrapper carrying members and a lock marked shared only by its op are both refused); the codec converting a wrapper's own key alongside its members; a wrapper's unset fields surviving truncation (the other reading, so both are pinned); and a lock on the empty logical key round-tripping through truncate-then-encode. 71 lib tests green. The coverage is unit-level by necessity: CI pins TIKV_VERSION v8.5.5, which predates shared locks, so no integration test in this repo can produce one, and the refusal path is unreachable against that server. v8.5.6 is the first release carrying shared locks, so a CI pin bump within the same release family would make these paths reachable — left to a separate change. Signed-off-by: Eduard R. <eduard@ralphovi.net>
eduralph
marked this pull request as ready for review
July 25, 2026 22:12
ti-chi-bot
Bot
removed
the
do-not-merge/work-in-progress
ti-chi-bot
Bot
added
size/L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why: the re-vendored kvproto exposes shared locks (
Op::SharedLock/Op::SharedPessimisticLock). Their contract is unusual: a shared lock's real holders live only inkvrpcpb.LockInfo.shared_lock_infos, and the proto comment is explicit that you must "DO NOT readfrom the wrapper LockInfo" — whose own
lock_versionandprimary_lockare left unset (itskeyis not; see below).
What: this client doesn't implement shared-lock resolution, and every partial handling of one
is worse than none — resolving the wrapper checks transaction 0; filtering on the wrapper's fields
(
use_async_commit,lock_type) silently drops the real members; and the pessimistic-lock specialcases in the resolver don't know
SharedPessimisticLock. So resolution refuses them with anexplicit error: in
resolve_locks, inLockResolver's crate-public entry point, and inCleanupLocks::executebefore any filter runs. Until real support lands, an explicit error is theonly answer that cannot roll back a live transaction or skip a dead one. Servers predating shared
locks never produce them, so this is a no-op there.
The API-v2 keyspace codecs are made shared-lock-aware separately, because they run on scan results
that never reach the resolver. They now take the shape of client-go's
codecV2.decodeLockInfo(
internal/apicodec/codec_v2.go), which decodes aLockInfo's ownKey/PrimaryLock/Secondariesand then recurses intoSharedLockInfos, with no wrapper special case. This clientnow does the same: convert the lock's own key fields, then recurse into the members.
What the wrapper actually contains. An earlier revision skipped a wrapper's own fields entirely,
reading the proto's "DO NOT read from the wrapper LockInfo" as covering them. Checking the writer —
SharedLocks::into_lock_infoin TiKV'scomponents/txn_types/src/lock.rs, identical atv8.5.7and on
master— that's only half right, and the half it gets wrong matters:A wrapper sets
lock_type,shared_lock_infosandkey— the key it locks — and leavesprimary_lock/lock_versionat their defaults. Each member is built from that same raw key plusits own primary and version. So the caveat scopes the per-transaction fields, not the locked key.
Skipping the wrapper wholesale would have handed
scan_locksa physical key beside decoded memberkeys; converting it wholesale would have hit the length assertion on the unset primary. Hence no
wrapper special case, just a per-field guard — both fields are exercised by tests.
I'd still welcome a correction if that reading of the writer is off. It can't be settled by test at
today's pin: CI runs
TIKV_VERSION: v8.5.5, which predates shared locks, so no integration test inthis repo can produce one — the coverage below is unit-level by necessity, and the refusal path is
unreachable against that server. Worth noting for a separate change:
v8.5.6is the first releasecarrying shared locks, so bumping the CI pin within the same release family would make these paths
reachable. Not proposing it here, given this PR is already a split-out.
The two directions guard differently, on purpose:
prefix), so unset fields are skipped rather than run into
pretruncate_bytes' length assertion.nothing is skipped.
scan_locks→resolve_locksround-trips locks through truncate-then-encode,and skipping empties there would strand a lock on the empty key with no prefix, sending resolution
after an empty physical key. Wrappers need no encode-side guard because resolution refuses them
first.
The panic hazard predates the re-vendor: a wrapper arrives the same way whichever proto vintage
parses it.
Full shared-lock support is deliberately follow-up work.
Verification: 4 new tests — the resolver refusal (a plain lock passes; a wrapper carrying members
and a lock marked shared only by its op are both refused); the codec converting a wrapper's own key
alongside its members; a wrapper's unset fields surviving truncation (the other reading, so both are
pinned); and a lock on the empty logical key round-tripping through truncate-then-encode. 71 lib
tests;
make checkgreen; txn/raw/failpoint integration suites green against a local api-v2 cluster;and behaviour compared against client-go by driving both clients through identical scenarios — no
observable change.
Summary by CodeRabbit
New Features
Bug Fixes