Skip to content

HDDS-15888. Stamp lastTransactionInfo on snapshot deletion to avoid premature GC#10788

Draft
smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-15888-stamp
Draft

HDDS-15888. Stamp lastTransactionInfo on snapshot deletion to avoid premature GC#10788
smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-15888-stamp

Conversation

@smengcl

@smengcl smengcl commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Unlike create/moveTableKeys/purge, OMSnapshotDeleteRequest never stamps lastTransactionInfo. areSnapshotChangesFlushedToDB() therefore reads the stale create-time stamp and reports an unflushed deletion as flushed, so SnapshotDeletingService does not defer it and can purge the snapshot while the delete and purge exist only in the double buffer. The purge empties the in-memory chain, ReclaimableFilter finds no previous snapshot (areSnapshotChangesFlushedToDB(null) is vacuously true), and KeyDeletingService hands blocks to SCM irreversibly while the on-disk snapshotInfoTable still shows the snapshot ACTIVE and referencing them.

Ratis log                 applied (cache + chain)          RocksDB (flushed marker)
idx 1: CreateSnapshot s1  s1 ACTIVE, lastTxnInfo = idx 1
idx 2: DeleteKey k        tombstone k in AOS deletedTable  <flushed through idx 2>
idx 3: DeleteSnapshot s1  s1 DELETED, lastTxnInfo NOT stamped (still idx 1)
   SDS gate: lastTxnInfo idx 1 <= flushed idx 2 => "flushed" (WRONG, delete is not durable);
   with the stamp: idx 3 > idx 2 => defer. Instead:
idx 4-5: moveTableKeys + purge s1   in-memory chain now empty  (idx 3-5 unflushed)
   KDS: no previous snapshot => k reclaimable => blocks deleted via SCM (irreversible)
   On-disk state: s1 ACTIVE, k captured in s1, k's blocks gone.

Severity

Low. The deletion was user-requested and Ratis-acked, DELETED snapshots cannot be unmarked, and crash recovery replays idx 3-5, so no supported flow reads the lost blocks. The inconsistency surfaces only via recovery that drops acked transactions (RocksDB restored without the matching Ratis log). Still worth fixing because both existing flush gates exist precisely to keep irreversible block deletion behind locally durable state, and the missing stamp silently defeats them. Found by model-checking the snapshot chain with TLA+, then reproduced in unit tests.

Fix

Stamp lastTransactionInfo in OMSnapshotDeleteRequest.validateAndUpdateCache, matching the other snapshot requests. The existing SDS gate then defers move/purge until the deletion is durable, so a chain-removed snapshot is at worst DELETED on disk, never ACTIVE.

Cost: snapshot GC waits at most one double-buffer flush cycle.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15888

How was this patch tested?

  • New TestOMSnapshotDeleteRequest#testSnapshotDeleteIsNotReportedFlushedUntilFlushed: fails without the fix, passes with it.
  • New TestReclaimableKeyFilter boundary tests: empty-chain reclamation is only legal with the on-disk row absent or DELETED; javadoc records the invariant.
  • Tests are Generated-by: Claude Code (Fable 5)

smengcl added 2 commits July 16, 2026 23:05
Generated-by: Claude Code (Fable 5)
@smengcl smengcl added the bug Something isn't working label Jul 17, 2026
Copilot AI review requested due to automatic review settings July 17, 2026 06:13
@smengcl smengcl added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jul 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a snapshot-GC durability gap in Ozone Manager by stamping lastTransactionInfo during snapshot deletion, ensuring flush-gating logic (areSnapshotChangesFlushedToDB) correctly defers snapshot GC until the delete is durably flushed.

Changes:

  • Stamp SnapshotInfo.lastTransactionInfo in OMSnapshotDeleteRequest.validateAndUpdateCache on deletion.
  • Add a regression test ensuring snapshot deletion is not reported “flushed” until the transaction marker advances.
  • Add boundary tests documenting/covering reclaimability behavior when the in-memory snapshot chain is empty.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotDeleteRequest.java Stamp deletion lastTransactionInfo so flush gates correctly defer snapshot GC.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotDeleteRequest.java Adds regression coverage for the flush-marker check around snapshot deletion.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableKeyFilter.java Adds boundary tests around reclaimability when the snapshot chain appears empty.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +284 to +292
/**
* Repro for the flush-lag reclamation window (companion to
* TestReclaimableKeyFilter#testKeyNotReclaimableWhenChainEmptyingPurgeIsUnflushed): snapshot deletion
* updates only status and deletionTime and never stamps lastTransactionInfo (unlike create, moveTableKeys,
* purge and key/dir purge), so areSnapshotChangesFlushedToDB() keys off the stale create-time stamp and
* reports an applied-but-unflushed deletion as flushed. SnapshotDeletingService#shouldIgnoreSnapshot relies
* on that method to defer processing until a snapshot's latest change is durable; the missing stamp lets
* the deletion be processed (moveTableKeys + purge submitted) before the double buffer has flushed it.
*/
Comment on lines +325 to +327
assertFalse(OmSnapshotManager.areSnapshotChangesFlushedToDB(getOmMetadataManager(), key),
"snapshot deletion applied at index 2 is unflushed (marker at the create transaction) but is"
+ " reported flushed because OMSnapshotDeleteRequest does not stamp lastTransactionInfo");
Comment on lines +344 to +345
// ...but the purge is not flushed: the applied/cache view of snapshotInfoTable has no row while the
// on-disk (skip-cache) view still holds the row -- as DELETED, guaranteed by the SDS flush gate.
@adoroszlai adoroszlai changed the title HDDS-15888. Stamp lastTransactionInfo on snapshot deletion so snapshot GC waits for the deletion to be flushed HDDS-15888. Stamp lastTransactionInfo on snapshot deletion to avoid premature GC Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants