Skip to content

[flink] Support remove_orphan_blobs action and procedure. - #10014

Open
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:support-clean-orphan-blob-flink
Open

Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:support-clean-orphan-blob-flink

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

Add Flink remove_orphan_blobs (action + procedure) for unreferenced primary-key .managed.blob packs, stacked on the landed core cleaner and Spark procedure.

  • local reuses LocalManagedBlobOrphanFilesClean
  • distributed freezes the validated used-pack set with executeAndCollect before listing candidates, then filters with a captured HashSet. Deletion does not reconnect the sidecar/mark DAG, so a lost used-pack shuffle cannot drop a live pack
  • Abort if the used-pack set changes (SKIP_MANAGED_BLOB_GC)
  • database.* runs tables one after another so one table’s freeze cannot execute another table’s uncommitted deletion graph
  • Bounded used/candidate joins keep ALL_EXCHANGES_BLOCKING
  • Docs: docs/docs/flink/procedures/repair.md, procedure index, and blob-storage.md

Tests

  • RemoveOrphanBlobsActionITCase / RemoveOrphanBlobsActionITCaseBase
    • testDeleteUnreferencedManagedBlobPack
    • testMissingManagedBlobSidecarSkipsPackGc
    • testDistributedDeleteWithEmptyUsedPackSet
    • testDistributedUnresolvableRelativeCandidateSkipsGc
    • testDistributedMarkReadsSharedMetadataOncePerPass
    • testDistributedCleanupDoesNotHangWithPipelinedShuffle
    • testDistributedDeletionAccountingSurvivesCleanupTaskRestartAfterDelete
    • testDistributedCanonicalAliasCandidatesAreCountedOnce
    • testActionFactoryDryRunParsing
    • testActionFactoryRejectsInvalidDryRun
    • testActionFactoryRejectsNonPositiveParallelism
    • testProcedureRejectsNonPositiveParallelism
    • testFrozenUsedMarkIsNotRecomputedDuringDeletion
    • testUsedPackSetChangeSkipsGc
    • testRemoveDatabaseOrphanBlobs
    • testProcedureDryRun
  • RemoveOrphanFilesActionITCaseBase
    • testDistributedCleanupForcesBlockingShuffle
  • Flink 1.18 / 1.19 subclasses of RemoveOrphanBlobsActionITCase

Expose managed blob pack GC on Flink and freeze the used-pack set before deletion so a lost mark shuffle cannot drop a live pack.
@Stephen0421
Stephen0421 force-pushed the support-clean-orphan-blob-flink branch from 8702823 to f695bb7 Compare September 20, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant