Skip to content

[KYUUBI #7708][SERVER] Reconcile zero-count metadata updates - #7709

Open
ruanwenjun wants to merge 1 commit into
apache:masterfrom
ruanwenjun:kyuubi-7708-metadata-update-retry
Open

[KYUUBI #7708][SERVER] Reconcile zero-count metadata updates#7709
ruanwenjun wants to merge 1 commit into
apache:masterfrom
ruanwenjun:kyuubi-7708-metadata-update-retry

Conversation

@ruanwenjun

Copy link
Copy Markdown
Member

Why are the changes needed?

A JDBC metadata UPDATE may report an update count of zero when the row is missing or when the requested values are already present. Kyuubi currently treats every zero count as a retryable failure. A persistent failure therefore remains in the retry queue and can block batch-session timeout cleanup while it waits for metadata retries to finish.

When an UPDATE returns zero, this patch verifies only the updated fields on the same JDBC connection:

  • An exact match is treated as an idempotent success.
  • A missing or mismatched row is retried once through the existing FIFO queue, then discarded if the postcondition still fails. This preserves recovery for an earlier queued INSERT without retrying a permanently missing row forever.
  • Database access failures remain retryable, and the existing close-time retry wait is preserved.

Closes #7708.

How was this patch tested?

  • Added a SQLite trigger-based regression test that forces executeUpdate() to return zero for both matching and mismatching rows.

  • Added retry coverage showing that a persistently missing row is removed from the retry queue.

  • Kept the existing FIFO INSERT-then-UPDATE retry coverage passing.

  • Ran:

    build/mvn test -pl kyuubi-server -am \
      -Pspark-provided,flink-provided,hive-provided \
      -Dtest=none \
      -DwildcardSuites=org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite,org.apache.kyuubi.server.metadata.MetadataManagerSuite

    All 13 tests passed.

Was this patch assisted by generative AI tooling?

Assisted-by: OpenAI Codex with GPT-5

@pan3793

pan3793 commented Sep 3, 2026

Copy link
Copy Markdown
Member

the change lgtm, cc @turboFei, who authored this feature, to take a look

@pan3793 pan3793 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR: the queued InsertMetadata retry path has a similar amplification pattern. A queued insert that hits a duplicate-key error rethrows to the outer catch and stays at the queue head forever - the foreground path fails fast on unrecoverableDBErr, but the retry loop does not apply the same check. Worth a follow-up issue.

}
}

private def verifyMetadataUpdatePostcondition(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #7708 says the original cause of the zero update count is still under investigation, please log a warn here when the row already matches, e.g. "update for {} returned 0 but all requested values already match, treating as idempotent success". Otherwise this case becomes invisible, while the old behavior at least surfaced an exception.

extends MetadataUpdatePostconditionException(
s"Metadata row $identifier was not found after an update returned 0")

private[metadata] class MetadataUpdateMismatchException(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: consider including expected vs actual values in the message, not just column names. When the discard is logged in MetadataManager, that makes the error actionable without reproducing the DB state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Persistent metadata update failures can block batch session timeout cleanup

2 participants