Skip to content

Skip ALTER ROLE when the stored SCRAM verifier already matches the password - #3171

Open
g2px1 wants to merge 2 commits into
zalando:masterfrom
g2px1:fix/scram-password-sync-churn
Open

Skip ALTER ROLE when the stored SCRAM verifier already matches the password#3171
g2px1 wants to merge 2 commits into
zalando:masterfrom
g2px1:fix/scram-password-sync-churn

Conversation

@g2px1

@g2px1 g2px1 commented Aug 13, 2026

Copy link
Copy Markdown

Fixes #3170.

With password_encryption: scram-sha-256, role sync compared the stored rolpassword with a freshly generated hash by string equality. SCRAM-SHA-256 verifiers embed a random salt, so the comparison never matched and every sync cycle re-issued ALTER ROLE ... PASSWORD for every managed role, re-salting the verifier each time. Besides WAL/audit noise, each re-salt invalidates SCRAM pass-through credentials cached by connection poolers (pgbouncer behind auth_query, including the operator-deployed pooler), producing a short window of password authentication failed server logins after every sync.

Changes

  • util.PGUserPasswordUpToDate(user, storedPassword, encryption): verifies the stored hash against the desired plaintext instead of comparing strings. For SCRAM verifiers the salt and iteration count are parsed from the stored value (SCRAM-SHA-256$<iter>:<salt>$<storedKey>:<serverKey>) and the derived server/stored keys are compared with hmac.Equal.
  • Behavior is preserved everywhere else:
    • a stored hash whose type does not match the configured password_encryption is reported as outdated, so switching between md5 and scram-sha-256 still re-hashes roles;
    • empty and pre-hashed desired passwords are compared verbatim, mirroring the early return in PGUserPassword;
    • malformed stored hashes are treated as outdated.
  • users.go uses the new check in produceSyncRequests; the ALTER statement itself is generated exactly as before.
  • Extracted isMD5Hash/isScramHash helpers from PGUserPassword (no behavior change) and added a table test covering match/mismatch, both encryptions, migration between them, pre-hashed and malformed inputs.

After the fix, a role whose verifier already corresponds to the secret is left untouched; pg_authid.rolpassword stays stable across sync cycles and pooler SCRAM pass-through keeps working.

…ssword

With password_encryption = scram-sha-256, syncSecrets compared the stored
rolpassword with a freshly generated verifier. SCRAM verifiers embed a
random salt, so the strings never match and every sync cycle re-issued
ALTER ROLE ... PASSWORD for every managed role, re-salting the verifier
each time. Besides the WAL and audit noise, this invalidates SCRAM
pass-through credentials cached by connection poolers (e.g. pgbouncer
behind auth_query), causing a short window of 'password authentication
failed' server logins after every sync.

Verify the stored hash against the desired password instead: for SCRAM
verifiers the salt and iteration count are taken from the stored value
and the derived keys are compared. Hashes whose type does not match the
configured password_encryption are still reported as outdated, so
switching between md5 and scram-sha-256 keeps re-hashing roles as
before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@FxKu FxKu added the bugfix label Aug 14, 2026
@FxKu FxKu added this to the 2.0.2 milestone Aug 14, 2026
Comment thread pkg/util/util.go
func (e *Encryptor) PGUserPassword(user spec.PgUser) string {
if (len(user.Password) == md5.Size*2+len(md5prefix) && user.Password[:3] == md5prefix) ||
(len(user.Password) > len(scramsha256prefix) && user.Password[:len(scramsha256prefix)] == scramsha256prefix) || user.Password == "" {
if isMD5Hash(user.Password) || isScramHash(user.Password) || user.Password == "" {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be a function because you also repeat it in PGUserPasswordUpToDate

Comment thread pkg/util/util.go
return false
}
return scramVerifierMatches(user.Password, storedPassword)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it's me but I find this switch case kinda hard to read

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh, I'm C++ programmer and I suddenly find that bug in production. So it might be not the best code you've seen

Comment thread pkg/util/util.go
Comment on lines +213 to +218
mac := hmac.New(sha256.New, key)
mac.Write([]byte("Server Key"))
derivedServerKey := mac.Sum(nil)
mac = hmac.New(sha256.New, key)
mac.Write([]byte("Client Key"))
derivedStoredKey := sha256.Sum256(mac.Sum(nil))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mac := hmac.New(sha256.New, key)
mac.Write([]byte("Server Key"))
derivedServerKey := mac.Sum(nil)
mac = hmac.New(sha256.New, key)
mac.Write([]byte("Client Key"))
derivedStoredKey := sha256.Sum256(mac.Sum(nil))
serverMAC := hmac.New(sha256.New, key)
serverMAC.Write([]byte("Server Key"))
derivedServerKey := serverMAC.Sum(nil)
clientMAC := hmac.New(sha256.New, key)
clientMAC.Write([]byte("Client Key"))
derivedStoredKey := sha256.Sum256(clientMAC.Sum(nil))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Role sync issues ALTER ROLE ... PASSWORD on every sync cycle when password_encryption is scram-sha-256

4 participants