feat(operator): persistent read-only users recreated on every restore - #125
Open
julianam-w wants to merge 5 commits into
Open
feat(operator): persistent read-only users recreated on every restore#125julianam-w wants to merge 5 commits into
julianam-w wants to merge 5 commits into
Conversation
Each restore is a fresh cluster built from the snapshot, so a role created on a previous restore is gone after switchover and any role that did survive in pg_authid has its password cleared during init. Only analyticsUsername was rebuilt each cycle. Add persistentUsers: a list of read-only login roles the operator recreates on each switchover, with per-user Secrets whose passwords stay stable for the life of the replica so downstream consumers keep working across restores. Provisioning runs after the persistentSchemas migration Job and before the new restore is labelled ready for traffic: the schemas these roles read from are written by that Job, so granting any earlier would apply to nothing, and the window has no external clients connected. Default privileges name the schema owner read from pg_namespace via ALTER DEFAULT PRIVILEGES FOR ROLE, rather than relying on SET ROLE, which silently covers nothing when run as anyone but the owner. Schemas absent from a restore are skipped with a PersistentUserSchemaMissing Warning event instead of blocking the switchover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The <replica>- prefix exists because Secrets are namespaced: two replicas in one namespace resolving to the same Secret would share a password, and the ownerReference means deleting either one cascades the Secret away from the other. With a single replica per namespace that is unreachable and the prefix is just noise in consumer manifests. Add secretName to opt out, keeping the scoped default. The operator refuses to adopt a Secret whose pgro.bes.au/replica label names a different replica, so a collision surfaces as InvalidSpec instead of two replicas silently sharing a credential. Validation now dedupes on resolved secret names rather than role names, so an override colliding with another user default is caught too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
readOnly is the default for a replica, and the restore init script writes default_transaction_read_only = on into postgresql.conf. Provisioning issues CREATE ROLE and GRANT on every replica, so on a read-only one every statement failed with "cannot execute ... in a read-only transaction" and the propagated error blocked switchover outright. The restore is a promoted standalone rather than a standby, so the setting is a plain USERSET GUC the operator session can turn off for itself; the replica stays read-only for every other client. Extract session_statements so the ordering is testable without a database, and add a read-only integration case, which is the configuration the existing two did not cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The canopy worklist syncer builds a replica spec from scratch, so it has to name the new field. A worklist entry has no notion of downstream consumer roles, so canopy-managed replicas get an empty list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
julianam-w
force-pushed
the
feat/persistent-users
branch
from
August 25, 2026 14:15
12c9f53 to
f075b09
Compare
… users readOnly defaults to true, and on PG >= 14 the init script then grants the analytics role only pg_read_all_data. That role cannot CREATE ROLE, so provisioning died on "permission denied to create role" and, because the error propagates, the switchover never completed. Redaction made it worse by demoting the role to NOSUPERUSER before provisioning ran. Neither has anything to do with the transaction default the earlier fix addressed: no session GUC rescues a missing privilege. Two of the five readOnly/persistentSchemas/redaction combinations were broken, including the plain one with no redaction anywhere in sight. The documented Tupaia config pairs persistentUsers with persistentSchemas, which is why it worked. Separate the database read-only posture from the analytics role privilege: the restore stays read-only for clients while the role keeps the rights the operator needs, and the operator demotes it once provisioning is done. Redaction defers its own demotion to that same tail step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
persistentUsers: read-only login roles the operator recreates on every restore, with per-user Secrets whose passwords stay stable for the life of the replica.Why
Every restore is a fresh cluster built from the snapshot, so a role created by hand on a previous restore is gone after switchover, and any role that did survive in
pg_authidhas its password cleared during init. OnlyanalyticsUsernamewas rebuilt each cycle, so a read-only consumer role had to be recreated manually after every restore.Shape
Notes for review
persistentSchemasmigration Job and before the new restore is labelled ready for traffic. The schemas these roles read from are written by that Job, so granting any earlier would apply to nothing, and the window has no external clients connected.pg_namespaceviaALTER DEFAULT PRIVILEGES FOR ROLE, rather thanSET ROLE. WithoutFOR ROLEthe statement applies to the connected role and silently covers nothing.readOnlydefaults to true, which putsdefault_transaction_read_only = oninpostgresql.confand fails everyCREATE ROLE/GRANT. The provisioning session clears that GUC for itself only; the replica stays read-only for everyone else.<replica>-prefix exists because Secrets are namespaced — two replicas sharing one Secret would share a password, and theownerReferencemeans deleting either cascades it away from the other.secretNameopts out; the operator refuses to adopt a Secret whosepgro.bes.au/replicalabel names a different replica.PersistentUserSchemaMissingWarning event rather than failing the switchover. Other provisioning failures do block switchover — worth a second opinion on that trade-off.Testing
163 unit tests pass locally, covering the SQL builder: role attributes, identifier/literal quoting,
FOR ROLEon default privileges, skip-on-missing-schema, and that the read-only GUC precedes all DDL.The provisioning path has never executed against a live Postgres — this PR is its first real run. Three integration cases added (
test-persistent-users,test-pu-missing-schema,test-pu-read-only) with a matrix entry inintegration.yml.🤖 Generated with Claude Code