Repository navigation
Fix gpactivatestandby after clean shutdown. - #1889
Conversation
8840626 to
03bb859
Compare
A forced activation previously relied on promote_trigger_file before starting the stopped standby, allowing startup to reach catalog access while recovery was still completing. Start the standby with GpStandbyStart and perform the normal pg_ctl promotion before reading the catalog. Share one 600-second monotonic deadline across startup, promotion, and the utility CHECKPOINT retry. Keep activation-specific handling out of gpstart and leave its CLI unchanged.
03bb859 to
2f8b71f
Compare
MisterRaindrop
left a comment
There was a problem hiding this comment.
Verified on a demo cluster (Rocky 8 build image, 3 primaries + 3 mirrors + standby coordinator), built from this branch. Each run starts from a freshly created cluster; gpactivatestandby is run the way the Behave step does it, with PGPORT and COORDINATOR_DATA_DIRECTORY pointing at the standby.
| Scenario | main |
this PR |
|---|---|---|
#1717: recovery_end_command='sleep 5' on the standby, gpstop -a, then gpactivatestandby -a -f |
rc 2, CRITICAL: Error activating standby coordinator; gpstart gets "the database system is in recovery mode" and the standby is left in recovery |
rc 0; new coordinator out of recovery, 2190/2190 rows, gpstate -Q rc 0 |
Existing "gpactivatestandby -f forces standby coordinator to start": coordinator and standby stopped -m immediate, gpactivatestandby -a then -a -f |
without -f: rc 2, as expected; with -f: rc 2, same failure as above |
without -f: rc 2; with -f: rc 0; new coordinator pu at 7001, 2190/2190 rows, gpstate -Q rc 0 |
The 7 tests in test_unit_gpactivatestandby.py pass.
So on main the force path never promotes at all: PostgreSQL 16 stopped honouring promote_trigger_file, and that GUC is the only thing the old path relied on. Starting the standby directly and promoting it with pg_ctl promote, as the non-force path already does, removes the dependency rather than restoring the GUC; #1639 had carried a server-side change that restored it (24faa8f), which this makes unnecessary. After this, nothing uses promote_trigger_file; removing the GUC can be a separate cleanup.
The existing Behave scenario above fails on main today, so #2170 will exclude it until this is in.
The suite comes from apache#1639, where it ran with that PR's fixes. Here it runs without them, and these scenarios fail until they land. Tag them @needs_fix and skip that tag in every job; each fix removes the tag from the scenarios it makes pass, which is also how it shows they do. A comment beside each tag names the fix. analyzedb 1 0977ac5 (identifiers and pg.escape_string) gpactivatestandby 1 apache#1889 (forced activation after a clean shutdown) gpaddmirrors 2 mirror postgresql.conf check; not pinned down yet, among b971d48, a3bfe47 and 8377856 gpcheckcat 11 0bfc1d7 gpinitstandby 2 apache#2172 (pg_basebackup EXCLUDE); without it the new standby promotes itself gpinitsystem 19 b971d48 gpmovemirrors 4 a3bfe47, 51955ac, 7056e02 (one of them a Scenario Outline with four examples) gpstop 2 45a1155, 51955ac (the gpstop lock) minirepro 3 3fccb75 (tables a view reads), 51955ac, apache#2175 (pg_dump --function-oids) The commit hashes are those in apache#1639. gprecoverseg is tagged as a whole: without its fixes a scenario hangs and the job reaches its 120 minute timeout, so the scenarios that do fail cannot be told apart from the ones never reached. It comes back as the gprecoverseg fixes land.
apache#1889 has landed, so the scenario it was tagged for can run.
Fixes #1717.
For forced activation of a stopped standby, start the postmaster directly with
GpStandbyStartinstead of usingpromote_trigger_fileandgpstart. Then promote it withpg_ctl promote, wait for a successful utilityCHECKPOINT, read the catalog, and restart the segments.The existing 600-second
pg_ctl promotetimeout is preserved. A single monotonic deadline covers standby startup, promotion, and the utilityCHECKPOINT, preventing these waits from accumulating while leaving segment restart to its existing timeout handling.