Skip to content

Fix gpactivatestandby after clean shutdown. - #1889

Merged
MisterRaindrop merged 4 commits into
apache:mainfrom
woblerr:fix_gpactivatestandby
Oct 23, 2026
Merged

MisterRaindrop merged 4 commits into
apache:mainfrom
woblerr:fix_gpactivatestandby

Conversation

@woblerr

@woblerr woblerr commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Fixes #1717.

For forced activation of a stopped standby, start the postmaster directly with GpStandbyStart instead of using promote_trigger_file and gpstart. Then promote it with pg_ctl promote, wait for a successful utility CHECKPOINT, read the catalog, and restart the segments.

The existing 600-second pg_ctl promote timeout is preserved. A single monotonic deadline covers standby startup, promotion, and the utility CHECKPOINT, preventing these waits from accumulating while leaving segment restart to its existing timeout handling.

Comment thread gpMgmt/test/behave/mgmt_utils/gpactivatestandby.feature

@leborchuk leborchuk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, LGTM

@woblerr
woblerr force-pushed the fix_gpactivatestandby branch from 8840626 to 03bb859 Compare August 26, 2026 08:29
A forced activation previously relied on promote_trigger_file before
starting the stopped standby, allowing startup to reach catalog access
while recovery was still completing.

Start the standby with GpStandbyStart and perform the normal pg_ctl
promotion before reading the catalog. Share one 600-second monotonic
deadline across startup, promotion, and the utility CHECKPOINT retry.
Keep activation-specific handling out of gpstart and leave its CLI
unchanged.
@tuhaihe
tuhaihe force-pushed the fix_gpactivatestandby branch from 03bb859 to 2f8b71f Compare September 28, 2026 06:43

@MisterRaindrop MisterRaindrop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on a demo cluster (Rocky 8 build image, 3 primaries + 3 mirrors + standby coordinator), built from this branch. Each run starts from a freshly created cluster; gpactivatestandby is run the way the Behave step does it, with PGPORT and COORDINATOR_DATA_DIRECTORY pointing at the standby.

Scenario main this PR
#1717: recovery_end_command='sleep 5' on the standby, gpstop -a, then gpactivatestandby -a -f rc 2, CRITICAL: Error activating standby coordinator; gpstart gets "the database system is in recovery mode" and the standby is left in recovery rc 0; new coordinator out of recovery, 2190/2190 rows, gpstate -Q rc 0
Existing "gpactivatestandby -f forces standby coordinator to start": coordinator and standby stopped -m immediate, gpactivatestandby -a then -a -f without -f: rc 2, as expected; with -f: rc 2, same failure as above without -f: rc 2; with -f: rc 0; new coordinator pu at 7001, 2190/2190 rows, gpstate -Q rc 0

The 7 tests in test_unit_gpactivatestandby.py pass.

So on main the force path never promotes at all: PostgreSQL 16 stopped honouring promote_trigger_file, and that GUC is the only thing the old path relied on. Starting the standby directly and promoting it with pg_ctl promote, as the non-force path already does, removes the dependency rather than restoring the GUC; #1639 had carried a server-side change that restored it (24faa8f), which this makes unnecessary. After this, nothing uses promote_trigger_file; removing the GUC can be a separate cleanup.

The existing Behave scenario above fails on main today, so #2170 will exclude it until this is in.

@MisterRaindrop
MisterRaindrop enabled auto-merge (squash) October 23, 2026 08:16
MisterRaindrop added a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 23, 2026
The suite comes from apache#1639, where it ran with that PR's fixes. Here it
runs without them, and these scenarios fail until they land.  Tag them
@needs_fix and skip that tag in every job; each fix removes the tag from
the scenarios it makes pass, which is also how it shows they do.  A
comment beside each tag names the fix.

  analyzedb      1  0977ac5 (identifiers and pg.escape_string)
  gpactivatestandby
                 1  apache#1889 (forced activation after a clean shutdown)
  gpaddmirrors   2  mirror postgresql.conf check; not pinned down yet,
                    among b971d48, a3bfe47 and 8377856
  gpcheckcat    11  0bfc1d7
  gpinitstandby  2  apache#2172 (pg_basebackup EXCLUDE); without it the new
                    standby promotes itself
  gpinitsystem  19  b971d48
  gpmovemirrors  4  a3bfe47, 51955ac, 7056e02 (one of them a
                    Scenario Outline with four examples)
  gpstop         2  45a1155, 51955ac (the gpstop lock)
  minirepro      3  3fccb75 (tables a view reads), 51955ac,
                    apache#2175 (pg_dump --function-oids)

The commit hashes are those in apache#1639.  gprecoverseg is tagged as a
whole: without its fixes a scenario hangs and the job reaches its
120 minute timeout, so the scenarios that do fail cannot be told apart
from the ones never reached.  It comes back as the gprecoverseg fixes
land.
@MisterRaindrop
MisterRaindrop merged commit d694984 into apache:main Oct 23, 2026
197 of 200 checks passed
@woblerr
woblerr deleted the fix_gpactivatestandby branch October 24, 2026 08:05
MisterRaindrop added a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 24, 2026
apache#1889 has landed, so the scenario it was tagged for can run.
@MisterRaindrop MisterRaindrop mentioned this pull request Oct 24, 2026
2 of 6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted

4 participants