Skip to content

Build/Test Tools: Wait for the database before running WP-CLI commands. - #12734

Open
adimoldovan wants to merge 4 commits into
WordPress:trunkfrom
adimoldovan:fix-db-readiness-race
Open

Build/Test Tools: Wait for the database before running WP-CLI commands.#12734
adimoldovan wants to merge 4 commits into
WordPress:trunkfrom
adimoldovan:fix-db-readiness-race

Conversation

@adimoldovan

@adimoldovan adimoldovan commented Jul 28, 2026

Copy link
Copy Markdown

Trac ticket: https://core.trac.wordpress.org/ticket/65742

npm run env:install intermittently fails on a cold database volume:

Error: Database connection error (2002) Connection refused
    at wp_cli (tools/local-env/scripts/install.js:62)

The script's first action, wp config create, connects to the database. Its only readiness wait runs six WP-CLI calls later and waits on the web port, never the database. Nothing retries.

cli does depend on mysql being service_healthy, but that healthcheck pings over the unix socket, which the entrypoint's temporary server answers while a cold volume is still initialising — before the real server listens on TCP.

This retries config create until it succeeds. Its own connectivity check exercises the path that matters, cli to mysql over TCP, so retrying it is the readiness probe.

Happy path is unchanged and adds no time. Forgetting npm run env:start still fails immediately with the existing hint. A database that never comes up fails after 2 minutes with the underlying error.

Testing Instructions

Only reproduces on a cold volume. npm run env:start hides it, because composer update takes long enough to warm the database:

docker compose down -v
docker compose up -d wordpress-develop cli
node ./tools/local-env/scripts/install.js

Timing-dependent, so repeat it. To force the window open, add a docker-compose.override.yml:

services:
  mysql:
    healthcheck:
      test: [ 'CMD-SHELL', 'true' ]
      interval: 1s
      retries: 1

With that, trunk fails every run and this branch passes every run. Also check npm run env:start && npm run env:install still works and is no slower.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5
Used for: Investigating the root cause, writing the change, and building the reproduction described above. The diagnosis was verified against container logs and probe timings rather than assumed. I reviewed the implementation and the test results and take responsibility for them.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

`npm run env:install` began with `wp config create`, which performs a database
connectivity check. The script's only readiness wait ran six WP-CLI calls later,
and waited on the web port rather than the database, so nothing actually
confirmed the database was reachable before it was used.

The `cli` service does depend on `mysql` being `service_healthy`, but that
healthcheck runs `mysqladmin ping -h localhost` inside the mysql container,
which connects over the unix socket. While a cold volume is initialising, that
socket is served by the entrypoint's temporary server, which listens on socket
only (`port: 0`). The healthcheck therefore reports healthy while the real
server is not yet listening on TCP, and connections from the `cli` container are
refused.

Retry the `config create` call until it succeeds. Its built-in connectivity
check exercises exactly the path that matters, cli container to host `mysql`
over TCP, so retrying it doubles as the readiness probe without adding a
separate wait or relying on a published host port.

Failures that retrying cannot fix, such as the environment never having been
started, still exit immediately, and the timeout reports the underlying error.
Copilot AI review requested due to automatic review settings July 28, 2026 11:03
@github-actions

Copy link
Copy Markdown

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the local Docker environment installer (npm run env:install) against intermittent cold-start database readiness races by retrying the initial WP-CLI config create step until the MySQL service is actually accepting TCP connections.

Changes:

  • Wraps wp config create in a new wp_cli_retry() helper to retry DB connectivity checks for up to 120s on cold volumes/slow CI.
  • Updates wp_cli() to support configurable stdio handling so retries can suppress noisy output while still preserving success output.
  • Adds timeout failure messaging with a pointer to MySQL container logs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tools/local-env/scripts/install.js
@adimoldovan adimoldovan self-assigned this Jul 28, 2026
@adimoldovan
adimoldovan marked this pull request as ready for review July 28, 2026 12:14
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props adrianmoldovanwp, lancewillett.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tools/local-env/scripts/install.js:101

  • err.stderr is treated as the sole source of output, but it's truthy even when it's an empty Buffer. In that case output becomes an empty string and you lose the actual failure details in err.message (and sometimes err.stdout), which can also cause the is not running detection to be missed and contradicts the goal of surfacing the underlying error after the timeout.
			const output = err.stderr ? err.stderr.toString() : err.message;

			// Retrying only helps while the environment is still starting up. A missing container
			// means it was never started, so there is nothing to wait for.
			if ( output.includes( 'is not running' ) ) {

@lancewillett lancewillett left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cold-database retry worked and all 197 CI checks passed or skipped.

But an empty stderr buffer discards available error details, potentially producing a silent two-minute retry.


Testing with head `8a369601bcb0921568a7ae2e030aac0f027540aa

The forced cold-database path waited and successfully generated wp-config.php; the missing-environment path also retained its fast failure.

The one blocker is in tools/local-env/scripts/install.js: err.stderr is a truthy Buffer even when empty. The current expression can therefore discard stdout and err.message, retry for two > minutes, then report no underlying error.

Could we select the first non-empty value from stderr, stdout, and err.message before checking and reporting the failure?

`err.stderr` is a Buffer, which is truthy even when empty, so an error with no
stderr output resolved to an empty string rather than falling back to
`err.message`. That discarded the underlying error at the end of the retry
window, and also defeated the `is not running` check, so a missing container
could retry for the full timeout and then report nothing.

Use the first of stderr, stdout, and `err.message` that actually captured
something.
Copilot AI review requested due to automatic review settings July 29, 2026 09:28
@adimoldovan

adimoldovan commented Jul 29, 2026

Copy link
Copy Markdown
Author

The one blocker is in tools/local-env/scripts/install.js: err.stderr is a truthy Buffer even when empty. The current expression can therefore discard stdout and err.message, retry for two > minutes, then report no underlying error.

Could we select the first non-empty value from stderr, stdout, and err.message before checking and reporting the failure?

Good catch. Fixed with 921d2a9 - now takes the first non-empty of stderr, stdout, err.message, with a fallback string.

Note: current failure in this run should be addressed by #12735

@adimoldovan
adimoldovan requested a review from lancewillett July 29, 2026 09:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tools/local-env/scripts/install.js:120

  • wp_cli_retry() retries on any WP-CLI failure (except the "is not running" case). That means deterministic failures (bad arguments, missing files/paths, permissions, etc.) will pause for up to 120s before exiting, which can be confusing and makes non-DB errors slower to diagnose. Consider only retrying on known transient DB-connectivity errors and failing fast for everything else.
			if ( ! notified ) {
				notified = true;
				console.log( waiting );
			}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants