Skip to content

Report the OpenDJ container healthy only once its bootstrap has succeeded - #898

Open
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/docker-health-after-bootstrap
Open

Report the OpenDJ container healthy only once its bootstrap has succeeded#898
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/docker-health-after-bootstrap

Conversation

@vharseko

Copy link
Copy Markdown
Member

Problem

The image's health check searches the root DSE:

HEALTHCHECK ... CMD opendj/bin/ldapsearch ... --baseDN "" --searchScope base "(objectClass=*)" 1.1 || exit 1

That answers as soon as the server is listening — and setup starts it in the middle of
bootstrap/setup.sh, before dsconfig create-backend userRoot and before the import-ldif
that creates the base entry. So a container reports healthy while a search of BASE_DN
still fails with 32 (No Such Entry), which is what depends_on: { condition: service_healthy }
promises users it will not.

CI hit exactly that in run 32392485054
— the Docker test custom password step of build-docker:

docker exec test_custom ... ldapsearch --baseDN "dc=example,dc=com" --searchScope base
The LDAP search request failed: 32 (No Such Entry)
##[error]Process completed with exit code 32.

The container logs the failure trap captured stop at Configuring Certificates ..... Done.,
i.e. the bootstrap had not reached the backend or the import; the step's poll had simply landed
in the window where the server answers and the instance is still being built. The build-docker-alpine
job ran the same step green in the same run — its poll landed 41 s in rather than 11 s in.

A second defect surfaced while fixing this: setup.sh reports the exit status of whatever
command ran last, so a dsconfig create-backend that failed left the script "successful",
and the container came up serving nothing.

Fix

  • run.sh writes /opt/opendj/.bootstrap-complete once the bootstrap is through, and the
    health check tests that marker before it probes — so a container still bootstrapping
    costs one test -f rather than a JVM per interval.
  • The marker is written only where the step it stands for succeeded: the bootstrap script,
    the replication MASTER_SERVER asked for, or the upgrade -n taken when starting over an
    instance that is already there. A bootstrap that fails leaves the container running to be
    looked at, with the reason in docker logs, but never healthy.
  • It is cleared first, because a restarted container replays run.sh over the writable
    layer the previous run left behind, and it lives outside ./data because it records what
    this container run has done, not what the volume holds.
  • What an instance is made of now fails setup.shsetup, dsconfig create-backend,
    makeldif and both import-ldif calls. The optional schema and data LDIFs keep the
    tolerance they were written with (--continueOnError).
  • --start-period 1s → 5m, which is what a bootstrap importing SAMPLE_DATA into a small
    container can take. A probe that succeeds ends it early; the trade-off is that a start that
    cannot come up at all is reported unhealthy later than it used to be.
  • The marker path has one definition, ENV BOOTSTRAP_COMPLETE, read by both run.sh and the
    probe. Both Dockerfiles are changed identically.

.github/workflows/build.yml is deliberately untouched: its four until ... healthy waits
become correct as they are, including the latent race in the Docker test step, where
dsconfig create-backend example2 could run while setup.sh was still creating userRoot.
On a runner the bootstrap takes ~40 s against the step's timeout 3m.

Verification

Both images built locally and driven through the CI steps verbatim:

check result
Docker test custom password step, noble exit 0, dn: dc=example,dc=com
Docker test step in full (create-backend, offline import, rebuild-index ×2, search) 10000 entries found, exit 0
the race itself, measured t+17s the server answers — where the old check turned healthy — and the search fails 32 (No Such Entry); t+31s the marker appears and it succeeds
a bootstrap that fails (BACKEND_TYPE=bogus) never healthy, no marker, setup.sh failed, this container will not report itself healthy in the log — while the root DSE answers, i.e. the old check would have said healthy
restart over an existing instance, noble and alpine marker cleared, rewritten by the upgrade branch, healthy again, base entry served
upgrade -n on an up-to-date instance exit 0, so gating the marker on it does not break the restart path
probe cadence ~5 s inside the start period, so the gate adds ~5–10 s after the bootstrap ends

The absolute timings above were taken on a box under load 150–450, where the bootstrap takes
120–280 s instead of ~35 s; the outcomes are unaffected.

…eded

The health check searched the root DSE, which answers as soon as setup starts the
server - in the middle of bootstrap/setup.sh, before the backend of BASE_DN is
created and its entries imported. A container therefore reported healthy while a
search of BASE_DN still failed with "No Such Entry": that is what the CI "Docker
test custom password" step hit when its poll landed in that window (run
32392485054), and what depends_on: service_healthy promises users it will not.

run.sh now writes /opt/opendj/.bootstrap-complete, which the health check tests
before it probes at all, so a container still bootstrapping costs no more than a
test -f. It is written only where the step it stands for succeeded - the bootstrap,
the replication it asked for, or the upgrade taken when starting over an instance
that is already there - and cleared first, so a restarted container cannot inherit
the marker of the previous run over the writable layer. What an instance is made of
is now left to fail setup.sh, which had reported the status of its last command
regardless of a backend it could not create; the optional schema and data LDIFs keep
the tolerance they were written with. A bootstrap that fails leaves the container
running to be looked at, but never healthy.

The start period covers a bootstrap importing SAMPLE_DATA into a small container,
which is minutes rather than the second it allowed for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant