Report the OpenDJ container healthy only once its bootstrap has succeeded - #898
Open
vharseko wants to merge 1 commit into
Open
Report the OpenDJ container healthy only once its bootstrap has succeeded#898vharseko wants to merge 1 commit into
vharseko wants to merge 1 commit into
Conversation
…eded The health check searched the root DSE, which answers as soon as setup starts the server - in the middle of bootstrap/setup.sh, before the backend of BASE_DN is created and its entries imported. A container therefore reported healthy while a search of BASE_DN still failed with "No Such Entry": that is what the CI "Docker test custom password" step hit when its poll landed in that window (run 32392485054), and what depends_on: service_healthy promises users it will not. run.sh now writes /opt/opendj/.bootstrap-complete, which the health check tests before it probes at all, so a container still bootstrapping costs no more than a test -f. It is written only where the step it stands for succeeded - the bootstrap, the replication it asked for, or the upgrade taken when starting over an instance that is already there - and cleared first, so a restarted container cannot inherit the marker of the previous run over the writable layer. What an instance is made of is now left to fail setup.sh, which had reported the status of its last command regardless of a backend it could not create; the optional schema and data LDIFs keep the tolerance they were written with. A bootstrap that fails leaves the container running to be looked at, but never healthy. The start period covers a bootstrap importing SAMPLE_DATA into a small container, which is minutes rather than the second it allowed for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The image's health check searches the root DSE:
That answers as soon as the server is listening — and
setupstarts it in the middle ofbootstrap/setup.sh, beforedsconfig create-backend userRootand before theimport-ldifthat creates the base entry. So a container reports
healthywhile a search ofBASE_DNstill fails with
32 (No Such Entry), which is whatdepends_on: { condition: service_healthy }promises users it will not.
CI hit exactly that in run 32392485054
— the
Docker test custom passwordstep ofbuild-docker:The container logs the failure trap captured stop at
Configuring Certificates ..... Done.,i.e. the bootstrap had not reached the backend or the import; the step's poll had simply landed
in the window where the server answers and the instance is still being built. The
build-docker-alpinejob ran the same step green in the same run — its poll landed 41 s in rather than 11 s in.
A second defect surfaced while fixing this:
setup.shreports the exit status of whatevercommand ran last, so a
dsconfig create-backendthat failed left the script "successful",and the container came up serving nothing.
Fix
run.shwrites/opt/opendj/.bootstrap-completeonce the bootstrap is through, and thehealth check tests that marker before it probes — so a container still bootstrapping
costs one
test -frather than a JVM per interval.the replication
MASTER_SERVERasked for, or theupgrade -ntaken when starting over aninstance that is already there. A bootstrap that fails leaves the container running to be
looked at, with the reason in
docker logs, but never healthy.run.shover the writablelayer the previous run left behind, and it lives outside
./databecause it records whatthis container run has done, not what the volume holds.
setup.sh—setup,dsconfig create-backend,makeldifand bothimport-ldifcalls. The optional schema and data LDIFs keep thetolerance they were written with (
--continueOnError).--start-period1s → 5m, which is what a bootstrap importingSAMPLE_DATAinto a smallcontainer can take. A probe that succeeds ends it early; the trade-off is that a start that
cannot come up at all is reported
unhealthylater than it used to be.ENV BOOTSTRAP_COMPLETE, read by bothrun.shand theprobe. Both Dockerfiles are changed identically.
.github/workflows/build.ymlis deliberately untouched: its fouruntil ... healthywaitsbecome correct as they are, including the latent race in the
Docker teststep, wheredsconfig create-backend example2could run whilesetup.shwas still creatinguserRoot.On a runner the bootstrap takes ~40 s against the step's
timeout 3m.Verification
Both images built locally and driven through the CI steps verbatim:
Docker test custom passwordstep, nobledn: dc=example,dc=comDocker teststep in full (create-backend, offline import, rebuild-index ×2, search)t+17sthe server answers — where the old check turned healthy — and the search fails32 (No Such Entry);t+31sthe marker appears and it succeedsBACKEND_TYPE=bogus)setup.sh failed, this container will not report itself healthyin the log — while the root DSE answers, i.e. the old check would have said healthyupgrade -non an up-to-date instanceThe absolute timings above were taken on a box under load 150–450, where the bootstrap takes
120–280 s instead of ~35 s; the outcomes are unaffected.