fix(ci): stop the e2e teardown from failing a green suite - #11816
Merged
Conversation
`docker stop` returns as soon as the container exits, but the daemon reaps a `--rm` container asynchronously after that. The `docker rmi localai-tests` that follows teardown-e2e then loses the race against the reaper and fails with "conflict: ... is using its referenced image", so make exits 1 and the job goes red after every spec has passed. This is why the E2E Backend Tests job fails at random across pull requests. Runs 33435319093, 33435332991, 33412165884 and 33444669207 all report "SUCCESS! -- 235 Passed | 0 Failed" and then die in teardown. `docker rm -f` is synchronous, so the image reference is gone before teardown-e2e returns. It also covers the case where no container is running, which `docker stop` could not because it rejects an empty argument list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code]
The tmate step runs on every failure and then holds the runner until GitHub cancels the job at the 6 hour limit. A one second cleanup race in the e2e teardown therefore costs a whole ubuntu-latest slot. The recent run list is full of 6h, 7h and 12h cancelled runs for that reason. The step now needs the `ci-debug` label on the pull request, so a session opens when somebody wants to debug and never otherwise. The 30 minute step timeout caps the cost when the label is left behind. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Two independent CI defects that together turn a passing E2E suite into a red check that burns a 6 hour runner slot.
1.
teardown-e2eraces the container reaper.run-e2e-imagestarts the container with-d --rm.docker stopreturns as soon as the container exits, but Docker reaps a--rmcontainer asynchronously after that. Thedocker rmi localai-testson the next line then loses the race and fails:Every spec had already passed.
docker rm -fis synchronous, so the image reference is gone beforeteardown-e2ereturns. The same change fixes a second latent bug on that line: with no matching container,docker stop $(docker ps -q ...)gets zero arguments and exits 1 with "requires at least 1 argument".2. The tmate step holds the runner for 6 hours. It runs on every failure and, despite
detached: true, goes silent until GitHub cancels the job at the wall. It now needs aci-debuglabel on the pull request, with a 30 minute step timeout as a backstop. Applied to all five active workflows that carry the step, since the trap is identical in each.Why
Sampled runs of
tests-e2e.ymlthat hung and were cancelled at the 6 hour limit:bump/BONSAI_VERSIONdocker rmiconflictbump/DS4_VERSIONdocker rmiconflictmasterdocker rmiconflictmasterdocker rmiconflicttest/distributed-e2e-ciThe run list for this workflow currently shows repeated 6h, 7h, 9h and 12h cancelled runs.
How to verify
The race reproduces against a container shaped like
localai-tests, one that resists SIGTERM and carries a large writable layer so the reap is slow:The empty-argument bug:
docker stop $(docker ps -q --filter ancestor=nonexistent)exits 1; the replacement exits 0.For the workflows, the E2E Backend Tests check on this PR should go green, and no tmate step should run because this PR carries no
ci-debuglabel.Not in scope
PR #11800 also has failing bonsai backend builds. Those are deterministic, not this flake: the fork bump makes
jsonresolve tocommon_json, which has noparse_error, whilebackend/cpp/llama-cpp/grpc-server.cppstill catchesconst json::parse_error&. That needs its own change.