Skip to content

fix(ci): stop the e2e teardown from failing a green suite - #11816

Merged
mudler merged 2 commits into
masterfrom
fix/e2e-teardown-race
Sep 1, 2026
Merged

fix(ci): stop the e2e teardown from failing a green suite#11816
mudler merged 2 commits into
masterfrom
fix/e2e-teardown-race

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

What changed

Two independent CI defects that together turn a passing E2E suite into a red check that burns a 6 hour runner slot.

1. teardown-e2e races the container reaper. run-e2e-image starts the container with -d --rm. docker stop returns as soon as the container exits, but Docker reaps a --rm container asynchronously after that. The docker rmi localai-tests on the next line then loses the race and fails:

Ran 235 of 236 Specs in 484.299 seconds
SUCCESS! -- 235 Passed | 0 Failed | 0 Pending | 1 Skipped
Test Suite Passed
...
docker rmi localai-tests
Error response from daemon: conflict: unable to remove repository reference "localai-tests"
  (must force) - container c8fca721d485 is using its referenced image 8be9d6846e5c
make: *** [Makefile:392: test-e2e] Error 1

Every spec had already passed. docker rm -f is synchronous, so the image reference is gone before teardown-e2e returns. The same change fixes a second latent bug on that line: with no matching container, docker stop $(docker ps -q ...) gets zero arguments and exits 1 with "requires at least 1 argument".

2. The tmate step holds the runner for 6 hours. It runs on every failure and, despite detached: true, goes silent until GitHub cancels the job at the wall. It now needs a ci-debug label on the pull request, with a 30 minute step timeout as a backstop. Applied to all five active workflows that carry the step, since the trap is identical in each.

Why

Sampled runs of tests-e2e.yml that hung and were cancelled at the 6 hour limit:

Run Branch Suite result Cause of the red X
33435319093 bump/BONSAI_VERSION 235 passed, 0 failed docker rmi conflict
33435332991 bump/DS4_VERSION 235 passed, 0 failed docker rmi conflict
33412165884 master 235 passed, 0 failed docker rmi conflict
33444669207 master 235 passed, 0 failed docker rmi conflict
33438683472 test/distributed-e2e-ci 6 failed genuine spec failure

The run list for this workflow currently shows repeated 6h, 7h, 9h and 12h cancelled runs.

How to verify

The race reproduces against a container shaped like localai-tests, one that resists SIGTERM and carries a large writable layer so the reap is slow:

cat > Dockerfile.race <<'DOCKER'
FROM busybox
CMD ["sh","-c","trap '' TERM; dd if=/dev/zero of=/big bs=1M count=800 2>/dev/null; while true; do sleep 1; done"]
DOCKER
docker build -q -t race-test -f Dockerfile.race .

# old teardown
docker run -d --rm --name r race-test
sleep 4
docker stop -t 1 $(docker ps -q --filter ancestor=race-test)
docker rmi race-test          # conflict, 5/5 trials

# new teardown
docker run -d --rm --name r race-test
sleep 4
CONTAINERS=$(docker ps -aq --filter ancestor=race-test); \
  [ -n "$CONTAINERS" ] && docker rm -f $CONTAINERS
docker rmi race-test          # succeeds, 5/5 trials

The empty-argument bug: docker stop $(docker ps -q --filter ancestor=nonexistent) exits 1; the replacement exits 0.

For the workflows, the E2E Backend Tests check on this PR should go green, and no tmate step should run because this PR carries no ci-debug label.

Not in scope

PR #11800 also has failing bonsai backend builds. Those are deterministic, not this flake: the fork bump makes json resolve to common_json, which has no parse_error, while backend/cpp/llama-cpp/grpc-server.cpp still catches const json::parse_error&. That needs its own change.

`docker stop` returns as soon as the container exits, but the daemon
reaps a `--rm` container asynchronously after that. The `docker rmi
localai-tests` that follows teardown-e2e then loses the race against the
reaper and fails with "conflict: ... is using its referenced image", so
make exits 1 and the job goes red after every spec has passed.

This is why the E2E Backend Tests job fails at random across pull
requests. Runs 33435319093, 33435332991, 33412165884 and 33444669207 all
report "SUCCESS! -- 235 Passed | 0 Failed" and then die in teardown.

`docker rm -f` is synchronous, so the image reference is gone before
teardown-e2e returns. It also covers the case where no container is
running, which `docker stop` could not because it rejects an empty
argument list.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
The tmate step runs on every failure and then holds the runner until
GitHub cancels the job at the 6 hour limit. A one second cleanup race in
the e2e teardown therefore costs a whole ubuntu-latest slot. The recent
run list is full of 6h, 7h and 12h cancelled runs for that reason.

The step now needs the `ci-debug` label on the pull request, so a
session opens when somebody wants to debug and never otherwise. The
30 minute step timeout caps the cost when the label is left behind.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
@mudler
mudler merged commit b4e16b9 into master Sep 1, 2026
69 of 70 checks passed
@mudler
mudler deleted the fix/e2e-teardown-race branch September 1, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants