Skip to content

fix: detect externally stopped and removed containers in up monitor - #13990

Open
glours wants to merge 2 commits into
docker:mainfrom
glours:13985-monitor-external-stop-detection
Open

fix: detect externally stopped and removed containers in up monitor#13990
glours wants to merge 2 commits into
docker:mainfrom
glours:13985-monitor-external-stop-detection

Conversation

@glours

@glours glours commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What I did
Since 2.39.3 the monitor relies solely on die events to stop tracking containers, but a container stopped while in restart backoff emits only stop and destroy events, so an attached up hangs forever after an external stop/down. Handle destroy as terminal, and on stop inspect the container to distinguish a definitive stop from the transient one emitted during a ContainerRestart, which keeps watch sync+restart (#13161) working.

Related issue
Fixes #13985

(not mandatory) A picture of a cute animal, if possible in relation to what you did
image

@glours
glours requested review from a team as code owners July 31, 2026 09:39
@glours
glours requested a review from ndeloof July 31, 2026 09:39

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟡 NEEDS ATTENTION

Two medium-confidence findings introduced by this PR's new ActionStop handling — both relate to how the monitor responds when ContainerInspect does or doesn't find the container.

Lower-confidence findings (not posted inline)

  • [low] pkg/compose/monitor.go:122 — ActionStart handler does not clear the restarting set (confidence: weak 10/100) — not a real bug given the current code flow; noted for future maintainability

Comment thread pkg/compose/monitor.go Outdated
Comment thread pkg/compose/monitor.go Outdated
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.76923% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/compose/monitor.go 80.76% 2 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@glours
glours requested a review from docker-agent July 31, 2026 10:15

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟢 APPROVE

The fix is well-structured and correct. The new ActionStop handler correctly distinguishes a definitive stop from a transient one by inspecting the container state, and ActionDestroy unconditionally terminates tracking without requiring an inspect (appropriate since there is nothing left to inspect). The isRestarting helper cleanly handles the NotFound case (container already removed) by returning false, nil, which is the right terminal disposition. The test suite covers the four key scenarios: destroy-only, die+stop canceling a restart backoff (#13985 repro), die+stop+start across a ContainerRestart cycle (#13161 guard), and stop racing a removal (inspect returns NotFound).

ndeloof
ndeloof previously approved these changes Aug 11, 2026
@ndeloof

ndeloof commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Considering the impact event management had in the past 😓 I'd like this new corner case is covered by an e2e test, based on reproduction scenario documented by #13985

ndeloof added a commit that referenced this pull request Aug 11, 2026
Regression test for #13985: a container stopped while in restart backoff
emits no die event (only stop then destroy), and since v2.39.3 the
attached up waits forever for a die event that never comes.

The test fails on main and passes with the monitor fix from #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 11, 2026
Regression tests for #13985: a container stopped while in restart backoff
emits no die event, and since v2.39.3 the attached up waits forever for a
die event that never comes. Two explicit scenarios:

- external stop alone: containers are stopped but not removed (stop event,
  no destroy). Up to v2.39.2 the attached up exited a few seconds after
  the stop.
- external stop+down: the container in backoff emits stop then destroy.

Both tests fail on main and pass with the monitor fix from #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 11, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 13, 2026
Regression tests for #13985: a container stopped while in restart backoff
emits no die event, and since v2.39.3 the attached up waits forever for a
die event that never comes. Two explicit scenarios:

- external stop alone: containers are stopped but not removed (stop event,
  no destroy). Up to v2.39.2 the attached up exited a few seconds after
  the stop.
- external stop+down: the container in backoff emits stop then destroy.

Both tests fail on main and pass with the monitor fix from #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 13, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 18, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 18, 2026
Regression tests for #13985: a container stopped while in restart backoff
emits no die event, and since v2.39.3 the attached up waits forever for a
die event that never comes. Two explicit scenarios:

- external stop alone: containers are stopped but not removed (stop event,
  no destroy). Up to v2.39.2 the attached up exited a few seconds after
  the stop.
- external stop+down: the container in backoff emits stop then destroy.

Both tests fail on main and pass with the monitor fix from #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 18, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof added a commit that referenced this pull request Aug 19, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
glours pushed a commit that referenced this pull request Aug 19, 2026
Map, per container lifecycle scenario, the exact event sequences the
engine emits and the monitor may receive: natural exit, policy-driven
restart (die/start only, no restart action, silence during backoff),
external stop of a running vs restarting container, ContainerRestart,
docker kill, OOM kill, rm -f and rm.

Mapped from moby's daemon sources (LogContainerEvent call sites and
handleContainerExit/containerStop/containerRestart/kill paths) and
verified against a live engine v29.6.1 with 'docker events'. Notably:
the relative order of stop and die on a container stop is not
guaranteed, and no event at all is emitted while a container sits in
restart backoff.

Related to #13985 / #13990.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Since 2.39.3 the monitor relies solely on die events to stop tracking
containers, but a container stopped while in restart backoff emits only
stop and destroy events, so an attached `up` hangs forever after an
external `stop`/`down`. Handle destroy as terminal, and on stop inspect
the container to distinguish a definitive stop from the transient one
emitted during a ContainerRestart, which keeps watch sync+restart
(docker#13161) working.

Fixes docker#13985

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Unit tests alone didn't cover the exact failure: an attached up
hanging when the project is stopped/removed externally while a
service sits in restart backoff. The new test reproduces that
sequence end to end, so a future regression on this path breaks
loudly instead of slipping through as a passing unit suite. A
RestartCount guard narrows the race with the backoff expiring, so a
missed window fails loudly instead of passing silently through the
pre-existing die-based path.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
@glours
glours force-pushed the 13985-monitor-external-stop-detection branch from c950d5f to c9cadaf Compare September 10, 2026 17:42
@glours
glours requested a review from docker-agent September 10, 2026 17:49

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟡 NEEDS ATTENTION

This PR correctly fixes the core #13985 issue — a container in restart backoff that is externally stopped no longer hangs up forever. Two semantic edge cases in the new event-handling logic are worth reviewing:

  1. The final event consumers receive in the stop-of-backoff scenario carries Restarting: true with no corrective follow-up
  2. A stop-before-die race on a normal container stop can leave a stale entry in the restarting set

Comment thread pkg/compose/monitor.go
// preceding die event, just stop tracking the container
logrus.Debugf("container %s stopped", ctr.Name)
restarting.Remove(ctr.ID)
containers.Remove(ctr.ID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] Final event consumers receive after external stop-of-backoff has Restarting: true, with no corrective follow-up

In the #13985 fix scenario (container in restart backoff, externally stopped), the event sequence is:

  1. dieonContainerDie: isRestarting returns true → emits ContainerEventExited{Restarting: true}, keeps container tracked
  2. stoponContainerStop: isRestarting returns false → calls restarting.Remove + containers.Remove, emits no event, monitor loop ends

The only ContainerEventExited consumers ever see for this scenario carries Restarting: true. No follow-up event corrects it to Restarting: false. The code comment ("the exit was already reported to listeners by the preceding die event") is accurate, but that prior event's flag was set optimistically at die-time and is now stale.

The test TestMonitorExitsWhenRestartingContainerStopped encodes this shape (got[0].Restarting == true), which means callers that check the flag to decide whether to wait for a restart may behave incorrectly — they see "exiting, restart expected" and then nothing. For callers that rely solely on monitor termination (not on the event flag) to detect the definitive stop this is harmless, but it is a semantic mismatch worth noting.

Fix: pass the stop-event timestamp into onContainerStop and emit a corrective ContainerEventExited{Restarting: false} from the definitive-stop branch before evicting the container:

// in the else branch of onContainerStop:
c.notify(newContainerEvent(stopTimeNano, ctr, api.ContainerEventExited))
restarting.Remove(ctr.ID)
containers.Remove(ctr.ID)

This also means the test assertion should flip to got[1].Restarting == false (two events: the die-time Restarting: true and the stop-time Restarting: false).

Confidence Score
🟡 moderate 67/100

Comment thread pkg/compose/monitor.go
// State.Restarting is set by engine when container is configured to restart on exit
// on ContainerRestart it doesn't (see https://github.com/moby/moby/issues/45538)
// container state still is reported as "running"
if willRestart {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] onContainerDie's non-restart path leaves a stale restarting entry when onContainerStop ran first

The new onContainerStop handler (added by this PR) can add a container to restarting when isRestarting returns true (i.e., State.Running == true, mid-exit during a normal docker stop). If this stop event arrives before the corresponding die event:

  1. stoponContainerStop: Running=true seen → restarting.Add(ctr.ID)
  2. dieonContainerDie: Exited seen → willRestart=false → calls containers.Remove(ctr.ID) but NOT restarting.Remove(ctr.ID)

The container is removed from containers (so the monitor eventually terminates), but the restarting set retains a stale entry. In the current session this is harmless — once the container leaves containers, no future start event for it is expected. However if the container is externally re-created with the same ID before the monitor session ends, onContainerStart would see restarting.Has(ctr.ID) == true and emit ContainerEventStarted{Restarting: true} incorrectly (suggesting it is recovering from a monitored crash rather than being a fresh start).

Fix: add restarting.Remove(ctr.ID) to onContainerDie's non-restart branch (alongside the existing containers.Remove):

// in onContainerDie, willRestart=false branch:
c.notify(newContainerEvent(event.TimeNano, ctr, api.ContainerEventExited))
restarting.Remove(ctr.ID)   // clean up any entry added by a preceding stop event
containers.Remove(ctr.ID)
return nil

This is a defensive cleanup that makes the two paths (stop-first and die-first) symmetric.

Confidence Score
🟡 moderate 57/100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Attached up never exits after external stop+down when a container is in restart backoff (regression in v2.39.3)

3 participants