plan the purge of stale pre_start hook runners - #14201
Conversation
e580984 to
7804577
Compare
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
7804577 to
b175494
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
5a83823 to
532e6cf
Compare
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
One medium-severity correctness finding confirmed (strong confidence, 100/100).
Summary
The PR correctly classifies hook containers into a dedicated HookContainers bucket so they no longer masquerade as replica-0, and the plan-level purge of stale hook runners is well-gated and best-effort. The refactoring of runPreStartHook into execPreStartHook + caller-managed lifecycle is clean.
One issue was found: the early continue in collectObservedState bypasses the orphan check for hook containers, so stale hook runners from deleted services escape compose up --remove-orphans cleanup.
532e6cf to
6682e70
Compare
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
Two findings introduced by this PR's + lines:
Inline comments (2):
pkg/compose/reconcile.go:774[medium] —planPurgeStaleHookRunnerspurges stale hook runners even whenscale=0, contrary to the stated contract (confidence: 🟡 moderate 57/100)pkg/compose/observed_state.go:180[medium] — Hook containers of disabled services are siloed intoHookContainerswith no cleanup path (confidence: 🟢 strong 97/100)
|
|
||
| for _, ctr := range raw { | ||
| svcName := ctr.Labels[api.ServiceLabel] | ||
| if ctr.Labels[api.HookLabel] != "" && knownServices[svcName] { |
There was a problem hiding this comment.
[medium] Hook containers of disabled services are siloed into HookContainers with no cleanup path
The new classification block guards on knownServices[svcName], and knownServices is populated for both active AND disabled services:
for _, ds := range project.DisabledServices {
knownServices[ds.Name] = true // ← disabled services included
}So a stale pre_start hook container whose ServiceLabel refers to a disabled service hits the new continue and lands in HookContainers[disabled-svc] — never reaching the orphan check below.
planPurgeStaleHookRunners is called only from reconcileService, which is driven by visitInDependencyOrder over NewGraph(project.Services) — disabled services are not in that graph. As a result, HookContainers["disabled-svc"] is filled at collection time and never read again.
Result: stale hook containers of disabled services have no cleanup path:
- Not an orphan →
--remove-orphansignores them. - Not processed by the plan →
planPurgeStaleHookRunnersnever fires for them.
They accumulate indefinitely until manually removed.
Fix: use a separate activeServices set (active services only, not disabled ones) for the hook-container gate, so disabled-service hook containers still reach the orphan check:
_, isActive := project.Services[svcName]
if ctr.Labels[api.HookLabel] != "" && isActive {Or inline:
| if ctr.Labels[api.HookLabel] != "" && knownServices[svcName] { | |
| if ctr.Labels[api.HookLabel] != "" && project.Services[svcName].Name != "" { |
| Confidence | Score |
|---|---|
| 🟢 strong | 97/100 |
There was a problem hiding this comment.
Deliberately kept as-is — the accumulation predates this PR rather than being introduced by it: before this change, a disabled service's hook runner sat in Containers[svc] as a bogus replica-0 entry (knownServices includes disabled services), never visited by reconcileService and never orphaned, so --remove-orphans did not clean it either. Routing it to the orphan check would be a behavior CHANGE: --remove-orphans would delete a retained-for-inspection runner while its service is merely profile-disabled — and while the service's real replicas are kept. The PR actually improves the cleanup path: once the service is re-enabled, reconcileService visits it and the plan purges the stale runner (as the imperative purge would). down keeps cleaning them via its label query. Leaving this thread open for the maintainer to weigh in.
pre_start hooks run in ephemeral containers. When a run fails, the runner is deliberately retained for post-mortem inspection, and the next run purges it — but that purge lived entirely inside the imperative primitive, invisible to the reconciliation plan. It is now a plan operation: when pre_start is going to run again (hooks declared, no replica running at observation — the imperative gating), the plan emits one best-effort RemoveContainer per stale runner, dropping its anonymous volumes, exactly the warn-only semantics of the imperative purge that remains in place as backstop. Observed state learns to tell hook containers apart: they carry no container-number label and previously classified as a service replica numbered 0. They now land in a dedicated HookContainers bucket the reconciler plans purges from. The imperative primitive is also split into its lifecycle-free execution piece (execPreStartHook: start, wait, log streaming, retain-on-failure) and the create/remove pieces around it, recomposed identically in runPreStart — locked by the existing characterization tests. This prepares moving hook-container creation and post-success removal into the plan once the start phase lands (docker#14200). Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
6682e70 to
b52def4
Compare
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟢 APPROVE
The PR is well-structured and the changes are correct.
Summary of changes reviewed:
observed_state.go: Hook containers (identified byHookLabel) are now correctly classified intoHookContainersmap keyed by service name, preventing stale runners from masquerading as replica 0. The guardknownServices[svcName]ensures containers for services removed from the model fall through to the orphan path as expected.reconcile.go:planPurgeStaleHookRunnerscorrectly mirrors the imperative gate (len(service.PreStart) > 0 && expectedScale > 0 && no running replica). The best-effort/warn-only semantics and theRemoveVolumes: trueflag are both correct.pre_start.go: The refactor cleanly separatesexecPreStartHook(start + wait + logs + retain-on-failure) fromrunPreStart(create → exec → remove-on-success). The success-path removal is now correctly owned by the caller, and the failure retention / cancellation removal paths are unchanged.executor_ops.go: TheBestEffortcheck andRemoveVolumesforwarding are placed correctly — a warn-only failure returnsnilso the plan continues, and the live-view comment explains the container stays in the cache.- Test coverage is thorough: executor, observed-state, and reconciler all have dedicated tests for the new behaviour.
What this PR does, in one sentence
The reconciliation plan now covers the purge of stale
pre_starthook runners, and observed state stops mistaking those runners for service replicas.Context
pre_starthooks run in ephemeral containers. When a hook fails, its runner is deliberately retained so the operator can inspect it, and the next run purges it — but that purge lives entirely inside the imperative primitive, invisible to the plan. Worse, observed-state collection had no notion of hook containers at all: carrying no container-number label, a leftover runner was classified as a service replica numbered 0.What the PR brings
HookContainersbucket — they no longer masquerade as replica 0.pre_startis going to run again (hooks declared, no replica running at observation — the exact imperative gating), the plan emits one best-effortRemoveContainerper stale runner, dropping its anonymous volumes. Failure is warn-only and a running service keeps its retained-for-inspection runner untouched, exactly like today; the imperative purge remains in place as backstop, so no behavior is lost in the cases the plan cannot see.execPreStartHook: start, wait, logs, retain-on-failure) and the create/remove pieces around it, recomposed identically inrunPreStart— locked by the existing characterization tests.Why this is the right next brick
Once the start phase lands in the plan (#14200), hook-container creation and post-success removal can join it as plan nodes chained around a run-only
RunPreStart— this PR puts the classification, the purge, and the split primitive in place without depending on it.