Repository navigation
fix(controller): answer /healthz without waiting for a running cycle - #47
Conversation
A cycle holds the controller lock for every GitHub and Railway call, and Status takes the same lock, so a health check that lands during the minute-long cycle times out. The test holds a cycle open with a blocking fake GitHub call and fails today after 100 ms.
Status took the lock a cycle holds for every GitHub and Railway call, so a health check during the minute-long cycle timed out. The cycle now publishes each finished Status through an atomic pointer that Status reads without the lock; the lock still serialises cycles and guards the cycle state. The alert test now takes the lock itself, since Status no longer would catch an alert sent under it. The scope scan moves out of record to keep it under CRAP 10, and its test now kills the two mutants that survived there. Proved with go test -race and a mutation run on the changed lines.
parsilver
left a comment
There was a problem hiding this comment.
No Blocker.
Status() now reads an atomic.Pointer[Status] that record() replaces whole at the end of each cycle, so /healthz never waits on the cycle lock. The snapshot is never mutated after Store, so sharing its slices with readers is safe. The new test holds a cycle open inside QueuedJobs and gets the previous cycle's status back within 100 ms; the 503-before-first-cycle case is pinned too. The alert test now takes c.mu itself, so it still proves alerts go out after the lock is released. CI runs go test -race ./... and is green on every job. The cause of the one-minute cycle (3 run listings per repository, all sequential) is written on #46.
Suggestion (follow-up): the cycle cost is 3 x repositories sequential calls. Listing runs once without a status filter, or reading the three statuses concurrently per repository, would cut the cycle well under a minute. Worth its own issue.
Merging.
Closes #46
/healthztimed out whenever it landed during a cycle:Cycle()holdsc.mufor every GitHub and Railway call (about a minute in production) andStatus()took the same lock.Change
Controller.statusis anatomic.Pointer[Status].record()builds a freshStatusand stores it once;Status()loads it withoutc.mu(zeroStatusbefore the first cycle, so/healthzstill answers 503 then).c.muis unchanged otherwise: it still runs one cycle at a time and guards the cycle state (failures,scopeFails,alerted,pending,lowWarn). A storedStatusis never changed afterwards./healthzpath does not wait on a cycle either:RateLimits()on the GitHub client takes the client's own lock only around a map copy, and that lock is never held across an HTTP call.TestAlertsAreSentWithoutHoldingTheLocknow takesc.muinside the alert, sinceStatus()no longer would catch an alert sent under the lock (checked: moving the alerts back under the lock makes it fail).record()intounreadableScopes()to keeprecordunder CRAP 10.[Unreleased]/ Fixed entry.Acceptance
/healthzanswers within 100 ms with the previous cycle's status:TestHealthzAnswersDuringACycleWithThePreviousStatus(blocking fakeQueuedJobs; expects 200 withqueued1 andservices1 from the first cycle). Before the fix it failed with/healthz did not answer within 100 ms while a cycle was running(first commit).TestHealthzIs503BeforeTheFirstCyclecovers the empty pointer.go test -race ./...passes.Checks run
Quality gates (changed code)
Status2.0,record8.0 (was 11.0),unreadableScopes4.0.Cycleis 18.5 on main and here; only its comment changed.dryer internal/controller: no candidates.n > 0andn >= FailAlertAfter);TestOneUnreadableScopeIsNamedAndTheOtherKeepsWorkingnow checks the scope is listed after one failed read and that the alert names it as unreadable that long.Risk
Low. The only shared state that changes hands is an immutable
Statusbehind an atomic pointer; cycles stay serialised. Rollback is a revert of this PR.Not verified
I did not run the controller against GitHub or Railway or read the production logs; the slow-cycle numbers on #46 are from the code path, and the
github_callsvalue on thecyclelog line will confirm them.