Skip to content

fix(controller): answer /healthz without waiting for a running cycle - #47

Merged
parsilver merged 2 commits into
mainfrom
fix/46-healthz-cycle-lock
Nov 16, 2026
Merged

parsilver merged 2 commits into
mainfrom
fix/46-healthz-cycle-lock

Conversation

@parsilver

@parsilver parsilver commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #46

/healthz timed out whenever it landed during a cycle: Cycle() holds c.mu for every GitHub and Railway call (about a minute in production) and Status() took the same lock.

Change

  • Controller.status is an atomic.Pointer[Status]. record() builds a fresh Status and stores it once; Status() loads it without c.mu (zero Status before the first cycle, so /healthz still answers 503 then).
  • c.mu is unchanged otherwise: it still runs one cycle at a time and guards the cycle state (failures, scopeFails, alerted, pending, lowWarn). A stored Status is never changed afterwards.
  • The rest of the /healthz path does not wait on a cycle either: RateLimits() on the GitHub client takes the client's own lock only around a map copy, and that lock is never held across an HTTP call.
  • TestAlertsAreSentWithoutHoldingTheLock now takes c.mu inside the alert, since Status() no longer would catch an alert sent under the lock (checked: moving the alerts back under the lock makes it fail).
  • The unreadable-scope scan moved out of record() into unreadableScopes() to keep record under CRAP 10.
  • CHANGELOG [Unreleased] / Fixed entry.

Acceptance

  • A test holds a cycle open and /healthz answers within 100 ms with the previous cycle's status: TestHealthzAnswersDuringACycleWithThePreviousStatus (blocking fake QueuedJobs; expects 200 with queued 1 and services 1 from the first cycle). Before the fix it failed with /healthz did not answer within 100 ms while a cycle was running (first commit).
  • The status still changes after each cycle: the existing health tests pass unchanged in what they assert; TestHealthzIs503BeforeTheFirstCycle covers the empty pointer.
  • go test -race ./... passes.
  • The cause of the minute-long cycle is in a comment on controller: /healthz waits for the cycle lock and times out during a cycle #46: sequential GitHub calls, three run listings per repository per cycle plus one jobs listing per active run.

Checks run

go vet ./...                      (clean)
gofmt -l internal                 (no files)
go test -race -count=3 ./...      ok for config, controller, ghapp, railway, reconcile, webhook
go test -race -count=1 ./...      same, after rebasing on main

Quality gates (changed code)

  • CRAP: Status 2.0, record 8.0 (was 11.0), unreadableScopes 4.0. Cycle is 18.5 on main and here; only its comment changed.
  • Duplication: dryer internal/controller: no candidates.
  • Mutation on the changed lines: 6 killed, 0 survived. The first run left two survivors in the scope scan (n > 0 and n >= FailAlertAfter); TestOneUnreadableScopeIsNamedAndTheOtherKeepsWorking now checks the scope is listed after one failed read and that the alert names it as unreadable that long.

Risk

Low. The only shared state that changes hands is an immutable Status behind an atomic pointer; cycles stay serialised. Rollback is a revert of this PR.

Not verified

I did not run the controller against GitHub or Railway or read the production logs; the slow-cycle numbers on #46 are from the code path, and the github_calls value on the cycle log line will confirm them.

A cycle holds the controller lock for every GitHub and Railway call,
and Status takes the same lock, so a health check that lands during
the minute-long cycle times out. The test holds a cycle open with a
blocking fake GitHub call and fails today after 100 ms.
Status took the lock a cycle holds for every GitHub and Railway call,
so a health check during the minute-long cycle timed out. The cycle
now publishes each finished Status through an atomic pointer that
Status reads without the lock; the lock still serialises cycles and
guards the cycle state.

The alert test now takes the lock itself, since Status no longer
would catch an alert sent under it. The scope scan moves out of
record to keep it under CRAP 10, and its test now kills the two
mutants that survived there. Proved with go test -race and a
mutation run on the changed lines.
@parsilver
parsilver marked this pull request as ready for review November 16, 2026 06:57

@parsilver parsilver left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Blocker.

Status() now reads an atomic.Pointer[Status] that record() replaces whole at the end of each cycle, so /healthz never waits on the cycle lock. The snapshot is never mutated after Store, so sharing its slices with readers is safe. The new test holds a cycle open inside QueuedJobs and gets the previous cycle's status back within 100 ms; the 503-before-first-cycle case is pinned too. The alert test now takes c.mu itself, so it still proves alerts go out after the lock is released. CI runs go test -race ./... and is green on every job. The cause of the one-minute cycle (3 run listings per repository, all sequential) is written on #46.

Suggestion (follow-up): the cycle cost is 3 x repositories sequential calls. Listing runs once without a status filter, or reading the three statuses concurrently per repository, would cut the cycle well under a minute. Worth its own issue.

Merging.

@parsilver
parsilver merged commit 7f53567 into main Nov 16, 2026
14 checks passed
@parsilver
parsilver deleted the fix/46-healthz-cycle-lock branch November 16, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

controller: /healthz waits for the cycle lock and times out during a cycle

1 participant