Repository navigation
fix(controller): a 404 scope in RUNNER_SCOPES no longer fails the cycle - #50
Conversation
A deleted repository in RUNNER_SCOPES answered 404 every cycle and the controller counted it as a failed cycle, so /healthz went 503 (#48). These tests pin the wanted behaviour: GitHub's 404 is ErrNotFound, and a not-found scope is warned about while the cycle still succeeds. They fail now: no call wraps ErrNotFound yet and the controller fails the cycle.
The controller could not tell a deleted repository from an outage: every non-2xx answer was the same opaque error. A 404 now wraps ErrNotFound. In an org scope a repository or run that vanishes between the listing and the read is skipped, so one deleted repository no longer hides the org's other repositories. QueuedJobs is split into orgRepos, repoJobs and runJobs to keep each piece small. Proved by the new notfound tests (repo runs 404, repo and org installation lookup 404, token request 404, an org repo and run that vanish mid-read) and the existing ghapp suite.
Three deleted repositories in RUNNER_SCOPES failed 112 cycles in a row and turned /healthz 503 (#48), so queued jobs of every other scope sat without runners. A scope GitHub answers 404 for is now skipped, listed in unreadable_scopes and warned about once a cycle, and the cycle still succeeds. It is also kept out of the failing alert's scopes unreadable that long, so an unrelated outage does not blame it. Every other read error fails the cycle as before. Proved by the controller notfound tests (jobs 404 and runners 404 with a readable scope still getting runners, a 502 still failing, the alert text) and the full suite with -race.
Operators need to know that a deleted or misspelled scope no longer turns /healthz 503 and where to see it instead: unreadable_scopes and the warn line. docs/deploy.md only says to watch /healthz and needs no change.
The 404 branch had pushed call's CRAP score from 19.2 to 20.2, above its score on main. statusError holds the status-to-error decision, so call is back at 19.2 and the 404 rule sits in one small function. Behaviour is unchanged: the ghapp suite passes with -race.
parsilver
left a comment
There was a problem hiding this comment.
Blocker:
-
A whole-pool 404 now looks healthy (
controller.gonoteRead). Its own comment lists "an App no longer installed" as a 404 case. If the App is uninstalled, or its installation id goes stale, every scope answers 404: each cycle returns nil,/healthzstays 200, and no alert fires, while no CI job gets a runner. #48's three deleted repos were the right case to stop failing the cycle, but not a total outage. Fix both:- When every scope in a cycle is 404 (no scope readable), fail the cycle as before, so
/healthzgoes 503 and the failing alert fires afterALERT_FAILED_CYCLES. - Track consecutive 404 cycles per scope, and send one alert per scope (not a failure) once it has been missing for
ALERT_FAILED_CYCLEScycles, with a recovered message when it reads again. A deleted repo left inRUNNER_SCOPESshould be noticed and removed, not warned about silently forever.
Tests: all scopes 404 fails the cycle and alerts. One scope 404 among readable ones keeps the cycle green, raises one missing-scope alert after N cycles, and clears it on recovery.
- When every scope in a cycle is 404 (no scope readable), fail the cycle as before, so
Checked and fine: ErrNotFound is set only on 404, not on 401, 403, 429 or 500, and that's tested. An org skips a repo or run that vanished mid-read. The token is forgotten on a 404. readScope/noteRead keep Cycle small, and other read errors fail the cycle as before. CI green.
A 404 scope stopped failing the cycle, but an uninstalled App makes every scope 404, which then looked healthy while no job got a runner. Such a cycle now fails as before. A single scope that stays 404 for ALERT_FAILED_CYCLES cycles raises one alert of its own, with a message when it reads again, so a deleted repo is noticed and removed instead of warned about forever.
A readable scope always has a zero 404 count, so the loop already returns there; the guard only hid the single-scope and empty-pool cases from the tests, which now cover them.
|
Review fixed and ready for review again.
Tests: all-404 fails and alerts, then recovers. A single-scope pool and an empty pool are covered. One 404 scope among readable ones alerts once and clears. The existing "does not blame a 404 scope" test still holds. Next: review and merge. Nothing is waiting on me. |
parsilver
left a comment
There was a problem hiding this comment.
No Blocker. The earlier Blocker is fixed:
- Every scope 404 fails the cycle.
noneReadablefails it only when every configured scope answered 404 this cycle. That includes a one-scope pool, but not an empty one. So/healthzgoes 503 and the failing alert fires afterALERT_FAILED_CYCLES. - One missing scope among readable ones keeps the cycle green. It sends one missing alert, guarded by
goneAlerted, and one "readable again" message when it recovers. - Other errors still fail the cycle, and the stuck list names only the scopes that failed for a reason other than 404.
The counter maps are keyed by cfg.Scopes and change only under c.mu. go test -race ./... passes. ErrNotFound wraps only 404, and DeleteRunner still treats an already-gone runner as before.
Suggestions:
controller.go:348-360: a scope that alternates between 404 and 500 resets both counters each time. It never reaches the missing alert or the stuck list, so it can stay unhealthy without ever alerting.controller.go:404: when every scope 404s, the failing alert names no scope and relies on the last error, which is cut at 300 characters. Opening it with "no scope readable: is the App installed?" would say what is wrong.ghapp.go:420: a 404 from one repo inside an org scope is now skipped with no log. A warning at least once per repo would surface a broken repo.
Nits: the counters live in memory only, so a restart repeats a missing alert and loses the "readable again" message; the README could say so. No test pins the case where every scope 404s and then some recover. Merging.
Closes #49
On 2026-11-17 three deleted repositories in
RUNNER_SCOPESanswered 404 every cycle, each cycle counted as failed, and/healthzwent 503 (#48). Every other scope's queued jobs waited too.What changed
ghapp: a GitHub answer of 404 is now an error wrapping the newErrNotFound(the message still names the method, path and status). This covers the installation lookup (after a restart), the token request, and the runs listing (while the installation id is still cached, which is what happened in controller: /healthz 503, 15 consecutive failed cycles since 2026-11-17 17:33Z #48). In an org scope, a repository or run that disappears between the listing and the read is skipped, so one deleted repository no longer hides the org's other repositories.QueuedJobsis split intoorgRepos,repoJobsandrunJobs, and the status-to-error decision moved intostatusErrorsocalldoes not grow.controller: reading a scope is one helper (readScope), reading them all isreadScopes, andnoteReaddecides what each result means. A 404 skips the scope, lists it inunreadable_scopesand logs one warn line per cycle (scope not found on GitHub; skipped), but does not fail the cycle. Such a scope is also kept out of the failing alert's "scopes unreadable that long", so an unrelated outage does not blame it. Every other read error fails the cycle as before.noneReadable), so/healthzgoes 503 and the failing alert fires afterALERT_FAILED_CYCLES; its last error names the scopes and asks whether the App is still installed. A scope that answers 404 forALERT_FAILED_CYCLEScycles in a row while others still read raises one alert of its own (controller: scope repo:acme/app not found on GitHub for 3 cycles in a row; remove it from RUNNER_SCOPES, or check the GitHub App is installed on it), never repeated, and oneis readable againmessage when it reads again (alertMissing). In an all-404 pool no per-scope alert is sent on top of the failing alert./healthzparagraph and a troubleshooting row) and CHANGELOG[Unreleased].docs/deploy.mdonly says to watch/healthzand needs no change.Acceptance criteria
unreadable_scopes, warned once per cycle, and/healthzstays 200.TestAScopeThatIsNotFoundIsWarnedAboutButTheCycleSucceedsrunsALERT_FAILED_CYCLES + 1cycles with a 404 scope: every cycle returns nil,/healthzis 200 withconsecutive_failures: 0anderror: false,unreadable_scopesis exactly the 404 scope, no alert, and exactly one warn line per cycle.TestAScopeWhoseRunnersAreNotFoundDoesNotFailTheCyclecovers the runners listing.TestAReadErrorOtherThanNotFoundStillFailsTheCycle(a 502), the existingTestEveryCycleFailingAlertsOnceAnd503sThenRecoversandTestOneUnreadableScopeIsNamedAndTheOtherKeepsWorking(unchanged, still pass), andTestOtherFailuresAreNotErrNotFoundin ghapp (500, 403, 401, 429 are notErrNotFound).TestTheFailingAlertDoesNotBlameANotFoundScopechecks the alert text.ghapp tests:
TestARepoWhoseRunsAnswer404IsErrNotFound,TestARepoWhoseInstallationLookupAnswers404IsErrNotFound,TestAnOrgWhoseInstallationLookupAnswers404IsErrNotFound,TestAnInstallationWhoseTokenRequestAnswers404IsErrNotFoundAndForgotten,TestAnOrgSkipsARepoOrRunThatVanishedMidRead.The tests went in first and failed for the expected reason:
err ghapp: GET /repos/acme/app/actions/runs: HTTP 404, want ErrNotFoundandcycle 1: repo:acme/app: ... HTTP 404 ..., want success.TestEveryScopeNotFoundFailsTheCycleAndAlerts: each ofALERT_FAILED_CYCLEScycles fails,/healthzis 503, exactly one failing alert naming both scopes, then one recovered message when the scopes read again.TestASingleScopePoolThatIsNotFoundFailsAndAnEmptyPoolDoesNotcovers a pool of one scope and a pool of none.TestAMissingScopeIsAlertedOnceAndClearedWhenItReadsAgain: no alert beforeALERT_FAILED_CYCLEScycles, exactly one after (not repeated over 2 more cycles), then one readable-again message and an emptyunreadable_scopes. The first test above now expects that one missing-scope alert.TestTheFailingAlertDoesNotBlameANotFoundScopestill checks that the failing alert does not name the 404 scope.The review tests went in first and failed for the expected reasons:
cycle 1 succeeded, want it to fail when no scope is readableandalerts [], want one starting "controller: scope repo:acme/app not found on GitHub for 3 cycles in a row".Checks
After the review fixes, on the head commit:
Quality gates, changed functions only (no
.quality/gate.pyin this repo, so the global tools were run locally):noteRead4.0,readScope3.0,unreadableScopes5.0,QueuedJobs8.0,orgRepos3.0,repoJobs7.0,runJobs5.0,statusError2.0), except two that were already over 10 on main and did not get worse:Cycle18.5 on main, 18.2 here;call19.2 on main, 19.2 here.QueuedJobswent from 16.1 on main to 8.0.readScopes5.0,noneReadable4.0,alertMissing6.0,unreadableScopes5.0,noteRead4.0;Cyclewent down to 15.2 (18.5 on main) because the read loop moved intoreadScopes. No duplicate candidates. Mutation on the lines changed after review: 19 of 19 killed. A first run left 5 survivors, which pointed at a guard innoneReadablethat did nothing and at missing single-scope, empty-pool and one-cycleunreadable_scopestests; the guard is gone and the tests are added.--no-coverage; CRAP shows the changed functions at 95 to 100% coverage.Risks and trade-offs
RUNNER_SCOPESor an App no longer installed on a repo or org. Those used to turn/healthz503 within three cycles; now they stay 200, show inunreadable_scopes, and log a warn line every cycle. The README troubleshooting row says so. A scope like that now raises one alert afterALERT_FAILED_CYCLEScycles, and if every scope answers 404 the cycle fails as before. A pool where most scopes are 404 and one still reads stays green; each missing scope gets its own alert.RUNNER_SCOPEShas the new name (the client never follows redirects).Not verified