Skip to content

fix(controller): a 404 scope in RUNNER_SCOPES no longer fails the cycle - #50

Merged
parsilver merged 9 commits into
mainfrom
fix/49-not-found-scopes
Nov 19, 2026
Merged

parsilver merged 9 commits into
mainfrom
fix/49-not-found-scopes

Conversation

@parsilver

@parsilver parsilver commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #49

On 2026-11-17 three deleted repositories in RUNNER_SCOPES answered 404 every cycle, each cycle counted as failed, and /healthz went 503 (#48). Every other scope's queued jobs waited too.

What changed

  • ghapp: a GitHub answer of 404 is now an error wrapping the new ErrNotFound (the message still names the method, path and status). This covers the installation lookup (after a restart), the token request, and the runs listing (while the installation id is still cached, which is what happened in controller: /healthz 503, 15 consecutive failed cycles since 2026-11-17 17:33Z #48). In an org scope, a repository or run that disappears between the listing and the read is skipped, so one deleted repository no longer hides the org's other repositories. QueuedJobs is split into orgRepos, repoJobs and runJobs, and the status-to-error decision moved into statusError so call does not grow.
  • controller: reading a scope is one helper (readScope), reading them all is readScopes, and noteRead decides what each result means. A 404 skips the scope, lists it in unreadable_scopes and logs one warn line per cycle (scope not found on GitHub; skipped), but does not fail the cycle. Such a scope is also kept out of the failing alert's "scopes unreadable that long", so an unrelated outage does not blame it. Every other read error fails the cycle as before.
  • After review: when every scope answers 404 (the App uninstalled, or its installation gone), the cycle fails as before (noneReadable), so /healthz goes 503 and the failing alert fires after ALERT_FAILED_CYCLES; its last error names the scopes and asks whether the App is still installed. A scope that answers 404 for ALERT_FAILED_CYCLES cycles in a row while others still read raises one alert of its own (controller: scope repo:acme/app not found on GitHub for 3 cycles in a row; remove it from RUNNER_SCOPES, or check the GitHub App is installed on it), never repeated, and one is readable again message when it reads again (alertMissing). In an all-404 pool no per-scope alert is sent on top of the failing alert.
  • README (the /healthz paragraph and a troubleshooting row) and CHANGELOG [Unreleased]. docs/deploy.md only says to watch /healthz and needs no change.

Acceptance criteria

  • A 404 scope is in unreadable_scopes, warned once per cycle, and /healthz stays 200. TestAScopeThatIsNotFoundIsWarnedAboutButTheCycleSucceeds runs ALERT_FAILED_CYCLES + 1 cycles with a 404 scope: every cycle returns nil, /healthz is 200 with consecutive_failures: 0 and error: false, unreadable_scopes is exactly the 404 scope, no alert, and exactly one warn line per cycle. TestAScopeWhoseRunnersAreNotFoundDoesNotFailTheCycle covers the runners listing.
  • Any other error still fails the cycle. TestAReadErrorOtherThanNotFoundStillFailsTheCycle (a 502), the existing TestEveryCycleFailingAlertsOnceAnd503sThenRecovers and TestOneUnreadableScopeIsNamedAndTheOtherKeepsWorking (unchanged, still pass), and TestOtherFailuresAreNotErrNotFound in ghapp (500, 403, 401, 429 are not ErrNotFound). TestTheFailingAlertDoesNotBlameANotFoundScope checks the alert text.
  • A 404 scope beside readable scopes, and runners are still created for them. The first test above queues a job in the readable org scope and asserts a runner service was created for it.

ghapp tests: TestARepoWhoseRunsAnswer404IsErrNotFound, TestARepoWhoseInstallationLookupAnswers404IsErrNotFound, TestAnOrgWhoseInstallationLookupAnswers404IsErrNotFound, TestAnInstallationWhoseTokenRequestAnswers404IsErrNotFoundAndForgotten, TestAnOrgSkipsARepoOrRunThatVanishedMidRead.

The tests went in first and failed for the expected reason: err ghapp: GET /repos/acme/app/actions/runs: HTTP 404, want ErrNotFound and cycle 1: repo:acme/app: ... HTTP 404 ..., want success.

  • After review: every scope 404 fails the cycle and alerts. TestEveryScopeNotFoundFailsTheCycleAndAlerts: each of ALERT_FAILED_CYCLES cycles fails, /healthz is 503, exactly one failing alert naming both scopes, then one recovered message when the scopes read again. TestASingleScopePoolThatIsNotFoundFailsAndAnEmptyPoolDoesNot covers a pool of one scope and a pool of none.
  • After review: one 404 scope among readable ones alerts once and clears. TestAMissingScopeIsAlertedOnceAndClearedWhenItReadsAgain: no alert before ALERT_FAILED_CYCLES cycles, exactly one after (not repeated over 2 more cycles), then one readable-again message and an empty unreadable_scopes. The first test above now expects that one missing-scope alert. TestTheFailingAlertDoesNotBlameANotFoundScope still checks that the failing alert does not name the 404 scope.

The review tests went in first and failed for the expected reasons: cycle 1 succeeded, want it to fail when no scope is readable and alerts [], want one starting "controller: scope repo:acme/app not found on GitHub for 3 cycles in a row".

Checks

go vet ./...                                            clean
gofmt -l .                                              nothing
go test -race -count=1 ./internal/controller ./internal/ghapp
ok  github.com/farzai/github-runner-railway/internal/controller  3.491s
ok  github.com/farzai/github-runner-railway/internal/ghapp       6.727s

After the review fixes, on the head commit:

go vet ./...                       clean
gofmt -l .                         nothing
go test -race -count=1 ./...
ok  github.com/farzai/github-runner-railway/internal/config      1.423s
ok  github.com/farzai/github-runner-railway/internal/controller  3.231s
ok  github.com/farzai/github-runner-railway/internal/ghapp       8.455s
ok  github.com/farzai/github-runner-railway/internal/railway     1.568s
ok  github.com/farzai/github-runner-railway/internal/reconcile   1.853s
ok  github.com/farzai/github-runner-railway/internal/webhook     1.446s

Quality gates, changed functions only (no .quality/gate.py in this repo, so the global tools were run locally):

  • CRAP: every new or changed function is under 10 (noteRead 4.0, readScope 3.0, unreadableScopes 5.0, QueuedJobs 8.0, orgRepos 3.0, repoJobs 7.0, runJobs 5.0, statusError 2.0), except two that were already over 10 on main and did not get worse: Cycle 18.5 on main, 18.2 here; call 19.2 on main, 19.2 here. QueuedJobs went from 16.1 on main to 8.0.
  • Duplication (dryer, threshold 0.8) on both changed files: no candidates.
  • After review: readScopes 5.0, noneReadable 4.0, alertMissing 6.0, unreadableScopes 5.0, noteRead 4.0; Cycle went down to 15.2 (18.5 on main) because the read loop moved into readScopes. No duplicate candidates. Mutation on the lines changed after review: 19 of 19 killed. A first run left 5 survivors, which pointed at a guard in noneReadable that did nothing and at missing single-scope, empty-pool and one-cycle unreadable_scopes tests; the guard is gone and the tests are added.
  • Mutation (mutator, changed lines only, first round): controller 9 of 9 killed, ghapp 20 of 20 killed, no survivors. The tool could not match Go's module-path coverage profile to the files, so it ran with --no-coverage; CRAP shows the changed functions at 95 to 100% coverage.

Risks and trade-offs

  • A 404 also means a typo in RUNNER_SCOPES or an App no longer installed on a repo or org. Those used to turn /healthz 503 within three cycles; now they stay 200, show in unreadable_scopes, and log a warn line every cycle. The README troubleshooting row says so. A scope like that now raises one alert after ALERT_FAILED_CYCLES cycles, and if every scope answers 404 the cycle fails as before. A pool where most scopes are 404 and one still reads stays green; each missing scope gets its own alert.
  • A renamed or transferred repository answers a redirect, not a 404, and still fails the cycle until RUNNER_SCOPES has the new name (the client never follows redirects).
  • If the App comes back for only some scopes after an all-404 stretch, the scopes still 404 get their missing alert at once, since their 404 count kept running through the outage.
  • Railway services of a 404 scope are left alone, like those of any unreadable scope, and its late-job alert marks are never cleared. Both are bounded and unchanged in kind.

Not verified

A deleted repository in RUNNER_SCOPES answered 404 every cycle and the
controller counted it as a failed cycle, so /healthz went 503 (#48).
These tests pin the wanted behaviour: GitHub's 404 is ErrNotFound, and
a not-found scope is warned about while the cycle still succeeds. They
fail now: no call wraps ErrNotFound yet and the controller fails the
cycle.
The controller could not tell a deleted repository from an outage: every
non-2xx answer was the same opaque error. A 404 now wraps ErrNotFound.
In an org scope a repository or run that vanishes between the listing
and the read is skipped, so one deleted repository no longer hides the
org's other repositories. QueuedJobs is split into orgRepos, repoJobs
and runJobs to keep each piece small.

Proved by the new notfound tests (repo runs 404, repo and org
installation lookup 404, token request 404, an org repo and run that
vanish mid-read) and the existing ghapp suite.
Three deleted repositories in RUNNER_SCOPES failed 112 cycles in a row
and turned /healthz 503 (#48), so queued jobs of every other scope sat
without runners. A scope GitHub answers 404 for is now skipped, listed
in unreadable_scopes and warned about once a cycle, and the cycle still
succeeds. It is also kept out of the failing alert's scopes unreadable
that long, so an unrelated outage does not blame it. Every other read
error fails the cycle as before.

Proved by the controller notfound tests (jobs 404 and runners 404 with
a readable scope still getting runners, a 502 still failing, the alert
text) and the full suite with -race.
Operators need to know that a deleted or misspelled scope no longer
turns /healthz 503 and where to see it instead: unreadable_scopes and
the warn line. docs/deploy.md only says to watch /healthz and needs no
change.
The 404 branch had pushed call's CRAP score from 19.2 to 20.2, above
its score on main. statusError holds the status-to-error decision, so
call is back at 19.2 and the 404 rule sits in one small function.
Behaviour is unchanged: the ghapp suite passes with -race.
@parsilver
parsilver marked this pull request as ready for review November 19, 2026 06:01

@parsilver parsilver left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker:

  1. A whole-pool 404 now looks healthy (controller.go noteRead). Its own comment lists "an App no longer installed" as a 404 case. If the App is uninstalled, or its installation id goes stale, every scope answers 404: each cycle returns nil, /healthz stays 200, and no alert fires, while no CI job gets a runner. #48's three deleted repos were the right case to stop failing the cycle, but not a total outage. Fix both:

    • When every scope in a cycle is 404 (no scope readable), fail the cycle as before, so /healthz goes 503 and the failing alert fires after ALERT_FAILED_CYCLES.
    • Track consecutive 404 cycles per scope, and send one alert per scope (not a failure) once it has been missing for ALERT_FAILED_CYCLES cycles, with a recovered message when it reads again. A deleted repo left in RUNNER_SCOPES should be noticed and removed, not warned about silently forever.

    Tests: all scopes 404 fails the cycle and alerts. One scope 404 among readable ones keeps the cycle green, raises one missing-scope alert after N cycles, and clears it on recovery.

Checked and fine: ErrNotFound is set only on 404, not on 401, 403, 429 or 500, and that's tested. An org skips a repo or run that vanished mid-read. The token is forgotten on a 404. readScope/noteRead keep Cycle small, and other read errors fail the cycle as before. CI green.

A 404 scope stopped failing the cycle, but an uninstalled App makes
every scope 404, which then looked healthy while no job got a runner.
Such a cycle now fails as before. A single scope that stays 404 for
ALERT_FAILED_CYCLES cycles raises one alert of its own, with a
message when it reads again, so a deleted repo is noticed and removed
instead of warned about forever.
A readable scope always has a zero 404 count, so the loop already
returns there; the guard only hid the single-scope and empty-pool
cases from the tests, which now cover them.
@parsilver

Copy link
Copy Markdown
Contributor Author

Review fixed and ready for review again.

  • Every scope answering 404 now fails the cycle as before, so /healthz goes 503 and the failing alert fires after ALERT_FAILED_CYCLES. Its last error names the scopes and asks whether the App is still installed.
  • A scope that answers 404 for ALERT_FAILED_CYCLES cycles in a row, while other scopes still read, raises one alert of its own. It is not repeated, and one "readable again" message follows when the scope reads again.
  • The scope read loop moved into readScopes, so Cycle went down from 18.2 to 15.2 CRAP. README and CHANGELOG describe the new alert and the all-404 failure.

Tests: all-404 fails and alerts, then recovers. A single-scope pool and an empty pool are covered. One 404 scope among readable ones alerts once and clears. The existing "does not blame a 404 scope" test still holds. go test -race ./... passes locally, and CI's test job and image builds pass on 35de70c. Mutation testing on the changed lines killed 19 of 19.

Next: review and merge. Nothing is waiting on me.

@parsilver parsilver left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Blocker. The earlier Blocker is fixed:

  • Every scope 404 fails the cycle. noneReadable fails it only when every configured scope answered 404 this cycle. That includes a one-scope pool, but not an empty one. So /healthz goes 503 and the failing alert fires after ALERT_FAILED_CYCLES.
  • One missing scope among readable ones keeps the cycle green. It sends one missing alert, guarded by goneAlerted, and one "readable again" message when it recovers.
  • Other errors still fail the cycle, and the stuck list names only the scopes that failed for a reason other than 404.

The counter maps are keyed by cfg.Scopes and change only under c.mu. go test -race ./... passes. ErrNotFound wraps only 404, and DeleteRunner still treats an already-gone runner as before.

Suggestions:

  • controller.go:348-360: a scope that alternates between 404 and 500 resets both counters each time. It never reaches the missing alert or the stuck list, so it can stay unhealthy without ever alerting.
  • controller.go:404: when every scope 404s, the failing alert names no scope and relies on the last error, which is cut at 300 characters. Opening it with "no scope readable: is the App installed?" would say what is wrong.
  • ghapp.go:420: a 404 from one repo inside an org scope is now skipped with no log. A warning at least once per repo would surface a broken repo.

Nits: the counters live in memory only, so a restart repeats a missing alert and loses the "readable again" message; the README could say so. No test pins the case where every scope 404s and then some recover. Merging.

@parsilver
parsilver merged commit 7969a10 into main Nov 19, 2026
14 checks passed
@parsilver
parsilver deleted the fix/49-not-found-scopes branch November 19, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

controller: a deleted repo in RUNNER_SCOPES must not fail the whole cycle

1 participant