Skip to content

feat: add log-based CoderdPanic alert via the Loki ruler - #84

Open
ibetitsmike wants to merge 1 commit into
mainfrom
mike/coderd-panic-alert
Open

feat: add log-based CoderdPanic alert via the Loki ruler#84
ibetitsmike wants to merge 1 commit into
mainfrom
mike/coderd-panic-alert

Conversation

@ibetitsmike

Copy link
Copy Markdown

Moves the CoderdPanic alert from dogfood-specific Loki values into the chart, as requested by @dannykopping in coder/dogfood#454 (review comment: https://github.com/coder/dogfood/pull/454#discussion_r3896503702).

What

Adds the chart's first log-based (Loki ruler) alerting rule:

  • CoderdPanic fires when a coderd pod logs a fatal panic: / fatal error:, or recovers from an HTTP handler panic (panic serving http request), within the configured period (default 5m).
  • New coder-logs-alerts ConfigMap (templates/configmap-loki-alerts.yaml + templates/_loki-alerts.tpl), labeled loki_rule: "" so the Loki rules sidecar loads it into the ruler. The ruler already ships alerts to Alertmanager (rulerConfig.alertmanager_url).
  • Configured via global.coder.alerts.coderd.groups.Panics (enabled/period/thresholds), following the existing alert group conventions, with a runbook_url label and a new runbook section.

Notes on the matcher

  • Panic output is untimestamped, so the collector's multiline stage folds tracebacks into the preceding timestamped log entry; (?m)^ matches the embedded panic line start.
  • coderd.aibridge* loggers are excluded: they echo LLM chat content, which can quote Go tracebacks pasted by users (observed false positives on dev.coder.com).

Validation

  • make lint passes (helm lint --strict, pint, helm-docs README regen, compiled/ regen with helm v3.17.1 + yq v4.42.1).
  • The matcher was backtested against 7 days of live dev.coder.com Loki data in coder/dogfood#454 (100% true positives). The chart's coderd-selector-based variant was verified to return identical per-pod results to the backtested job-based selector on the same data.

Once released, dogfood will bump the chart version and keep only its Slack routing for the alert (coder/dogfood#454 is being reworked accordingly).

Disclosure: this PR was created by Xum (an AI agent) on behalf of @ibetitsmike.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant