Add Actions dashboard for Grafana - #5824
Draft
backspace wants to merge 13 commits into
Draft
Conversation
GitHub's Actions UI cannot say how deep the queue is, how long jobs have been waiting, or which branches and people are holding the runners. Establishing that during the recent archive outage meant scripting the REST API by hand, and the answer disappeared as soon as the terminal scrolled. A collector samples the Actions API and prints one JSON line per observation, in the same shape every service here already logs, so the existing Alloy and FireLens pipelines carry it to Loki without special handling. A dashboard in the boxel-status folder reads it back: queue depth and wait percentiles over time, runners consumed by branch, author and workflow, and a table of in-progress jobs ranked by how long their current step has run — the signal that separates a wedged job from a slow one. Grouped depth is emitted as its own lines rather than nested on the snapshot, because LogQL flattens nested JSON into one label per key, which for branch names yields a label per branch instead of a series grouped by branch. Two constraints shape the collector. It must not run as a scheduled Actions workflow, since it would queue behind the backlog it measures and go blind during the incident it exists for. And a sample costs one request per active run, so at the ~60 runs this repository sustains, a one-minute interval would consume most of a token's hourly REST budget; the interval is two minutes and a sample is skipped rather than reported partially when the remaining budget nears its reserve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uction The collector is a single file that needs nothing beyond the Node runtime and global fetch, so its image is the runtime plus that file — no package install and no lockfile. Deployment is workflow_dispatch only. Putting the deploy of a queue sampler on a CI trigger would make it queue behind the backlog it reports on, which is the same reason the collector itself does not run inside Actions. The hosted task reads its credential from ACTIONS_COLLECTOR_GITHUB_TOKEN rather than a bare GITHUB_TOKEN, because its SSM path is shared with every other boxel service and a generic name there would be ambiguous. The plain name still works for local runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The token was documented as needing `actions: read`, which is wrong for this repository and misleading about why it exists. Both endpoints the collector reads answer unauthenticated on a public repository — the token's only job is the rate limit, which authentication raises from 60 requests an hour to 5,000, against a single sample that can cost sixty. So no permissions are needed while boxel is public. The private-repository requirement is stated separately, since that is when Actions: Read-only would start to matter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Observability diff (vs staging)Show diffdiff --git a/tmp/committed-canon.Sp4nKF/dashboards/boxel-status/actions-queue.json b/tmp/committed-canon.Sp4nKF/dashboards/boxel-status/actions-queue.json
new file mode 100644
index 0000000..6d70069
--- /dev/null
+++ b/tmp/committed-canon.Sp4nKF/dashboards/boxel-status/actions-queue.json
@@ -0,0 +1,1274 @@
+{
+ "apiVersion": "dashboard.grafana.app/v1beta1",
+ "kind": "Dashboard",
+ "metadata": {
+ "annotations": {
+ "grafana.app/folder": "defd2d156sav4d"
+ },
+ "name": "boxel-actions-queue"
+ },
+ "spec": {
+ "annotations": {
+ "list": []
+ },
+ "description": "GitHub Actions queue depth, wait times and runner consumption, sampled from the Actions REST API and shipped as JSON log lines on channel boxel:actions-queue.",
+ "editable": true,
+ "fiscalYearStartMonth": 0,
+ "graphTooltip": 0,
+ "links": [],
+ "panels": [
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "panels": [],
+ "title": "Queue depth",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Jobs waiting for a runner at the most recent sample.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 0,
+ "y": 1
+ },
+ "id": 3,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()",
+ "legendFormat": "queued",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Jobs queued now",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Jobs currently holding a runner.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 4,
+ "y": 1
+ },
+ "id": 4,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()",
+ "legendFormat": "running",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Jobs running now",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Workflow runs in queued or in_progress state.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 8,
+ "y": 1
+ },
+ "id": 5,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval]) by ()",
+ "legendFormat": "runs",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Active runs",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued depth rising while running stays flat means the pool is saturated, not slow.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 1
+ },
+ "id": 6,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()",
+ "legendFormat": "queued",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()",
+ "legendFormat": "running",
+ "queryType": "range",
+ "refId": "B"
+ }
+ ],
+ "title": "Queue depth over time",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Age of the oldest job still waiting for a runner.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 12,
+ "x": 0,
+ "y": 5
+ },
+ "id": 7,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "longest wait",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Longest current wait",
+ "type": "stat"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 18
+ },
+ "id": 8,
+ "panels": [],
+ "title": "Wait times",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "How long jobs have been waiting, measured from job creation.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ },
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 19
+ },
+ "id": 9,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.5, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.9, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "p90",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "max",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "Queue wait percentiles",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 27
+ },
+ "id": 10,
+ "panels": [],
+ "title": "Who is consuming the pool",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 28
+ },
+ "id": 11,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by branch",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 28
+ },
+ "id": 12,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by author",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 28
+ },
+ "id": 13,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by workflow",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by branch.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 36
+ },
+ "id": 14,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by branch",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by actor.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 36
+ },
+ "id": 15,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by author",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by workflow.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 36
+ },
+ "id": 16,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by workflow",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 44
+ },
+ "id": 17,
+ "panels": [],
+ "title": "Long-running steps",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "A job whose current step has run far longer than its peers is the wedge signal — compare against the same step on other shards before concluding it is stuck.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "job"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Open this job on GitHub",
+ "url": "https://github.com/cardstack/boxel/actions/runs/${__data.fields.run_id}/job/${__data.fields.job_id}"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "head_branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "actor"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "GitHub profile",
+ "url": "https://github.com/${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 45
+ },
+ "id": 18,
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step, run_id, job_id) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "In-progress jobs by current step duration",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true,
+ "job_id": true,
+ "run_id": true
+ },
+ "renameByName": {
+ "Value": "in step",
+ "Value #A": "in step"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 55
+ },
+ "id": 19,
+ "panels": [],
+ "title": "Collector health",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Sampling stops rather than reporting partial depth when this nears the reserve.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 0,
+ "y": 56
+ },
+ "id": 20,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "min_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap rate_limit_remaining [$__interval]) by ()",
+ "legendFormat": "remaining",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "GitHub API requests remaining",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Gaps in the series above should have a matching line here.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 12,
+ "y": 56
+ },
+ "id": 21,
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "{service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=~\"collector-error|collector-throttled\"",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Collector errors and throttling",
+ "type": "logs"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 24,
+ "x": 0,
+ "y": 62
+ },
+ "id": 1,
+ "options": {
+ "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*).",
+ "mode": "markdown"
+ },
+ "targets": [],
+ "title": "About this dashboard",
+ "type": "text"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 9
+ },
+ "id": 40,
+ "panels": [],
+ "title": "Right now",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "A point-in-time count, unlike the timeseries above — a branch whose jobs have finished disappears from this table instead of leaving a line behind.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 10
+ },
+ "id": 41,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued now, by branch",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "branch"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Which branches are occupying the pool at this moment.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 10
+ },
+ "id": 42,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Runners held now, by branch",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "branch"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "The same instant count grouped by whoever triggered the run.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "author"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "GitHub profile",
+ "url": "https://github.com/${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 10
+ },
+ "id": 43,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued now, by author",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "author"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ }
+ ],
+ "refresh": "1m",
+ "schemaVersion": 42,
+ "tags": [
+ "ci",
+ "actions",
+ "forensics"
+ ],
+ "templating": {
+ "list": [
+ {
+ "hide": 2,
+ "name": "env",
+ "query": "__ENV__",
+ "skipUrlSync": true,
+ "type": "constant"
+ }
+ ]
+ },
+ "time": {
+ "from": "now-6h",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "browser",
+ "title": "GitHub Actions Queue",
+ "weekStart": ""
+ }
+}
(Run: https://github.com/cardstack/boxel/actions/runs/32319536118) |
Contributor
Grafana previewPreview deployed for 1 dashboard in the staging Grafana.
Dashboards: Preview is torn down automatically when this PR is closed or merged. (Run: https://github.com/cardstack/boxel/actions/runs/32319536121) |
…hub-actions-backlog-visible-in-grafana
A workflow_dispatch workflow can only be dispatched once its file is on the default branch, so until this merges there is no way to build the collector — and with no collector running, the dashboard previews on this PR with every panel empty, which is indistinguishable from a query that is simply wrong. A push trigger scoped to this branch closes that loop, letting the queries be checked against real data before merging. The staging environment carries no branch policy and the OIDC trust policy for the deploy role admits any ref of this repository, so a branch build is permitted. `inputs.environment` is null on a push event, so the environment default is now explicit at each use — worth keeping after the trigger goes. Remove the push trigger before merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub returns 403 for a rejected credential, for a token whose permissions do not cover the resource, and for an exhausted rate limit alike, distinguishing them only in the response body. The emitted error carried just the status, so a collector sitting at 403 said nothing about which of those it was. Carrying a bounded slice of the body and the remaining request budget into the error puts that in the collector-error line, where the dashboard already shows it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every metric panel drew one series per sample rather than one line. `| json` promotes each field of a record to a label, so successive samples never shared a label set and a range aggregation without an explicit `by ()` grouped none of them together. The stat tiles showed a row of numbers instead of a value, and the panels reading per-job fields returned so many series they rendered as no data at all. Every one of these queries has now been run against staging Loki: each returns a single series, and the grouped panels return one per group. The long-running steps table gains the actor, since the first question asked of a job that has sat in one step for hours is whose it is. The temporary push trigger goes with them, having served its purpose — the collector is deployed and reporting, which is what let these queries be checked against real data rather than merged on the assumption they were right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hub-actions-backlog-visible-in-grafana
The grouped panels drew a series per branch and listed them alphabetically, so the branch holding most of the queue could sit below the fold while smaller ones were named above it — the panel answered "which branches have queued jobs" when the question being asked of it is "which branch has the most". The legend is now a table sorted by last value, showing last and max, and each grouped query is bounded to the ten largest series at any instant so a busy hour cannot push the ones that matter out of view. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The collector samples every two minutes, but Grafana derives $__interval from panel width and on a multi-hour window that lands well under the sample period. Most range windows therefore covered no sample at all — at thirty seconds the queries return nothing — so the panels rendered as isolated spikes rather than a series, and the wait panels in particular looked like noise. Flooring the panel interval at five minutes keeps every window covering at least one sample, and spanning nulls bridges a gap where a sample is missed rather than breaking the line. Every query was re-run against staging Loki at the floored interval: each returns a continuous series where it previously returned a handful of scattered points. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The description occupied the top of the page, so opening the dashboard during an incident meant scrolling past prose to reach the queue depth. It reads better as an appendix: useful once, when someone wants to know where the numbers come from, and in the way every other time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The grouped panels are timeseries over the dashboard's window, so a branch that was queue-heavy an hour ago keeps a line long after its jobs have gone. Read at a glance during an incident that says "this branch has the most queued", when what it means is "this branch had the most queued at some point in the last six hours" — twice now that has sent someone looking for jobs that had already finished. A "Right now" row of instant queries answers the other question directly: what is queued, and who is holding runners, at this moment. Rows with nothing queued are filtered out, so a branch drops off the table when its work is done rather than lingering at zero. The timeseries stay. "How did we get here" and "what is true now" are different questions and the history is the right shape for the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeing that a job has sat in one step for four hours raises exactly one question, and answering it meant copying a job name into the Actions search by hand. The rows now carry links on the text already in them, so no column shows a URL. A job opens its own page on GitHub, which needs the run and job identifiers, so the query carries them and the transformation hides the columns — they are identifiers rather than information. A branch opens the Actions runs filtered to it, and an author their profile. Every link was resolved against a live row and fetched; all six return 200. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.