feat(observability): add newrelic-prometheus-agent metrics ingestion pipeline

[mastermanas805]

The gap this closes

Prod (do-nyc3-instant-prod) shipped logs + APM + OTLP traces to New Relic but had no metrics ingestion pipeline — nothing scraped the services' /metrics. Verified read-only this session:

kubectl get pods -A | grep -iE 'prometheus|nri|otel-collector'   # → empty

Every backend emits a rich instant_* / provisioner_* Prometheus surface, but with no scraper the Metric event type was empty in NR. So ~46 NR alert conditions of the form FROM Metric WHERE metricName LIKE 'instant_%' were INERT — querying a stream that didn't exist:

grep -rl "FROM Metric" newrelic/alerts/ | wc -l   # → 46

What ships

The official newrelic-prometheus-agent (newrelic-prometheus-configurator 2.11.1 initContainer + Prometheus v3.12.0 --agent) as one stateless 2-container pod in the newrelic namespace. It scrapes the three backend /metrics and remote-writes instant_* series to NR's US Prometheus endpoint (metric-api.newrelic.com/prometheus/v1/write), lighting up every FROM Metric alert at once.

Chosen over the full nri-bundle (too heavy for the thin 6×2cpu/4GB pool) and over self-hosted Prometheus (another stateful pod to back up).

The 3 verified scrape targets (live, 2026-06-11)

Service	Reached via	Port / path	/metrics auth
instant-api	pod SD, ns instant, app=instant-api	8080 /metrics	Bearer required (401 without METRICS_TOKEN)
instant-worker	pod SD, ns instant-infra, app=instant-worker	8091 /metrics	open (200)
instant-provisioner	pod SD, ns instant-infra, app=instant-provisioner	8092 /metrics	open (200)

Pod service discovery, not static Service-DNS (deviation from the brief, flagged for the operator): the live topology rules out static targets — instant-worker has no Service; instant-provisioner's Service exposes only :50051 (gRPC), not the :8092 metrics sidecar. Pod SD discovers all three by app label and scrapes podIP:port directly, surviving pod evictions.

The METRICS_TOKEN bearer is sent to all three (required by api, harmlessly ignored by worker/provisioner's bare promhttp handlers) so the config stays correct if they gain gating later.

Secret wiring (referenced by name, never inlined)

One secret in the newrelic ns, newrelic-prometheus-agent-secrets, with copies of values that already exist:

Key	Live source
NEW_RELIC_LICENSE_KEY	instant-secrets (ns instant) + instant-infra-secrets (ns instant-infra) — same INGEST key the logging DaemonSet + go-agents use
METRICS_TOKEN	an inline value: env on all three Deployments (not a k8s secret key today) — same token across api/worker/provisioner

The manifest ships a Secret template with CHANGE_ME (repo convention). The operator populates real values out-of-band; no secret value is committed.

Validation (no real apply — operator-gated, CLAUDE.md rule 15)

• kubectl apply --dry-run=client → 6 objects "created (dry run)" ✓
• kubeconform -strict -ignore-missing-schemas -kubernetes-version 1.31.0 (exact CI validate.yml) → 6/6 valid ✓
• Full-repo kubeconform → 121 resources, 0 invalid, 0 errors ✓
• promtool check config on the rendered scrape config (real Prometheus v3.12.0) → SUCCESS ✓ (validates the k8s pod SD + bearer credentials_file + __address__ port-override relabels)

Post-apply verification gate

Within ~5 min of the pod running, run in NR:

FROM Metric SELECT count(*) WHERE metricName LIKE 'instant_%' SINCE 10 minutes ago

Non-zero → the 46 FROM Metric alerts go live. Highest-value ones that flip from inert to armed: razorpay-webhook-sig-fail, deploy-job-failed-detected, deploy-runtime-failed-detected, provisioner-circuit-open, pg-pool-saturation, email-delivery-ratio-low, auth-probe-fail, payment-probe-fail, redis-maxmemory-regrade-failed.

Accuracy correction: backup-stale-36h and customer-backup-failed are FROM Log alerts (already firing), not metric alerts — the brief grouped backup-stale with the metric set; it reads the backup CronJob's log line. Noted in the docs.

Files

• k8s/newrelic-prometheus-agent.yaml — SA + ClusterRole/Binding (least-priv pod SD read) + ConfigMap (configurator schema) + Deployment + Secret template
• OBSERVABILITY-PIPELINE.md — apply + verify runbook, secret prerequisites, the 46-alert gate
• k8s/APPLY-CHECKLIST.md — apply entry (incl. the placeholder-Secret clobber hazard)
• observability/METRICS-CATALOG.md — prerequisite note: the pipeline is a hard dependency of every Metric-based alert

Region assumption (operator to confirm)

NR datacenter assumed US (logging → log-api.newrelic.com, go-agents → otlp.nr-data.net, both US). The configurator auto-derives US vs EU from the license-key prefix, so no manifest change is needed if later confirmed EU.

🤖 Generated with Claude Code