obs(newrelic): P0 log-based alert for unsanctioned destructive DDL on postgres-customers (truehomie DDL trap, D3)

[mastermanas805]

What

Task D3 deliverable 4 — wires the postgres-customers DDL-logging trap (set during the 2026-06-03 truehomie-db incident) to a P0 NR alert, plus a dashboard page.

• newrelic/alerts/customer-db-destructive-ddl.json — CRITICAL, FROM Log (metric-based NR alerting is not live in prod — no Prometheus pipeline). Fires when the postgres-customers pod (k8s_namespace_name='instant-data', k8s_label_app='postgres-customers') logs a DROP DATABASE/ROLE/USER/OWNED statement not accounted for by a sanctioned provisioner drop (event=provisioner.drop — server.guardedDrop for RPC drops, pool.deprovisionBacking caller='pool_reaper' for hot-pool reaps) in the same 15-minute window, with a 4× DDL budget per sanctioned shared-pg drop (up to 3 retried DROP DATABASE attempts + 1 DROP USER). The truehomie signature — drops with ZERO provisioner.drop events — always pages. Full triage runbook in the condition description.
• newrelic/dashboards/admin-defense.json — new third page "customer-db DDL trap": unsanctioned-delta billboard, DDL-vs-sanctioned timeseries, raw trap lines table, sanctioned-drops + dropguard-refusals table. Purely additive (106 insertions, 0 deletions).
• newrelic/CHANGES.md — entry with the upstream dependency.

Discovered trap log shape (live, read-only kubectl on do-nyc3-instant-prod)

• Trap = ALTER SYSTEM SET log_statement='ddl' + log_connections=on (2026-06-03 incident response; persists in postgresql.auto.conf on the PVC — the connection received/authorized lines visible in the current pod log prove the settings survived the 2026-06-06 Recreate).
• DDL line shape: standard postgres stderr format, e.g.
  2026-06-10 18:55:14.433 UTC [1704166] LOG: statement: DROP DATABASE "db_96edf9eed8ed42929036b63298ec5b2b" (extended-protocol clients log LOG: execute <name>: DROP ...) — the NRQL matches the DROP fragment, not the prefix. 26,777 pod log lines in the last 96h contain ZERO statement: lines — i.e. no DDL ran in that window; the trap is armed and quiet.

Ordering dependency

Provisioner PR #56 (InstaNode-dev/provisioner) adds the pool_reaper ledger entry + dropguard. Merge #56 before applying this alert, or hot-pool reaps of failed postgres items will false-positive.

Rule-17 coverage block

Symptom:        unaudited DROP DATABASE/ROLE on postgres-customers pages nobody (truehomie 2026-06-03)
Enumeration:    kubectl logs (96h) + rg over provisioner/api/worker/common for all DROP emitters; existing alert JSONs greped for ddl/drop coverage (only metric-based provisioner-drop-*.json existed — not live)
Sites found:    1 alert gap (no log-based DDL alert), 1 dashboard gap
Sites touched:  2 (alert JSON + dashboard page) + CHANGES.md
Coverage test:  newrelic JSONs validated with python json.tool (apply.test.sh is pre-existing-broken on master — committed merge-conflict markers at line 174 from PR #14 + stale dry-run count; flagged, not fixed here)
Live verified:  awaiting operator apply (newrelic/apply.sh — this repo has no auto-apply by design)

Pre-existing finding (not fixed here)

newrelic/tests/apply.test.sh on master contains unresolved merge-conflict markers (line 174, from PR #14 squash) and a stale expected-count baseline (33 vs 98 JSON files) — the NR test suite has been un-runnable since that merge. Recommend a follow-up PR.

🤖 Generated with Claude Code