Alerts for skipped tests/models stay pending in alerts_v2 forever (not sent, not marked skipped) #2385

Describe the bug

When a dbt test is skipped (for example because its upstream model failed in dbt build), edr monitor inserts an alert for it into alerts_v2, but never processes it afterwards. The default status filter of edr monitor drops it before sending, and dropped alerts are not marked skipped either. So the row stays status = 'pending' forever.

In a long-running project these rows pile up indefinitely. Anyone who queries alerts_v2 sees a large number of "pending" alerts and can reasonably read it as "alerting is broken", although nothing will ever be sent for them.

As far as I can tell from the code (I may be missing something):

• The population query inserts every test result that is not pass / reused, and every model result that is not success / reused, so skipped is included:
  • test_alerts.sql#L124
  • model_alerts.sql#L123
• edr monitor filters by status fail / error / runtime error / warn by default:
  • schema.py#L138-L142
• Only alerts that survive the filter are passed to _sort_alerts, which decides between "send" and "skip". Alerts removed by the filter get neither:
  • data_monitoring_alerts.py#L433
• The internal test validate_alert_statuses_are_updated already carves these rows out ("By default we don't send skipped models"), which suggests the leftover pending rows are known:
  • validate_alert_statuses_are_updated.sql#L13
• The dbt package vars disable_skipped_test_alerts / disable_skipped_model_alerts (both true by default, added in Disable skipped tests alerts #354 / Disable skipped model alerts #300) do not seem to affect alerts_v2 population. Setting them explicitly made no difference in the reproduction below.

#2363 fixed the same pattern for the reused status. skipped seems to be the remaining case.

To Reproduce

A self-contained project on DuckDB. No real Slack is needed: a local stand-in webhook answers 200 ok.

1. Install: pip install "elementary-data[duckdb]==0.26.0" "dbt-core==1.10.*" "dbt-duckdb==1.10.*"
2. Create the project:

dbt_project.yml

name: repro
version: "1.0"
profile: repro
models:
  elementary:
    +schema: elementary

packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.26.0

models/upstream_broken.sql (fails at run time, so the test on it is skipped)

select * from table_that_does_not_exist

models/healthy.sql

select 1 as id

models/schema.yml (accepted_values on healthy is a control that fails normally)

version: 2
models:
  - name: upstream_broken
    columns:
      - name: id
        data_tests:
          - not_null
  - name: healthy
    columns:
      - name: id
        data_tests:
          - accepted_values:
              values: [2]

profiles.yml (the path must be absolute, because edr runs dbt from its own bundled project directory)

repro:
  target: dev
  outputs:
    dev: {type: duckdb, path: "{{ env_var('REPRO_DB') }}", schema: main}
elementary:
  target: dev
  outputs:
    dev: {type: duckdb, path: "{{ env_var('REPRO_DB') }}", schema: main_elementary}

fake_slack_webhook.py

from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        self.send_response(200); self.end_headers(); self.wfile.write(b"ok")
HTTPServer(("127.0.0.1", 8765), H).serve_forever()

3. Run:

export DBT_PROFILES_DIR=. REPRO_DB="$PWD/repro.duckdb"
dbt deps
dbt run -s elementary
dbt build            # upstream_broken: ERROR, not_null_upstream_broken_id: SKIP, accepted_values_healthy_id__2: FAIL
python fake_slack_webhook.py &
edr monitor --profiles-dir . --project-dir . --slack-webhook http://127.0.0.1:8765/hook

4. Check alerts_v2:

select type,
       json_extract_string(data, '$.test_name')       as test_name,
       json_extract_string(data, '$.model_unique_id') as model,
       json_extract_string(data, '$.status')          as result_status,
       status                                          as alert_status
from main_elementary.alerts_v2;

Result:

The monitor log says Update skipped alerts (0) and Update sent alerts (2).

Running dbt build and edr monitor a second time (also with --dbt-vars '{"disable_skipped_test_alerts": true, "disable_skipped_model_alerts": true}') adds one more skipped / pending row, and the first one is still pending. Once detected_at falls out of --days-back, nothing will ever read these rows again.

Expected behavior

An alert that edr monitor has decided not to send should not stay pending. Either of these would work for us:

1. Do not insert alerts for skipped results when disable_skipped_test_alerts / disable_skipped_model_alerts are on, the same way fix: treat reused status as success/pass in alert population #2363 handles reused, or
2. Mark alerts removed by the status filter as skipped in alerts_v2, so pending only means "not processed yet".

Option 1 seems closer to the existing vars. But the status filter is configurable (--filters statuses:...), so option 2 would also cover other statuses that a filter excludes.

Screenshots

N/A

Environment (please complete the following information):

• Elementary CLI (edr) version: 0.26.0
• Elementary dbt package version: 0.26.0
• dbt version you're using: dbt-core 1.10.23 (reproduction), also seen with dbt-core 1.12.5 on BigQuery
• Data warehouse: DuckDB with dbt-duckdb 1.10.1 (reproduction), BigQuery with dbt-bigquery 1.12.1 (where we first noticed it)
• Infrastructure details: reproduction on macOS, local. In production, edr monitor runs daily on a container job with --days-back 7

Additional context

As a workaround, we run a daily job that sets status = 'skipped' on these rows, using the same UPDATE that update_skipped_alerts issues.

Would you be willing to contribute a fix for this issue?

Yes, I'd be happy to open a PR if you can confirm which direction you prefer.