fix(logmq): filter unsubscribed operator events before suppression

[samadalishah]

Summary

Filter unsubscribed operator events before entering their delivery suppression window.

This prevents an event that will never be emitted from:

• reading or writing suppression state in Redis;
• failing log processing because of an unrelated Redis error;
• producing a misleading opevent delivery failed log;
• nacking and redelivering an otherwise successfully persisted log message.

Problem

Operator-event topic filtering currently happens inside Emitter.Emit.

For events with a suppression key, logmq.BatchProcessor.send enters the Redis-backed suppression window before calling Emitter.Emit. This means the topic filter is applied too late.

For example:

OPERATOR_EVENTS_TOPICS=alert.destination.disabled

When an attempt exhausts its retries, the alert evaluator can still plan an alert.attempt.exhausted_retries event. Although that topic is not subscribed, the previous delivery path was:

1. Build the exhausted-retries event.
2. Enter its Redis suppression window.
3. Only then call Emitter.Emit.
4. Emitter.Emit sees that the topic is disabled and discards it.

As a result, an unsubscribed event could fail before reaching the component responsible for filtering it:

opevent delivery failed
topic=alert.attempt.exhausted_retries
error=context deadline exceeded

The log message was then nacked even though no exhausted-retries event was configured for delivery.

Root cause

Topic eligibility was checked after optional delivery-layer behavior:

suppression window -> emitter -> topic filter

Filtering needs to happen before any behavior specific to delivering the event:

topic filter -> suppression window -> emitter

Fix

BatchProcessor.send now checks Emitter.Enabled before entering the suppression window:

if !bp.alerts.Emitter.Enabled(de.event.Topic) {
    return nil
}

Emitter.Emit retains its existing topic check as a boundary safeguard.

Subscribed events continue through the same suppression and delivery path as before.

Tests

Added a regression test covering the reported configuration:

• only alert.destination.disabled is subscribed;
• the evaluated attempt also exhausts its retries;
• the exhausted-retries suppression dependency is configured to return context.DeadlineExceeded;
• the log message is acknowledged;
• alert.destination.disabled is delivered;
• alert.attempt.exhausted_retries is not delivered;
• the exhausted-retries suppression window is never called.

The existing delivery-suppression tests continue to cover subscribed exhausted-retry events, including:

• sequential window suppression;
• per-destination suppression;
• tenant isolation;
• concurrent suppression conflicts;
• suppression cleanup after delivery failure.

Impact

• No configuration or API changes.
• No behavior change for subscribed operator-event topics.
• Unsubscribed topics no longer perform delivery-related Redis work.
• Redis failures for unsubscribed topics can no longer nack log messages.
• Avoids misleading delivery-failure logs for events that were intentionally filtered.

Test command

go test ./internal/logmq

For maintainers

Please review if the change makes sense in a broader scope that I may be missing here as I have found this issue in my use case where I am only interested in alert.destination.disabled events running 3 replicas of outpost-log service specifically.

[alexluong]

Thanks for opening the PR. Sorry for the delayed review.

Overall LGTM. Great catch as well!

One small question here. Once that is settled we can merge and include in the next release!

[alexluong]

I wonder if we should move this logic to plan step instead of send?

[samadalishah]

Thank you for taking a look!

Good question. I placed the check in send intentionally because topic filtering and suppression are both delivery concerns, while plan handles domain decisions and actions. In particular, disabling a destination must still happen even when the corresponding notification topic is not subscribed.
Also, not every event passes through plan. For example, attempt.success and the alerts-disabled attempt.failed path construct deliveryEvents directly. Keeping the check at the final delivery boundary guarantees that no unsubscribed event can enter suppression, regardless of where it was produced.

[alexluong]

Makes sense! Merging now.

Also from your error report, I discovered the root cause of the context deadline exceeded issue here #1063