fix(logmq): don't wait on in-flight exhausted-retries suppression - #1063
Open
alexluong wants to merge 1 commit into
Open
fix(logmq): don't wait on in-flight exhausted-retries suppression#1063alexluong wants to merge 1 commit into
alexluong wants to merge 1 commit into
Conversation
The window was an idempotence.Idempotence with only SuccessfulTTL set, so its in-flight conflict path inherited the 5s default: a contender that saw "processing" slept 5s, the same as the emit budget, and then failed its recheck on the expired ctx. send()'s ErrConflict branch was unreachable; every concurrent exhaustion on one destination nacked with "context deadline exceeded" and was redelivered. Replace it with a SETNX-per-window type in logmq. Losing the claim is the suppression: return immediately, no in-flight state, no wait. A failed emit releases the claim so the next exhaustion re-alerts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
opevent delivery failed … context deadline exceedednack reported in #1050 for subscribers ofalert.attempt.exhausted_retries. #1050 keeps unsubscribed events out of the window entirely; this covers the case where the event is subscribed.The failure
A destination goes down. Its queued events retry and exhaust around the same time, spread across the log service replicas. Every exhaustion on that destination wants to send one
exhausted_retriesalert through the same suppression key,opevents:exhausted:<tenant>:<destination>, so that only the first one in the window goes out.The window was an
idempotence.Idempotenceconfigured with only the window TTL, so it inherited the package's 5s conflict wait, meant for delivery idempotency where the claimed work is an HTTP call. When two exhaustions overlap:processing, A emits (a few ms) and marks the keyprocessed.processing, and sleeps 5s before rechecking.context deadline exceeded.send()returns the error,sendAlllogsopevent delivery failed, and the whole log message nacks.MarkProcessednever runs.attempt.failedordestination.disabledcan go out twice. The window key isprocessedby then, so the retry acks.The
ErrConflict → deliveredbranch insend()was written for this case but is unreachable: the sleep always outlives the ctx. One collision costs a goroutine parked 5s, an error log, a nack and redelivery, and possible duplicate alerts. The loser had nothing to wait for; the window's contract is one alert per destination per window, and A's alert is already on its way.The fix
Replaces the window with a SETNX-per-window type in
logmq: losing the claim is the suppression, return immediately. A failed emit releases the claim so the next exhaustion re-alerts.send()drops itsErrConflicthandling. Live windows reset once on upgrade since the key value changes, at worst one extra exhausted-retries alert per destination mid-window.