test(qwp): fix flaky SenderErrorDispatcher overflow test - #88
Open
bluestreak01 wants to merge 1 commit into
Open
test(qwp): fix flaky SenderErrorDispatcher overflow test#88bluestreak01 wants to merge 1 commit into
bluestreak01 wants to merge 1 commit into
Conversation
testFullInboxDropsOldestAndCounts slept a fixed 50 ms after the first offer, assuming the lazily-started dispatcher thread had taken error 0 out of the capacity-4 inbox by then. On a loaded Windows CI agent (questdb/questdb macwin build 260325) the thread started late, the inbox still held error 0 when the fill began, and offers 4-6 each evicted the head: getDroppedNotifications() returned 3 and the test failed with expected:<2> but was:<3>. The handler now counts down a handlerEntered latch on entry, and the test awaits the latch instead of sleeping. The dispatcher thread only invokes the handler after it removes the head from the inbox, so the latch orders "error 0 taken" before the fill: the queue provably starts empty and exactly two overflow drops follow on any scheduler. The delivered-tail assertion depends on the same handoff, so the latch deflakes both assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SenderErrorDispatcherTest.testFullInboxDropsOldestAndCountsfails intermittently on loaded CI agents:Observed in questdb/questdb
macwinbuild 260325 (windows-other-2job), which runs this suite against the client.Root cause
The test offers error 0 to a capacity-4 dispatcher whose handler blocks on a latch, then sleeps a fixed 50 ms and assumes the lazily-started dispatcher thread has taken error 0 out of the inbox by then. The drop count the test asserts is fully determined by consumer progress:
When the dispatcher thread wins the race, 7 - 1 - 4 = 2 drops. When thread start-up exceeds 50 ms, the inbox still holds error 0 during the fill, offers 4-6 each evict the head, and 7 - 0 - 4 = 3 drops fail the assertion. The failing run's close-time WARN ("abandoning 3 queued errors") matches the losing timeline exactly: the thread had taken one entry into the blocked handler, leaving three of the surviving four in the queue.
The production dispatcher behaves correctly in both timelines - it evicts the oldest entry and counts each eviction. Only the test's starting-state assumption races. The delivered-tail assertion (
{0, 3, 4, 5, 6}) rests on the same assumption; the drop-count assertion just fires first.Fix
The handler counts down a
handlerEnteredlatch on entry, and the test awaits that latch instead of sleeping. The dispatcher thread only invokes the handler after removing the head from the inbox, so once the latch releases, the fill provably starts from an empty queue and exactly two overflow drops follow on any scheduler.The await bounds the wait at 5 s: if the dispatcher ever fails to start its thread or deliver the head, the test fails with an explicit "handler should take the head within 5s" message instead of a misleading drop-count mismatch. In the common case the latch releases well under the old 50 ms sleep, so the class runs a bit faster locally (~0.07 s vs ~0.34 s).
Test plan
mvn -pl core -Dtest=SenderErrorDispatcherTest test- 11/11 pass, three consecutive runs