Skip to content

test(qwp): fix flaky SenderErrorDispatcher overflow test - #88

Open
bluestreak01 wants to merge 1 commit into
mainfrom
fix/error-dispatcher-test-race
Open

test(qwp): fix flaky SenderErrorDispatcher overflow test#88
bluestreak01 wants to merge 1 commit into
mainfrom
fix/error-dispatcher-test-race

Conversation

@bluestreak01

Copy link
Copy Markdown
Member

Problem

SenderErrorDispatcherTest.testFullInboxDropsOldestAndCounts fails intermittently on loaded CI agents:

java.lang.AssertionError: expected:<2> but was:<3>
    at SenderErrorDispatcherTest.testFullInboxDropsOldestAndCounts(SenderErrorDispatcherTest.java:136)

Observed in questdb/questdb macwin build 260325 (windows-other-2 job), which runs this suite against the client.

Root cause

The test offers error 0 to a capacity-4 dispatcher whose handler blocks on a latch, then sleeps a fixed 50 ms and assumes the lazily-started dispatcher thread has taken error 0 out of the inbox by then. The drop count the test asserts is fully determined by consumer progress:

drops = offers - consumed_before_overflow - capacity

When the dispatcher thread wins the race, 7 - 1 - 4 = 2 drops. When thread start-up exceeds 50 ms, the inbox still holds error 0 during the fill, offers 4-6 each evict the head, and 7 - 0 - 4 = 3 drops fail the assertion. The failing run's close-time WARN ("abandoning 3 queued errors") matches the losing timeline exactly: the thread had taken one entry into the blocked handler, leaving three of the surviving four in the queue.

The production dispatcher behaves correctly in both timelines - it evicts the oldest entry and counts each eviction. Only the test's starting-state assumption races. The delivered-tail assertion ({0, 3, 4, 5, 6}) rests on the same assumption; the drop-count assertion just fires first.

Fix

The handler counts down a handlerEntered latch on entry, and the test awaits that latch instead of sleeping. The dispatcher thread only invokes the handler after removing the head from the inbox, so once the latch releases, the fill provably starts from an empty queue and exactly two overflow drops follow on any scheduler.

The await bounds the wait at 5 s: if the dispatcher ever fails to start its thread or deliver the head, the test fails with an explicit "handler should take the head within 5s" message instead of a misleading drop-count mismatch. In the common case the latch releases well under the old 50 ms sleep, so the class runs a bit faster locally (~0.07 s vs ~0.34 s).

Test plan

  • mvn -pl core -Dtest=SenderErrorDispatcherTest test - 11/11 pass, three consecutive runs
  • the fix removes the timing assumption by construction (the latch orders "head taken" before the fill) rather than enlarging the sleep, so repetition is corroboration, not the argument

testFullInboxDropsOldestAndCounts slept a fixed 50 ms after the first
offer, assuming the lazily-started dispatcher thread had taken error 0
out of the capacity-4 inbox by then. On a loaded Windows CI agent
(questdb/questdb macwin build 260325) the thread started late, the
inbox still held error 0 when the fill began, and offers 4-6 each
evicted the head: getDroppedNotifications() returned 3 and the test
failed with expected:<2> but was:<3>.

The handler now counts down a handlerEntered latch on entry, and the
test awaits the latch instead of sleeping. The dispatcher thread only
invokes the handler after it removes the head from the inbox, so the
latch orders "error 0 taken" before the fill: the queue provably
starts empty and exactly two overflow drops follow on any scheduler.
The delivered-tail assertion depends on the same handoff, so the
latch deflakes both assertions.
@bluestreak01 bluestreak01 added the bug Something isn't working label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant