Skip to content

Flaky tests in agent.orchestration.node under CI load: AgentToolNodeExecutionLeaseTest and NeutralToolRuntimeTest #379

Open

Description

@XianReallyHot-ZZH

Summary

Two concurrency-sensitive test classes in argi-agent-framework (io.github.agentic.ai.agent.orchestration.node) fail intermittently on GitHub Actions runners while passing consistently on local machines. The red test checks they produce block otherwise-green PRs, and fork contributors cannot re-run workflows (re-run requires admin rights), so the only workaround is empty retrigger commits.

Incident ledger

Date Where Run Failing test Outcome
2026-11-15 main 43727198824 NeutralToolRuntimeTest.interruptionCancelsSourceAndPreservesInterruptFlag next runs on main green with no change
2026-11-16 PR #371 43885490710 AgentToolNodeExecutionLeaseTest.parallelDispatchStopsPermitWaiterAfterLeaseLoss (expected: <0> but was: <1>) green on retrigger
2026-11-19 PR #371 44434068638 NeutralToolRuntimeTest.interruptionCancelsSourceAndPreservesInterruptFlag green on retrigger

The failures are provably non-deterministic

The 2026-11-19 red landed on an empty commit (c52ed72) whose tree is identical to 1c4346f — and that same tree had passed the test job green about an hour earlier in run 44430136438. Same code, opposite outcomes, ~1h apart.

Locally both classes are stable on the same tree: AgentToolNodeExecutionLeaseTest 10/10 consecutive runs, NeutralToolRuntimeTest 5/5.

Failure modes

  • AgentToolNodeExecutionLeaseTest.parallelDispatchStopsPermitWaiterAfterLeaseLoss — a permit-waiter race: with maxParallelTools=1, the queued second tool can acquire the just-released permit and start before the dispatch loop observes the lease loss and stops the waiters. assertEquals(0, secondToolCalls.get()) then fails. The class already received a determinism pass in 668d157 ("test(agent): make lease-loss cancellation checks deterministic"), but this race persists.
  • NeutralToolRuntimeTest.interruptionCancelsSourceAndPreservesInterruptFlag — a timing-sensitive interruption assertion that loses the race under runner load.

Both widen their race windows on loaded shared runners, which matches the pattern: red only under CI, green everywhere else.

Offer

Happy to investigate making both tests deterministic (latch-based handoffs instead of scheduler timing, in the spirit of 668d157) if maintainers agree that is the right direction.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions