Skip to content

tests: run-engine and session tests read the box's real load and time out on a busy machine #1525

Description

@AbirAbbas

What happened

Several run-engine and session tests start real run seats, which pass through the machine gate (task.max_load, default 1.5 per core). On a busy box they wait machine busy · load per core at or above task.max_load 1.5, then time out. They are green in CI and alone, and red when another suite is running beside them. CLAUDE.md treats that as a bug, not a known flake.

Seen on 2026-09-25 on an 8-core box at load 11–24:

  • internal/session TestUnderOneModelEveryRunSeatRidesTheConversationsModel/the_launch_model: timed out waiting for the run to end on dev@691ade547 at load 12, and 11 s alone. Added in task room, landing and --one-model: real file names, one note sentence, every seat on one model, no build caches #1517.
  • internal/run TestSupervisorLaunchesEveryReadyLeafAtOnceWithNoSlotBound and TestAnUnboundedRunEndedEarlyStillDrainsEveryWorker: fail even alone at load about 11.
  • Also timing out only under load: TestPlandbCliWorkerOwnDone, TestARunCorrectedPastItsRoundLimitStopsWithoutMergingOrSayingDone, TestBeltRunCarriesMachineGateAndShowsItsHold, TestPlandbCliLoopThroughBash, TestACarriedOverDesignRaisesItsCardAndSavesOnYes, TestTheDelegationDoorsCrossFromTheEnginesProfile (session); TestHostedTasksToolStopEndsTheRunAndReportsWhatHappened (remote).

Replication

Deterministic:

stress-ng --cpu $(( $(nproc) * 2 )) --timeout 600s &   # or any load above 1.5 per core
go test -count=1 -run 'TestUnderOneModelEveryRunSeatRidesTheConversationsModel' ./internal/session
go test -count=1 -run 'TestSupervisorLaunchesEveryReadyLeafAtOnceWithNoSlotBound' ./internal/run

Both fail today. Without the load, both pass.

The fix

A test that is not about the machine gate runs with the gate out of its way, through its fixture profile or an injected load reader, never the box's real /proc/loadavg. A test that IS about the gate injects the load it needs.

Acceptance

  • Unit: each named test passes with stress-ng holding the box above 1.5 per core.
  • A law test, or a single fixture helper, so that a new run-engine test cannot read the real load by accident.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testsThe suite itself — flakes, harnesses, laws, CI redsbugSomething the code does that it should not

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions