SeededScript's weight test samples a distribution: 1 run in ~550 fails under an overriding MORPH_STRESS_SEED

[Yaraslaut]

Summary

SeededScript can pick the last generator, which absorbs the leftover weight
(examples/common/testkit/test_action_driver.cpp:173) asserts an outcome that is
only probably true of the sequence its generator draws. With the default
hard-coded seed it passes, but SeededScript lets the environment replace that
seed for every script in the process, and under an overriding seed the case fails
about 1 run in 1,000 — measured, with three reproducers below.

Found while fixing #365 (pastebin's id-collision retry test), which was the same
shape: an assertion whose truth is a property of a sampled distribution rather
than of the logic under test. This one is filed separately and is not fixed by
that change.

The assertion

SeededScript<int> script{/*seed=*/2'024,
                         /*generators=*/{{1, [] { return 100; }}, {9, [] { return 200; }}},
                         /*burstSize=*/100, /*onBurst=*/[](const std::vector<int>&) {}};

for (int i = 0; i < 60; ++i) {
    generated.push_back(script.next());
}

// Both generators must actually have been selected, or this test would
// silently stop covering one of the two paths it exists to cover.
CHECK(std::count(generated.begin(), generated.end(), 100) > 0);
CHECK(std::count(generated.begin(), generated.end(), 200) > 0);

The light generator is weighted 1 against 9, so it is drawn with p = 0.1. Over 60
draws, count(... 100) > 0 is false with probability 0.9^60 ≈ 1.8e-3, i.e.
about 1 run in 550.

The default seed 2024 is not itself the problem — it is a fixed sequence that
passes. The problem is that the seed is not actually fixed:

[[nodiscard]] static std::uint64_t resolveSeed(std::uint64_t defaultSeed) {
    if (const char* env = std::getenv("MORPH_STRESS_SEED"); env != nullptr && *env != '\0') {
        return std::stoull(env);
    }
    return defaultSeed;
}

(examples/common/testkit/action_driver.hpp:98-104)

MORPH_STRESS_SEED is documented in examples/TESTING.md:369 as the knob for
re-running the ledger/kanban stress runs under a different schedule. Exporting
it also silently reseeds this unit case — which is not a stress test at all — and
re-rolls the 1-in-550 lottery. Catch2's --rng-seed does not reach it either
way.

Verification status

Reproduced, on 5d3c8690 (branch scenario-corpus-both-axes), Linux, clang
22.1.8, clang-release preset with -DMORPH_BUILD_QT=ON -DMORPH_BUILD_LADDER=ON -DMORPH_LADDER_RUNGS=pastebin, binary ladder_common_tests.

Swept MORPH_STRESS_SEED over 1..3000, one process per seed:

FAIL seed=314
FAIL seed=779
FAIL seed=2522
scanned=3000 failing=3

3/3000 = 1.0e-3 measured, against 1.8e-3 predicted by 0.9^60 (3 failures is a
small sample; the two are consistent, the sweep is not a precise estimate of the
rate).

Real output for one of them:

$ MORPH_STRESS_SEED=314 QT_QPA_PLATFORM=offscreen ./ladder_common_tests "SeededScript can pick the last generator*"

-------------------------------------------------------------------------------
SeededScript can pick the last generator, which absorbs the leftover weight
-------------------------------------------------------------------------------
examples/common/testkit/test_action_driver.cpp:173
...............................................................................

examples/common/testkit/test_action_driver.cpp:193: FAILED:
  CHECK( std::count(generated.begin(), generated.end(), 100) > 0 )
with expansion:
  0 > 0

test cases:  1 |  0 passed | 1 failed
assertions: 62 | 61 passed | 1 failed

Not verified:

• Whether any CI job exports MORPH_STRESS_SEED. I did not audit the workflows
  for it, so I do not claim this has ever fired in CI. On the default seed the
  case is deterministic and green, which is why it is a developer-facing hazard
  (re-running a stress schedule with an exported seed) rather than a per-run one.
• Whether seed 2024 also passes on libstdc++ != this one, libc++ or MSVC.
  std::uniform_int_distribution is not specified by the standard, so the pinned
  seed is not a portable guarantee that both branches are drawn — the same 1.8e-3
  lottery is re-rolled by a standard-library change. Inferred from the standard,
  not measured on another toolchain.

What would change the verdict

Close it when the case no longer samples: either drive the two branches
deterministically (a SeededScript whose draws are scripted, or two scripts with
weights that make each branch certain), or make this case immune to
MORPH_STRESS_SEED and assert the exact expected sequence for its own fixed
seed. Re-open if a seeded re-run of the ladder stress suites reports this case as
a failure again.

Related, weaker findings (not measured, filed here for the record rather than lost)

Same class — an assertion that is a probability rather than a property — but at
odds that make a real failure implausible:

• tests/test_opaque_model_ids.cpp:106-115, 96-105, 119-129 — "two independently
  keyed OpaqueIdGenerators disagree" / "output is not sequential" (~2^-64 each;
  the code comment states the probability). Keys come from std::random_device
  per instance, so a failure would be unreproducible. The practical hazard is not
  the 2^-64 but a platform whose std::random_device is a deterministic PRNG
  restarting per construction, where these fail 100% of the time.
• examples/polls/tests/test_poll_model.cpp:120-124, 143-151 and
  examples/polls/tests/test_app.cpp:79 — "two CreatePoll calls never collide
  on pollId/adminToken/participantToken" (2^-128, fresh std::random_device per
  call, no retry loop in CreatePoll unlike pastebin's).
• tests/net/test_ws_handshake.cpp:49 — REQUIRE(key != generateClientKey()),
  with "vanishingly unlikely to collide" written in the line's own comment.
• examples/ledger/tests/test_multiclient.cpp:213-214 — CHECK(checking.numerator != 0) is a "some work happened" proxy that is zero exactly when the seeded
  3:1 split lands on R = 2F. Impossible at the default nActions = 40, but both
  MORPH_LADDER_ACTIONS and MORPH_STRESS_SEED are environment knobs, and at an
  action count divisible by 3 an unlucky seed fires it on a healthy run.
  Inferred from reading, not reproduced.

These were surveyed by reading, not by sweeping seeds; only the
test_action_driver.cpp case above was reproduced.

[Yaraslaut]

Triage: valid — implement as written. Independently replicated, seed for seed.

Rather than re-running the sweep against ladder_common_tests, I replicated SeededScript::next()'s draw exactly in a standalone program — one std::mt19937_64 seeded per run, a fresh std::uniform_int_distribution<int>{0, totalWeight - 1} per call, weights {1, 9}, 60 draws — and swept the same range:

FAIL seed=314 (light generator never drawn)
FAIL seed=779 (light generator never drawn)
FAIL seed=2522 (light generator never drawn)
scanned=3000 failing=3
default seed 2024: light drawn 4 times in 60 -> PASS

The same three seeds, from an independent implementation. That confirms both the rate and the mechanism: nothing about the test's surroundings is involved, the failure is entirely in the draw, and MORPH_STRESS_SEED reaches it through resolveSeed (action_driver.hpp:100-105) with no opt-out.

Two details worth adding to the issue:

• The default seed's margin is thin. Seed 2024 draws the light generator 4 times in 60. The case is not comfortably green on the pinned seed; it is one of the more marginal passes. That sharpens the portability point — a libstdc++/libc++/MSVC change to uniform_int_distribution re-rolls from a starting position that is four draws from failing, not from a safe one.
• The two figures in the issue are both right. 0.9^60 = 1.79e-3 (≈ 1 in 558) is the predicted rate; 3/3000 = 1.0e-3 is a three-sample measurement of it. The summary's "about 1 run in 1,000" and the analysis' "about 1 run in 550" are the measurement and the prediction, not a contradiction — worth saying so in the text, since they read as inconsistent.

Both proposed directions fit. Asserting the exact expected sequence for a fixed, un-overridable seed is the stronger of the two: it makes the case measure next()'s weight walk (including the "last generator absorbs the remainder" path the comment says it exists to cover) rather than a property of a sample, and it removes the MORPH_STRESS_SEED coupling that makes a stress re-run reseed a unit test.

On the "related, weaker findings" section: correct to record rather than lose, but do not fold them into this fix. examples/ledger/tests/test_multiclient.cpp:213-214 is the only other one with an environment knob that can actually fire it (MORPH_LADDER_ACTIONS at a count divisible by 3, plus MORPH_STRESS_SEED); the 2^-64 / 2^-128 ones are a different concern — their real hazard is a platform whose std::random_device is deterministic, which is a portability check, not a flake. If either is pursued it should be its own issue.

What would change this verdict: a sweep showing no failing seed in 1..3000. Mine finds 314, 779 and 2522, matching the report exactly.