Seed random number generator

[emitra17]

Allow specification of seed in config so run is reproducible

[emitra17]

Thinking about this more, since the jobs return in a nondeterministic order, this might not actually be possible.
Would only work for synchronous algorithms, and even then, need to be careful that randomizations each iteration are done in an order independent of the order in which jobs return, and hope that failing jobs is deterministic.
How did BNF1 manage this feature?

[ryants]

It didn't handle it as far as I know. All it could do was set the seed for Perl's random number generator.

[ryants]

AFAIK exactly reproducing a run would require the exact same machine setup, which is highly unlikely on a cluster

[emitra17]

I guess for stochastic simulations, we would have to (using our own seeded random number generator) generate a seed to feed into BioNetGen or NFSim. Which would be annoying, but if you did that, then I think results should be the same regardless of machine setup. And I think PyBNF only pulls random numbers on the main thread, which should be reproducible if seeded.

If BNF1 didn't actually support this then I guess there's less pressure for us to do so. But it could be useful for having examples for a paper that are reproducible, and for tracking down bugs that occur under rare conditions.

[ryants]

I suppose that's true, since none of the algorithms are timing dependent. If we were generating new PSets before certain jobs finished, then we'd probably be screwed.

I agree that it's not a huge issue currently

[wshlavacek]

Implementation Plan

Overview

Add a random_seed config key that seeds np.random at startup, making runs reproducible. All algorithmic randomness already flows through np.random.* on the main thread, so a single seed call is sufficient for the PyBNF side. Stochastic simulators (BioNetGen/NFSim) need separate handling.

Step 1: Add random_seed config key

In config.py:

• Add random_seed to the set of recognized keys (integer, optional, default None)
• Validate it's a non-negative integer if provided

Step 2: Seed np.random at startup

In pybnf.py, early in the run (before algorithm construction):

• If random_seed is set, call np.random.seed(random_seed)
• If random_seed is not set, generate a seed from system entropy via np.random.randint(0, 2**31), call np.random.seed() with it, and log it so the user can reproduce after the fact
• Log the effective seed at INFO level: "Random seed: %d"

This is sufficient because:

• All random draws (parameter sampling, algorithm proposals, acceptance decisions, latin hypercube, bootstrap) use np.random.*
• All random draws happen on the main thread — the Dask workers only run simulators and don't call np.random

Step 3: Handle job return order (the hard part)

The main run loop (Algorithm.run()) processes results via custom_as_completed(), meaning got_result() is called in whatever order simulations finish. For algorithms where got_result() triggers random draws (proposing new candidates), the sequence of random numbers consumed depends on return order.

Synchronous/generation-based algorithms (PSO, island DE, Scatter Search, Simplex, DREAM, all Bayesian MCMCs): These wait for all jobs in a generation/iteration to complete before proposing new candidates. The got_result() calls that trigger new random draws happen only after all results for the current batch are in. Within a batch, results are just stored — no random numbers are drawn. These should be reproducible with just a seed.

However, verify this claim for each algorithm by auditing whether got_result() draws random numbers before the generation boundary (e.g. for bookkeeping). If any do, the draws could be order-dependent.

Asynchronous DE (ade): Each got_result() immediately proposes a new candidate using np.random.choice and np.random.random. The order results arrive determines which random numbers go to which proposals. This algorithm cannot be made reproducible without deeper changes (e.g., per-index RNG streams or deterministic result ordering). Document this limitation.

Step 4: Pass seeds to stochastic simulators (optional, separate PR)

For full reproducibility with stochastic simulators:

• When random_seed is set, use np.random.randint() to generate a per-simulation seed
• For BNGL: inject seed=>VALUE into the action command (PyBNF already detects and parses seed=> — see BNGLModel.seeded)
• For NFSim: similar mechanism via command-line -seed flag
• SBML/libRoadRunner: deterministic ODE solvers, no action needed

This is the most invasive part and could be a follow-up PR. Without it, the algorithmic trajectory is reproducible but stochastic simulation outputs may differ.

Step 5: Documentation

• Document random_seed in the config reference
• Note the ade limitation
• Note that stochastic simulator reproducibility requires Step 4
• Note that the logged seed can be used to reproduce any run after the fact

Step 6: Tests

Add tests that:

1. Set a seed, run random_pset() twice, verify identical results
2. Set a seed, run random_latin_hypercube_psets() twice, verify identical results
3. Set a seed, run 2-3 iterations of a synchronous algorithm (e.g. PSO), verify identical proposal sequences
4. Verify the config parser accepts and validates random_seed
5. Verify the seed is logged

---

Pre-implementation test needs

Before starting this work, we should add baseline characterization tests that exercise the random→result pipeline for each algorithm. Currently, the algorithm tests (test_particle_swarm.py, test_diff_evolution.py, etc.) verify that iterations run without crashing and produce reasonable outputs, but they use loose statistical assertions like "some but not all proposals were accepted" or "values are within bounds."

These tests won't catch a subtle bug where seeding appears to work but the random stream is actually consumed in a nondeterministic order. We need tests that:

1. Verify generation-boundary determinism for each synchronous algorithm: Set a seed, feed results to got_result() in two different orders (simulating different job completion orders within a generation), and confirm the next generation's proposals are identical regardless. This is the key invariant that makes seeding work, and it's currently untested.

2. Verify ade is order-dependent: Same test as above but for async DE, confirming proposals do differ — documenting the known limitation.

These tests validate the assumptions the seed feature relies on. Without them, we could implement seeding and have it silently break for certain algorithms without any test catching it.

[wshlavacek]

Pre-implementation tests added

tests/test_seed_determinism.py — 20 tests validating the assumptions the seed feature depends on. Run with:

cd tests
python3 -m pytest test_seed_determinism.py -v

No BioNetGen installation is required; the tests mock _load_simulators and _initialize_models.

What the tests cover

TestSeedReproducibility (10 tests) — For each algorithm, verifies that identical seed + identical result-processing order produces identical proposals. This is the core invariant: np.random.seed(N) at startup will produce a reproducible run as long as simulation results arrive in the same order.

Algorithms covered: pso, de, ade, ss, dream, mh, pt, dream_zsp, am, sim

TestResultOrderIndependence (3 tests) — For algorithms where got_result() does not draw random numbers until the generation boundary, verifies that proposals are identical regardless of the order results are fed. These algorithms will be reproducible with just a seed, even with nondeterministic job completion order.

Algorithms confirmed order-independent: de (single island), ss, sim

TestResultOrderDependence (7 tests) — For algorithms where got_result() draws random numbers on each call (e.g., MH acceptance), documents that different result orders produce different proposals. These algorithms are only reproducible when simulations return in the same order.

Algorithms confirmed order-dependent: pso, ade, dream, mh, pt, dream_zsp, am

Not yet covered

• sa (simulated annealing) — uses the same BasicBayesMCMCAlgorithm class as mh/pt, so behavior is the same, but a dedicated test should be added.
• scream — excluded because it's being renamed. Tests should be added for whatever replaces it.
• Multi-island de — the current test uses islands=1. With multiple islands, the order in which islands complete their generations can affect migration setup random draws, making it partially order-dependent. A multi-island test should be added.
• Bootstrap resampling (data.py) — uses np.random.choice and would be seeded by the same global seed, but is not yet tested for determinism.