Count only inputs as already set when a set_input helper splits a longer period

[MaxGhenis]

Fixes #580.

Summary

set_input_divide_by_period and set_input_dispatch_by_period spread an input given for a longer period over a variable's months (or years). They treated any value stored for a sub-period as already set, including values the simulation had calculated (a cached default, a formula result, a carried-over or uprated value). So the same input gave different values depending on what had been calculated before it was set.

Executed on master b78b0ba, with CountryTaxBenefitSystem plus formula-less person variables and one person:

Variable (helper)	Steps	New simulation	After calculating first
flow_m: float, MONTH (divide)	calculate("2013-01"), then set_input("2013", [1200]); read May and January	100, 100	109.09, 0
flow_m	calculate("2013"), then set_input("2013", [1200])	accepted	ValueError: Inconsistent input
stock_m: int, MONTH (dispatch)	calculate("2013-01"), then set_input("2013", [7]); read May	7	0
stock_m	calculate("2013"), then set_input("2013", [7]); read December and the year	7, 7	0, 0
flow_y: float, YEAR (divide)	calculate("2013"), then set_input("month:2013-01:24", [2400]); read 2013 and 2014	1200, 1200	0, 2400
flow_y	calculate("2013-05"), then set_input("month:2013-01:24", [2400]); read 2013-05	100	0

(repro/helper_order.py in the evidence folder; with this branch every "after calculating first" cell equals the "new simulation" cell.)

The helpers now tell inputs from calculated values using the simulation's record of inputs (Simulation._user_input_keys). That record has to be right for them to be right. Where master gets it wrong, this PR either fixes it or names the open PR that does (see "Merge order" below).

restore_simulation. It stored every restored value with put_in_cache, so a restored simulation had an empty record and the helpers would have replaced restored inputs. The dumper now writes which values were inputs (inputs.txt), and restore records exactly those; a dump without the file restores every value as an input. This is #576's simulation_dumper.py change, byte for byte: the two PRs merge without conflict on that file, and whichever lands second changes nothing there.

The rule

A sub-period counts as already set only if its value was stored as an input, meaning it is in the record. Entries are (variable, branch, period); an input set for twelve months from the first of a month is stored under the year, so both forms are recognised.

• A recorded input is kept, as before:

  • divide subtracts it from the value given and shares the rest among the other sub-periods;
  • dispatch applies it to the sub-periods after it (see below).

• A calculated value is replaced, as if nothing were stored.

• Calculated values over overlapping periods are dropped after storing. These are the same variable's values over periods that overlap the input period:

  • the sums calculate_add caches (a monthly variable's year);
  • the twelfths calculate_divide caches (a yearly variable's month);
  • calculated own-period values a branch would read before the new input.

  The search covers the input's branch name and the names its simulation reads through (its ancestors, then default), in that simulation's own memory and disk storage. Inputs are never dropped, and the simulation a branch was created from keeps its values.

• Bookkeeping. What the helpers store is recorded even when a helper is called directly. calculate's fast cache drops the replaced and dropped periods, but only for a branch the simulation reads: a value stored under a branch it does not read changes nothing it returned.

• No record. A simulation without one (_user_input_keys missing), or a holder without a simulation, keeps master's behaviour: every stored value counts as set.

The dispatch helper's "reuse the existing array" branch (the TODO)

This branch is kept for inputs, and the docstring now documents it. A sub-period that already holds an input keeps it, and that input, not the value given for the longer period, is applied to the later sub-periods that have none. With 3 set for March, setting 7 for the year gives 7 in January and February and 3 from March to December: a stock's value known at a month holds for the rest of the period.

That is the existing behaviour for inputs set on top of inputs, which the issue requires to keep working. Changing it would change results for sequences with no calculation at all, a separate semantic question. The asymmetry stays: the reverse order gives 7 in every month but March. A calculated value is no longer reused this way; that was the bug, where one calculated January set every later month to the default.

Invariants

Each holds for any variables, inputs and calculation requests in the property tests' scope (tests/core/test_set_input_helper_order_property.py):

1. Order independence. Take inputs I1, then any lists of plain calculate requests (on the simulation and on up to two nested branches, a list per level), then inputs I2 on the last of these. The result matches setting I1 and I2 on a simulation that calculated nothing:
   • the same inputs of I2 are accepted or refused;
   • the same inputs are recorded;
   • every own period of the variables returns a byte-identical array, in memory or on disk;
   • so do periods of another size (sums, twelfths), for variables that I2 gives no input for one of their own periods (set_input for one period leaves the variable's cached sum or twelfth over an overlapping period in place #579).
2. Inputs only. Those values equal what a model holding inputs alone gives (Reference in tests/fixtures/set_input_helper_order.py), with the two helper rules above applied to inputs.
3. Conservation (divide). A divided input is accepted whenever a sub-period has no input yet. Once accepted, the variable's values over its sub-periods add up to the value given, and a plain read of that period returns it. Both qualifications below predate this PR:
   • the sum holds to float32 tolerance within the tested magnitudes (inputs up to 1,200); with an existing input of 1e9, float32 rounding loses more than that;
   • an input already stored at a longer period inside the input period (twelve months set as one value) is kept, so reading that period returns that input.
4. Inputs are kept. A helper never changes an input already set for one of the variable's own periods. Afterwards, every sub-period of the input period holds a recorded input.
5. Restore keeps the record. These are example tests. A restored simulation records exactly the dumped simulation's inputs: an input set later over a longer period keeps the restored inputs, and apply_reform recalculates restored calculated values.

Outside the properties, because they are other order dependences with their own PRs or issues:

• auto-carry-over and uprating (Carry over only inputs, the latest at or before the requested period #562, Uprate only from inputs, so uprated values don't depend on calculation order #563);
• calculate_add/calculate_divide called directly (Cache ADD and DIVIDE results only where a plain read returns them #571);
• other variables calculated from the input (Make set_input on a branch drop values calculated from the input it replaces #560, on branches);
• an input for one of a variable's own periods leaving the variable's cached sum or twelfth in place (set_input for one period leaves the variable's cached sum or twelfth over an overlapping period in place #579, filed with this PR; it reproduces on Make set_input on a branch drop values calculated from the input it replaces #560, Carry over only inputs, the latest at or before the requested period #562, Cache ADD and DIVIDE results only where a plain read returns them #571 and Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 as well).

Tests

• tests/core/test_set_input_helper_order.py, 39 example tests. On master cbfdedf, 26 fail and 13 pass. The 13 that pass pin behaviour that must not change:

  • month inputs kept under a yearly input;
  • a situation giving both a month and its year;
  • a contradicting total refused;
  • values outside the input period kept;
  • inputs stored at a longer period kept, from January or another month;
  • dispatch on top of inputs (2 tests);
  • the no-record fallback;
  • the fast cache kept for an input under a branch the simulation does not read;
  • restored values kept and restored calculated values recalculated (3 tests).

  The restore and fast-cache tests pass on master and fail on this branch's helpers without the matching change.

• tests/core/test_set_input_helper_order_property.py, invariants 1-4, Hypothesis with 500, 300 and 300 examples. The module starts with pytest.importorskip("hypothesis") for the smoke job. All three fail on master within the first examples. Soak runs passed: 8,000/4,000/4,000 examples at af877e8 and 6,000/3,000/3,000 at 1f5e3ec. The helper logic they exercise has not changed since, apart from the fast-cache condition, which the example test above covers.

• Mutation check (mutation/run.py): 25/25 mutants killed at 917f0c2. They include the two mutants the review found surviving at 22a9b31 (twelve-month inputs recognised only from January; intermediate ancestor branches skipped), the restore change, and the fast-cache condition.

• Full suite at 917f0c2 (merged with master cbfdedf): 1,208 passed, 4 skipped, 1 xfailed. Country-template YAML tests: 39 passed.

• Windows: on disk, a value calculated for a period like year:2013:2 goes in a file named after the period, and Windows rejects : in file names (Storage keys need a structured (branch, period) scheme: str(period) is lossy and separators collide with branch names #526). The property's on-disk examples skip such periods. With : blocked in file names the way Windows does it, the property fails at 1f5e3ec and passes at f7ac58b.

Review

Independent adversarial review, GPT-6.1 Sol, four rounds (review/ in the evidence folder):

• Round 1 (22a9b31): REQUEST CHANGES. Restored inputs were replaced; fixed. Disk clones and stale records are pre-existing bugs fixed by Give each disk-backed holder storage its own directory #558 and Keep the record of set_input values in step with storage #561 (next section). Test gaps for nested branches and twelve-month inputs from other months are now covered, and the conservation claim is qualified.
• Round 2 (1f5e3ec): REQUEST CHANGES.
  • Restoring every value as an input froze calculated values through a reform. Fixed: the dumper is Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576's.
  • A calculated write over an input left the input's record entry.
• Round 3 (29410c3): REQUEST CHANGES. My fix for round 2's second finding, dropping record entries on calculated writes, lost real inputs: an ADD at a variable's own period re-stores the input unchanged; clones share their source's record until Keep the record of set_input values in step with storage #561; and ETERNITY entries come in several forms. It is removed, and holder.py is master's again. That case now depends on Keep the record of set_input values in step with storage #561 and Cache ADD and DIVIDE results only where a plain read returns them #571 together (next section). The same round's composition with Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 found the helpers evicting the fast cache for branches the simulation does not read; fixed.
• Round 4 (917f0c2, delta): pending.

Merge order: #561 first; #558 and #571 before or with this

The cases below come from the review, executed on master, on this branch at 917f0c2, and on this branch merged with #561, #571 or #558 (review/review_lifecycle.py, review/round2_scripts/replacement_probe.py; outputs review/lifecycle_917f0c23.out, compose/round4/replacement_561_571.out):

Case	master	this branch	with #561	with #558
A clone shares its source's record (#559); calculated January, then an annual input on the source	wrong (January 0)	same as master	right	same as master
A custom set_input handler calculates while storing	wrong	same as master	right	same as master
Input deleted with delete_arrays, recalculated, then an annual input	wrong	same as master	right	same as master
Twelve-month input deleted (or set on a clone), year recalculated, then an annual input	Inconsistent input	stale year, months right	right	same as this branch
Disk-backed clone sets an annual input over a month its source calculated	source unchanged	source's file overwritten	same as this branch	source unchanged

• Record drift (rows 1-4). The first four rows come from the record drifting from storage (_user_input_keys drifts from stored values after delete_arrays and clone #559): shared between clones, kept after delete_arrays, written during handlers. Keep the record of set_input values in step with storage #561 fixes that. In rows 1-3 this branch is exactly as wrong as master. In row 4 master raises and this branch returns a stale year. So Keep the record of set_input values in step with storage #561 should merge first.
• calculate_add over an input (round 2, finding 2). Calling calculate_add directly over an input set for twelve months, then setting an annual input, is the same kind of case. This branch alone, and with Keep the record of set_input values in step with storage #561 alone or Cache ADD and DIVIDE results only where a plain read returns them #571 alone, gives a stale annual [0, 0]. With Keep the record of set_input values in step with storage #561 and Cache ADD and DIVIDE results only where a plain read returns them #571 together it matches a simulation that never called calculate_add: the annual read keeps the twelve-month input, the months take the annual one, and both survive invalidation. Cache ADD and DIVIDE results only where a plain read returns them #571 stops ADD caching over an input, but it looks the input up by the year, and only Keep the record of set_input values in step with storage #561 records a twelve-month input in that form.
• Disk-backed clones (row 5). On master a disk-backed clone shares its source's files (Give each disk-backed holder storage its own directory #558), so any write a clone makes to a key its source stores overwrites the source's file. An exact-period clone.set_input(...) already does this on master, as does a default-branch holder input on a get_branch branch (executed, repro/disk_clone_exact_input.py). This branch adds one more such write. With Give each disk-backed holder storage its own directory #558 merged none of them reaches the source. Disk storage is opt-in (MemoryConfig, marked experimental).

How it composes with the other open core PRs

Each PR head was scratch-merged with this branch at 917f0c2, and the full suite was run on the merge (compose/round4/):

PR	Merge	Full suite on the merge
#576 (restore input record, holder fast-cache eviction)	clean	1,268 passed, 4 skipped, 1 xfailed
#561 (_user_input_keys drift)	clean	1,239 passed, 4 skipped, 1 xfailed
#571 (ADD/DIVIDE caches)	clean	1,224 passed, 4 skipped, 1 xfailed
#558 (disk storage per holder)	clean	1,240 passed, 4 skipped, 1 xfailed
#566 (fast cache by containment)	clean	1,217 passed, 4 skipped, 1 xfailed
#562, #563, #560, #552	conflicts	see below

• Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576. Its restore property excludes, from the comparison with a new simulation, the examples where an annual input follows a calculation, because of this bug. With the exclusion removed (compare_with_fresh = True) at 3,000 examples, it fails on Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 alone on exactly this case and passes merged with this branch. Once both are in, _split_after_calculating can go.

• Carry over only inputs, the latest at or before the requested period #562, Uprate only from inputs, so uprated values don't depend on calculation order #563, Make set_input on a branch drop values calculated from the input it replaces #560, Read only periods the current branch can see when uprating or carrying over #552. None of these conflicts is this PR's own:

  • In simulation_dumper.py, they are Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576's conflicts (git merge-tree of each head with Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 conflicts on the same file), since this branch's dumper is Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576's.
  • Make set_input on a branch drop values calculated from the input it replaces #560 and Uprate only from inputs, so uprated values don't depend on calculation order #563 also conflict with master itself in in_memory_storage.py, because of Give a storage a set of shared keys only while it shares an array #578.

  At an earlier head, each was resolved and its full suite passed: Carry over only inputs, the latest at or before the requested period #562 (1,243) and Uprate only from inputs, so uprated values don't depend on calculation order #563 (1,277) write and read both sidecars, Make set_input on a branch drop values calculated from the input it replaces #560 (1,269) takes its own dumper (it already records inputs), and Read only periods the current branch can see when uprating or carrying over #552 (1,718) dumps the branch-visible values with the input record. The resolved files are in compose/round3/.

• Carry over only inputs, the latest at or before the requested period #562. Adds a per-key derived mark in storage. After Carry over only inputs, the latest at or before the requested period #562, the helpers could read that mark for the exact branch they store to. Its is_derived reads through ancestors, which is the wrong question here: a parent's calculated value under default must not stop a branch from taking an input. The drop step deletes keys from _arrays/_files directly; Carry over only inputs, the latest at or before the requested period #562's is_derived requires the key to still be stored, so a stale _derived entry is harmless.

• Make set_input on a branch drop values calculated from the input it replaces #560. Keeps its own _input_keys and _sequence_numbers in storage. The drop step removes only non-input keys. A stale _sequence_numbers entry for a dropped key is ignored by every reader and cleared by the next delete.

Downstream

A/B, real runs. Core master cbfdedf against this branch at 917f0c2, with the same country code and data, every array compared byte for byte:

Run	Arrays	Result	Helper events with the fix
policyengine-uk c7e826ea, enhanced FRS 2024-25 (sha256 03fe15e4…), full sample, 2026	36	bitwise identical	0 helper calls
policyengine-us fbe24ad1, enhanced CPS 2024 (sha256 0a6b961a…), 3,000-household subsample, 2026 (income tax and the itemizing / SALT branches, state taxes, CTC, EITC, SNAP, net income, MTRs)	25	bitwise identical	2 helper calls, 24 sub-periods stored, 0 calculated values replaced, 0 dropped

The same comparison at 22a9b31 and 29410c3, against master b78b0ba (the PE-US run on another enhanced CPS 2024 file), was also bitwise identical.

On these runs the only code that differs from master is the helpers' body, which runs twice in PE-US and not at all in PE-UK. Timings were within run-to-run noise on a loaded host:

• PE-UK: 7.4 s on both.
• PE-US: 50.9 s on master against 68.1 s with the fix. At 29410c3 it was 102.8 s against 90.9 s, the other way round.

Code that sets inputs over longer periods. Every non-test set_input call was listed, with the definition period of the variable it sets (downstream/set_input_sites.py; it makes no changes to the repos it reads):

Repo	Ref	Calls
policyengine-us	upstream/main 615a0f2d	38
policyengine-uk	origin/main 84da2863	30
policyengine-canada	upstream/master 389648ad	1

Every call that names its variable passes a period of that variable's own unit, so no helper runs. The calls whose variable is a parameter fall into these groups:

• Dataset and situation loaders (PE-UK simulation.py build_from_*, PE-US system.py, policyengine.py main 6a9c878 us/model.py). These set inputs on freshly built populations before anything is calculated.
• Behavioural-response copies (PE-UK move_values and dynamics, PE-US behavioral_response_measurements, tob_revenue_*). All set YEAR variables at years.
• PE-UK reform and budget loops. These name none of PE-UK's 8 MONTH variables.
• PE-UK's 37 explicit set_input = set_input_dispatch_by_period attributes. All are on YEAR variables.
• PE-UK utils/dependencies.py calculate_dependency_contributions. This is the only site where a helper meets calculated values: it zeroes and then restores a dependency with set_input(var, year, ...). For a MONTH float dependency it fails either way: on master the zeroing call raises Inconsistent input, and with this branch the restoring call does (repro/uk_dependencies_pattern.py). Nothing relied on the old behaviour. Fixing the utility is a separate PE-UK task.

No downstream code calls dump_simulation or restore_simulation.

Checks

• CI on the head commit.
• ruff format and ruff check clean.
• Not run: multi-year country runs.

Evidence (scripts, outputs, A/B arrays, mutation results, compositions, reviews): ~/reviews/core-set-input-helper-order-2026-10-02/ on the author's machine.

axiom: n/a: core engine input handling, no policy encoding

🤖 Generated with Claude Code