Carry over only inputs, the latest at or before the requested period

[MaxGhenis]

Problem

With auto_carry_over_input_variables = True (policyengine-us, -uk, -canada), a variable with no formula result for a period takes a value from another period. Simulation._calculate took the latest-starting stored period of any kind and returned the default if that period started after the requested one. So the result depended on which periods had been calculated first:

• Any later stored period blocked carry-over. Inputs for 2012 and 2014 gave the default for 2013. With only a 2012 input, calculating 2014 first (which carries 2012 and caches 2014) made 2013 the default too.
• Calculated values carried like inputs. A month's twelfth cached by calculate_divide carried into later years (the policyengine-us monthly_age bug Carry over only values stored at the variable's definition period #557 fixes). So did a value masked by defined_for (its mask came along), a default cached where defined_for was false everywhere (it replaced the input), and a formula result from before the formula's end.

Executed on master b78b0ba vs this branch. Each row compares a simulation that calculated something first with a fresh one asked only the target (one-person YEAR variables; script explore.py in the review folder):

Scenario	Fresh	Master after earlier request	This PR
Input 2012 = 7; ask 2014, then 2013	7	0	7
Inputs 2012 = 7 and 2014 = 9; ask 2013	0 on master, 7 here	0	7
defined_for false for person 0 in 2013 only; ask 2013, then 2014	[7, 7]	[0, 7]	[7, 7]
defined_for false for everyone in 2013; ask 2013, then 2014	7	0	7
Formula ends 2013; ask 2013, then 2014	0	6	0
Yearly flow input 12 for 2012; ask 2012-06, then 2013 (#557)	12	1	12
Branch sets the 2012 input to 70 after the parent carried 2013; branch asks 2014	70	7	70
No input; ask 2013; then set 2012 = 7 and ask 2014	7	0	7

A policyengine-uk household from another session shows the same thing on PE-UK main: with region set for 2025, asking 2040 and then 2026 gives LONDON (the default) for 2026; asking 2026 first gives SCOTLAND. On this branch both orders give SCOTLAND.

Intended rule (from history)

• 2022-10 (fdc8ee6, 5d3bc59): carry-over introduced as "carry the last known period", sorted(known_periods)[-1].
• 2024-08 (Small updates #231): added if last_known_period.start > period.start: return default, with no test or changelog entry. Reading it as "don't carry a later input backwards" fits; but because it only looked at the single latest period, it also blocked forward carry whenever anything later was stored, calculated values included.
• 2026-04 (Stop cross-branch contamination in Holder.get_array + fix auto-carry-over #457, H1/H2): ordered by period.start and passed the branch through; kept the guard.
• 2026-09 (Fix variable uprating when the uprating parameter is undefined, and pick the latest earlier period #551): fixed the same "wrong base period" class in the uprating path by taking the latest earlier period.

No country package relies on "a later stored period means default" in its tests: the PE-US and PE-UK YAML suites were run with a shadow core that logs every carry-over decision where this rule and master's differ (results below).

Rule now

A period with no formula result takes the input stored for the latest-starting period that starts no later than it (on a tie, the one that ends last, then the larger unit), preferring the variable's own definition-period unit and using another unit only when there is none (as #557 does, and as before 3.24.0); masked by defined_for at the period; with no such input, the default. An input never carries backwards. Values the simulation calculated never carry. This is documented on TaxBenefitSystem.auto_carry_over_input_variables.

Fix

• Provenance, stored with the value. Every value Simulation caches (formula, carried, uprated and default values; defined_for defaults; calculate_divide's month twelfth; calculate_add's multi-period sums) is stored with derived=True. InMemoryStorage and OnDiskStorage keep the mark per storage key: every put sets or clears it, delete drops it, clone copies it (including Share cached arrays with simulation branches and copy them on first read #556's shared-array clones), apply_reform's cache wipe clears it, OnDiskStorage.restore starts without it, and storages pickled before it existed default to none (__setstate__). A value calculated while an input is being set (by a set_input helper that calculates) stays on its own branch and is not recorded as an input.
• Which periods are inputs. Holder.get_input_periods(branch_name) makes one pass over the stored keys and, for each period, takes the key get_array(period, branch_name) reads first (the branch, its parent_branch ancestors, default; memory before disk), keeping it if it is not derived. A period stored only under a branch this one cannot read is never taken for an input, and no per-period branch walk is needed. Holder.is_derived(period, branch_name) answers the same question for one period; both read no array, so branches still copy only what they read (Share cached arrays with simulation branches and copy them on first read #556).
• Carry-over takes, of those input periods starting no later than the requested one, the latest: own unit first, then by start, then by end, then the larger unit, then the period's string form. The choice depends only on the inputs, not on the order they were stored. With none, the default; it is cached (derived) except when a later period is stored, where it is returned uncached exactly as master did, so formulas that test whether a value is stored (get_array(period.last_year) is not None) see what they saw before and the uprating path never uprates from a cached default.
• Inputs are never replaced by derived values. put_in_cache(..., derived=True) stores nothing when the branch already reads an input at that period. calculate_add/calculate_divide used to overwrite inputs stored in another unit (helper-less variables) or over several years; calculate_add now caches only sums over several sub-periods (a single sub-period is already stored by calculate).
• Dump/restore persists the marks (derived_periods.txt per variable), read in the same loop and on the same branch as the values dumped (so Read only periods the current branch can see when uprating or carrying over #552's branch dumps, once merged, keep matching marks: the merge conflicts there on purpose).

Relation to open PRs

• Carry over only values stored at the variable's definition period #557 (carry over values at the variable's own unit): this PR fixes the same month-twelfth bug through provenance and keeps Carry over only values stored at the variable's definition period #557's own-unit preference. All 253 tests in Carry over only values stored at the variable's definition period #557's current test file (bab4325) pass on this branch. Either merge order works: if Carry over only values stored at the variable's definition period #557 lands first, this PR's carry-over block replaces Carry over only values stored at the variable's definition period #557's in the rebase.
• Read only periods the current branch can see when uprating or carrying over #552 (branch-visible known periods): composes; it narrows known_periods, which this filter reads. The only textual conflict is in _dump_holder, where the resolution passes Read only periods the current branch can see when uprating or carrying over #552's branch to is_derived as it does to get_array.
• Make set_input on a branch drop values calculated from the input it replaces #560 (branch set_input invalidation): complementary, and it also records provenance in the storages (is_input). The two flags sit side by side (derived here: the simulation calculated it; is_input there: set_input stored it), and the merge conflict in put/_set is mechanical: keep both keyword arguments and both sets.
• Uprate only from inputs, so uprated values don't depend on calculation order #563 (uprating, stacked on this PR): the uprating path has the same class of order dependence (int truncation, float32 drift and defined_for masks compound when a later year is uprated from a calculated intermediate year); Uprate only from inputs, so uprated values don't depend on calculation order #563 uprates only from inputs with Holder.is_derived.

Invariants

For every input set and every sequence of earlier requests (calculate in either unit, calculate_add, calculate_divide, on the simulation and on branches forked from it):

1. Order independence. The value a simulation (or a branch or nested branch) returns for a period equals what a fresh simulation given the same inputs returns for that period alone. Bitwise.
2. Reference rule. It equals the rule above, computed directly from the inputs (tests/fixtures/carry_over.py::reference).
3. No backward carry. With inputs only after the period, the default.
4. Inputs survive options. After calculate_add/calculate_divide, every input a branch reads is still stored and still carries.
5. Unchanged where master already agreed. Single-year PE-US and PE-UK outputs are bitwise identical to master (measured below).

Outside this PR's invariant (pre-existing on master, filed separately): uprating (#563, stacked on this PR); ADD/DIVIDE results read back by STOCK variables; the spiral depth cutoff; subsample exporting calculated values as inputs; on-disk derivative overwriting the parent's input file; _fast_cache eviction on period containment; deleted inputs left in _user_input_keys; on-disk keys with _ in the branch name (#552).

Tests

• tests/core/test_carry_over_order.py: 56 regressions, no Hypothesis import; on master b78b0ba most fail, and those that pass are controls for behaviour that must not change (no backward carry, ADD over an input period, an off-unit input starting with the period, the year-first tie, reform replay of an ancestor input, the copy-on-write read count, defaults as master cached them).
• tests/core/test_carry_over_order_property.py: the Hypothesis property for invariants 1 and 2, 400 derandomized examples plus an @example per regression shape, with branch-side requests and variables with no set_input helper. Skipped where Hypothesis is not installed (the smoke job).
• tests/fixtures/carry_over.py: shared variables and the reference rule.
• Mutation check (mutation_check10.py plus targeted reruns, 33 mutants of the carry-over rule, get_input_periods, the derived marks in holder and both storages, the input guard, ADD/DIVIDE, default caching, the cache wipe, index rebuilds, old pickles and dump/restore): all 31 non-equivalent mutants killed, 12 by the property alone (the rest pin branch, storage, dump and caching paths outside its domain). The 2 survivors are equivalent for current callers: caching a single sub-period's ADD result as derived, and has() implemented through get.
• Randomized soak: 20,000 non-derandomized examples passed on the final commit.
• uv run pytest tests: 1200 passed, 4 skipped, 1 xfailed (local, Python 3.14t). CI runs on Ubuntu and Windows, Python 3.11-3.14, and the smoke job.
• Independent reviews (all executed reproductions): round 1, soundness (GPT-6.1 Sol) and code review (Opus 5.5); rounds 2 and 3, soundness (GPT-6.1 Sol). Every finding in this change's scope is fixed with a regression test that fails without the fix: ADD/DIVIDE overwriting inputs (multi-year, other-unit, anchored, in branches), per-branch provenance, provenance after deletion, uncached defaults, a cached default becoming an uprating base, ties, dump/restore and its branch, the input-helper branch context, copies of Share cached arrays with simulation branches and copy them on first read #556's shared arrays, the per-decision scan cost, the smoke-job import, the property's weakness, the changelog length, unreadable branches, marks across the cache wipe, index rebuilds and old pickles. Pre-existing issues outside this change are filed separately (listed under Invariants). Round 4 (GPT-6.1 Sol, on the round-3 fixes): APPROVE, with two non-blocking notes. First, a P3 performance tradeoff: _calculate decodes stored keys twice per decision (get_known_periods and get_input_periods), which costs extra only for histories beyond the period parser's cache (about 1,024 periods); left as a follow-up. Second, an insertion-order tie between equal-extent inputs, now broken by unit.

Measured impact

Every number below is from a real run: policyengine-us 6c5170fd and policyengine-uk 7b9fc379 (both main on 2026-10-01), core master b78b0ba versus this PR (35e20d9; later commits are tests only). Scripts, logs and arrays are in the review folder (us_bench.py, uk_bench.py, compare.py, full/, round5/).

PE-US, full Enhanced CPS, 2026, single year: 23 of 24 benchmark arrays bitwise identical: income tax $2,141.0853bn, state income tax $521.6485bn and household net income $14,462.0966bn in both, and every other tax, credit, SNAP and household array. One differs, spm_unit_net_income, for 14 SPM units (6,223 weighted), +$3.01m. That is a fix:

• aca_magi_fraction reads the prior-year guideline, tax_unit_fpg for 2025, which reads state_fips for 2025. The dataset stores state_fips for 2024, and the run caches 2026 first. On master, the cached 2026 blocks carry-over to 2025, so every household gets the default state_fips 6 (California) for 2025.
• With California's (contiguous) guideline instead of their own, Alaska and Hawaii households had too low a guideline. With this PR, tax_unit_fpg for 2025 changes for 320 SPM units, aca_magi_fraction for 292, and aca_ptc for 21 tax units: +$3.38m, from $40.8246bn to $40.8280bn (7,130 weighted). Medical out-of-pocket spending net of the credit falls $3.01m for 14 SPM units, raising their SPM net income by the same amount. aca_ptc enters household health benefits, not household net income.
• A shadow core that logs every carry-over decision where this rule and master's differ recorded 3,474 decisions on the full run, 2,675 with a different source. Only this one changed a value; the rest returned the same value (master carrying a calculated default that equals the default).

PE-US, 3,000-household subsample, 2026: 25 of 25 arrays bitwise identical (including marginal tax rates and the itemization branches).

PE-UK, full Enhanced FRS 2024-25, 2026: 36 of 36 arrays bitwise identical (household net income £1,757.970bn, Universal Credit £78.494bn, income tax £313.139bn). The shadow log recorded 6 carry-over decisions, 3 with a different source and none with a different value.

YAML suites under the shadow core:

• PE-UK: 1,370 passed; 0 decisions with a different value.
• PE-US baseline: two of three shards completed, 19,838 tests passed, 0 failed (the third was stopped for host memory). Across 170,432 carry-over decisions, 473 changed a value, all of one kind: a monthly input stored for January, and a later month the run had already calculated (December). Master returned the default for the months in between; this PR carries the January input.
  • Variables: ssi_lives_in_medical_treatment_facility, ssi_medicaid_pays_majority_of_care, receives_ssi.
  • Tests: 8 files (Kansas SSPP, Indiana SSP, California San Bernardino general relief, Texas CEAP). Rerun alone, all 8 pass on master and on this PR: 61 tests each.
  • The third shard, and running policy/reform in one process, run out of memory on stock master too (reform/ passed a 12 GB footprint within 205 s on b78b0ba; both cores grew about 0.6 GB per completed reform test: 12.0 GB after 21 tests on master, 8.1 GB after 13 on this PR). A separate investigation into the YAML test runner's memory growth owns that; it is not caused by this PR.

Multi-year in one simulation (3,000-household subsample):

• 2025 then 2026: master gives 2026 income tax $32.87bn (the monthly age/12 carried by Carry over only values stored at the variable's definition period #557's bug), against $1,962.456bn for 2026 alone. This PR gives $1,962.456bn either way, along with state income tax $494.342bn and household net income $14,012.652bn. What still differs from the single-year run is float32 drift in uprated values (weights and employment income about 1e-7 relative; fixed by Uprate only from inputs, so uprated values don't depend on calculation order #563, stacked on this PR), plus a few tax units in the persistent itemization branches (fixed by policyengine-us#9738).
• 2025, 2026, then 2027: master gives 2027 income tax $31.72bn and Carry over only values stored at the variable's definition period #557 alone gives $2,038.34bn (equal to 2027 alone). This PR raises in 2027 (ParameterNotFoundError ... md.msde.ccs.payment.informal.rates.UNKNOWN): in the itemizing/not_itemizing branches forked at 2025 and reused at 2027, 27 Maryland households get county UNKNOWN.
  • The cause is policyengine-us's county formula. It emulates carry-over by returning holder.get_array(sorted(holder.get_known_periods())[-1]) from the default branch, which inside a branch reused across years returns None.
  • Core then fell back to carry-over. Master and Carry over only values stored at the variable's definition period #557 carry an earlier calculated county; this PR carries only inputs, and the eCPS stores county_fips, not county.
  • Land the policyengine-us fix (read the branch's own latest stored county, or fall through to county_fips), or policyengine-us#9738's per-period branches, before or with the core release that includes this PR. Production paths (policyengine.py, the APIs) build one simulation per year and are unaffected.

Behaviour changes

• Sparse inputs carry forward. Monthly inputs for January and June now give January's value for February to May (master: the default). Each period uses the latest earlier input, as a January-only input already did. A package that wants gaps to default needs explicit gap values.
• Carried and calculated values no longer carry from one another; results now equal what the period gives alone. One visible side effect: with an input for 2011 and a later stored period (say 2014), calculating 2012 now carries the 2011 input and caches it, so get_array(2012) is set; master returned the default uncached. With no input to carry, caching is exactly as on master.
• A helper-less variable with an own-unit input and a later input in another unit carries the own-unit one (as Carry over only values stored at the variable's definition period #557, and as before 3.24.0); no country package declares set_input = None.

After #578 (head 2580d43b)

#578 merged while this PR was open and conflicted with it in in_memory_storage.py. The Fix section above describes the head round 4 reviewed (912a4ac5). Three commits follow it. None changes which input carries for a period with a finite end.

• 12445111: merge of master. Give a storage a set of shared keys only while it shares an array #578 gives a storage a set of shared keys only while it shares an array, because an empty set in every InMemoryStorage cost about 1.3 MB per policyengine-us simulation and ran PE-US CI out of memory (3.32.12 gives every holder's storage an empty set: about 1.3 MB more per policyengine-us simulation #577). This PR had added a second set to every storage and a __setstate__ that gave every unpickled storage its own _shared set, which Give a storage a set of shared keys only while it shares an array #578's tests reject. The merge keeps Give a storage a set of shared keys only while it shares an array #578's code and drops that __setstate__.
• d89a2ea9: the in-memory storage records its inputs, not its derived values. is_derived is now "stored and not an input". A simulation calculates far more values than it is given: in a policyengine-us household, 5,201 of 5,204 stored values are derived. A storage has a set of inputs only while it holds one, and it shares one frozenset with its clones until either changes its own. A pickled state records that inputs were recorded; a state from before counts every stored value as an input, as it did then. The on-disk storage keeps its set of derived keys.
• 6ddbaf9f, 2580d43b: periods ending after 9999-12-31. The carry-over key called period.stop, which raises for a period that ends after the last date datetime has (day:9999-12-30:3), where master carried the input. The key now uses _end_order: the number of the day after the period's last day, by integer arithmetic. It equals stop plus one day wherever stop has a value, and orders every other period by its true end.

Retained memory per simulation. Measured on a policyengine-us 2.18.3 household with one household_net_income calculation: 6,185 storages, and 30,925 holder clones across 5 branches.

Core	Retained	Storages with an index set of their own
master with #578	20.94 MB	0
merge, one set of derived keys per storage that needs one	24.35 MB	1,514
same, shared with clones	22.57 MB	1,514
this head (inputs recorded)	21.25 MB	3 (648 bytes)

The remaining 0.31 MB is one more 8-byte attribute slot in each of the 37,110 storage objects.

Results unchanged. policyengine-us, eCPS 2024 (us-data 1.112.3), 3,000-household subsample, 2026: 25 of 25 arrays bitwise identical between master cbfdedf7 and 6ddbaf9f. 2580d43b changes only _end_order, which the round-2 review checked against the old key on 440,000 candidate sets of positive-sized periods with a finite end: all matched.

Tests.

• tests/core/test_storage_input_index.py pins the input index.
• Give a storage a set of shared keys only while it shares an array #578's differential (test_storage_shared_index_differential.py) now also stores derived values and compares is_derived with a storage that always had both sets.
• Regressions and properties cover the end order, checked against Period.stop and against numpy's calendar.
• Full suite on 2580d43b: 1252 passed, 4 skipped, 1 xfailed. Country-template YAML: 39 passed. CI: 18 of 18.

Reviews of the delta (GPT-6.1 Sol, hard tier):

• Round 1, on 6ddbaf9f: REQUEST CHANGES. Periods ending after 9999-12-31 all sorted equal. Fixed in 2580d43b. It found the storage change sound over 60,000 operations against an independent model.
• Round 2, on 2580d43b: APPROVE WITH NITS. It checked 574,173 endpoints against an independent calendar model, and every earlier surviving mutant is now killed.
• Its two nits:
  1. A day period of size zero or less (outside the Period contract of positive sizes) now ends before a one-day period, where stop treated it as one day.
  2. The ETERNITY test would accept a large finite value. Production returns infinity; the assertion is tightened in a follow-up.

axiom: n/a: engine fix to carry-over in policyengine-core; no policy rule changes

🤖 Generated with Claude Code

[MaxGhenis]

Like-for-like PE-US benchmark: core #557 vs #562. Setup: Enhanced CPS 2024, 3,000-household subsample (fixed seed), policyengine-us main 20ccd5ac with and without policyengine-us#9738. Each cell compares all 24 benchmark arrays bitwise (income tax, itemization, both branch liabilities, CTC, EITC, DE/ID/VA and state income tax, household net income, SNAP and more).

Comparison	core master b78b0ba (with #556)	#557 at 7af22b1	#562 at a80a3b9
2024 single-year vs master	n/a	identical	identical
2025 single-year vs master	n/a	identical	identical
2025 from a 2024-then-2025 run vs 2025-only, PE-US main	20 of 24 differ (age at 1/12: 1,114 tax units' income tax change)	8 differ (formula-branch residual: 124 tax units' ctc_limiting_tax_liability, 1 tax unit's income tax)	8 differ (same residual)
Same, with PE-US #9738	n/a	identical	identical
#557 vs #562, 2025 from the two-year run with #9738	n/a	identical	identical

So on this model the two core PRs are interchangeable. Both remove the age/12 error, both leave single-year results bitwise unchanged, and both need PE-US #9738 for the second year to match exactly. They differ in scope:

• Carry over only values stored at the variable's definition period #557 is an 18-line stateless filter for helper variables.
• Carry over only inputs, the latest at or before the requested period #562 reworks carry-over to take only inputs, with derived marks held in the storages and dumper. That also covers helper-less variables, and it changes the "latest at or before" selection.

Scripts and arrays: ~/reviews/pe-us-second-year-2026-10-01/cmp_bench.sh, rerun_a2.sh, cmp/ on the investigation host.

[MaxGhenis]

Merged at the reviewed head 2580d43b (squash 58e23f56). Audit trail:

• Decision: d820, ruled yes by Max on 2026-10-02: merge with or after Calculate each year's formula branches fresh so multi-year simulations match single-year ones policyengine-us#9738, which merged 2026-10-03 14:11Z.
• Reviews:
  • Round 4 (GPT-6.1 Sol) approved 912a4ac5.
  • Delta round 1 on 6ddbaf9f: REQUEST CHANGES. Periods ending after 9999-12-31 all sorted equal. Fixed in 2580d43b.
  • Delta round 2 on 2580d43b: APPROVE WITH NITS.
• The two nits, both P3:
  1. A day period of size zero or less now ends before a one-day period, where Period.stop treated it as one day. Such sizes are outside the Period contract (positive sizes), so no change.
  2. The ETERNITY test would accept a large finite value. Production returns infinity; the assertion is tightened in a follow-up.
• Checks on 2580d43b:
  • CI: 18 of 18.
  • Local: 1252 passed, 4 skipped, 1 xfailed; country-template YAML: 39 passed.
• Results: policyengine-us eCPS 2024 (us-data 1.112.3), 3,000-household subsample, 2026: 25 of 25 arrays bitwise identical between master cbfdedf7 and 6ddbaf9f. 2580d43b changes only the end order, which round 2 matched against the old key on 440,000 candidate sets.
• Memory: 21.25 MB retained per policyengine-us household simulation, against 20.94 MB on master with Give a storage a set of shared keys only while it shares an array #578.

#563 and #582 follow.

🤖 Generated with Claude Code