Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for

[MaxGhenis]

Three bugs found by the review of #563 (uprating order). None comes from #563: each reproduces on master b78b0ba.

Draft: this changes core behaviour, so the merge decision is Max's.

The bugs, on master and here

	Case	master	This PR
1	Input [1001, 77] for an uprated int variable in 2012; calculate 2013; dump_simulation; restore_simulation; apply_reform with a reform that changes nothing; calculate 2015	[0, 0] (the 2012 input is gone too)	[1116, 85], as in a new simulation
2	Calculate a variable for 2013, then simulation.get_holder(name).set_input(period("2013"), [300, 400]); calculate 2013 again	[1038, 79] while the holder holds [300, 400]	[300, 400]
3a	An Enum variable with uprating, input for 2012, calculate 2015	registers; TypeError: Forbidden operation. The only operations allowed on EnumArrays are '==' and '!='.	ValueError when the variable is registered
3b	A float variable whose defined_for names an Enum variable on the same entity; calculate	registers; the same TypeError	ValueError when the second of the two variables is registered

Two configurations ran on master without an error and are now rejected as well (both executed on master):

• A group variable defined_for a person Enum. Mapping to the group sums the members' Enum indices, so master masked on that sum: a household was kept if any member's value was not the Enum's first member ([20.0, 0.0] for one household with a present member and one without). This branch raises at registration.
• An Enum variable with uprating that is only ever read where it has an input. It never reached the multiplication on master. This branch raises at registration.

No variable in policyengine-us, -uk, -canada, -il, -ng or -au is in either case (see "Country packages").

Fixes

1. restore_simulation restores inputs as inputs

apply_reform keeps the values recorded in Simulation._user_input_keys and drops everything else. restore_simulation put every value back with put_in_cache, which records nothing, so the next apply_reform dropped the restored inputs.

• dump_simulation writes inputs.txt next to each variable's arrays: the periods, one per line, whose dumped value the record names as an input. The record is read the way _invalidate_all_caches reads it back through storage, so an ETERNITY variable's one value is an input whatever period its entry names.
• restore_simulation stores exactly those periods as inputs (through Holder._set inside a _user_input_contexts entry, the path set_input uses, so they land in _user_input_keys), and every other value with put_in_cache, as before.
• A dump with no inputs.txt (written by an earlier version) does not say which values were calculated. Every value in it is restored as an input, and restore_simulation warns. Before, apply_reform dropped every value in it; now it drops none, so the inputs survive, but a reform applied afterwards does not recalculate the values the dump had calculated. That is the rule Make set_input on a branch drop values calculated from the input it replaces #560 and Carry over only inputs, the latest at or before the requested period #562 use for such dumps. The alternative, treating only formula-less variables' values as inputs, would recalculate under a reform but lose any input that had been set on a formula variable.
• to_input_dataframe reads the same record, so a restored simulation now exports its inputs (it exported no input variable before).

2. Holder writes and deletes drop the fast-cache entries they change

Simulation.calculate answers a repeated request from _fast_cache before it reads the holder. Only Simulation.set_input and Simulation.delete_arrays dropped an entry, and only the exact (variable, period) key.

Holder._set (every write: set_input, each store a set_input helper makes, put_in_cache) and Holder.delete_arrays now call Holder._evict_fast_cache:

• Scope. Only the fast cache of the holder's own simulation. A branch has its own holders and its own fast cache and keeps the values it started with. Nothing is dropped when the write is under a branch name that simulation does not read (Simulation._get_visible_branch_names).

• A write drops the entry for the period written, or every entry of the variable if it is defined for ETERNITY (one stored value answers every period).

• A delete drops the entries for every period the deleted one contains, or all of the variable's entries when no period is given: the same rule the storage uses.

• Cost. A write is a dict lookup, and returns at once when the key is not cached, which is the case for every store calculate itself makes (it stores, then caches). Only deletes and ETERNITY writes scan the cache. Measured in the policyengine-us A/B run below (with marginal_tax_rate), eviction took 0.085 s of a 69 s run:

  Kind	Calls	Time	Fast-cache state
  Writes	75,870	0.081 s	caches of up to 20,374 entries
  ETERNITY writes	5	0.002 s	scanning about 20,000 entries
  Deletes	24,060	0.002 s	always on an empty fast cache

  User CPU for the whole run: 61.45 s on master, 61.11 s here.

3. uprating and defined_for need values that support arithmetic

Uprating multiplies the earlier value by an index ratio; defined_for keeps values where another variable is > 0. Enum arrays allow only == and !=, and str and date arrays cannot be multiplied by a float or compared with a number (str and date reproduce the same way on master).

• uprating on an Enum, str or date variable raises ValueError in Variable.__init__, next to the computation-mode check, and in the uprating setter (policyengine-us assigns variable.uprating after loading). It covers an uprating inherited from a baseline variable, since that is what calculate uses; in that case the message points to replace_variable, because update_variable cannot drop an inherited attribute. bool, int and float are unchanged.
• defined_for must name a bool, int or float variable. TaxBenefitSystem.load_variable checks both directions (the variable's own defined_for, and every registered variable defined for it), so whichever of the two is registered second raises, in any load order. While __init__ loads variables_dir, the checks are left to one pass over every variable at the end; that only saves a scan of the variables for each non-numeric one loaded.
• An Enum member is still accepted: defined_for = StateCode.CA names the variable CA.
• calculate repeats both checks with the same message, for what registration cannot see: a defined_for assigned or a variable put into system.variables afterwards, and an uprating assigned on a class that declares uprating itself (such a class replaces the property that checks assignments).
• replace_variable keeps the existing variable when the replacement is rejected. It deleted the old one first, so a rejection would have left the system without the variable.

I did not require a strict bool: policyengine-us has 14 variables whose defined_for names a float (12) or int (2) variable.

Invariants

1. Restore round trip. A restored simulation stores exactly the dumped simulation's default-branch values, byte for byte, and records the same storage keys as inputs.
2. Restore is invisible to requests and reforms. Any sequence of calculate requests gives the same bytes (or the same error) on the restored and the dumped simulation, before and after apply_reform with a reform that changes nothing. After that reform both also equal a new simulation given only the inputs, except in one existing case (see "Not in this PR").
3. The fast cache never changes what calculate returns. For random sequences of calculate, holder set_input / put_in_cache / delete_arrays (under the simulation's own branch name, default, or a branch it does not read), Simulation.set_input, Simulation.delete_arrays and branch creation, every result equals that of a twin tree whose fast caches are emptied before each calculate; so does every stored value at the end.
4. A holder write or delete drops only what it changes: entries of its own simulation and variable, for a branch that simulation reads, for the period written or a period the deleted one contains. Other simulations' fast caches are untouched. (An input set on a branch is left open, because Make set_input on a branch drop values calculated from the input it replaces #560 makes it drop what was calculated from it.)
5. Registration is total for these two attributes. A system that loads has no non-numeric variable with uprating, and no defined_for naming a registered non-numeric variable, whatever order the variables were added in.

Tests

54 new tests; on master b78b0ba 39 fail and 15 pass. The 15 are the cases that must keep working: numeric uprating (3) and defined_for (4), the Enum-member form, an unregistered defined_for, an Enum input without uprating carrying over, a variables directory with a bool condition, a dump with an input record restoring without a warning, and three fast-cache guards (a write under a branch the simulation does not read keeps the entry; branch and parent do not touch each other; calculate's own store keeps its entry).

• tests/core/test_restore_input_registry.py (9, including an input set on a branch, which shares the record the dump reads, and the warning for a dump without a record) and test_restore_input_registry_property.py (invariants 1 and 2, 200 examples, with a branch-input step).
• tests/core/test_holder_write_fast_cache.py (10) and test_holder_write_fast_cache_property.py (invariants 3 and 4, 500 examples).
• tests/core/variables/test_non_numeric_uprating_and_defined_for.py (33): every non-numeric type, both registration orders, a variables directory in both load orders, the setter and its rollback, a reform that turns an uprated or a defined_for variable into an Enum, a group variable defined_for a person Enum, a rejected replace_variable, and the calculate checks for every non-numeric type.
• Shared system: tests/fixtures/uprated_inputs.py. The property modules start with pytest.importorskip("hypothesis").
• Mutation check: 36 mutants of the new code (ignore the input record, record every period or any branch's entries, no ETERNITY normalisation, no legacy-dump warning, no eviction on write or on delete, exact-key delete, no or inverted branch check, evict the whole variable, reversed containment, each registration check removed, each type added to or removed from the numeric set, each half of the calculate checks, no replace_variable rollback), each run against the new tests plus test_fast_cache.py and test_dump_restore.py: 36 killed.

Country packages

No variable in any PolicyEngine country package is rejected. policyengine-us, -uk and -canada were loaded at their canonical heads with this branch's core:

Package	Head	Variables	With uprating	defined_for targets
policyengine-us	d123302f	6,197	366, all float	4,413 bool, 12 float, 2 int
policyengine-uk	3c48247e	976	85, all float	72 bool
policyengine-canada	389648ad	405	none	213 bool

policyengine-il (cf3c7fc), -ng (31182d5) and -au (93ae379) have about 20 variables each, none with defined_for or uprating (static scan at their canonical heads).

In a policyengine-us load the new checks took about 17 ms of 20 s (13,276 uprating checks, 2 passes over the variables, 128 single-variable checks).

Single-year outputs are bitwise identical to master (same country code and data on both arms, core b78b0ba against this branch at 1c24abd, whose production code the later commits do not change):

• policyengine-uk (main c7e826ea, Enhanced FRS 2024-25, year 2026): 36 of 36 arrays identical, including marginal_tax_rate; income tax £313.139bn on both.
• policyengine-us (main fbe24ad1, Enhanced CPS, 3,000-household subsample, 2026, with marginal_tax_rate, which branches per adult and writes and deletes through holders): 25 of 25 arrays identical; income tax $1,962.456bn on both.
• Harness: ~/reviews/core-restore-cache-enum-2026-10-02/ab/ (run_ab.sh, compare.py, outputs in out/).

Relation to open PRs

Each was merged with this branch in a scratch worktree and the full suite run on the result.

• Drop every contained period from the fast cache when deleting or setting a period #566 (fast-cache eviction by contained periods, from chip task_a9863c4b): merges cleanly; 1,206 pass. It changes Simulation.delete_arrays, set_input and the spiral purge; this PR changes the holder, which those all go through. So Drop every contained period from the fast cache when deleting or setting a period #566's 9 tests also pass on this branch without Drop every contained period from the fast cache when deleting or setting a period #566's production change. Merging both is safe: both only drop fast-cache entries, and anything Drop every contained period from the fast cache when deleting or setting a period #566 drops on top (a contained period the holder did not write) is read again from storage. Drop every contained period from the fast cache when deleting or setting a period #566's tests are worth keeping either way.
• Carry over only inputs, the latest at or before the requested period #562 (carry-over) and Uprate only from inputs, so uprated values don't depend on calculation order #563 (uprating, stacked on it): conflicts in Holder._set (keep derived= and the eviction call), at the end of Holder (both add methods; keep both) and in the dumper, where both sidecars are written and read: a value is restored as an input if inputs.txt lists it and derived_periods.txt does not, and a dump with only derived_periods.txt restores every value it does not list as an input. 1,254 pass.
• Read only periods the current branch can see when uprating or carrying over #552 (branch-visible known periods): conflict in _dump_holder only. The resolution reads the input record for the branch that actually stores the dumped value. With it, dumping a branch that set its own 2013 input restores to [2151, 190] for 2015, equal to a new simulation; on this branch alone that case still gives [1116, 85], because master dumps the default branch's values. 1,729 pass.
• Keep the record of set_input values in step with storage #561 (_user_input_keys drift): conflict in Holder._set only (keep is_input and the eviction call). Keep the record of set_input values in step with storage #561 makes a branch copy the input record instead of sharing it; this PR's branch-input test checks the outcome, so it holds either way. 1,228 pass.
• Make set_input on a branch drop values calculated from the input it replaces #560 (branch set_input invalidation): it also writes inputs.txt, in the same format, and restores inputs through its per-array flag, so its dumper replaces this PR's; Holder._set keeps both. This PR's restore tests pass on Make set_input on a branch drop values calculated from the input it replaces #560's dumper once its restore_simulation also warns about a dump without inputs.txt (a dozen lines, in compose/560_simulation_dumper.py). 1,280 pass.
• Give each disk-backed holder storage its own directory #558 (disk storage directories): merges cleanly.

The resolved files are in ~/reviews/core-restore-cache-enum-2026-10-02/compose/.

Not in this PR

• A value calculated from an input that later changes stays stale (the first half of the review's finding 1: set the 2013 input after calculating 2015). Make set_input on a branch drop values calculated from the input it replaces #560 covers it for branches.
• Dumping a branch dumps the default branch's values (the second half of finding 4). Read only periods the current branch can see when uprating or carrying over #552 fixes it; see above.
• set_input helpers count a calculated month as already set. Found by this PR's property test, on master: calculate one month of a monthly variable, then set an annual input, and the input is divided by 11 instead of 12 ([109.09] instead of [100]); a stock variable gives [0] instead of [7]. The property leaves that case out of its comparison with a new simulation. Chip task_fd52e7d5.

Checks run

• uv run pytest tests: 1,197 passed, 4 skipped, 1 xfailed (Python 3.14 free-threaded build, numpy 2.4.2, Hypothesis 6.168.3).
• policyengine-core test policyengine_core/country_template/tests -c policyengine_core.country_template: 39 passed.
• ruff format --check . and ruff check .: clean.
• Independent review (Opus 5.5 on Subfleet): APPROVE WITH NITS, static only. Every finding it raised is addressed in a later commit; the ones that needed running were executed here first (the group-entity and never-uprated Enum cases on master, the replace_variable loss, the setter bypass).
• Not run: make documentation (no docs page changes; the restore_simulation API page is generated from its docstring).

axiom: n/a: core engine, no policy change

🤖 Generated with Claude Code