Evaluate recursions over periods in full when they reach an input or the start of a formula

[MaxGhenis]

Summary

max_spiral_loops caps how many frames of one variable the calculation stack holds; beyond it _check_for_cycle raised SpiralError and the deepest frame got the variable's default. The cap counts uncached frames only, so a recursion over periods gave a different result depending on what had been calculated before:

recursive(t) = recursive(t - 1) + 1 (formula_2010), input 1 in 2010	New simulation	After the previous year
max_spiral_loops = 1, 2012	1	3
default (10), 2021	10	12

The true values are 3 and 12. On real data, policyengine-us weeks_worked (formula_2025: last year's value; eCPS input in 2024) is cut in a new 2035 simulation and returns the default from 2025, which feeds SPM work expenses. Found while reviewing #562 (soundness review, finding 6).

The rule

1. Anchored recursions are evaluated in full. When a recursion reaches the cap and one turn of it (the frames since the variable's previous frame) heads towards something that ends it whatever is cached (a period with no formula: before a dated formula's start going back, after end going forward; or an input of the variable's own unit), the outermost calculate calculates that deepest period first, with an empty stack, and then starts its own request again. Deferred periods are kept in a list, so a chain of any length is evaluated max_spiral_loops frames at a time without growing the Python stack. The control-flow exception, _DeeperPeriodFirst, derives from BaseException so a formula catching Exception doesn't swallow it.
2. A frame with no formula and no adds/subtracts at its period ends a recursion and is never treated as a spiral.
3. Other recursions are cut as before, but no value calculated while a cut is on the stack outlives that calculation: frames on the stack, and everything cached until every active stack of the related simulations empties, are purged then. Such values used to stay cached above the cut (OpenFisca ac83748, test_spiral_cache) and gave later calculations results that depended on order. They are still reused within the calculation that made them.
4. Purges hit only what was calculated. They delete exactly the cut periods (master deleted every period they contain, inputs included) under the calculating simulation's branch name (master deleted under default, so a branch kept its cut values), including branches made since. Each clone has its own invalidated_caches (master's branches shared the parent's set object, so a branch's cut could later delete the parent's values).

Known residual, same as master: a recursion whose only base case is a condition inside a formula, or a defined_for mask, is still cut at the cap.

Invariants

• For any system of recursive variables built from dated or undated formulas and inputs, any max_spiral_loops and any earlier requests, a request returns what it returns in a new simulation. Property test: tests/core/test_spiral_order_property.py (Hypothesis, 200 examples: one or two yearly variables, lags 0-2 years, dated or undated formulas, inputs, limits 1, 2, 3, 10). It fails on master. A broader scratch exploration over the same space found 80 order-dependent systems in 2,000 on master and 0 in 7,500 with this change.
• An anchored recursion's value equals the value from calculating every year in order.
• Cycles still raise CycleError; recursions with no anchor still return the same cut value as master on a new simulation.

Downstream impact

Real runs on master b78b0ba and on all three of #571/#572/#573 merged together (scratch merge 37aadada), each output compared array by array (compare.py). The PE-US 2026 run was also repeated on each branch alone, and the 2035 run on #572 alone; all identical too.

Run	Outputs compared	Result
policyengine-us 6c5170fd, eCPS 2024, 3,000-household subsample, 2026 (income tax, itemizing and SALT branches, state taxes, CTC, EITC, SNAP, household and SPM net income, weights, marginal tax rates)	25 arrays	bitwise identical
same, 2035	25 arrays	bitwise identical
policyengine-uk 7b9fc379, enhanced FRS 2024-25, full sample, 2026 (income tax, NI, UC and legacy benefits, Pension Credit, HB, council tax, HBAI income, poverty, marginal tax rates)	36 arrays	bitwise identical

A probe of the PE-US 2026 and 2035 and PE-UK 2026 and 2042 runs on master found one recursion cut by the limit: PE-US weeks_worked in 2035 (below).

PE-US weeks_worked (formula_2025: last year's value) is the one recursion the probe saw reach the limit on real data: a new 2035 simulation cuts it at 2025. The eCPS file carries no weeks_worked, so every year is the default 0 and the cut changes nothing above. Where the input exists it does. A real policyengine-us run on a household with one worker (40 weeks entered for 2024):

2035	New simulation	After calculating 2030
master: weeks_worked, spm_unit_work_expenses	0, $0	40, $1,800.14
this branch	40, $1,800.14	40, $1,800.14

So household calculations from 2035 on (ten years after the 2025 formula start) change, to the value calculating year by year gives.

Tests

• tests/core/test_spiral_order.py: 19 regressions (anchored by input, by a dated formula, by end, through another variable; unanchored cut; values reached by and above a cut not kept; skipped input does not loop; cycles; except Exception formulas; the deferral cap; branches traced and untraced, called from formulas, made after a cut; a branch's cut leaving the parent's input alone). 14 fail on master; the other 5 pin behaviour kept (unanchored cut value, cycles, a skipped input).
• tests/core/test_cycles.py::test_spiral_cache now asserts the value above a cut is not kept, and that a later request returns the same value.
• tests/core/test_fast_cache_guards.py: its stub holder gains delete_array.
• Mutation check: 12 mutants of the guards (no deferral, no futility guard, which hangs, default-branch purge, no marking after a cut, no formula-less exemption, own-variable-only anchors, no descendant purge, shared invalidated set, no related stacks, no input anchor, no formula anchor, no branch purge), all killed.
• Full suite: 1163 passed, 4 skipped, 1 xfailed.

API

New: Holder.delete_array(period, branch_name) (exact period), InMemoryStorage.delete_exact, OnDiskStorage.delete_exact, module constant MAX_SPIRAL_DEFERRALS. calculate now delegates to _calculate_traced. invalidate_spiral_variables marks every frame on the stack. Country packages: PE-UK's Simulation.__init__ sets invalidated_caches itself (unaffected); PE-US tools/branched_simulation.py (unused) replaces _check_for_cycle.

Composition

#566 also edits purge_cache_of_invalid_values (fast-cache eviction line); the conflict is textual. Merges cleanly with the ADD/DIVIDE and subsample PRs from the same review.

axiom: n/a: core engine, no policy encoding

🤖 Generated with Claude Code