Impute UC deductions and private school attendance in constituency and local-authority runs

[MaxGhenis]

Fixes #1902

Stacked on #1899. Do not merge before it. This branch builds on #1899's head (c46893e), which adds Simulation.built_from_dataset. Until #1899 merges, the diff below includes its commits. This PR's own changes are the last two commits.

Summary

Three variables chose between microdata imputation and household defaults by testing whether total weight was below 1e6:

Variable	Old test	Default below 1e6
uc_deduction_random_draw	benefit-unit weight	draw of 1.0, so no UC deductions
uc_deduction_type_random_draw	benefit-unit weight	1.0
attends_private_school	household weight projected to each person (person.household(...)), so in effect people	no private school attendance

policyengine.py builds constituency and local-authority simulations by filtering rows from the national data. src/policyengine/countries/uk/regions.py uses RowFilterStrategy on constituency_code_oa / la_code_oa. filter_dataset_by_household_ids in src/policyengine/utils/entity_utils.py keeps rows without rescaling weights. run() in src/policyengine/tax_benefit_models/uk/model.py wraps them in UKSingleYearDataset and calls Microsimulation. So every constituency and local authority fell below the threshold and got household defaults: no UC deductions, and no private school attendance even under a private school VAT reform.

Each variable now gates on getattr(<entity>.simulation, "built_from_dataset", False), which is true for any simulation built from data, however little weight it carries. attends_private_school also loses a dead hasattr(person.simulation, "dataset") check (Simulation.dataset is a class attribute, so it always held).

filter_dataset now carries each person's attends_private_school into the household it extracts, as #1899 does for months_since_last_birthday. A household alone ranks at the 100th income percentile (rate 0.47 × 0.85), so without this about 40% of children in an extract would be assigned to private school. UC draws need no carrying: they hash benunit_id, which the extract keeps.

attends_private_school also no longer raises when no household has weight. The old gate returned before ranking, and MicroSeries cannot rank zero total weight. Households without weight stay at percentile 0, as before.

The attends_private_school YAML cases were vacuous under the new gate (situations always return False), so they are replaced with situation cases: a household with 1e9 of weight and the top income attends no private school unless set, and a set value is kept.

clone(), get_branch() and subsample() copy or keep the instance __dict__, so the flag survives into baseline and branch simulations.

Invariants (stated and tested)

policyengine_uk/tests/test_data_built_imputations.py:

1. Construction decides, not weight. In a data-built simulation, scaling every weight by 2^k for any k in [-14, 30] (totals from about a hundred to about 10^15) leaves the draws, uc_has_deduction, uc_deduction_combination, uc_deductions and attends_private_school unchanged (Hypothesis). Powers of two scale exactly in floating point, so the property holds exactly and can't flake on percentile boundaries.
2. Row-filter invariance for UC deductions (differential test). A region filtered from a national-scale data-built simulation (about 2.3m of weight, the region about 0.6m), with rows kept as RowFilterStrategy keeps them, gives every benefit unit the draws, deduction flag, combination and amount it has in the full simulation.
3. Hashed draws in small data. A data-built simulation with under a million units of weight gets splitmix64_uniform(benunit_id) draws, some deductions and some private school attendance.
4. Extracts keep their imputations. A filter_dataset extract reproduces the full simulation's UC deductions and private school attendance, and the test asserts both sets are non-empty.
5. Data without weight. A data-built simulation where every household has zero weight attends no private school and still gets hashed draws.
6. Situations get defaults whatever their weight. A household situation with 1e9 of weight gets draws of 1.0, no deductions and no private school attendance.

All six fail on #1899's head, and the first new YAML case fails there too. The extract test also fails with the new gates but without the filter_dataset carry.

Intended exception: private school attendance is not row-filter invariant. It ranks incomes within the simulated population, so a constituency ranks against itself (see caveats).

Constituency and local-authority runs, before and after

Real runs, following policyengine.py's path: filter the national tables by constituency_code_oa / la_code_oa / region, build UKSingleYearDataset + Microsimulation, and calculate 2026.

• Data: enhanced FRS 2024-25, policyengine-uk-data-private 1.56.16, the policyengine.py uk-6.2.0 bundle.
• Before: Set State Pension age from date of birth, including the rise to 67 #1899's head, c46893e. After: 40aaab0. The later review commit changes only the zero-weight path, tests and docs.
• National run: the unfiltered simulation (identical before and after on every metric), grouped by the same geography.
• Runs: 632 GB constituencies (Northern Ireland records carry no constituency or LA code), 363 GB local authorities and Northern Ireland, each run under both versions: 1,992 filtered simulations plus 2 national runs.

632 constituencies (sums over constituency runs)

	Before	After	National run
Constituencies with any UC deductions	0	596	597
UC benefit units with deductions (k)	0	2,938	2,940
UC deductions (£m)	0	1,901	1,902
UC paid (£m)	79,609	77,708	77,618
Private school pupils (k)	0	811	865
Constituencies with private school pupils	0	578	532

363 local authorities

	Before	After	National run
LAs with any UC deductions	0	346	346
UC benefit units with deductions (k)	0	2,939	2,940
UC deductions (£m)	0	1,901	1,902
Private school pupils (k)	0	809	865

• Household net income falls by £1,901m summed over constituencies, equal to the deductions.
• UC deductions match the national run to within float32 rounding (relative gap ≤ 1e-6) in 619 of 632 constituencies and 350 of 363 LAs.
• The rest differ because UC entitlement differs between the filtered and national runs, not the draws. In the one case examined (E14001101, one differing benefit unit), a 66-year-old is under State Pension age in the constituency run and over it nationally. Set State Pension age from date of birth, including the rise to 67 #1899's months_since_last_birthday spreads birthdays within the simulated population. UC paid differs by more than 0.1% in 168 constituencies. I did not trace every case.
• Northern Ireland as a filtered region is unchanged: in 2026 it carries 1,006,938 of benefit-unit weight and about 2.0m people, both above the old threshold.

Caveats and follow-ups

• Private school attendance ranks locally. Constituency totals come to 811k against 865k from the national run. Per constituency, the correlation with the national run is 0.61. The ratio has a median of 1.09 and a 90th percentile of 24, as poorer areas give their top local earners top-percentile rates. The previous behaviour was zero everywhere. Ranking by national income needs the percentile, or attendance itself, carried in the data (follow-up task).
• Separate issue found while measuring, not changed here. Several incidence variables spread a national total over the simulated population by each household's share of a weighted sum, e.g. shareholding and corporate_land_value. In a filtered constituency this puts the whole national total on the constituency. For E14001063 (87 records), corporate_tax_incidence is £34,863m in the constituency run vs £28m for the same households nationally, and business_rates is £31,733m vs £25.6m. Household net income is −£35,634m vs £2,328m. This affects every constituency run in policyengine.py (follow-up task). The net-income figures above are differences, in which it cancels.
• UC draws and filter_dataset affect only data-built simulations. Household calculators (situations) are unchanged: they got the defaults before and still do.

Tests run

• test_data_built_imputations.py: 6 passed.
• test_uc_deductions.py + test_state_pension_age.py: 37 passed.
• contrib/labour/attends_private_school.yaml + private_school_vat.yaml: 5 passed.
• Independent review: an Opus 5.5 peer approved with nits. I executed and fixed the zero-weight crash it found, and addressed its test nits.
• ruff format / ruff check clean.
• The full suite was not run locally (the targeted tests cover the changed variables); CI runs it.

axiom: n/a: microsimulation imputation

🤖 Generated with Claude Code