Stop SPI-synthetic rows carrying benefit claims nobody observed

[MaxGhenis]

Draft. This changes the published dataset, so merging it is a data release and Max's call. No dataset was uploaded or released from this branch.

Problem

impute_income stacks a 10,000-household copy of the FRS (household_is_spi_synthetic), replaces its incomes with SPI draws, and calibration gives those rows weight. On the main build (b45c373, policyengine-uk 2.93.0), they hold 8.62m of 31.09m households (27.7%).

Their benefit columns come from two places:

• Second-stage QRF draws. impute_frs_only_variables fills every *_reported column in FRS_ONLY_PERSON_VARIABLES, drawing each person's reports from their age, gender, region and six incomes. It sees nothing of their benefit unit (partner, children, rent, capital), their health or their history.
• Copies of the FRS donor. create_frs builds would_claim_uc, would_claim_pc and would_claim_child_benefit (anchored on the donor's reports), receives_benefits_in_own_right and ssmg_reported before the SPI rows exist.

policyengine-uk 2.93.0 treats a positive report as an existing claim, in three ways:

• Continuing-award gate: housing_benefit_eligible, income_support_eligible, is_WTC_eligible, is_CTC_eligible, esa_income_eligible, jsa_income_eligible, sda and ssmg.
• Claim flag: would_claim_housing_benefit, would_claim_IS, would_claim_WTC, would_claim_CTC and would_claim_council_tax_reduction. Each is claims_all_entitled_benefits | report > 0. claims_all_entitled_benefits sums reports over the whole simulation, so it is False in any dataset run.
• Paid amount:
  • esa_income and jsa_income: the reported award less tariff income;
  • council_tax_benefit, where there is no scheme;
  • esa_contrib, jsa_contrib, incapacity_benefit, iidb, afcs and bsp, by adds.

What that does on the main build (b45c373)

Data file (2024-25), calibrated weights:

	FRS rows	SPI rows
Child Benefit reporters in a benefit unit with no child or 16–19 qualifying young person, by weight	0.0%	34%
ESA (contributory) reporters earning over ESA's permitted-work limit, by weight	not shown (under 10 records)	41%
UC / PC / CB reporters whose unit doesn't claim, weighted benefit units	0 / 0 / 0	462k / 29k / 106k
People whose receives_benefits_in_own_right disagrees with their own reports, weighted	0	2.48m

SPI report rates by weight, relative to FRS rows:

Benefit	SPI rate ÷ FRS rate
IIDB	6.2x
Bereavement support	4.6x
ESA (contributory)	3.6x
AFCS	2.4x
State pension	1.09x

Benefits paid on SPI rows, policyengine-uk 2.93.0, 2025:

Benefit	Total £bn	On SPI rows £bn	SPI share
Universal Credit	75.60	7.36	9.7%
ESA (contributory)	5.06	2.90	57%
Child Benefit	16.90	4.56	27%
Council tax reduction	2.21	0.49	22%
ESA (income-related)	2.15	0.18	8.2%
Pension Credit	6.21	0.16	2.6%
IIDB	0.82	0.53	65%
AFCS	2.40	1.49	62%
Bereavement support	0.59	0.32	53%
State pension	125.90	38.46	31%
PIP	27.70	5.89	21%

Change

apply_spi_donor_benefit_rules (imputations/frs_only.py) runs on the SPI rows after the QRF draw. It runs before the disability categories and flags are derived, because ESA, IIDB and AFCS feed is_disabled_for_benefits.

Reports zeroed on SPI rows:

Reports	Why
UC, Pension Credit, Housing Benefit, council tax reduction, Income Support, WTC, CTC, income-related ESA and JSA, Sure Start Maternity Grant	Entitlement turns on the unit's joint means and make-up, which here come from the imputed incomes. UC and Pension Credit keep a claim route through the redrawn take-up flags (below). The others need a report in 2.93.0, so SPI rows no longer receive them. policyengine-uk 2.93.0 switches tax credits off from 6 April 2025, and IS and income-related JSA from 1 April 2026 (active.yaml). Pension-age HB on SPI rows was £0.03bn. CTR is the one that matters; see "Measured with real builds" and the follow-up. SSMG needs one of these awards and was a donor copy.
ESA and JSA (contributory), incapacity benefit, SDA	Paid only to people out of work or incapable of it, but the QRF gives them to earners. ESA and JSA caseloads are calibrated to DWP figures (620k and 71k), so FRS rows must carry them. FRS rows held 304k of the main build's 724k ESA (contributory) caseload.
Child Benefit	The model reads it only through the take-up flag, and the draw ignores the children.

Donor values restored: IIDB, AFCS and bereavement support. They follow from an injury, military service or a death, not income, and the QRF over-draws them (6.2x, 2.4x and 4.6x by weight). Nothing calibrates them.

Draws kept:

• State pension: paid as reported over pension age. 99.5% of SPI people aged 67 or over report it, against 98.3% on FRS rows (records).
• Winter fuel payment: not read by the model.
• AA, DLA and PIP: drawn below the FRS rates, as the income gradient implies.
• Carer's allowance: the draws respect its earnings limit (no SPI reporter is above it), though policyengine-uk has no earnings test.

Flags rebuilt from the rows' own reports:

• would_claim_uc and would_claim_pc are redrawn with assign_reported_takeup, the helper create_frs now uses too. With no reports, every SPI unit draws at the take-up rate, because whether a synthetic family claims a means-tested benefit at its imputed income is unobserved. The draws use their own seed (101; create_frs uses 100) and don't depend on which columns exist.
• would_claim_child_benefit keeps the donor's flag. The award doesn't depend on the replaced incomes, and the donor's claim is for the same children.
• receives_benefits_in_own_right is rebuilt from the rows' own UC, JSA and ESA reports. That makes it False on every SPI row, so SPI 16–19-year-olds can count as qualifying young people.

The QRF chain is unchanged. Every column stays in it, so the kept draws are bit-identical, and a test checks this. The cost is that the kept values were drawn alongside values that are later zeroed or restored. create_frs keeps its draw order, so the base FRS doesn't change; the builds below confirm it table by table.

Other effects of zeroing that follow from this change:

• ESA and JSA (contributory) are taxable, so SPI rows' total income falls, and that feeds income tax and the HMRC income-band targets.
• Disability flags feed UC's LCWRA element.
• The 16–19 bursary flags and Scottish Child Payment eligibility read some of these reports.

Invariants (tested; Hypothesis for 2–6)

1. The three rule sets are pinned exactly. Changing a column's treatment is a deliberate edit.
2. Zeroed columns are zero, restored columns equal the donor's values, nothing else changes, and the input isn't mutated.
3. A unit with a member reporting UC or Pension Credit claims it, and receives_benefits_in_own_right matches the person's own reports. This is checked with those reports left unzeroed, so the test can fail.
4. The rules are deterministic and idempotent.
5. Draws don't depend on which other columns are present.
6. Ids are shuffled and gapped, so matching by position would fail.
7. With no reports, UC and PC are claimed at the take-up rate (40,000 units, ±1pp), and the Child Benefit flag is untouched.
8. Through impute_frs_only_variables:
   • kept draws are bit-identical with and without the rules;
   • restored columns equal the donor's values;
   • the disability flags match the final reports.

I broke the code in seven ways, and each failure was caught:

Code broken	Test that fails
Rules not applied	8
No donor restore	2, 8
Child Benefit flag redrawn	1, 7, 8
receives_benefits_in_own_right not rebuilt	3, 8
Anchors not rebuilt	3, 7
A draw skipped when a column is missing	5
A column dropped from a rule set	1

test_frs_only_reported_values_correlate_with_training_pattern now checks the QRF's income gradient on carer's allowance, because UC is zeroed on these rows.

Measured with real builds

Two production-settings builds (512 epochs, PE_UK_DATA_OA_CLONES=1), measured with policyengine-uk 2.93.0 (the lock):

• Before: b45c373 + Seed the calibration weight dropout so builds are reproducible #507's seed.
• After: eb14fd0 + the same seed.

Both builds served every target download from the same frozen cache (0 misses). Scripts and outputs are in analysis/ of the evidence folder (local; the h5 files are FRS-derived).

The base FRS is unchanged. All three tables are identical column by column (34,966 people, 18,850 benefit units, 16,288 households). The h5 checksums differ only in file metadata.

Coherence on SPI rows, after:

• 0 Child Benefit, UC, PC or ESA/JSA (contributory) reporters.
• 0 units whose take-up flag disagrees with their own reports (before: 462k / 30k / 106k weighted for UC / PC / CB).
• 0 people whose receives_benefits_in_own_right disagrees with their reports (before: 2.48m weighted).
• IIDB, AFCS and bereavement support now appear on SPI rows at 0.90x, 0.96x and 1.06x the FRS rate (records).

Benefits, 2025:

Benefit	Before £bn	After £bn	Change	SPI rows before → after £bn	Caseload before → after (k)
Universal Credit	75.67	75.41	−0.26	7.34 → 5.42	6,373 → 6,395
Pension Credit	6.20	6.20	+0.01	0.16 → 0.14	1,703 → 1,696
Housing Benefit	12.97	12.96	−0.00	0.03 → 0	1,560 → 1,568
Council tax reduction	2.20	1.73	−0.47 (−21%)	0.49 → 0	3,316 → 2,519
ESA (income-related)	2.15	2.11	−0.04	0.17 → 0	190 → 190
ESA (contributory)	5.08	5.19	+0.12	2.89 → 0	726 → 721 (DWP target 620)
JSA (contributory)	0.30	0.30	+0.00	0.01 → 0	69 → 69
Child Benefit	16.95	16.88	−0.07	4.58 → 4.25	8,400 → 8,363
State pension	125.82	125.82	+0.01	38.47 → 37.50	11,475 → 11,484
PIP	27.73	27.58	−0.15	5.87 → 3.50	3,651 → 3,627
DLA	8.26	8.60	+0.34	1.47 → 1.46	1,375 → 1,414
IIDB	0.83	0.61	−0.22	0.53 → 0.12	255 → 135
AFCS	2.40	0.95	−1.45	1.49 → 0.09	218 → 120
Bereavement support	0.59	0.78	+0.19	0.32 → 0.51	133 → 111

2026 moves the same way; for example, CTR is −£0.47bn and UC −£0.22bn.

Headline:

	2025 before	2025 after	Change	2026 change
Income tax £bn	294.35	293.80	−0.55	−0.60
Household benefits £bn	546.69	544.79	−1.90	−1.94
Household net income £bn	1,709.79	1,708.31	−1.48	−1.49
Relative poverty, BHC	18.42%	18.27%	−0.15pp	−0.25pp
Relative poverty, AHC	21.78%	22.12%	+0.34pp	+0.23pp
Child relative poverty, AHC	29.79%	30.60%	+0.81pp	+0.47pp
Pensioner relative poverty, AHC	15.30%	15.30%	0.00pp	+0.25pp

SPI rows' share of household weight fell from 8.63m to 8.20m: calibration moved weight onto FRS rows to carry what SPI rows no longer report.

What to read into these:

• CTR −21% is the cost of no claim route on SPI rows. It's the follow-up below; until it lands, CTR on SPI rows is understated rather than drawn at random.
• AFCS falls 60%. The QRF had put 62% of it on SPI rows. Whether £0.95bn is closer to the truth than £2.40bn hasn't been checked against MoD statistics here, and nothing calibrates it.
• Bereavement support rises 32%. Restoring donor values fixed the imputation (1.06x the FRS rate by record), but calibration weights those SPI households at 3.7x the FRS rate by weight, and no target constrains it.
• ESA (contributory) caseload stays about 100k above DWP's 620k in both builds. FRS rows now carry all of it.
• DLA rises 4% without any report changing. That is reweighting, because nothing calibrates DLA.
• Child AHC poverty rises 0.81pp in 2025 (0.47pp in 2026). I haven't broken this down by benefit or by reweighting.

Interactions

• Calibrate pension-age Housing Benefit to DWP's GB figures #490 adds a pension-age exclusion to would_claim_uc in create_frs. Whichever of Calibrate pension-age Housing Benefit to DWP's GB figures #490 and Stop SPI-synthetic rows carrying benefit claims nobody observed #514 merges second must apply the same exclusion to the SPI redraw, or Calibrate pension-age Housing Benefit to DWP's GB figures #490's test_built_dataset_has_no_pension_age_uc_claimants fails. This is noted on Calibrate pension-age Housing Benefit to DWP's GB figures #490.
• Solve Pension Credit take-up over entitled benefit units #510 overwrites would_claim_pc for every unit after imputation, filling SPI rows like non-reporters. So it supersedes this PR's PC redraw, and its spi_synthetic_benunits exclusion becomes redundant.
• Build council_tax before council tax reduction; stop imputing reported CTR on SPI rows #499 (SPI-synthetic rows carry imputed reported council tax reduction on every person #497) zeroes council_tax_benefit_reported on SPI rows by taking it out of the QRF chain. This PR zeroes it after the draw instead. Build council_tax before council tax reduction; stop imputing reported CTR on SPI rows #499's test asserting that the zeroed and imputed lists don't overlap contradicts this design, so whichever merges second drops that assertion. Both add hypothesis>=6.168.3 the same way.
• Seed the calibration weight dropout so builds are reproducible #507: both builds above include its seeded calibration dropout (applied locally, not in this branch) and the same cached target downloads, so the comparison isn't swamped by run-to-run noise.

Not in this PR

• Council tax reduction claim route. SPI rows now have no way to claim CTR. A follow-up would give them a claim propensity: the donor's, or a take-up rate derived from FRS rows.
• Donor work status. SPI rows keep the donor's employment_status, hours_worked and the ESA/JSA proxy flags built from them, which can contradict the imputed incomes.
• Other donor copies. Healthy Start vouchers and free school meals are still copied from the donor. ssmg_reported is annualised ×52 on a one-off grant; that's harmless, because only > 0 is read.

Reviews:

• Round 1 (Opus, 96672af): REQUEST_CHANGES, addressed in eb14fd0.
• Round 2 (eb14fd0): APPROVE. Its remaining comments are handled in 90377e0, which drops the small-cell comparator and adds tests for the take-up year and the stage-two UC redraw. Calibrate pension-age Housing Benefit to DWP's GB figures #490's exclusion is an integration requirement and is recorded on Calibrate pension-age Housing Benefit to DWP's GB figures #490.

axiom: n/a: dataset construction in uk-data; no policy rule encoded.

🤖 Generated with Claude Code