Keep FRS-reported dividends and key them on person_id

[MaxGhenis]

Part of #495. Fixes the data cause of PolicyEngine/policyengine-uk#1948.

Summary

The enhanced FRS gives Universal Credit claimants six-figure dividends that no respondent reported (PolicyEngine/policyengine-uk#1948). Two defects in the FRS half cause it:

1. frs.py keys the survey's dividends on row position. It sums investment-account dividends to person.index, while account.person_id holds household_id * 1000 + person. Almost nothing matches: in the 2024-25 FRS about £40m of dividends survives, against £8.0bn across 1,717 people when keyed on person_id (gross4 weights). The interest lines a few rows above key on person_id correctly.
2. impute_income overwrites them anyway. It replaces every FRS respondent's dividend_income with a draw from the SPI income model, whose only predictors are age, gender and region. So dividends land on people without regard to their investments, earnings or benefits. In the published data, UC claimants receive them about as often as anyone else.

This PR fixes both.

• Keying. The dividend calculation moves into a tested helper, frs_dividend_income, keyed on person_id and annualised with WEEKS_IN_YEAR like the savings lines beside it. It was * 52, a 0.35% difference.
• No FRS-half overwrite. impute_income no longer imputes the FRS half. The SPI-donor half still carries the SPI income distribution, and calibration to the HMRC dividend band targets reweights between the two halves.

Evidence the dividends were not the survey's

Linking the published-style artifact (enhanced_frs_2024_25.h5, sha256 e433e532…, run with policyengine-uk main 44240bd8, 2026) to the raw FRS 2024-25 gives these aggregates:

2026	Model dividend_income	Raw FRS investment-account dividends in the same households
UC units with over £50k of dividends, FRS half (15 rows from fewer than ten households, each appearing twice through the capital-gains stacking)	£3.13bn	£0.008bn
FRS-reported UC benefit units	£2.91bn	£0.011bn
Whole FRS half	£47.3bn	£18.3bn

Those households report no dividends from a company they direct, so these are not owner-managers either. PolicyEngine/policyengine-uk#1965 adds UC's owner-manager rule (reg 77) separately.

Measured: rebuilds of main and this branch

Builds. Both are real local builds with production settings (512 epochs, PE_UK_DATA_OA_CLONES=1). They used the same trained imputation models, the same raw inputs and the same environment (uk-data uv.lock, policyengine-uk 2.93.0 for the build).

• Main: b45c373, dataset sha256 6baf2953….
• Branch: c9b203e, dataset sha256 c9878b7f….

Both logs show the same failure to download HMRC's salary sacrifice CSV (410 Gone), so that target source is missing from both builds equally (below).

Simulation. Each dataset was simulated with policyengine-uk main (44240bd8) and with PolicyEngine/policyengine-uk#1950 (473adff9). #1950 applies UC Regs reg 66(1): dividends, interest and rent are never UC unearned income. Stored weights; aggregates only.

Dividends (policyengine-uk main, 2026)

	main build	this branch	change
Dividends, all	£64.80bn	£62.85bn	−£1.95bn
FRS half	£47.13bn	£17.88bn	−£29.25bn
SPI-donor half	£17.67bn	£44.97bn	+£27.30bn
Dividend recipients	5.31m	4.67m	−0.64m
Share of UC units with dividends (all units: 13.7% → 11.3%)	11.1%	2.0%
Dividends in UC units	£3.65bn	£1.23bn	−£2.42bn
Dividends in FRS-half units reporting UC in the survey	£3.89bn	£0.0001bn
Dividend income tax	£14.01bn	£13.37bn	−£0.63bn
Income tax	£312.9bn	£314.1bn	+£1.19bn

The #1948 cohort (2026)

UC units with dividends above…	£50k: main build	£50k: branch	£20k: main build	£20k: branch
policyengine-uk main: units (records)	16.1k (12)	0 (0)	40.2k (23)	24.7k (6)
policyengine-uk main: their UC	£0.213bn	£0	£0.537bn	£0.359bn
#1950: units (records)	35.8k (51)	6.6k (13)	82.6k (118)	33.5k (25)
#1950: their UC	£0.501bn	£0.110bn	£1.139bn	£0.472bn

On this branch, no FRS-half record has UC and dividends above £20k, under either rule set. What remains is SPI-donor rows only: 25 records over £20k and 13 over £50k under #1950. Those rows take their household wealth and would_claim_uc from the FRS donor household they were copied from. impute_frs_only_variables re-imputes benefit reports and pension contributions, but not wealth or take-up. Making those consistent with the SPI incomes is the remaining capital-coherence work in #495; the microcosm build has uc_capital_coherence and uc_reporter_redraw stages for this.

The data fix also halves what #1950 costs: +£1.93bn of UC on the main build against +£0.90bn on this branch (2026). Much of #1950's measured gain on the published data went to imputed dividends.

Everything else (policyengine-uk main)

2026	main build	this branch
UC	£79.79bn	£80.16bn
UC caseload	6.346m	6.340m
UC-eligible benefit units	10.33m	10.06m
Housing Benefit	£13.95bn	£14.23bn
Council tax reduction	£2.25bn	£2.48bn
Pension Credit	£6.91bn	£6.96bn
Household net income	£1,756.1bn	£1,768.2bn
Relative poverty BHC / AHC	14.69% / 18.84%	14.87% / 18.91%
Child poverty AHC	23.94%	24.02%
Effective sample size: households / UC recipients	1,218 / 711	1,177 / 745
Largest record's share of UC	0.91%	0.67%
SPI-donor half's share of benefit units (weighted)	9.79m	10.37m

These moves come from calibration. With FRS respondents no longer carrying SPI-scale dividends, the optimiser puts more weight on the SPI-donor half to meet the HMRC dividend targets (+0.58m benefit units), and that half has higher incomes. I have not decomposed them further.

Calibration fit (final epoch, 637 national targets)

	main build	this branch
Within 10% of target	84.3%	84.1%
Median abs(ratio − 1)	0.040	0.042
Mean squared log ratio	6.535	6.534

HMRC dividend targets, estimate ÷ target, by total-income band:

Income band	Dividend amounts, main	Dividend amounts, branch	Recipients, main	Recipients, branch
£12,570–£200k (10 bands)	1.009–1.050	1.033–1.054	1.004–1.048	1.023–1.064
£200k–£300k	1.024	1.073	1.026	1.046
£300k–£500k	1.021	0.510	1.069	1.027
£500k–£1m	0.485	0.565	0.761	1.118
£1m and over	0	0	0	0

The fit worsens in the £300k–£500k amount band and improves in the £500k–£1m band. The £1m-and-over band has an estimate of zero in both builds, a pre-existing gap I have not investigated.

Other targets that moved most, from main to branch, are listed in national_fit.txt among the evidence below. Examples: property income count at £500k–£1m, 2.25 → 1.11; self-employment count at £500k–£1m, 2.01 → 1.04; private pension count at £300k–£500k, 1.02 → 1.28; ONS savings interest, 0.80 → 0.77.

Tests

• test_frs_dividend_income.py (new): accounts land on the person whose person_id holds them, not on whoever sits at that row. It also checks the account-type and gross-up rules and that dividends are never negative.
• test_imputation_source_flags.py::test_impute_income_keeps_reported_dividends_on_the_frs_half (new): only the SPI-donor copy is imputed, and FRS-half dividends and earnings pass through unchanged. It fails on main.
• Full suite locally: 546 passed, 91 skipped, 1 xfailed.

Not in this PR

• Director dividends. The FRS job table's DIVIDEND is not read. Despite its label, the stored value is a weekly net amount: it equals the adult table's DIVIDNET for 176 of 177 people. The FRS puts directors' dividends at £12-20bn a year net (DIVIDNET and DIVIDEND × 52, gross4). Adding them needs a net-to-gross conversion and a check that they are not already in other income fields or in the investment accounts.
• Owner-manager inputs. Filling the new owned_company_* inputs from Model Universal Credit reg 77 for owner-managers of a company policyengine-uk#1965 (from DIRECTOR, DIRPERC and DIVIDEND) needs that release first.
• SPI-donor wealth and take-up. SPI-donor rows keep the donor household's wealth and would_claim_uc (Capital stocks disagree with income from capital, so UC capital tests miss rent, share and savings holders #495).
• microcosm. Its hmrc_spi_income_spine stage redraws base-FRS adults' dividends from the same age/gender/region forest (spi_income.py, required by uk/spec/sources.yaml). It needs the matching change; that is tracked separately.
• Salary sacrifice target. HMRC's salary sacrifice CSV (Tables 6.1 and 6.2) now returns 410 Gone, so both builds ran without that target.

This changes a published dataset, so the merge is a data-release decision.

Evidence (local, aggregates only): ~/reviews/uc-dividends-1948-2026-09-30/ (impact/rebuild/, data-build/*.log, diag/).

🤖 Generated with Claude Code