Build council_tax before council tax reduction; stop imputing reported CTR on SPI rows

[MaxGhenis]

Fixes #496
Fixes #497

Draft. This changes the published dataset, so merging and releasing it is a release decision for Max. No dataset was built, uploaded or released from this branch.

What changes

1. council_tax is the bill after discounts and before council tax reduction (CTR) (council_tax is the FRS bill net of council tax reduction, but the model uses it as the gross liability #496). A new derive_council_tax in datasets/frs.py replaces the inline council tax block in create_frs:
   • Households reporting a reduction (CTREB = 1) get CTANNUAL + CTREBAMT × 365.25/7. This includes CTANNUAL = 0, a full reduction.
   • If a recipient's CTREBAMT is missing or not positive, the reduction is unknown. The bill is the larger of CTANNUAL and the (region, band, single-adult) mean of non-recipients.
   • Missing or negative bills are imputed from the same cell mean. The pool now contains non-recipients only, because recipients' CTANNUAL is net of their CTR.
   • Scotland, from 2024-25: water and sewerage are netted using the gross DVs CWATAMT1 and CSEWAMT1, because CSEWAMT is blank in 2024-25. Non-recipients are netted at 100% of gross and recipients at 65% (see the evidence below). Earlier releases keep the old CSEWAMT + CWATAMTD netting.
   • The stale comment ("Only ~25% of household report Council Tax bills") is gone. In 2024-25, 91.5% of GB households (weighted) have CTANNUAL > 0, 7.5% have 0 and 1.0% are missing.
2. SPI-synthetic rows get zero reported CTR (SPI-synthetic rows carry imputed reported council tax reduction on every person #497). council_tax_benefit_reported is removed from FRS_ONLY_PERSON_VARIABLES and listed in a new SPI_DONOR_ZEROED_PERSON_VARIABLES. impute_frs_only_variables sets it to 0 on the target (SPI-donor) rows.
3. Tests, a changelog fragment, hypothesis as a dev extra (with the uv.lock refresh, which also updates the lock's stale self-version), and .hypothesis/ in .gitignore.

axiom: n/a: dataset construction change in uk-data, no policy rule encoded.

Evidence for #496

DWP's definition. The FRS 2024-25 derived-variable summary (UKDS SN 9563, 9563_dv_summary_2425.xlsx, sheet DV 2425, row 118) defines CTANNUAL as "Annual CT amount after discounts/reduction". It says the DV is calculated using "... GVTREGNO, CTBAND, GROSSCT, WHYNOCT, CTREB and CTREBAMT", and CTREB and CTREBAMT are the reported reduction. policyengine-uk documents council_tax as "Gross annual Council Tax liability before Council Tax Reduction", and council_tax_reduction_maximum_eligible_liability returns it unchanged.

The add-back identity. Raw FRS, gross4-weighted. Recipients have CTANNUAL ≥ 0 and CTREBAMT > 0. Each ratio is their mean divided by the mean CTANNUAL of non-recipients in the same (region, band, single-adult) cell.

FRS year	England, CTANNUAL only	England, + CTREBAMT × 365.25/7	Wales, + CTREBAMT × 365.25/7
2022-23	0.438	1.002	1.040
2023-24	0.454	0.998	1.047
2024-25	0.586	0.989	1.010

The identity holds in the older interview-based years too. Their retired question asked for the amount "after deducting any discounts or reduction". So the add-back applies to every vintage that has CTREB and CTREBAMT. Only the Scottish gross water DVs are limited to 2024-25 onwards.

Scottish water and sewerage. All measured on raw FRS 2024-25 Scottish households, gross4-weighted.

• Status discount. Non-recipients with a 25% status discount were compared with 2+ adult undiscounted non-recipients in the same band.
  • Netting the gross charges gives a tax-only ratio of 0.749-0.762 in every band from A to G.
  • Discounting the charges first gives 0.835-0.865.
  • So DWP's CTANNUAL applies the status discount to council tax only, and the charges are netted at gross.
• Recipients. Recipients were checked against the same identity, using non-recipient tax-only cells by (band, discount status).
  • Netting 100% of the gross charges gives a ratio of 0.83.
  • Netting 65% gives 1.00 overall, 0.98-1.01 in every band from A to F, and 1.00 and 1.02 for the discounted and undiscounted groups.
  • This matches a flat reduction at the Water Charges Reduction Scheme's 35% maximum. Consumer Scotland describes the scheme as a water charge cut of 35% of the household's CTR share, capped at 35% combined with the single-occupancy discount.
  • The real scheme scales with the award, but DWP's derivation behaves like the flat maximum.
• Band D check. Non-recipients in Band D with 2+ adults and no discount:
  • CTANNUAL is £1,890.
  • Netting the gross charges gives £1,335.
  • The old CWATAMTD-only netting left £1,681, which still contains sewerage.
  • The 2024-25 Scottish average Band D excluding water and sewerage is £1,418.11 (gov.scot, via the CTR shortfall investigation notes).

Effect on the base FRS 2024-25. Raw household file, gross4-weighted survey-year £. This is the same code path create_frs now calls. It is not the calibrated EFRS.

Group	Mean council_tax before	after	Share at £0 before	after
England, reported recipients	£779	£1,275	17.8%	0.0%
Wales, reported recipients	£791	£1,306	13.3%	0.0%
Scotland, reported recipients	£758	£928	0.0%	0.0%
England, non-recipients	£1,601	£1,601	8.1%	8.1%
Wales, non-recipients	£1,626	£1,626	3.4%	3.4%
Scotland, non-recipients	£1,551	£1,221	4.2%	4.2%

Non-recipients with a bill outside Scotland are unchanged, record for record. The Scottish non-recipient fall is the sewerage that the old netting missed.

Calibration. The OBR council tax targets use council_tax_less_benefit, which is council_tax minus CTR. For reported recipients, council_tax was already net of CTR, so CTR came off twice. With a gross bill it comes off once. Scottish council_tax_less_benefit also falls by the sewerage, so the Scottish council tax fit will move on the next build.

Evidence for #497

The FRS build puts reported CTR (household CTREBAMT) on the household reference person only. The stage-2 QRF is person-level and predicted it for every person on SPI-donor rows:

• 140 synthetic households have 2-3 reporters;
• no FRS-original household has more than one.

The imputed receipt also runs the wrong way with income. Measured on the 1.56.16 file (base-year values, stored household_weight). Household market income is the sum of employment, self-employment, private pension, savings interest, dividend and property income. Each group includes its capital-gains clones.

Household market income	FRS-original: share reporting CTR	SPI-synthetic: share "reporting" CTR
< £10k	33.9%	5.0%
£10-20k	15.3%	14.1%
£20-30k	10.2%	8.6%
£30-50k	10.5%	15.9%
£50-100k	2.2%	16.7%
£100k+	0.2%	29.3%

policyengine-uk does not treat this as noise:

• would_claim_council_tax_reduction is claims_all_entitled_benefits | council_tax_benefit_reported > 0;
• where council_tax_reduction_scheme_supported is false, council_tax_benefit is the reported amount.

#497 gives the 2026 effect: synthetic rows hold £0.44bn of England's £1.35bn working-age CTR.

Why zero rather than an HRP-only household imputation. The stage-2 model predicts from person-level age, gender, region and personal incomes. A credible CTR imputation would need a separate household-level model (household income, tenure, household composition) and its own validation. Zero is the conservative fix: synthetic households can still get CTR through the model's own eligibility and take-up.

The cost is that zero also removes imputed receipt from low-income synthetic rows: £0.16bn (base year, stored weights) in the under-£20k bands, out of £0.72bn on synthetic rows. If that proves too low, the household-level imputation in #497's first option is the follow-up.

Downstream: policyengine-uk

The consumer is PolicyEngine/policyengine-uk#1966, the Scotland and Wales working-age CTR rules. Its diagnostic run is a real 2026 microsimulation. It used a diagnostic copy of the 1.56.16 EFRS with reported CTR added back to FRS-original rows' council_tax, not a dataset rebuilt from this branch. It gave:

2026 CTR	1.56.16 release data	Diagnostic copy with gross bills
Scotland working-age	£0.133bn	£0.201bn
Wales working-age	£0.009bn	£0.045bn
England pensioners	£0.716bn	£1.026bn

(Source: the CTR shortfall investigation of 2026-09-30, impact/branch_v1.json and impact/branch_v1_grossbill.json.)

This branch will not reproduce those numbers exactly:

• it also nets Scottish sewerage, which lowers Scottish bills;
• it zeroes synthetic reported CTR;
• a rebuild recalibrates the weights.

A rebuilt dataset is needed for the release numbers.

Invariants (stated and tested)

For every household table:

1. council_tax is finite, non-negative and has one value per household.
2. A non-recipient with a non-negative CTANNUAL outside Scotland keeps it exactly.
3. A recipient with a known reduction outside Scotland gets exactly CTANNUAL + CTREBAMT × 365.25/7, so it is never below CTANNUAL.
4. Recipients' bills never change any non-recipient's value, because the imputation pool excludes them.
5. On SPI-donor rows, council_tax_benefit_reported is 0.

Invariants 1-4 are Hypothesis property tests in test_council_tax_gross_bill.py, alongside example tests for each case in the issue.

Differential check (one-off, on the licensed raw files, aggregates only; not committed because the data are licensed). With CTREB and CTREBAMT removed, derive_council_tax equals the old inline block exactly:

• for every household in FRS 2020-21, 2022-23 and 2023-24;
• for every household outside Scotland in FRS 2024-25.

Mutation check. Each of these regressions fails at least one test:

• dropping the add-back;
• putting recipients back in the pool;
• netting recipients' water at 100%;
• replacing the unknown-reduction max;
• removing the SPI zeroing.

Tests

Local, Python 3.13 with uv sync --frozen --all-extras:

• test_council_tax_gross_bill.py: 12 passed. Two of these are Hypothesis properties, at 300 and 200 examples.
• test_frs_only_imputation.py: 6 passed, including the new SPI zeroing test.
• test_legacy_benefit_proxies.py: 14 passed. The end-to-end create_frs fixture now also checks that a full reduction leaves the gross bill in council_tax.
• Full suite (TESTING=1 pytest, no built datasets locally): 530 passed, 113 skipped, 1 xfailed and 4 failed.
  • The skips are tests that need built .h5 datasets.
  • The failures are two OBR NICs tests in test_obr_nic_signal.py and two OBR income tax tests in test_target_registry.py. They fail because the live OBR download returns a non-zip file (the HMRC salary sacrifice CSV also returns 410 Gone).
  • The same four fail identically on unmodified origin/main (b45c373), so they are not caused by this PR.
• ruff format --check . passes with ruff 0.15.5 (locked) and 0.16.9 (latest, as CI installs it).

Not done here

• Scottish water and sewerage charges are zeroed in FRS 2024-25 (CSEWAMT retired) #467 is not fixed. water_and_sewerage_charges is still zero for Scotland in 2024-25. The sewerage netted out of council_tax here does not land there yet, as was already true of the water charge netted since Net Scottish Water charges out of FRS council tax (fixes ~25% Scottish overstatement) #449.
• Recipients with a zero amount keep zero reported CTR. These are CTREB = 1 households with CTREBAMT = 0, 5.1% of weighted GB recipients in 2024-25. They get an imputed bill here but keep council_tax_benefit_reported = 0, as before.
• The wealth imputation predictor moves for recipients. council_tax is a predictor in the WAS wealth QRF (imputations/wealth.py), so recipients' predictor rises to the gross bill. Separately, and predating this PR, the WAS side of that predictor looks mismatched:
  • CTAmtR8 ("Council Tax Amount") is mostly a per-instalment amount. In round 8 its median is £174, 0.1 of DVCTaxAmtAnnualR8 ("Household Annual Council Tax paid"), whose median is £1,520.
  • The FRS side is annual.
  • This needs its own issue.
• CTPAID is still unused.
• No dataset was rebuilt locally, so there are no post-calibration numbers. CI's Test job builds a reduced TESTING dataset.
• uv.lock (and .gitignore, for Calibrate pension-age Housing Benefit to DWP's GB figures #490) may conflict with Drop ons/council_tax_band_d/{code} target emission (#483) #484 and Calibrate pension-age Housing Benefit to DWP's GB figures #490. A relock resolves that.

🤖 Generated with Claude Code