SPI: keep Marriage Allowance on recipients (MAIND) and give transferors their cut allowance

[MaxGhenis]

Depends on PolicyEngine/policyengine-uk#1963 (fixes policyengine-uk#1947). Do not merge until #1963 is merged.

This PR does not bump the policyengine-uk lock (2.93.0). A lock bump changes the dataset build, so it belongs in its own data release. Models before #1963 ignore the new marriage_allowance_relinquished column and treat marriage_allowance as before, so merging this changes no model output until the lock moves.

What #1963 changes

marriage_allowance becomes the transferable amount a person receives from their spouse's election. It gives a Step 6 tax reduction instead of coming off taxable income. The transferor side is makes_marriage_allowance_election / marriage_allowance_relinquished, which cuts the transferor's personal allowance. datasets/spi.py sets marriage_allowance from the SPI variable MAIND, so the question is which spouse MAIND marks.

Which spouse MAIND marks: the recipient

HMRC's wording alone does not say. From the SPI 2022-23 Public Use Tape documentation (UK Data Service SN 9422):

• Annex A, p. 15: "MAIND — Marriage allowance claimant indicator — -1 – Not classified; 0 – No claims; 1 – Claimant".
• Annex A, p. 16: "PAS — Personal allowance (includes 10% marriage allowance transfer if applicable). — 0 – 13,830".
• Section 3, p. 7: "HMRC collects data regarding claimants (receivers and transferers) of Marriage Allowance through coding adjustments for those in NPS or via SA returns."

The 2020-21 documentation (SN 9121, p. 19) has the same label, marked "NEW FOR 2021".

The tape settles it (aggregates only):

	2022-23	2020-21
MAIND == 1 records (weighted)	30,888 (2,358,560)	28,847 (2,143,161)
HMRC published claimants (non-structural tax relief statistics, December 2024, s. 5.13)	2,350,000	2,170,000
MAIND == 1 with PAS = allowance + transferable amount	100% (£13,830)	79.2% (£13,750); the rest have £12,500
MAIND == 1 with PAS = allowance − transferable amount	0	0
MAIND == 0 with PAS = allowance − transferable amount	13,673 (1,197,068)	10,373 (969,794)

So MAIND == 1 is one spouse per couple, the one whose allowance HMRC raises: the recipient. The decisive fact is that no MAIND == 1 record has the cut allowance. The count match is weaker evidence, because HMRC calibrates the tape to its published claimants (p. 7: "To calibrate to published Marriage Allowance claimants, estimated values are imputed to cases"), so MAIND and the transferors' PAS are partly imputed. That publication describes the relief from the recipient's side: "Gives a tax reduction to a person whose spouse or civil partner has elected for a reduced Personal Allowance." In 2020-21, 20.8% of MAIND == 1 records have the plain £12,500 allowance; I have not established why. Transferors are in the tape unflagged, with PAS cut by the transferable amount. That is the cluster policyengine-uk's docs/book/validation/spi-validation.ipynb describes ("the SPI removes the transferred amount from the individual's Personal Allowance, but does not indicate that the subsequent lower amount is driven by the Marriage Allowance").

What changes here

• marriage_allowance stays on MAIND == 1: the input is consistent with #1963. The stale comments now point to marriage_allowance_transferable_amount and ITA 2007 s. 55B(4)-(5). One of them said HMRC rounds down; the code and the statute round up.
• New: records with MAIND == 0, PAS equal to the personal allowance less the transferable amount, and TI under the £100,000 taper threshold get marriage_allowance_relinquished. Each SPI record is one person, so the model cannot derive an election from a spouse; setting makes_marriage_allowance_election would do nothing, because marriage_allowance_relinquished also requires is_marriage_allowance_spouse.
• No double counting: create_spi does not input PAS, and a test now guards that. The model computes the allowance itself, so the recipient's SPI allowance uplift and the transferor's cut each enter once, through these two inputs.
• The 26 records in 2022-23 with PAS = £11,310 and TI ≥ £100,000 are left alone. All 26 have a higher-rate marginal band (MAR), which rules out an election (s. 55C(1)(c), (ca)). None has TI − PENSRLF − GIFTAID − GIFTINV under £100,000, and for 19 that figure is within £2 of £102,520, where the taper gives exactly £11,310.
• _get_allowances no longer falls back to 2020-21 values when policyengine-uk cannot be imported. On a 2022-23 tape that fallback would have found no transferors and raised no error.

Nothing published moves. No production code reads the SPI h5: the income imputation reads the tab directly and never uses MAIND, and the income projection now uses HMRC's aggregate tables. load_spi_dataset has no caller outside tests.

SPI validation (2022-23 tape, real Microsimulation runs)

Share of records whose PolicyEngine income_tax is within £10 of the SPI's TOTTAX_DEVO_TXP. 834,538 non-composite records (MAIND != -1); 4 runs. Unknown region codes were mapped to SOUTH_EAST so the runs complete (see the note below).

Group	Records	policyengine-uk main (44240bd8d), spi.py on main	#1963 (292f7c8b2), spi.py on main	#1963 + this PR
All	834,538	91.05%	91.00%	91.15%
All, weighted		96.59%	96.50%	96.74%
Recipients (MAIND == 1)	30,888	94.08%	92.82%	92.82%
Transferors (PAS = £11,310, TI < £100,000)	13,647	90.33%	90.33%	99.68%
Everyone else	790,003	90.94%	90.94%	90.94%
PolicyEngine income tax (SPI: £204.35bn)		£203.70bn	£203.69bn	£203.71bn

Why recipients match less often under #1963 (run 1 vs run 2, same dataset):

Recipients	Records	Old model	#1963
Scottish (region or SCOT_TXP)	2,075	94.2%	52.7%
– of which marginal rate STIR (intermediate)	845	96.8%	0.8%
Not Scottish, with dividends	3,448	74.3%	79.1%
Not Scottish, with savings income, no dividends	10,824	95.6%	96.5%
Not Scottish, other	14,541	97.6%	99.0%

• Scottish recipients: the SPI's tax matches the old allowance treatment, where an intermediate-rate payer saves 21% × £1,260 = £264.60. #1963 gives 20% × £1,260 = £252, the Scottish basic rate that s. 55B(3) requires. The £12.60 gap is over the £10 tolerance. HMRC's documentation is consistent with an allowance treatment in the SPI: PAS "includes 10% marriage allowance transfer", and its variable relationships (p. 19) list "Personal Allowances = PAS + BPADUE" with no separate Marriage Allowance reduction. This is a difference between the SPI's calculation and the statute, not a dataset error, so nothing here tries to reproduce it.
• Other recipients: 485 more records match and 13 fewer.

Invariants and tests

For every record: each of marriage_allowance and marriage_allowance_relinquished is 0 or the transferable amount; no record has both; only MAIND == 1 receives; exactly the unflagged records (MAIND == 0) with the cut PAS and income under the taper threshold relinquish; personal_allowance is never a dataset input.

• test_create_spi_marriage_allowance_invariants: exhaustive over 60 combinations of MAIND × PAS × TI. Hypothesis is not a dependency of this repo. Mutation checks: changing the transferor condition from MAIND == 0 to MAIND != 1, or narrowing the income condition to TI < 50_000, fails it.
• test_create_spi_marriage_allowance_sides: named cases (recipient, transferor, no transfer, tapered look-alike, composite record).
• test_spi_marriage_allowance_matches_statute_in_policyengine_uk: runs SPI-shaped records through policyengine-uk. Recipient saves £252 in England and as a Scottish intermediate-rate payer; the recipient's own allowance stays £12,570; the transferor's is £11,310 and their tax is 20% × £690. Passes on #1963's head. On policyengine-uk main it would fail at £264.60, which is what pins the new semantics; so it skips, with that reason, until the installed model defines marriage_allowance_relinquished. In CI it starts running when the lock reaches a release with #1963.
• Existing fake tapes gain PAS and TI. The fiscal-year test now asserts £1,260 exactly instead of "not £1,250".

Locally (test_spi_build.py, test_spi_allowance_deductions.py): 20 passed against #1963's head; 19 passed and 1 skipped against policyengine-uk main.

Review

Two independent rounds (Opus 5.5, Subfleet lanes, read-only tools, so the reviewers read the code and sources but did not run the tests).

• Round 1, approve with changes: the statute test's skip on main, the silent 2020-21 fallback, the imputation caveat, the 26 high-income records, and the MAIND == 0 gap in the invariant grid. Addressed in ba07c78 and in this description.
• Round 2, approve at ba07c78, with one more assertion requested (that exactly the qualifying records relinquish), added in the next commit, and two wording fixes made here.

Because the statute test skips in CI under the current lock, green CI does not show the new inputs work with the merged model. Before merging, both test files must be run against merged policyengine-uk main: 20 passed, none skipped.

Before merge

• policyengine-uk#1963 merged
• 20 passed, 0 skipped against merged policyengine-uk main
• CI green

Note for a follow-up

create_spi maps GORCODE 13 ("Address abroad") and 14 ("Address unknown or not available"), 16,547 records, to "UNKNOWN", and policyengine-uk cannot simulate that region (ParameterNotFoundError on private_rental_prices.UNKNOWN). load_spi_dataset works around it; create_spi output on its own does not simulate.

axiom: n/a: data (SPI dataset inputs; the rules change is policyengine-uk#1963).

🤖 Generated with Claude Code