Map FRS EMPSTATI 11 to OTHER_INACTIVE and refuse unknown adult codes (uk-data#526 parity)

[MaxGhenis]

What was wrong

The UK frs_employment stage mapped FRS EMPSTATI with a table that kept the incumbent's truncated-map artifact on purpose: code 11 went to LONG_TERM_DISABLED, and a post-map .fillna("LONG_TERM_DISABLED") sent any code above the domain there too. In the UKDS FRS 2024-25 data dictionary (SN 9563, adult table, EMPSTATI "Adult - Employment Status - ILO definition"), 11 is "Other Inactive" and 9 is "Permanently sick/disabled". policyengine-uk's EmploymentStatus has OTHER_INACTIVE.

In the pinned FRS 2024-25 adult.tab, 916 adults (2.05m grossed) have code 11, beside 1,678 (3.24m) with code 9. All 2,594 came out LONG_TERM_DISABLED. Working-age code-11 adults with a reported status therefore entered the ESA health-condition proxy, and those with no hours worked also the support-group proxy. Code-11 adults at State Pension age were in neither, because both proxies require working age.

This ports the incumbent's fix, PolicyEngine/policyengine-uk-data#526, so the two stay at parity.

The fix

• FRS_EMPSTATI_EMPLOYMENT_STATUS (immutable) maps codes 1-11 one data-dictionary label per line; 11 is OTHER_INACTIVE.
• derive_employment_status_from_frs(empstati, is_adult_record):
  • People who are not in adult.tab are CHILD. The stage still aligns adult.tab by person_id, so child-table people arrive as NaN; they are now recognised by adult.tab membership rather than by the NaN → 0 → CHILD map entry.
  • An adult whose code is blank or outside 1-11 refuses the build, as Two-signal FLSA overtime incidence: usual-hours leg joins the reference-week snapshot (#451 item 4 residual) #526 does, instead of falling back to a guessed status. The error names the offending codes and never prints a count: adults are not survey households, so even 10 or more adults could come from fewer than 10 households.
• frs_legacy_proxies is unchanged. It reads the frame's employment_status, and ESA_HEALTH_EMPLOYMENT_STATUSES is (LONG_TERM_DISABLED, SHORT_TERM_DISABLED), so code-11 adults leave both ESA proxies. legacy_jobseeker_proxy reads UNEMPLOYED only.
• The stage notes in uk/spec/sources.yaml, the source_stages.json mirror and the charter-H2 fixture's copy no longer describe the truncated-map artifact. They cite uk-data#526, because the source manifest refuses the incumbent package name in stage text.

Impact on the licensed input (aggregates only)

Every one of the 27,714 adults in the pinned adult.tab (sha256 4eaea080…658d) carries a code from 1 to 11: none is blank, non-integer or out of range. So the new refusal does not stop the real build. child.tab has no EMPSTATI column.

	Before	After
LONG_TERM_DISABLED adults	2,594	1,678
OTHER_INACTIVE adults	0	916 (2.05m grossed)
Codes 1-8 and 10	unchanged	unchanged

The full per-code table is in experiments/uk-frs-empstati-other-inactive/README.md.

Differential against uk-data#526 (differential_vs_uk_data_526.py, run once by hand; uk-data is not a microcosm dependency and #526 is unreleased). It loads #526's code table and derive function from the PR head (1832adeb; its mapping and accept/refuse logic are unchanged since d3984002, and only the refusal message has changed) by AST:

• the code tables are identical;
• 34 fixed cases (codes 0-12, -1, 11.5, NaN and 99, as adult and as child rows) agree;
• 2,000 Hypothesis examples agree, refusing together, and both sides refuse with ValueError (the differential's run at fb026659 found that Two-signal FLSA overtime incidence: usual-hours leg joins the reference-week snapshot (#451 item 4 residual) #526's message raised TypeError under pandas 3 for a blank code next to another unknown code; its owner fixed that in 89c48e07);
• all 34,966 licensed FRS 2024-25 people get the same status (0 mismatches).

Who reads employment_status (code read this session):

• policyengine-uk 2.100.0, microcosm's lock: no variable formula reads it. dynamics/labour_supply.py reads only the self-employed and STUDENT statuses. The three proxies are not engine variables. Engine outputs and calibration do not move.
• Three instruments see the column, and none of them can tell the two statuses apart:
  • The eFRS parity reference (uk-data 1.56.16) records it as an unweighted non-empty-string share (_nonzero_share).
  • The release input-coverage gate counts rows that differ from the engine default, UNEMPLOYED (_nondefault_signal_mask), so OTHER_INACTIVE counts the same as LONG_TERM_DISABLED did.
  • The input-mass parity skips string columns entirely (input_mass.py), and its reference totals leave out the proxies, which are not engine variables.
• No enum-domain gate covers employment_status.

Timing against uk-data#526

No microcosm test, gate or CI job compares employment_status categories or the ESA proxies with a released uk-data H5:

• policyengine-uk-data is not in uv.lock.
• The only pinned uk-data output is the 1.56.16 eFRS parity reference. The engine-uk parity-reference tests load its committed extraction, which holds employment_status only as a non-empty share.

So this PR does not have to wait for #526's release.

Until the uk-data batch (d833) ships #526, microcosm's code-11 adults and ESA proxies differ from the pinned incumbent. No committed instrument measures that difference. A microcosm merge is not a data release; the next licensed UK build picks the change up.

Pins

• release_input_coverage_manifest.json: all 15 source_manifest_sha256 values (sha256 of source_stages.json), regenerated with tools/build_uk_release_input_coverage_manifest.py; --check passes.
• Nothing else moves:
  • The notes are on the spec's documentation surface, outside stage_contract_sha256. uk_spine.json is unchanged. Regenerating the whole charter-H2 fixture with tools/graph_uk_spine_fixture.py --output <scratch> reproduces every committed file byte for byte (diff -rq exit 0), including the hand-edited fixture.json notes.
  • The kernel implementation hash of uk.stage.frs_employment@1 moves at run time, but nothing commits it.
  • No spec-engine attested module, UK spec digest or gate-battery digest covers this module or the notes.
• The charter-H2 synthetic adult.tab carries codes 1-8 only, so its outputs and the integration-uk smoke build are unchanged.

Invariants (property-tested with Hypothesis)

• The mapping is row-wise: each person's status depends only on their own code and adult flag, and it commutes with row order.
• Non-adults are CHILD whatever their code. Adults with codes 1-11 get exactly their data-dictionary status.
• Any blank or unknown adult code, anywhere in the input, refuses the build. The refusal never prints a count.
• Adult codes plus CHILD are a bijection onto policyengine-uk's EmploymentStatus (engine-uk test).
• Only code 11 moves relative to the truncated map.
• For any ages, State Pension ages, hours and reported flags, the ESA health proxy is exactly reported ∧ 16 ≤ age < SPA ∧ code ∈ {9, 10}. The support group is a subset of it, and only code 9 can be in it. Code 11 never sets either proxy.

Tests

• tests/engine_free/uk/test_uk_frs_employment.py:
  • codes 0-11, as adult and as child rows;
  • the code table pinned to the data-dictionary labels;
  • immutability;
  • unknown adult codes (0, -1, 12, 11.5, NaN, non-numeric) refused, through both the function and the stage derivation;
  • the suppressed error message;
  • the Hypothesis properties above.
• tests/engine_free/uk/test_uk_frs_legacy_proxies.py: every code 1-11 through the stage derivation into the proxies, and the ESA proxy property.
• tests/engine/uk/test_uk_frs_employment.py: the bijection onto the engine enum, and the ESA statuses are engine statuses.
• Local, on the final tree:
  • Engine-free UK and shared suites: 7,222 passed. 8 failed only because the notes and manifest were mid-edit during the run, and all 8 pass when re-run on the final tree. The one remaining failure, test_engine_is_reported_unavailable_without_the_uk_extra, asserts the UK engine is not installed; my local venv has the uk extra, and the engine-free CI job does not install it.
  • Engine-uk: the new enum test, test_uk_release_input_coverage.py (including test_shipped_manifest_is_current), the manifest tests and the charter-H2 test_acceptance_h_parity.py: 56 passed.
  • integration-uk: the synthetic smoke build, run with --run-integration: 2 passed.
  • tools/ci_test_plan.py verify passes, and ruff check is clean on every touched file.
• Mutation check (experiments/uk-frs-empstati-other-inactive/mutation_check.py, run against a temporary copy of the sources): 11 of 11 mutations are killed by the three test files. They include the old behaviour (11 → LONG_TERM_DISABLED), unknown adult codes defaulting to LONG_TERM_DISABLED, every row treated as an adult, adult records found by code presence instead of adult.tab membership, 9 and 10 swapped, an adult count printed, children given a non-CHILD status, code 0 accepted for adults, non-integer codes truncated, the support group ignoring hours worked, and OTHER_INACTIVE added to the ESA health statuses.

Review

Both rounds were independent reviews through subfleet, served by GPT-6.1 Sol after the Opus lane hit its limit.

• Round 1, at f4154ce64: REQUEST_CHANGES, one should-fix and four nits. All five are fixed in c3761888:

  • small mismatch counts suppressed in the differential;
  • each parity instrument described separately;
  • the claim about what CI reads corrected;
  • the PR body reconciled with the reviewed head;
  • the support-group property made exact, with an 11th mutation for it.

  The response is in the session's review notes. CI's first engine-free run also caught the changelog fragment naming the incumbent package, which the live-tree scan refuses; that is fixed in the same commit.

• Round 2, at exactly c37618889a300e6014ee4d7509b4184264071218: APPROVE, with no code findings. Its two PR-body nits (the proxy-impact wording above and a leftover drafting marker) are fixed in this body.

axiom: n/a: data mapping of an FRS survey code to an input column; no policy rule changes.

🤖 Generated with Claude Code