Observed firm-microdata readers: Form 5500 and OSHA ITA (#192)

[vahid-ahmadi]

Every firm-side source in the repository so far is a published aggregate — SUSB, BDS, QWI, J2J. This adds the first observed firm records: DOL Form 5500 / 5500-SF plan-sponsor filings and OSHA ITA Form 300A establishment summaries, both public US federal works.

Why this changes the firm-side design

The design has assumed a synthetic firm population because no public firm microdata was believed to exist. It does. The decisive test is whether real records cover the SUSB calibration grid densely enough to reweight rather than generate:

SUSB 2022 target cells (20 NAICS sectors x 5 canonical bands)	97
Cells with zero real records	0
Cells with fewer than 10 real records	0
SUSB employment in uncovered cells	0.00%
Real firm records available	987,039

Implied cell weights (SUSB firms / real records): median 1.7, p90 14.7, max 225.2 (NAICS 99 1-9, an 8,332-firm cell). Density is thinnest in 1-9 (74 records per 1,000 SUSB firms) and near-census in 100-499 (941 per 1,000).

A reweighted real-firm frame is therefore viable and no synthetic firm row is required. This PR is the reader layer only; the frame construction, the unit reconciliation and the calibration are separate work.

Exact banding — the reason these sources are worth having

Both carry integer employment counts, so firms.banding.band_of_count maps them onto the canonical IC2 bands exactly. Neither ever produces a straddling BandSpan, unlike CPS NOEMP, SIPP EJB1_EMPSIZE, BDS fsize ("20 to 99") or LEHD firmsize ("0-19"). Tests pin the intended band for every canonical edge independently of banding.py's own tables.

Three findings the readers are built around

1. The Form 5500 sponsor EIN is not the SUSB enterprise. 23,813 sponsors report 500+ active participants while SUSB 2022 counts 21,041 firms with 500+ employees — more large sponsors than large firms exist, which is only possible if the units differ. Active participants are additionally a lower bound on employment (plan-covered workers only). 122,173 of 828,330 sponsors file more than one form and the counts overlap rather than add, so read_sponsors aggregates by maximum and never by sum.

2. The OSHA employment field is dirty in a concentrated way. The raw annual_average_employees column sums to 321,889,014 — 237% of total US employment. The objective physical test (annual hours per employee cannot exceed 8,760) removes only 1,996 records holding 235,875 employees. About a dozen records carry the rest, and the cap chosen to remove them swings the national total by 32%:

Rule	Surviving employment
raw	321,889,014
after hours/employee <= 8,760	321,653,139
+ cap 50,000	47,838,489
+ cap 613,000	61,188,048
+ cap 1,000,000	62,912,901

That makes the cap a referee choice with a quantified effect on every downstream margin, not a loader detail. read_ita therefore applies no cleaning at all and attaches hours_per_employee / hours_implausible / bandable flags; apply_quality_rule takes max_employees as a required argument with no default and records exactly what it dropped on the returned frame. This mirrors BandSpan, which carries ambiguity explicitly rather than resolving it by hidden convention.

3. Two administrative firm-size measures disagree one time in three. 43,001 EINs appear in both universes; they agree on the canonical band for 66.3% of them, median ratio 0.89, with disagreement running in both directions (1,451 firms are OSHA 100-499 but 5500 500+ — impossible if participants were merely a subset of employees). Registered here as evidence, not as a threshold. It is a measurement-floor observation and it bounds how tightly any firm-size gate can be set: a threshold finer than 66.3% band agreement is finer than the instruments resolve.

Staging and provenance

Raw files stay outside Git (the SIPP/PSID convention) under POPULACE_DYNAMICS_FIRM_DIR, pinned by sha256 recorded at fetch time by scripts/fetch_firm_microdata.py. Provenance sidecar: data/external/firm_microdata_sources.md.

The DOL .../Latest/... URLs are refreshed in place as amended filings arrive, so a digest identifies the vintage actually read, not a stable publisher artifact; a mismatch after a refresh requires a deliberate pin update and a rebuild of anything that read the old bytes.

All three sources are works of the US federal government published without stated redistribution restriction — unlike the Kauffman Firm Survey (#339), whose archive carries no explicit redistribution license.

Verification

• 26 new tests pass, including real-data tests that reproduce every pinned figure in the provenance sidecar against the staged files (231,725 / 763,552 filings; 790,028 sponsors; 23,813 at 500+; 383,283 OSHA rows; raw sum 321,889,014; 1,996 hours-implausible; 61,188,048 at cap 613,000)
• full unit tier: 1,549 passed, 7 skipped
• tier manifest and README updated: unit 1,530 -> 1,556; total 5,551 -> 5,577
• Ruff and Black (79) clean on every changed file

Scope

Reader and provenance work only. It does not modify gates.yaml, IC3 targets, floors, thresholds, partitions, calibration, or candidate fitting, and authorizes no candidate run. Based on master rather than stacked on the IC3 queue, so it can be reviewed in parallel.

tests/tier_counts.json will conflict trivially with #212/#235/#276 — whoever merges second recounts from current master, per the existing convention.

Relates to #192, #282. Supersedes the firm-microdata role of #339.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
social-security-model	Ready	Preview	Aug 12, 2026 11:18am