microcosm

The population stack: one kernel datatype — the Frame, a weighted sampling frame of entity tables — and packages as operators on it. One PEP 420 microcosm namespace, shipped as shard distributions; a microcosm metapackage will pin the constellation.

package	import	role	succeeds
microcosm-frame	microcosm.frame	the kernel: Frame, typed weights, strata, links, weighted accounting, unit structure, rules-engine protocol	microdf, microunit
microcosm-fit	microcosm.fit	conditional models (weight-aware by construction)	ad hoc imputation scripts
microcosm-calibrate	microcosm.calibrate	representation: targets → calibrated weights (APG / L0)	microcalibrate
microcosm-build	microcosm.build	population build plans, donor graphs, release gates, and country build stages	one-off build drivers
microcosm-data	microcosm.data	published population registry and lazy engine loaders	country-specific data packages

Firm support is experimental. The frame kernel can declare firm entity tables and validate person-firm jobs link tables, but link-aware operators, firm calibration targets, and firm release pipelines are not production surfaces yet.

See DESIGN.md for the charter: why the rebuild, the kernel semantics, the RulesEngine protocol (policyengine-us today, Axiom rulespec-us next), longitudinal design (one weight per trajectory), and the process rules (behavioral contract tests, constellation versioning, environment-carrying artifacts).

Incumbent comparisons and historical replacement benchmarks live outside this repo. The live Microcosm repo owns the library, build contracts, published population registry, and acceptance gates.

Development

uv sync --all-packages   # workspace install (all members + dev groups)
uv run pytest            # all packages, incl. behavioral contract tests
uv run ruff check .

Packaged country specs load once per process: load_country_spec("uk") returns the same object on every call. In a notebook or other long-lived session, edits to a country package under packages/microcosm-build/src/microcosm/build/<country>/ are not picked up until you restart the kernel, load it by path with load_country_spec(Path(...)) (always re-read), or call microcosm.build.country_spec._load_packaged_country_spec.cache_clear().

Staging build telemetry

US fiscal refresh builds emit pre-release staging telemetry by default: progress JSON is uploaded to policyengine/populace-us-staging while the build runs (best-effort — a missing token or failed upload never fails the build), so every candidate shows up on the staging dashboard before it is published. Disable with --no-staging, or point elsewhere with --staging-repo-id / POPULACE_STAGING_REPO_ID. An empty POPULACE_STAGING_REPO_ID is ignored rather than read as off, and staging with no destination at all is an argparse error — --no-staging is the only way a build produces no telemetry.

The build manifest records what staging did: the run id, the destination, and how many files actually reached it, or an explicit enabled: false for a declared opt-out. microcosm-publish-release refuses a release whose build meant to stage and delivered nothing (--allow-missing-staging overrides); a declared --no-staging build publishes without the flag:

python tools/build_us_fiscal_refresh_release.py \
  --ledger-facts consumer_facts.jsonl \
  --out /tmp/microcosm-build

This writes progress.json, events.ndjson, calibration_progress.json, and final candidate diagnostics under runs/<run_id>/ without updating production latest.json.

The UK commands (tools/build_uk_frs_spine.py, a shim over the package's uk_runtime.spine_build, and microcosm-build-uk / tools/build_uk_full.py, whose --release-role builds either the national or the dense line; tools/build_uk_rowwise_candidate.py is a stub over the same driver) stage version 2 telemetry to policyengine/populace-uk-staging under the same switch. The build command also stages the finished dataset bundle it built, national, dense or exact-count, under staged/<run_id>/ in the private policyengine/populace-uk-private repository so the team can inspect it without publishing it: releases/ and latest.json are untouched, the release contract is not consulted, and a releasable: false size run stages like any other. Fetch a bundle with tools/fetch_uk_staged_dataset.py; re-stage a finished run directory with tools/stage_uk_rowwise_candidate.py. See docs/uk-staging-operations.md.

See SYSTEM_REQUIREMENTS.md for the measured memory, disk, and CPU footprint of developing and building locally (and what to budget on a build machine — RAM is the binding constraint).

Release-gate preflight

Several release gates fail on facts that are already determined by the base pool, the frozen selection, and the target/coverage registry — no calibration solve needed to see them. tools/preflight_us_release_gates.py recovers those signals in minutes so a two-hour release launch is not the first place a knowable defect surfaces:

uv run python tools/preflight_us_release_gates.py \
  --base-h5 out/base-m/base_populace_us_2024_puf_support.h5 \
  --selection-source-manifest inputs/buildm_keogh_swap_selection_source.json \
  --export-input-mass-reference-h5 forensics/populace_us_2024.h5

It is read-only against the H5 artifacts and reports, per check, PASS / FAIL / AT-RISK with the measured numbers (exit 1 on any FAIL, 2 on AT-RISK only, 0 clean):

1. Selection carryover — the frozen selection-source manifest maps cleanly onto the base pool (the frozen-support recovery contract, run pre-solve).
2. Zero-support preview — compiled positive fiscal targets whose materialized support is ~0 under the selection at base weights stay a structural zero after the solve. Direct-column targets are checked; engine-derived measures are marked not statically checkable (pass --ledger-facts to compile the target surface).
3. Export-mass parity risk — each export-mass column's pool mass at base weights against its reference band, honoring the release tool's US_EXPORT_INPUT_MASS_REVIEWED_EXCLUSIONS register (reused, never re-declared). A column out of band pre-solve is flagged for review.
4. Smoke-probe support audit — every reform-coverage probe leaf's pool vs selected nonzero support and pool sign-leg decomposition. A leaf with pool support but zero selected support fails (the input the frozen selection cannot express); a thin selection or a signed leaf whose net sign contradicts the probe's expected_sign is AT-RISK.

A new lineage — a release built on a fresh base with no selection source (docs/us-release-build-rule.md §3) — has no frozen selection to carry over. Say so explicitly with --new-lineage in place of --selection-source-manifest (the two are refused together; with neither, the manifest is required as before):

uv run python tools/preflight_us_release_gates.py \
  --base-h5 out/base/base_populace_us_2024_puf_support.h5 \
  --new-lineage \
  --ledger-facts inputs/consumer_facts.jsonl

The report records selection_carryover as SKIPPED with reason new_lineage. The one refusal inside that check that belongs to the base rather than to a selection — the base must carry the materialized PUF capital-gains own-tail, which the release tool also requires on every arm — still runs, as capital_gains_tail_presence. Every other check runs unchanged on the whole base, which is what a release without a selection calibrates. Given --release-manifest, --new-lineage also requires that release to record no selection source.

Run it at base-build exit, before any release launch, and after any change to the selection-source manifest or the target/coverage registry. The synthetic-fixture unit tests (packages/microcosm-build/tests/engine_free/us/test_us_release_gate_preflight.py) run in the normal uv run pytest suite; the real-H5 mode above is a local/runbook step.

Releasing & alerts

The native SPM role source-enrichment lane creates a new US H5 from the exact reviewed BuildP parent, preserves its original variables and schema-5 calibration evidence, and requires fresh country/wrapper compatibility checks. It has a local candidate builder and uses the regular publisher's contract with --parent-h5 and --preflight-only. The same release type publishes the reported-receipt child of the national default as a tag-only donor for the ACS local chain. It is never the latest.json default.

The non-default ACS local-area chain (tools/build_us_acs_local_release.py) calibrates to the SOI state surface by default, the 4,459-target contract of Build O and Build P; --soi-mode totals and --soi-mode full are explicit opt-ins. See the ACS local-area SOI target surface for what each mode contains and where the build records it.

National and ACS local-area builds now use the same typed schema-8 calibration diagnostics writer. The local builder adds its Census population marginals to a versioned TargetRegistry, including provider, category, geography, and target hierarchy, before calibration. Both builders always attempt diagnostics after the calibrated dataset exists. If construction, validation, serialization, or writing fails, the release manifest records the failure and publication emits a warning without discarding the dataset release.

Current UK national and rowwise builders use that same schema and writer. The UK extension is fully typed: weight summaries, zero-weight strata, geography-level pass rates, local fit summaries, and rotated holdout evidence are validated at construction, including their cross-field reconciliations. UK release workflows stop when diagnostics are unavailable because their later release checks require that evidence; historical schema-6 and schema-7 UK artifacts remain readable through isolated compatibility validation.

Standard publication uploads the locally built releases/<id>/ artifacts to the Hugging Face dataset, tags the release, and updates latest.json. It runs on the build machine (it needs the freshly built H5), so it isn't a CI step:

tools/publish_release.sh releases/<id> --repo-id policyengine/populace-us

tools/publish_release.sh is a thin wrapper around microcosm-publish-release (all arguments pass straight through). The moment latest.json goes live, the publish CLI posts a release alert to Slack — #populace-us or #populace-uk, chosen from the repo id.

Promotable UK release lines use pointers named latest-<line>.json. Publish a cut for inspection with --no-latest --tag-name <cut-tag>, then promote the reviewed cut with --promote-line <line> --tag-name <cut-tag>. Promotion moves only that line pointer; the UK repository-global latest.json remains frozen on the June 2023 release. Promotion reuses the immutable cut tag the inspect publication created (it checks the tagged manifest is byte-identical) and writes only the pointer commit, so the two-step sequence and a retry after a failed pointer commit both work. A line's registry entry is registered off the default variant until its first promotion; the default flips in a follow-up after the pointer exists.

The publisher uploads only the contract files, the release manifest's artifacts and any --extra-file. The US fiscal-refresh tool therefore binds its terminal gate verdicts as manifest artifacts: input_coverage.json, input_mass_parity.json, qrf_tail_concentration.json and reform_coverage_smoke.json. Both manifests also record the per-run QRF tail register (qrf_tail_register) and the export-mass reference, so a waiver ships with the release it waives, and a gate_evidence block that says of each verdict whether it is bound, skipped by flag or never evaluated.

A US release may not store a column that looks like a policyengine-us variable (lowercase snake_case) unless the engine it is certified against defines that variable or microcosm.data.stored_inputs.US_STORED_NON_VARIABLE_COLUMNS registers the column with a reviewed reason (microcosm#1026: the engine ignores such a column, which is how a renamed WIC take-up input shipped unread). Three release seams refuse one, each against the engine the release records as built-with:

• the fiscal-refresh tool, in its batched pre-export gates, and it grades the written H5, which must earn the same verdict (the exact-k ladder lane runs this tool);
• the source-enrichment probe, at certification, validation and publication;
• the ACS local-area chain's package stage, before it assembles the release directory.

A refusal names each column: rename it to its live input, or add a reviewed register entry. The published default, its reported-receipt child and the 2026-09-23 ACS local-area release built on that child all store two retired engine inputs, would_claim_wic and medicare_part_b_premiums (replaced by takes_up_wic_if_eligible and medicare_part_b_premiums_reported), so each would now be refused. Check local files from HDF metadata alone with uv run python -m microcosm.data.stored_inputs path/to/populace_us_2024.h5.

US exact-k ladder candidates use a tag-only lane. Run tools/build_us_exact_k_ladder_release.py, then execute the publish_command recorded in package_result.json. That command includes --create-tag, --no-latest, and --tag-only: it uploads the immutable release and creates its tag without committing candidate artifacts or release copies to the production main branch. The launcher also forces --no-staging, so the build writes neither a production nor a staging pointer. The candidate is therefore available only by its explicit release id or tag until a separate promotion updates latest.json. Because Slack alerts are coupled to that production pointer update, tag-only publication sends no release alert. The promotion is the standard publish of the same release directory, without --no-latest and --tag-only: it reuses the existing release-id tag once the tagged release_manifest.json is byte-identical to the local one, and writes only the main commit that carries latest.json (microcosm#450). A tag that describes another cut refuses before any commit.

Evidence-tier releases (microcosm#506) are the third lane: the best available artifact when terminal gates failed, built with tools/build_us_fiscal_refresh_release.py --evidence-release (which records every gate failure with an owner issue in the release manifest's known_failures block, or refuses) and published with

tools/publish_release.sh releases/<id> --repo-id policyengine/populace-us --evidence

The --evidence flag validates against the evidence release contract — a certified-shape release is refused under it and vice versa — tags the immutable release as usual, and moves only latest-evidence.json; the certified latest.json pointer and the pe.py certification path never see evidence artifacts. Each evidence publish supersedes the last, so latest-evidence.json always names the best current evidence artifact (consumers: microcosm.data.latest_evidence_release). Its Slack alert is labeled as an evidence-tier publish.

The alert is a no-op unless the channel's incoming-webhook URL is set, so configure it once on the build machine:

cp tools/release.env.example tools/release.env   # then paste the webhook URLs

tools/release.env is gitignored; the wrapper loads it (or you can just export SLACK_WEBHOOK_POPULACE_US / SLACK_WEBHOOK_POPULACE_UK in your shell) and warns if neither is set. After that, every release publishes with an automatic Slack alert.

Canonical UK exact-k builds also require a stable, base64-encoded 32-byte release key. Source tools/release.env before the national build as well as publication. Two variables carry it during the report-format migration — export both from the same key material:

• MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY — what the national build signs with (the gate-battery executor) and what schema-4 report verification reads.
• POPULACE_UK_TERMINAL_GATE_SIGNING_KEY — what schema-3 (legacy-format) report verification reads; retires with the legacy format.

The gate battery authenticates the complete report, canonical release id, and exact calibration-diagnostics digest with HMAC-SHA256; the persistence seam cannot sign caller-composed gate results, and publication independently verifies the report from the same out-of-band key. If the key is missing or malformed, a full-scale build persists the unsigned report and then refuses to stage, and publication rejects unsigned reports.