Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,17 @@ A sealed deny-list in `microcosm.build.us_runtime.h5_io` overrides this opt-in
for known-excluded publications while preserving their scoring-only diagnostic
path.

The independent US annual static-aging candidate builder lives in
`microcosm.build.us_annual_static_aging`; it consumes a pinned published parent
and writes local annual H5 files without running the base graph or publishing.
See [the annual candidate guide](docs/us-annual-static-aging.md). Its completion
manifest is build evidence, not release certification.
Optional annual release metadata invokes additional artifact, identity, and
acceptance checks within the normal release gates. Annual cuts use one pinned
`<base_release>-annual-<YYYYMMDDTHHMMSSZ>-<hex8>` tag and cannot update latest
pointers. Qualify source-enrichment bases before adding annual metadata; use
the candidate guide's qualification order and tag-only publication route.

## Root journals are history, not state

The root `PROGRESS*.md`, `FINAL_REPORT.md`, `*_COVERAGE_PROGRESS.md`, and
Expand Down
21 changes: 21 additions & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,27 @@ Logbook attempt row beside the output before the tool returns. The retiring
two-spine lineage remains available only through its explicit compatibility
flag for byte-reproducible historical builds.

## Annual cross-sectional projections

Static aging supplies independent annual cross-sections for budget-window
estimates. It runs downstream of an accepted base population: demographic
projections change household weights, and monetary factors preserve projected
input aggregates under those weights. It preserves the base records, entity
IDs, and memberships. These repeated IDs identify source records; they do not
describe individual trajectories.

The US release path exports one single-year H5 per supported year with the
base dataset's entity-table layout. Each annual artifact records its source
year, projection year, parent release and dataset hash, model and projection
inputs, and annual acceptance results. Base-year calibration evidence applies
to the base population; each projected year requires its own demographic,
aggregate, and runtime checks. Consumers select a declared annual artifact and
reject requests outside its published coverage.

The base graph and calibration remain the source of the population. Annual
projection artifacts retain that source identity and do not certify a new
base population. See [static aging](docs/static-aging.md) and issue #333.

## Longitudinal (the social-security-model direction)

This section names kernel changes the current `Frame` does NOT yet support;
Expand Down
1 change: 1 addition & 0 deletions changelog.d/333-annual-projection-contract.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Adopt static aging for annual budget-window cross-sections and define the downstream annual-file release contract, separate from base graph certification and individual trajectories.
1 change: 1 addition & 0 deletions changelog.d/annual-static-aging-export.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add a pinned local US annual static-aging candidate builder with preserved base artifacts, native single-year H5 files, source provenance, and checked round trips.
57 changes: 48 additions & 9 deletions docs/static-aging.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Static aging

`microcosm.calibrate.static_aging` projects a base-year frame to later years.
It reweights a fixed cross-section independently for each year. The proposed
operator remains subject to the cross-sectional-versus-longitudinal design
decision in [#333](https://github.com/PolicyEngine/microcosm/issues/333).
It reweights a fixed cross-section independently for each year and supplies
annual budget-window estimates. Dynamics remains the separate path for
individual trajectories. See the design decision in
[#333](https://github.com/PolicyEngine/microcosm/issues/333).

## The split

Expand Down Expand Up @@ -85,12 +86,12 @@ Both `frame_for` and the US dataset exporter apply factors in float64.

## What it is not

The base cross-section is reweighted once per year with no person identity
across years. No transition happens: employment status stays at its
base-year distribution by age, so a projected downturn appears only as slower
per-capita income growth spread over everyone. That is the Dynamics
operator's job (sequencing step 6), and this step is replaced, not extended,
when it lands.
Static aging reweights the base cross-section once per year. Each output keeps
the source record IDs and memberships, but those IDs do not track individual
lives across years. The operator leaves employment status and other
demographic columns unchanged; weights change their representation. Monetary
factors change income amounts without simulating employment transitions.
Dynamics will model those transitions and individual trajectories.

## Country adapters

Expand All @@ -101,3 +102,41 @@ series, including the source totals behind derived `_per_capita` parameters,
come back as totals; other series come back as indices.
`multi_year_dataset` exports a base-year bundle plus its projected years as a
`USMultiYearDataset`, which the engine treats as already extended.

## Annual release integration

The publication layout uses one H5 per year with the existing single-year
entity-table format. The in-memory multi-year container can supply each
year's `USSingleYearDataset`; consumers do not need a combined on-disk file.
Each annual file keeps the base tables, columns, row counts, IDs, and
memberships. The year, household weights, and mapped monetary values change.
Floating-point precision and compression can change the file's byte size.

Annual projections run downstream of an explicitly pinned accepted base.
A new graph build must pass its own release gates before it can supply that
base. Projection evidence records the parent release and H5 hash, source year,
projection year, model and projection-input identities, and annual checks.
The base's calibration receipt does not certify its projected years.

Producer release metadata maps a dataset family to its annual artifact keys:

```json
{
"dataset_years": {
"populace_us_2024": {
"2024": "populace_us_2024",
"2030": "populace_us_2030",
"2035": "populace_us_2035"
}
}
}
```

This example abbreviates the mapping; a release through 2035 lists every
supported year. Each value references a normal revision- and hash-pinned
artifact. The wrapper validates the mapping during certification, checks the
file's stored year, and refuses unavailable years. Cache identities include
the actual artifact and relevant runtime versions.

Local annual candidates are build evidence. Publication and wrapper
certification follow their separate acceptance checks.
106 changes: 106 additions & 0 deletions docs/us-annual-static-aging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# US annual static-aging candidates

`microcosm.build.us_annual_static_aging.build_annual_static_aging` creates a local
candidate containing one native single-year H5 for each year from the source year
through the requested end year (2035 by default). It reuses `static_aging` and
`multi_year_dataset`; it does not run the base calibration graph or publish data.

The inputs are an exact base H5 SHA256 and parent release identifier, an exact SSA
population CSV SHA256, and the version, full commit, and clean source checkout of
the imported PolicyEngine-US model. The caller supplies the parent release claim;
release certification must authenticate that release and its base SHA separately.
The numerical environment must contain the reviewed model and Microcosm packages.

After setting the input paths and reviewed pins, use the CLI in that environment:

```sh
uv run --no-sync python -m microcosm.build.us_annual_static_aging \
--base-h5 "$BASE_H5" --base-sha256 "$BASE_SHA256" \
--parent-release "$PARENT_RELEASE" \
--ssa-csv "$SSA_CSV" --ssa-sha256 "$SSA_SHA256" \
--model-source "$MODEL_SOURCE" --model-commit "$MODEL_COMMIT" \
--model-version "$MODEL_VERSION" --output-dir "$NEW_OUTPUT_DIR" \
--base-year 2024 --end-year 2035
```

The builder uses the frame-anchored demographic targets, SSA ages 80–84 pooled at
80 and 85+ pooled at 85, 300 calibration epochs, seed 0, and a maximum weight ratio
of 5. It records all settings in the manifest. Every projected year starts from
the original frame; years are calculated and serialized individually to avoid
retaining the entire budget window in memory.

The source-year H5 is copied byte for byte. Projected files preserve all six
entities, columns, rows, IDs, and unscaled inputs. Household weights and mapped
monetary inputs change according to the existing static-aging operator. Signed
income inputs retain their separate positive and negative scale factors. Each
annual H5 uses root entity tables and an explicit `_time_period` equal to its
year. Logical and native-loader round trips compare all table values. Consumers
must select the exact annual file and year; its inputs are already projected.
The current publication contract requires table-format storage with direct HDF
fields. Fixed-format inputs, including missing nullable booleans that require
that storage format, are rejected before building and need a separate certified
layout contract.

The output directory must be new. A failed attempt retains its partial files and
`build_status.json`; rerunning requires another directory. `annual_manifest.json`
is written last, after all years finish and source/input hashes are rechecked.
It contains:

- `base`: source dataset, year, parent release claim, path, and SHA256.
- `inputs`, `model`, and `runtime`: SSA pin, model commit and source hashes, actual
implementation hashes, and dependency versions.
- `metadata.dataset_years`: a map from the source dataset to year-keyed annual
dataset names, including the preserved source year.
- `artifacts`: annual names mapped to relative H5 paths, hashes, years, entity row
counts, ordered columns, and round-trip receipts.
- Per-year projection receipt paths and hashes, containing the actual factors,
parameter series, demographic targets and achievements, and calibration fit.

A complete candidate is not a certified release. Release preparation must add
normal immutable repository/revision pins, authenticate the parent, and run
independent demographic, monetary, identity, and runtime acceptance checks.
That process supplies the separate annual projection acceptance report and
publication decision; this module supplies no certification override.

## Qualification and publication order

1. Qualify the base release under its existing contract with the intended
country, Core, wrapper, and calculator wheels. Source-enrichment bases also
require the exact original parent H5 and their producer-source evidence.
Merely downloading an older bundle and rerunning compatibility does not
update its producer-source pins. If those pins no longer qualify, reproduce
the base bundle with reviewed producer code and prove preservation before
adding annual artifacts.
2. Run independent checks on the saved annual files. Record schema, source
identity, year, demographics, input aggregates, and runtime acceptance for
every declared year, including the base year. Runtime checks must exercise
the intended wrapper and model with the earlier annual inputs that policy
formulas need. Bind the acceptance report to the candidate manifest hash,
model commit and source hash, and country/Core versions.
3. Add `metadata.dataset_years`, `metadata.annual_projection_manifest`, and
`metadata.annual_projection_acceptance` to the qualified release manifest.
The latter two values name artifact keys for the candidate manifest and
the separate schema-1 `us_annual_projection_acceptance` report. Keep H5
paths at the repository root; give both reports and every per-year
projection receipt their exact `releases/<base_release>/filename.json`
paths. The gate verifies each per-year receipt's hash and years against
the candidate manifest and requires every year from the base through the
last declared year, so policy lookbacks retain their annual inputs.
Pin all artifacts to one
`<base_release>-annual-<YYYYMMDDTHHMMSSZ>-<hex8>` tag and their exact hashes.
Complete source-enrichment certification before this step: that operation
writes base-tag compatibility metadata.
4. Run normal publisher preparation with the artifact root, annual tag, and
`update_latest=False`. For source enrichment, also supply the original
parent H5 and compatibility wheels. The annual extension adds checks; the
publisher still enforces every original base-release gate. Publish with
`tag_only=True` to preserve the existing main-branch files as well as both
latest pointers.
5. Certify the wrapper against the explicit annual revision and release
manifest, then release the wrapper and update consumers. Certification
must retain the complete year map and exact annual artifact pins.

The annual tag cannot become `latest.json` through this route. It augments the
same accepted base population; it does not promote another agent's new graph
candidate. A later graph release can supply a new base only after passing its
own qualification, followed by a fresh annual build and acceptance.
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,17 @@ class HdfWriteExclusion:
version_owner="PolicyEngineUSAdapter payload contract",
nullable_boolean_storage="numpy_bool_or_object_pd_na_v1",
),
FrameSerializerSpec(
serializer_id="us_annual_static_aging",
writer=HdfWriteSite(
"packages/microcosm-build/src/microcosm/build/us_annual_static_aging.py",
"_write_year",
),
backend="pandas.HDFStore table with direct fields",
routes=("US annual static-aging candidate",),
version_owner="schema-1 annual static-aging candidate native layout",
nullable_boolean_storage="numpy_bool_missing_rejected_v1",
),
FrameSerializerSpec(
serializer_id="legacy_us_two_spine",
writer=HdfWriteSite(
Expand Down
Loading
Loading