Follow-up to #530 / #531.
Problem
ensure_datasets() reuses every US year file ({data_folder}/{stem}_year_{year}.h5) that already exists and passes its staleness check. load_datasets() loads them. A year file that lost a renamed stored input gives wrong results without any error:
A subset check ("the file's register covers today's") would not close this either. A file cut by a newer release under a larger register stores the new live name. An older release whose engine does not define that name would skip it (_build_simulation_from_dataset sets only columns in system.variables). A removed or retargeted entry fails the same way.
Proposal
create_datasets() stamps each year file last with policyengine_year_file_stamp, stored as UTF-8 JSON {"register": <the whole LEGACY_INPUT_RENAMES>, "revision": <year-file format revision>}.
ensure_datasets() regenerates, and load_datasets() refuses, any year file whose stamp does not equal this release's stamp exactly, or whose renames record is missing, malformed or not covered by its stamped register. The error names the renames the file may lack, generated from the register with no WIC special case.
- The check reads only the stamp and the record, never the tables or the source dataset.
- Permission, directory and file-lock errors propagate instead of starting a full regeneration.
Saved simulation outputs (Simulation.load()) still gate only on the record's presence. That is left to a separate issue.
Follow-up to #530 / #531.
Problem
ensure_datasets()reuses every US year file ({data_folder}/{stem}_year_{year}.h5) that already exists and passes its staleness check.load_datasets()loads them. A year file that lost a renamed stored input gives wrong results without any error:create_datasets()exported onlysim.input_variables. policyengine-us 2.x renamedwould_claim_wictotakes_up_wic_if_eligible, so these files store neither name. Every WIC-eligible person then takes WIC up: about $11.5B instead of $6.6B in 2024 (Published US releases store would_claim_wic, which policyengine-us 2.x ignores, so WIC take-up is 100% under policyengine.py 6.1.1 microcosm#1026).policyengine_legacy_input_renames).ensure_datasets()regenerates, andload_datasets()refuses, a file with no record. The record shows that a file was cut with the mapping, not under which register. Map the stored WIC take-up draw onto takes_up_wic_if_eligible when loading US data #531's own body and the comment onRENAMES_H5_DATASETsay so. When policyengine-us next renames an input andLEGACY_INPUT_RENAMESgains an entry, year files cut before that entry will still carry a record and be reused, and they can lose the newly renamed input the same way.A subset check ("the file's register covers today's") would not close this either. A file cut by a newer release under a larger register stores the new live name. An older release whose engine does not define that name would skip it (
_build_simulation_from_datasetsets only columns insystem.variables). A removed or retargeted entry fails the same way.Proposal
create_datasets()stamps each year file last withpolicyengine_year_file_stamp, stored as UTF-8 JSON{"register": <the whole LEGACY_INPUT_RENAMES>, "revision": <year-file format revision>}.ensure_datasets()regenerates, andload_datasets()refuses, any year file whose stamp does not equal this release's stamp exactly, or whose renames record is missing, malformed or not covered by its stamped register. The error names the renames the file may lack, generated from the register with no WIC special case.Saved simulation outputs (
Simulation.load()) still gate only on the record's presence. That is left to a separate issue.