Skip to content

Stamp US year files with the register of renamed stored inputs they were cut under #532

Description

@MaxGhenis

Follow-up to #530 / #531.

Problem

ensure_datasets() reuses every US year file ({data_folder}/{stem}_year_{year}.h5) that already exists and passes its staleness check. load_datasets() loads them. A year file that lost a renamed stored input gives wrong results without any error:

A subset check ("the file's register covers today's") would not close this either. A file cut by a newer release under a larger register stores the new live name. An older release whose engine does not define that name would skip it (_build_simulation_from_dataset sets only columns in system.variables). A removed or retargeted entry fails the same way.

Proposal

  • create_datasets() stamps each year file last with policyengine_year_file_stamp, stored as UTF-8 JSON {"register": <the whole LEGACY_INPUT_RENAMES>, "revision": <year-file format revision>}.
  • ensure_datasets() regenerates, and load_datasets() refuses, any year file whose stamp does not equal this release's stamp exactly, or whose renames record is missing, malformed or not covered by its stamped register. The error names the renames the file may lack, generated from the register with no WIC special case.
  • The check reads only the stamp and the record, never the tables or the source dataset.
  • Permission, directory and file-lock errors propagate instead of starting a full regeneration.

Saved simulation outputs (Simulation.load()) still gate only on the record's presence. That is left to a separate issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions