Skip to content

_user_input_keys drifts from stored values after delete_arrays and clone #559

Description

@MaxGhenis

Simulation._user_input_keys, the (variable, branch, period) record of values set with set_input, drifts from the stored arrays in two ways (3.32.9):

  1. delete_arrays keeps the entries for the arrays it deletes. A formula result calculated later for the same period is then treated as a user input. _invalidate_all_caches (run by apply_reform) replays it, so the stale formula output survives a reform, and to_input_dataframe would export it.
  2. clone() shares the set with the original (the shallow __dict__ copy) while the holders' stored arrays are copied. An input set later on the clone is then recorded for the original as well, and on a parent after get_branch it is recorded for branches created earlier, whose storage never received it.

Reproduction with the country template:

from policyengine_core.country_template import CountryTaxBenefitSystem
from policyengine_core.simulations import SimulationBuilder

tbs = CountryTaxBenefitSystem()
def sim():
    return SimulationBuilder().build_from_entities(
        tbs, {"persons": {"a": {"salary": {"2025-01": 1000}}}, "households": {"h": {"parents": ["a"]}}})

s = sim()
s.set_input("income_tax", "2025-01", [5.0])
s.delete_arrays("income_tax")
s.calculate("income_tax", "2025-01")          # formula output, 150
s._invalidate_all_caches()
print(s.get_holder("income_tax").get_array("2025-01"))   # [150.] replayed as if it were input

s = sim()
s.calculate("income_tax", "2025-01")
c = s.clone()
c.set_input("income_tax", "2025-01", [7.0])
print(s._user_input_keys is c._user_input_keys)          # True

Suggested fix: in delete_arrays, drop the entries for the deleted variable, branches and periods; in clone, copy the set (set(self._user_input_keys)). A per-stored-array input flag copied with storage would also fix both.

PolicyEngine/policyengine-uk#2025 works around this in the UK Simulation subclass (clone and delete_arrays overrides), because income_support_eligible uses the record to decide whether an award was entered directly. Those overrides can go once core keeps the record in step.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions