Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/scripts/stage12-validate-infrastructure.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ for account in \
done
uk_geography_lookup_objects=(
"gs://policyengine-uk-data-private/constituencies_2024.csv"
"gs://policyengine-uk-data-private/local_authorities_2021.csv"
)
for secret_name in \
"${STAGE12_MODAL_TOKEN_ID_SECRET_NAME}" \
Expand Down
44 changes: 40 additions & 4 deletions docs/migration/stage-12-modal-worker-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,11 @@ with the live API-owned schema.
`stage12-evaluation-gcp-credentials` Modal secret. The worker application
does not receive the existing general GCP credential secret. The credential's
service account can read and write its environment-specific Stage 12 artifact
bucket. A conditional IAM binding separately permits it to read only
`constituencies_2024.csv` and `local_authorities_2021.csv` from
`policyengine-uk-data-private`; deployment validation downloads both lookup
files using the worker credential.
bucket. A conditional IAM binding separately permits it to read
`constituencies_2024.csv` from `policyengine-uk-data-private`; deployment
validation downloads that lookup file using the worker credential. Stage 12
local-authority output uses packaged metadata and does not require access to
`local_authorities_2021.csv`.
- `STAGE12_DATABASE_URL` is delivered from the environment-specific Secret
Manager resource named by `STAGE12_DATABASE_URL_SECRET_NAME`. It authenticates
as the existing shared `policyengine_v2_runtime` account; Stage 12 does not
Expand Down Expand Up @@ -207,6 +208,41 @@ metadata so PolicyEngine can correctly recognize conditional labor-supply
analysis. `include_cliffs=true` is supported by this path and is carried through
both output planning and aggregation.

### Temporary UK local-authority display metadata

Stage 12 detects the local-authority boundary configuration from the complete
source dataset before applying any requested regional scope. It examines the
dataset's `la_code_oa` values: the 17 predecessor English authorities identify
LAD22, while `E06000063` through `E06000066` identify LAD23. A mixed boundary
configuration, an unknown code, or a dataset without a distinguishing code
fails the Stage 12 run. Dataset names and release labels do not select a
boundary version, and there is no fallback to the newest known configuration.

The executor temporarily packages three display-only resources:

- one common `code,name` file containing the union of supported authority
codes;
- one `code,x,y` file for the 374-authority LAD22 configuration; and
- one `code,x,y` file for the 361-authority LAD23 configuration.

The Enhanced FRS geography is LAD22: its 2021 output areas are crosswalked to
2022 local-authority boundaries. The output-area vintage does not make the
local-authority boundary version LAD21. Names and coordinates are attached only
after numeric local-authority impacts have been calculated; these resources
never assign a household to an authority.

The detected boundary version is stored as typed metadata in both child Parquet
artifacts. The coordinator requires baseline and reform to agree before it
enriches the aggregate output. Existing non-Stage-12 calculations continue to
use the existing GCS lookup and are not changed by this path.

This resource ownership is temporary. A future boundary version must add its
codes to the common names file, add a distinct coordinate file, and register a
typed detector definition. Longer term, display-coordinate rendering should
move to the front end, and Stage 14 or later work should remove these temporary
executor resources when the authoritative v2 report architecture supersedes
Stage 12.

## Temporary direct runner endpoint

> **Temporary Stage 12 interface:** The authenticated routes in this section
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,20 @@ class SimulationRole(StrEnum):
STANDALONE = "standalone"


class UKLocalAuthorityBoundaryVersion(StrEnum):
"""Supported UK local-authority boundary versions."""

LAD22 = "lad22"
LAD23 = "lad23"


class UKLocalAuthorityMetadata(StrictContractModel):
"""Local-authority boundary version detected from a UK simulation dataset."""

country: Literal["uk"] = "uk"
boundary_version: UKLocalAuthorityBoundaryVersion


class ArtifactMediaType(StrEnum):
JSON = "application/json"
PARQUET = "application/vnd.apache.parquet"
Expand Down Expand Up @@ -85,6 +99,9 @@ class SimulationParquetPayloadContract(StrictContractModel):
calculation_provenance_metadata_key: Literal[
"policyengine.stage12.calculation_provenance"
] = "policyengine.stage12.calculation_provenance"
uk_local_authority_metadata_key: Literal[
"policyengine.stage12.uk_local_authority_metadata"
] = "policyengine.stage12.uk_local_authority_metadata"


SIMULATION_PARQUET_PAYLOAD_CONTRACT = SimulationParquetPayloadContract()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@
import pytest

from policyengine_simulation_contract.stage12_execution import (
SIMULATION_PARQUET_PAYLOAD_CONTRACT,
EntityOutputPlan,
ReportAggregate,
ReportOutputRequirements,
Stage12OutputPlan,
UKLocalAuthorityMetadata,
UKLocalAuthorityBoundaryVersion,
stage12_output_plan_sha256,
)

Expand Down Expand Up @@ -115,3 +118,24 @@ def test_entity_output_plan_separates_calculated_and_dataset_variables() -> None
additional_variables=("constituency_code_oa",),
dataset_variables=("constituency_code_oa",),
)


def test_uk_local_authority_metadata_is_strict_and_typed() -> None:
metadata = UKLocalAuthorityMetadata(
boundary_version=UKLocalAuthorityBoundaryVersion.LAD22
)

assert metadata.country == "uk"
assert metadata.boundary_version is UKLocalAuthorityBoundaryVersion.LAD22

with pytest.raises(ValueError, match="extra_forbidden"):
UKLocalAuthorityMetadata.model_validate(
{"country": "uk", "boundary_version": "lad23", "dataset": "microcosm"}
)


def test_parquet_contract_names_uk_local_authority_metadata() -> None:
assert (
SIMULATION_PARQUET_PAYLOAD_CONTRACT.uk_local_authority_metadata_key
== "policyengine.stage12.uk_local_authority_metadata"
)
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ def test_storage_validation_uses_only_the_modal_worker_credentials() -> None:
assert source.count(f"runtime_gcloud {modal_account} storage cp") == 3
assert source.count(f"runtime_gcloud {modal_account} storage rm") == 2
assert "gs://policyengine-uk-data-private/constituencies_2024.csv" in source
assert "gs://policyengine-uk-data-private/local_authorities_2021.csv" in source
assert "gs://policyengine-uk-data-private/local_authorities_2021.csv" not in source
assert "STAGE12_UK_GEOGRAPHY_BUCKET" not in source


Expand Down
Loading
Loading