minimal data requirements and small pixi fix - #54
Merged
Conversation
Snakemake's notebook: directive shells out to papermill to execute prepare_wacc, prepare_political_stability, prepare_chokepoints, prepare_steel_demand, and the plotting notebooks. It was missing from every pixi environment, so none of those rules could run.
rules/trade_model.smk hardcoded cost_year=[2050] when pulling supply curves into model_trade, ignoring the actual requested wildcard, so trade runs at any other cost_year silently used stale 2050 supply curves. Dropping the hardcode lets it flow through allow_missing like region/interone/intertwo already do. prepare-steel-demand.ipynb called mock_snakemake(cost_year=2050) unconditionally (missing the `if "snakemake" not in globals():` guard used by the other prep notebooks), so it overwrote Snakemake's real injected object on every run and always wrote steel_demand_clustered_2050.csv regardless of the wildcard requested. Verified: dry-running a cost_year~2030 trade target now resolves supply-curve dependencies at cost_year~2030 (previously 2050), and a real run of prepare_steel_demand for cost_year=2030 now produces resources/steel_demand_clustered_2030.csv with genuinely different data from the 2050 file.
download_labour_data had no input: block, so the script's read of data/labour/unido-raw/data.csv was invisible to the DAG. If that file was missing, the rule ran "successfully" but silently skipped the UNIDO merge and never wrote resources/merged_labour_inputs.csv, surfacing only as a confusing MissingOutputException instead of a clear MissingInputException naming the real cause. Declares the file as input.unido_raw and has the script read the Snakemake-provided path instead of its own hardcoded module constant, so a missing file is now caught at DAG-build time. Verified: dry-run now lists data/labour/unido-raw/data.csv as an input; temporarily removing it correctly raises MissingInputException; restoring it and running for real still produces resources/merged_labour_inputs.csv as before.
JanTautorus
changed the title
Owner
|
Thank you @JanTautorus. I have reviewed your PR and accept it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
necessary external input data:
there was a silent dependeny on unido-raw data without an explicity mention in the snakemake rule that it requires this input. Also added papermill to pixi to run preperation notebooks properly via snakemake.