Check the simulation tables in CI

.github/workflows/simulate.yml

@@ -3,6 +3,13 @@ on:
   push:
     branches:
       - master
+  pull_request:
+    paths:
+      - 'problems/**'
+      - 'src/python/simulate.py'
+      - 'src/python/solver_settings.yaml'
+      - 'src/python/benchmark_models_petab/**'
+      - '.github/workflows/simulate.yml'
   workflow_dispatch:
   merge_group:
   schedule:
@@ -25,17 +32,26 @@ jobs:
         AMICI_PARALLEL_COMPILE: ""
       run: |
         pip install --upgrade pip
-        # TODO: once amici 1.0 is released, replace with:
-        # pip install amici[petab]
-        pip install petab
-        pip install "git+https://github.com/dweindl/amici.git@6a85422ad233fc268fb71272bba39f5b82db387a#egg=amici&subdirectory=python/sdist"
+        pip install "amici[petab]>=1.2.0"
 
-    - name: Simulate with nominal parameters
+    - name: Simulate and compare to the simulation tables
       env:
         AMICI_PARALLEL_COMPILE: ""
+      # The problems are simulated with the solver settings of
+      # src/python/solver_settings.yaml.
       run: |
+        failed=()
         for problem in `ls -1 problems/`; do
-            python src/python/simulate.py "$problem" -j
+            python src/python/simulate.py "$problem" -j --check-simulations \
+                || failed+=("$problem")
             echo
             echo
         done
+
+        {
+          echo "### Comparison to the simulation tables"
+          echo
+          echo "Failed: ${failed[*]:-none}"
+        } | tee -a "$GITHUB_STEP_SUMMARY"
+
+        [ ${#failed[@]} -eq 0 ]

CONTRIBUTING.md

@@ -4,8 +4,9 @@ Simply create a new branch and open a pull request with your files. We will then
 
 Each problem lives under `problems/<ProblemID>/`, with at least one of `v1/`/`v2/` present:
 - `v1/` holds the problem in PEtab v1 format: `problem.yaml`, `model.xml`,
-  `conditions.tsv`, `measurements.tsv`, `observables.tsv`, `parameters.tsv`, and optionally
-  `simulations.tsv` and `visualizations.tsv`.
+  `conditions.tsv`, `measurements.tsv`, `observables.tsv`, `parameters.tsv`,
+  `simulations.tsv` (see [Simulation table](#simulation-table)), and optionally
+  `visualizations.tsv`.
 - `v2/` holds the same problem in PEtab v2 format.
 - `metadata.yaml` contains version-independent metadata. Currently, this is a single-sentence
   `short_description` of the problem, stating the type of model (e.g., biochemical reaction network,
@@ -22,6 +23,20 @@ Each problem lives under `problems/<ProblemID>/`, with at least one of `v1/`/`v2
   PEtab files themselves (e.g. raw data, scripts, notebooks, figures). There's no prescribed
   structure — organize it however fits your problem.
 
+## Simulation table
+
+`v1/simulations.tsv` holds the simulations of the model at the nominal parameters of the
+parameter table, one row per row of the measurement table: the measurement table with the
+column `measurement` replaced by `simulation`. It can be generated with AMICI:
+```bash
+python src/python/simulate.py <ProblemID> --output problems/<ProblemID>/v1/simulations.tsv
+```
+The [Simulate](.github/workflows/simulate.yml) workflow checks that every problem reproduces
+its simulation table (`simulate.py <ProblemID> --check-simulations`), with the solver settings
+of [`solver_settings.yaml`](src/python/solver_settings.yaml). Check that these settings are
+accurate enough for a new problem with `simulate.py <ProblemID> --check-simulation-tolerances`,
+and add problem-specific settings to `solver_settings.yaml` if they are not.
+
 ## v1/v2 equivalence
 
 A problem's `v1/` and `v2/` encodings must represent *exactly* the same mathematical problem:

problems/Blasi_CellSystems2016/v1/simulations.tsv

@@ -119,33 +119,6 @@ observable_k5k12		control	0.047962399802456	inf		sigma
 observable_k5k12		control	0.047962399802456	inf		sigma
 observable_k5k12		control	0.047962399802456	inf		sigma
 observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k12		control	0.047962399802456	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k5k16		control	0.003900604877395	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
@@ -158,15 +131,6 @@ observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
 observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k12		control	0.007670374014446	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
-observable_k8k16		control	0.004314605919164	inf		sigma
 observable_k8k16		control	0.004314605919164	inf		sigma
 observable_k8k16		control	0.004314605919164	inf		sigma
 observable_k8k16		control	0.004314605919164	inf		sigma