Add per-problem metadata files with descriptions

.github/pull_request_template.md

@@ -21,6 +21,7 @@
   - [ ] A visualization table is included, that can be used with the simulated data to reproduce figures from the original publication
   - [ ] The PEtab problem is valid (check with e.g. `petablint -vy problem.yaml`)
   - [ ] Labels (e.g. `parameterName` in the parameter table or `xLabel` in the visualization table) encode special characters with tex-style strings instead of unicode, e.g. use `H$_2$O` for water.
+- [ ] `problems/<ProblemID>/metadata.yaml` contains a single-sentence `short_description` of the problem and passes `bmp-check-metadata`
 - [ ] The PEtab problem author(s) are assigned to the GitHub issue
 - [ ] The README has been updated with `bmp-create-overview  --update` (requires `pip install -e src/python` from the repository root)
   - [ ] The new PEtab problem row in the generated table has the correct reference (and other entries)

.github/workflows/tests.yml

@@ -37,6 +37,9 @@ jobs:
     - name: Check SBML metadata
       run: bmp-check-sbml-metadata
 
+    - name: Check problem metadata files
+      run: bmp-check-metadata
+
     - name: Check all PEtab YAML files under problems/
       # informational only for now -- some pre-existing, unrelated problems
       # are known to fail this check

CONTRIBUTING.md

@@ -6,6 +6,17 @@ Each problem lives under `problems/<ProblemID>/`:
 - `v1/` contains the PEtab problem itself: `problem.yaml`, `model.xml`, `conditions.tsv`,
   `measurements.tsv`, `observables.tsv`, `parameters.tsv`, and optionally `simulations.tsv`
   and `visualizations.tsv`.
+- `metadata.yaml` contains version-independent metadata. Currently, this is a single-sentence
+  `short_description` of the problem, stating the type of model (e.g., biochemical reaction network,
+  compartmental epidemiological model), the biological system, and, where informative, the data.
+  For example:
+  ```yaml
+  short_description: Reaction network model of Epo-induced STAT5A/STAT5B phosphorylation and homo- and heterodimerization, fitted to quantitative mass spectrometry data.
+  ```
+  The file is validated against the `ProblemMetadata` model in
+  [`metadata.py`](src/python/benchmark_models_petab/metadata.py)
+  (check with `bmp-check-metadata`); the corresponding JSON schema is
+  [`metadata.schema.json`](src/python/benchmark_models_petab/schemas/metadata.schema.json).
 - `resources/` (optional) holds anything else related to the problem that isn't part of the
   PEtab files themselves (e.g. raw data, scripts, notebooks, figures). There's no prescribed
   structure — organize it however fits your problem.

problems/Alkan_SciSignal2018/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of the DNA damage response and cell fate in cancer cells treated with chemotherapeutics, fitted to signaling and cell count data.

problems/Armistead_CellDeathDis2024/metadata.yaml

@@ -0,0 +1 @@
+short_description: Small reaction network model of sphingolipid metabolism with ceramide feedback, fitted to lipidomics data of wild-type and mutant zebrafish embryos.

problems/Bachmann_MSB2011/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of Epo-induced JAK2/STAT5 signaling with two negative feedback regulators (SOCS3, CIS) in murine erythroid progenitor cells.

problems/Beer_MolBioSystems2014/metadata.yaml

@@ -0,0 +1 @@
+short_description: Small kinetic model of bacterial growth and production of the blue pigment indigoidine by engineered variants of a non-ribosomal peptide synthetase.

problems/Bertozzi_PNAS2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: SIR compartmental epidemiological model of the early COVID-19 outbreak in New York and California.

problems/Blasi_CellSystems2016/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of combinatorial histone H4 acetylation states generated by motif-specific acetylation reactions.

problems/Boehm_JProteomeRes2014/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of Epo-induced STAT5A/STAT5B phosphorylation and homo- and heterodimerization, fitted to quantitative mass spectrometry data.

problems/Borghans_BiophysChem1997/metadata.yaml

@@ -0,0 +1 @@
+short_description: Three-variable ODE model of complex calcium oscillations with Ca2+-activated IP3 degradation, fitted to measured calcium bursting in a single hepatocyte.

problems/Brannmark_JBC2010/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of insulin receptor and IRS1 phosphorylation with receptor endocytosis and feedback in primary human adipocytes.

problems/Bruno_JExpBot2016/metadata.yaml

@@ -0,0 +1 @@
+short_description: Enzyme kinetic model of the in vitro cleavage of carotenoids and apocarotenoids by Arabidopsis carotenoid cleavage dioxygenase 4 (CCD4).

problems/Chen_MSB2009/metadata.yaml

@@ -0,0 +1 @@
+short_description: Large mass-action reaction network model of ErbB receptor signaling to ERK and AKT in response to EGF and heregulin.

problems/Crauste_CellSystems2017/metadata.yaml

@@ -0,0 +1 @@
+short_description: Cell population model of CD8 T cell differentiation (naive, early/late effector, memory), fitted to cell counts in mice after vaccinia virus infection.

problems/Elowitz_Nature2000/metadata.yaml

@@ -0,0 +1 @@
+short_description: Gene regulatory network model of the repressilator, a synthetic three-gene transcriptional oscillator in E. coli, fitted to a fluorescence time course.

problems/Fiedler_BMCSystBiol2016/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of RAF/MEK/ERK signaling with ERK feedback in HeLa cells, fitted to phospho-MEK/ERK data without and with RAF or MEK inhibitors.

problems/Froehlich_CellSystems2018/metadata.yaml

@@ -0,0 +1 @@
+short_description: Large-scale reaction network model of cancer signaling pathways, parameterized by cell-line-specific gene expression and fitted to drug response viability data.

problems/Fujita_SciSignal2010/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of EGF-induced EGFR, Akt and S6 phosphorylation in PC12 cells under different EGF stimulation profiles.

problems/Giordano_Nature2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: Compartmental epidemiological model (SIDARTHE, distinguishing detected/undetected and symptom severity) of the COVID-19 epidemic in Italy.

problems/Isensee_JCB2018/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of cAMP-induced PKA activation and phosphorylation of its regulatory subunit in rat sensory neurons under pharmacological perturbations.

problems/Lang_PLOSComputBiol2024/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of the human cell cycle, generated from a rule-based model and fitted to compartment-resolved single-cell imaging data of RPE-1 cells.

problems/Laske_PLOSComputBiol2019/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of intracellular influenza A virus replication, fitted to viral RNA, genome import and virus release data in human A549 cells.

problems/Liu_IFACPapersOnLine2025/metadata.yaml

@@ -0,0 +1 @@
+short_description: Population model of oncolytic virus therapy with uninfected and infected tumor cells and virus, fitted to tumor size data in zebrafish.

problems/Lucarelli_CellSystems2018/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of TGF-beta-induced SMAD complex formation and target gene expression.

problems/Okuonghae_ChaosSolitonsFractals2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: Compartmental epidemiological model of COVID-19 in Lagos, Nigeria, including asymptomatic and detected infections.

problems/Oliveira_NatCommun2021/metadata.yaml

@@ -0,0 +1 @@
+short_description: Compartmental epidemiological model of COVID-19 in Bahia, Brazil (state excluding the capital Salvador), with non-detected cases, hospitalization and deaths.

problems/Perelson_Science1996/metadata.yaml

@@ -0,0 +1 @@
+short_description: Within-host model of HIV-1 viral dynamics, fitted to plasma HIV-1 RNA levels after treatment with the protease inhibitor ritonavir.

problems/Rahman_MBS2016/metadata.yaml

@@ -0,0 +1 @@
+short_description: Compartmental epidemiological model of HIV transmission with CD4+ T cell count-based infection stages and treatment, fitted to HIV prevalence in South Africa.

problems/Raia_CancerResearch2011/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of IL-13-induced JAK2/STAT5 signaling in the primary mediastinal B-cell lymphoma cell line MedB-1.

problems/Raimundez_PCB2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of EGFR and MET signaling to ERK and AKT in two gastric cancer cell lines in response to EGF and the anti-EGFR antibody cetuximab.

problems/SalazarCavazos_MBoC2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of EGFR dimerization, multisite phosphorylation and binding of the adaptor proteins GRB2 and SHC1, generated from a rule-based model.

problems/Schwen_PONE2014/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of insulin binding, internalization and degradation in two subpopulations of isolated hepatocytes with low and high insulin binding.

problems/Smith_BMCSystBiol2013/metadata.yaml

@@ -0,0 +1 @@
+short_description: Large reaction network model of insulin signaling in rodent adipocytes and its regulation by oxidative stress.

problems/Sneyd_PNAS2002/metadata.yaml

@@ -0,0 +1 @@
+short_description: Kinetic model of the type-2 IP3 receptor, fitted to time courses of calcium release from liver microsomes at varying IP3 and calcium concentrations.

problems/Weber_BMC2015/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of interactions of protein kinase D, a lipid kinase (PI4KIIIbeta) and a ceramide transfer protein (CERT) at the trans-Golgi network.

problems/Zhao_QuantBiol2020/metadata.yaml

@@ -0,0 +1 @@
+short_description: Compartmental epidemiological model of COVID-19 in Wuhan, Hubei and the rest of China, with quarantined and unquarantined infected individuals.

problems/Zheng_PNAS2012/metadata.yaml

@@ -0,0 +1 @@
+short_description: Reaction network model of the combinatorial methylation of lysines 27 and 36 on histone H3.

src/python/MANIFEST.in

@@ -1 +1,2 @@
 recursive-include benchmark_models_petab/problems *
+include benchmark_models_petab/schemas/*.json

src/python/benchmark_models_petab/C.py

@@ -14,6 +14,7 @@
 V1_DIRNAME: str = "v1"
 PROBLEM_FILENAME: str = "problem.yaml"
 SIMULATIONS_FILENAME: str = "simulations.tsv"
+METADATA_FILENAME: str = "metadata.yaml"
 
 GITHUB_REPO: str = "Benchmarking-Initiative/Benchmark-Models-PEtab"
 GITHUB_DEFAULT_BRANCH: str = "master"

src/python/benchmark_models_petab/__init__.py

@@ -6,6 +6,7 @@
 
 from .base import get_problem, get_problem_yaml_path, get_simulation_df
 from .C import MODEL_DIRS, MODELS, MODELS_DIR
+from .metadata import ProblemMetadata, get_metadata
 from .overview import get_overview_df
 from importlib.metadata import PackageNotFoundError, version
 

src/python/benchmark_models_petab/check_metadata.py

@@ -0,0 +1,53 @@
+"""Check that every problem has a metadata file that matches the schema."""
+
+import sys
+from pathlib import Path
+
+import pydantic
+import yaml
+
+from .C import MODELS
+from .metadata import ProblemMetadata, get_metadata_path
+
+
+def check_metadata_file(path: Path) -> list[str]:
+    """Check a single metadata file.
+
+    Returns
+    -------
+    A list of error messages, empty if the file is valid.
+    """
+    if not path.is_file():
+        return [f"Missing metadata file `{path.name}`."]
+
+    try:
+        ProblemMetadata.from_yaml(path)
+    except yaml.YAMLError as e:
+        return [f"Invalid YAML: {e}"]
+    except pydantic.ValidationError as e:
+        return [
+            f"{'/'.join(map(str, error['loc'])) or '<root>'}: {error['msg']}"
+            for error in e.errors()
+        ]
+
+    return []
+
+
+def main():
+    """Check the metadata files of all problems."""
+    num_failures = 0
+
+    for problem_id in MODELS:
+        errors = check_metadata_file(get_metadata_path(problem_id))
+
+        print(problem_id)
+        if errors:
+            print("\n".join("\t" + error for error in errors))
+            num_failures += 1
+        print("=" * 100)  # just for output readability
+
+    num_passed = len(MODELS) - num_failures
+    print(f"Result: {Path(__file__).stem}")
+    print(f"{num_passed} out of {len(MODELS)} passed.")
+    print(f"{num_failures} out of {len(MODELS)} failed.")
+    sys.exit(num_failures)

src/python/benchmark_models_petab/metadata.py

@@ -0,0 +1,103 @@
+"""Version-independent per-problem metadata."""
+
+import json
+from importlib.resources import files
+from pathlib import Path
+from typing import Any
+
+import yaml
+from pydantic import BaseModel, ConfigDict, Field
+
+from .C import METADATA_FILENAME, MODELS_DIR
+
+METADATA_SCHEMA_FILENAME = "metadata.schema.json"
+
+
+class ProblemMetadata(BaseModel):
+    """Version-independent metadata of a benchmark problem."""
+
+    model_config = ConfigDict(
+        extra="forbid",
+        title="Benchmark problem metadata",
+        json_schema_extra={
+            "$schema": "https://json-schema.org/draft/2020-12/schema"
+        },
+    )
+
+    short_description: str = Field(
+        min_length=20,
+        max_length=160,
+        pattern=r"^[A-Z][\x20-\x7E]*\.$",
+        description=(
+            "Single-sentence summary of the problem: the type of model "
+            "(e.g., biochemical reaction network, compartmental "
+            "epidemiological model), the biological system, and, where "
+            "informative, the data. Plain ASCII text, no line breaks, "
+            "ending with a period."
+        ),
+    )
+
+    @classmethod
+    def from_yaml(cls, path: str | Path) -> "ProblemMetadata":
+        """Load and validate a metadata file.
+
+        Raises
+        ------
+        pydantic.ValidationError
+            If the metadata is invalid.
+        """
+        with open(path, encoding="utf-8") as f:
+            return cls.model_validate(yaml.safe_load(f))
+
+
+def get_metadata_schema() -> dict[str, Any]:
+    """Get the JSON schema for the per-problem metadata files."""
+    return ProblemMetadata.model_json_schema()
+
+
+def get_metadata_path(id_: str) -> Path:
+    """Get the path to the metadata file of the given problem.
+
+    Parameters
+    ----------
+    id_: Problem name, as in `benchmark_models_petab.MODELS`.
+
+    Returns
+    -------
+    The path to the metadata file. The file is not guaranteed to exist.
+    """
+    return Path(MODELS_DIR, id_, METADATA_FILENAME)
+
+
+def get_metadata(id_: str) -> ProblemMetadata:
+    """Get the metadata of a benchmark problem.
+
+    The metadata is not specific to a PEtab version.
+
+    Parameters
+    ----------
+    id_: Problem name, as in `benchmark_models_petab.MODELS`.
+
+    Returns
+    -------
+    The problem metadata.
+    """
+    path = get_metadata_path(id_)
+    if not path.is_file():
+        raise ValueError(f"Could not find metadata for problem `{id_}`.")
+    return ProblemMetadata.from_yaml(path)
+
+
+def write_metadata_schema(path: str | Path | None = None) -> None:
+    """Write the JSON schema for the metadata files.
+
+    By default, the packaged schema file is overwritten. Run after changing
+    `ProblemMetadata`.
+    """
+    if path is None:
+        path = Path(
+            str(files(__package__) / "schemas" / METADATA_SCHEMA_FILENAME)
+        )
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(get_metadata_schema(), f, indent=2)
+        f.write("\n")

src/python/benchmark_models_petab/schemas/metadata.schema.json

@@ -0,0 +1,20 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "description": "Version-independent metadata of a benchmark problem.",
+  "properties": {
+    "short_description": {
+      "description": "Single-sentence summary of the problem: the type of model (e.g., biochemical reaction network, compartmental epidemiological model), the biological system, and, where informative, the data. Plain ASCII text, no line breaks, ending with a period.",
+      "maxLength": 160,
+      "minLength": 20,
+      "pattern": "^[A-Z][\\x20-\\x7E]*\\.$",
+      "title": "Short Description",
+      "type": "string"
+    }
+  },
+  "required": [
+    "short_description"
+  ],
+  "title": "Benchmark problem metadata",
+  "type": "object"
+}

src/python/pyproject.toml

@@ -19,6 +19,8 @@ authors = [
 urls = { "Homepage" = "https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab" }
 dependencies = [
     "petab>=0.4.0",
+    "pydantic>=2",
+    "pyyaml",
     "numpy",
     "pandas",
     "python-libsbml",
@@ -33,6 +35,7 @@ site = ["bokeh>=3.7.3"]
 [project.scripts]
 bmp-petablint = "benchmark_models_petab.check_petablint:main"
 bmp-check-petab-yaml = "benchmark_models_petab.check_petab_yaml:main"
+bmp-check-metadata = "benchmark_models_petab.check_metadata:main"
 bmp-check-sbml-metadata = "benchmark_models_petab.check_sbml_metadata:main"
 bmp-check-tsv-format = "benchmark_models_petab.check_tsv_format:main"
 bmp-create-overview = "benchmark_models_petab.overview:main"