diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 28d3db9..a14642d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,8 +20,19 @@ jobs: - uses: cvmfs-contrib/github-action-cvmfs@72f2a8d366043978d6aebee154bc9ed321c2f2a8 # v5.6 with: cvmfs_repositories: sft.cern.ch - - name: Build and test - run: ci/github-ci.sh /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh + # Also follows the model's own SPINE tag (tools/release/follow_spine.py), which must come + # out "identical"; its artifacts are uploaded when the job fails. + - name: Build and test, then follow the model's SPINE tag + run: >- + ci/github-ci.sh --follow-spine "$RUNNER_TEMP/artifacts" + /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh + - name: Upload the follow_spine artifacts + if: failure() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: follow-spine-artifacts + path: ${{ runner.temp }}/artifacts + if-no-files-found: ignore - name: Tag matches project(VERSION) if: startsWith(github.ref, 'refs/tags/') run: ci/check_tag_version.sh "$GITHUB_REF_NAME" diff --git a/EGM.md b/EGM.md index 06e0ca7..ac0784b 100644 --- a/EGM.md +++ b/EGM.md @@ -76,8 +76,10 @@ Build and packaging (`CMakeLists.txt`): own writer (the `current_*` set by any tag, the frozen cases by old releases), plus their expected JSON, `manifest.json` and `tests/golden_members.inc`. - [./tools/release/EGM.md](./tools/release/EGM.md) - Release tooling: - `scan_consumer_fields.py` (the model fields a CAFMaker build reads, against the contract - table of `tests/test_contracts.cpp`). + `follow_spine.py` (one SPINE tag, end to end: identical, changed, not followed or a + failure, with `status.json`; `--apply` writes a change into the tree), `classify.py` (the + change and the proposed bump) and `scan_consumer_fields.py` (the model fields a CAFMaker + build reads, against the contract table of `tests/test_contracts.cpp`). - examples: [./examples/EGM.md](./examples/EGM.md) - The `spine-dump` command-line tool. - ci: [./ci/EGM.md](./ci/EGM.md) - the GitHub Actions workflow and `github-ci.sh` (HDF5 1.14 and 1.12 from LCG views on CVMFS), `check_tag_version.sh` (release tag against @@ -118,8 +120,11 @@ Build and packaging (`CMakeLists.txt`): - Developer tools only: `tools/gen` rewrites the generated model; `tools/fixtures` clones SPINE commits and rewrites `tests/data/fixtures/` (cases and manifest) and `tests/golden_members.inc`; `tools/release/scan_consumer_fields.py` runs a CAFMaker's compile commands with - `-fsyntax-only` in a work folder; `ci/` scripts create `build-ci/`, `build-asan/`, - `build-tsan/` and `build-sl7/`. + `-fsyntax-only` in a work folder; `tools/release/follow_spine.py` clones SPINE, and builds + and tests an export in its artifacts folder. With `--apply`, after a `changed` outcome, it + writes the new `schema/spine-*.json`, the regenerated model, `tests/golden_members.inc`, the + current fixtures and the manifest into the working tree, and commits nothing. `ci/` scripts + create `build-ci/`, `build-asan/`, `build-tsan/` and `build-sl7/`. - CI: the GitHub Actions workflow builds and tests on GitHub's hosted runners, with read-only access to the repository and no secrets. @@ -129,12 +134,16 @@ Build and packaging (`CMakeLists.txt`): `cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build --output-on-failure` - The ctest entries include `spine_reader_tests` (unit tests), `spine_reader_contracts` (what the CAFMakers rely on: `tests/test_contracts.cpp`), `spine_reader_generated_model` - (`tools/gen/generate.py --check`), `spine_reader_golden_members`, `spine_dump_output`, and + (`tools/gen/generate.py --check`), `spine_reader_golden_members`, + `spine_reader_release_tools` (the unit tests of `tools/release`), `spine_dump_output`, and `spine_reader_real_files` (skipped unless `SPINE_READER_REAL_FILES` names a directory). - `spine_reader_tests` excludes the `real files*` and `contract*` test cases, which - `spine_reader_real_files` and `spine_reader_contracts` run. `spine_reader_generated_model` and `spine_reader_golden_members` are registered only - when CMake finds Python 3 (with `SPINE_READER_REQUIRE_PYTHON=ON`, a missing Python fails the - configuration), and `spine_dump_output` only when the tools are built. + `spine_reader_real_files` and `spine_reader_contracts` run. `spine_reader_generated_model` + and `spine_reader_golden_members` are registered only when CMake finds Python 3 (with + `SPINE_READER_REQUIRE_PYTHON=ON`, a missing Python fails the configuration), + `spine_reader_release_tools` only for Python 3.10 or newer, and `spine_dump_output` only + when the tools are built. - CI: `.github/workflows/ci.yml` runs `ci/github-ci.sh` with HDF5 1.14 (LCG_110, Ubuntu 24.04) - and HDF5 1.12 (LCG_105, Ubuntu 22.04) on every push and pull request. + and HDF5 1.12 (LCG_105, Ubuntu 22.04) on every push and pull request. The 1.14 job then runs + `tools/release/follow_spine.py` on the model's own SPINE tag and requires `identical`. - Other toolchains: `ci/sl7-ups.sh`, `ci/sanitize.sh`, `ci/sanitize.sh thread`. diff --git a/README.md b/README.md index fa4c845..afa807e 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Status: under construction. | `src/model/` | Generated field tables. Do not edit by hand. | | `tools/gen/` | Generator: dumps SPINE `FieldMetadata` to `schema/` and writes the model. | | `tools/fixtures/` | Writes the test files in `tests/data/` with SPINE's own writer, plus expected values. | -| `tools/release/` | Tooling for following SPINE releases: `scan_consumer_fields.py` lists the model fields a CAFMaker build reads. | +| `tools/release/` | Tooling for following SPINE releases: `follow_spine.py` runs one SPINE tag end to end, `classify.py` proposes the version bump, `scan_consumer_fields.py` lists the model fields a CAFMaker build reads ([README](tools/release/README.md)). | | `tests/` | doctest unit tests (`test_*.cpp`). | ## Build @@ -55,6 +55,7 @@ cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build --outpu | `spine_reader_real_files` | Reads every `*.h5` in `$SPINE_READER_REAL_FILES` (or `-DSPINE_READER_REAL_FILES=dir`). Without a directory it reports **Skipped**, never Passed. | | `spine_reader_generated_model` | `tools/gen/generate.py --check`: the generated model is what the committed SPINE schema produces. | | `spine_reader_golden_members` | `tests/golden_members.inc` (the member list that `test_golden.cpp` compares object by object, and that `test_model.cpp` and `test_contracts.cpp` walk) is current. | +| `spine_reader_release_tools` | The unit tests of `tools/release` (`classify.py`, `follow_spine.py`), without SPINE or the network. Registered for Python 3.10 or newer. | | `spine_dump_output` | `spine-dump` prints what it should for the Reader test files. | When a CAFMaker starts reading a model field, add its row to the contract table. @@ -71,6 +72,8 @@ CI (`.github/workflows/ci.yml`) builds and runs ctest on every push and pull req 1.14 (LCG_110, Ubuntu 24.04) and HDF5 1.12 (LCG_105, Ubuntu 22.04) from the LCG views on CVMFS. Both jobs run `ci/github-ci.sh`, which configures with `-DSPINE_READER_REQUIRE_PYTHON=ON`: a missing Python 3 then fails the configuration instead of silently leaving out the Python-based checks. +The 1.14 job then runs `tools/release/follow_spine.py` on the model's own SPINE tag, which must +come out `identical`; its artifacts are uploaded when it fails. `ci/EGM.md` shows how to run the same jobs locally. ## Conventions diff --git a/ci/EGM.md b/ci/EGM.md index 11d9e62..e52de5a 100644 --- a/ci/EGM.md +++ b/ci/EGM.md @@ -13,13 +13,17 @@ Every script finds the repository from its own location, so it can be run from a directory. ## Interface -- `github-ci.sh [build directory]` +- `github-ci.sh [--follow-spine DIR] [build directory]` - Sources the given LCG view, then configures `/` (default `build-ci`) with `-DSPINE_READER_REQUIRE_PYTHON=ON`, builds with `-j$(nproc)`, and runs `ctest --output-on-failure`. No build type is given, so the project default (`RelWithDebInfo`) applies, as in CI. + - With `--follow-spine DIR`, it then reads `spine_version` from the one `schema/spine-*.json` + and runs `tools/release/follow_spine.py v --artifacts-dir DIR --jobs + $(nproc)`. It needs exit 0 (`identical`). - Prints the OS, the compiler (`$CXX`, set by the LCG view), CMake, Python and the view. - - Exit codes: 2 on bad arguments; otherwise the first failing command's code. + - Exit codes: 2 on bad arguments; otherwise the first failing command's code (for + `follow_spine.py`, its own: 10, 20, 30 or 2). - `check_tag_version.sh vX.Y.Z` - Reads `project(spine-reader VERSION X.Y.Z ...)` from `CMakeLists.txt`. - Exits 0 when the argument equals `vX.Y.Z`, 1 when it does not or when no version line is @@ -29,13 +33,22 @@ directory. - Runs on every push (branches and tags), every pull request and manual dispatch, with `permissions: contents: read` and no secrets. - Job `hdf5-1_14`: `ubuntu-24.04`, LCG_110 `x86_64-ubuntu2404-gcc13-opt` (HDF5 1.14.6, gcc 13 - from the runner, CMake 3.30, Python 3.13). On a tag push it also runs - `check_tag_version.sh "$GITHUB_REF_NAME"`. + from the runner, CMake 3.30, Python 3.13). + - It runs `github-ci.sh --follow-spine "$RUNNER_TEMP/artifacts"`: the build, ctest, then + `follow_spine.py` on the model's own SPINE tag, which clones SPINE from GitHub and must + give `identical`. The tag must still point at the schema's `spine_sha`, so CI only ever + runs that recorded SPINE commit. + - When the job fails, it uploads `$RUNNER_TEMP/artifacts` as `follow-spine-artifacts` + (`actions/upload-artifact`). + - On a tag push it also runs `check_tag_version.sh "$GITHUB_REF_NAME"`. - Job `hdf5-1_12`: `ubuntu-22.04`, LCG_105 `x86_64-ubuntu2204-gcc11-opt` (HDF5 1.12.2, gcc 11 from the runner, CMake 3.26, Python 3.9). - Each job checks out the repository, mounts `sft.cern.ch` with - `cvmfs-contrib/github-action-cvmfs`, and runs `github-ci.sh` with its view. The two actions - are pinned by commit SHA, with the version in a comment. + `cvmfs-contrib/github-action-cvmfs`, and runs `github-ci.sh` with its view. The actions + (`actions/checkout`, the CVMFS action and `actions/upload-artifact`) are pinned by commit + SHA, with the version in a comment. + - The release-tool unit tests (`spine_reader_release_tools`) need Python 3.10, so they run in + the 1.14 job only; the 1.12 job (Python 3.9) registers six ctest entries. - `sanitize.sh [mode]` - With no argument (`mode=address`, the default) it configures with `-DSPINE_READER_SANITIZE=ON`, which enables ASan and UBSan with @@ -84,6 +97,9 @@ directory. - Read from CVMFS: the LCG views, `setup_larsoft.sh` and UPS products, and the container images. `sl7-ups.sh` and the dry run below need `apptainer` on the host. - The workflow runs on GitHub's hosted runners, with read-only access to the repository. +- `github-ci.sh --follow-spine DIR` clones SPINE (from GitHub, into + `$TMPDIR/spine-reader-fixtures`) and writes `DIR`: the artifacts, an export of HEAD and its + build. ## Verification - Run: `ci/sanitize.sh` (ASan and UBSan), `ci/sanitize.sh thread` (TSan), and @@ -94,6 +110,7 @@ directory. ```bash img=/cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/sft/docker apptainer exec -B /cvmfs -B /tmp $img/ubuntu24:latest bash -e ci/github-ci.sh \ + --follow-spine /tmp/follow-spine-artifacts \ /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh build-ci-1.14 apptainer exec -B /cvmfs -B /tmp $img/ubuntu22:latest bash -e ci/github-ci.sh \ /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-ubuntu2204-gcc11-opt/setup.sh build-ci-1.12 diff --git a/ci/github-ci.sh b/ci/github-ci.sh index 6b9b56d..4deb717 100755 --- a/ci/github-ci.sh +++ b/ci/github-ci.sh @@ -1,13 +1,24 @@ #!/bin/bash # Build and test spine-reader the way the GitHub workflow (.github/workflows/ci.yml) does: # source an LCG view, configure with SPINE_READER_REQUIRE_PYTHON=ON, build, run ctest. +# With --follow-spine DIR, then run tools/release/follow_spine.py on the model's own SPINE tag, +# with its artifacts in DIR, and require "identical" (exit 0). # The same script is used for the local dry run (see ci/EGM.md). # -# ci/github-ci.sh [build directory, default build-ci] +# ci/github-ci.sh [--follow-spine DIR] [build directory, default build-ci] set -uo pipefail +follow= +if [[ ${1:-} == --follow-spine ]]; then + if [[ $# -lt 2 ]]; then + echo "usage: $0 [--follow-spine DIR] [build directory]" >&2 + exit 2 + fi + follow=$2 + shift 2 +fi if [[ $# -lt 1 || $# -gt 2 ]]; then - echo "usage: $0 [build directory]" >&2 + echo "usage: $0 [--follow-spine DIR] [build directory]" >&2 exit 2 fi view=$1 @@ -26,3 +37,15 @@ echo "== $(cmake --version | head -1) | $(python3 --version) | view $view" cmake -S "$repo" -B "$build" -DSPINE_READER_REQUIRE_PYTHON=ON cmake --build "$build" -j"$(nproc)" ctest --test-dir "$build" --output-on-failure + +if [[ -n $follow ]]; then + # the model's SPINE version, from the one committed schema + version=$(python3 -c 'import glob, json, sys +found = glob.glob(sys.argv[1] + "/schema/spine-*.json") +assert len(found) == 1, found +print(json.load(open(found[0]))["spine_version"])' "$repo") + echo "== follow_spine.py v$version (artifacts in $follow)" + "$repo/tools/release/follow_spine.py" "v$version" --artifacts-dir "$follow" --jobs "$(nproc)" || + { code=$?; echo "follow_spine.py v$version exited $code; identical (0) is required" >&2 + exit "$code"; } +fi diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 909f8e8..be3c897 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -54,6 +54,14 @@ if(Python3_Interpreter_FOUND) add_test(NAME spine_reader_golden_members COMMAND ${Python3_EXECUTABLE} ${PROJECT_SOURCE_DIR}/tools/fixtures/make_member_list.py --check) + # The unit tests of tools/release (classify.py, follow_spine.py): no SPINE, no network. The + # release tools need Python 3.10, as SPINE does; older interpreters leave them out. + if(Python3_VERSION VERSION_GREATER_EQUAL 3.10) + add_test(NAME spine_reader_release_tools + COMMAND ${Python3_EXECUTABLE} -m unittest discover + -s ${PROJECT_SOURCE_DIR}/tools/release/tests + -t ${PROJECT_SOURCE_DIR}/tools/release) + endif() else() message(WARNING "No Python 3 interpreter: the generated-model checks are not registered") endif() diff --git a/tools/EGM.md b/tools/EGM.md index 6b57631..60a61a5 100644 --- a/tools/EGM.md +++ b/tools/EGM.md @@ -7,10 +7,13 @@ library. They are tied to SPINE and to the single committed `schema/spine-*.json`. `gen` turns SPINE's object model into the generated C++ model. `fixtures` produces the golden HDF5 files and their decoded expectations, which the C++ tests read. `release` holds the tooling for following SPINE -releases; today `scan_consumer_fields.py`, which lists the model fields a -CAFMaker build reads. `gen` and `fixtures` each register a ctest check: -`spine_reader_generated_model` (`gen/generate.py --check`) and -`spine_reader_golden_members` (`fixtures/make_member_list.py --check`). +releases: `follow_spine.py` (one SPINE tag, end to end, to one outcome), +`classify.py` (what changed, and the proposed bump) and +`scan_consumer_fields.py` (the model fields a CAFMaker build reads). Each +registers a ctest check: `spine_reader_generated_model` (`gen/generate.py +--check`), `spine_reader_golden_members` (`fixtures/make_member_list.py +--check`) and `spine_reader_release_tools` (the unit tests of `release`, only +for Python 3.10 or newer). `tests/CMakeLists.txt` registers them when CMake finds a Python 3 interpreter; with `-DSPINE_READER_REQUIRE_PYTHON=ON` (CI turns it on) a missing interpreter fails the configuration instead of leaving them out. Run the tools in the @@ -36,7 +39,18 @@ CAFMaker's environment. resolves tags and clones commits. `make_member_list.py` writes or checks `tests/golden_members.inc` from the schema. Legacy notes: [./fixtures/README.md](./fixtures/README.md). -- release: [./release/EGM.md](./release/EGM.md) - `scan_consumer_fields.py` - (compiles a CAFMaker's sources with `-fsyntax-only` against model headers whose - members are marked `[[deprecated]]`, and lists the fields read; with - `--contracts`, the ones that have no row in `tests/test_contracts.cpp`). +- release: [./release/EGM.md](./release/EGM.md) - `follow_spine.py TAG` runs the + following stages and ends in one outcome: + - it dumps the tag's schema; + - it compares it with the model (identical or changed); + - it writes and checks the tag's fixtures; + - it classifies the change; + - it builds and tests an export of HEAD with them. + + The exit code is 0 identical, 10 changed, 20 not followed, 30 failure, or 2 usage. + `--apply` writes a change into the working tree. + + `classify.py` proposes the version bump. `scan_consumer_fields.py` compiles a CAFMaker's + sources with `-fsyntax-only` against model headers whose members are marked + `[[deprecated]]`, and lists the fields read; with `--contracts`, it lists the ones that have + no row in `tests/test_contracts.cpp`. Guide: [./release/README.md](./release/README.md). diff --git a/tools/gen/EGM.md b/tools/gen/EGM.md index 9678b51..904eece 100644 --- a/tools/gen/EGM.md +++ b/tools/gen/EGM.md @@ -171,7 +171,8 @@ SPINE clone --dump_schema.py--> schema/spine-.json --generate.py--> C++ changed files (written or deleted). - `check_names.py` compiles a temporary `all.cpp` with ` -std=gnu++17 -dM -E -x c++` (`check=True`). It reads - `include/spine/model*` and `src/model/*`, and prints the clashes and a summary. + the `.hpp` and `.cpp` files of `include/spine/model*` and `src/model/` (not the `EGM.md` + files), and prints the clashes and a summary. ## Verification diff --git a/tools/gen/README.md b/tools/gen/README.md index cb8e136..dacb074 100644 --- a/tools/gen/README.md +++ b/tools/gen/README.md @@ -10,6 +10,11 @@ SPINE clone at a tag --dump_schema.py--> schema/spine-.json --generate. ## Regenerate for a new SPINE release +`tools/release/follow_spine.py vX.Y.Z` does all of this, and checks the result: it dumps the +tag, compares the schema with the model, writes and checks the tag's fixtures, builds and tests +the regenerated model, and with `--apply` writes it into the working tree +([../release/README.md](../release/README.md)). The manual commands below are what it runs: + ```bash source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc13-opt/setup.sh git clone -q --branch vX.Y.Z /some/dir/spine-X.Y.Z @@ -50,7 +55,9 @@ mismatch also exits 1. enumerator by a trailing underscore (`NULL` -> `NULL_`; the descriptor keeps `"NULL"`) from its `RESERVED` list; if the checker finds a new clash, add the name there. `--include` adds headers (HDF5, ROOT) to the scan. -4. **Build and test.** `cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build`. +4. **Build and test.** Read `classification.md` from the `follow_spine.py` run first: it lists + what changed, kind by kind, and the proposed bump. + `cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build`. The `static_assert`s in `src/model/*.cpp` fail the build if a member's C++ type disagrees with its descriptor row. No test holds field counts, defaults or enum member counts: they follow the schema. Write the new tag's fixtures first (`tools/fixtures/make_fixtures.py --tag `), diff --git a/tools/gen/check_names.py b/tools/gen/check_names.py index 8da58a3..01bdc5f 100644 --- a/tools/gen/check_names.py +++ b/tools/gen/check_names.py @@ -60,7 +60,7 @@ def main(): macros = system_macros(args.cxx, args.include) files = sorted( [*(root / "include/spine").glob("model/*.hpp"), root / "include/spine/model.hpp"] - + [*(root / "src/model").glob("*")] + + [p for p in (root / "src/model").glob("*") if p.suffix in (".cpp", ".hpp")] ) clashes = {} for path in files: diff --git a/tools/release/EGM.md b/tools/release/EGM.md index 5ccd29d..d67208d 100644 --- a/tools/release/EGM.md +++ b/tools/release/EGM.md @@ -2,14 +2,96 @@ ## Overview -Tooling for following SPINE releases. Today it holds `scan_consumer_fields.py`, which keeps the -contract table of `tests/test_contracts.cpp` current: it lists the model fields one CAFMaker -build reads, and, with `--contracts`, the ones that have no row in the table. A row is added -when a CAFMaker starts reading a field. The tool runs the CAFMaker's own compile commands, so it -is run by hand, in that CAFMaker's environment; no ctest entry runs it. +Tooling for following SPINE releases. The hand-written [README.md](./README.md) is the guide: +the environment, the outcomes and what to do for each, replays, `--apply`. + +- `follow_spine.py` runs the whole chain for one SPINE tag and ends in one outcome: + - it dumps the tag's schema and compares it with the model; + - it writes and checks the tag's fixtures; + - it classifies the change; + - it builds and tests an export of HEAD with the new schema and fixtures. + + CI runs it on the model's own tag and requires `identical`. +- `classify.py` describes the change between two schemas and proposes a version bump. +- `scan_consumer_fields.py` keeps the contract table of `tests/test_contracts.cpp` current. It + lists the model fields one CAFMaker build reads, and, with `--contracts`, the ones that have no + row. It runs the CAFMaker's own compile commands, so it is run by hand, in that CAFMaker's + environment. +- `tests/`: the unit tests of the first two (ctest `spine_reader_release_tools`). ## Interface +### `follow_spine.py` (CLI; Python 3.10+; mode 100755) + +``` +follow_spine.py TAG [--baseline SCHEMA] [--apply] [--repo-root DIR] [--spine-repo URL] + [--workdir DIR] [--artifacts-dir DIR] [--jobs N] +``` + +- Defaults: + - `--repo-root`: the checkout the script is in; + - `--spine-repo`: `$SPINE_REPO`, else `https://github.com/DeepLearnPhysics/spine`; + - `--workdir`: `$SPINE_FIXTURE_WORKDIR`, else `$TMPDIR/spine-reader-fixtures`, shared with + `make_fixtures.py`; + - `--artifacts-dir`: a new `follow_spine--*` temporary folder; + - `--jobs`: the number of CPUs. +- Stages, in order: `s0_checks`, `s1_tag`, `s2_dump`, `s3_compare`, `s4_fixtures`, + `s5_classify`, `s6_build`, `s6_test`, `s8_apply` (README.md describes each). +- Outcomes and exit codes: + - `identical` 0; + - `changed` 10, with `contracts_failed` true when `spine_reader_contracts` is the only + failed ctest entry; + - `not_followed` 20; + - 30 for `environment_error`, `tag_moved`, `dump_failed`, `version_mismatch`, + `generate_failed`, `fixture_tooling_failed`, `fixture_check_failed`, `build_failed`, + `test_failed` and `internal_error`; + - 2 for a usage error, which writes no `status.json`. +- `/status.json` keys: + - `status_format_version` (1), `tag`, `spine_sha`, `spine_version`, `baseline {path, + spine_version, spine_sha}`; + - `outcome`, `exit_code`, `failed_stage`, `failure_detail`, `failure_reason`; + - `contracts_failed`, `needs_attention`, `replay`, `partial` (false); + - `comparison {result, files, type_check_changed, writer_format_changed, error}`, + `proposed_bump`, `change_kinds`; + - `stages {: {status, duration_s, log, ...}}`, `warnings`, `substitutions`; + - `tools {python, numpy, h5py, hdf5, cmake, compiler, git}`; + - `spine_reader {commit, dirty}`, `artifacts`, `reproduce`. +- The other artifacts: `logs/.log`, `schema/spine-X.Y.Z.json`, `fixtures/` (the new + current set and merged manifest), `head/fixtures/`, `classification.{json,md}`, + `warnings.json`, `baseline-warnings.json`, `export/`, `build/`, `ctest-junit.xml`, + `previous/`, `effects.txt`, `fixture-run/`. +- Functions the unit tests call, needing neither git nor the network nor SPINE: + - `check_usage(args, repo)`; + - `outcome_for(stages)`, which returns `(outcome, exit code, failed stage, + contracts_failed)`; + - `compare_schemas(baseline, new, generator=None)`; + - `failed_tests(junit_path)`. + +### `classify.py` (CLI and library; Python 3.10+, standard library only) + +``` +classify.py BASELINE NEW [--old-columns DIR --new-columns DIR] [--warnings FILE] + [--baseline-warnings FILE] [--effects FILE] --json OUT --md OUT +``` + +- `classify(baseline, new, old_columns, new_columns, warnings, baseline_warnings, effects, ...)` + returns these keys: + - `baseline` and `new` `{path, spine_version, spine_sha}`; + - `classes {: {added, removed, retyped, reordered, changed}}`, `classes_added`, + `classes_removed`; + - `enums {: {added, removed, renumbered, aliases, doc_changed}}`, `enums_added`, + `enums_removed`, `scheme_tables_changed`; + - `type_check_changed`, `writer_format_changed`, `column_diffs`, `default_only_fields`, + `unwritten_classes`; + - `warnings_new`, `substitutions`, `effects_on_previous_files`; + - `change_kinds`, `module_only`, `proposed_bump`, `reasons`, `follow_up`, `needs_attention`. +- `BUMP_TABLE` maps every change kind to `none`, `patch`, `minor` or `person`. `bump_of()` gives + `person` for a kind the table lacks. The highest bump wins. +- `render_markdown(classification)`: the text of `classification.md`, ending with the checklist + of `tools/gen/README.md`. +- Exit codes: 0 written; 2 bad input (an unreadable file, or a schema without `spine_version`, + `classes`, `enums` or `scheme_tables`). + ### `scan_consumer_fields.py` (CLI; Python 3.9+, standard library only) ``` @@ -56,6 +138,39 @@ python3 tools/release/scan_consumer_fields.py --name ND \ ## Invariants +- `follow_spine.py` never runs `git commit` and never edits `CMakeLists.txt`. Without + `--apply` it writes only in the artifacts folder and the `--workdir` clones. +- Stage `s0_checks` runs before any network access: + - a tag `vX.Y.Z` is followed; + - `vX.Y.Z` followed by more characters is a pre-release, and is `not_followed`; + - anything else is a usage error; + - a tag below v1.2.1 is `not_followed`. +- `--apply` needs a clean working tree (no changes, no untracked files), no `--baseline`, and a + tag newer than the model's; otherwise it is a usage error before any network access. +- A run is a replay when its `--baseline` is not the committed schema, or its tag is older than + the model's. A replay stops after `s5_classify`. Without the default baseline, the + classification has no column comparison, because HEAD's fixtures belong to the committed + schema. +- `s1_tag`: + - For the baseline's own version, a missing tag, or a commit other than the baseline's + `spine_sha`, is `tag_moved`. + - Another missing tag is a usage error. + - A `git ls-remote` that fails is `environment_error`. +- `s3_compare` is identical exactly when `generate.generate()` gives the same files for the + baseline and for the new schema with its `spine_version` set to the baseline's, and + `type_check` and `writer_format` are equal. Warnings do not count. +- `s6_test`: + - When only `spine_reader_contracts` fails and the comparison is `changed`, the outcome is + `changed` with `contracts_failed`. + - Any other failed entry, or a contract failure with an identical comparison, is + `test_failed`. + - `spine_reader_real_files` is skipped: `SPINE_READER_REAL_FILES` is removed from ctest's + environment. +- The effects text and the second classification never change an outcome. +- `classify.py` never decides "identical". A warning is new when its `(kind, name)` is not among + the baseline schema's warnings or `--baseline-warnings`. Substitutions (kind `substitution`) + are listed apart. + - The marking: - The copy of `include/` marks every member of every schema class `[[deprecated("SPINEFIELD .")]]`, and every `Field` enumerator likewise. @@ -82,6 +197,19 @@ python3 tools/release/scan_consumer_fields.py --name ND \ ## Side Effects +- `follow_spine.py`: + - It writes the artifacts folder. + - It clones SPINE commits into `--workdir`, through `tools/fixtures/spine_checkout.py`. + - It runs `dump_schema.py`, `make_fixtures.py`, `classify.py`, `generate.py`, + `check_names.py`, `make_member_list.py`, CMake, the compiler and ctest as subprocesses. + - It accesses the network through `git ls-remote` and `git clone` of `--spine-repo`. + - With `--apply`, after `changed`, it also writes into the working tree of `--repo-root`: + - `schema/spine-*.json`, replaced; + - `include/spine/model/`, `include/spine/model.hpp` and `src/model/`, regenerated; + - `tests/golden_members.inc`; + - `tests/data/fixtures/current_*` and `manifest.json`. +- `classify.py` writes only `--json` and `--md`. + - Creates the work folder and writes the files listed under `--workdir`; deletes it after a successful run unless `--keep` is given, and keeps it when the scan fails. - Runs `make -n -B` in the build folder and the CAFMaker's compilers, through `--wrap` when @@ -91,6 +219,24 @@ python3 tools/release/scan_consumer_fields.py --name ND \ ## Verification +- `python3 -m unittest discover -s tools/release/tests -t tools/release` (ctest + `spine_reader_release_tools`, registered for Python 3.10 or newer). +- `follow_spine.py v` gives `identical` (exit 0). CI runs it in the HDF5 1.14 job + (`ci/github-ci.sh --follow-spine`). +- 2026-11-06, local SPINE tags in a scratch clone (`--spine-repo `) gave the outcomes of + the release design's verification table. For example: + - a field added: `changed`, patch; + - a default, unit or field the CAFMakers read: `changed` with `contracts_failed`; + - two enum values swapped: `test_failed`; + - a reader that raises: `fixture_check_failed`; + - `FieldMetadata.pdg` deleted: `dump_failed`; + - `version.py` not updated: `version_mismatch`; + - a renamed writer parameter: `fixture_tooling_failed`; + - a moved v1.4.0: `tag_moved`. + + Also checked: the replays of v1.2.1, v1.2.2 and v1.2.3, and `--apply` on the added field in a + clean clone. + - On 2026-11-06 the three ports were scanned in the SL7 container. Each was compared with the scan of 2026-11-05 from which the contract table was written: - ND (DUNE/ND_CAFMaker#213; its ML filler is unchanged since `fbda59e`): 47 fields; diff --git a/tools/release/README.md b/tools/release/README.md new file mode 100644 index 0000000..e6abcd6 --- /dev/null +++ b/tools/release/README.md @@ -0,0 +1,192 @@ +# tools/release: following SPINE releases + +When SPINE tags a release, `follow_spine.py` answers one question: does spine-reader need a new +release for it? It dumps the tag's schema, compares it with the model, writes and checks the +tag's own fixtures, describes the change, and builds and tests spine-reader with it. It changes +nothing in the repository unless `--apply` is given, and it never commits. + +| Tool | What it does | +| --- | --- | +| `follow_spine.py` | The whole chain for one SPINE tag; one outcome and exit code | +| `classify.py` | Describes the change between two schemas and proposes a version bump | +| `scan_consumer_fields.py` | Lists the model fields a CAFMaker build reads, for the contract table | + +## What you need + +`follow_spine.py` needs the LCG_110 view, git, network access to the SPINE repository and a +C++ compiler: + +```bash +source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc13-opt/setup.sh +``` + +In detail, it needs: + +- Python 3.10 or newer (SPINE's own requirement), with numpy, h5py, yaml and scipy; +- git; +- CMake 3.21 or newer, for `ctest --output-junit`; +- a C++ compiler. + +Stage `s0_checks` checks all of this before any network access, and reports +`environment_error` when something is missing. + +## Running it + +```bash +tools/release/follow_spine.py v1.4.1 +``` + +It prints the artifacts folder, a new temporary folder unless `--artifacts-dir` is given. That +folder holds: + +- `status.json`; +- `classification.md` and `classification.json`; +- one log per stage in `logs/`; +- the new schema in `schema/` and the new current set in `fixtures/`; +- the export and its build. + +The last lines it prints name the outcome and give the command that reproduces the run. + +Options: + +- `--spine-repo URL` (default `https://github.com/DeepLearnPhysics/spine`, or `$SPINE_REPO`): + a git URL or a local path. +- `--workdir DIR`: where the SPINE clones live, one per commit. It is shared with + `tools/fixtures/make_fixtures.py` (default `$TMPDIR/spine-reader-fixtures`). +- `--jobs N`: parallel build jobs (default: the number of CPUs). +- `--repo-root DIR`: the spine-reader checkout (default: the one the script is in). +- `--baseline SCHEMA`, `--apply`: see below. + +What a run does, stage by stage: + +| Stage | What it does | +| --- | --- | +| `s0_checks` | The tag's form, `--apply`'s conditions, the environment | +| `s1_tag` | The tag's commit (`git ls-remote`). The model's own tag must still point at the schema's `spine_sha`. One clone per commit | +| `s2_dump` | `tools/gen/dump_schema.py --spine-sha`, without `--strict`. The tag must equal `v` + `spine.__version__` | +| `s3_compare` | **identical** when `generate.py` gives the same files for both schemas once the version is normalised, and `type_check` and `writer_format` are unchanged; otherwise **changed** | +| `s4_fixtures` | `make_fixtures.py --tag`: the tag's current set, written by its own writer. Every cross-check runs, and the tag's own reader reads the files back | +| `s5_classify` | `classify.py`: what changed, and the proposed bump. A replay stops here | +| `s6_build` | `git archive HEAD` into the artifacts folder, with the new current set and manifest. If changed, it adds the new schema, the regenerated model, `check_names.py` and `golden_members.inc`. Then it configures and builds | +| `s6_test` | ctest with `--output-junit`, without `SPINE_READER_REAL_FILES`. Then the regenerated `spine-dump` reads HEAD's previous `current_v1`/`current_v2`, and `classify.py` runs again with that text | +| `s8_apply` | With `--apply`, after `changed` only: the new schema, model and fixtures go into the working tree | + +Uncommitted changes are not tested: the build uses `git archive HEAD`. + +## Outcomes + +| Exit | Outcome | What to do | +| --- | --- | --- | +| 0 | `identical` | Nothing. The tag needs no spine-reader release, and the working tree is unchanged. If `needs_attention` is true, read the warnings in `classification.md` (for example a class `dump_schema.py` does not list) | +| 10 | `changed` | Read `classification.md`, then rerun with `--apply` and open a pull request with the proposed bump in `project(VERSION)`. With `contracts_failed`, a unit, default or enum value a CAFMaker relies on changed: talk to the CAFMaker maintainers before editing `tests/test_contracts.cpp` | +| 20 | `not_followed` | Nothing. The tag is a pre-release (`v1.5.0rc1`) or older than v1.2.1 | +| 30 | `environment_error` | Fix the environment (the log of `s0_checks` or `s1_tag` says what is missing), or check the network and `--spine-repo` | +| 30 | `tag_moved` | The model's own tag is gone, or points to another commit than the schema's `spine_sha`. Ask the SPINE maintainers; never re-dump over it | +| 30 | `dump_failed` | `dump_schema.py` cannot read the tag. `failure_reason` is the exception class (`AttributeError`, ...); fix the dumper | +| 30 | `version_mismatch` | The tag and `spine.__version__` differ: a SPINE release mistake. Report it upstream | +| 30 | `generate_failed` | `generate.py` or `check_names.py` refuses the new schema: fix the generator | +| 30 | `fixture_tooling_failed` | A fixture worker cannot drive the tag's API (`make_fixtures.py` exit 3): adapt `tools/fixtures/spine_worker.py` | +| 30 | `fixture_check_failed` | A cross-check failed, or the tag's own reader disagrees with the files it wrote. See `logs/s4_fixtures.log` | +| 30 | `build_failed` | The regenerated model does not build. See `logs/s6_build.log` | +| 30 | `test_failed` | A ctest entry other than `spine_reader_contracts` failed in the export. `stages.s6_test.failed_tests` names them | +| 30 | `internal_error` | A bug in the tools. The stage log has the traceback | +| 2 | (usage error) | Bad arguments, a malformed tag, a tag that does not exist (other than the model's own), or `--apply` that cannot apply. No `status.json` is written | + +`status.json` records: + +- `status_format_version` (1), and the tag, its `spine_sha` and `spine_version`; +- the baseline; +- `outcome`, `exit_code`, `failed_stage`, `failure_detail` and `failure_reason`; +- `contracts_failed`, `needs_attention`, `replay` and `partial` (always false); +- `comparison`, `proposed_bump` and `change_kinds`; +- `stages`: each with its `status`, `duration_s` and `log`; +- the warnings and substitutions; +- `tools`: the Python, numpy, h5py, HDF5, CMake, compiler and git versions; +- the spine-reader commit; +- `reproduce`. + +## Replays + +A replay checks an old pair of tags without building or applying anything: it stops after +`s5_classify`. A run is a replay when it is given a `--baseline` other than the committed +schema, or a tag older than the model's: + +```bash +tools/release/follow_spine.py v1.2.3 --baseline tools/release/tests/data/spine-1.2.2.json +``` + +Its outcome is `identical` or `changed`, with `replay: true`. +`tools/release/tests/data/spine-1.2.2.json` and `spine-1.2.3.json` are dumps of those tags made +with the current dumper. Older tags still write their own fixtures, so a replay of v1.2.1 or +v1.2.2 also checks them, with the enum labels those tags lack listed as substitutions. + +## `--apply` + +`--apply` needs three things: + +- a clean working tree, with no changes and no untracked files; +- no `--baseline`; +- a tag newer than the model's. + +It writes only after a `changed` outcome: + +- the new `schema/spine-X.Y.Z.json`, replacing the old one (git keeps it); +- the regenerated model, through `generate.py --root`, which also deletes the files the + generator no longer writes; +- `tests/golden_members.inc`; +- the new current set (`tests/data/fixtures/current_*`) and the merged manifest. The frozen + cases keep their entries. + +It does not edit `project(VERSION)`, and does not commit. It prints the proposed bump and the +follow-up list. `git checkout . && git clean -fd` undoes it. + +## `classify.py` + +```bash +tools/release/classify.py BASELINE NEW [--old-columns DIR --new-columns DIR] [--warnings FILE] \ + [--baseline-warnings FILE] [--effects FILE] --json OUT --md OUT +``` + +`follow_spine.py` runs it twice. It can also be run by hand on any two schema dumps. It lists the +change kinds it finds, and proposes the bump of the highest one: + +| Change kind | Proposed bump | +| --- | --- | +| nothing | `none` | +| field or enum docs, module paths (`doc_changed`, `module_changed`) | `none`: the regenerated model may merge without a tag | +| fields, enum members, enums or scheme tables added | `patch`, provisional until decision D6 (strict mode under a patch release) | +| a field, class, enum, enum member or scheme table removed; a shape or element retype; a renumbered enum member | `minor`: a 0.x release's breaking change under `SameMinorVersion` | +| a default, unit, flag or reference changed; a field reordered; a class added; enum aliases; a scheme table changed; `type_check`; `writer_format`; a stored width or type in the decoded columns; columns no schema change explains | `person`: a person decides | +| any kind not listed | `person` | + +The order is none < patch < minor < person. `needs_attention` is true when a class was added, +when a warning is new (by its `(kind, name)`), or when the fixtures needed substitutions. Exit +codes: 0, or 2 on bad input. + +## `scan_consumer_fields.py` + +When a CAFMaker starts reading a model field, the field needs a row in +`tests/test_contracts.cpp`. Run the scan on that CAFMaker's build, in its own environment: + +```bash +python3 tools/release/scan_consumer_fields.py --name ND \ + --source ~/ND_CAFMaker --build ~/ND_CAFMaker/build \ + --setup 'cd ~/ND_CAFMaker && source ./ndcaf_setup_deps.sh prof' \ + --wrap 'apptainer exec -B /cvmfs -B /tmp -B /home /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest' \ + --contracts tests/test_contracts.cpp +``` + +It lists the fields the build reads, and the ones that have no row (exit 1). See +[EGM.md](./EGM.md) for its options. + +## Tests + +`spine_reader_release_tools` (ctest, Python 3.10 or newer) runs +`python3 -m unittest discover -s tools/release/tests -t tools/release`. The tests need neither +SPINE nor the network: + +- `test_classify.py` checks every change kind on mutated copies of the frozen + `tests/data/spine-1.4.0.json`, the precedence of the bumps, the real pair 1.2.2 → 1.2.3, and + that every change kind has a row in the bump table. +- `test_outcomes.py` checks `outcome_for()`, `check_usage()` and `compare_schemas()`. Its + `--apply` test needs the git binary. diff --git a/tools/release/classify.py b/tools/release/classify.py new file mode 100755 index 0000000..80b1ae4 --- /dev/null +++ b/tools/release/classify.py @@ -0,0 +1,622 @@ +#!/usr/bin/env python3 +"""Describe the change between two schema dumps of SPINE, and propose a version bump. + + classify.py BASELINE NEW [--old-columns DIR --new-columns DIR] [--warnings FILE] + [--baseline-warnings FILE] [--effects FILE] --json OUT --md OUT + +* BASELINE and NEW are ``schema/spine-*.json`` files written by ``tools/gen/dump_schema.py``. +* ``--old-columns`` and ``--new-columns`` are folders of ``*.expected.json`` files (the decoded + fixtures): the committed current set and the new tag's. The cases both folders hold are + compared column by column; that is where a change of a stored width shows. Without them there + is no column comparison. +* ``--warnings`` is a JSON list of ``{kind, name, detail}``: the fixture run's warnings and + substitutions (``kind`` ``"substitution"``). The new schema's own ``warnings`` are added to it. + ``--baseline-warnings`` is the same for the baseline (the committed manifest's), added to the + baseline schema's. A warning is new when its ``(kind, name)`` is not among the baseline's. +* ``--effects`` is text embedded verbatim: what the regenerated ``spine-dump`` prints for the + previous current set. + +The classifier never decides "identical": ``follow_spine.py`` does that by generating the C++ +from both schemas. This tool describes what differs, kind by kind, and proposes the bump of the +highest kind found, in the order none < patch < minor < person (BUMP_TABLE; a kind it does not +list is "person"). A person writes the bump into ``project(VERSION)``. + +Exit codes: 0 written; 2 bad input (an unreadable file or a missing key). +""" + +import argparse +import json +import sys +from pathlib import Path + +BUMPS = ["none", "patch", "minor", "person"] + +# Every change kind the classifier reports, with the bump it proposes. A kind missing here is +# "person" (bump_of). test_classify.py checks that each kind classify() can report has a row. +BUMP_TABLE = { + # comments and descriptor doc strings only: the regenerated model may merge without a tag + "doc_changed": "none", + "module_changed": "none", + # additions (patch is provisional until the strict-mode decision, D6) + "field_added": "patch", + "enum_member_added": "patch", + "enum_added": "patch", + "scheme_table_added": "patch", + # removals and retypes: a minor release, the breaking change of a 0.x library + "field_removed": "minor", + "class_removed": "minor", + "enum_member_removed": "minor", + "enum_removed": "minor", + "scheme_table_removed": "minor", + "shape_retyped": "minor", + "element_retyped": "minor", + "enum_member_renumbered": "minor", + # a person decides + "default_changed": "person", + "units_changed": "person", + "flags_changed": "person", + "reference_changed": "person", + "field_reordered": "person", + "class_added": "person", + "enum_aliases_changed": "person", + "scheme_table_changed": "person", + "type_check_changed": "person", + "writer_format_changed": "person", + "stored_width_changed": "person", + "stored_type_changed": "person", + "stored_columns_changed": "person", +} + +TYPE_KEYS = ("storage", "element", "length", "enum") +CHANGED_KEYS = { # field key -> change kind + "default": "default_changed", + "units": "units_changed", + "flags": "flags_changed", + "reference": "reference_changed", + "reference_space": "reference_changed", + "doc": "doc_changed", +} +MODULE_KEYS = ("defined_in", "doc_source") +DOC_ONLY = {"doc_changed", "module_changed"} + + +class InputError(Exception): + """An input file cannot be read or lacks a key.""" + + +def bump_of(kind): + return BUMP_TABLE.get(kind, "person") + + +def kind_of_type(element): + if element in ("bool", "string"): + return element + if element.startswith(("int", "uint")): + return "integer" + if element.startswith("float"): + return "float" + return element + + +# --------------------------------------------------------------------------------------------- +# Schemas +# --------------------------------------------------------------------------------------------- + +def diff_class(old, new): + """The changes of one class present in both schemas.""" + old_fields = {f["name"]: (i, f) for i, f in enumerate(old["fields"])} + new_fields = {f["name"]: (i, f) for i, f in enumerate(new["fields"])} + out = {"added": [], "removed": [], "retyped": [], "reordered": False, "changed": []} + for name, (i, f) in new_fields.items(): + if name not in old_fields: + out["added"].append({"name": name, "position": i, "storage": f["storage"], + "element": f["element"], "length": f["length"]}) + out["removed"] = [name for name in old_fields if name not in new_fields] + common_old = [n for n in old_fields if n in new_fields] + common_new = [n for n in new_fields if n in old_fields] + out["reordered"] = common_old != common_new + for name in common_old: + a, b = old_fields[name][1], new_fields[name][1] + type_a = {k: a.get(k) for k in TYPE_KEYS} + type_b = {k: b.get(k) for k in TYPE_KEYS} + if type_a != type_b: + out["retyped"].append({ + "name": name, "from": type_a, "to": type_b, + "shape_changed": (a["storage"], a["length"]) != (b["storage"], b["length"]), + }) + for key in list(CHANGED_KEYS) + list(MODULE_KEYS): + if a.get(key) != b.get(key): + out["changed"].append({"name": name, "key": key, "from": a.get(key), + "to": b.get(key)}) + for key in ("module", "mro", "role", "doc"): + if old.get(key) != new.get(key): + out["changed"].append({"name": None, "key": key, "from": old.get(key), + "to": new.get(key)}) + return out + + +def class_kinds(diff): + """The change kinds of one class diff.""" + kinds = set() + if diff["added"]: + kinds.add("field_added") + if diff["removed"]: + kinds.add("field_removed") + if diff["reordered"]: + kinds.add("field_reordered") + for r in diff["retyped"]: + kinds.add("shape_retyped" if r["shape_changed"] else "element_retyped") + for c in diff["changed"]: + if c["name"] is None: + kinds.add("doc_changed" if c["key"] in ("role", "doc") else "module_changed") + elif c["key"] in MODULE_KEYS: + kinds.add("module_changed") + else: + kinds.add(CHANGED_KEYS[c["key"]]) + return kinds + + +def diff_enum(old, new): + members_a, members_b = dict(map(tuple, old["members"])), dict(map(tuple, new["members"])) + out = { + "added": [{"name": n, "value": v} for n, v in members_b.items() if n not in members_a], + "removed": [{"name": n, "value": v} for n, v in members_a.items() if n not in members_b], + "renumbered": [{"name": n, "from": members_a[n], "to": v} + for n, v in members_b.items() if n in members_a and members_a[n] != v], + "aliases": None, + "doc_changed": old.get("doc") != new.get("doc") or old.get("module") != new.get("module"), + } + if old.get("aliases", []) != new.get("aliases", []): + out["aliases"] = {"from": old.get("aliases", []), "to": new.get("aliases", [])} + return out + + +def diff_scheme_tables(old, new): + out = [] + for attr in sorted(set(old) | set(new)): + a, b = old.get(attr, {}), new.get(attr, {}) + for scheme in sorted(set(a) | set(b)): + if a.get(scheme) != b.get(scheme): + out.append({"attribute": attr, "scheme": scheme, "from": a.get(scheme), + "to": b.get(scheme)}) + return out + + +def diff_writer_format(old, new): + if old is None or new is None: + return [] if old == new else [{"key": "*", "from": old, "to": new}] + return [{"key": k, "from": old.get(k), "to": new.get(k)} + for k in sorted(set(old) | set(new)) if old.get(k) != new.get(k)] + + +def diff_type_check(old, new): + def entries(tc): + return {json.dumps(m, sort_keys=True) for m in (tc or {}).get("mismatches", [])} + a, b = entries(old), entries(new) + if a == b: + return None + return {"added": [json.loads(x) for x in sorted(b - a)], + "removed": [json.loads(x) for x in sorted(a - b)]} + + +# --------------------------------------------------------------------------------------------- +# Decoded columns +# --------------------------------------------------------------------------------------------- + +def read_columns(folder): + """{case: {product: {"class_name", "columns": {name: info}}}} of the *.expected.json files.""" + out = {} + for path in sorted(Path(folder).glob("*.expected.json")): + doc = load_json(path) + out[path.name[: -len(".expected.json")]] = { + key: {"class_name": p.get("class_name"), "columns": p.get("columns", {})} + for key, p in doc.get("products", {}).items() + } + return out + + +def diff_columns(old, new, retyped, added, removed): + """Column differences of the cases both folders hold. ``retyped``, ``added`` and ``removed`` + are sets of (class, field) the schema diff explains.""" + diffs, kinds = [], set() + for case in sorted(set(old) & set(new)): + for product in sorted(set(old[case]) | set(new[case])): + a, b = old[case].get(product), new[case].get(product) + if a is None or b is None: + diffs.append({"case": case, "product": product, + "change": "product added" if a is None else "product removed"}) + kinds.add("stored_columns_changed") + continue + cls = b["class_name"] or a["class_name"] + for name in sorted(set(a["columns"]) | set(b["columns"])): + ca, cb = a["columns"].get(name), b["columns"].get(name) + if ca is None or cb is None: + explained = (cls, name) in (added if ca is None else removed) + diffs.append({"case": case, "product": product, "column": name, + "change": "column added" if ca is None else "column removed", + "explained_by_schema": explained}) + if not explained: + kinds.add("stored_columns_changed") + continue + for key in ("type", "count", "variable", "enum"): + if ca.get(key) == cb.get(key): + continue + if key == "type" and kind_of_type(ca["type"]) == kind_of_type(cb["type"]): + change = "width" + else: + change = key + explained = (cls, name) in retyped + diffs.append({"case": case, "product": product, "column": name, + "change": change, "from": ca.get(key), "to": cb.get(key), + "explained_by_schema": explained}) + if key == "enum" or explained: + continue # enum labels follow the enum diff + kinds.add("stored_width_changed" if change == "width" + else "stored_type_changed") + return diffs, kinds + + +def default_only(new_columns, new_schema, fields): + """Of ``fields`` ((class, field) pairs), those that no row of any case holds at a value other + than the schema's default: the fixtures test them only at their default.""" + varied = set() + defaults = {(c, f["name"]): f["default"] for c, d in new_schema["classes"].items() + for f in d["fields"]} + for path in sorted(Path(new_columns).glob("*.expected.json")): + doc = load_json(path) + for product in doc.get("products", {}).values(): + if product.get("kind") != "objects": + continue + cls = product.get("class_name") + for rows in product.get("events", []): + for row in rows: + for key in fields: + if key[0] == cls and key[1] in row and row[key[1]] != defaults.get(key): + varied.add(key) + return [f"{c}.{n}" for (c, n) in fields if (c, n) not in varied] + + +def written_classes(columns): + return {p["class_name"] for case in columns.values() for p in case.values() if p["class_name"]} + + +# --------------------------------------------------------------------------------------------- +# The classification +# --------------------------------------------------------------------------------------------- + +def load_json(path): + try: + return json.loads(Path(path).read_text()) + except (OSError, ValueError) as e: + raise InputError(f"{path}: {e}") from e + + +def warning_key(w): + return (w.get("kind"), w.get("name")) + + +def classify(baseline, new, old_columns=None, new_columns=None, warnings=(), + baseline_warnings=(), effects=None, baseline_path=None, new_path=None): + """The classification of the change from schema ``baseline`` to ``new`` (dicts).""" + for doc, label in ((baseline, "baseline"), (new, "new")): + for key in ("spine_version", "classes", "enums", "scheme_tables"): + if key not in doc: + raise InputError(f"{label} schema has no '{key}'") + kinds = set() + out = { + "baseline": {"path": baseline_path, "spine_version": baseline["spine_version"], + "spine_sha": baseline.get("spine_sha")}, + "new": {"path": new_path, "spine_version": new["spine_version"], + "spine_sha": new.get("spine_sha")}, + } + + # classes and fields + old_classes, new_classes = baseline["classes"], new["classes"] + out["classes_added"] = [c for c in new_classes if c not in old_classes] + out["classes_removed"] = [c for c in old_classes if c not in new_classes] + if out["classes_added"]: + kinds.add("class_added") + if out["classes_removed"]: + kinds.add("class_removed") + out["classes"] = {} + added, removed, retyped = set(), set(), set() + for name in old_classes: + if name not in new_classes: + continue + diff = diff_class(old_classes[name], new_classes[name]) + kinds |= class_kinds(diff) + added |= {(name, f["name"]) for f in diff["added"]} + removed |= {(name, f) for f in diff["removed"]} + retyped |= {(name, r["name"]) for r in diff["retyped"]} + if any(diff[k] for k in ("added", "removed", "retyped", "reordered", "changed")): + out["classes"][name] = diff + for name in out["classes_added"]: + added |= {(name, f["name"]) for f in new_classes[name]["fields"]} + for name in out["classes_removed"]: + removed |= {(name, f["name"]) for f in old_classes[name]["fields"]} + + # enums and scheme tables + old_enums, new_enums = baseline["enums"], new["enums"] + out["enums_added"] = [e for e in new_enums if e not in old_enums] + out["enums_removed"] = [e for e in old_enums if e not in new_enums] + if out["enums_added"]: + kinds.add("enum_added") + if out["enums_removed"]: + kinds.add("enum_removed") + out["enums"] = {} + for name in old_enums: + if name not in new_enums: + continue + diff = diff_enum(old_enums[name], new_enums[name]) + for key, kind in (("added", "enum_member_added"), ("removed", "enum_member_removed"), + ("renumbered", "enum_member_renumbered"), + ("aliases", "enum_aliases_changed"), ("doc_changed", "doc_changed")): + if diff[key]: + kinds.add(kind) + if any(diff.values()): + out["enums"][name] = diff + out["scheme_tables_changed"] = diff_scheme_tables(baseline["scheme_tables"], + new["scheme_tables"]) + for change in out["scheme_tables_changed"]: + if change["from"] is None: + kinds.add("scheme_table_added") + elif change["to"] is None: + kinds.add("scheme_table_removed") + else: + kinds.add("scheme_table_changed") + + # what the dump records beyond the model + out["type_check_changed"] = diff_type_check(baseline.get("type_check"), new.get("type_check")) + if out["type_check_changed"]: + kinds.add("type_check_changed") + out["writer_format_changed"] = diff_writer_format(baseline.get("writer_format"), + new.get("writer_format")) + if out["writer_format_changed"]: + kinds.add("writer_format_changed") + + # decoded columns + out["column_diffs"] = [] + out["default_only_fields"] = [] + out["unwritten_classes"] = [] + if old_columns and new_columns: + out["column_diffs"], column_kinds = diff_columns( + read_columns(old_columns), read_columns(new_columns), retyped, added, removed) + kinds |= column_kinds + if new_columns: + out["default_only_fields"] = default_only(new_columns, new, sorted(added)) + written = written_classes(read_columns(new_columns)) + out["unwritten_classes"] = [c for c in new_classes if c not in written] + + # warnings and substitutions + new_warnings = list(new.get("warnings", [])) + list(warnings) + known = {warning_key(w) for w in list(baseline.get("warnings", [])) + list(baseline_warnings)} + out["warnings_new"], seen = [], set() + for w in new_warnings: + key = warning_key(w) + if key not in known and key not in seen and w.get("kind") != "substitution": + out["warnings_new"].append(w) + seen.add(key) + out["substitutions"] = [w for w in warnings if w.get("kind") == "substitution"] + out["effects_on_previous_files"] = effects + + # the bump + out["change_kinds"] = sorted(kinds) + out["module_only"] = bool(kinds) and kinds <= DOC_ONLY + bump = max((bump_of(k) for k in kinds), key=BUMPS.index, default="none") + out["proposed_bump"] = bump + out["reasons"] = [{"kind": k, "bump": bump_of(k)} + for k in sorted(kinds, key=lambda k: (-BUMPS.index(bump_of(k)), k))] + out["needs_attention"] = bool(out["classes_added"] or out["warnings_new"] + or out["substitutions"]) + out["follow_up"] = follow_up(out, added) + return out + + +def follow_up(out, added): + items = [] + for field in out["default_only_fields"]: + items.append(f"the fixtures hold {field} only at its default: set it in " + "tools/fixtures/content.py") + for cls in out["unwritten_classes"]: + items.append(f"no current fixture writes {cls}: its fields are not checked against a file") + for w in out["warnings_new"]: + items.append(f"new warning {w.get('kind')} {w.get('name')}: {w.get('detail', '')}".rstrip()) + for s in out["substitutions"]: + items.append(f"substitution {s.get('name')}: {s.get('detail', '')}".rstrip()) + if added: + items.append("a CAFMaker that starts reading an added field needs a row in " + "tests/test_contracts.cpp (tools/release/scan_consumer_fields.py)") + items.append("after an added field, ReaderOptions::strict refuses files of the previous " + "SPINE release (decision D6 is open)") + if out["proposed_bump"] == "patch": + items.append("the patch bump for additions is provisional until decision D6") + if out["classes_added"]: + items.append("a new class needs a product name in the Reader and, if a CAFMaker reads " + "it, contract rows: " + ", ".join(out["classes_added"])) + return items + + +# --------------------------------------------------------------------------------------------- +# Markdown +# --------------------------------------------------------------------------------------------- + +def render_markdown(c): + b, n = c["baseline"], c["new"] + lines = [f"# SPINE {b['spine_version']} -> {n['spine_version']}", ""] + lines.append(f"- baseline: `{b['path']}` (SPINE {b['spine_version']}, `{b['spine_sha']}`)") + lines.append(f"- new: `{n['path']}` (SPINE {n['spine_version']}, `{n['spine_sha']}`)") + lines.append(f"- proposed bump: **{c['proposed_bump']}**" + + ("" if c["proposed_bump"] != "person" else " (a person decides)")) + lines.append(f"- needs attention: {'yes' if c['needs_attention'] else 'no'}") + lines.append("") + lines.append("## Why") + lines.append("") + if c["reasons"]: + lines += [f"- `{r['kind']}`: {r['bump']}" for r in c["reasons"]] + else: + lines.append("- no change kind found") + lines.append("") + + lines.append("## Classes and fields") + lines.append("") + for key, label in (("classes_added", "classes added"), ("classes_removed", "classes removed")): + if c[key]: + lines.append(f"- {label}: " + ", ".join(f"`{x}`" for x in c[key])) + for cls, d in c["classes"].items(): + lines.append(f"- `{cls}`") + for f in d["added"]: + length = f"[{f['length']}]" if f["length"] else "" + lines.append(f" - added `{f['name']}` at {f['position']}: {f['storage']} " + f"{f['element']}{length}") + for f in d["removed"]: + lines.append(f" - removed `{f}`") + for r in d["retyped"]: + lines.append(f" - retyped `{r['name']}`: {r['from']} -> {r['to']}" + + (" (shape)" if r["shape_changed"] else "")) + if d["reordered"]: + lines.append(" - field order changed") + for ch in d["changed"]: + where = f"`{ch['name']}` " if ch["name"] else "class " + lines.append(f" - {where}{ch['key']}: {json.dumps(ch['from'])} -> " + f"{json.dumps(ch['to'])}") + if not (c["classes"] or c["classes_added"] or c["classes_removed"]): + lines.append("- none") + lines.append("") + + lines.append("## Enums") + lines.append("") + for key, label in (("enums_added", "enums added"), ("enums_removed", "enums removed")): + if c[key]: + lines.append(f"- {label}: " + ", ".join(f"`{x}`" for x in c[key])) + for name, d in c["enums"].items(): + parts = [] + if d["added"]: + parts.append("added " + ", ".join(f"{m['name']}={m['value']}" for m in d["added"])) + if d["removed"]: + parts.append("removed " + ", ".join(m["name"] for m in d["removed"])) + if d["renumbered"]: + renumbered = [f"{m['name']} {m['from']}->{m['to']}" for m in d["renumbered"]] + parts.append("renumbered " + ", ".join(renumbered)) + if d["aliases"]: + parts.append(f"aliases {d['aliases']['from']} -> {d['aliases']['to']}") + if d["doc_changed"]: + parts.append("doc or module changed") + lines.append(f"- `{name}`: " + "; ".join(parts)) + for t in c["scheme_tables_changed"]: + lines.append(f"- scheme table {t['attribute']}[{t['scheme']}]: {t['from']} -> {t['to']}") + if not (c["enums"] or c["enums_added"] or c["enums_removed"] or c["scheme_tables_changed"]): + lines.append("- none") + lines.append("") + + lines.append("## Writer and type check") + lines.append("") + if c["writer_format_changed"]: + lines += [f"- writer_format {w['key']}: {json.dumps(w['from'])} -> {json.dumps(w['to'])}" + for w in c["writer_format_changed"]] + else: + lines.append("- writer_format unchanged") + if c["type_check_changed"]: + tc = c["type_check_changed"] + lines.append(f"- type_check: {len(tc['added'])} mismatch(es) added, " + f"{len(tc['removed'])} removed") + lines += [f" - {m.get('class')}.{m.get('field')}: {m.get('detail')}" for m in tc["added"]] + else: + lines.append("- type_check unchanged") + lines.append("") + + lines.append("## Decoded columns of the current set") + lines.append("") + explained = [d for d in c["column_diffs"] if d.get("explained_by_schema")] + for d in c["column_diffs"]: + if d.get("explained_by_schema"): + continue + what = d["change"] + if "from" in d: + what += f" {d['from']} -> {d['to']}" + col = f".{d['column']}" if "column" in d else "" + lines.append(f"- {d['case']} {d['product']}{col}: {what}") + if explained: + columns = sorted({d["column"] for d in explained}) + lines.append(f"- {len(explained)} difference(s) the schema changes above explain " + f"({', '.join(columns)}; all listed in classification.json)") + if not c["column_diffs"]: + lines.append("- no difference (or no column folders given)") + lines.append("") + + if c["follow_up"]: + lines.append("## Follow-up") + lines.append("") + lines += [f"- [ ] {item}" for item in c["follow_up"]] + lines.append("") + + if c["effects_on_previous_files"]: + lines.append("## The regenerated spine-dump on the previous current set") + lines.append("") + lines.append("```text") + lines.append(c["effects_on_previous_files"].rstrip("\n")) + lines.append("```") + lines.append("") + + lines.append("## Checklist (tools/gen/README.md, What to check afterwards)") + lines.append("") + lines += [ + "- [ ] the type check printed by `dump_schema.py` says `0 mismatch(es)`", + "- [ ] `git diff schema/`: every difference above is expected", + "- [ ] `python3 tools/gen/check_names.py` passes", + "- [ ] build and ctest; if `spine_reader_contracts` fails, talk to the CAFMaker " + "maintainers before editing `tests/test_contracts.cpp`", + "- [ ] gcc 12, C++17 in the SL7 container (`ci/sl7-ups.sh`)", + "- [ ] docs and units (`doc_source` in the JSON)", + f"- [ ] write the bump (proposed: {c['proposed_bump']}) into `project(VERSION)`", + ] + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------------------------- +# Command line +# --------------------------------------------------------------------------------------------- + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + p.add_argument("baseline") + p.add_argument("new") + p.add_argument("--old-columns") + p.add_argument("--new-columns") + p.add_argument("--warnings") + p.add_argument("--baseline-warnings") + p.add_argument("--effects") + p.add_argument("--json", required=True) + p.add_argument("--md", required=True) + args = p.parse_args(argv) + if bool(args.old_columns) != bool(args.new_columns): + p.error("--old-columns and --new-columns go together") + try: + for folder in (args.old_columns, args.new_columns): + if folder and not Path(folder).is_dir(): + raise InputError(f"no folder {folder}") + warnings = load_json(args.warnings) if args.warnings else [] + baseline_warnings = load_json(args.baseline_warnings) if args.baseline_warnings else [] + if not isinstance(warnings, list) or not isinstance(baseline_warnings, list): + raise InputError("--warnings and --baseline-warnings hold JSON lists") + effects = None + if args.effects: + try: + effects = Path(args.effects).read_text() + except OSError as e: + raise InputError(f"{args.effects}: {e}") from e + result = classify(load_json(args.baseline), load_json(args.new), args.old_columns, + args.new_columns, warnings, baseline_warnings, effects, + args.baseline, args.new) + except (InputError, KeyError, TypeError) as e: + print(f"classify: {type(e).__name__}: {e}", file=sys.stderr) + return 2 + Path(args.json).write_text(json.dumps(result, indent=1) + "\n") + Path(args.md).write_text(render_markdown(result)) + print(f"classify: SPINE {result['baseline']['spine_version']} -> " + f"{result['new']['spine_version']}: proposed bump {result['proposed_bump']}" + f" ({', '.join(result['change_kinds']) or 'no change kind'})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/release/follow_spine.py b/tools/release/follow_spine.py new file mode 100755 index 0000000..da03f69 --- /dev/null +++ b/tools/release/follow_spine.py @@ -0,0 +1,755 @@ +#!/usr/bin/env python3 +"""Follow one SPINE tag: dump its schema, compare it with the model, write and check its +fixtures, classify the change, then build and test spine-reader with it. + + follow_spine.py TAG [--baseline SCHEMA] [--apply] [--repo-root DIR] [--spine-repo URL] + [--workdir DIR] [--artifacts-dir DIR] [--jobs N] + +The stages (tools/release/README.md has the whole story): + +* s0_checks: the tag form, ``--apply``'s conditions and the environment; no network. +* s1_tag: the tag's commit (``git ls-remote``) and a checkout of it (tools/fixtures). +* s2_dump: ``tools/gen/dump_schema.py`` on that checkout. +* s3_compare: identical (the generated C++ is the same once the version is normalised, and + ``type_check`` and ``writer_format`` are unchanged) or changed. +* s4_fixtures: ``tools/fixtures/make_fixtures.py`` writes the tag's current set with its own + writer, runs every cross-check and reads the files back with the tag's own reader. +* s5_classify: ``classify.py``. A replay (a ``--baseline`` other than the committed schema, or a + tag older than the model's) stops here. +* s6_build and s6_test: a ``git archive HEAD`` export in the artifacts folder, with the new + current set (and, if changed, the new schema and the regenerated model), built and tested. + Then the regenerated ``spine-dump`` reads HEAD's previous ``current_v1``/``current_v2``, and + ``classify.py`` runs again with that text. +* s8_apply (``--apply``, after ``changed`` only): what s6 overlaid is written into the + repository's working tree. Nothing is committed and ``project(VERSION)`` is not edited. + +Exit codes: 0 identical; 10 changed (``contracts_failed`` says whether +``spine_reader_contracts`` failed); 20 not followed (a pre-release tag, or one below v1.2.1); 30 +a failure (``failed_stage`` names the stage); 2 usage error. Every outcome but a usage error +writes ``status.json`` in the artifacts folder. + +The caller sets up the LCG view: Python 3.10 or newer with numpy, h5py, yaml and scipy, git, +CMake 3.21 or newer and a C++ compiler, plus network access to the SPINE repository. +""" + +import argparse +import importlib.util +import io +import json +import os +import platform +import re +import shlex +import shutil +import subprocess +import sys +import tarfile +import tempfile +import time +import traceback +import xml.etree.ElementTree as ElementTree +from pathlib import Path + +HERE = Path(__file__).resolve().parent +DEFAULT_REPO = HERE.parents[1] +sys.path.insert(0, str(DEFAULT_REPO / "tools" / "fixtures")) +import spine_checkout # noqa: E402 (imports neither h5py nor numpy) + +STATUS_FORMAT_VERSION = 1 +DEFAULT_SPINE_REPO = "https://github.com/DeepLearnPhysics/spine" +DEFAULT_WORKDIR = os.path.join(tempfile.gettempdir(), "spine-reader-fixtures") +TAG_RE = re.compile(r"^v(\d+)\.(\d+)\.(\d+)$") +PRERELEASE_RE = re.compile(r"^v\d+\.\d+\.\d+\S+$") +OLDEST_FOLLOWED = (1, 2, 1) +MIN_CMAKE = (3, 21) +MODULES = ("numpy", "h5py", "yaml", "scipy") + +STAGES = ["s0_checks", "s1_tag", "s2_dump", "s3_compare", "s4_fixtures", "s5_classify", + "s6_build", "s6_test", "s8_apply"] +FAILURES = ["environment_error", "tag_moved", "dump_failed", "version_mismatch", + "generate_failed", "fixture_tooling_failed", "fixture_check_failed", "build_failed", + "test_failed", "internal_error"] +EXIT_CODES = {"identical": 0, "changed": 10, "not_followed": 20, "usage_error": 2, + **{f: 30 for f in FAILURES}} +CONTRACTS_TEST = "spine_reader_contracts" + + +class UsageError(Exception): + """Bad arguments: exit 2, and no status.json.""" + + +class StageFailure(Exception): + """A stage ended in a failure outcome.""" + + def __init__(self, outcome, detail, reason=None): + super().__init__(outcome, detail, reason) + self.outcome, self.detail, self.reason = outcome, detail, reason + + def __str__(self): + return f"{self.outcome}: {self.detail}" + + +# --------------------------------------------------------------------------------------------- +# Pure functions (the unit tests call these; they need neither git, the network, nor SPINE) +# --------------------------------------------------------------------------------------------- + +def parse_tag(tag): + """(major, minor, patch) of a final tag vX.Y.Z, or None.""" + m = TAG_RE.match(tag) + return tuple(int(x) for x in m.groups()) if m else None + + +def version_tuple(text): + return tuple(int(x) for x in text.split(".")) + + +def committed_schema(repo): + """The one schema/spine-*.json of ``repo``.""" + found = sorted(Path(repo, "schema").glob("spine-*.json")) + if len(found) != 1: + raise UsageError(f"{repo}/schema holds {len(found)} spine-*.json files, not one") + return found[0] + + +def git(repo, *args, check=True): + proc = subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True) + if check and proc.returncode != 0: + raise UsageError(f"git {' '.join(args)} in {repo} failed: {proc.stderr.strip()}") + return proc.stdout + + +def check_usage(args, repo): + """The checks of s0 that need no network. Raises UsageError; returns a dict with the tag's + version, the baseline, the model's version, ``replay`` and ``not_followed`` (a reason or + None).""" + out = {"tag": args.tag, "version": None, "not_followed": None} + version = parse_tag(args.tag) + if version is None: + if PRERELEASE_RE.match(args.tag): + out["not_followed"] = f"{args.tag} is a pre-release tag" + return out + raise UsageError(f"{args.tag!r} is not a tag of the form vX.Y.Z") + out["version"] = version + if version < OLDEST_FOLLOWED: + out["not_followed"] = (f"{args.tag} is older than v{'.'.join(map(str, OLDEST_FOLLOWED))}, " + "which the dumper cannot import") + return out + committed = committed_schema(repo) + try: + model = json.loads(committed.read_text()) + out["model_version"] = version_tuple(model["spine_version"]) + except (ValueError, KeyError, AttributeError) as e: + raise UsageError(f"{committed}: not a schema dump ({type(e).__name__}: {e})") from e + baseline = Path(args.baseline).resolve() if args.baseline else committed.resolve() + if not baseline.is_file(): + raise UsageError(f"--baseline: no file {args.baseline}") + try: + out["baseline"] = json.loads(baseline.read_text()) + except ValueError as e: + raise UsageError(f"--baseline {baseline}: not JSON ({e})") from e + if "spine_version" not in out["baseline"]: + raise UsageError(f"--baseline {baseline}: no spine_version") + out["baseline_path"] = baseline + out["replay"] = baseline != committed.resolve() or version < out["model_version"] + if args.apply: + if out["replay"]: + raise UsageError("--apply follows the model forward: it takes no --baseline and no tag " + "older than the model's") + if version <= out["model_version"]: + raise UsageError(f"--apply needs a tag newer than the model's " + f"(v{model['spine_version']})") + if git(repo, "status", "--porcelain").strip(): + raise UsageError(f"--apply needs a clean working tree in {repo} (git status shows " + "changes or untracked files)") + return out + + +def outcome_for(stages): + """(outcome, exit code, failed stage, contracts_failed) from the stage results. + + ``stages`` maps a stage name to a dict with ``status`` (``passed``, ``failed`` or + ``skipped``) and, for a failed stage, ``outcome``. ``s0_checks`` may carry + ``not_followed``; ``s3_compare`` carries ``comparison`` (identical or changed); + ``s6_test`` carries ``failed_tests``. + """ + comparison = stages.get("s3_compare", {}).get("comparison") + for name in STAGES: + result = stages.get(name) + if result is None: + continue + if result.get("not_followed"): + return "not_followed", EXIT_CODES["not_followed"], None, False + if result.get("status") != "failed": + continue + if name == "s6_test": + failed = set(result.get("failed_tests", [])) + if failed == {CONTRACTS_TEST} and comparison == "changed": + return "changed", EXIT_CODES["changed"], None, True + return "test_failed", EXIT_CODES["test_failed"], name, False + outcome = result.get("outcome", "internal_error") + if outcome not in EXIT_CODES: + outcome = "internal_error" + return outcome, EXIT_CODES[outcome], name, False + if comparison in ("identical", "changed"): + return comparison, EXIT_CODES[comparison], None, False + return "internal_error", EXIT_CODES["internal_error"], None, False + + +def load_generator(repo): + """tools/gen/generate.py of ``repo``, imported as a module.""" + path = Path(repo, "tools", "gen", "generate.py") + spec = importlib.util.spec_from_file_location("spine_reader_generate", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def compare_schemas(baseline, new, generator=None): + """s3: ``identical`` when the generated files of both schemas are the same once the new + schema's spine_version is set to the baseline's, and type_check and writer_format are + unchanged; otherwise ``changed``. ``generate_failed`` when the generator refuses either + schema (with ``error``).""" + generator = generator or load_generator(DEFAULT_REPO) + out = {"result": None, "files": [], "type_check_changed": False, + "writer_format_changed": False, "error": None} + normalised = dict(new, spine_version=baseline["spine_version"]) + try: + old_files = generator.generate(baseline) + new_files = generator.generate(normalised) + except generator.GenerationError as e: + out["result"], out["error"] = "generate_failed", str(e) + return out + out["files"] = sorted(p for p in set(old_files) | set(new_files) + if old_files.get(p) != new_files.get(p)) + out["type_check_changed"] = baseline.get("type_check") != new.get("type_check") + out["writer_format_changed"] = baseline.get("writer_format") != new.get("writer_format") + changed = out["files"] or out["type_check_changed"] or out["writer_format_changed"] + out["result"] = "changed" if changed else "identical" + return out + + +def failed_tests(junit_path): + """Names of the failed tests of a ctest JUnit file.""" + root = ElementTree.parse(junit_path).getroot() + names = [] + for case in root.iter("testcase"): + if case.get("status") == "fail" or case.find("failure") is not None: + names.append(case.get("name")) + return names + + +# --------------------------------------------------------------------------------------------- +# The run +# --------------------------------------------------------------------------------------------- + +class Run: + def __init__(self, args, repo, artifacts): + self.args, self.repo, self.artifacts = args, repo, artifacts + self.logs = artifacts / "logs" + self.logs.mkdir(parents=True, exist_ok=True) + self.stages = {} + self.info = {"warnings": [], "substitutions": []} + + def log_path(self, stage): + return self.logs / f"{stage}.log" + + def say(self, stage, text): + with open(self.log_path(stage), "a") as fh: + fh.write(text.rstrip("\n") + "\n") + + def run(self, stage, cmd, env=None, cwd=None, stdout_file=None): + """Run ``cmd``, appending its output to the stage's log. Returns (code, stdout, stderr).""" + self.say(stage, "$ " + shlex.join(str(c) for c in cmd)) + proc = subprocess.run([str(c) for c in cmd], capture_output=True, env=env, cwd=cwd) + if stdout_file is not None: + Path(stdout_file).write_bytes(proc.stdout) + stdout = "" + else: + stdout = proc.stdout.decode(errors="replace") + stderr = proc.stderr.decode(errors="replace") + with open(self.log_path(stage), "a") as fh: + if stdout: + fh.write(stdout if stdout.endswith("\n") else stdout + "\n") + if stderr: + fh.write(stderr if stderr.endswith("\n") else stderr + "\n") + fh.write(f"[exit {proc.returncode}]\n") + return proc.returncode, stdout, stderr + + def stage(self, name, function, *args): + """Run one stage; record its status, duration and log. Returns the function's value, or + None when the stage failed.""" + start = time.monotonic() + print(f"follow_spine: {name} ...", flush=True) + result = {"status": "passed", "log": str(self.log_path(name).relative_to(self.artifacts))} + self.log_path(name).touch() + value = None + try: + value = function(*args) + if isinstance(value, dict): + result.update({k: v for k, v in value.items() if k in + ("comparison", "failed_tests", "not_followed", "detail")}) + except UsageError: + raise + except StageFailure as e: + result.update(status="failed", outcome=e.outcome, detail=e.detail) + if e.reason: + result["reason"] = e.reason + if isinstance(e, TestsFailed): + result["failed_tests"] = e.names + self.say(name, f"FAILED: {e}") + except Exception as e: # anything unexpected is internal_error + result.update(status="failed", outcome="internal_error", + detail=f"{type(e).__name__}: {e}") + self.say(name, traceback.format_exc()) + result["duration_s"] = round(time.monotonic() - start, 3) + self.stages[name] = result + if result["status"] == "failed": + print(f"follow_spine: {name} failed: {result['outcome']}: {result['detail']}", + flush=True) + return None + return value + + # --- s0 ------------------------------------------------------------------------------------ + + def s0_environment(self): + problems = [] + if sys.version_info < (3, 10): + problems.append(f"Python {platform.python_version()}: SPINE needs 3.10 or newer") + for module in MODULES: + if importlib.util.find_spec(module) is None: + problems.append(f"no Python module {module}") + for tool in ("git", "cmake", "ctest"): + if shutil.which(tool) is None: + problems.append(f"no {tool} on PATH") + if shutil.which("cmake"): + text = subprocess.run(["cmake", "--version"], capture_output=True, text=True).stdout + m = re.search(r"(\d+)\.(\d+)", text) + if not m or (int(m.group(1)), int(m.group(2))) < MIN_CMAKE: + problems.append(f"CMake {m.group(0) if m else '?'}: 3.21 or newer is needed " + "(ctest --output-junit)") + cxx = os.environ.get("CXX") or shutil.which("c++") or shutil.which("g++") + if not cxx or not shutil.which(cxx.split()[0]): + problems.append("no C++ compiler (CXX, c++ or g++)") + self.info["tools"] = self.tool_versions(cxx) + for p in problems: + self.say("s0_checks", p) + if problems: + raise StageFailure("environment_error", "; ".join(problems)) + return {} + + def tool_versions(self, cxx): + def first_line(cmd): + try: + text = subprocess.run(cmd, capture_output=True, text=True, timeout=60).stdout + return text.splitlines()[0].strip() if text.strip() else None + except (OSError, subprocess.SubprocessError): + return None + probe = ("import json, numpy, h5py; print(json.dumps({'numpy': numpy.__version__, " + "'h5py': h5py.__version__, 'hdf5': h5py.version.hdf5_version}))") + versions = {"python": platform.python_version()} + try: + proc = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True, + timeout=120) + versions.update(json.loads(proc.stdout) if proc.returncode == 0 else {}) + except (OSError, ValueError, subprocess.SubprocessError): + pass + versions["cmake"] = first_line(["cmake", "--version"]) if shutil.which("cmake") else None + versions["compiler"] = first_line(shlex.split(cxx) + ["--version"]) if cxx else None + versions["git"] = first_line(["git", "--version"]) if shutil.which("git") else None + return versions + + # --- s1 ------------------------------------------------------------------------------------ + + def s1_tag(self, usage): + tag, baseline = self.args.tag, usage["baseline"] + own = tag == "v" + baseline["spine_version"] + try: + sha = spine_checkout.resolve_tag(self.args.spine_repo, tag) + except spine_checkout.LsRemoteError as e: + raise StageFailure("environment_error", str(e)) from e + except spine_checkout.TagNotFound as e: + if own: + raise StageFailure("tag_moved", f"{tag}, the baseline's own tag, is gone from " + f"{self.args.spine_repo}") from e + raise UsageError(str(e)) from e + self.say("s1_tag", f"{tag} -> {sha}") + recorded = baseline.get("spine_sha") + if own and recorded and sha != recorded: + raise StageFailure("tag_moved", f"{tag} points to {sha}, but the baseline was dumped " + f"from {recorded}") + if own and not recorded: + self.say("s1_tag", "the baseline records no spine_sha: the tag is not checked") + try: + src = spine_checkout.ensure_clone(tag, sha, self.args.spine_repo, self.args.workdir) + except spine_checkout.CheckoutError as e: + raise StageFailure("environment_error", str(e)) from e + self.say("s1_tag", f"checkout: {src}") + return {"sha": sha, "src": src} + + # --- s2 ------------------------------------------------------------------------------------ + + def s2_dump(self, tag, sha, src): + out_dir = self.artifacts / "schema" + shutil.rmtree(out_dir, ignore_errors=True) + env = dict(os.environ, PYTHONHASHSEED="0") + code, _, stderr = self.run("s2_dump", [ + sys.executable, self.repo / "tools/gen/dump_schema.py", "--spine-src", src, + "--spine-sha", sha, "--out-dir", out_dir], env=env) + if code != 0: + lines = [x for x in stderr.splitlines() if x.startswith("dump_schema: ")] + m = re.match(r"^dump_schema: (\w+):", lines[-1]) if lines else None + detail = lines[-1][len("dump_schema: "):] if lines else stderr.strip()[-500:] + raise StageFailure("dump_failed", detail or f"exit {code}", + reason=m.group(1) if m else f"exit {code}") + found = sorted(out_dir.glob("spine-*.json")) + if len(found) != 1: + raise StageFailure("internal_error", f"the dump wrote {len(found)} schema files") + new = json.loads(found[0].read_text()) + if tag != "v" + new["spine_version"]: + raise StageFailure("version_mismatch", f"{tag} says spine.__version__ is " + f"{new['spine_version']}") + self.info["schema_warnings"] = new.get("warnings", []) + return {"path": found[0], "schema": new, "spine_version": new["spine_version"]} + + # --- s3 ------------------------------------------------------------------------------------ + + def s3_compare(self, baseline, new): + result = compare_schemas(baseline, new, load_generator(self.repo)) + self.say("s3_compare", json.dumps(result, indent=1)) + if result["result"] == "generate_failed": + raise StageFailure("generate_failed", result["error"]) + self.info["comparison"] = result + return {"comparison": result["result"], **result} + + # --- s4 ------------------------------------------------------------------------------------ + + def head_files(self): + """HEAD's manifest and current_*.expected.json, in /head.""" + head = self.artifacts / "head" + (head / "fixtures").mkdir(parents=True, exist_ok=True) + names = git(self.repo, "ls-tree", "--name-only", "HEAD", "tests/data/fixtures/").split() + for name in names: + base = os.path.basename(name) + if base == "manifest.json" or (base.startswith("current_") + and base.endswith(".expected.json")): + data = subprocess.run(["git", "-C", str(self.repo), "show", f"HEAD:{name}"], + capture_output=True, check=True).stdout + (head / "fixtures" / base).write_bytes(data) + return head / "fixtures" + + def s4_fixtures(self, tag, sha, schema_path): + head = self.head_files() + out = self.artifacts / "fixtures" + shutil.rmtree(out, ignore_errors=True) + code, stdout, _ = self.run("s4_fixtures", [ + sys.executable, self.repo / "tools/fixtures/make_fixtures.py", "--tag", tag, + "--sha", sha, "--schema", schema_path, "--out", out, + "--manifest-base", head / "manifest.json", "--spine-repo", self.args.spine_repo, + "--workdir", self.args.workdir, "--artifacts-dir", self.artifacts / "fixture-run"]) + summary = {} + for line in reversed(stdout.splitlines()): + if line.startswith("{"): + try: + summary = json.loads(line) + except ValueError: + pass + break + self.info["warnings"] = summary.get("warnings", []) + self.info["substitutions"] = summary.get("substitutions", []) + if code == 3: + raise StageFailure("fixture_tooling_failed", "a worker could not drive SPINE's API") + if code == 1: + raise StageFailure("fixture_check_failed", + f"{summary.get('failures', '?')} check(s) failed") + if code != 0: + raise StageFailure("internal_error", f"make_fixtures.py exited {code}") + return {"fixtures": out, "head": head} + + # --- s5 ------------------------------------------------------------------------------------ + + def classify(self, stage, baseline_path, new_path, old_columns, new_columns, effects=None): + head_manifest = json.loads((self.artifacts / "head/fixtures/manifest.json").read_text()) + warnings = self.artifacts / "warnings.json" + warnings.write_text(json.dumps(self.info["warnings"] + self.info["substitutions"], + indent=1) + "\n") + baseline_warnings = self.artifacts / "baseline-warnings.json" + baseline_warnings.write_text(json.dumps(head_manifest.get("warnings", []) + + head_manifest.get("substitutions", []), + indent=1) + "\n") + cmd = [sys.executable, self.repo / "tools/release/classify.py", baseline_path, new_path, + "--warnings", warnings, "--baseline-warnings", baseline_warnings, + "--json", self.artifacts / "classification.json", + "--md", self.artifacts / "classification.md"] + if old_columns is not None: + cmd += ["--old-columns", old_columns, "--new-columns", new_columns] + if effects is not None: + cmd += ["--effects", effects] + code, _, stderr = self.run(stage, cmd) + if code != 0: + raise StageFailure("internal_error", f"classify.py exited {code}: {stderr.strip()}") + self.info["classification"] = json.loads( + (self.artifacts / "classification.json").read_text()) + return {} + + # --- s6 ------------------------------------------------------------------------------------ + + def s6_build(self, changed, new_schema_path, fixtures): + export = self.artifacts / "export" + shutil.rmtree(export, ignore_errors=True) + export.mkdir(parents=True) + archive = subprocess.run(["git", "-C", str(self.repo), "archive", "--format=tar", "HEAD"], + capture_output=True, check=True).stdout + safe = {"filter": "data"} if hasattr(tarfile, "data_filter") else {} + with tarfile.open(fileobj=io.BytesIO(archive)) as tar: + tar.extractall(export, **safe) + self.say("s6_build", f"exported HEAD ({git(self.repo, 'rev-parse', 'HEAD').strip()}) " + f"to {export}") + target = export / "tests/data/fixtures" + for old in target.glob("current_*"): + old.unlink() + for new in sorted(Path(fixtures).iterdir()): + if new.name == "manifest.json" or new.name.startswith("current_"): + shutil.copy2(new, target / new.name) + self.say("s6_build", "overlaid the new current set and the merged manifest") + if changed: + for old in (export / "schema").glob("spine-*.json"): + old.unlink() + shutil.copy2(new_schema_path, export / "schema" / Path(new_schema_path).name) + for cmd in ([sys.executable, export / "tools/gen/generate.py", "--root", export], + [sys.executable, export / "tools/gen/check_names.py", "--root", export], + [sys.executable, export / "tools/fixtures/make_member_list.py"]): + code, _, stderr = self.run("s6_build", cmd) + if code != 0: + raise StageFailure("generate_failed", f"{Path(cmd[1]).name} exited {code}: " + f"{stderr.strip()[-500:]}") + build = self.artifacts / "build" + shutil.rmtree(build, ignore_errors=True) + for cmd in (["cmake", "-S", export, "-B", build, "-DSPINE_READER_REQUIRE_PYTHON=ON", + f"-DPython3_EXECUTABLE={sys.executable}"], + ["cmake", "--build", build, "-j", str(self.args.jobs)]): + code, _, _ = self.run("s6_build", cmd) + if code != 0: + raise StageFailure("build_failed", f"{shlex.join(str(c) for c in cmd[:2])} " + f"exited {code}") + return {"export": export, "build": build} + + def s6_test(self, build): + junit = self.artifacts / "ctest-junit.xml" + env = {k: v for k, v in os.environ.items() if k != "SPINE_READER_REAL_FILES"} + code, _, _ = self.run("s6_test", ["ctest", "--test-dir", build, "--output-on-failure", + "--output-junit", junit], env=env) + failed = failed_tests(junit) if junit.exists() else [] + self.say("s6_test", f"failed tests: {failed or 'none'}") + if failed: + raise TestsFailed(failed) + if code != 0: + raise StageFailure("test_failed", f"ctest exited {code} and no test is marked failed") + return {"failed_tests": []} + + def effects(self, build): + """What the regenerated spine-dump prints for HEAD's previous current_v1 and v2.""" + previous = self.artifacts / "previous" + previous.mkdir(exist_ok=True) + files = [] + for name in ("current_v1.h5", "current_v2.h5"): + data = subprocess.run(["git", "-C", str(self.repo), "show", + f"HEAD:tests/data/fixtures/{name}"], + capture_output=True, check=True).stdout + (previous / name).write_bytes(data) + files.append(previous / name) + dumps = sorted(p for p in Path(build).rglob("spine-dump") if p.is_file()) + text = self.artifacts / "effects.txt" + if not dumps: + text.write_text("spine-dump was not built\n") + return text + code, stdout, stderr = self.run("s6_test", [dumps[0], "--events", "1", *files]) + text.write_text(f"$ spine-dump --events 1 " + f"\n{stdout}{stderr}[exit {code}]\n") + return text + + # --- s8 ------------------------------------------------------------------------------------ + + def s8_apply(self, new_schema_path, fixtures): + repo = self.repo + for old in (repo / "schema").glob("spine-*.json"): + old.unlink() + shutil.copy2(new_schema_path, repo / "schema" / Path(new_schema_path).name) + for cmd in ([sys.executable, repo / "tools/gen/generate.py", "--root", repo], + [sys.executable, repo / "tools/fixtures/make_member_list.py"]): + code, _, stderr = self.run("s8_apply", cmd) + if code != 0: + raise StageFailure("internal_error", f"{Path(cmd[1]).name} exited {code}: " + f"{stderr.strip()[-500:]}") + target = repo / "tests/data/fixtures" + for old in target.glob("current_*"): + old.unlink() + for new in sorted(Path(fixtures).iterdir()): + if new.name == "manifest.json" or new.name.startswith("current_"): + shutil.copy2(new, target / new.name) + self.say("s8_apply", git(repo, "status", "--short")) + return {} + + +class TestsFailed(StageFailure): + """s6_test: these ctest entries failed.""" + + def __init__(self, names): + super().__init__("test_failed", "failed: " + ", ".join(names)) + self.names = names + + +def parse_args(argv): + p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + p.add_argument("tag", help="the SPINE tag, vX.Y.Z") + p.add_argument("--baseline", help="schema to compare with (default: the committed one); " + "another one makes the run a replay") + p.add_argument("--apply", action="store_true", + help="after 'changed', write the new schema, model and fixtures into the tree") + p.add_argument("--repo-root", default=str(DEFAULT_REPO), + help="the spine-reader checkout (default: the one this script is in)") + p.add_argument("--spine-repo", default=os.environ.get("SPINE_REPO", DEFAULT_SPINE_REPO), + help="git URL or path of the SPINE repository") + p.add_argument("--workdir", default=os.environ.get("SPINE_FIXTURE_WORKDIR", DEFAULT_WORKDIR), + help="where the SPINE clones live (shared with make_fixtures.py)") + p.add_argument("--artifacts-dir", help="where the artifacts, the export and its build go " + "(default: a new temporary folder)") + p.add_argument("--jobs", type=int, default=os.cpu_count() or 1, + help="parallel build jobs (default: the number of CPUs)") + return p.parse_args(argv) + + +def main(argv=None): + argv = sys.argv[1:] if argv is None else argv + args = parse_args(argv) + repo = Path(args.repo_root).resolve() + try: + if not (repo / "tools/release/follow_spine.py").is_file(): + raise UsageError(f"--repo-root: {repo} is not a spine-reader checkout") + usage = check_usage(args, repo) + except UsageError as e: + print(f"follow_spine: usage error: {e}", file=sys.stderr) + return EXIT_CODES["usage_error"] + + artifacts = Path(args.artifacts_dir).resolve() if args.artifacts_dir else \ + Path(tempfile.mkdtemp(prefix=f"follow_spine-{args.tag}-")) + artifacts.mkdir(parents=True, exist_ok=True) + print(f"follow_spine: {args.tag}; artifacts in {artifacts}", flush=True) + run = Run(args, repo, artifacts) + reproduce = shlex.join([str(Path(__file__).resolve()), *argv]) + try: + status = follow(run, usage) + except UsageError as e: + print(f"follow_spine: usage error: {e}", file=sys.stderr) + return EXIT_CODES["usage_error"] + status["reproduce"] = reproduce + (artifacts / "status.json").write_text(json.dumps(status, indent=1, default=str) + "\n") + report(status, artifacts) + return status["exit_code"] + + +def follow(run, usage): + """All the stages; returns the status.json document.""" + args = run.args + status = {"status_format_version": STATUS_FORMAT_VERSION, "tag": args.tag, + "spine_sha": None, "spine_version": None, "replay": bool(usage.get("replay")), + "partial": False} + if "baseline" in usage: + b = usage["baseline"] + status["baseline"] = {"path": str(usage["baseline_path"]), + "spine_version": b.get("spine_version"), + "spine_sha": b.get("spine_sha")} + stages = run.stages + + def finish(): + outcome, code, failed_stage, contracts = outcome_for(stages) + classification = run.info.get("classification", {}) + status.update( + outcome=outcome, exit_code=code, failed_stage=failed_stage, + failure_detail=stages[failed_stage].get("detail") if failed_stage else None, + failure_reason=stages[failed_stage].get("reason") if failed_stage else None, + contracts_failed=contracts, + needs_attention=bool(classification.get("needs_attention")), + comparison=run.info.get("comparison"), + proposed_bump=classification.get("proposed_bump"), + change_kinds=classification.get("change_kinds"), + stages=stages, warnings=run.info.get("schema_warnings", []) + run.info["warnings"], + substitutions=run.info["substitutions"], tools=run.info.get("tools"), + spine_reader={"commit": git(run.repo, "rev-parse", "HEAD", check=False).strip(), + "dirty": bool(git(run.repo, "status", "--porcelain", + check=False).strip())}, + artifacts=str(run.artifacts)) + return status + + if usage["not_followed"]: + stages["s0_checks"] = {"status": "passed", "not_followed": usage["not_followed"], + "duration_s": 0.0, "log": None} + run.info["tools"] = None + return finish() + if run.stage("s0_checks", run.s0_environment) is None: + return finish() + tag = run.stage("s1_tag", run.s1_tag, usage) + if tag is None: + return finish() + status["spine_sha"] = tag["sha"] + dump = run.stage("s2_dump", run.s2_dump, args.tag, tag["sha"], tag["src"]) + if dump is None: + return finish() + status["spine_version"] = dump["spine_version"] + compare = run.stage("s3_compare", run.s3_compare, usage["baseline"], dump["schema"]) + if compare is None: + return finish() + fixtures = run.stage("s4_fixtures", run.s4_fixtures, args.tag, tag["sha"], dump["path"]) + if fixtures is None: + return finish() + # columns are compared only with HEAD's set, which belongs to the committed baseline + default_baseline = usage["baseline_path"] == committed_schema(run.repo).resolve() + old_columns = fixtures["head"] if default_baseline else None + if run.stage("s5_classify", run.classify, "s5_classify", usage["baseline_path"], + dump["path"], old_columns, fixtures["fixtures"]) is None: + return finish() + if usage["replay"]: + return finish() + changed = compare["comparison"] == "changed" + built = run.stage("s6_build", run.s6_build, changed, dump["path"], fixtures["fixtures"]) + if built is None: + return finish() + run.stage("s6_test", run.s6_test, built["build"]) + try: # informational: a failure here changes no outcome + effects = run.effects(built["build"]) + run.classify("s6_test", usage["baseline_path"], dump["path"], old_columns, + fixtures["fixtures"], effects) + except Exception as e: + run.say("s6_test", f"the effects or the second classification failed: " + f"{type(e).__name__}: {e}") + outcome = outcome_for(stages)[0] + if args.apply and outcome == "changed": + run.stage("s8_apply", run.s8_apply, dump["path"], fixtures["fixtures"]) + elif args.apply: + print(f"follow_spine: --apply: nothing written ({outcome})", flush=True) + return finish() + + +def report(status, artifacts): + print(f"follow_spine: {status['tag']} -> {status['outcome']} (exit {status['exit_code']})" + + (f", failed at {status['failed_stage']}: {status['failure_detail']}" + if status["failed_stage"] else "") + + (", contracts failed" if status["contracts_failed"] else "") + + (", replay" if status["replay"] else "") + + (", needs attention" if status["needs_attention"] else "")) + if status.get("proposed_bump"): + print(f"follow_spine: proposed bump: {status['proposed_bump']} " + f"({', '.join(status.get('change_kinds') or []) or 'no change kind'})") + if (artifacts / "classification.md").exists(): + print(f"follow_spine: classification: {artifacts / 'classification.md'}") + if status["stages"].get("s8_apply", {}).get("status") == "passed": + classification = json.loads((artifacts / "classification.json").read_text()) + print("follow_spine: --apply wrote the new schema, model, golden_members.inc, current " + "set and manifest; nothing is committed. Follow-up:") + for item in classification.get("follow_up", []): + print(f" - {item}") + print(f"follow_spine: status: {artifacts / 'status.json'}") + print(f"follow_spine: reproduce: {status['reproduce']}") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/release/tests/__init__.py b/tools/release/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tools/release/tests/data/spine-1.2.2.json b/tools/release/tests/data/spine-1.2.2.json new file mode 100644 index 0000000..cb37e38 --- /dev/null +++ b/tools/release/tests/data/spine-1.2.2.json @@ -0,0 +1,8384 @@ +{ + "schema_format": 1, + "spine_version": "1.2.2", + "spine_sha": "d4de053fd33fffddd031a3c8b58c10004baaee20", + "generator": "tools/gen/dump_schema.py", + "notes": [ + "Field order is the order of as_dict(lite=False): the HDF5 compound member order.", + "Types are the declared ones (annotation or FieldMetadata.dtype), not read from files.", + "interaction_mode and interaction_type are plain int64 with no enum: SPINE interprets them under interaction_scheme (see scheme_dependent_enum on those fields and enums[*].scheme_dependent).", + "derived fields (SPINE stored_property) take as default the value on a default-constructed object (default_source = default_instance).", + "float32/float64 defaults 'nan', 'inf' and '-inf' are strings in this file.", + "spine_sha is the commit given with --spine-sha (null without it).", + "writer_format: HDF5Writer's format attributes and the default of its format_version parameter; null when SPINE has no such attribute or parameter.", + "warnings: what the dumper sees but does not map (unmapped_attribute, unlisted_class)." + ], + "classes": { + "RecoParticle": { + "module": "spine.data.out.particle", + "mro": [ + "RecoParticle", + "ParticleBase", + "OutBase", + "PosDataBase", + "DataBase", + "RecoBase" + ], + "role": "reconstructed particle (main reco product)", + "doc": "Reconstructed particle information.", + "num_fields": 55, + "fields": [ + { + "name": "is_truth", + "defined_in": "RecoBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "Whether this is a truth object (always False for reconstructed)", + "doc_source": "RecoBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "units", + "defined_in": "OutBase", + "storage": "string", + "element": "string", + "length": null, + "python_type": "str", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": "cm", + "default_source": "dataclass", + "doc": "Units in which coordinates are expressed", + "doc_source": "OutBase", + "writer": { + "storage": "string", + "dtype": "utf8_vlen_string" + } + }, + { + "name": "id", + "defined_in": "OutBase", + "storage": "scalar", + "element": "int64", + "length": null, + "python_type": "int", + "flags": [ + "index" + ], + "units": null, + "enum": null, + "reference": "object", + "reference_space": "same", + "default": -1, + "default_source": "dataclass", + "doc": "Unique index of the object within the object list", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1 TPC in one module", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "is_matched", + "defined_in": "OutBase", + "storage": "scalar", + "element": "bool", + "length": null, + "python_type": "bool", + "flags": [], + "units": null, + "enum": null, + "reference": null, + "reference_space": null, + "default": false, + "default_source": "dataclass", + "doc": "True if a true object match was found", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": "|b1" + } + }, + { + "name": "cathode_offset", + "defined_in": "OutBase", + "storage": "scalar", + "element": "float64", + "length": null, + "python_type": "float", + "flags": [], + "units": "cm", + "enum": null, + "reference": null, + "reference_space": null, + "default": "nan", + "default_source": "dataclass", + "doc": "If the particle is a cathode crosser, this is how far in cm one needs to move its points along the drift direction to reconcile at the cathode. This is directly proportional to time through time=offset/vdrift", + "doc_source": "OutBase", + "writer": { + "storage": "scalar", + "dtype": " 1.2.3, and the bump table. No SPINE, no network.""" + +import contextlib +import copy +import io +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE.parent)) + +import classify # noqa: E402 + +DATA = HERE / "data" + + +def frozen(version="1.4.0"): + """A copy of tools/release/tests/data/spine-.json, never the committed schema.""" + return json.loads((DATA / f"spine-{version}.json").read_text()) + + +BASE = frozen() + + +def mutated(change): + """A copy of the frozen 1.4.0 schema with ``change(schema)`` applied.""" + schema = copy.deepcopy(BASE) + change(schema) + return schema + + +def field(schema, cls, name): + return next(f for f in schema["classes"][cls]["fields"] if f["name"] == name) + + +def add_field(schema, cls="RecoParticle", at=10, name="new_score"): + fields = schema["classes"][cls]["fields"] + new = dict(field(schema, cls, "csda_ke"), name=name, doc="A new score") + fields.insert(at, new) + schema["classes"][cls]["num_fields"] = len(fields) + + +def remove_field(schema, cls="RecoParticle", name="crt_scores"): + c = schema["classes"][cls] + c["fields"] = [f for f in c["fields"] if f["name"] != name] + c["num_fields"] = len(c["fields"]) + + +def set_members(schema, enum, members): + schema["enums"][enum]["members"] = members + + +# Each case: (description, change, expected change kinds, expected bump) +CASES = [ + ("a field added in the middle", add_field, {"field_added"}, "patch"), + ("a field removed", remove_field, {"field_removed"}, "minor"), + ("a shape retype", + lambda s: field(s, "RecoParticle", "csda_ke").update(storage="fixed_array", length=3), + {"shape_retyped"}, "minor"), + ("an element retype", + lambda s: field(s, "RecoParticle", "csda_ke").update(element="float32"), + {"element_retyped"}, "minor"), + ("a reordering", + lambda s: s["classes"]["RunInfo"]["fields"].reverse(), {"field_reordered"}, "person"), + ("an enum member added", + lambda s: s["enums"]["ParticlePID"]["members"].append(["NEUTRON", 6]), + {"enum_member_added"}, "patch"), + ("an enum member removed", + lambda s: s["enums"]["ParticlePID"]["members"].pop(), {"enum_member_removed"}, "minor"), + ("an enum member renumbered", + lambda s: set_members(s, "ParticlePID", [ + [n, 7 if n == "MUON" else v] for n, v in s["enums"]["ParticlePID"]["members"]]), + {"enum_member_renumbered"}, "minor"), + ("an enum alias added", + lambda s: s["enums"]["ParticlePID"]["aliases"].append(["MU", 2, "MUON"]), + {"enum_aliases_changed"}, "person"), + ("an enum added", + lambda s: s["enums"].__setitem__("NewEnum", copy.deepcopy(s["enums"]["NuCurrentType"])), + {"enum_added"}, "patch"), + ("an enum removed", + lambda s: s["enums"].pop("NeutNuInteractionType"), {"enum_removed"}, "minor"), + ("a scheme table added", + lambda s: s["scheme_tables"]["interaction_mode"].__setitem__("MARLEY", + "GenieNuInteractionType"), + {"scheme_table_added"}, "patch"), + ("a scheme table removed", + lambda s: s["scheme_tables"]["interaction_type"].pop("NEUT"), {"scheme_table_removed"}, + "minor"), + ("a scheme table changed", + lambda s: s["scheme_tables"]["interaction_mode"].__setitem__("NUWRO", + "NuWroNuInteractionType"), + {"scheme_table_changed"}, "person"), + ("a default changed", + lambda s: field(s, "RecoParticle", "csda_ke").update(default=0.0), {"default_changed"}, + "person"), + ("a unit changed", + lambda s: field(s, "RecoParticle", "csda_ke").update(units="GeV"), {"units_changed"}, + "person"), + ("a flag changed", + lambda s: field(s, "RecoParticle", "csda_ke")["flags"].append("vector"), {"flags_changed"}, + "person"), + ("a reference changed", + lambda s: field(s, "RecoParticle", "interaction_id").update(reference_space="opposite"), + {"reference_changed"}, "person"), + ("a doc-only change", + lambda s: field(s, "RecoParticle", "csda_ke").update(doc="Range-based kinetic energy"), + {"doc_changed"}, "none"), + ("a module-only change", + lambda s: s["classes"]["RecoParticle"].update(module="spine.data.out.reco"), + {"module_changed"}, "none"), + ("a class added", + lambda s: s["classes"].__setitem__("NewThing", copy.deepcopy(s["classes"]["RunInfo"])), + {"class_added"}, "person"), + ("a class removed", lambda s: s["classes"].pop("FlashHypothesis"), {"class_removed"}, + "minor"), + ("a type_check entry", + lambda s: s["type_check"]["mismatches"].append( + {"class": "RecoParticle", "field": "csda_ke", "what": "writer dtype", + "detail": "dtype: declared '' + '' + '' + '' + '' + '') + self.assertEqual(follow_spine.failed_tests(junit), ["spine_reader_contracts"]) + + def test_a_usage_error_writes_no_status(self): + with tempfile.TemporaryDirectory() as tmp: + artifacts = Path(tmp) / "artifacts" + with contextlib.redirect_stderr(io.StringIO()) as err: + code = follow_spine.main(["1.4.0", "--artifacts-dir", str(artifacts)]) + self.assertEqual(code, 2) + self.assertIn("usage error", err.getvalue()) + self.assertFalse(artifacts.exists()) + + def test_a_tag_not_followed_writes_a_status(self): + with tempfile.TemporaryDirectory() as tmp: + artifacts = Path(tmp) / "artifacts" + with contextlib.redirect_stdout(io.StringIO()): + code = follow_spine.main(["v1.4.0rc1", "--artifacts-dir", str(artifacts)]) + self.assertEqual(code, 20) + status = json.loads((artifacts / "status.json").read_text()) + self.assertEqual((status["outcome"], status["exit_code"]), ("not_followed", 20)) + self.assertEqual(status["status_format_version"], 1) + + +if __name__ == "__main__": + unittest.main()