Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,19 @@ jobs:
- uses: cvmfs-contrib/github-action-cvmfs@72f2a8d366043978d6aebee154bc9ed321c2f2a8 # v5.6
with:
cvmfs_repositories: sft.cern.ch
- name: Build and test
run: ci/github-ci.sh /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh
# Also follows the model's own SPINE tag (tools/release/follow_spine.py), which must come
# out "identical"; its artifacts are uploaded when the job fails.
- name: Build and test, then follow the model's SPINE tag
run: >-
ci/github-ci.sh --follow-spine "$RUNNER_TEMP/artifacts"
/cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh
- name: Upload the follow_spine artifacts
if: failure()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: follow-spine-artifacts
path: ${{ runner.temp }}/artifacts
if-no-files-found: ignore
- name: Tag matches project(VERSION)
if: startsWith(github.ref, 'refs/tags/')
run: ci/check_tag_version.sh "$GITHUB_REF_NAME"
Expand Down
27 changes: 18 additions & 9 deletions EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,10 @@ Build and packaging (`CMakeLists.txt`):
own writer (the `current_*` set by any tag, the frozen cases by old releases), plus their
expected JSON, `manifest.json` and `tests/golden_members.inc`.
- [./tools/release/EGM.md](./tools/release/EGM.md) - Release tooling:
`scan_consumer_fields.py` (the model fields a CAFMaker build reads, against the contract
table of `tests/test_contracts.cpp`).
`follow_spine.py` (one SPINE tag, end to end: identical, changed, not followed or a
failure, with `status.json`; `--apply` writes a change into the tree), `classify.py` (the
change and the proposed bump) and `scan_consumer_fields.py` (the model fields a CAFMaker
build reads, against the contract table of `tests/test_contracts.cpp`).
- examples: [./examples/EGM.md](./examples/EGM.md) - The `spine-dump` command-line tool.
- ci: [./ci/EGM.md](./ci/EGM.md) - the GitHub Actions workflow and `github-ci.sh` (HDF5 1.14 and
1.12 from LCG views on CVMFS), `check_tag_version.sh` (release tag against
Expand Down Expand Up @@ -118,8 +120,11 @@ Build and packaging (`CMakeLists.txt`):
- Developer tools only: `tools/gen` rewrites the generated model; `tools/fixtures` clones SPINE
commits and rewrites `tests/data/fixtures/` (cases and manifest) and `tests/golden_members.inc`;
`tools/release/scan_consumer_fields.py` runs a CAFMaker's compile commands with
`-fsyntax-only` in a work folder; `ci/` scripts create `build-ci/`, `build-asan/`,
`build-tsan/` and `build-sl7/`.
`-fsyntax-only` in a work folder; `tools/release/follow_spine.py` clones SPINE, and builds
and tests an export in its artifacts folder. With `--apply`, after a `changed` outcome, it
writes the new `schema/spine-*.json`, the regenerated model, `tests/golden_members.inc`, the
current fixtures and the manifest into the working tree, and commits nothing. `ci/` scripts
create `build-ci/`, `build-asan/`, `build-tsan/` and `build-sl7/`.
- CI: the GitHub Actions workflow builds and tests on GitHub's hosted runners, with read-only
access to the repository and no secrets.

Expand All @@ -129,12 +134,16 @@ Build and packaging (`CMakeLists.txt`):
`cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build --output-on-failure`
- The ctest entries include `spine_reader_tests` (unit tests), `spine_reader_contracts` (what
the CAFMakers rely on: `tests/test_contracts.cpp`), `spine_reader_generated_model`
(`tools/gen/generate.py --check`), `spine_reader_golden_members`, `spine_dump_output`, and
(`tools/gen/generate.py --check`), `spine_reader_golden_members`,
`spine_reader_release_tools` (the unit tests of `tools/release`), `spine_dump_output`, and
`spine_reader_real_files` (skipped unless `SPINE_READER_REAL_FILES` names a directory).
- `spine_reader_tests` excludes the `real files*` and `contract*` test cases, which
`spine_reader_real_files` and `spine_reader_contracts` run. `spine_reader_generated_model` and `spine_reader_golden_members` are registered only
when CMake finds Python 3 (with `SPINE_READER_REQUIRE_PYTHON=ON`, a missing Python fails the
configuration), and `spine_dump_output` only when the tools are built.
`spine_reader_real_files` and `spine_reader_contracts` run. `spine_reader_generated_model`
and `spine_reader_golden_members` are registered only when CMake finds Python 3 (with
`SPINE_READER_REQUIRE_PYTHON=ON`, a missing Python fails the configuration),
`spine_reader_release_tools` only for Python 3.10 or newer, and `spine_dump_output` only
when the tools are built.
- CI: `.github/workflows/ci.yml` runs `ci/github-ci.sh` with HDF5 1.14 (LCG_110, Ubuntu 24.04)
and HDF5 1.12 (LCG_105, Ubuntu 22.04) on every push and pull request.
and HDF5 1.12 (LCG_105, Ubuntu 22.04) on every push and pull request. The 1.14 job then runs
`tools/release/follow_spine.py` on the model's own SPINE tag and requires `identical`.
- Other toolchains: `ci/sl7-ups.sh`, `ci/sanitize.sh`, `ci/sanitize.sh thread`.
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Status: under construction.
| `src/model/` | Generated field tables. Do not edit by hand. |
| `tools/gen/` | Generator: dumps SPINE `FieldMetadata` to `schema/` and writes the model. |
| `tools/fixtures/` | Writes the test files in `tests/data/` with SPINE's own writer, plus expected values. |
| `tools/release/` | Tooling for following SPINE releases: `scan_consumer_fields.py` lists the model fields a CAFMaker build reads. |
| `tools/release/` | Tooling for following SPINE releases: `follow_spine.py` runs one SPINE tag end to end, `classify.py` proposes the version bump, `scan_consumer_fields.py` lists the model fields a CAFMaker build reads ([README](tools/release/README.md)). |
| `tests/` | doctest unit tests (`test_*.cpp`). |

## Build
Expand All @@ -55,6 +55,7 @@ cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build --outpu
| `spine_reader_real_files` | Reads every `*.h5` in `$SPINE_READER_REAL_FILES` (or `-DSPINE_READER_REAL_FILES=dir`). Without a directory it reports **Skipped**, never Passed. |
| `spine_reader_generated_model` | `tools/gen/generate.py --check`: the generated model is what the committed SPINE schema produces. |
| `spine_reader_golden_members` | `tests/golden_members.inc` (the member list that `test_golden.cpp` compares object by object, and that `test_model.cpp` and `test_contracts.cpp` walk) is current. |
| `spine_reader_release_tools` | The unit tests of `tools/release` (`classify.py`, `follow_spine.py`), without SPINE or the network. Registered for Python 3.10 or newer. |
| `spine_dump_output` | `spine-dump` prints what it should for the Reader test files. |

When a CAFMaker starts reading a model field, add its row to the contract table.
Expand All @@ -71,6 +72,8 @@ CI (`.github/workflows/ci.yml`) builds and runs ctest on every push and pull req
1.14 (LCG_110, Ubuntu 24.04) and HDF5 1.12 (LCG_105, Ubuntu 22.04) from the LCG views on CVMFS.
Both jobs run `ci/github-ci.sh`, which configures with `-DSPINE_READER_REQUIRE_PYTHON=ON`: a missing
Python 3 then fails the configuration instead of silently leaving out the Python-based checks.
The 1.14 job then runs `tools/release/follow_spine.py` on the model's own SPINE tag, which must
come out `identical`; its artifacts are uploaded when it fails.
`ci/EGM.md` shows how to run the same jobs locally.

## Conventions
Expand Down
29 changes: 23 additions & 6 deletions ci/EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,17 @@ Every script finds the repository from its own location, so it can be run from a
directory.

## Interface
- `github-ci.sh <LCG view setup.sh> [build directory]`
- `github-ci.sh [--follow-spine DIR] <LCG view setup.sh> [build directory]`
- Sources the given LCG view, then configures `<repo>/<build directory>` (default `build-ci`)
with `-DSPINE_READER_REQUIRE_PYTHON=ON`, builds with `-j$(nproc)`, and runs
`ctest --output-on-failure`. No build type is given, so the project default
(`RelWithDebInfo`) applies, as in CI.
- With `--follow-spine DIR`, it then reads `spine_version` from the one `schema/spine-*.json`
and runs `tools/release/follow_spine.py v<spine_version> --artifacts-dir DIR --jobs
$(nproc)`. It needs exit 0 (`identical`).
- Prints the OS, the compiler (`$CXX`, set by the LCG view), CMake, Python and the view.
- Exit codes: 2 on bad arguments; otherwise the first failing command's code.
- Exit codes: 2 on bad arguments; otherwise the first failing command's code (for
`follow_spine.py`, its own: 10, 20, 30 or 2).
- `check_tag_version.sh vX.Y.Z`
- Reads `project(spine-reader VERSION X.Y.Z ...)` from `CMakeLists.txt`.
- Exits 0 when the argument equals `vX.Y.Z`, 1 when it does not or when no version line is
Expand All @@ -29,13 +33,22 @@ directory.
- Runs on every push (branches and tags), every pull request and manual dispatch, with
`permissions: contents: read` and no secrets.
- Job `hdf5-1_14`: `ubuntu-24.04`, LCG_110 `x86_64-ubuntu2404-gcc13-opt` (HDF5 1.14.6, gcc 13
from the runner, CMake 3.30, Python 3.13). On a tag push it also runs
`check_tag_version.sh "$GITHUB_REF_NAME"`.
from the runner, CMake 3.30, Python 3.13).
- It runs `github-ci.sh --follow-spine "$RUNNER_TEMP/artifacts"`: the build, ctest, then
`follow_spine.py` on the model's own SPINE tag, which clones SPINE from GitHub and must
give `identical`. The tag must still point at the schema's `spine_sha`, so CI only ever
runs that recorded SPINE commit.
- When the job fails, it uploads `$RUNNER_TEMP/artifacts` as `follow-spine-artifacts`
(`actions/upload-artifact`).
- On a tag push it also runs `check_tag_version.sh "$GITHUB_REF_NAME"`.
- Job `hdf5-1_12`: `ubuntu-22.04`, LCG_105 `x86_64-ubuntu2204-gcc11-opt` (HDF5 1.12.2, gcc 11
from the runner, CMake 3.26, Python 3.9).
- Each job checks out the repository, mounts `sft.cern.ch` with
`cvmfs-contrib/github-action-cvmfs`, and runs `github-ci.sh` with its view. The two actions
are pinned by commit SHA, with the version in a comment.
`cvmfs-contrib/github-action-cvmfs`, and runs `github-ci.sh` with its view. The actions
(`actions/checkout`, the CVMFS action and `actions/upload-artifact`) are pinned by commit
SHA, with the version in a comment.
- The release-tool unit tests (`spine_reader_release_tools`) need Python 3.10, so they run in
the 1.14 job only; the 1.12 job (Python 3.9) registers six ctest entries.
- `sanitize.sh [mode]`
- With no argument (`mode=address`, the default) it configures with
`-DSPINE_READER_SANITIZE=ON`, which enables ASan and UBSan with
Expand Down Expand Up @@ -84,6 +97,9 @@ directory.
- Read from CVMFS: the LCG views, `setup_larsoft.sh` and UPS products, and the container
images. `sl7-ups.sh` and the dry run below need `apptainer` on the host.
- The workflow runs on GitHub's hosted runners, with read-only access to the repository.
- `github-ci.sh --follow-spine DIR` clones SPINE (from GitHub, into
`$TMPDIR/spine-reader-fixtures`) and writes `DIR`: the artifacts, an export of HEAD and its
build.

## Verification
- Run: `ci/sanitize.sh` (ASan and UBSan), `ci/sanitize.sh thread` (TSan), and
Expand All @@ -94,6 +110,7 @@ directory.
```bash
img=/cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/sft/docker
apptainer exec -B /cvmfs -B /tmp $img/ubuntu24:latest bash -e ci/github-ci.sh \
--follow-spine /tmp/follow-spine-artifacts \
/cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-ubuntu2404-gcc13-opt/setup.sh build-ci-1.14
apptainer exec -B /cvmfs -B /tmp $img/ubuntu22:latest bash -e ci/github-ci.sh \
/cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-ubuntu2204-gcc11-opt/setup.sh build-ci-1.12
Expand Down
27 changes: 25 additions & 2 deletions ci/github-ci.sh
Original file line number Diff line number Diff line change
@@ -1,13 +1,24 @@
#!/bin/bash
# Build and test spine-reader the way the GitHub workflow (.github/workflows/ci.yml) does:
# source an LCG view, configure with SPINE_READER_REQUIRE_PYTHON=ON, build, run ctest.
# With --follow-spine DIR, then run tools/release/follow_spine.py on the model's own SPINE tag,
# with its artifacts in DIR, and require "identical" (exit 0).
# The same script is used for the local dry run (see ci/EGM.md).
#
# ci/github-ci.sh <LCG view setup.sh> [build directory, default build-ci]
# ci/github-ci.sh [--follow-spine DIR] <LCG view setup.sh> [build directory, default build-ci]
set -uo pipefail

follow=
if [[ ${1:-} == --follow-spine ]]; then
if [[ $# -lt 2 ]]; then
echo "usage: $0 [--follow-spine DIR] <LCG view setup.sh> [build directory]" >&2
exit 2
fi
follow=$2
shift 2
fi
if [[ $# -lt 1 || $# -gt 2 ]]; then
echo "usage: $0 <LCG view setup.sh> [build directory]" >&2
echo "usage: $0 [--follow-spine DIR] <LCG view setup.sh> [build directory]" >&2
exit 2
fi
view=$1
Expand All @@ -26,3 +37,15 @@ echo "== $(cmake --version | head -1) | $(python3 --version) | view $view"
cmake -S "$repo" -B "$build" -DSPINE_READER_REQUIRE_PYTHON=ON
cmake --build "$build" -j"$(nproc)"
ctest --test-dir "$build" --output-on-failure

if [[ -n $follow ]]; then
# the model's SPINE version, from the one committed schema
version=$(python3 -c 'import glob, json, sys
found = glob.glob(sys.argv[1] + "/schema/spine-*.json")
assert len(found) == 1, found
print(json.load(open(found[0]))["spine_version"])' "$repo")
echo "== follow_spine.py v$version (artifacts in $follow)"
"$repo/tools/release/follow_spine.py" "v$version" --artifacts-dir "$follow" --jobs "$(nproc)" ||
{ code=$?; echo "follow_spine.py v$version exited $code; identical (0) is required" >&2
exit "$code"; }
fi
8 changes: 8 additions & 0 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,14 @@ if(Python3_Interpreter_FOUND)
add_test(NAME spine_reader_golden_members
COMMAND ${Python3_EXECUTABLE} ${PROJECT_SOURCE_DIR}/tools/fixtures/make_member_list.py
--check)
# The unit tests of tools/release (classify.py, follow_spine.py): no SPINE, no network. The
# release tools need Python 3.10, as SPINE does; older interpreters leave them out.
if(Python3_VERSION VERSION_GREATER_EQUAL 3.10)
add_test(NAME spine_reader_release_tools
COMMAND ${Python3_EXECUTABLE} -m unittest discover
-s ${PROJECT_SOURCE_DIR}/tools/release/tests
-t ${PROJECT_SOURCE_DIR}/tools/release)
endif()
else()
message(WARNING "No Python 3 interpreter: the generated-model checks are not registered")
endif()
Expand Down
30 changes: 22 additions & 8 deletions tools/EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,13 @@ library. They are tied to SPINE and to the single committed
`schema/spine-*.json`. `gen` turns SPINE's object model into the generated C++
model. `fixtures` produces the golden HDF5 files and their decoded expectations,
which the C++ tests read. `release` holds the tooling for following SPINE
releases; today `scan_consumer_fields.py`, which lists the model fields a
CAFMaker build reads. `gen` and `fixtures` each register a ctest check:
`spine_reader_generated_model` (`gen/generate.py --check`) and
`spine_reader_golden_members` (`fixtures/make_member_list.py --check`).
releases: `follow_spine.py` (one SPINE tag, end to end, to one outcome),
`classify.py` (what changed, and the proposed bump) and
`scan_consumer_fields.py` (the model fields a CAFMaker build reads). Each
registers a ctest check: `spine_reader_generated_model` (`gen/generate.py
--check`), `spine_reader_golden_members` (`fixtures/make_member_list.py
--check`) and `spine_reader_release_tools` (the unit tests of `release`, only
for Python 3.10 or newer).
`tests/CMakeLists.txt` registers them when CMake finds a Python 3 interpreter;
with `-DSPINE_READER_REQUIRE_PYTHON=ON` (CI turns it on) a missing interpreter
fails the configuration instead of leaving them out. Run the tools in the
Expand All @@ -36,7 +39,18 @@ CAFMaker's environment.
resolves tags and clones commits. `make_member_list.py` writes or checks
`tests/golden_members.inc` from the schema. Legacy notes:
[./fixtures/README.md](./fixtures/README.md).
- release: [./release/EGM.md](./release/EGM.md) - `scan_consumer_fields.py`
(compiles a CAFMaker's sources with `-fsyntax-only` against model headers whose
members are marked `[[deprecated]]`, and lists the fields read; with
`--contracts`, the ones that have no row in `tests/test_contracts.cpp`).
- release: [./release/EGM.md](./release/EGM.md) - `follow_spine.py TAG` runs the
following stages and ends in one outcome:
- it dumps the tag's schema;
- it compares it with the model (identical or changed);
- it writes and checks the tag's fixtures;
- it classifies the change;
- it builds and tests an export of HEAD with them.

The exit code is 0 identical, 10 changed, 20 not followed, 30 failure, or 2 usage.
`--apply` writes a change into the working tree.

`classify.py` proposes the version bump. `scan_consumer_fields.py` compiles a CAFMaker's
sources with `-fsyntax-only` against model headers whose members are marked
`[[deprecated]]`, and lists the fields read; with `--contracts`, it lists the ones that have
no row in `tests/test_contracts.cpp`. Guide: [./release/README.md](./release/README.md).
3 changes: 2 additions & 1 deletion tools/gen/EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,8 @@ SPINE clone --dump_schema.py--> schema/spine-<version>.json --generate.py--> C++
changed files (written or deleted).
- `check_names.py` compiles a temporary `all.cpp` with
`<cxx> -std=gnu++17 -dM -E -x c++` (`check=True`). It reads
`include/spine/model*` and `src/model/*`, and prints the clashes and a summary.
the `.hpp` and `.cpp` files of `include/spine/model*` and `src/model/` (not the `EGM.md`
files), and prints the clashes and a summary.

## Verification

Expand Down
9 changes: 8 additions & 1 deletion tools/gen/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ SPINE clone at a tag --dump_schema.py--> schema/spine-<version>.json --generate.

## Regenerate for a new SPINE release

`tools/release/follow_spine.py vX.Y.Z` does all of this, and checks the result: it dumps the
tag, compares the schema with the model, writes and checks the tag's fixtures, builds and tests
the regenerated model, and with `--apply` writes it into the working tree
([../release/README.md](../release/README.md)). The manual commands below are what it runs:

```bash
source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc13-opt/setup.sh
git clone -q --branch vX.Y.Z <SPINE repository> /some/dir/spine-X.Y.Z
Expand Down Expand Up @@ -50,7 +55,9 @@ mismatch also exits 1.
enumerator by a trailing underscore (`NULL` -> `NULL_`; the descriptor keeps `"NULL"`) from its
`RESERVED` list; if the checker finds a new clash, add the name there. `--include` adds headers
(HDF5, ROOT) to the scan.
4. **Build and test.** `cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build`.
4. **Build and test.** Read `classification.md` from the `follow_spine.py` run first: it lists
what changed, kind by kind, and the proposed bump.
`cmake -S . -B build && cmake --build build -j8 && ctest --test-dir build`.
The `static_assert`s in `src/model/*.cpp` fail the build if a member's C++ type disagrees with
its descriptor row. No test holds field counts, defaults or enum member counts: they follow
the schema. Write the new tag's fixtures first (`tools/fixtures/make_fixtures.py --tag <tag>`),
Expand Down
2 changes: 1 addition & 1 deletion tools/gen/check_names.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def main():
macros = system_macros(args.cxx, args.include)
files = sorted(
[*(root / "include/spine").glob("model/*.hpp"), root / "include/spine/model.hpp"]
+ [*(root / "src/model").glob("*")]
+ [p for p in (root / "src/model").glob("*") if p.suffix in (".cpp", ".hpp")]
)
clashes = {}
for path in files:
Expand Down
Loading