Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ Build and packaging (`CMakeLists.txt`):
- [./tools/gen/EGM.md](./tools/gen/EGM.md) - Model generator (`dump_schema.py`,
`generate.py`, `check_names.py`).
- [./tools/fixtures/EGM.md](./tools/fixtures/EGM.md) - Golden test files written by SPINE's
own writer, plus their expected JSON and `tests/golden_members.inc`.
own writer (the `current_*` set by any tag, the frozen cases by old releases), plus their
expected JSON, `manifest.json` and `tests/golden_members.inc`.
- examples: [./examples/EGM.md](./examples/EGM.md) - The `spine-dump` command-line tool.
- ci: [./ci/EGM.md](./ci/EGM.md) - the GitHub Actions workflow and `github-ci.sh` (HDF5 1.14 and
1.12 from LCG views on CVMFS), `check_tag_version.sh` (release tag against
Expand Down Expand Up @@ -112,7 +113,7 @@ Build and packaging (`CMakeLists.txt`):
- Temporarily changes HDF5's library-global automatic error printing (`h5::ErrorSilencer`),
restoring it afterwards.
- Developer tools only: `tools/gen` rewrites the generated model; `tools/fixtures` clones SPINE
tags and rewrites `tests/data/fixtures/` and `tests/golden_members.inc`; `ci/` scripts
commits and rewrites `tests/data/fixtures/` (cases and manifest) and `tests/golden_members.inc`; `ci/` scripts
create `build-ci/`, `build-asan/`, `build-tsan/` and `build-sl7/`.
- CI: the GitHub Actions workflow builds and tests on GitHub's hosted runners, with read-only
access to the repository and no secrets.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "incomplete",
"case": "current_incomplete",
"writer": {"spine_version":"1.4.0","format_version":1,"lite":false,"complete":false},
"layout": "V1",
"num_events": 2,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "v1.4.0_v1",
"case": "current_v1",
"writer": {"spine_version":"1.4.0","format_version":1,"lite":false,"complete":true},
"layout": "V1",
"num_events": 5,
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "v1.4.0_v1_lite",
"case": "current_v1_lite",
"writer": {"spine_version":"1.4.0","format_version":1,"lite":true,"complete":true},
"layout": "V1",
"num_events": 5,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "v1.4.0_v2",
"case": "current_v2",
"writer": {"spine_version":"1.4.0","format_version":2,"lite":false,"complete":true},
"layout": "V2Products",
"num_events": 5,
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "v1.4.0_v2_fixed_only",
"case": "current_v2_fixed_only",
"writer": {"spine_version":"1.4.0","format_version":2,"lite":false,"complete":true},
"layout": "V2Products",
"num_events": 5,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"case": "v1.4.0_v2_lite",
"case": "current_v2_lite",
"writer": {"spine_version":"1.4.0","format_version":2,"lite":true,"complete":true},
"layout": "V2Products",
"num_events": 5,
Expand Down
90 changes: 90 additions & 0 deletions tests/data/fixtures/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"cases": {
"current_incomplete": {
"derived_from": null,
"expected": "current_incomplete.expected.json",
"h5": "current_incomplete.h5",
"layout": "V1",
"lite": false,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"current_v1": {
"derived_from": null,
"expected": "current_v1.expected.json",
"h5": "current_v1.h5",
"layout": "V1",
"lite": false,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"current_v1_lite": {
"derived_from": null,
"expected": "current_v1_lite.expected.json",
"h5": "current_v1_lite.h5",
"layout": "V1",
"lite": true,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"current_v2": {
"derived_from": null,
"expected": "current_v2.expected.json",
"h5": "current_v2.h5",
"layout": "V2Products",
"lite": false,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"current_v2_fixed_only": {
"derived_from": "current_v2",
"expected": "current_v2_fixed_only.expected.json",
"h5": "current_v2_fixed_only.h5",
"layout": "V2Products",
"lite": false,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"current_v2_lite": {
"derived_from": null,
"expected": "current_v2_lite.expected.json",
"h5": "current_v2_lite.h5",
"layout": "V2Products",
"lite": true,
"role": "current",
"sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"tag": "v1.4.0"
},
"v0.11.1_v1": {
"derived_from": null,
"expected": "v0.11.1_v1.expected.json",
"h5": "v0.11.1_v1.h5",
"layout": "V1",
"lite": false,
"role": "frozen",
"sha": "b78a6d64f60ce0be5a24611541df19033620b95a",
"tag": "v0.11.1"
},
"v2root": {
"derived_from": null,
"expected": "v2root.expected.json",
"h5": "v2root.h5",
"layout": "V2Root",
"lite": false,
"role": "frozen",
"sha": "a9ad6c63ce5532429c26b3374ec73c7ecc2f1cea",
"tag": "v0.16.3"
}
},
"partial": false,
"spine_sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"spine_tag": "v1.4.0",
"spine_version": "1.4.0",
"substitutions": [],
"warnings": []
}
82 changes: 73 additions & 9 deletions tests/test_golden.cpp
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
// Golden tests: every fixture in tests/data/fixtures was written by SPINE's own writer and
// decoded independently with h5py (tools/fixtures/decode_raw.py). The core must reproduce the
// decoded file, product by product, column by column, row by row.
//
// The cases are the *.expected.json files of the directory: the current set (current_*, written
// by the model's SPINE tag) and the frozen cases (written by older releases, never rewritten).
// tools/fixtures/make_fixtures.py writes them.

#include <doctest.h>
#include <json.hpp>

#include <algorithm>
#include <array>
#include <cmath>
#include <filesystem>
#include <fstream>
#include <functional>
#include <map>
Expand All @@ -29,11 +35,6 @@ using nlohmann::json;
using spine::ScalarType;
using spine::detail::ColumnBuffer;

const std::vector<std::string> kCases = {
"v1.4.0_v1", "v1.4.0_v2", "v1.4.0_v1_lite", "v1.4.0_v2_lite",
"v1.4.0_v2_fixed_only", "v2root", "v0.11.1_v1",
};

std::string fixture(const std::string& name, const std::string& extension) {
return std::string(SPINE_READER_TEST_DATA) + "/fixtures/" + name + extension;
}
Expand All @@ -44,6 +45,69 @@ json expected(const std::string& name) {
return json::parse(in);
}

/// One fixture: its name, its role (from the name) and the writer flags its .expected.json holds.
struct Case {
std::string name;
bool current = false; ///< "current_*": written by the model's SPINE tag; otherwise frozen
bool complete = true; ///< info@complete is not false (a missing attribute counts as complete)
};

/// Every case of tests/data/fixtures, sorted by name.
const std::vector<Case>& all_cases() {
static const std::vector<Case> cases = [] {
const std::string suffix = ".expected.json";
std::vector<Case> found;
for (const auto& entry :
std::filesystem::directory_iterator(std::string(SPINE_READER_TEST_DATA) + "/fixtures")) {
const std::string file = entry.path().filename().string();
if (file.size() <= suffix.size() ||
file.compare(file.size() - suffix.size(), suffix.size(), suffix) != 0) {
continue;
}
Case c;
c.name = file.substr(0, file.size() - suffix.size());
c.current = c.name.rfind("current_", 0) == 0;
const json want = expected(c.name);
const json& complete = want["writer"]["complete"];
c.complete = !(complete.is_boolean() && !complete.get<bool>());
found.push_back(c);
}
std::sort(found.begin(), found.end(),
[](const Case& a, const Case& b) { return a.name < b.name; });
return found;
}();
return cases;
}

/// The cases a reader opens without allow_incomplete: those whose writer.complete is not false
/// (a null, as in files of SPINE v0.11.1, counts as complete).
std::vector<std::string> complete_cases() {
std::vector<std::string> names;
for (const Case& c : all_cases()) {
if (c.complete) names.push_back(c.name);
}
return names;
}

} // namespace

TEST_CASE("the fixture directory holds the current set and the frozen cases") {
std::set<std::string> current, frozen;
for (const Case& c : all_cases()) (c.current ? current : frozen).insert(c.name);
// the current set: the names the other tests and tools/fixtures/make_fixtures.py use
CHECK(current == std::set<std::string>{"current_incomplete", "current_v1", "current_v1_lite",
"current_v2", "current_v2_fixed_only",
"current_v2_lite"});
CHECK_FALSE(frozen.empty());
for (const Case& c : all_cases()) {
INFO("fixture " << c.name);
CHECK(c.complete == (c.name != "current_incomplete"));
CHECK(expected(c.name)["case"].get<std::string>() == c.name);
}
}

namespace {

bool same_number(const json& want, double got) {
if (want.is_string()) {
const std::string s = want.get<std::string>();
Expand Down Expand Up @@ -152,7 +216,7 @@ void check_values(spine::core::ProductSource& product, const json& want, std::ui

TEST_CASE("the core reproduces every golden fixture") {
std::lock_guard<std::recursive_mutex> lock(spine::core::hdf5_mutex());
for (const std::string& name : kCases) {
for (const std::string& name : complete_cases()) {
INFO("fixture " << name);
const json want = expected(name);
auto file = spine::core::FileSource::open(fixture(name, ".h5"));
Expand Down Expand Up @@ -191,7 +255,7 @@ TEST_CASE("the core reproduces every golden fixture") {

TEST_CASE("incomplete files are refused unless allowed") {
std::lock_guard<std::recursive_mutex> lock(spine::core::hdf5_mutex());
const std::string path = fixture("incomplete", ".h5");
const std::string path = fixture("current_incomplete", ".h5");
CHECK_THROWS_AS(spine::core::FileSource::open(path), spine::Error);
spine::core::OpenOptions options;
options.allow_incomplete = true;
Expand Down Expand Up @@ -304,7 +368,7 @@ const json* by_id(const json& rows, std::int64_t id) {

TEST_CASE("the reader fills every member of every model object as the fixtures say") {
std::size_t objects_checked = 0;
for (const std::string& name : kCases) {
for (const std::string& name : complete_cases()) {
INFO("fixture " << name);
const json want = expected(name);
spine::Reader reader(fixture(name, ".h5"));
Expand Down Expand Up @@ -383,7 +447,7 @@ std::vector<T> list_of(const json& row, const char* column) {

TEST_CASE("event relations agree with the ids the fixtures hold") {
std::size_t resolved = 0, matched = 0, members = 0;
for (const std::string& name : kCases) {
for (const std::string& name : complete_cases()) {
INFO("fixture " << name);
const json want = expected(name);
spine::Reader reader(fixture(name, ".h5"));
Expand Down
10 changes: 5 additions & 5 deletions tests/test_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -586,12 +586,12 @@ TEST_CASE("strict mode refuses a product that lacks model fields, and only then"
ReaderOptions strict;
strict.strict = true;

// a complete SPINE v1.4.0 file reads
CHECK_NOTHROW(Reader(fixture_file("v1.4.0_v1.h5"), strict));
CHECK_NOTHROW(Reader(fixture_file("v1.4.0_v2.h5"), strict));
// a complete file of the current set reads
CHECK_NOTHROW(Reader(fixture_file("current_v1.h5"), strict));
CHECK_NOTHROW(Reader(fixture_file("current_v2.h5"), strict));

// the same file written in lite mode lacks index and orig_index; the oldest release lacks more
for (const char* name : {"v1.4.0_v1_lite.h5", "v0.11.1_v1.h5", "v1.4.0_v2_fixed_only.h5"}) {
for (const char* name : {"current_v1_lite.h5", "v0.11.1_v1.h5", "current_v2_fixed_only.h5"}) {
CAPTURE(name);
CHECK_NOTHROW(Reader(fixture_file(name))); // not strict: fine
try {
Expand All @@ -611,7 +611,7 @@ TEST_CASE("strict mode refuses a product that lacks model fields, and only then"
}

TEST_CASE("incomplete files are refused unless ReaderOptions::allow_incomplete") {
const std::string path = fixture_file("incomplete.h5");
const std::string path = fixture_file("current_incomplete.h5");
try {
Reader reader(path);
FAIL("the file is incomplete");
Expand Down
4 changes: 2 additions & 2 deletions tests/test_reader_locking.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,11 @@ std::vector<bool> finished_while_locked(const std::vector<std::function<void()>>
} // namespace

TEST_CASE("the public operations of a Reader that touch HDF5 wait for the HDF5 lock") {
const std::string path = fixture("v1.4.0_v1.h5");
const std::string path = fixture("current_v1.h5");
Reader reader(path);
auto doomed = std::make_unique<Reader>(path); // destroyed by an action
Reader target(path); // overwritten by an action
Reader source(fixture("v1.4.0_v2.h5")); // moved from by an action
Reader source(fixture("current_v2.h5")); // moved from by an action
std::unique_ptr<Reader> constructed; // built by an action (and so destroyed after)

const std::vector<bool> finished = finished_while_locked({
Expand Down
11 changes: 7 additions & 4 deletions tools/EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ in the LCG_110 view (`source
[./gen/README.md](./gen/README.md).
- fixtures: [./fixtures/EGM.md](./fixtures/EGM.md) - `make_fixtures.py` writes
`tests/data/fixtures/<case>.h5` with SPINE's real writers (one `spine_worker.py`
subprocess per write, litify or read call, each importing a single SPINE tag;
`PYTHONHASHSEED=0`), decodes them with the SPINE-free `decode_raw.py` into
`<case>.expected.json`, and cross-checks them. `make_member_list.py` writes or
checks `tests/golden_members.inc` from the schema. Legacy notes:
subprocess per write, litify or read call, each importing a single SPINE commit;
`PYTHONHASHSEED=0`): the current set (`current_*`) with any tag, by default the
model's, and the frozen cases with their own releases. It decodes them with the
SPINE-free `decode_raw.py` into `<case>.expected.json`, cross-checks them, and
records each case's SPINE tag and commit in `manifest.json`. `spine_checkout.py`
resolves tags and clones commits. `make_member_list.py` writes or checks
`tests/golden_members.inc` from the schema. Legacy notes:
[./fixtures/README.md](./fixtures/README.md).
Loading