Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions schema/spine-1.4.0.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
{
"schema_format": 1,
"spine_version": "1.4.0",
"spine_sha": "4726e068f17e433e2fec068dc8ee498fa9a0658f",
"generator": "tools/gen/dump_schema.py",
"notes": [
"Field order is the order of as_dict(lite=False): the HDF5 compound member order.",
"Types are the declared ones (annotation or FieldMetadata.dtype), not read from files.",
"interaction_mode and interaction_type are plain int64 with no enum: SPINE interprets them under interaction_scheme (see scheme_dependent_enum on those fields and enums[*].scheme_dependent).",
"derived fields (SPINE stored_property) take as default the value on a default-constructed object (default_source = default_instance).",
"float32/float64 defaults 'nan', 'inf' and '-inf' are strings in this file."
"float32/float64 defaults 'nan', 'inf' and '-inf' are strings in this file.",
"spine_sha is the commit given with --spine-sha (null without it).",
"writer_format: HDF5Writer's format attributes and the default of its format_version parameter; null when SPINE has no such attribute or parameter.",
"warnings: what the dumper sees but does not map (unmapped_attribute, unlisted_class)."
],
"classes": {
"RecoParticle": {
Expand Down Expand Up @@ -8782,5 +8786,22 @@
"type_check": {
"method": "declared type vs numpy dtype of HDF5Writer.get_object_dtype(cls())",
"mismatches": []
}
},
"writer_format": {
"format_name": "spine_hdf5",
"legacy_format_version": 1,
"current_format_version": 2,
"supported_format_versions": [
1,
2
],
"default_format_version": 1
},
"warnings": [
{
"kind": "unmapped_attribute",
"name": "cat",
"detail": "FieldMetadata.cat is not read; set on RecoParticle.index, RecoParticle.orig_index, RecoParticle.fragment_ids, TruthParticle.index_adapt, TruthParticle.index_g4, TruthParticle.index, TruthParticle.orig_index, TruthParticle.fragment_ids, RecoInteraction.index, RecoInteraction.orig_index, TruthInteraction.index_adapt, TruthInteraction.index_g4, TruthInteraction.index, TruthInteraction.orig_index"
}
]
}
53 changes: 44 additions & 9 deletions tools/gen/EGM.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,26 @@ SPINE clone --dump_schema.py--> schema/spine-<version>.json --generate.py--> C++
- `--out FILE` (overrides `--out-dir`)
- `--classes NAME...` (default `DEFAULT_CLASSES`)
- `--strict`
- `--spine-sha SHA` (40 hexadecimal characters, recorded as `spine_sha`)
- `DEFAULT_CLASSES`: RecoParticle, TruthParticle, RecoInteraction,
TruthInteraction, RunInfo, Trigger, ImageMeta3D, Flash, CRTHit, FlashHypothesis.
- `EXCLUDED_CLASSES`: the classes of `spine.data` with `attr_metadata` that are
not dumped on purpose: RecoFragment, TruthFragment, Neutrino, ImageMeta2D,
Particle (larcv), ParticleLabel, and the bases DataBase and PosDataBase.
- `CLASS_ROLE`, `FLAG_NAMES` (derived, position, vector, index, categorical, pdg,
pdg_minus_one_valid, pointwise, lite_skip), `SCALAR_ELEMENT`.
pdg_minus_one_valid, pointwise, lite_skip), `MAPPED_ATTRIBUTES` (the flags but
`derived`, plus return_type, length, dtype, units, enum, reference,
reference_space, skip), `WRITER_FORMAT_ATTRIBUTES`, `SCALAR_ELEMENT`.
- Helpers: `declared_type`, `dump_class`, `collect_enums`, `scheme_tables`,
`field_doc`, `compare_writer`.
- Output keys: `schema_format` (1), `spine_version`, `generator`, `notes`,
`classes`, `enums`, `scheme_tables`, `type_check {method, mismatches}`.
`field_doc`, `compare_writer`, `writer_format`, `unmapped_attributes`,
`unlisted_classes`.
- Output keys: `schema_format` (1), `spine_version`, `spine_sha` (`null` without
`--spine-sha`), `generator`, `notes`, `classes`, `enums`, `scheme_tables`,
`type_check {method, mismatches}`, `writer_format {format_name,
legacy_format_version, current_format_version, supported_format_versions,
default_format_version}` and `warnings [{kind, name, detail}]`.
- Exit codes: 0 written; 1 SPINE could not be imported or read, or `--strict` and
a type mismatch; 2 usage error.

### `generate.py` (CLI; standard library only)

Expand Down Expand Up @@ -80,6 +92,9 @@ SPINE clone --dump_schema.py--> schema/spine-<version>.json --generate.py--> C++
`<root>/schema/spine-*.json`, otherwise it exits with
`generate: pass --schema (found N files in schema/)` [generate.py:main]. The
model describes one SPINE release.
- The model version (`SPINE_READER_MODEL_SPINE_VERSION`, the descriptors'
`schema_version`) names the schema the model was generated from, not the newest
SPINE it reads [generate.py:generate].
- Output is deterministic: generated files and globs are processed in sorted
order, so a diff after regeneration is a real change [generate.py:main].
- Stale-file rule: any `.hpp`/`.cpp` directly under `include/spine/model/` or
Expand Down Expand Up @@ -111,13 +126,33 @@ SPINE clone --dump_schema.py--> schema/spine-<version>.json --generate.py--> C++
dataclass defaults are checked against instance values. Mismatches are recorded
under `type_check.mismatches`. They fail the run (exit 1) only with `--strict`
[dump_schema.py:declared_type, dump_class, main].
- `dump_schema.py` stops on:
- an unknown `--classes` name (`dump_schema: spine.data has no class <NAME>`);
- `dump_schema.py` stops, writing nothing, on any exception while it imports or
reads SPINE. It prints one line to stderr, `dump_schema: <exception class>:
<message>`, and exits 1. The cases it raises itself:
- an unknown `--classes` name (`LookupError: spine.data has no class <NAME>`);
- a field type it cannot map (`TypeError`: an enum not declared `int`, an array
without a declared dtype, an unsupported type);
- a stored name that is neither a field nor a stored property (`RuntimeError`);
- a scheme whose interaction enum cannot be identified uniquely (`RuntimeError`)
- a scheme whose interaction enum cannot be identified uniquely (`RuntimeError`).

A SPINE that lacks an attribute the dumper reads gives `AttributeError` (for
example a `FieldMetadata` without `pdg`, or a SPINE older than `attr_metadata`)
[dump_schema.py:declared_type, dump_class, scheme_tables, main].
- `writer_format` reads each attribute with `getattr(HDF5Writer, name, None)`, and
`default_format_version` from the signature of `HDF5Writer.__init__`, so a
missing attribute or parameter is `null`, not an error. A renamed
`format_version` parameter gives `null` [dump_schema.py:writer_format].
- `warnings` lists, first the unmapped attributes in the order `FieldMetadata`
declares them, then the unlisted classes sorted by name:
- `unmapped_attribute`: each `FieldMetadata` attribute outside
`MAPPED_ATTRIBUTES`; `detail` names the dumped fields that set it (at SPINE
1.4.0, only `cat`);
- `unlisted_class`: each class of `vars(spine.data)` with `attr_metadata` that is
in neither `DEFAULT_CLASSES`, `--classes` nor `EXCLUDED_CLASSES`; an alias of a
listed class (`Meta` is `ImageMeta3D`) is listed.

A comparison of two dumps compares warnings by `(kind, name)`
[dump_schema.py:unmapped_attributes, unlisted_classes].
- Fields that `as_dict` does not store go to `skipped_fields`; fields dropped by
`as_dict(lite=True)` go to `lite_dropped` [dump_schema.py:dump_class].
- `check_names.py` exits 1 if any model identifier, after comments and strings are
Expand All @@ -128,8 +163,8 @@ SPINE clone --dump_schema.py--> schema/spine-<version>.json --generate.py--> C++

- `dump_schema.py` imports `spine.data`, `spine.version`, `HDF5Writer` and
`spine.constants.enums`. It reads SPINE source files (for the docstrings) and
writes the JSON with `mkdir(parents=True)`. It prints field counts and the
mismatch list to stdout.
writes the JSON with `mkdir(parents=True)`. It prints field counts, the
mismatch list, the writer format and the warnings to stdout.
- `generate.py` reads the schema. Unless `--check` is given, it deletes stale
files and writes only the files whose content changed. It prints
`SPINE <version>: N files, M written (C classes, E enums)`, where M counts the
Expand Down
26 changes: 24 additions & 2 deletions tools/gen/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ SPINE clone at a tag --dump_schema.py--> schema/spine-<version>.json --generate.
source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc13-opt/setup.sh
git clone -q --branch vX.Y.Z <SPINE repository> /some/dir/spine-X.Y.Z

PYTHONPATH=/some/dir/spine-X.Y.Z/src python3 tools/gen/dump_schema.py # -> schema/spine-X.Y.Z.json
PYTHONPATH=/some/dir/spine-X.Y.Z/src python3 tools/gen/dump_schema.py \
--spine-sha $(git -C /some/dir/spine-X.Y.Z rev-parse HEAD) # -> schema/spine-X.Y.Z.json
python3 tools/gen/generate.py --schema schema/spine-X.Y.Z.json # -> include/spine/model*, src/model
```

Expand All @@ -26,6 +27,11 @@ one): the model describes one release.
`generate.py --check` writes nothing and exits 1 if a committed generated file is stale. Output is
deterministic, so a diff after regeneration is a real change.

`dump_schema.py` exits 1 when SPINE cannot be imported or read, after one line on stderr:
`dump_schema: <exception class>: <message>` (for example `dump_schema: AttributeError:
'FieldMetadata' object has no attribute 'pdg'`). It writes nothing then. With `--strict`, a type
mismatch also exits 1.

## What to check afterwards

1. **The type check printed by `dump_schema.py`.** Each field's declared type (annotation, or
Expand All @@ -34,7 +40,9 @@ deterministic, so a diff after regeneration is a real change.
under `type_check` in the JSON. The same section also lists any field whose dataclass default
differs from the value on an instance. `--strict` makes mismatches an error.
2. **`git diff schema/`.** Fields added, removed or retyped, enum members, new `FieldMetadata`
flags. A class SPINE started storing is added to `DEFAULT_CLASSES` in `dump_schema.py`; the
flags. A change of `writer_format` (for example a new default layout) or a new entry under
`warnings` (an attribute the dumper does not map, a class it does not list) has no effect on the
generated C++, and needs a look all the same. A class SPINE started storing is added to `DEFAULT_CLASSES` in `dump_schema.py`; the
header, the source and the `SPINE_READER_FOR_EACH_CLASS` entry then appear by themselves. A
field or class SPINE removed disappears and `generate.py` deletes the stale files.
3. **`python3 tools/gen/check_names.py`.** No identifier of the model may be a macro of a common C,
Expand All @@ -59,6 +67,20 @@ deterministic, so a diff after regeneration is a real change.

## What is in the JSON

Top level:

| key | meaning |
| --- | --- |
| `schema_format`, `generator`, `notes` | the format of this file (1), the tool, and notes on reading it |
| `spine_version`, `spine_sha` | `spine.__version__`, and the commit given with `--spine-sha` (`null` without it) |
| `classes`, `enums`, `scheme_tables` | the model: what `generate.py` reads |
| `type_check` | the declared types against `HDF5Writer.get_object_dtype`, and dataclass defaults against instance values |
| `writer_format` | `format_name`, `legacy_format_version`, `current_format_version` and `supported_format_versions` of `HDF5Writer`, and `default_format_version`, the default of its `format_version` parameter; each is `null` when SPINE has no such attribute or parameter |
| `warnings` | `{kind, name, detail}` for what the dumper sees but does not map: `unmapped_attribute` (a `FieldMetadata` attribute outside `MAPPED_ATTRIBUTES`; at 1.4.0, `cat`) and `unlisted_class` (a class of `spine.data` with `attr_metadata` in neither `DEFAULT_CLASSES` nor `EXCLUDED_CLASSES`) |

`generate.py` ignores `spine_sha`, `type_check`, `writer_format` and `warnings`: they describe the
dump, for comparing two dumps, and do not change the generated C++.

`classes.<Name>.fields[]`, in the order of `as_dict(lite=False)` (the order of the HDF5 compound
members), minus fields with `skip=True`:

Expand Down
Loading