Skip to content

data_format.md

Latest commit

 

History

History
59 lines (52 loc) · 3.41 KB

data_format.md

File metadata and controls

59 lines (52 loc) · 3.41 KB
Raw

Derived data format

The versioned data/features/ directory contains the complete processed inputs: NPZ, CSV, JSON, Markdown and checksum files. Every NPZ loads with allow_pickle=False. Normal use requires no ZIP or separate data download.

  • manifest.json: format version, file inventory, hashes and experiment list.
  • experiments//experiment.json: frozen model parameters, group mask, selected feature names, GA configuration and historical fold metrics.
  • experiments///features.npz: ordered block_000... arrays, X_selected, selected_columns, compound_id, fe_idx, y_observed, y, y_encoded, test_fold and train_val_fold.
  • samples.csv: original compound IDs, Fe indices and exact row order.
  • feature_manifest.json: block names/types, dtypes, column ranges, groups, selected columns, origin and per-array hashes.
  • labels.json: target definition, units and many-to-one mappings.
  • splits.json: explicit train/test/validation indices and compound IDs.
  • augmentations//: derived blocks and parent_indices, sample manifest, training-fold eligibility and the historical PCA policy.
  • baselines/: processed calibration inputs, labels, original per-fold coefficients and saved predictions; no calculation output files.
  • external/: 31-case paper mapping, processed historical predictions, and selected feature vectors for five tasks and 31 UMA/16 SC-XRD structures. Each subset identifies its public training contract and encoded labels. reference_fold_predictions describe current persisted-model predictions, not silently substituted manuscript predictions.
  • historical_shap/: 16 numeric SHAP matrices with exactly verified sample and feature bindings. Missing/unbound artifacts are enumerated in index.json.
  • figure6_inputs/: six exactly bound historical category-plot inputs and an explicit provenance ledger.
  • review_notes.json, validation.json, model_verification.json: historical discrepancies, validation scope and fixed-parameter checks.

Numeric feature arrays are not rounded, normalized again, shuffled, noised or converted to lower precision. Unicode arrays replace only internal pickled string metadata. The full candidate pool and selected matrix are both kept.

The private exporter uses original V4 label/split methods and augmentors without invoking the normal feature-extracting TaskData constructor. Main cache IDs must agree; augmentation identity is reconstructed from original_indices. The known stale augmentation ID sidecars are not used as row labels. Missing feature calculations reuse existing descriptor code only; expensive DFT/UMA calculations are not rerun. Source paths, model objects, original summaries, structures, references and point-cloud sidecars are excluded. Detailed private input hashes/paths remain in a sibling private audit file, outside the public dataset.

The public adapter validates sample IDs, Fe indices, encoded labels, block dimensions, selected-column ordering, saved folds and augmentation parents. File checksums additionally reject added/changed files. Archive extraction rejects path traversal and symlinks.

Included, self-contained inputs

The repository includes this dataset under data/features/. Training reads the actual NPZ arrays, labels, partitions and JSON contracts. No private input recipe, cache resolution, descriptor extraction or user-supplied integrity token is part of the public training workflow. Automatic file/array checks detect accidental input changes.