Skip to content

Latest commit

 

History

History
107 lines (93 loc) · 6.48 KB

protocol.md

File metadata and controls

107 lines (93 loc) · 6.48 KB

Historical experiment protocol and limits

The default is a frozen historical refit, not a methodological improvement. The original traditional ML inference functions, train-only augmentation helpers, group-mask mapping, metrics and genetic operators are reused through PrecomputedTaskData. No descriptor extraction, data filtering or new splitting occurs in the public workflow.

IS uses 623 target-complete Fe-site rows; QS, OS and SPFE use 624. Molecular SP uses the 376-row pure-Fe subset, partitioned into odd (274) and even (102) multiplicity subsets. IS is corrected to 80 K; QS is absolute splitting in mm/s. SP and SPFE are 2S+1. OS and SP mappings merge some physical labels into a shared encoded class. Both observed labels and actual encodings are kept.

The exported five-fold PredefinedSplit assignments are authoritative. Rows marked -1 are always in training, including any historical rare-class treatment. Alternative DFT/XRD features enter training only if their original parent is in that fold's training set. Tests remain unaugmented. Original train/validation roles are exported even though frozen ML refits use the historical five-fold evaluation.

Feature blocks and selected columns retain source precision and order. Historical SOAP global PCA and independently fitted augmentation PCA are retained. Odd/even PCA bases must not be interchanged. These arrays must not be described as fold-local PCA or used to claim a nested-validation result. The private nested-validation workflow is a separate method and is not silently substituted here.

The original site-level splits allow different Fe sites of a molecule to appear in training and test sets. Parameter/feature selection used the same fold system as historical evaluation. These facts limit generalization claims and can lead to optimistic estimates; this release does not change them. A future molecule-grouped or nested experiment must be reported separately.

Known unresolved historical details

  • SPFE UMA-AUG: corrected frozen inputs use an explicitly verified existing XRD SOAP-PCA cache. All saved fold metrics now reproduce, including third-fold accuracy 0.728 and mean 0.7500774194. The original SPFE augmentation cache is absent, so identity to that missing historical matrix remains unproven. No PCA or other feature generation occurs in public training.
  • SP DFT with-MB: the author confirmed that the paper row reuses the full 021 configuration, selected features and model parameters, shared with the DFT without-MB SP row. Both give accuracy 0.9150766019. The public ID remains SP_dft_with_mb to identify the paper row; experiment.json records the shared provenance. This is not an independent SP feature-ablation/optimization result. The original private 031 search (0.9068401594) and older public local results are preserved as a separate historical run.
  • IS calibration: frozen fold coefficients reproduce saved predictions exactly, but fitting ordinary least squares on current training rows produces different coefficients. Its original fitting procedure needs author confirmation. Default evaluation therefore uses the original coefficients.
  • QS calibration files encode a direct computed prediction, with slope 1 and intercept 0. They are not treated as newly fitted affine calibrations.
  • Historical GA random seeds/initial populations are missing. Frozen selections are preserved; new searches carry their own seed and mode.
  • External historical OS percentages use the numerically reported physical predictions (30 UMA/15 SC-XRD). New encoded evaluation includes the "other" class (31/16). The source plotting script contains a special FeOS judgment for original entry 68. Historical records and fresh scoring stay separate.
  • Plotting inputs labeled UMA do not all use the same training contract: exact binding establishes UMA-AUG for IS/QS and UMA for the three classification tasks. The original normalization is preserved.

Only precomputed-feature ML reproduction is covered. This dataset does not independently reproduce literature extraction, feature generation from original structures, quantum calculations, or the original end-to-end timing claims. Derived descriptors may still reveal information about source molecules.

Validation of the included inputs

The current complete input set contains 283 files. All 20 frozen experiment contracts were fitted with public code, producing 120 fold models; every saved fold metric agrees within 1e-10 (largest observed difference 1.11e-16). The four frozen-coefficient baselines reproduce the saved predictions exactly. New public models predict all 31 UMA cases and 16 SC-XRD overlaps with zero per-fold difference from the exported model-reference predictions. The 85 available corresponding older fold models also have identical learned tree and ensemble parameters to their refits.

For SPFE UMA-AUG, the corrected 514-by-6 XRD SOAP-PCA block was copied from an explicitly verified existing cache without recomputing PCA. Only that block and its corresponding selected matrix changed from the earlier candidate. All other 206 primary experiment/calibration files were unchanged. Agreement with recorded metrics does not prove byte identity to the absent historical SPFE augmentation matrix.

The default public data are complete and require no private project or feature-generation step. Input validation records remain inside data/features/validation.json and data/features/model_verification.json. Detailed development reports are kept only in an ignored local archive. Your new model and analysis outputs are saved under the --output directories. The table mapping used by summarize is moss_predict/configs/paper_tables.json; there is no second root-level copy. The current SP DFT-with-MB paper contract uses author-confirmed shared 021 inputs rather than the independent 031 search. Only that experiment's public contract and data metadata were changed; the remaining 19 experiment inputs are unchanged. Historical numerical verification records in earlier local directories describe their original input versions.

Engineering compatibility

The Figure 6 helper resolves its default project root using parents[1], matching its actual location under scripts/. Explicit and default input paths now refer to data/features/figure6_inputs. The GA --init-mask path reserves the seed slot before drawing population-1 masks, matching the original random draw order and subsequent random state. Genetic operators and scientific settings are unchanged; historical unknown seeds still prevent replaying the old search.