Make the repo the source for the analysis scripts#6
Merged
Conversation
scripts/pipeline/analysis/ had forked. The versions actually used to process the July
2026 beam time lived outside the repo and had gained a scan registry, environment-driven
paths, and several new tools; the repo copies were stale (parentbeta_validate 133 vs 153
lines, parentbeta_reconstruct 214 vs 218). Anyone running from the repo got the old code.
This makes the repo the source.
Updated:
parentbeta_validate.py scan registry + env-driven paths; runs on any host/scan
parentbeta_reconstruct.py output prefix parameterised
Added (all driven by LAUE_WORK / LAUE_SCAN_DATA / LAUE_SCAN_{ALPHA,BETA} / LAUE_OUT_PREFIX):
run_analysis_chain.sh the whole sequence, one command per scan
null_model.py measures the random-orientation null ON THAT SCAN
empirical_gate.py re-scores the validated set against that measured null
exclusion_null.py measured null for the beta alpha-exclusion test
anchor_null.py tests whether the retained-beta anchor beats chance
variant_coherence.py spatial coherence of the variant map vs a shuffle null
regrain.py grains as CONTIGUOUS regions, not orientation-only clusters
tolerance_sensitivity.py how counts move with the clustering tolerance
big_grain_diagnostic.py is one large "grain" actually several?
big_grain_split_test.py do its lobes have distinct orientations?
collect_scan_metrics.py cross-scan summary JSON
scan_map.py unvalidated catalog map (was hardcoded to one scan)
catalog_figures.py report plates
validated_figures.py
Naming: the id6_* prefix named a single scan and these tools are scan-agnostic, so they
follow the directory's existing convention (null_model.py, not id6_null_model.py). The
"id6_10x10" registry key is kept for backward compatibility with existing outputs and now
carries a comment noting it is specimen ID26 at 20x20 um, not ID6 at 10x10.
README gains the chain documentation, the three methodological rules it enforces (measure
the null per scan; a grain is contiguous; check that a corroborating statistic beats
chance), and a gotchas section covering tcsh noclobber, ARG_MAX, CUDA_DEVICE_ORDER, pgrep
self-match, late-batch outputs, and blank frames.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/pipeline/analysis/had forked. The versions actually used to process the July 2026beam time (13 scans, ~150k frames, four machines) lived outside the repo and had gained a
scan registry, environment-driven paths and several new tools. The repo copies were stale —
parentbeta_validate133 vs 153 lines,parentbeta_reconstruct214 vs 218 — so anyonerunning from the repo got the old code.
This makes the repo the source.
Updated
parentbeta_validate.py— scan registry + env-driven paths; runs on any host, any scanparentbeta_reconstruct.py— output prefix parameterisedAdded
All driven by
LAUE_WORK/LAUE_SCAN_DATA/LAUE_SCAN_{ALPHA,BETA}/LAUE_OUT_PREFIX:run_analysis_chain.shnull_model.pyempirical_gate.pyexclusion_null.pyanchor_null.pyvariant_coherence.pyregrain.pytolerance_sensitivity.pybig_grain_diagnostic.py,big_grain_split_test.pycollect_scan_metrics.pyscan_map.pycatalog_figures.py,validated_figures.pyWhy these matter, not just that they exist
Three findings from this beam time are encoded here, each of which changed a number we would
otherwise have published:
p<1e-4gate under-rejects. It assumes uniformly scattered peaks; realLaue peak fields are clustered, so the true null has much heavier tails. Across nine scans
the measured α null maximum ranged 14–17 and β 11–16 — one inherited value misstates the
others. On one dataset this moved the defensible β count by two orders of magnitude.
merges spatially disjoint regions; the largest "grain" in one scan proved to be two
crystallites 0.44° apart (73σ past a label-shuffle null). Splitting on connectivity moved
α 325→614 and β 40→27 — opposite directions, which is what shows it is a definitional fix
rather than a tuning knob.
with few β clusters and worthless with many: against 2,537 candidates a random
orientation lands within 1.74° of one 9% of the time.
Naming
The
id6_*prefix named one scan; these tools are scan-agnostic, so they now follow thedirectory's existing convention (
null_model.py, notid6_null_model.py). Theid6_10x10registry key is kept for backward compatibility with existing outputs and carries a comment
noting it is specimen ID26 at 20×20 µm — not ID6 at 10×10. The folder name is wrong;
seven of these scans are mislabelled the same way.
README
Adds the chain documentation, the three rules above, and a gotchas section: tcsh
noclobbersilently refusing writes,
ls | wc -lreturning 0 past ARG_MAX,ls -dtmatching.launch.log,CUDA_DEVICE_ORDER,pgrep -fmatching its own command line, outputsappearing in one late batch, and blank frames legitimately producing no output.
Depends on nothing in #5, but both come from the same campaign.