Vectorized fast paths: trailing stack, moving-reference stretching, filter-once ensembles - #34
Merged
Merged
Conversation
run_pipeline(..., return_cc=True) returns (dvv, valid, cc) with the per-epoch stretching correlation coefficient, the input needed by uq_measurement.weaver_stretching_error. CC is collected for the fixed and moving references with the stretching estimator (the moving loop was already computing it and discarding it); NaN for other estimators and the inversion reference. Behavior-preserving: the default two-tuple return is unchanged, and CC-gating stays fixed-reference-only (the moving-reference CC is returned for error modelling but does not alter the valid mask). Verified against the golden expected-metrics suite (22 passed). Motivation: the noisepy-dvv-cloud pipeline needs the real per-epoch CC to replace a placeholder in its dvv_err_within (Weaver/Clarke) column. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-once ensembles Three performance fast paths, each verified to reproduce the per-day loop it replaces to ~1e-15 in dv/v (regression tests at rtol=0, atol=1e-12): - _trailing_stack: difference of float64 cumulative sums, O(ndays*nlag) independent of the stack length instead of O(ndays*k*nlag) (~2.3x at k=45). - measure_stretching_trailing: vectorized stretching against a trailing reference. The stretched sample positions t/(1+eps) are data-independent, so the linear-interpolation gather indices/weights are computed once per epsilon and applied to all days at once; trailing references come from a cumulative sum and the band-pass runs once over the whole matrix. deviations._moving_reference dispatches to it for "stretching (TS)" (measured 4.7x on the 3-year volcano synthetic, 12.2 -> 2.6 s), keeping the generic per-day loop for other estimators. collect_cc behavior from the return_cc branch is preserved. - run_pipeline(..., prefiltered=True): callers evaluating several stack/reference variants at the same band can band-pass the raw CCF matrix once; the estimators skip their internal band-pass. Exact because the band-pass is linear and commutes with linear stacking; restricted to the estimators whose band usage is that one linear filter (stretching, WCC, DTW, MWCS), ValueError otherwise. Combined, a 5-member same-band ensemble drops ~4x per pair-band. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot flagged assert_allclose(cc_b, cc_a, ...) as failing on the all-NaN cc arrays (inversion/mwcs cases, where no CC is collected) because equal_nan defaults to False. That's true of plain np.allclose, but numpy.testing.assert_allclose already defaults equal_nan=True -- verified all 4 parametrized cases (fixed/moving/inversion/mwcs) were already passing. Not a real bug, but making the default explicit (with a comment on why) so the next reader doesn't hit the same false alarm.
Same issue as codameter#32: this file grants Skill(deep-research) permissions, unrelated to the vectorized fast-paths change, isn't tracked on master, and doesn't belong bundled in here.
There was a problem hiding this comment.
🟢 Ready to approve
The changes are well-tested and behavior-preserving; only a small non-blocking performance nit was identified.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR adds several vectorized “fast paths” for dv/v measurement pipelines and exposes two new pipeline options (return_cc, prefiltered) to support coherence-based error models and filter-once ensembles, while keeping numerical output effectively unchanged (to float-rounding).
Changes:
- Adds
run_pipeline(..., return_cc=True)to optionally return per-epoch stretching CC (fixed and moving references), with fixed-reference-only CC gating preserved. - Adds
run_pipeline(..., prefiltered=True)plusprefilteredplumbing through applicable estimators to enable filter-once ensemble execution (and rejects wavelet estimators). - Implements vectorized moving-reference stretching (
measure_stretching_trailing) and replaces_trailing_stackwith a cumulative-sum-based implementation; adds regression tests covering all fast paths.
File summaries
| File | Description |
|---|---|
| tests/test_deviations.py | Adds regression tests for the new fast paths, return_cc, and prefiltered behavior (including wavelet rejection). |
| src/codameter/synthetic_demo.py | Adds prefiltered support to several estimators and introduces measure_stretching_trailing; rewrites _trailing_stack using cumulative sums. |
| src/codameter/deviations.py | Wires in return_cc/prefiltered, adds estimator allowlist for prefiltering, and dispatches moving-reference stretching to the new vectorized implementation. |
| CHANGELOG.md | Documents the new options and performance optimizations under Unreleased. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Comment on lines
109
to
112
| ndays = ccfs.shape[0] | ||
| out = np.full(ndays, np.nan) | ||
| cc_out = np.full(ndays, np.nan) | ||
| for d in range(ref_days, ndays): |
…ict from v0.3.0 release) # Conflicts: # CHANGELOG.md # src/codameter/deviations.py # tests/test_deviations.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the three performance optimizations verified bit-exact by prototypes during the 2026-08-02 audit (max dv/v difference ~1e-15 vs the current code on the 1095-day volcano synthetic).
Supersedes #33: that PR was stacked on #32 (
feat/run-pipeline-return-cc) and was supposed to auto-retarget tomasteronce #32 merged, but--delete-branchon #32's squash-merge deleted the base branch before that could happen, and GitHub auto-closed #33 as a result (base branch gone). Same branch, same commits, just re-opened directly againstmaster.Changes
_trailing_stackvia cumulative sums (synthetic_demo.py). Difference of float64 cumsums: O(ndays x nlag) independent of stack length, replacing the O(ndays x k x nlag) per-day loop. Measured 2.2x at k=45 (roughly at parity below k~15, where the loop was already cheap).Vectorized moving-reference stretching (
measure_stretching_trailing, dispatched fromdeviations._moving_referencefor"stretching (TS)"). The stretched sample positionst/(1+eps)are data-independent, so the linear-interpolation gather indices/weights are computed once per epsilon and applied to every day's band-passed trailing reference at once; trailing references are built by cumsum and the band-pass runs once over the whole matrix (2 FFTs total instead of 2 per day). Measured 4.7x on the 3-year volcano synthetic (12.2 -> 2.6 s), max |dv/v| difference 1.4e-15. Other estimators keep the generic per-day loop;collect_ccfrom Add return_cc option to run_pipeline #32 is preserved (regression-tested).Filter-once ensembles:
run_pipeline(..., prefiltered=True). Band-passing is linear, so it commutes exactly with linear stacking; callers evaluating several stack/reference variants at the same band can band-pass the raw CCF matrix once and the estimators skip their internal band-pass. Only valid at an identical band and only for estimators whose band usage is that one linear filter (stretching, WCC, DTW, MWCS); the wavelet estimators raiseValueError. Measured 1.7x on a 5-member same-band ensemble on top of the other fast paths.Combined effect on a 5-member ensemble: ~4x per pair-band.
Tests
TestFastPathRegressionsintests/test_deviations.py: each fast path vs the replaced per-day loop (replicated inline), andprefiltered=Truevs the internal band-pass for fixed / moving / inversion references and MWCS -- all asserted atrtol=0, atol=1e-12(observed ~1e-15), plus theValueErrorfor wavelet estimators..claude/settings.json(out-of-scope, unrelated permissions grant) dropped; madeequal_nan=Trueexplicit in the prefiltered-CC regression test (verifiednumpy.testing.assert_allclosealready defaults to it -- not a real bug, but silences a legitimate-looking false alarm for the next reader).CHANGELOG updated under Unreleased.