Feature/noisepy eval - #3
Open
mdenolle wants to merge 7 commits into
Open
Conversation
… trim - parse_mseed: libmseed MS3TraceList fast path assembles contiguous segments in C (bench.mseed: 217 ms -> 37 ms, now faster than obspy.read at 42 ms; bit-identical output); per-record fallback kept for malformed v2 headers; collect_flags now opt-in; encoding taken from each segment's first record; num_segments now counts true segments (was: miniSEED records) - TraceBundle: segments(), to_dict(fill_value=) with true sample placement (validated exactly equal to obspy merge(method=1, fill_value=0) on gapped + overlap fixtures; later-segment-overwrites), trim(start_ns, end_ns), overlaps(); gappy default to_dict() warns - bundle_to_xarray routes through the same placement (NaN fill default) so the time coordinate is honest across gaps - S3OpenClient/S3AuthClient/FDSNMultiClient join chunks in submission order (was as_completed: non-deterministic byte order); bulk results sorted to submission order after live progress - SeisfetchClient.get_numpy/get_xarray: sample-precise window trim by default (trim=False restores whole-object behavior) - pymseed pinned >=0.6,<0.9; private-API sid fallback isolated+guarded - new fixtures (<25 KB each) + generator; 16 new tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…adapter, precision bank - benchmarks/: runner.py writes results/<tag>_<date>.json; suites.py (parse, cold_import, memory, footprint, s3_pull --live); render_results regenerates RESULTS.md. Committed m1-native results: parse 33.6 ms vs obspy 38.3 ms on the 11 MB day file (was 217 ms before Phase 1), tracemalloc peak 27.7 vs 52.0 MB, installed 80 vs 311 MB. Finding: cold import seisfetch 0.27 s vs obspy 0.13 s (top-level boto3 import). - seisfetch/contrib/noisepy_adapter.py: numpy/scipy ports of the obspy ops NoisePy's preprocess_raw uses at rm_resp=NO (check_sample_gaps, hann taper with in-place dtype semantics, merge(method=1,fill=0) via to_dict placement, obspy-recipe Fourier resample exact to the last ulp, trim(pad=True)), NpChannelData duck type, SeisfetchS3RawStore (URLs owned by seisfetch.s3), preprocess_raw_np full chain; [noisepy] extra (scipy only, no obspy) - to_dict(fill_value=): keep data dtype when fill fits (0 in float32 stays float32 — obspy merge semantics); promote to float64 otherwise - tests/precision/: parse identity vs obspy per segment across all encodings+gap topologies (obspy contiguous-run splits and 32 ns UTCDateTime float rounding normalized in the test harness); per-op equivalence EXACT for taper/merge/bandpass/resample/trim; full preprocess chain EXACT vs the obspy chain on gapped+float32 fixtures Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… report - benchmarks/noisepy_eval/run_ccf_eval.py: identical SCEDC bytes through (A) obspy.read + noisepy preprocess_raw and (B) seisfetch parse + adapter ports, then noisepy's own compute_fft/correlate. Result on CI.PASC 2022-01-02 (EN/EZ/NZ/ZZ daily CCFs): max abs diff 0.0 — bit-identical — and identical stretching-dv/v grid cells; results JSON committed. Integration-marked wrapper in tests/precision/. - adapter: segment_interpolate_np port; sub-sample alignment wired inside the resample branch only (mirrors noisepy placement) - seisfetch/__init__: transport imports (boto3/httpx) now lazy via PEP 562 — cold import 0.27 -> 0.10 s, 2.4x faster than obspy - benchmarks/docker/: Dockerfile.bench + run_matrix.sh (fargate-class 2cpu/4g, lambda-class 1g and 512m). Finding: obspy publishes no linux/aarch64 wheels, so arm64 (Graviton) installs must compile from source; seisfetch installs from wheels - docs/noisepy-obspy-replacement-report.md: full evaluation report; machine-matrix table pending the container runs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fargate-class (2cpu/4g), lambda-1g, lambda-512m runs committed. Both stacks parse a day file in 156 MB RSS -> 512 MB Lambda feasible; cold import seisfetch 1.5-1.7x faster everywhere; parse loses its native edge under CPU throttling (1.3-1.6x slower than obspy; documented honestly with the pymseed bulk-accessor lever). Verdict: justified — bit-identical CCFs, 80 vs 311 MB footprint, no aarch64 obspy wheels vs pure-wheel seisfetch install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds an evaluation-grade, ObsPy-free data-path for running NoisePy-style preprocessing on seisfetch-parsed miniSEED, alongside determinism fixes (byte/result ordering) and expanded precision/benchmark coverage to validate bit-level equivalence against ObsPy/NoisePy.
Changes:
- Extend
parse_mseedto emit true continuous segments (fast path via libmseed trace list + guarded per-record fallback) and add gap/overlap/trim + gap-awareto_dict(fill_value=...). - Make multi-source fetch/join outputs deterministic (submission order) across S3, FDSN multi-client, and bulk helpers; add client-level window trimming default.
- Add NoisePy adapter ports + precision/integration tests, benchmarks, and an evaluation report; introduce lazy top-level imports and a CI workflow.
Reviewed changes
Copilot reviewed 29 out of 37 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_segments_api.py | Validates new segment/gap/overlap/trim + to_dict(fill_value=...) semantics. |
| tests/test_s3_determinism.py | Tests deterministic multi-day S3 concatenation order and client-side trimming (moto). |
| tests/precision/test_preprocess_equivalence.py | Exact-equality checks for numpy/scipy ports vs ObsPy operations and full chain. |
| tests/precision/test_parse_identity.py | Asserts bit-identical decode vs ObsPy per segment across fixtures/encodings. |
| tests/precision/test_ccf_equivalence.py | Integration-marked end-to-end CCF equivalence harness runner check. |
| tests/precision/init.py | Declares precision test package. |
| tests/fixtures/make_fixtures.py | Script to generate committed miniSEED fixtures used by tests. |
| seisfetch/s3.py | Deterministic submission-order joining for concurrent S3 fetches. |
| seisfetch/fdsn.py | Deterministic submission-order joining across multiple FDSN providers. |
| seisfetch/convert.py | Segment-aware parsing, gap/overlap/trim APIs, gap-aware to_dict, xarray merge behavior. |
| seisfetch/contrib/noisepy_adapter.py | Adds ObsPy-free numpy/scipy ports + NoisePy-shaped adapters/store. |
| seisfetch/contrib/init.py | Declares contrib namespace as optional integrations. |
| seisfetch/client.py | Adds trim option (default on) for window-precise parsing results. |
| seisfetch/bulk.py | Ensures deterministic bulk result ordering (submission order) after concurrency. |
| seisfetch/init.py | Introduces PEP 562 lazy imports for transport layers to improve cold-start. |
| pyproject.toml | Pins pymseed range and adds noisepy optional extra (scipy). |
| docs/noisepy-obspy-replacement-report.md | Documents evaluation results, methodology, risks, and recommendation. |
| benchmarks/suites.py | Adds offline benchmark suites and supporting helpers. |
| benchmarks/runner.py | CLI runner to execute suites and persist results JSON. |
| benchmarks/results/m1-native_2026-08-03.json | Captured benchmark results for m1-native run. |
| benchmarks/results/lambda-512m_2026-08-03.json | Captured benchmark results under lambda-512m limits. |
| benchmarks/results/lambda-1g_2026-08-03.json | Captured benchmark results under lambda-1g limits. |
| benchmarks/results/fargate-class_2026-08-03.json | Captured benchmark results under fargate-class limits. |
| benchmarks/results/ccf_equivalence_m1_2026-08-03.json | Captured CCF equivalence metrics JSON. |
| benchmarks/RESULTS.md | Rendered benchmark summary markdown from results JSON. |
| benchmarks/render_results.py | Deterministic renderer for benchmarks/RESULTS.md. |
| benchmarks/noisepy_eval/run_ccf_eval.py | Harness to compare obspy+noisepy vs seisfetch+adapter through real noisepy. |
| benchmarks/docker/run_matrix.sh | Script to run benchmark matrix under cgroup limits via Docker. |
| benchmarks/docker/Dockerfile.bench | Benchmark container (installs both seisfetch and obspy). |
| .github/workflows/test.yml | Adds CI job running offline pytest matrix (ubuntu/macos, py3.10/3.12). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+219
to
+230
| # keep the data dtype when fill_value is exactly representable | ||
| # in it (0 in float32 stays float32 — obspy merge semantics); | ||
| # promote otherwise (NaN into int data -> float64) | ||
| data_dtype = segs[0].data.dtype | ||
| cast = np.asarray(fill_value).astype(data_dtype, casting="unsafe") | ||
| try: | ||
| fits = bool(cast == fill_value) or ( | ||
| np.isnan(fill_value) and np.isnan(cast) | ||
| ) | ||
| except (TypeError, ValueError): | ||
| fits = False | ||
| dtype = data_dtype if fits else np.result_type(data_dtype, np.float64) |
Comment on lines
+283
to
+289
| sr = cur.sampling_rate | ||
| if sr <= 0: | ||
| continue | ||
| sample_interval_ns = int(1e9 / sr) | ||
| expected_next_ns = cur.endtime_ns + sample_interval_ns | ||
| overlap_ns = expected_next_ns - nxt.starttime_ns | ||
| overlap_samples = overlap_ns / sample_interval_ns |
Comment on lines
+13
to
+16
| def _day_bytes(day_marker: int) -> bytes: | ||
| # deterministic distinct content per day via npts | ||
| np.random.seed(day_marker) | ||
| return make_mseed(network="IU", station="ANMO", npts=500 + day_marker) |
seisfetch/contrib/response.py (~450 lines, numpy + stdlib xml): - evalresp-equivalent evaluate_response (analog/Hz/digital PZ, FIR with DC normalization and CorrectionApplied phase, stage gains, (iw)^n unit conversion, CM/MM/NM scaling): max rel diff 1.6e-10 vs compiled evalresp on both CI.PASC epochs, all outputs - discovery, proven by perturbation: evalresp IGNORES the XML NormalizationFactor/Frequency and recomputes A0 at the STAGE GAIN's frequency (2007 epoch: f_norm 0.03 Hz vs gain freq 1.0 Hz reproduces evalresp's 0.997788169 scalar to 9 digits) - remove_response_np: full obspy Trace.remove_response port (SAC taper, _npts2nfft, pre_filt, water level) — 6.6e-16 of peak vs obspy on the real 6.9M-sample Tohoku day, and 1.9 s vs obspy's 3.6 s - translate_resp_np + damped_oscillator_response: SeisIO.jl-style translation (eps-guard, target-response stabilization, PZ-only) - mode='paz' quantified: 0.7-1.3% below 4 Hz, unusable >= 16 Hz - design doc docs/response-removal-design.md; 10 precision tests; 19 KB two-epoch real StationXML fixture Container parse gap (profiled, fixed): - root cause was np_datasamples.copy() of the borrowed C buffer (27-74 ms under cgroups), not per-segment Python; fixed by decoding via create_numpy_array_from_recordlist into a numpy-owned array: 21 ms in EVERY environment (obspy 32-39 ms), parse RSS 92 -> 37 MiB - benchmarks/profile_parse.py micro-harness; docs/pymseed-issue-draft.md (upstream enhancement proposal, NOT submitted); report hypothesis bullet corrected Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matrix rerun: parse now fastest in every environment (21.4 ms native / ~20 ms all containers vs obspy 32-37 ms). bench suite gains an untimed warmup call per timing loop (first-touch page faults skewed macOS min-of-5 by ~2x). Report tables updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.