Reported by a first external consumer (v1.3.0), confirmed twice (observe run + verify run). Method gap in run-scenario / create-observe-run-report.
Part 1 — a clean backend is not a clean run
odd_stack_reset gives a clean backend; the observation window then opens on an application process that has been running for an arbitrary time. Every cumulative counter/histogram read inside the window includes pre-window activity, while traces and logs are correctly window-scoped — the two signal families disagree about what "the run" is, and nothing in the method warns about it. Measured: a process 7 minutes older than the window with 2 pre-window failed jobs made the job counters and duration histograms unusable as run totals; the observer had to reconcile per-batch deltas and a byte-exact sum identity just to attribute points to the run.
Part 2 — restarting the process is not enough either: pin the instance identity
The verify run showed the sharper version. The backend was recreated, the app force-recreated ~60 s later — and in that gap the outgoing pre-fix process got one final periodic export, flushing its whole cumulative history into the brand-new store. A store meant to hold only post-fix data contained two service.instance.ids (the old one still carrying the pre-fix histogram boundaries — unambiguous proof). Unfiltered queries mix the two; two verification checks would have failed for reasons unrelated to the code under test. The instrumentation is blameless — service.instance.id is exactly what made the separation possible; the gap is that no field of the report records which instance its numbers belong to, and no step of the protocol pins it.
Suggested fix
run-scenario: state that a clean backend is not a clean run, and prescribe the ordering that avoids the overlap — restart the observed process first, then reset the backend (the reverse of what is natural; the reason is not guessable).create-observe-run-report: record the observed instance identity (service.instance.id or backend equivalent) and the process start time / restart-performed next to window in the frontmatter, and state that metric queries in the protocol must be qualified by it. A before-value that cannot be attributed to a process is not a before-value.
Reported by a first external consumer (v1.3.0), confirmed twice (observe run + verify run). Method gap in
run-scenario/create-observe-run-report.Part 1 — a clean backend is not a clean run
odd_stack_resetgives a clean backend; the observation window then opens on an application process that has been running for an arbitrary time. Every cumulative counter/histogram read inside the window includes pre-window activity, while traces and logs are correctly window-scoped — the two signal families disagree about what "the run" is, and nothing in the method warns about it. Measured: a process 7 minutes older than the window with 2 pre-window failed jobs made the job counters and duration histograms unusable as run totals; the observer had to reconcile per-batch deltas and a byte-exact sum identity just to attribute points to the run.Part 2 — restarting the process is not enough either: pin the instance identity
The verify run showed the sharper version. The backend was recreated, the app force-recreated ~60 s later — and in that gap the outgoing pre-fix process got one final periodic export, flushing its whole cumulative history into the brand-new store. A store meant to hold only post-fix data contained two
service.instance.ids (the old one still carrying the pre-fix histogram boundaries — unambiguous proof). Unfiltered queries mix the two; two verification checks would have failed for reasons unrelated to the code under test. The instrumentation is blameless —service.instance.idis exactly what made the separation possible; the gap is that no field of the report records which instance its numbers belong to, and no step of the protocol pins it.Suggested fix
run-scenario: state that a clean backend is not a clean run, and prescribe the ordering that avoids the overlap — restart the observed process first, then reset the backend (the reverse of what is natural; the reason is not guessable).create-observe-run-report: record the observed instance identity (service.instance.idor backend equivalent) and the process start time / restart-performed next towindowin the frontmatter, and state that metric queries in the protocol must be qualified by it. A before-value that cannot be attributed to a process is not a before-value.