ci(memtrack): benchmark memtrack's own tracking overhead - #517
ci(memtrack): benchmark memtrack's own tracking overhead#517not-matthias wants to merge 5 commits into
Conversation
`codspeed-memtrack track` pays a fixed cost per invocation (BPF program load plus uprobe/uretprobe attaches) on top of the tracked command, and nothing measured it so far, so wall-clock regressions in that overhead went unnoticed. Add a walltime config with three exec targets covering distinct workloads (read-only, allocation-heavy, I/O-heavy) and a CI job that runs them with the CLI and memtrack built from source.
The listing and dd's stderr were captured into the runner log once per round, which made the uploaded log 4.2 MB of noise. The tracked command string is run through `bash -c`, so a redirect inside it works.
Run each walltime workload with rmap disabled and enabled so the rmap contribution can be measured directly.
Exec harness treats the first token as the executable, so invoke env to apply the rmap setting before starting memtrack.
Greptile SummaryThe PR adds a required Linux CI job that measures
Confidence Score: 5/5The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issue identified. The workflow installs and configures the same memtrack executable used by the configured targets, and the benchmark commands have bounded execution settings and compatible output behavior.
|
| Filename | Overview |
|---|---|
| .github/workflows/ci.yml | Adds the memtrack benchmark setup and execution job and makes it part of the required aggregate check; no actionable defect was established. |
| crates/memtrack/codspeed.yml | Defines six wall-time targets comparing RSS-only and RSS+rmap tracker overhead with bounded warmup and measurement times. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
CI[GitHub Actions job] --> Install[Install memtrack and grant capabilities]
Install --> Build[Build release CodSpeed CLI]
Build --> Runner[Run wall-time benchmark configuration]
Runner --> RSS[RSS-only ls, tar, and dd]
Runner --> RMap[RSS plus rmap ls, tar, and dd]
RSS --> Check[Required aggregate CI check]
RMap --> Check
Reviews (1): Last reviewed commit: "fix(memtrack): launch rmap benchmarks th..." | Re-trigger Greptile
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| 🆕 | WallTime | memtrack track dd |
N/A | 1.5 s | N/A |
| 🆕 | WallTime | memtrack track ls |
N/A | 1.5 s | N/A |
| 🆕 | WallTime | memtrack track tar |
N/A | 5 s | N/A |
| 🆕 | WallTime | memtrack track dd (rmap) |
N/A | 1.6 s | N/A |
| 🆕 | WallTime | memtrack track ls (rmap) |
N/A | 1.6 s | N/A |
| 🆕 | WallTime | memtrack track tar (rmap) |
N/A | 5 s | N/A |
Comparing memtrack-walltime-benches (a2d6c6f) with main (8868af5)
The tar benchmark's first measured round created a multi-GB archive from scratch, so its reported value was dominated by one-off disk allocation instead of tracking overhead. On a runner this turned a single warmup round into ~95s and made the RSS+rmap variant (which ran later and overwrote the already-allocated file) look 18x faster. Interleave the rmap variants so each pair runs back to back over the same page-cache state, and create the archives before measurement so every round measures steady-state overwrite I/O.
Add walltime benchmarks that measure
codspeed-memtrack's own tracking overhead.The memory benchmarks only report the tracked program's numbers, so the tracker's
fixed costs (BPF load, uprobe attach, ring-buffer drain, event encoding) were
invisible: regressions in the tracker itself looked like noise in the workload.
Each benchmark times a full
codspeed-memtrack trackinvocation of arepresentative command (
ls,dd,tar) through the walltime instrument.Every workload runs twice — RSS-only and RSS+rmap — so rmap's overhead is directly
comparable in CI instead of being benchmarked by hand on random machines. The
tracked commands redirect their output away and run with generous warmup/max
times, since one tracked run already pays the fixed BPF setup cost.
Stacked follow-up will use these benches to drive and verify tracking-overhead
reductions.
Refs COD-3179