Skip to content

ci(memtrack): benchmark memtrack's own tracking overhead - #517

Open
not-matthias wants to merge 5 commits into
mainfrom
memtrack-walltime-benches
Open

ci(memtrack): benchmark memtrack's own tracking overhead#517
not-matthias wants to merge 5 commits into
mainfrom
memtrack-walltime-benches

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Add walltime benchmarks that measure codspeed-memtrack's own tracking overhead.

The memory benchmarks only report the tracked program's numbers, so the tracker's
fixed costs (BPF load, uprobe attach, ring-buffer drain, event encoding) were
invisible: regressions in the tracker itself looked like noise in the workload.
Each benchmark times a full codspeed-memtrack track invocation of a
representative command (ls, dd, tar) through the walltime instrument.

Every workload runs twice — RSS-only and RSS+rmap — so rmap's overhead is directly
comparable in CI instead of being benchmarked by hand on random machines. The
tracked commands redirect their output away and run with generous warmup/max
times, since one tracked run already pays the fixed BPF setup cost.

Stacked follow-up will use these benches to drive and verify tracking-overhead
reductions.

Refs COD-3179

`codspeed-memtrack track` pays a fixed cost per invocation (BPF program
load plus uprobe/uretprobe attaches) on top of the tracked command, and
nothing measured it so far, so wall-clock regressions in that overhead
went unnoticed.

Add a walltime config with three exec targets covering distinct
workloads (read-only, allocation-heavy, I/O-heavy) and a CI job that
runs them with the CLI and memtrack built from source.
The listing and dd's stderr were captured into the runner log once per
round, which made the uploaded log 4.2 MB of noise. The tracked command
string is run through `bash -c`, so a redirect inside it works.
Run each walltime workload with rmap disabled and enabled so the rmap contribution can be measured directly.
Exec harness treats the first token as the executable, so invoke env to apply the rmap setting before starting memtrack.
@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds a required Linux CI job that measures codspeed-memtrack startup and tracking overhead through the wall-time runner.

  • Installs memtrack, grants its BPF capabilities, and builds the release CodSpeed CLI.
  • Adds RSS-only and RSS+rmap benchmarks for ls, tar, and dd.
  • Makes the new benchmark job a dependency of the aggregate CI check.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issue identified.

The workflow installs and configures the same memtrack executable used by the configured targets, and the benchmark commands have bounded execution settings and compatible output behavior.

Important Files Changed

Filename Overview
.github/workflows/ci.yml Adds the memtrack benchmark setup and execution job and makes it part of the required aggregate check; no actionable defect was established.
crates/memtrack/codspeed.yml Defines six wall-time targets comparing RSS-only and RSS+rmap tracker overhead with bounded warmup and measurement times.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CI[GitHub Actions job] --> Install[Install memtrack and grant capabilities]
  Install --> Build[Build release CodSpeed CLI]
  Build --> Runner[Run wall-time benchmark configuration]
  Runner --> RSS[RSS-only ls, tar, and dd]
  Runner --> RMap[RSS plus rmap ls, tar, and dd]
  RSS --> Check[Required aggregate CI check]
  RMap --> Check
Loading

Reviews (1): Last reviewed commit: "fix(memtrack): launch rmap benchmarks th..." | Re-trigger Greptile

@codspeed-hq

codspeed-hq Bot commented Aug 25, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks
🆕 6 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime memtrack track dd N/A 1.5 s N/A
🆕 WallTime memtrack track ls N/A 1.5 s N/A
🆕 WallTime memtrack track tar N/A 5 s N/A
🆕 WallTime memtrack track dd (rmap) N/A 1.6 s N/A
🆕 WallTime memtrack track ls (rmap) N/A 1.6 s N/A
🆕 WallTime memtrack track tar (rmap) N/A 5 s N/A

Comparing memtrack-walltime-benches (a2d6c6f) with main (8868af5)

Open in CodSpeed

The tar benchmark's first measured round created a multi-GB archive from
scratch, so its reported value was dominated by one-off disk allocation
instead of tracking overhead. On a runner this turned a single warmup
round into ~95s and made the RSS+rmap variant (which ran later and
overwrote the already-allocated file) look 18x faster.

Interleave the rmap variants so each pair runs back to back over the same
page-cache state, and create the archives before measurement so every
round measures steady-state overwrite I/O.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant