Skip to content

feat(memory): apply kernel memory tunables before memory-mode runs - #507

Open
not-matthias wants to merge 3 commits into
mainfrom
cod-3300-kernel-memory-tunables
Open

feat(memory): apply kernel memory tunables before memory-mode runs#507
not-matthias wants to merge 3 commits into
mainfrom
cod-3300-kernel-memory-tunables

Conversation

@not-matthias

@not-matthias not-matthias commented Aug 18, 2026

Copy link
Copy Markdown
Member

Benchmark repos running codspeed --mode memory currently need a hand-written CI step to stabilise kernel memory behaviour before measuring: disable THP, zero vm.compaction_proactiveness / vm.swappiness / kernel.numa_balancing, swapoff -a, and drop the page cache before each suite.

The runner now does this itself at the start of every memory-mode execution.

Behaviour

  • Applied only when a CI run environment is detected, so developer machines are not silently mutated. Skipped entirely when privileges cannot be elevated without a password prompt.
  • Best effort: a knob that cannot be applied is a warning, never fatal.
  • MemoryTunables::apply() returns a guard holding the previous value of every knob it changed, and restores it on drop — including on the error paths, where teardown() never runs. Only knobs that were not already at the target are captured, and THP is restored to the exact mode it held (madvise stays madvise).
  • swapoff -a is guarded: skipped on zram devices (it permanently resets their disksize) and whenever the swapped pages would not fit in MemAvailable. Restoring re-enables the recorded entries one by one, since swapon -a would miss a swap file absent from /etc/fstab.
  • The page cache drop has no previous state to restore; the node is a write-only trigger.
  • Applied in run() rather than setup(), so the page cache is dropped before every suite and --skip-setup does not bypass it. The sysctl/THP writes are no-ops on later suites since ensure_sysctl compares before writing.

Commits

  1. refactor(executor): move linux_sysctl into executor helpers — pure move, the sysctl primitive is no longer walltime-specific.
  2. feat(memory): apply kernel memory tunables before memory-mode runs

Verification

  • cargo clippy --all-targets clean, cargo test --lib (pre-existing failures on this host are the sudo-requiring tests, which write kernel.kptr_restrict and are identical on main).
  • Local non-CI run logs Not running in CI, skipping kernel memory tunables; host THP unchanged.
  • Privileged container: THP enabled/defrag become [never], vm.compaction_proactiveness = 0, vm.swappiness = 0, and the swap guard correctly logged Leaving swap enabled: swapped pages do not fit in available memory.

@codspeed-hq

codspeed-hq Bot commented Aug 18, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks


Comparing cod-3300-kernel-memory-tunables (fc2fe71) with main (8868af5)

Open in CodSpeed

@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch from febfce2 to a219b76 Compare August 19, 2026 15:40
@not-matthias
not-matthias marked this pull request as ready for review August 19, 2026 15:45
@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch from a219b76 to 3568cec Compare August 19, 2026 15:46
@greptile-apps

greptile-apps Bot commented Aug 19, 2026

Copy link
Copy Markdown

Greptile Summary

The PR applies and restores Linux kernel memory tunables around memory-mode benchmark runs and centralizes reusable sysctl guards for profiling. The current restoration path does not preserve swap priorities, while privileged local runs remain able to mutate host memory state.

  • Adds best-effort THP, swap, and page-cache stabilization around memory runs.
  • Records changed THP and swap state in a drop guard.
  • Moves profiling sysctl handling into shared executor helpers and restores prior values.

Confidence Score: 3/5

The PR is not yet safe to merge because privileged local runs mutate host memory state and swap restoration changes non-default priorities.

Memory tuning is reached locally whenever root or passwordless sudo is available, including an irreversible page-cache drop, and restored swap areas lose their configured priority because only their paths are recorded.

Files Needing Attention: src/executor/memory/tunables.rs and src/executor/memory/executor.rs

Important Files Changed

Filename Overview
src/executor/memory/tunables.rs Adds host memory tuning and restoration, but swap restoration loses each area's original priority and the existing local-run admission defect remains.
src/executor/memory/executor.rs Applies the tuning guard for every memory execution, including privileged local runs.
src/executor/helpers/linux_sysctl.rs Centralizes sysctl mutation behind a guard that restores changed values on drop.
src/executor/wall_time/executor.rs Retains profiling sysctl guards on the executor so prior values survive setup and are restored when the executor is dropped.

Sequence Diagram

sequenceDiagram
  participant E as MemoryExecutor
  participant T as MemoryTunables
  participant K as Linux kernel
  participant B as Benchmark
  E->>T: apply()
  T->>K: Set THP and disable eligible swap
  T->>K: Drop page cache
  E->>B: Run memory benchmark
  B-->>E: Exit
  E->>T: Drop guard
  T->>K: swapon(path)
  Note over T,K: Original swap priority is not restored
  T->>K: Restore THP mode
Loading

Fix all with Greploop Fix All in Claude Code Fix All in Codex

Reviews (3): Last reviewed commit: "fix(walltime): restore profiling sysctls" | Re-trigger Greptile

Comment thread src/run_environment/mod.rs Outdated
Comment thread src/executor/memory/tunables.rs Outdated
Comment thread src/executor/memory/tunables.rs
Comment thread src/executor/memory/tunables.rs
Comment thread src/executor/memory/tunables.rs Outdated
Comment thread src/executor/memory/tunables.rs Outdated
Comment thread src/executor/memory/tunables.rs Outdated
@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch 2 times, most recently from 755df11 to 8bfb28b Compare August 24, 2026 13:07
The sysctl read-then-write-with-sudo primitive is not walltime-specific;
memory mode needs it too.
Disables transparent huge pages, sets vm.compaction_proactiveness,
vm.swappiness and kernel.numa_balancing to 0, disables swap and drops the
page cache, so benchmark repos no longer need a hand-written CI step.

Applied only in CI, best-effort: a knob that cannot be set is a warning.
swapoff is skipped on zram devices and whenever the swapped pages would not
fit in available memory.
@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch from 77ad6fc to 87a02e9 Compare August 24, 2026 17:30

@art049 art049 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it was the case before but it hurts readability a bit

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to gate this whole mod behind the feature flag but not every single line, because now we end up gating almost every single line with #[cfg(target_os = "linux")] 😅

Comment thread src/executor/memory/tunables.rs
Comment thread src/executor/memory/tunables.rs
@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch 2 times, most recently from dcd2e8f to 758c564 Compare August 26, 2026 13:54
@not-matthias

Copy link
Copy Markdown
Member Author

@greptileai review again

Comment thread src/executor/memory/tunables.rs Outdated
Keep the original profiler sysctl values in the walltime executor and restore them when it is dropped. This prevents a local or containerized run from leaving host-global profiling access enabled.
@not-matthias
not-matthias force-pushed the cod-3300-kernel-memory-tunables branch from 758c564 to fc2fe71 Compare August 26, 2026 14:34
@not-matthias
not-matthias requested a review from art049 August 26, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants