[lvaroqui]

Make exec-harness declare the pid of the command it benchmarks, so its own spawning cost can be left out of simulation results.

Since #531, exec-harness turns instrumentation on in its own process and spawns the command, which inherits that state across fork and exec. The dumped part therefore also holds the harness's cost of spawning the command and waiting for it. That is a fixed overhead on every benchmark, about 207k instructions per command locally.

Changes:

• instrument-hooks binding: new set_executed_benchmark_for_pid(pid, uri). set_executed_benchmark keeps its behavior and delegates to it.
• exec-harness: spawns the command and passes its pid, instead of calling Command::status(). In memory mode the pid only travels to the runner's FIFO, where nothing reads it for exec-harness.
• Submodule bump: feat(valgrind): declare the pid a benchmark ran in instrument-hooks#32 writes desc: Benchmark pid: <pid> in the part when the pid is not the caller's. This bump also brings the thread-safe C API exports and the callgrind_toggle_collect helper.

Checked locally under the patched valgrind, with the runner's simulation flags on sh -c '/bin/true; /bin/true; :':

part: 2
desc: Spawned pid: 289126
desc: Benchmark pid: 289126
desc: Trigger: Client Request: exec_harness::true_twice

Still to do before this is ready:

• Bump the valgrind pin to 0codspeed8 once feat(callgrind): add CALLGRIND_ADD_DESC client request valgrind-codspeed#43 is released. Until then the request is ignored with a warning and results keep the harness's cost.
• Repoint the submodule once feat(valgrind): declare the pid a benchmark ran in instrument-hooks#32 merges.
• The backend support for leaving out the declared launcher cost isn't released yet.

Closes COD-3722

[codspeed]

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 41 untouched benchmarks
⏩ 6 skipped benchmarks¹

---

_{Comparing cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation (fbb67ac) with spike/cod-3440-memtrack-musl (fd51df2)}

Footnotes

1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge.

Summary

exec-harness now records the spawned command’s PID with its benchmark URI.

• Simulation profiles name the process that ran each command.
• Valgrind allows the runner executable’s file capabilities.

No actionable issues were found. lvaroqui acknowledges that backend support for removing the harness’s cost is not released yet. The description also lists pending Valgrind and submodule work; the current code already pins 0codspeed9 and the updated hooks revision.

Diagram

sequenceDiagram
    participant Harness as exec-harness
    participant Command as Benchmark command
    participant Hooks as Instrument hooks
    participant Valgrind
    Harness->>Hooks: Start benchmark
    Harness->>Command: Spawn and save PID
    Command-->>Harness: Exit status
    Harness->>Hooks: Stop benchmark
    Harness->>Hooks: Declare command PID and URI
    Hooks->>Valgrind: Add Benchmark pid description
    Hooks->>Valgrind: Dump benchmark part

Loading

_{Reviews (2) · Last reviewed commit: "feat(instrument-hooks): declare benchmar..." · Reviewed by Greptile}