Skip to content

feat: add MacOS support#81

Open
not-matthias wants to merge 3 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos
Open

feat: add MacOS support#81
not-matthias wants to merge 3 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos

Conversation

@not-matthias

Copy link
Copy Markdown
Member

No description provided.

@codspeed-hq

codspeed-hq Bot commented Jun 2, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 16.23%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 12 regressed benchmarks
✅ 206 untouched benchmarks
🆕 1 new benchmark

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime iterative fibo 15 312 ns 900 ns -65.33%
WallTime iterative fibo 20 360 ns 900 ns -60%
WallTime short body 1.7 µs 2.8 µs -37.66%
WallTime short body 1.7 µs 2.6 µs -33.64%
WallTime short body 1.8 µs 2.5 µs -31.13%
WallTime short body 1.8 µs 2.5 µs -30%
WallTime short body 1.7 µs 2.4 µs -28.86%
WallTime short body 1.8 µs 2.5 µs -27.96%
Simulation switch 2 8.3 µs 11.2 µs -25.47%
Memory short body 408 B 536 B -23.88%
WallTime fibo 10 2.1 µs 2.8 µs -23.04%
Simulation wait 1ms 20 µs 22.8 µs -11.95%
Memory wait 1ms 16 B 7 B ×2.3
Simulation short body 121.2 µs 76.1 µs +59.37%
Simulation long body 382.1 µs 336.1 µs +13.69%
WallTime switch 1 324 ns 288 ns +12.5%
WallTime test sync baz 10 360 ns 324 ns +11.11%
WallTime test sync baz 100 480 ns 432 ns +11.11%
WallTime test sync baz 10 384 ns 348 ns +10.34%
🆕 WallTime fibo darwin N/A 7.9 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos (c184ad6) with main (4dae798)

Open in CodSpeed

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 2 times, most recently from f0b1349 to 3982c34 Compare June 2, 2026 10:36
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 32c3b31 to 1d75a61 Compare June 3, 2026 13:23
@not-matthias not-matthias marked this pull request as ready for review June 3, 2026 15:22
@greptile-apps

greptile-apps Bot commented Jun 3, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds macOS support to the CodSpeed Node.js benchmarking plugins by centralizing the __codspeed_root_frame__ wrapper into a shared wrapWithRootFrame/wrapWithRootFrameSync utility in @codspeed/core, and introduces benchmark timeline markers (BenchmarkStart/BenchmarkEnd) so the macOS walltime runner can bracket perf samples. CI is updated to remove the old walltime-macos-test job and replace it with a proper codspeed-walltime-macos job in codspeed.yml.

  • packages/core/src/rootFrame.ts (new): Centralizes the __codspeed_root_frame__ wrapper that was previously inlined in every plugin; all four plugins now import and use it.
  • packages/tinybench-plugin/src/shared.ts and vitest-plugin/src/walltime/index.ts: Add marker emission (addMarker with BenchmarkStart/BenchmarkEnd timestamps) so the runner can identify the benchmark window inside a profiling sample — critical for macOS walltime support.

Confidence Score: 4/5

The macOS walltime support is functionally sound in the vitest plugin path, but the tinybench plugin emits benchmark markers after closing the sample window, which both breaks the new integration test and produces incorrect profiling data on the tinybench walltime path.

In packages/tinybench-plugin/src/shared.ts, sendBenchmarkMarkers() is called after stopBenchmark() and setExecutedBenchmark(). The code's own comment, the vitest-plugin reference implementation, and the newly added integration test all require the markers to be emitted before stopBenchmark(). The test will fail as written, and any tinybench walltime benchmark run on macOS will have misattributed or missing flame graph data.

packages/tinybench-plugin/src/shared.ts — both wrapWithInstrumentHooks and wrapWithInstrumentHooksAsync call sendBenchmarkMarkers after stopBenchmark(), inverting the required ordering.

Important Files Changed

Filename Overview
packages/tinybench-plugin/src/shared.ts Adds marker emission to wrapWithInstrumentHooks/wrapWithInstrumentHooksAsync, but sendBenchmarkMarkers is called after stopBenchmark(), inverting the required ordering and breaking the newly added integration test.
packages/tinybench-plugin/tests/index.integ.test.ts Adds test that asserts markers land between startBenchmark and stopBenchmark — but the current shared.ts implementation violates this invariant, so the test will fail.
packages/vitest-plugin/src/walltime/index.ts Refactors tinybench hook to use wrapWithRootFrame; markers are correctly emitted before stopBenchmark(), restores task.fn in finally, and captures pid once outside the hot path.
packages/core/src/rootFrame.ts New utility cleanly centralizes the codspeed_root_frame wrapper; both sync and async variants are well-typed and correct.
.github/workflows/codspeed.yml Adds codspeed-walltime-macos job; previous threads already flagged the staging URL, diagnostic step, and mutable branch pin.
packages/tinybench-plugin/src/walltime.ts Simplifies wrapTaskFunction to use shared wrapWithRootFrame/wrapWithRootFrameSync; logic is unchanged.
packages/vitest-plugin/src/analysis.ts Swaps inline codspeed_root_frame for the shared helper; no behavioral change.
packages/benchmark.js-plugin/src/index.ts Uses shared wrapWithRootFrame/wrapWithRootFrameSync; test mock updated to pass through the real implementations.
packages/vitest-plugin/benches/macos.bench.ts New macOS-only benchmark, skipped via describe.skipIf on non-Darwin platforms.

Sequence Diagram

sequenceDiagram
    participant Plugin as Plugin (tinybench/vitest/benchmark.js)
    participant Core as @codspeed/core
    participant Runner as CodSpeed Runner

    Note over Plugin,Runner: Correct order (vitest walltime / this PR intent)
    Plugin->>Core: startBenchmark()
    Plugin->>Core: currentTimestamp() → runStart
    Plugin->>Plugin: execute benchmark fn() under __codspeed_root_frame__
    Plugin->>Core: currentTimestamp() → runEnd
    Plugin->>Core: addMarker(BenchmarkStart, runStart)
    Plugin->>Core: addMarker(BenchmarkEnd, runEnd)
    Plugin->>Core: stopBenchmark()
    Plugin->>Core: setExecutedBenchmark(pid, uri)
    Core-->>Runner: "FIFO: SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd"

    Note over Plugin,Runner: Bug in shared.ts (tinybench plugin)
    Plugin->>Core: startBenchmark()
    Plugin->>Core: currentTimestamp() → runStart
    Plugin->>Plugin: execute benchmark fn()
    Plugin->>Core: currentTimestamp() → runEnd
    Plugin->>Core: stopBenchmark() closes sample window FIRST
    Plugin->>Core: setExecutedBenchmark(pid, uri)
    Plugin->>Core: addMarker(BenchmarkStart, runStart) outside window
    Plugin->>Core: addMarker(BenchmarkEnd, runEnd) outside window
    Core-->>Runner: Markers outside sample - flame graph attribution breaks
Loading

Reviews (20): Last reviewed commit: "fixup: stop before sending timestamps" | Re-trigger Greptile

Comment thread packages/vitest-plugin/src/walltime/index.ts Outdated
Comment thread packages/vitest-plugin/benches/macos.bench.ts Outdated
Comment thread .github/workflows/codspeed.yml Outdated
Comment thread .github/workflows/codspeed.yml Outdated
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 5 times, most recently from 762c0d8 to 93cf800 Compare June 4, 2026 16:41

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm

Comment thread packages/vitest-plugin/src/walltime/index.ts Outdated
Comment thread .github/workflows/codspeed.yml Outdated
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 93cf800 to e08c11f Compare June 10, 2026 10:19
Comment thread .github/workflows/codspeed.yml Outdated
@avalanche-staging

avalanche-staging Bot commented Jun 10, 2026

Copy link
Copy Markdown

Congrats! CodSpeed is installed 🎉

🆕 1 new benchmark was detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Detected benchmark


Open in CodSpeed

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 792ee5f to fea06fe Compare June 10, 2026 16:36
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from fea06fe to 48a2b46 Compare June 10, 2026 16:42
Comment thread packages/vitest-plugin/src/walltime/index.ts

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@GuillaumeLagrange

Copy link
Copy Markdown
Contributor

Although CI is failing.
Some of it is due to github instabilities of yesterday, but one of the job failed typechecking, I think because of core library. I'll elt you check

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 2 times, most recently from 79e8686 to 39e7a35 Compare June 11, 2026 13:31
Emit benchmark start/end markers for the tinybench plugin and the vitest
walltime runner, wrapping the measured function in a root frame.

The runner consumes the instrument-hooks FIFO stream in order and expects
SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd nesting per
benchmark, so the markers must land inside the sample window:

- vitest: emit the marker pair before stopBenchmark(), and move
  stopBenchmark() plus the markers into a finally block so a throwing
  benchmark cannot leave the profiler started-but-never-stopped
- tinybench: emit markers per task between start/stop instead of a single
  run-level pair; wrap the body in try/finally to keep start/stop balanced
  when a benchmark throws
- benchmark.js: bind wrapWithRootFrame/wrapWithRootFrameSync to the real
  implementations in the integ test's core mock

Add a regression test asserting both markers land between startBenchmark
and stopBenchmark in walltime mode.
Add benches/macos.bench.ts guarded with describe.skipIf(!darwin) so it
only runs on macOS, and wire the walltime-macos-test CI job to execute it
via a direct `node vitest.mjs bench --run macos` invocation.
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 7a06ec3 to c184ad6 Compare June 11, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants