feat(perf): benchmark shelltime track latency on every PR and main push

[AnnatarHe]

Why

The shell hooks run shelltime track in the foreground before and after every command, so its latency is felt at every prompt. That latency covers the whole process: exec, Go init of a large dependency tree, main's config read, then the track logic. Nothing measured it until now.

What

perf/ is a new, separate Go module. Keeping it separate means golang.org/x/perf never affects version selection for the shipped binaries, and the root ./..., mockery, codecov and goreleaser never see it.

• Harness (harness_test.go, track_bench_test.go):

  • Execs the real binary once per iteration, with the exact flags the zsh hook passes, in an isolated $HOME.

  • Scenarios:

    Scenario	What it runs
    Startup/floor	true, as an A/A runner-noise check
    Startup/version	shelltime --version
    Track/daemon/{pre,post}	track against a fake daemon on /tmp/shelltime.sock
    Track/direct/pre	track -p=pre with no daemon
    Track/direct/post	track -p=post with no daemon, over 500 seeded store pairs
    Track/direct/post-sync	the post that reaches flushCount, sending its batch to a fake API

  • Every scenario checks that it took its intended path (daemon events, store growth, API batches, no errors in log.log). track always exits 0, so without these checks a broken run would just look fast.

  • Besides wall time, it reports user/sys CPU, p50/p95 and peak RSS.

  • The Track scenarios are skipped when a real daemon owns the socket.

• cmd/perfreport:
  • Compares base and head using benchmath: medians with a 95% confidence interval, and a Mann-Whitney U test.
  • Flags ⚠️ when a scenario is >10% and >0.25 ms slower with p < 0.05, and 🚀 for the same rule in the other direction.
  • Marks a scenario that failed on head with ❌, and the noisy-runner case with 🎲.
  • Output: markdown, ::warning annotations, and regression= / skipped= step outputs.
  • Covered by golden tests.
• compare.sh:
  • Builds base and head with identical release-like flags (-trimpath -buildvcs=false -s -w).
  • Skips the run when the two binaries are byte-identical, unless the harness itself changed.
  • Runs one harness against both binaries in 10 interleaved ABBA rounds × 100 execs.
• comment.sh creates or updates a single sticky PR comment through gh api. No third-party action gets a write token.

.github/workflows/perf.yaml ("Track Perf"):

• Runs on PRs to main, pushes to main (compared against the previous tip) and workflow_dispatch.
• The report goes to the job summary and to the PR comment. Fork PRs get the summary only.
• A regression is a warning, not a failed check.

Docs: perf/README.md, plus short sections in CLAUDE.md and AGENTS.md.

Verification (local, Linux container)

• go -C perf vet ./... && go -C perf test ./... pass. shellcheck and actionlint are clean.
• Full-size A/A run (identical code, FORCE=1, 10×100): ✅ no significant change in any scenario. It took about 1m50s.
• Regression drill (a temporary time.Sleep(3ms) in commandTrack, not committed):
  • All 5 Track scenarios were flagged ⚠️ (+22% to +62%) with annotations and regression=true.
  • Startup/version was correctly not flagged.
• Path checks: pointing the harness at /usr/bin/true fails every Track scenario with a clear message.
• Daemon guard: a live listener on the socket skips the Track scenarios, and a stale socket file is removed.
• Identical builds produce byte-identical binaries, so a docs-only PR is a quick "binary unchanged" report.

This PR only adds the harness, so its own run is effectively a forced A/A check on CI runners.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QuSfoWGrqsxjw5Yyegzzyt

---

Generated by Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
unittests	85.83% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

⏱️ shelltime track performance

✅ No significant change in shelltime track latency

Comparing main @ a5b2991 → #317 @ 8cbf2f4. Each scenario execs the binary the way the shell hooks do; values are the median of 10 interleaved rounds. Lower is better.

Scenario	base	head	Δ	p
Startup/floor	481 µs ±3%	478 µs ±4%	~	0.579	A/A
Startup/version	3.58 ms ±2%	3.53 ms ±3%	~	0.631	✅
Track/daemon/pre	3.83 ms ±2%	3.83 ms ±3%	~	0.853	✅
Track/daemon/post	3.83 ms ±2%	3.81 ms ±2%	~	0.436	✅
Track/direct/pre	3.78 ms ±2%	3.77 ms ±3%	~	0.684	✅
Track/direct/post	7.21 ms ±1%	7.15 ms ±1%	~	0.190	✅
Track/direct/post-sync	8.73 ms ±1%	8.60 ms ±3%	~	0.165	✅

Binary size: 19.1 MiB → 19.1 MiB (+0.0%).

CPU, tail latency and memory

Per exec, base → head (Δ when significant).

Scenario	p95 wall	user CPU	sys CPU	peak RSS
Startup/floor	538 µs → 537 µs	280 µs → 280 µs	147 µs → 154 µs	12.2 MiB → 12.1 MiB
Startup/version	3.91 ms → 3.88 ms	1.73 ms → 1.65 ms	2.29 ms → 2.27 ms	18.3 MiB → 18.3 MiB
Track/daemon/pre	4.19 ms → 4.17 ms	1.88 ms → 1.85 ms	2.44 ms → 2.49 ms	19.1 MiB → 19.1 MiB
Track/daemon/post	4.18 ms → 4.20 ms	1.86 ms → 1.85 ms	2.45 ms → 2.42 ms	18.9 MiB → 18.9 MiB
Track/direct/pre	4.14 ms → 4.13 ms	1.88 ms → 1.85 ms	2.39 ms → 2.38 ms	19.1 MiB → 19.1 MiB
Track/direct/post	7.83 ms → 7.77 ms	5.07 ms → 5.01 ms	3.30 ms → 3.35 ms	21.3 MiB → 21.2 MiB
Track/direct/post-sync	9.61 ms → 9.49 ms (-1.24%)	6.63 ms → 6.54 ms	4.44 ms → 4.43 ms	23.4 MiB → 23.5 MiB

benchstat

goos: linux
goarch: amd64
pkg: github.com/malamtime/cli/perf
cpu: AMD EPYC 7763 64-Core Processor                
                         │    base     │                head                │
                         │   sec/op    │   sec/op     vs base               │
Startup/floor-4            481.3µ ± 3%   478.2µ ± 4%       ~ (p=0.579 n=10)
Startup/version-4          3.581m ± 2%   3.526m ± 3%       ~ (p=0.631 n=10)
Track/daemon/pre-4         3.833m ± 2%   3.827m ± 3%       ~ (p=0.853 n=10)
Track/daemon/post-4        3.835m ± 2%   3.809m ± 2%       ~ (p=0.436 n=10)
Track/direct/pre-4         3.777m ± 2%   3.773m ± 3%       ~ (p=0.684 n=10)
Track/direct/post-4        7.214m ± 1%   7.150m ± 1%       ~ (p=0.190 n=10)
Track/direct/post-sync-4   8.735m ± 1%   8.604m ± 3%       ~ (p=0.165 n=10)
geomean                    3.468m        3.440m       -0.79%

                         │    base     │                head                │
                         │ p50-sec/op  │ p50-sec/op   vs base               │
Startup/floor-4            474.1µ ± 3%   467.6µ ± 3%       ~ (p=0.436 n=10)
Startup/version-4          3.557m ± 3%   3.516m ± 2%       ~ (p=0.684 n=10)
Track/daemon/pre-4         3.814m ± 3%   3.839m ± 2%       ~ (p=0.796 n=10)
Track/daemon/post-4        3.815m ± 2%   3.768m ± 2%       ~ (p=0.165 n=10)
Track/direct/pre-4         3.771m ± 2%   3.779m ± 2%       ~ (p=0.912 n=10)
Track/direct/post-4        7.143m ± 2%   7.087m ± 1%       ~ (p=0.123 n=10)
Track/direct/post-sync-4   8.744m ± 1%   8.582m ± 3%       ~ (p=0.105 n=10)
geomean                    3.447m        3.419m       -0.80%

                         │    base     │                head                │
                         │ p95-sec/op  │ p95-sec/op   vs base               │
Startup/floor-4            537.7µ ± 3%   537.3µ ± 4%       ~ (p=0.912 n=10)
Startup/version-4          3.913m ± 1%   3.881m ± 2%       ~ (p=0.631 n=10)
Track/daemon/pre-4         4.194m ± 1%   4.172m ± 3%       ~ (p=0.739 n=10)
Track/daemon/post-4        4.182m ± 1%   4.196m ± 2%       ~ (p=0.529 n=10)
Track/direct/pre-4         4.139m ± 2%   4.131m ± 1%       ~ (p=0.631 n=10)
Track/direct/post-4        7.830m ± 2%   7.765m ± 1%       ~ (p=0.075 n=10)
Track/direct/post-sync-4   9.611m ± 1%   9.493m ± 2%  -1.24% (p=0.043 n=10)
geomean                    3.802m        3.784m       -0.48%

                         │     base     │                 head                 │
                         │  peak-rss-B  │  peak-rss-B    vs base               │
Startup/floor-4            12.19Mi ± 2%   12.14Mi ± 16%       ~ (p=0.796 n=10)
Startup/version-4          18.28Mi ± 1%   18.28Mi ±  1%       ~ (p=0.912 n=10)
Track/daemon/pre-4         19.12Mi ± 1%   19.09Mi ±  1%       ~ (p=0.579 n=10)
Track/daemon/post-4        18.92Mi ± 1%   18.90Mi ±  1%       ~ (p=0.971 n=10)
Track/direct/pre-4         19.08Mi ± 0%   19.08Mi ±  1%       ~ (p=0.853 n=10)
Track/direct/post-4        21.29Mi ± 1%   21.23Mi ±  1%       ~ (p=0.315 n=10)
Track/direct/post-sync-4   23.42Mi ± 0%   23.46Mi ±  0%       ~ (p=0.218 n=10)
geomean                    18.59Mi        18.57Mi        -0.11%

                         │     base     │                head                 │
                         │  sys-sec/op  │  sys-sec/op   vs base               │
Startup/floor-4            147.1µ ± 13%   153.5µ ± 19%       ~ (p=0.796 n=10)
Startup/version-4          2.293m ±  9%   2.275m ±  6%       ~ (p=0.739 n=10)
Track/daemon/pre-4         2.441m ±  7%   2.493m ±  6%       ~ (p=0.315 n=10)
Track/daemon/post-4        2.449m ±  5%   2.415m ±  5%       ~ (p=0.631 n=10)
Track/direct/pre-4         2.393m ±  5%   2.383m ±  7%       ~ (p=0.853 n=10)
Track/direct/post-4        3.304m ±  4%   3.345m ±  3%       ~ (p=0.631 n=10)
Track/direct/post-sync-4   4.440m ±  3%   4.426m ±  5%       ~ (p=0.579 n=10)
geomean                    1.838m         1.850m        +0.66%

                         │    base     │                head                 │
                         │ user-sec/op │ user-sec/op   vs base               │
Startup/floor-4            280.5µ ± 6%   280.4µ ±  4%       ~ (p=0.912 n=10)
Startup/version-4          1.728m ± 7%   1.654m ±  7%       ~ (p=0.063 n=10)
Track/daemon/pre-4         1.885m ± 6%   1.847m ± 11%       ~ (p=0.436 n=10)
Track/daemon/post-4        1.859m ± 3%   1.847m ±  5%       ~ (p=0.393 n=10)
Track/direct/pre-4         1.879m ± 4%   1.853m ±  6%       ~ (p=0.529 n=10)
Track/direct/post-4        5.068m ± 4%   5.009m ±  3%       ~ (p=0.165 n=10)
Track/direct/post-sync-4   6.631m ± 2%   6.544m ±  3%       ~ (p=0.436 n=10)
geomean                    1.950m        1.920m        -1.55%

_{Δ compares medians; ~ means no significant difference (Mann-Whitney U, p ≥ 0.05). A scenario is flagged when it is more than 10% and more than 250µs slower with p < 0.05. Startup/floor runs true, not shelltime: it is an A/A check of runner noise. Runner: linux/amd64, AMD EPYC 7763 64-Core Processor . go1.27.1, 10 rounds × 100 execs per scenario and binary.}

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.