diff --git a/.claude/skills/perf-benchmarking/SKILL.md b/.claude/skills/perf-benchmarking/SKILL.md index 5c74b3fe4f..728e28492f 100644 --- a/.claude/skills/perf-benchmarking/SKILL.md +++ b/.claude/skills/perf-benchmarking/SKILL.md @@ -5,6 +5,8 @@ description: Measure React Native performance on-device for stream-chat-react-na # Perf benchmarking (on-device, stream-chat-react-native) +> For a full multi-hour base-vs-head comparison of every major flow (release builds, FlatList and FlashList, statistics), use the `perf-gauntlet` skill instead. + How to measure SDK performance for real, against the **SampleApp running on a physical Android device**, using the committed `perf/` toolkit. This skill exists because the methodology is full of traps that produce confidently-wrong numbers; follow the rules and recipes here instead of improvising. Prerequisites for almost everything below: diff --git a/.claude/skills/perf-gauntlet/SKILL.md b/.claude/skills/perf-gauntlet/SKILL.md new file mode 100644 index 0000000000..c1e7519ce0 --- /dev/null +++ b/.claude/skills/perf-gauntlet/SKILL.md @@ -0,0 +1,272 @@ +--- +name: perf-gauntlet +description: Full on-device performance gauntlet for stream-chat-react-native - a multi-hour, base-vs-head comparison of every major SDK flow (channel list, channel open with 100 messages, live events, composer, scroll, threads, context menu, cold start) on FlatList AND FlashList, in RELEASE builds with React's profiling renderer, with per-component render/re-render numbers, CPU per thread, frames, and statistics that survive emulator noise. Use when asked for "a full gauntlet", "perf smoke tests", "does this regress performance", "compare develop and this branch", "performance before release", or "run the perf thing again". Not for quick single-change checks (use perf-benchmarking for those). +--- + +# Perf gauntlet (base vs head, release builds, every major flow) + +This is the procedure behind the October 2026 develop-vs-V10 comparison. It produces a table of "JS work" per +scenario for two builds (base = usually `origin/develop`, head = usually the working tree), with mount and +re-render numbers per major component, FlatList vs FlashList, CPU per thread, frame stats, and confidence +intervals. The harness lives in `perf/gauntlet/` (see its README for the file map). Read this whole file before +starting: almost every rule below exists because breaking it produced confidently wrong numbers once. + +## 0. Before touching anything: tell the user what this costs, and ask two things + +Say this up front (adjust numbers if the plan differs): + +| Step | Time | +|---|---| +| Export base tree + install deps | 2-5 min | +| First release build of the native project | 10-15 min (later builds: 1-2 min each, JS only) | +| Seed one fresh user per session (~700 API calls each) | ~5 min per user, 4-8 users | +| Dry runs (1 run per scenario, both sides) | ~30 min | +| Main gauntlet: 4 sessions, ABBA, 5 runs per scenario | ~3.5 h (~50 min per session) | +| Extra samples for small row/scroll deltas (12 short sessions) | ~1.6 h | +| Diagnostic builds + root-causing | 0.5-2 h | +| **Total** | **~5-7 h**, the emulator is busy the whole time | + +Ask (or default and say so): +1. **What is base and what is head?** Default: base = `origin/develop`, head = the current working tree. +2. **Which device?** Default: the Pixel 7 AVD (1080x2400; the driver's taps assume this size). + +Nothing in the repo is edited: every source change the measurement needs is applied at bundle time by the Metro +config/transformer, and all output goes to `perf/gauntlet/out/` (gitignored). Never commit, never stash, never +check out other refs in the user's tree (base is exported with `git archive`). + +## 1. The non-negotiable rules (each one cost hours once) + +1. **Release builds with React's profiling renderer, never debug.** Debug React is 3-4x slower and does dev-only + work that differs between versions; an attached CDP profiler doubles dev work again. The Metro config swaps + `ReactFabric-prod` for `ReactFabric-profiling`, which keeps `fiber.actualDuration` and Profiler commit hooks + in an otherwise normal release bundle. +2. **The emulator's speed depends on the HOST, and it can silently drop 5-7x for React work.** Confirmed + mechanism: host memory pressure (gradle builds, yarn install, Metro/hermesc, a big IDE) makes macOS compress the + emulator's RAM (3.3 GB compressed right after two builds); memory-heavy work (React rendering, Fabric) then runs + 5-7x slower until the emulator is cold-booted, while tight CPU loops look almost normal. It can also flip + mid-session. Therefore: + - do ALL builds first, `./gradlew --stop`, then COLD-BOOT the emulator, then measure; never build during a run; + - start the emulator from the user's Terminal panel (`run_in_terminal`), never from a background Bash task + (that gives it nice 5); `ps -o nice -p $(pgrep -f qemu-system | head -1)` must be 0; + - `preflight.sh` (run by `run.sh` before every session) waits for host load < 6 and refuses to measure when the + host has < 20% free memory, > 512 MB of the emulator is compressed, or an on-device CPU loop is > 1.4x its + calibration. When it stops a run: free memory, cold-boot the emulator, resume with the remaining users; + - inside the data, the per-window speed indices (`arith`, `mem` in `report.py main`) FLAG a mode change (they + rise ~60-70% when React work is 7x slower) but do not scale with it: never "normalize" across modes. If a + scenario's windows split into two clusters (e.g. cold start 230 ms vs 1550 ms with the same mounts), compare + only within one mode, and say so. +3. **One native binary for both sides.** Build both APKs from THIS checkout's Android project; only the JS bundle + differs (`build.sh` points the RN gradle plugin at each side's SampleApp). `check-native-parity.cjs` must pass; + if native modules differ, each side needs its own native build and you must say the comparison includes native + changes. +4. **A fresh, identically seeded user per session; ABBA side order; nothing else running.** Sessions mutate the + user's channels (new messages, reorders, unread counts), so a reused user is a different dataset. `run.sh` + alternates base/head as ABBA so slow drift cancels. Never seed, build or run Metro/gradle while a session runs. +5. **Nothing that touches the UI hierarchy inside a measurement window.** `uiautomator dump` is slow and can hang + for minutes; all dumps happen before a window opens. Waits inside windows are fixed `settle` sleeps. +6. **Never edit a script bash is executing.** Bash reads scripts incrementally; an edit mid-run corrupts the run. + `run.sh` freezes `session.sh` into `out//frozen/` first. To change scripts mid-gauntlet, write a new + file and `mv` it over (the running bash keeps the old inode). +7. **Grant notifications after `pm clear`** (`session.sh` does). The Android 13+ permission dialog pauses the + activity AND the JS timers; a cold start behind it measures a paused app. +8. **Cold start is `am force-stop; am start` back to back, with the window opened BEFORE the stop.** A plain + force-stop + wait lets Firebase's datatransport job restart the process ~0.4 s later, so the launch lands in a + half-initialized background process and cold starts become bimodal (1x vs 7x CPU). +9. **Counts first, time second.** Render counts, mounts, hook counts and context changes are deterministic and + immune to emulator speed. Time on an emulator varies +/-10-15% between sessions; that is bigger than most real + effects. A timing claim of a few percent needs: (a) the session-level CI (`report.py ci`), (b) the result + normalized by the speed index, (c) JS-thread CPU agreeing. If they disagree, say the effect is below what the + emulator can resolve. +10. **Before naming a cause, check the code is actually different.** If the "slower" components are unchanged + code with identical hook counts, the difference is the runtime, not the code. Use diagnostic builds (section + 6) to prove a cause; never report a correlation as a cause. +11. **FlatList windowing pollutes event windows.** About 3 in 10 FlatList event windows coincide with VirtualizedList + mounting a batch of ~30 cells (~1 s). `report.py` splits those out as `~batch`; report them + separately, never in the event's median. +12. **Don't flip-flop in front of the user.** Report interim numbers as interim ("not significant yet"), and only + give a verdict once the counts, CPU and the session-level CI agree. Say how big an effect the data can and + cannot rule out. + +## 2. How the harness measures (so you can interpret and extend it) + +`perf/gauntlet/runtime/probe.js` loads before anything else in the bundle (the Metro config replaces index.js's +first import with `runtime/entry-shim.js`). It installs a fake `__REACT_DEVTOOLS_GLOBAL_HOOK__` before the renderer +loads, so React reports every commit: + +| Metric | Source | Meaning | +|---|---|---| +| `render_ms` | HostRoot `actualDuration` | render phase incl. host-node creation | +| `wall_ms` | hook time - HostRoot `actualStartTime` | render + Fabric commit (Yoga layout on the JS thread) + layout effects | +| `layout_fx_ms` / `passive_fx_ms` | root `` (AppRegistry patch) | effect time | +| **`js_react_ms`** | `wall + passive` | **"JS work", the headline number** | +| per component | fiber walk over reconciled subtrees | self ms, inclusive ms (outermost instance), renders (PerformedWork), mounts | +| `dispatch_ms` | `client.dispatchEvent` wrapper (`probe-llc.js`, ES import so it hits the app's stream-chat build) | LLC event handling, per event type | +| `cpu_js/main/rt/proc_ms` | `/proc//task/*/stat` via `devmark.sh` at window edges | thread CPU (10 ms ticks), immune to preemption | +| `speed_ms`, `mem_speed_ms` | a fixed arithmetic loop and a fixed allocate-and-walk loop, timed once a second | runtime state during the window; a jump flags a mode change (rule 2); lower = faster | +| `frames` | `dumpsys gfxinfo` reset/dump per window | UI-thread frames, jank %, p50/p90/p99 (rough on emulators) | +| `gc_n` | `HermesInternal.getInstrumentedStats()` once a second | GC count (coarse) | +| counts | probe `M` lines | loaded messages per Bench channel: proves "100 loaded" on both sides | + +Everything is buffered and flushed to logcat once a second (`BENCH `, < 3.5k chars per line). Windows +are opened/closed by `devmark.sh` (`PERFMARK S|E