docs(run-engine): add CK virtual-time benchmark results

Real flag OFF-vs-ON A/B from a local homelab with a production-like
multi-cluster topology (not production, no prod data or traffic). Micro
arm: a light key behind a backlog goes from first-served at step 480 to
step 4, wait p95 down ~70%, contention fairness 0.34 to 1.0, with the
balanced and lone-key cases unchanged and drain steps identical
(work-conserving). End-to-end arm across three worker clusters: the
victim tenant start latency drops ~38% mean while the flood tenant is
unchanged. Relative numbers only; single box, not prod scale.

Author: 1stvamp

internal-packages/run-engine/design/benchmarks/results-2026-07-27.md

@@ -0,0 +1,80 @@
+# CK virtual-time scheduling: A/B benchmark results (2026-07-27)
+
+Run on a **local homelab box, not production**. It is a *production-like*
+topology only in shape: one shared control plane (Postgres + one shared run-queue
+Redis + ClickHouse + the webapp/engine on this branch) and three separate managed
+worker clusters registered as distinct worker groups, all as containers on a
+single machine. No production data, traffic, or infrastructure was involved.
+Method and harness: `2026-07-26-ck-vtime-benchmark.md`.
+
+All numbers are **relative** (flag OFF vs ON, identical load, same box). Absolute
+throughput on this single-box homelab is nowhere near prod scale and is not
+reported as such; only the OFF-vs-ON delta is meaningful.
+
+## Arm 1: queue-level micro-benchmark (the isolated, defensible numbers)
+
+Real `RunQueue` driven directly against a dedicated Redis, flag OFF (age-ordered
+CK dequeue) vs ON (virtual time), 5 trials, quantum 1 / window multiplier 3.
+Every arm served every message exactly once (conservation checked), so the
+comparison is sound. Wait is in logical dequeue steps.
+
+| scenario | metric | baseline (OFF) | vtime (ON) | delta |
+| --- | --- | --- | --- | --- |
+| **ckSkew** (heavy backlog + light keys) | light-key first-serve step | 480 | 4 | **−99%** |
+| | light-key wait p50 | 556 | 96 | −83% |
+| | light-key wait p95 | 628 | 188 | −70% |
+| | light-key wait p99 | 636 | 196 | −69% |
+| | Jain fairness (contention) | 0.34 | 1.00 | +192% |
+| | drain step (work conservation) | 636 | 636 | 0% |
+| **ckTrickle** (bulk + trickle keys) | trickle first-serve step | 480 | 4 | −99% |
+| | trickle wait p95 | 592 | 172 | −71% |
+| | Jain fairness | 0.50 | 1.00 | +100% |
+| **ckSybil** (20 keys flood + 1 light; caps can't fix) | light first-serve step | 25 | 2 | −92% |
+| | light wait p95 | 34 | 27 | −21% |
+| **ckManyKeys** (61 variants > pass-1 window) | light first-serve step | 72 | 9 | −88% |
+| | drain step (no starvation) | 81 | 79 | drains fully |
+| **ckBalanced** (4 symmetric keys, no-harm) | worst-key wait p95 | 92 | 92 | 0% |
+| **ckHeavyIdle** (lone key, work conservation) | drain step | 59 | 59 | 0% (exact) |
+| (all scenarios) | Redis ops per dequeue+ack | baseline | +7% to +23% | cost |
+
+Reading it: a light key behind a backlog goes from waiting out the whole backlog
+(first-serve step 480) to being served almost immediately (4), wait p95 drops
+~70%, and contention fairness goes from lopsided (Jain 0.34) to even (1.00). The
+symmetric and lone-key cases are untouched, and drain steps are identical, so the
+fair order is work-conserving and does no harm. The cost is a modest per-dequeue
+Redis op increase.
+
+## Arm 2: end-to-end on the three worker regions (realism check)
+
+Deployed task on one shared base queue, per-run concurrency keys, environment
+concurrency ceiling pinned to 5 so the CK dequeue order is the bottleneck.
+Noisy-neighbor load: tenant A floods across 30 keys (240 runs), tenant B sends 30
+runs across 3 keys, runs spread across the three regions, 1.5s hold. Metric is
+per-run start latency (`startedAt - createdAt`, ms). Flag flipped OFF vs ON with a
+control-plane redeploy between arms; identical load each arm.
+
+| tenant | metric | baseline (OFF) | vtime (ON) | delta |
+| --- | --- | --- | --- | --- |
+| **B (victim, 3 keys)** | start latency mean | 221,476 | 136,344 | **−38%** |
+| | start latency p50 | 219,901 | 133,930 | **−39%** |
+| | start latency p95 | 259,092 | 221,061 | −15% |
+| | start latency p99 | 259,136 | 221,078 | −15% |
+| A (flood, 30 keys) | start latency mean | 95,130 | 100,136 | +5% |
+| A | start latency p50 | 95,049 | 95,689 | ~0% |
+
+Reading it: under age order the light tenant waits behind the flood (mean ~221s);
+with virtual time it takes fair turns and drops to ~136s (mean −38%, p50 −39%),
+while the flood tenant is essentially unchanged (+5%, it stops jumping the queue).
+The smaller p95/p99 gain reflects per-**key** fairness: tenant B's keys carry
+deeper per-key backlogs here (10 runs/key vs the flood's 8), so B's last runs
+stay in the fair rotation longer. Absolute latencies are the cap-5 serialization
+plus per-run worker startup on a single box; only the OFF-vs-ON delta is the
+signal.
+
+## Bottom line
+
+Both arms agree: the fair order removes the starvation of a light concurrency key
+behind a large or sharded backlog, does not harm the balanced or lone-key cases,
+stays work-conserving, and costs a small, bounded per-dequeue Redis overhead. The
+micro-benchmark isolates the scheduler (the defensible numbers); the end-to-end
+run confirms the same effect on real deployed runs across the worker regions.