perf(gemm): start workers once per product, synchronise on work done

[tannevaled]

Diagnosis (hardware counters, Zen 3 KVM guest cfarm421)

At 16 workers a 1024² product kept the CPUs busy 7.1 of 16 of the time. The instruction count stayed flat and so did L1 misses. The old driver forked and joined goroutines twice per (jc, pc) block. A fork/join wakes parked threads one after another; a micro-benchmark on the same guest measured it at 26 µs at 2 workers, 165 µs at 8 and 330 µs at 16, which is about 40% of a 12.5 ms product.

Refuted first: SMT siblings (pairwise interference is 0–10%, never half) and the size of the A block versus L2 (an MC sweep over 72–256 gave nothing clear).

docs/perf.md had said "not scheduling overhead: 83% of the profile is the micro-kernel". That was wrong, because a cycles profile cannot see idle cores. It is now struck through and corrected in place.

Change (internal/kernels/gemm.go)

• Workers. They start once per product and synchronise on work done: a block's bands start once its B chunks are packed and all bands of the previous block are done. The first version waited at a barrier for every worker, which put the wake-up chain back on the critical path (96³ on 8 POWER9 cores at 0.57×), so it was replaced.
• B buffer. It is double-buffered, and packing is shared in claims of 4 NR-wide panels. Claimed work never waits, so the scheme cannot deadlock. Each C tile is still accumulated in pc order, so results are bit-identical to the serial product.
• Pools. The pack buffers are split into an A pool and a B pool. A full pair per worker had held 16 MiB of unused B buffers at 16 workers, refilled and zeroed after each GC.
• Removed. packBP/packBParMin and the barrier type are gone. spinUntil spins, then yields with runtime.Gosched, and never parks.

Tests

• TestSpinUntil and TestGemmParallelYielding cover every wait yielding immediately and more workers than CPUs. The existing GEMM tests run under -race, and coverage stays at 100%.
• Sabotage, detected: dropping the "previous block's bands done" condition produces DATA RACE reports.
• Sabotage, not detected: dropping the "block i-2 done before repacking its buffer" condition broke nothing, because a worker reaching block i has already passed block i-1's wait, which requires it. That condition was dead, so it was removed and the implication is written down.

Measured (medians of interleaved runs vs the previous driver)

	2 workers	8	16
Zen 3 guest 1024²	1.06×	1.33×	1.39×
Zen 3 guest 512²	1.20×	1.20×	1.33×
Zen 3 guest 256²	0.92×	1.19×	1.24×
Zen 3 guest 128²	0.89×	1.50×	1.16×
Zen 3 guest (1024×256)·(256×1024)	0.82×	1.07×	0.89× (p25 1.00×)
POWER9 bare metal 512²	1.34×	1.37×
POWER9 bare metal 256²	1.61×	1.78× (24 rounds)

Against OpenBLAS-16, measured in one interleaved run: at 256² we are now 1.44× faster (the previous driver was 1.06×); at 512² 0.42× (was 0.29×); at 1024² 0.51× (was 0.43×).

Known loss: 2 workers on the KVM guest lose 8–18% on mid sizes. Bare-metal POWER9 gains at 2 workers. Not measured on Apple M, because the machine was shared at the time.

Version: v0.4.1 (performance only, no API change).

🤖 Generated with Claude Code