[jkaczman]

Makes a change to measure microbench performance regressions based on instruction counts instead of latency.

Latency can shift when unrelated changes move the binary's code layout, which can cause false regressions. Instruction counts don't depend on layout, therefore making it less prone to flakiness.

Latency still shows up in the logs but will not cause PR checks to fail.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...dog/src/frontend/client/query_engine/test/bench.rs	0.00%	20 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Metal bench (arm64) passed. All 103 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	ctx/txn
pooler		21.48 (-0.5% ± 0.5%)	22,244 (-0.5% ± 0.7%)	35,167 (+0.0% ± 0.10%)	32,266 (-0.7% ± 0.6%)
lb		22.15 (-0.1% ± 0.4%)	22,961 (-0.3% ± 0.5%)	36,067 (-0.0% ± 0.08%)	33,067 (-0.3% ± 1.0%)
sharding-w2		37.69 (+0.2% ± 1.3%)	51,775 (-0.1% ± 1.3%)	101,222 (+0.0% ± 0.01%)	45,163 (+0.5% ± 2.7%)	1.545 (+0.0% ± 0.5%)
sharding-w8	228,396 (-0.1% ± 0.5%)	31.84 (+0.2% ± 0.6%)	49,550 (-0.1% ± 0.5%)	57,040 (+0.0% ± 0.01%)	38,421 (+0.5% ± 1.1%)	0.1514 (-0.2% ± 1.7%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	10,682 (-1.4% ± 1.8%)	3,058 (-0.7% ± 1.0%)	2,804 (+0.7% ± 1.2%)
lb	11,180 (-0.7% ± 1.2%)	3,133 (-0.3% ± 0.6%)	2,846 (+0.4% ± 0.9%)
sharding-w2	23,311 (-0.4% ± 3.7%)	4,763 (-0.2% ± 2.8%)	7,287 (+0.9% ± 5.4%)
sharding-w8	23,615 (+0.0% ± 1.1%)	4,414 (+0.2% ± 0.4%)	10,482 (-0.4% ± 1.2%)

Microbench: all 90 measures within bounds.

Microbench latency: 3 benches changed more than 3%

benchmark	latency	baseline	change	instructions change
extended_cached::begin	1,554 ns	1,629 ns	-4.63% ± 2.12%	-
extended_cached::set	1,635 ns	1,704 ns	-4.03% ± 1.61%	-
simple_cold::set	2,367 ns	2,444 ns	-3.14% ± 1.98%	-

lb flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 775a972 with 97fc159 on m8g.metal-24xl (spot).

[github-actions]

Metal bench (x86) passed. All 103 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	ctx/txn
pooler		35.41 (+0.1% ± 0.2%)	24,495 (+0.3% ± 0.2%)	36,691 (-0.0% ± 0.01%)	58,054 (-0.0% ± 0.3%)
lb		36.4 (+0.0% ± 0.1%)	25,649 (-0.1% ± 0.3%)	37,645 (+0.0% ± 0.02%)	58,920 (+0.1% ± 0.2%)
sharding-w2		58.05 (-0.1% ± 0.4%)	56,265 (-0.0% ± 0.09%)	106,028 (-0.0% ± 0.07%)	73,513 (-0.0% ± 0.7%)	1.381 (+0.1% ± 0.2%)
sharding-w8	144,594 (-0.0% ± 0.7%)	50.32 (-0.1% ± 0.5%)	62,414 (-0.2% ± 0.7%)	61,431 (+0.0% ± 0.02%)	65,363 (-0.0% ± 0.7%)	0.1574 (+0.3% ± 1.0%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	77,052 (+0.4% ± 0.5%)	3,761 (+0.1% ± 0.1%)	26,391 (+0.2% ± 0.9%)
lb	80,358 (+0.2% ± 0.5%)	3,892 (-0.0% ± 0.2%)	28,793 (-1.0% ± 1.7%)
sharding-w2	166,927 (+0.2% ± 0.7%)	6,362 (-0.1% ± 0.3%)	52,097 (-0.9% ± 2.0%)
sharding-w8	131,857 (-0.3% ± 0.5%)	5,098 (-0.1% ± 0.2%)	168,420 (-0.2% ± 1.3%)

Microbench: all 90 measures within bounds.

Microbench latency: 4 benches changed more than 3%

benchmark	latency	baseline	change	instructions change
extended_cached::begin	1,830 ns	1,760 ns	+3.98% ± 1.89%	-
extended_cached::set	1,944 ns	1,876 ns	+3.64% ± 2.93%	-
extended_cold::insert	7,048 ns	6,830 ns	+3.19% ± 0.59%	-
extended_cached::insert	2,776 ns	2,694 ns	+3.05% ± 1.30%	-

lb flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs 97fc159

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 775a972 with 97fc159 on c7a.metal-48xl (spot).