Add declarative benchmark matrix with profiles

[connortsui20]

Rationale for this change

SQL benchmark coverage was duplicated across hand-maintained pr_targets / develop_targets arrays and inline full / base matrices in several workflow files. That made it difficult to see what each CI lane ran and allowed PR, develop, and nightly coverage to drift.

This PR moves the SQL benchmark matrix into one declarative Python registry. The workflows select one of the three profiles they actually use (develop, pr, or nightly) and receive the corresponding GitHub Actions matrix from vx-bench.

The scope is intentionally narrow: this PR does not introduce a generic predicate or tag language, and Rust micro-benchmark workflow consolidation is split into a follow-up PR.

What changes are included in this PR?

• Declare each SQL benchmark and its supported targets once in bench_orchestrator/matrix.py.
• Add vx-bench matrix <profile> for the three workflow-backed profiles.
• Resolve the SQL PR, develop, and nightly matrices dynamically instead of duplicating them in YAML.
• Preserve develop and nightly coverage while making the PR lane consistent across benchmarks.

What APIs are changed? Are there any user-facing changes?

No Vortex library APIs change. vx-bench gains the matrix subcommand; its existing commands are unchanged. This addresses #4935.

Validation

• uv run ruff check bench-orchestrator
• uv run ruff format --check bench-orchestrator
• uv run pytest bench-orchestrator/tests (37 passed)
• yamllint --strict -c .yamllint.yaml on the changed workflows
• actionlint on the SQL and nightly workflows

https://claude.ai/code/session_01PQg1cgPLKJg4QVJvE2xa3Q

[joseph-isaacs]

Do we not use yml for the ci ones? Is this hook into there?

[connortsui20]

@joseph-isaacs Yes — it's now hooked into the workflow YAML, not just a standalone module. The benchmark workflows resolve their matrix from a plan job that runs vx-bench matrix <profile> and feed it into the bench job via fromJSON:

• sql-benchmarks.yml — the two inline full/base JSON matrices are deleted; it now takes a profile input and resolves it.
• bench-core.yml (new) — reusable micro-benchmark workflow, replacing the duplicated steps that were copy-pasted across bench.yml/bench-pr.yml.
• bench.yml, bench-pr.yml, sql-pr.yml, nightly-bench.yml — thin callers passing a profile (develop/pr/nightly/micro).

Net −549 lines across the workflows. develop/nightly resolve to the same target sets as before; the source of truth is bench-orchestrator/bench_orchestrator/matrix.py.

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will degrade performance by 14.81%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 1697 untouched benchmarks
⏩ 9 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	cuda/bitpacked_u8/unpack/3bw[100M]	299.3 µs	351.3 µs	-14.81%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/bench-declarative (2be6840) with develop (8a1f91c)}

Footnotes

1. 9 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩