cluster spec sheet: make the MySQL seeding instructions bloat-proof

[peterdukelarsen]

Motivation

Old single-threaded snapshots would run count() then read all rows in a table in just 4m. Now it takes 3m30s just to run a count(), which likely accounts for the performance regression flagged here: https://linear.app/materializeinc/issue/QAR-144/mysql-ingestion-performance-extremely-bad-on-large-cluster-50x-bad-on.

Specific trigger is unclear but I suspect the data was re-inserted fragmented whereas historically someone had run OPTIMIZE TABLE tbl before. I want to try to get the table into a consistent state across reseeds.

Changes

Add an OPTIMIZE TABLE tbl to ensure the table is more consistently densly packed.

[def-]

QA LLM Review

1. MEDIUM -- reseeding the MySQL table without bumping SourceIngestionScenario.VERSION

test/cluster-spec-sheet/mzcompose.py:2115

Details

The point of this change is to alter the physical layout of the external MySQL table so the source_ingestion / mysql hydration measurement drops from roughly 8m back to roughly 4m, but VERSION stays at 1.1.0. Rows recorded before and after the reseed therefore carry the same scenario_version, so the tracked series takes a silent ~2x step that is indistinguishable from a product change and that can mask a genuine regression landed in the same window.

ClusterScalingScenario states the rule at test/cluster-spec-sheet/mzcompose.py:547: bump when the scenario "changes in a way that makes comparing results between versions useless." The immediately preceding external-environment change did exactly that (# 1.1.0: MySQL RDS instance type changed), and #38377 landed specifically so that a per-workload bump reaches the recorded value. The value flows scenario.version() -> ScenarioRunner (test/cluster-spec-sheet/mzcompose.py:4157) -> the scenario_version column in misc/python/materialize/test_analytics/setup/tables/15-cluster-spec-sheet.sql, which is the only thing downstream consumers have to tell the two data regimes apart.

Suggest bumping to 1.2.0 with a matching # 1.2.0: ... note, landed at the same time as the actual reseed so the version boundary lines up with the data boundary.

2. MEDIUM -- OPTIMIZE TABLE targets page-split bloat, but the recorded DDL creates no primary key

test/cluster-spec-sheet/mzcompose.py:2167

Details

OPTIMIZE TABLE can only recover space that page splits wasted, and page splits from out-of-order customer_id values require a clustered index on customer_id. The create table tbl (...) recorded 28 lines above declares no PRIMARY KEY, so InnoDB clusters on a hidden monotonic row id, every insert appends, and a scattered-order load never bloats. Either that DDL is stale and the live table has a key these instructions do not reproduce, or the bloat mechanism cannot occur and OPTIMIZE addresses the wrong cause. Both readings leave the same defect: a reseed that follows this block verbatim still does not reproduce the table being measured, which is what the change set out to fix.

• InnoDB with no PRIMARY KEY and no NOT NULL UNIQUE index synthesizes a 6-byte DB_ROW_ID clustered index that increments per insert, so leaf pages fill densely regardless of the order customer_id arrives in.
• The 11.5 GiB vs 4.6 GiB figure is the signature of random insertion into a customer_id-keyed clustered index, which is evidence that the live table does carry that key and that test/cluster-spec-sheet/mzcompose.py:2139 is out of date.
• The scatter itself is incidental: n is assembled from independent loop counters, so the emitted order depends on the join order the optimizer happens to pick for the ten-way CROSS JOIN. Recording PRIMARY KEY (customer_id) in the DDL and adding ORDER BY n to the INSERT ... SELECT makes the load dense by construction instead of relying on a post-hoc rebuild; keeping the OPTIMIZE on top is still cheap insurance.
• Related, on the count(*) cost named in the motivation: the snapshot size gauge reads information_schema.tables.table_rows and falls back to an exact SELECT COUNT(*) when that estimate is absent or <= mysql_source_snapshot_exact_count_max_rows (default 1,000,000) at src/storage/src/source/mysql/snapshot.rs:1287 and src/storage-types/src/dyncfgs.rs:238. A freshly bulk-loaded table whose InnoDB stats were never recalculated can report a stale-low table_rows and take that O(rows) path. The Postgres block ends with analyze tbl;; the MySQL block has no counterpart. OPTIMIZE TABLE recalculates stats so it covers this incidentally, but recording ANALYZE TABLE tbl; too would make that part of the seed state explicit rather than a side effect.