Consolidate test categories and enable parallel executions

[johnsimons]

I decided to tackle CI timings again, and this time I went down the path of doing the opposite of what I did last time.

The last round split test categories apart so jobs could run side by side, which bought wall clock by spending runners. We kept adding categories after that, and a run had grown to 52 jobs against our 60 job org wide cap. So a single run was using 87% of the budget for every Particular repo, and any second run in flight sat in the queue for minutes.

I started out looking at parallelising more steps inside the jobs, but the step timings killed that idea. Where a job provisions infrastructure the backgrounded build is already completely hidden behind it, and where it does not there is nothing to overlap with. The win was in the other direction: the critical path was 870s while 34 of 41 jobs finished under 420s, and that slack is what you spend to get runners back.

What I changed

Set concurrency on the workflow. We were never cancelling superseded runs. Over the last 120 runs, zero were cancelled and 23 pairs overlapped, the worst holding a full run for 27 minutes. This was the biggest single cause of the queueing and it costs nothing. It cancels on master too, deliberately. Releases are unaffected, they run from release.yml on a tag and this workflow never triggers on tags.

Merged the categories that share infrastructure, 20 down to 11. The merge is declared in the projects, by sharing a <TestCategory> value, not in the workflow. I tried it in the matrix first and it was worse: you end up string matching category names in YAML and duplicating the mapping across the csproj files. Doing it in the project meant select-test-projects.ps1 needed no change and ServiceControl_TESTS_FILTER kept working.

Category	Now covers	Projects
Raven	audit tests plus the Raven acceptance and persistence suites	7
RabbitMQ	all four classic/quorum and conventional/direct combinations	4
SqlServer	transport, persistence and acceptance	3
PostgreSql	transport, persistence and acceptance	3

One database server per category instead of two. Both provisioning actions hardcode their host port, so a job can only invoke each once, and it turns out it only needs to. The persistence and acceptance suites create a database per test and the transport suite uses randomly suffixed queue tables, so they share a server fine. We also stop paying for the SQL Server full text install twice.

Added -MaxParallel to run-tests.ps1 so a multi-project category costs the slowest assembly rather than the sum. Defaults to 1, so anything single-project behaves as before.

Results

	Jobs	Runner-min	Wall clock	Peak concurrent
Before	52	208.4	14.6 min	43
After	34	141.5	12.1 min	28
	-35%	-32%	-17%	-35%

I expected to trade wall clock for runners and instead got both. Merging the SQL Server categories turned out faster merged than the old critical path was split: Windows-SqlServer runs all three suites in 723s on one runner, where Windows-PrimarySqlServerAcceptance alone took 870s across three.

Before merging

The required status checks need updating, otherwise this cannot go green.

Remove, each in Windows- and Linux- form: DefaultAudit, PrimaryRavenAcceptance, PrimaryRavenPersistence, PrimarySqlServerAcceptance, PrimaryPostgreSqlAcceptance, SqlServerPersistence, PostgreSqlPersistence, RabbitMQClassicConventional, RabbitMQClassicDirect, RabbitMQQuorumConventional, RabbitMQQuorumDirect.

Add: Windows-Raven, Linux-Raven, Windows-RabbitMQ, Linux-RabbitMQ. SqlServer and PostgreSql stay, they just cover more now.

[rbev]

Should this plan be converted into as built documentation or removed?

[johnsimons]

I will remove it