[rominf]

Stacked on #517 (base branch test/e2e-simulated-machines, so the diff is this PR alone). Draft until #517 merges; it then moves to main.

Summary

On every pull request, and again in the merge queue, the self-hosted GPU lanes run almost the whole suite (~170 scenarios) to cover the dozen that need a GPU, and they're what sets merge latency. After #517, every GPU scenario that only needs the CLI to see a GPU runs on a simulated machine on the hosted lane, and the rest of the suite needs no GPU at all. So:

• The four GPU lanes run only GPU scenarios. MI300X, Strix Ubuntu, Strix Windows and W7900 set E2E_GPU_ONLY=1 on every event, so they run only scenarios that need the real hardware (@requires-real-gpu, @requires-gpu, @requires-gfx-target). Everything else resolves to a skip that names the reason, so the report grid shows it as not applicable there rather than missing. A skip that has already resolved keeps its own, more precise reason.
• On PRs they run a smoke test. On pull_request and merge_group the lanes also set E2E_GPU_SMOKE_ONLY=1. That narrows the real-GPU set to the cheap per-engine canaries, now tagged @gpu-smoke: vLLM inference, lemonade inference, and the lemonade HF-checkpoint serve. In the queue the @merge-queue serves run too, as before (MI300X does not run in the queue at all since ci: supersede PR runs and skip MI300X in the merge queue #696). Pushes to main and release/**, dispatches and nightly run the full real-GPU set.
• New GitHub-hosted E2E tests (Windows) job. It runs the suite on windows-latest. Until now the Strix Windows GPU lane was the only place the non-GPU scenarios met native Windows.
  • It reports under its own platform (mock-windows, rendered "Mock / Windows") and writes its own summary.
  • The required E2E consolidated report deliberately doesn't wait for it, so a cold Windows release build doesn't sit in front of every merge. That job now downloads only the Linux artifact by exact name, so whether Windows shows up no longer depends on which job finished first.
• The WSL lane is unchanged. It has no usable GPU, so it keeps running the whole non-GPU suite inside a real WSL2 distribution, our only real-distro coverage.

Tradeoffs for maintainers

• Real-GPU regressions are caught later. The merge queue no longer runs the full real-GPU suite. A regression outside the canaries and the @merge-queue serves shows up on the next push-to-main run or nightly, not before merge. The @gpu-smoke set is deliberately the unmasked per-engine checks: the lemonade default-engine serves are flaky-xfail on Linux lemonade (EAI-7423), so on their own they could never turn a lane red. Tag a scenario @gpu-smoke to keep it blocking.
• Native-Windows non-GPU coverage is advisory for now. It moves from the required Strix Windows lane to the new hosted job, which isn't required yet. Once it has a green record, an admin should add E2E tests (Windows) to the required checks; a workflow edit can't do that.
• Two chat scenarios lose a bonus real serve. chat-06 and chat-07 used to do a real serve on GPU lanes and a mock serve elsewhere. Now they only run with the mock. Their own comments note that real inference is covered by the serve-*-inference scenarios.
• The Windows job starts immediately. It depends only on changes, not on the Linux build-and-test, and uses the sccache Azure backend under its own prefix (rocm-cli/e2e-windows), written only on push/merge_group, like every other job.

Making E2E tests (Windows) required

Checked against what can make a required check unsafe here:

• It always reports. The job-level if: is the same always() form the required e2e job uses, with the work gated at step level. On a non-heavy PR it reports success after a few seconds, and on push/merge_group changes forces heavy, so the merge queue always runs it for real.
• It runs in the merge queue. ci.yml triggers on merge_group.
• No name collision. No other job is called E2E tests (Windows).
• Fork-safe. No secret is required; without the cache credential it compiles without a shared cache.
• Stable so far. 6 of 6 runs green on windows-latest (the PR run plus 5 parallel dispatches): identical 106 scenarios every time, 0 unexpected failures, and the same 2 deterministic xfails.
• Latency. Cold (cache empty, as on every PR so far) it takes about 19–20 minutes, finishing about 6.5 minutes after the last required hosted check. About 14 of those minutes are compile, which the cache should cut once a push or merge_group run has written it. That's not measurable before merge, since PRs only read the cache. Either way it's well short of the self-hosted lanes, which are what bound merge latency.

Promote it only after this PR is on main. A required check that doesn't exist on the base blocks every PR that doesn't contain the job, and open PRs based before the merge need a rebase to produce it. I'd also let it build a short record on main first: the self-hosted lanes were promoted on dozens of runs, and six is a small sample for network-dependent scenarios. The admin step is POST /repos/ROCm/rocm-cli/branches/main/protection/required_status_checks/contexts with ["E2E tests (Windows)"], after backing up the current protection.

Scenario coverage (AGENTS.md §3)

CI plumbing only; no CLI behaviour changes. The selection rule is unit-tested (a_gpu_lane_runs_only_its_gpu_scenarios_narrowed_to_the_smoke_test_on_prs), and the lane wiring, WSLENV forwarding and report artifact names are pinned by xtask contract tests.

Test plan

• On the current head (rebased on test(e2e): describe GPU hosts with simulated machines (ROCMAI-677) #517 at 124b181, itself on main 1c33248): cargo test -p e2e-cucumber --lib --bins --test feature_naming (187 + 1 + 4), cargo test -p xtask (318) and cargo test -p e2e-report (66) all pass. Clippy (both invocations, Linux and x86_64-pc-windows-gnu) and fmt are clean.

• Earlier, before the rebase: E2E_GPU_ONLY=1 cargo xtask e2e locally on a GPU-less host: 0 scenarios run. All 224 resolve to skip; 166 carry the new reason, and the others keep their own (lifecycle, nightly, real GPU, docker).

• Earlier, before the rebase: the new Windows job was 6 of 6 green on windows-latest; see above. The GPU lanes ran under the new rule (Strix Windows: 2 smoke scenarios in 7 minutes).

• If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (n/a)

• If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (n/a)

• Every new or changed user-facing message was read against the code path that runs after it. (n/a)

[rominf]

Rebased onto #517's new head (124b181, on main at 1c33248). Head is now 7c98a8a. It stays a draft until #517 merges.

What changed to fit main:

• main dropped the per-PR R9700 and MI350P lanes (ci(e2e): trim per-PR GPU lanes and add nightly-channel leg to strix-ubuntu (ROCMAI-125) #463) and added W7900 (ci(e2e): add a W7900 self-hosted E2E lane (EAI-8763) #454). E2E_GPU_ONLY=1 is now on MI300X, Strix Ubuntu, Strix Windows and W7900. E2E_GPU_SMOKE_ONLY is on those four and on WSL, and WSL still forwards it through WSLENV.
• MI300X no longer runs in the merge queue (ci: supersede PR runs and skip MI300X in the merge queue #696), so it has no E2E_MERGE_QUEUE. The contract test self_hosted_gpu_lanes_run_only_the_gpu_smoke_test_on_prs_and_in_the_merge_queue covers the new lane list and expects no E2E_MERGE_QUEUE on MI300X. I checked that it fails when W7900 loses E2E_GPU_ONLY.
• docs/ci-hardware-testing.md follows main's lane list and artifact names, and adds e2e-windows-report.
• ci: run Linux CI on both Ubuntu LTS images; fork-free reclaim decoys with liveness checks (ROCMAI-670) #718 is not on main, so the new E2E tests (Windows) job keeps windows-latest, and the Linux jobs stay on ubuntu-latest. Its sccache probe step matches windows-build-and-test on main.

Local checks passed: xtask 318, e2e-cucumber lib 187, feature_naming 4, e2e-report 66, and clippy and fmt on Linux and Windows-gnu. While a PR is stacked, CI does not run on it.