[VideoGenX] run matched H3 workloads on SGLang and vLLM-Omni

.github/workflows/test-h3-spec.yml

@@ -0,0 +1,43 @@
+name: Test H3 Spec
+
+on:
+  pull_request:
+    paths: &h3-spec-paths
+      - 'experimental/video-generation/specs/**'
+      - 'experimental/video-generation/tests/**'
+      - '.github/workflows/test-h3-spec.yml'
+  push:
+    paths: *h3-spec-paths
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: H3 spec loader and hashes
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+      - name: Test spec loader and hashes
+        run: >-
+          uv run --no-project --python 3.12
+          --with 'pyyaml>=6,<7' --with 'pytest>=8,<9'
+          python -m pytest experimental/video-generation/tests/test_h3_spec.py -q
+      - name: Lint and format
+        # The infx config never lints tests/, so assert (S101) is allowed there.
+        # These standalone modules have no package, so INP001 does not apply.
+        run: |
+          uvx ruff@0.14.0 check --config inferencex-e2e/infx/ruff.toml --ignore INP001 \
+            experimental/video-generation/specs/h3_spec.py
+          uvx ruff@0.14.0 check --config inferencex-e2e/infx/ruff.toml --ignore INP001,S101 \
+            experimental/video-generation/tests/test_h3_spec.py
+          uvx ruff@0.14.0 format --check --config inferencex-e2e/infx/ruff.toml \
+            experimental/video-generation/specs/h3_spec.py experimental/video-generation/tests/test_h3_spec.py

experimental/video-generation/evaluator/cli.py

@@ -100,6 +100,33 @@ def _mvp_command(arguments: argparse.Namespace) -> int:
                 host=arguments.host, record=arguments.record,
             )
 
+        if arguments.command == "srt-yaml-serve":
+            from .mvp_srt_selected import selected_plan, serve
+
+            plan = selected_plan(arguments.spec, arguments.set, arguments.plan_sha256)
+            serve(
+                plan, runtime=arguments.runtime, runtime_revision=arguments.runtime_revision,
+                model_path=arguments.model_path, port=arguments.port,
+                gpus=arguments.gpus, output=arguments.record,
+            )
+            return 0
+
+        if arguments.command == "srt-yaml-client":
+            from .mvp_srt_selected import run_matrix, selected_plan
+
+            plan = selected_plan(arguments.spec, arguments.set, arguments.plan_sha256)
+            runs = run_matrix(
+                plan, runtime=arguments.runtime, runtime_revision=arguments.runtime_revision,
+                endpoint=arguments.endpoint, output=arguments.output,
+                concurrencies=arguments.concurrencies, gpus=arguments.gpus,
+                window_dir=arguments.window_dir,
+            )
+            for concurrency, run in runs.items():
+                print(json.dumps({"concurrency": concurrency, "summary": run.get("summary"),
+                                  "status": run.get("status")}, indent=2, sort_keys=True))
+            return 0 if all(run["summary"]["valid"] == run["summary"]["scheduled"]
+                            and run["summary"]["scheduled"] for run in runs.values()) else 1
+
         if arguments.command == "srt-client":
             from .mvp_srt import run_srt_client, run_srt_matrix
 
@@ -264,6 +291,29 @@ def _parser() -> argparse.ArgumentParser:
     srt_client.add_argument("--window-dir", type=Path,
                             help="srt-slurm measurement-window directory (SRT_MEASUREMENT_WINDOW_DIR)")
 
+    selected_server = subparsers.add_parser("srt-yaml-serve", help="serve a selected YAML video workload")
+    selected_server.add_argument("--spec", required=True, type=Path)
+    selected_server.add_argument("--set", required=True)
+    selected_server.add_argument("--plan-sha256", required=True)
+    selected_server.add_argument("--runtime", required=True, choices=("sglang", "vllm-omni"))
+    selected_server.add_argument("--runtime-revision", required=True)
+    selected_server.add_argument("--model-path", required=True, type=Path)
+    selected_server.add_argument("--port", required=True, type=int)
+    selected_server.add_argument("--gpus", required=True, type=int)
+    selected_server.add_argument("--record", required=True, type=Path)
+
+    selected_client = subparsers.add_parser("srt-yaml-client", help="measure a selected YAML video workload")
+    selected_client.add_argument("--spec", required=True, type=Path)
+    selected_client.add_argument("--set", required=True)
+    selected_client.add_argument("--plan-sha256", required=True)
+    selected_client.add_argument("--runtime", required=True, choices=("sglang", "vllm-omni"))
+    selected_client.add_argument("--runtime-revision", required=True)
+    selected_client.add_argument("--endpoint", required=True)
+    selected_client.add_argument("--output", required=True, type=Path)
+    selected_client.add_argument("--concurrencies", required=True, nargs="+", type=int)
+    selected_client.add_argument("--gpus", required=True, type=int)
+    selected_client.add_argument("--window-dir", required=True, type=Path)
+
     srt_finalize = subparsers.add_parser(
         "srt-finalize",
         help="assemble the app bundle from an srt-slurm H3 job and its power validation",

experimental/video-generation/evaluator/mvp_srt_selected.py

@@ -0,0 +1,145 @@
+"""Run a selected YAML video workload through the native srt-slurm service."""
+
+from __future__ import annotations
+
+import os
+import re
+import time
+from pathlib import Path
+
+from specs import h3_spec
+
+from .mvp_runner import run_plan, validate_plan
+from .mvp_srt import BENCHMARK_STEM, CELL_DIR, _write, benchmark_window
+
+
+def selected_plan(path: Path, set_id: str, expected_sha256: str) -> dict:
+    spec = h3_spec.load_spec(path)
+    if set_id not in spec["sets"]:
+        raise ValueError(f"unknown selected workload set: {set_id}")
+    plan = validate_plan(h3_spec.plan_for(spec, set_id))
+    if h3_spec.plan_sha256(plan) != expected_sha256:
+        raise ValueError("selected workload plan does not match the bound plan_sha256")
+    return plan
+
+
+def h3_vllm_model_view(snapshot: Path, output: Path) -> Path:
+    """Expose only the tested FL2VA files without copying weights or mutating the cache."""
+    required = ("FL2VA", "model_index.json", "LICENSE", "README.md", "Ref2VA/model_index.json")
+    for name in required:
+        if not (snapshot / name).exists():
+            raise ValueError(f"H3 model snapshot lacks {name}")
+    output.mkdir(parents=True, exist_ok=False)
+    for name in required:
+        target = output / name
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.symlink_to(snapshot / name)
+    return output
+
+
+def server_argv(plan: dict, runtime: str, model_path: Path, port: int, gpus: int) -> list[str]:
+    if plan["model_id"] != "MiniMaxAI/MiniMax-H3" or gpus != 2:
+        raise ValueError("this H3 native profile requires two GPUs")
+    if runtime == "sglang":
+        from .mvp_gpu_job import _server_argv
+
+        spec = {
+            "baseline": {"python": "/usr/bin/python3"},
+            "model": {"path": str(model_path), "revision": plan["model_revision"]},
+            "plan": plan,
+            "server": {"ulysses_degree": 2, "tp_size": 1, "encoder_parallel": "auto",
+                       "performance_mode": "speed", "dit_cpu_offload": False},
+            "gpu_uuids": ["assigned"] * gpus,
+            "port": port,
+        }
+        return _server_argv(spec, "baseline", host="0.0.0.0")
+    if runtime == "vllm-omni":
+        return [
+            "vllm", "serve", str(model_path), "--served-model-name", plan["model_id"],
+            "--omni", "--host", "0.0.0.0", "--port", str(port),
+            "--trust-remote-code", "--task-type", "fl2va", "--num-gpus", "2",
+            "--tensor-parallel-size", "2", "--usp", "1", "--ring", "1",
+            "--text-encoder-tp-size", "2", "--vae-patch-parallel-size", "2",
+            "--vae-parallel-mode", "tile", "--vae-use-tiling", "--enforce-eager",
+        ]
+    raise ValueError(f"unsupported native video runtime: {runtime}")
+
+
+def serve(
+    plan: dict, *, runtime: str, runtime_revision: str, model_path: Path,
+    port: int, gpus: int, output: Path,
+) -> None:
+    if not re.fullmatch(r"[0-9a-f]{40}", runtime_revision):
+        raise ValueError("runtime revision must be an immutable commit")
+    if not model_path.is_dir():
+        raise ValueError("the pinned model snapshot is unavailable")
+    if runtime == "vllm-omni":
+        model_path = h3_vllm_model_view(model_path, output / "model-view")
+    argv = server_argv(plan, runtime, model_path, port, gpus)
+    from .mvp_gpu_job import cuda_devices
+
+    uuids = cuda_devices()
+    if len(uuids) != gpus:
+        raise RuntimeError("visible GPU count differs from the native recipe")
+    env = os.environ.copy()
+    env["VGBENCH_GPU_UUIDS"] = ",".join(uuids)
+    env["HF_HUB_OFFLINE"] = "1"
+    env["TRANSFORMERS_OFFLINE"] = "1"
+    if runtime == "vllm-omni":
+        env["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
+        env["VLLM_OMNI_VIDEO_SYNC_TIMEOUT"] = "14400"
+        storage = output / "storage"
+        storage.mkdir(parents=True, exist_ok=True)
+        env["VLLM_OMNI_SERVER_STORAGE__PATH"] = str(storage)
+    _write(output / "spec.json", {
+        "schema_version": "0.1.0",
+        "plan": plan,
+        "gpu_uuids": uuids,
+        "baseline": {
+            "name": runtime,
+            "revision": runtime_revision,
+            "model_path": str(model_path),
+            "model_revision": plan["model_revision"],
+            "argv": argv,
+        },
+        "serving": {"mode": "closed_loop"},
+    })
+    _write(output / "launch.json", {
+        "runtime": runtime, "runtime_revision": runtime_revision,
+        "model_id": plan["model_id"], "model_revision": plan["model_revision"],
+        "plan_sha256": h3_spec.plan_sha256(plan), "gpu_uuids": uuids,
+        "argv": argv, "slurm_job_id": env.get("SLURM_JOB_ID"),
+    })
+    os.execvpe(argv[0], argv, env)
+
+
+def run_matrix(
+    plan: dict, *, runtime: str, runtime_revision: str, endpoint: str,
+    output: Path, concurrencies: list[int], gpus: int, window_dir: Path,
+    request_seconds: float = 900,
+) -> dict[int, dict]:
+    if runtime not in {"sglang", "vllm-omni"}:
+        raise ValueError("unsupported native video runtime")
+    if not concurrencies or len(set(concurrencies)) != len(concurrencies) or any(c < 1 for c in concurrencies):
+        raise ValueError("concurrencies must be distinct positive integers")
+    results = {}
+    for concurrency in concurrencies:
+        cell = output / CELL_DIR.format(concurrency=concurrency)
+        before = time.time() - time.monotonic()
+        run = run_plan(
+            plan, cell / "baseline", endpoint=endpoint, runtime=runtime,
+            runtime_revision=runtime_revision,
+            hardware_label=f"native srt-slurm {gpus} GPUs",
+            model_revision=plan["model_revision"], timeout_seconds=request_seconds,
+            serving_concurrency=concurrency,
+        )
+        after = time.time() - time.monotonic()
+        contract = benchmark_window(run, utc_minus_monotonic=(before + after) / 2)
+        contract["clock"]["offset_spread_seconds"] = abs(after - before)
+        path = cell / f"{BENCHMARK_STEM.format(concurrency=concurrency)}.json"
+        _write(path, contract)
+        from infx.results.power.window import write_window
+
+        write_window(path, concurrency, window_dir)
+        results[concurrency] = run
+    return results

experimental/video-generation/pyproject.toml

@@ -7,7 +7,7 @@ name = "inferencex-h3-video-ci"
 version = "0.1.0"
 description = "Controlled H3 runtime regression measurements for InferenceX CI"
 requires-python = ">=3.12"
-dependencies = ["av==16.1.0", "numpy==2.3.5"]
+dependencies = ["av==16.1.0", "numpy==2.3.5", "pyyaml==6.0.3"]
 
 [project.optional-dependencies]
 dev = ["pytest==8.4.2"]

experimental/video-generation/specs/README.md

@@ -0,0 +1,43 @@
+# H3 workload spec
+
+`h3.yaml` defines the MiniMax-H3 text-to-video-and-audio workload that SGLang and vLLM-Omni both run.
+`h3_spec.py` loads and validates it, then builds the plan object that the runner from PR #2916 executes and hashes.
+`h3-vllm-omni-serving.md` records the pinned vLLM-Omni image, server command, parallelism and attention backend that served this workload on 2x H100.
+
+## What the YAML records
+
+- `model` and every `generation` field hold a `value` and a `source` citation of the form `<ref>@<commit>:<path>:<line>`.
+- `sent.sglang` and `sent.vllm-omni` say whether that engine's request carries the field.
+  SGLang rejects caller-set fps and frame counts, runs at a fixed 24 fps, and derives the frame count and dimensions from its `target`.
+  Only vLLM-Omni receives fps, frame count, width, and height.
+- `checked_on_clip` says whether the delivered clip is compared to the value.
+  The runner checks the clip's duration against `frame_count / fps` (107 / 24 = 4.458 s), so `duration_seconds` itself is not checked.
+- `prompts` defines each prompt once with its motion and sound requirements.
+  Each entry in `sets` lists cases that pair a prompt with a seed, and its `seed_sources` cites every seed.
+- `reference` records the SGLang reference run. It is informational, and no code reads or hashes it.
+  The run used workflow commit f7c7f58, whose spec, `mvp_runner.py`, and `mvp_media.py` match 0cb4c847d byte for byte.
+
+## Hash contract
+
+`canonical_json_bytes` is a byte-for-byte copy of the function in #2916's `evaluator/mvp_runner.py`.
+It serializes with sorted keys, compact separators, UTF-8, and no NaN.
+
+- `plan_sha256(plan_for(spec, set_id))` equals the `plan_sha256` that #2916's runner records for the same plan.
+  The `frozen-c124` set reproduces #2916's `h3-serving-2gpu-c124-v1` plan, whose hash is `8d51a472caa7db0f5d4c33adec1e9c9f852e7144cf25356085802edaf7d80d68`.
+- `spec_sha256(spec)` covers the model, the generation values, and each set's resolved cases and counts.
+- Citations, `sent`, `checked_on_clip`, and `reference` are outside both hashes, as are YAML key order, comments, and formatting.
+  Neither hash includes GPU UUIDs, job IDs, ports, or hosts.
+- `plan_for(spec, set_id, repetitions=..., warmup_runs=...)` overrides the counts.
+  If either count differs from the set's own, the plan ID gains a `-r<repetitions>-w<warmup_runs>` suffix, so a shortened run never shares an ID or a hash with the full one.
+
+## Run the tests and print the hashes
+
+From the repository root:
+
+```bash
+uv run --no-project --python 3.12 --with 'pyyaml>=6,<7' --with 'pytest>=8,<9' python -m pytest experimental/video-generation/tests/test_h3_spec.py -q
+uv run --no-project --python 3.12 --with 'pyyaml>=6,<7' python experimental/video-generation/specs/h3_spec.py
+```
+
+The second command prints `spec_sha256` and each set's `plan_id` and `plan_sha256` as JSON.
+To hash another spec file, pass its path as the argument.

experimental/video-generation/specs/h3-vllm-omni-serving.md

@@ -0,0 +1,49 @@
+# Serving MiniMax-H3 on vLLM-Omni
+
+Configuration observed for one MiniMax-H3 request on 2x H100 80GB on 2026-10-17.
+
+## Pins
+
+| Component | Value |
+| --- | --- |
+| vLLM-Omni | release v0.30.0, commit `a8576ccb725c4e21cd13c3eb5f9a546b21149d2b` |
+| Image | `vllm/vllm-omni:v0.30.0`, manifest digest `sha256:fc77cfaac7b1b43c30cbf32eaea214864434c6e0f303e6e9eb9f64104241a994`, linux/amd64 `sha256:b09c9eab7d2e30817208de92ccb482756d13d4339377d7cc4c4ce9f47bf04174` |
+| In-image versions | vllm 0.30.0, vllm-omni 0.30.0, torch 2.13.0+cu130 |
+| Model | `MiniMaxAI/MiniMax-H3` revision `42ed227ee7df40d41602854ae760620d6eb651fe` (same as `specs/h3.yaml` and PR #2916) |
+
+Only the `FL2VA/` partition plus root `model_index.json`, `LICENSE`, `README.md` and `Ref2VA/model_index.json` need to be on disk (129 GB); `--task-type fl2va` loads nothing else. Leave the root `modular_model_index.json` out of a local copy: with it present, vllm-omni misdetects the directory as FastH3 (upstream issue [#8796](https://github.com/vllm-project/vllm-omni/issues/8796)).
+
+## Server command
+
+```
+vllm serve <snapshot dir> --served-model-name MiniMaxAI/MiniMax-H3 --omni --host 0.0.0.0 --port 30283 \
+  --trust-remote-code --task-type fl2va --num-gpus 2 --tensor-parallel-size 2 --usp 1 --ring 1 \
+  --text-encoder-tp-size 2 --vae-patch-parallel-size 2 --vae-parallel-mode tile --vae-use-tiling \
+  --enforce-eager
+```
+
+`<snapshot dir>` is the Hugging Face cache snapshot directory for the revision above. Serving the Hub ID with `--revision` while `HF_HUB_OFFLINE=1` fails at startup with `huggingface_hub.errors.IncompleteSnapshotError` because `snapshot_download` requires every file of the repository and resolves `refs/main`; the directory path avoids that. `--served-model-name` keeps the model name the evaluator sends.
+
+Environment:
+
+```
+HF_HUB_OFFLINE=1
+TRANSFORMERS_OFFLINE=1
+HF_HUB_CACHE=<cache root>
+VLLM_WORKER_MULTIPROC_METHOD=spawn
+VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400
+```
+
+## Parallelism
+
+Tensor parallel 2 across both GPUs with Ulysses and ring degree 1, text encoder TP 2, VAE patch-parallel 2 in tile mode with tiling. This is the upstream recipe's only two-GPU resident layout; pure Ulysses replicates the whole DiT on every rank, so SGLang's `ulysses_degree 2 / tp_size 1` layout from #2916 does not fit 80 GB here. The observed request used `--enforce-eager`.
+
+## Attention backend
+
+Resolved by the server at startup, no flag passed: `FLASH_ATTN` for `role='minimax_h3.token_refiner'` and for `role='self'` (log line from `selector.py:84/93`).
+
+## Observed on 2x H100 (informational, not a benchmark)
+
+- Model load: 61.3 GiB and 282 s per TP worker; `/health` 200 about 5 min 20 s after launch.
+- One request from the `frozen-c124` set with `repetitions=1, warmup_runs=0`: 50 denoise steps at 2.35 s/it, 125.4 s submit to validated clip, both GPUs at the 700 W cap with 70.2 GB used.
+- The request carries width, height, fps and num_frames plus `extra_params` (`task`, `target`, `audio_flow_shift`); the server logged `Applied extra_params` and produced 1344x768, 107 frames, 24 fps, 32 kHz stereo, matching the spec's clip checks.