diff --git a/README.md b/README.md
index 2687de1..db4a70d 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,5 @@
+
@@ -141,6 +142,10 @@ uv sync --extra tutorials # or: pip install -e ".[tutorials]"
uv run jupyter lab tutorials/
```
+Continue with [Tutorial 2: multi-GPU quickstart](tutorials/02_multi_gpu_quickstart.ipynb)
+to decode on up to eight GPUs using the process backend, check decoding
+quality, and compare GPU counts with 1,024 shots per GPU.
+
## Benchmarking
`pelagos_benchmark.py` is a full-featured benchmark CLI. Run
diff --git a/tutorials/02_multi_gpu_quickstart.ipynb b/tutorials/02_multi_gpu_quickstart.ipynb
new file mode 100644
index 0000000..c1cb65d
--- /dev/null
+++ b/tutorials/02_multi_gpu_quickstart.ipynb
@@ -0,0 +1,324 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-00",
+ "metadata": {},
+ "source": [
+ "Copyright (c) 2026 Advanced Micro Devices, Inc. Licensed under the Apache License, Version 2.0; see the `LICENSE` file in the repository root.\n",
+ "\n",
+ "# Tutorial 2 — Decoding a qLDPC code on multiple GPUs\n",
+ "\n",
+ "This notebook extends the [single-GPU quickstart](01_single_gpu_quickstart.ipynb)\n",
+ "to multiple GPUs:\n",
+ "\n",
+ "1. Build a decoding problem for the [[72,12,6]] bivariate bicycle code.\n",
+ "2. Decode a batch using one worker process per GPU.\n",
+ "3. Check the syndrome solutions and logical error rate.\n",
+ "4. Compare throughput with 1, 2, 4, and 8 GPUs, keeping 1,024 shots per GPU.\n",
+ "\n",
+ "Pelagos's process backend keeps one worker process per GPU and splits\n",
+ "each batch across them. Each worker runs the existing Triton decoder. Reusing the\n",
+ "decoder keeps those workers alive between calls.\n",
+ "\n",
+ "**Requirements:** at least two AMD GPUs with ROCm on the same machine, and a clone of\n",
+ "this repository with the tutorial dependencies. Use a ROCm PyTorch/Triton environment,\n",
+ "then run from the clone:\n",
+ "\n",
+ "```bash\n",
+ "pip install -e \".[tutorials]\"\n",
+ "jupyter lab tutorials/\n",
+ "```\n",
+ "\n",
+ "Run cells in order. First decodes include compilation and autotuning and may take\n",
+ "longer.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-01",
+ "metadata": {},
+ "source": [
+ "## 0. Setup\n",
+ "\n",
+ "GPU IDs below refer to the devices visible to this notebook. Set any GPU visibility\n",
+ "variables before starting Jupyter. The default selects up to eight visible GPUs.\n",
+ "With eight GPUs, the scaling section runs all four counts: 1, 2, 4, and 8.\n",
+ "With fewer GPUs, it runs the available counts.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-02",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gzip\n",
+ "import time\n",
+ "from pathlib import Path\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import stim\n",
+ "import torch\n",
+ "\n",
+ "from pelagos.process import ProcessRelayBPDecoder\n",
+ "from pelagos.stim.sinter.check_matrices import CheckMatrices\n",
+ "\n",
+ "assert torch.cuda.is_available(), \"Pelagos needs a GPU (CUDA or ROCm); there is no CPU fallback.\"\n",
+ "DEVICES = list(range(min(8, torch.cuda.device_count())))\n",
+ "assert len(DEVICES) >= 2 and len(set(DEVICES)) == len(DEVICES)\n",
+ "assert all(0 <= d < torch.cuda.device_count() for d in DEVICES), \"Allocate at least two GPUs.\"\n",
+ "\n",
+ "def find_repo_root():\n",
+ " for p in [Path.cwd(), *Path.cwd().parents]:\n",
+ " if (p / \"tests/testdata/bicycle_bivariate\").is_dir():\n",
+ " return p\n",
+ " raise FileNotFoundError(\"Run this notebook from inside the pelagos clone.\")\n",
+ "\n",
+ "REPO = find_repo_root()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-03",
+ "metadata": {},
+ "source": [
+ "## 1. Load a circuit and sample shots\n",
+ "\n",
+ "As in Tutorial 1, we load a bundled stim circuit at physical error rate 0.003,\n",
+ "then construct its check matrix $H$, observable matrix $O$, and error priors.\n",
+ "Sampling also gives the true logical observable flips for each shot.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-04",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "paths = list((REPO / \"tests/testdata/bicycle_bivariate\").glob(\n",
+ " \"*bicycle_bivariate_72_12_6_memory_X,*error_rate=0.003,*.stim.gz\"\n",
+ "))\n",
+ "assert len(paths) == 1, f\"Expected one circuit, found {len(paths)}.\"\n",
+ "circuit = stim.Circuit(gzip.decompress(paths[0].read_bytes()).decode())\n",
+ "dem = circuit.detector_error_model(decompose_errors=False)\n",
+ "cm = CheckMatrices.from_dem(dem)\n",
+ "H = cm.check_matrix.tocsr()\n",
+ "O = cm.observables_matrix.tocsr()\n",
+ "print(\"Circuit:\", paths[0].name)\n",
+ "print(\"H:\", H.shape, \"| nonzeros:\", H.nnz, \"| logical observables:\", O.shape[0])\n",
+ "\n",
+ "PER_GPU_BATCH = 1024\n",
+ "NUM_SHOTS = PER_GPU_BATCH * len(DEVICES)\n",
+ "detectors, observables, _ = dem.compile_sampler(seed=2026).sample(NUM_SHOTS)\n",
+ "syndromes = np.ascontiguousarray(detectors, dtype=np.uint8)\n",
+ "\n",
+ "CONFIG = dict(\n",
+ " H_csr=H, error_priors=cm.error_priors,\n",
+ " algo=\"relay\", perf=\"throughput\", data_type=\"float32\",\n",
+ " pre_iter=80, num_sets=100, set_max_iter=60,\n",
+ " gamma0=0.65, gamma_dist_interval=(-0.24, 0.66), stop_nconv=1,\n",
+ " seed=1234, gamma_draw=\"broadcast\", scheduling=\"flood-fill\",\n",
+ ")\n",
+ "print(\"Decoder settings:\", {k: v for k, v in CONFIG.items() if k not in (\"H_csr\", \"error_priors\")})\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-05",
+ "metadata": {},
+ "source": [
+ "## 2. Decode on multiple GPUs\n",
+ "\n",
+ "Pass the syndromes as a NumPy array or CPU tensor of shape `[batch, checks]`.\n",
+ "The workers decode their assigned shots on the GPUs and return the results in\n",
+ "the original input order.\n",
+ "\n",
+ "With 1,024 shots per GPU and eight GPUs, the global batch contains 8,192 shots.\n",
+ "Reuse the decoder across batches. The `with` block automatically closes its\n",
+ "worker processes when the block finishes.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-06",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "with ProcessRelayBPDecoder(**CONFIG, device=DEVICES) as decoder:\n",
+ " decoder.decode(syndromes) # Warm this batch shape; workers persist.\n",
+ " result = decoder.decode(syndromes)\n",
+ "\n",
+ "# Returned CPU tensors remain valid after the decoder is closed.\n",
+ "assert result[\"errors\"].device.type == \"cpu\"\n",
+ "errors = result[\"errors\"].numpy()\n",
+ "assert errors.shape == (NUM_SHOTS, H.shape[1])\n",
+ "assert np.all((errors == 0) | (errors == 1))\n",
+ "print(\"Decoded error patterns:\", errors.shape)\n",
+ "print(\"Result fields:\", sorted(result))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-07",
+ "metadata": {},
+ "source": [
+ "The result fields have the same meaning as in Tutorial 1:\n",
+ "\n",
+ "| key | shape | meaning |\n",
+ "|---|---|---|\n",
+ "| `errors` | `[B, V]` | estimated error pattern for each shot |\n",
+ "| `found_count` | `[B]` | number of valid solutions found; `> 0` means the shot converged |\n",
+ "| `iterations` | `[B]` | BP iterations spent on each shot |\n",
+ "| `weights` | `[B]` | negative log-likelihood of the returned solution |\n",
+ "\n",
+ "The GPU workers return the decoded results in the original input order.\n",
+ "\n",
+ "## 3. Check the result\n",
+ "\n",
+ "A syndrome is satisfied when $H\\hat e=s\\pmod 2$. Logical decoding succeeds when\n",
+ "$O\\hat e$ matches the sampled observable flips. These are different checks: a valid\n",
+ "syndrome solution can still have a logical error. We report both, plus the fraction\n",
+ "of shots for which the decoder found a valid solution.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-08",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "satisfied = np.all(((H @ errors.T) % 2).T == syndromes, axis=1)\n",
+ "predicted_observables = ((O @ errors.T) % 2).T.astype(bool)\n",
+ "logical_failures = np.any(predicted_observables != observables, axis=1)\n",
+ "converged = result[\"found_count\"].numpy() > 0\n",
+ "failure_count = int(logical_failures.sum())\n",
+ "print(f\"Syndrome satisfied: {satisfied.mean():.3%}\")\n",
+ "print(f\"Converged: {converged.mean():.3%}\")\n",
+ "print(f\"Logical failures: {failure_count}/{NUM_SHOTS}\")\n",
+ "print(f\"Logical error rate: {failure_count / NUM_SHOTS:.5f} per shot (all 12 logical qubits, 6 rounds)\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-09",
+ "metadata": {},
+ "source": [
+ "Decoding succeeds when the predicted observable flips match the true flips.\n",
+ "\n",
+ "## 4. Scale from one to eight GPUs\n",
+ "\n",
+ "We keep 1,024 shots per GPU and increase the global batch with\n",
+ "GPU count. For 1, 2, 4, and 8 GPUs, the global batches are 1,024, 2,048, 4,096,\n",
+ "and 8,192 shots. Every run uses the process backend, including the one-GPU baseline.\n",
+ "\n",
+ "Each setting gets a warm-up at its actual batch shape, followed by three measured\n",
+ "repetitions. Workers are reused. The timer covers the blocking\n",
+ "`decode()` calls: host dispatch, transfers, GPU decoding, and returned CPU results.\n",
+ "Sampling, worker startup, warm-up, and quality calculations are outside the timer.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-10",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "GPU_COUNTS = [n for n in (1, 2, 4, 8) if n <= len(DEVICES)]\n",
+ "REPEATS = 3\n",
+ "# Reuse prefixes of the sampled shots as the global batch grows.\n",
+ "rows = []\n",
+ "\n",
+ "for count in GPU_COUNTS:\n",
+ " batch_size = PER_GPU_BATCH * count\n",
+ " batch = syndromes[:batch_size]\n",
+ " with ProcessRelayBPDecoder(**CONFIG, device=DEVICES[:count]) as decoder:\n",
+ " decoder.decode(batch) # Warm up once at this batch shape.\n",
+ " seconds = []\n",
+ " for _ in range(REPEATS):\n",
+ " started = time.perf_counter()\n",
+ " output = decoder.decode(batch)\n",
+ " seconds.append(time.perf_counter() - started)\n",
+ " rates = batch_size / np.asarray(seconds)\n",
+ " rows.append(dict(gpus=count, batch=batch_size,\n",
+ " median=float(np.median(rates)), low=float(rates.min()),\n",
+ " high=float(rates.max()), seconds=seconds))\n",
+ "\n",
+ "baseline = rows[0][\"median\"]\n",
+ "for row in rows:\n",
+ " row[\"speedup\"] = row[\"median\"] / baseline\n",
+ "\n",
+ "print(f\"{'GPUs':>4} {'shots/GPU':>10} {'global batch':>12} {'shots/s':>10} {'speedup':>8}\")\n",
+ "for row in rows:\n",
+ " print(f\"{row['gpus']:>4} {PER_GPU_BATCH:>10} {row['batch']:>12} \"\n",
+ " f\"{row['median']:>10.0f} {row['speedup']:>7.2f}x\")\n",
+ " print(\" measured seconds:\", [round(t, 4) for t in row[\"seconds\"]])\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "multi-gpu-11",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "fig, ax = plt.subplots(figsize=(6, 3.5))\n",
+ "y = np.array([r[\"speedup\"] for r in rows])\n",
+ "low = np.array([r[\"low\"] / baseline for r in rows])\n",
+ "high = np.array([r[\"high\"] / baseline for r in rows])\n",
+ "ax.errorbar(GPU_COUNTS, y, yerr=[y - low, high - y],\n",
+ " marker=\"o\", capsize=3, label=\"multi-GPU scaling\")\n",
+ "ax.plot(GPU_COUNTS, GPU_COUNTS, \"k--\", alpha=0.4, label=\"ideal linear scaling\")\n",
+ "ax.set(xticks=GPU_COUNTS, xlabel=\"GPUs\", ylabel=\"Throughput / one-GPU median\",\n",
+ " title=\"Scaling: 1,024 shots per GPU\")\n",
+ "ax.grid(alpha=0.3)\n",
+ "ax.legend()\n",
+ "plt.tight_layout()\n",
+ "plt.show()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "multi-gpu-12",
+ "metadata": {},
+ "source": [
+ "The points show median throughput relative to the corresponding one-GPU median.\n",
+ "Bars show the observed minimum and maximum rates divided by that same baseline.\n",
+ "\n",
+ "Each additional GPU handles another 1,024 shots. The speedup compares the total\n",
+ "shots decoded per second with the one-GPU run. The dashed line shows proportional\n",
+ "growth: twice as many GPUs would give twice the aggregate throughput.\n",
+ "\n",
+ "For the full multi-GPU design and performance evaluation, see our paper,\n",
+ "*Scaling Pêlagos: Multi-GPU Orchestration for Relay Belief-Propagation Decoding*.\n",
+ "\n",
+ "## Next steps\n",
+ "\n",
+ "* To select particular GPUs, edit `DEVICES`, for example `[0, 2, 4, 6]`.\n",
+ " `GPU_COUNTS` selects how many of those devices each run uses.\n",
+ "* Lower `PER_GPU_BATCH` if GPU memory is limited. Larger codes also need smaller\n",
+ " per-GPU batches; keep `H.nnz × per-GPU batch` below the decoder's indexing limit.\n",
+ "* For the higher-level `RelayDecoder` / `ObservableDecoderRunner` API, see\n",
+ " [process-backend documentation](../docs/process-backend.md).\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/tutorials/README.md b/tutorials/README.md
index e1bb7fa..a5760d1 100644
--- a/tutorials/README.md
+++ b/tutorials/README.md
@@ -1,3 +1,5 @@
+
+
# Tutorials
Hands-on Jupyter notebooks for Pelagos. Run them from a clone of the repository, because they
@@ -14,5 +16,9 @@ cross-check against the `relay-bp` CPU reference.
| # | Notebook | What you learn | Runtime* |
|---|---|---|---|
| 1 | [Single-GPU quickstart](01_single_gpu_quickstart.ipynb) | Decoder API; from a stim circuit to a logical error rate on the [[72,12,6]] bivariate bicycle code; plain BP vs. Relay-BP; throughput vs. batch size; sweeping the physical error rate; comparing against the Rust reference | ~3–4 min |
+| 2 | [Multi-GPU quickstart](02_multi_gpu_quickstart.ipynb) | Process-per-GPU decoding; host inputs and outputs; syndrome and logical-error checks; scaling with 1, 2, 4, and 8 GPUs at 1,024 shots per GPU | Depends on GPU count and hardware |
\* Measured on a laptop RTX 3000 Ada (8 GB). A GPU (CUDA or ROCm) is required.
+
+Tutorial 2 uses up to eight AMD GPUs with ROCm on the same machine, with at
+least two needed for the multi-GPU examples. It adapts to the available GPU count.