From aad6e337730e2c8c8726a9b504f54effe012084b Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Wed, 12 Aug 2026 20:42:01 -0700 Subject: [PATCH 1/7] skills: consolidate EAGLE3 skills into a speculative-decoding skill The four eagle3-* skills each baked EAGLE3 into a stage of the same draft-model pipeline, so adding DFlash or DSpark would have meant four more near-duplicate skills. Collapse them into a single speculative-decoding skill shaped like ptq (SKILL.md + references/), split along two axes: references/stages/ holds the algorithm-independent procedure, references/algorithms/ holds the per-algorithm data sheet. Stage docs cite algorithm sheet sections by heading; algorithms/README.md defines the contract for adding a sheet. Also drops the assumption of a fixed 4-task pipeline, which does not hold across algorithms and variants. Signed-off-by: Ye Yu --- .claude/skills/eagle3-new-model | 1 - .claude/skills/eagle3-review-logs | 1 - .claude/skills/eagle3-triage | 1 - .claude/skills/eagle3-validate | 1 - .claude/skills/speculative-decoding | 1 + .../modelopt/skills/eagle3-new-model/SKILL.md | 46 ----- .../modelopt/skills/eagle3-triage/SKILL.md | 168 ------------------ .../modelopt/skills/eagle3-validate/SKILL.md | 119 ------------- .../skills/speculative-decoding/SKILL.md | 85 +++++++++ .../references/algorithms/README.md | 36 ++++ .../references/algorithms/eagle3.md | 105 +++++++++++ .../references/stages/configure.md | 53 ++++++ .../references/stages/review-logs.md} | 60 +++---- .../references/stages/triage.md | 102 +++++++++++ .../references/stages/validate.md | 107 +++++++++++ tools/launcher/docs/claude_code.md | 2 +- 16 files changed, 516 insertions(+), 372 deletions(-) delete mode 120000 .claude/skills/eagle3-new-model delete mode 120000 .claude/skills/eagle3-review-logs delete mode 120000 .claude/skills/eagle3-triage delete mode 120000 .claude/skills/eagle3-validate create mode 120000 .claude/skills/speculative-decoding delete mode 100644 plugins/modelopt/skills/eagle3-new-model/SKILL.md delete mode 100644 plugins/modelopt/skills/eagle3-triage/SKILL.md delete mode 100644 plugins/modelopt/skills/eagle3-validate/SKILL.md create mode 100644 plugins/modelopt/skills/speculative-decoding/SKILL.md create mode 100644 plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md create mode 100644 plugins/modelopt/skills/speculative-decoding/references/algorithms/eagle3.md create mode 100644 plugins/modelopt/skills/speculative-decoding/references/stages/configure.md rename plugins/modelopt/skills/{eagle3-review-logs/SKILL.md => speculative-decoding/references/stages/review-logs.md} (50%) create mode 100644 plugins/modelopt/skills/speculative-decoding/references/stages/triage.md create mode 100644 plugins/modelopt/skills/speculative-decoding/references/stages/validate.md diff --git a/.claude/skills/eagle3-new-model b/.claude/skills/eagle3-new-model deleted file mode 120000 index 9db34648539..00000000000 --- a/.claude/skills/eagle3-new-model +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/eagle3-new-model \ No newline at end of file diff --git a/.claude/skills/eagle3-review-logs b/.claude/skills/eagle3-review-logs deleted file mode 120000 index f4a028fdd4f..00000000000 --- a/.claude/skills/eagle3-review-logs +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/eagle3-review-logs \ No newline at end of file diff --git a/.claude/skills/eagle3-triage b/.claude/skills/eagle3-triage deleted file mode 120000 index 86f9b97821f..00000000000 --- a/.claude/skills/eagle3-triage +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/eagle3-triage \ No newline at end of file diff --git a/.claude/skills/eagle3-validate b/.claude/skills/eagle3-validate deleted file mode 120000 index 29446f36dec..00000000000 --- a/.claude/skills/eagle3-validate +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/eagle3-validate \ No newline at end of file diff --git a/.claude/skills/speculative-decoding b/.claude/skills/speculative-decoding new file mode 120000 index 00000000000..4318d4ea853 --- /dev/null +++ b/.claude/skills/speculative-decoding @@ -0,0 +1 @@ +../../.agents/skills/speculative-decoding \ No newline at end of file diff --git a/plugins/modelopt/skills/eagle3-new-model/SKILL.md b/plugins/modelopt/skills/eagle3-new-model/SKILL.md deleted file mode 100644 index 65a34bdacfc..00000000000 --- a/plugins/modelopt/skills/eagle3-new-model/SKILL.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -name: eagle3-new-model -description: > - Add a new model to the EAGLE3 offline pipeline. Generates an hf_offline_eagle3.yaml - launcher config for a new model checkpoint, choosing the right hidden state dump - backend (TRT-LLM / HF / vLLM) and GPU configuration. - Use when user wants to run EAGLE3 on a model that does not yet have a YAML in - tools/launcher/examples/ or asks how to configure the pipeline for a new checkpoint. -user_invocable: true ---- - -# EAGLE3 New Model Configuration - -Create `tools/launcher/examples///hf_offline_eagle3.yaml` by **copying the -closest existing example and adapting it**. Pick a reference with the same shape as the -target (dense vs MoE, similar size) from `tools/launcher/examples/` — e.g. the Qwen3-8B -config for a dense model. - -The pipeline is a 4-task config (`task_0` data synthesis → `task_1` hidden-state dump → -`task_2` train → `task_3` benchmark). The task structure, args, containers, and GPU/node -sizing are all visible in the existing examples — infer them from a reference rather than -hand-rolling. This file documents only the two things that are **not** obvious from the -examples: which dump backend to pick, and the model-specific gotchas. - -## Choosing the `task_1` hidden-state dump backend - -| Backend | Script | When to use | -|---------|--------|-------------| -| vLLM | `common/eagle3/dump_offline_data_vllm.sh` | **Default.** Broad coverage via vLLM's native hidden-state extractor. | -| HF | `common/eagle3/dump_offline_data_hf.sh` | VLMs / multimodal, custom-code models, sliding-window attention (TRT-LLM can't serve these). | -| TRT-LLM | `common/eagle3/dump_offline_data.sh` | Pure-text models with TRT-LLM support; pass `--tp ` and `--moe-ep `. | - -Rule of thumb: **HF** if the model is a VLM or uses sliding-window attention; **vLLM** -otherwise. TRT-LLM only when you specifically want its kernels for a supported plain-text model. - -## Model-specific adjustments - -These are the non-obvious knobs that vary per model: - -| Situation | What to change | -|---|---| -| Requires `--trust-remote-code` | Add to `task_0` vLLM args (before the `--` separator) and to `task_3` benchmark args | -| MoE with large expert hidden dim | Increase `intermediate_size` in `eagle_config.json` to match `moe_intermediate_size` | -| Custom tokenizer (e.g. tiktoken) | Set `TIKTOKEN_RS_CACHE_DIR` env var in `task_0` and `task_1` | - -After adapting the config, preview it with `--dryrun` before submitting. diff --git a/plugins/modelopt/skills/eagle3-triage/SKILL.md b/plugins/modelopt/skills/eagle3-triage/SKILL.md deleted file mode 100644 index 45d3b5607d7..00000000000 --- a/plugins/modelopt/skills/eagle3-triage/SKILL.md +++ /dev/null @@ -1,168 +0,0 @@ ---- -name: eagle3-triage -description: > - Triage a failed EAGLE3 pipeline run. Identifies which step failed (data synthesis, - hidden state dump, training, or benchmark), diagnoses root cause from logs, and - suggests fixes. Use when user reports an EAGLE3 pipeline failure or asks why a - specific step failed. Also helps debug new model support issues. -user_invocable: true ---- - -# EAGLE3 Pipeline Triage - -Diagnose failures in the 4-step EAGLE3 offline pipeline. This skill walks through -each step, identifies the failure point, and provides actionable fixes. - -## Pipeline Overview - -| Step | Script | Purpose | Common failure area | -|------|--------|---------|---------------------| -| task_0 | `common/vllm/query.sh` | Data synthesis via vLLM server | Server startup, model loading, OOM | -| task_1 | `common/eagle3/dump_offline_data_vllm.sh` (or `_hf.sh` / `.sh`) | Dump hidden states | Backend selection, OOM, unsupported arch | -| task_2 | `common/eagle3/train_eagle.sh` | Train EAGLE3 draft head | Dependencies, training crash, export | -| task_3 | `common/specdec_bench/quick_check.sh` | Benchmark acceptance rate | Engine startup, draft model loading | - -## Step 0 — Locate the experiment - -Ask the user for one of: -- Experiment directory (e.g., the `--job-dir` passed to `launch.py` or `slurm.py`) -- The model name / YAML they ran - -Find recent experiments under the job directory: - -```bash -ls -td experiments/cicd/cicd_* | head -10 -# or wherever --job-dir was pointed -``` - -Each experiment directory contains one subdirectory per task (task_0 through task_3), -each with a log file whose name varies by launch mode (Slurm: `sbatch_*.out`, local -Docker: `*.log`). - -## Step 1 — Fetch logs for the failed task - -Match the log files generally and read the tail of each — errors appear at the end: - -```bash -find experiments// -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do - echo "=== $f ==="; tail -200 "$f"; echo -done -``` - -Look for the first task with a non-zero exit code or error message. - -## Step 2 — Diagnose by step - -### task_0 failures (Data Synthesis) - -**How it works:** Launches a vLLM OpenAI-compatible server, polls `/health` until ready, -then runs `query.py` to generate synthetic prompt/response pairs. -Output goes to `/scratchspace/data/`. - -| Error pattern | Root cause | Fix | -|---|---|---| -| Server never becomes healthy (hangs at health check) | Model too large for allocated GPUs, or vLLM startup crash | Check BF16 weight size vs total allocated GPU memory; increase TP and/or nodes. | -| `CUDA out of memory` during model load | Insufficient GPU memory | Reduce `--max-model-len` or increase `--tensor-parallel-size` | -| `trust_remote_code` error | Model requires custom code but flag not set | Add `--trust-remote-code` before the `--` separator in task_0 args | -| Vocab / tokenizer error | Missing tokenizer cache (e.g., GPT-OSS-20B needs `TIKTOKEN_RS_CACHE_DIR`) | Set `TIKTOKEN_RS_CACHE_DIR` to a pre-populated cache path in the environment | -| Architecture not supported | vLLM version doesn't support this model | Try a newer vLLM container (`vllm/vllm-openai:latest`) | -| `CANCELLED ... DUE TO TIME LIMIT` | Job wall-clock limit too short | Increase Slurm `--time`. Note: `afterany` deps let task_1 still start. | -| Empty `/scratchspace/data/` | query.py ran but produced no output | Check `--data` path exists and contains prompts. Check query.py logs. | - -### task_1 failures (Hidden State Dump) - -**How it works:** Loads the target model and runs a forward pass on each conversation, -saving hidden states as `.pt` files in `/scratchspace/offline_hidden_states/`. - -Three backends are available: - -| Backend | Script | When to use | -|---------|--------|-------------| -| vLLM | `dump_offline_data_vllm.sh` | Broad model coverage; uses vLLM's native hidden-state extractor | -| HF | `dump_offline_data_hf.sh` | VLMs, custom-code models, SWA attention; uses `device_map="auto"` | -| TRT-LLM | `dump_offline_data.sh` | Pure-text models with TRT-LLM support; needs `--tp`/`--moe-ep` args | - -| Error pattern | Root cause | Fix | -|---|---|---| -| `No such file or directory: dump_offline_data_vllm.sh` | Wrong script path in YAML | Use the correct path under `common/eagle3/` | -| `FileNotFoundError: /scratchspace/data` | task_0 failed or produced no output | Re-run task_0 first, or point `--input-data` to existing data | -| `CUDA out of memory` | Model too large | Switch to `_hf.sh` (device_map="auto") or increase TP | -| `RuntimeError` / unsupported arch | Model not supported by TRT-LLM backend | Switch to `dump_offline_data_hf.sh` or `dump_offline_data_vllm.sh` | -| `NCCL timeout` / `NCCL error` | Multi-node communication failure | Retry. Reduce EP. | -| No `.pt` files in output dir | Script ran but extraction produced nothing | Check `--max-seq-len` and input data format | -| `pyxis: child terminated with signal 15` | SIGTERM — likely OOM | Increase TP or switch backends | - -### task_2 failures (Training) - -**How it works:** Installs requirements, runs `launch_train.sh` (Accelerate + FSDP) with the -config from `modelopt_recipes/general/speculative_decoding/eagle3.yaml`, then exports via -`export_hf_checkpoint.py`. Output: `/scratchspace/eagle3/` and `/scratchspace/export/`. - -| Error pattern | Root cause | Fix | -|---|---|---| -| `FileNotFoundError: /scratchspace/offline_hidden_states` | task_1 failed or produced no output | Re-run task_1 first | -| `CUDA out of memory` during training | Batch size too large | Reduce `training.train_bs` or `training.training_seq_len` | -| `KeyError` / `AttributeError` in model loading | Model architecture not recognized by EAGLE3 | Model may need code changes in modelopt for this architecture | -| Loss is NaN or diverges | LR too high or data quality issue | Reduce `training.lr`. Check hidden state data. | -| `export_hf_checkpoint.py` fails | Training produced incomplete checkpoint | Check `/scratchspace/eagle3/` for `model.safetensors` | - -### task_3 failures (Benchmark) - -**How it works:** Launches vLLM with the target + draft model, runs acceptance rate and -throughput benchmarks. Output: JSON files. - -| Error pattern | Root cause | Fix | -|---|---|---| -| `FileNotFoundError: /scratchspace/export` | task_2 failed or export step failed | Re-run task_2. Check export output. | -| `trust_remote_code` error at benchmark | Model requires it but `quick_check.sh` doesn't forward the flag | Pass `--trust-remote-code` in task_3 args | -| Server fails with draft model | Draft model config incompatible with engine | Check `eagle_config.json` and engine version | -| AR below threshold / exit code 1 | Draft model quality too low | More epochs, data, or hyperparameter tuning | -| `CUDA out of memory` | Target + draft exceeds GPU memory | Increase TP | -| vLLM EAGLE3 not supported | vLLM version too old | Use a newer vLLM container | - -## Step 3 — Check for new-model-specific issues - -If the user is adding support for a new model, also check: - -1. **Is the model a VLM?** → Use `dump_offline_data_hf.sh` (text-only path, no vision encoder invoked) -2. **Does the model use sliding window attention (SWA)?** → TRT-LLM backend won't work; use HF or vLLM -3. **Does the model need `trust_remote_code`?** → Add to task_0 args AND task_3 args -4. **Is the model MoE?** → Check `eagle_config.json` `intermediate_size` matches model's `moe_intermediate_size` -5. **Is the model architecture recognized by EAGLE3 training?** → may need code changes in `modelopt/torch/speculative/` -6. **Custom tokenizer?** → May need additional environment vars (e.g., `TIKTOKEN_RS_CACHE_DIR`) - -## Step 4 — Suggest fix and next steps - -After diagnosis, provide: - -1. **Root cause** — one-line summary -2. **Fix** — specific config change, code edit, or command to run -3. **How to re-run** — skip earlier successful steps by pointing to existing scratchspace artifacts - -To skip task_0 and task_1 and re-run from task_2: - -```bash -uv run launch.py --yaml examples///hf_offline_eagle3.yaml \ - pipeline.task_0.skip=true \ - pipeline.task_1.skip=true \ - --yes -``` - -To run only task_1 standalone (using existing task_0 data): - -```bash -uv run launch.py --yaml examples///hf_offline_eagle3.yaml \ - pipeline.task_0.skip=true \ - pipeline.task_2.skip=true \ - pipeline.task_3.skip=true \ - --yes -``` - -If the fix requires code changes in ModelOpt (e.g., supporting a new model architecture), -note that a separate PR in the modelopt repo is needed. - -## Step 5 — Record the failure pattern - -If you encounter a failure pattern not seen before, capture it in the team's internal -triage tracker — the symptom, root cause, and fix — so the next engineer debugging the -same issue benefits. diff --git a/plugins/modelopt/skills/eagle3-validate/SKILL.md b/plugins/modelopt/skills/eagle3-validate/SKILL.md deleted file mode 100644 index 08edc749aef..00000000000 --- a/plugins/modelopt/skills/eagle3-validate/SKILL.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -name: eagle3-validate -description: > - Validate that an EAGLE3 pipeline run completed successfully end-to-end. - Checks all 4 steps produced expected artifacts, verifies acceptance rate - meets threshold (>= 2.1), and produces a summary report. - Use when user wants to verify a pipeline run or check benchmark results. -user_invocable: true ---- - -# EAGLE3 Pipeline Validation - -Verify that an EAGLE3 pipeline run completed successfully and meets quality criteria. - -## Step 0 — Identify the experiment - -Find the most recent experiment directory (or ask the user for the path): - -```bash -ls -td experiments/cicd/cicd_* | head -5 -``` - -Each experiment directory has one subdirectory per task (numbered 0–3), each containing a -log file whose name varies by launch mode (Slurm: `sbatch_*.out`, local Docker: `*.log`). - -## Step 1 — Check task outcomes - -Match the log files generally and read the tail of each: - -```bash -find experiments// -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do - echo "=== $f ==="; tail -50 "$f"; echo -done -``` - -All 4 tasks must complete without error. Look for: -- `exit code: 0` or no error — success -- `DUE TO TIME LIMIT` — timeout -- `FAILED` / `signal` / exception traceback — failure - -If any task failed, suggest running `/eagle3-triage` instead. - -## Step 2 — Verify artifacts exist - -Check each step produced the expected output (artifacts live on the cluster at `/scratchspace/`). -Confirm via log messages: - -| Step | Expected log evidence | Artifact | -|------|-----------------------|----------| -| task_0 | "Saved N samples" or progress bar completing | `/scratchspace/data/*.jsonl` | -| task_1 | "Successfully processed N conversations" | `/scratchspace/offline_hidden_states/*.pt` | -| task_2 | Training loss decreasing, "export complete" | `/scratchspace/eagle3/model.safetensors`, `/scratchspace/export/` | -| task_3 | `Average Acceptance Length ... ratio: X.XX` | JSON result files | - -## Step 3 — Check acceptance rate - -In the task_3 log, find: - -```text -Average Acceptance Length {'accept': X, 'count': Y, 'ratio': Z.ZZ} -``` - -The `ratio` field is the acceptance rate (AR). - -| Criterion | Threshold | Status | -|-----------|-----------|--------| -| AR (MT-Bench) | >= 2.1 | PASS / FAIL | - -If the log shows `AR ... < lower bound`, the run already triggered a threshold failure (exit code 1). - -## Step 4 — Check training quality - -In the task_2 log look for: -- **Final training loss** — should be decreasing, not NaN -- **AR validation during training** (if `training.ar_validate_steps` was set) -- **Number of training steps** — confirms full training duration - -## Step 5 — Produce validation report - -```markdown -## EAGLE3 Pipeline Validation Report - -**Experiment:** -**Model:** -**Date:** -**Pipeline config:** - -### Step Status -| Step | Task | Status | Notes | -|------|------|--------|-------| -| 0 | Data synthesis | PASS/FAIL/TIMEOUT | N samples generated | -| 1 | Hidden state dump | PASS/FAIL | N .pt files | -| 2 | Training + export | PASS/FAIL | Final loss: X.XX | -| 3 | Benchmark | PASS/FAIL | AR: X.XX | - -### Acceptance Rate -- MT-Bench AR: X.XX (threshold: >= 2.1) — PASS/FAIL - -### Training Summary -- Final loss: X.XX -- Training steps: N -- AR during training: X.XX (if validated) - -### Overall: PASS / FAIL - -``` - -## Step 6 — Suggest next steps - -**If PASS:** -- Record the verified result (and checkpoint path) in the team's internal triage tracker -- This model is now a candidate to add as a launcher example in a dedicated PR - -**If FAIL:** -- Identify which step or metric failed -- Suggest running `/eagle3-triage` for diagnosis -- For a low AR, diagnose the specific cause from the run (training loss curve, data - volume/quality, draft-head capacity, hyperparameters) and suggest fixes targeted to that - scenario — low AR can have many causes, so avoid a generic checklist. diff --git a/plugins/modelopt/skills/speculative-decoding/SKILL.md b/plugins/modelopt/skills/speculative-decoding/SKILL.md new file mode 100644 index 00000000000..cfe9d7b9693 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/SKILL.md @@ -0,0 +1,85 @@ +--- +name: speculative-decoding +description: >- + Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, + DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants + to add a new model to a draft-training pipeline, asks why a pipeline run failed, + wants experiment logs reviewed, or wants to check whether a run's acceptance rate + meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "draft model", + "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a + model (use ptq) or serving a checkpoint (use deployment). +user_invocable: true +--- + +# Speculative Decoding Draft-Model Training + +Everything needed to take a target model from "no draft head" to "validated +acceptance rate" lives in this directory. Work through the stages below in order +for a new model; jump straight to a stage when you already know which one you need. + +## Two axes: stage and algorithm + +The pipeline is the same shape for every draft-model algorithm — synthesize data, +dump base-model hidden states, train the draft, benchmark acceptance rate. What +changes between algorithms is which training script and recipe run, which knobs +matter, and which failures are typical. + +So this skill is split along those two axes, and **you almost always read one file +from each**: + +| Axis | Directory | What it holds | +| --- | --- | --- | +| Stage | `references/stages/` | The procedure — algorithm-independent | +| Algorithm | `references/algorithms/` | The data sheet — scripts, recipe, knobs, thresholds, known failures | + +Read the stage file for *what to do*, and the algorithm sheet for *the values to +plug in*. When a stage file says "see the algorithm sheet", it means the section of +`references/algorithms/.md` with the matching heading. + +## Stages + +| Stage | Reference | Use when | +| --- | --- | --- | +| 1. Configure | `references/stages/configure.md` | Adding a model that has no pipeline YAML yet | +| 2. Review logs | `references/stages/review-logs.md` | A run finished (or died) and you want a pass/fail summary | +| 3. Triage | `references/stages/triage.md` | A task failed and you need root cause plus a fix | +| 4. Validate | `references/stages/validate.md` | All tasks passed and you need to confirm the acceptance rate gate | + +Review-logs and triage overlap by design: review-logs is the fast sweep across all +tasks, triage is the deep dive into one failing task. Start with review-logs unless +the user already knows which task broke. + +## Algorithms + +| Algorithm | Sheet | Recipe | +| --- | --- | --- | +| EAGLE3 | `references/algorithms/eagle3.md` | `modelopt_recipes/general/speculative_decoding/eagle3.yaml` | +| DFlash | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/dflash.yaml` | +| DSpark | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/dspark.yaml` | +| Domino | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/domino.yaml` | + +If the user's algorithm has no sheet yet, the stage procedures still apply — derive +the missing values from an existing launcher example for that algorithm +(`tools/launcher/examples/*/*/hf_*_.yaml`) and its recipe, then write the +sheet as you go. `references/algorithms/README.md` defines what a sheet must contain. + +## End-to-end: a new model + +1. Confirm the algorithm and find the closest existing launcher example. +2. **Configure** — write the pipeline YAML (`references/stages/configure.md`). +3. Preview with `--dryrun`, then submit: + + ```bash + cd tools/launcher + uv run launch.py --yaml examples///.yaml --yes + ``` + +4. Register the job and set up monitoring per the **monitor skill**. +5. **Review logs** when it finishes (`references/stages/review-logs.md`). +6. **Triage** anything that failed (`references/stages/triage.md`), fix, re-run only + the failed tasks onward via `pipeline.task_N.skip=true`. +7. **Validate** once all tasks pass (`references/stages/validate.md`). + +Model-support gaps that need code changes land in `modelopt/torch/speculative/` and +require a separate ModelOpt PR — the pipeline YAML alone cannot fix an unrecognized +architecture. diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md new file mode 100644 index 00000000000..a8e37f7f6b1 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md @@ -0,0 +1,36 @@ +# Algorithm sheets + +One file per draft-model algorithm. A sheet holds only what differs between +algorithms; the procedure lives in `../stages/`. `eagle3.md` is the worked example. + +Keep sheets short. If something is true for every algorithm, it belongs in the +stage file instead. + +## Required sections + +Stage files reference these by heading, so use the headings verbatim. + +| Heading | Contents | +| --- | --- | +| `## Pipeline tasks` | Table of task → script → purpose → output path. Task *count* varies per config — EAGLE3 offline is 4 tasks, DFlash offline is 2 — so describe the tasks this algorithm's examples actually use. | +| `## Recipe and training knobs` | The `modelopt_recipes/general/speculative_decoding/.yaml` path, plus the per-model overrides that usually need tuning. | +| `## Per-model adjustments` | The non-obvious knobs that vary by target model (attention type, MoE dims, tokenizer, `trust_remote_code`). | +| `## Success markers` | Per task, the log line that proves it worked, and the artifact it should leave behind. Consumed by review-logs and validate. | +| `## Quality gate` | The metric, where it appears in the log, and the pass threshold. | +| `## Known failures` | Error pattern → root cause → fix, for failures specific to this algorithm. Generic failures (OOM, NCCL, time limit) live in `../stages/triage.md`. | + +## Adding a sheet + +Source the facts from the repo rather than from memory: + +- Launcher examples: `tools/launcher/examples/*/*/hf_*_.yaml` — task layout, + scripts, container images, GPU sizing. +- Scripts: `tools/launcher/common/` — `eagle3/` and `specdec/` hold the training and + hidden-state-dump entry points. +- Recipe: `modelopt_recipes/general/speculative_decoding/.yaml` — defaults. +- Implementation: `modelopt/torch/speculative/plugins/` — `hf_.py` and + `modeling_.py`. + +Then add a row to the algorithm table in `../../SKILL.md`. + +Algorithms with launcher examples but no sheet yet: DFlash, DSpark, Domino. diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/eagle3.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/eagle3.md new file mode 100644 index 00000000000..62610504d71 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/eagle3.md @@ -0,0 +1,105 @@ +# EAGLE3 + +Draft head trained on hidden states dumped from the target model. Examples: +`tools/launcher/examples/*/*/hf_offline_eagle3.yaml` (and the `hf_online_*`, +`hf_streaming_*` variants). + +## Pipeline tasks + +The offline configuration is 4 tasks; each passes artifacts to the next through a +shared `/scratchspace`. + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/vllm/query.sh` or `common/tensorrt_llm/query.sh` | Data synthesis — serve the target model, generate prompt/response pairs | `/scratchspace/data/*.jsonl` | +| task_1 | `common/eagle3/dump_offline_data_vllm.sh` (or `_hf.sh` / `dump_offline_data.sh`) | Forward the target model, save hidden states | `/scratchspace/offline_hidden_states/*.pt` | +| task_2 | `common/eagle3/train_eagle.sh` | Train the draft head, then export | `/scratchspace/eagle3/model.safetensors`, `/scratchspace/export/` | +| task_3 | `common/specdec_bench/quick_check.sh` | Benchmark acceptance rate and throughput | JSON result files | + +### Choosing the task_1 dump backend + +| Backend | Script | When to use | +| --- | --- | --- | +| vLLM | `common/eagle3/dump_offline_data_vllm.sh` | **Default.** Broad coverage via vLLM's native hidden-state extractor. | +| HF | `common/eagle3/dump_offline_data_hf.sh` | VLMs / multimodal, custom-code models, sliding-window attention (TRT-LLM can't serve these). Uses `device_map="auto"`. | +| TRT-LLM | `common/eagle3/dump_offline_data.sh` | Pure-text models with TRT-LLM support; pass `--tp ` and `--moe-ep `. | + +Rule of thumb: **HF** if the model is a VLM or uses sliding-window attention; **vLLM** +otherwise. TRT-LLM only when you specifically want its kernels for a supported +plain-text model. + +## Recipe and training knobs + +`modelopt_recipes/general/speculative_decoding/eagle3.yaml`, passed to +`train_eagle.sh` via `--config` with dotted overrides: + +| Override | Note | +| --- | --- | +| `model.model_name_or_path` | Target checkpoint | +| `data.offline_data_path` | task_1 output directory | +| `training.output_dir` | Draft checkpoint destination | +| `training.training_seq_len` | Lower it first when training OOMs | +| `training.train_bs` | Lower it next when training OOMs | +| `training.lr` | Lower it when loss is NaN or diverging | +| `training.ar_validate_steps` | Set to run AR validation during training | + +`task_3` selects the algorithm at benchmark time with +`--speculative_algorithm EAGLE3`. + +## Per-model adjustments + +| Situation | What to change | +| --- | --- | +| Requires `--trust-remote-code` | Add to `task_0` server args (before the `--` separator) **and** to `task_3` benchmark args | +| MoE with large expert hidden dim | Increase `intermediate_size` in `eagle_config.json` to match the model's `moe_intermediate_size` | +| Custom tokenizer (e.g. tiktoken) | Set `TIKTOKEN_RS_CACHE_DIR` to a pre-populated cache path in `task_0` and `task_1` | +| VLM | Use `dump_offline_data_hf.sh` — the text-only path, no vision encoder invoked | +| Sliding-window attention | TRT-LLM backend won't work; use HF or vLLM | +| Architecture unrecognized by training | Needs code changes in `modelopt/torch/speculative/` — a separate ModelOpt PR | + +## Success markers + +| Task | Log evidence | Artifact | +| --- | --- | --- | +| task_0 | "Saved N samples", or a progress bar completing | `/scratchspace/data/*.jsonl` | +| task_1 | "Successfully processed N conversations" | `/scratchspace/offline_hidden_states/*.pt` | +| task_2 | Training loss decreasing, "export complete" | `/scratchspace/eagle3/model.safetensors`, `/scratchspace/export/` | +| task_3 | `Average Acceptance Length ... ratio: X.XX` | JSON result files | + +## Quality gate + +The `task_3` log prints: + +```text +Average Acceptance Length {'accept': X, 'count': Y, 'ratio': Z.ZZ} +``` + +The `ratio` field is the acceptance rate (AR). + +| Criterion | Threshold | +| --- | --- | +| AR (MT-Bench) | >= 2.1 | + +If the log shows `AR ... < lower bound`, the run already tripped the threshold check +and exited non-zero. + +## Known failures + +Generic infrastructure failures are in `../stages/triage.md`. These are +EAGLE3-specific: + +| Error pattern | Root cause | Fix | +| --- | --- | --- | +| `No such file or directory: dump_offline_data_vllm.sh` | Wrong script path in YAML | Use the correct path under `common/eagle3/` | +| `FileNotFoundError: /scratchspace/data` | task_0 failed or produced no output | Re-run task_0, or point `--input-data` at existing data | +| `FileNotFoundError: /scratchspace/offline_hidden_states` | task_1 failed or produced no output | Re-run task_1 | +| `FileNotFoundError: /scratchspace/export` | task_2 or its export step failed | Re-run task_2; check export output | +| `RuntimeError` / unsupported arch during dump | Model not supported by the TRT-LLM backend | Switch to `dump_offline_data_hf.sh` or `dump_offline_data_vllm.sh` | +| No `.pt` files in the dump output dir | Extraction produced nothing | Check `--max-seq-len` and the input data format | +| `KeyError` / `AttributeError` loading the model in task_2 | Architecture not recognized by EAGLE3 training | Needs code changes in `modelopt/torch/speculative/` | +| Loss is NaN or diverging | LR too high, or data quality issue | Reduce `training.lr`; check the hidden-state data | +| `export_hf_checkpoint.py` fails | Training produced an incomplete checkpoint | Check `/scratchspace/eagle3/` for `model.safetensors` | +| Empty `/scratchspace/data/` after task_0 | `query.py` ran but wrote nothing | Check `--data` path exists and contains prompts; check `query.py` logs | +| Server fails to load the draft model in task_3 | Draft config incompatible with the engine | Check `eagle_config.json` and the engine version | +| vLLM reports EAGLE3 not supported | vLLM version too old | Use a newer vLLM container | +| AR below threshold / exit code 1 | Draft quality too low | More epochs or data, or hyperparameter tuning | diff --git a/plugins/modelopt/skills/speculative-decoding/references/stages/configure.md b/plugins/modelopt/skills/speculative-decoding/references/stages/configure.md new file mode 100644 index 00000000000..59b541bc6e1 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/configure.md @@ -0,0 +1,53 @@ +# Stage 1 — Configure a new model + +Create `tools/launcher/examples///.yaml` by **copying the closest +existing example and adapting it**. Pick a reference with the same algorithm and the +same shape as the target (dense vs MoE, similar size) from `tools/launcher/examples/` +— e.g. the Qwen3-8B config for a dense model. + +The task structure, args, containers, and GPU/node sizing are all visible in the +existing examples — infer them from a reference rather than hand-rolling. This file +covers only what the examples don't make obvious. + +## Step 1 — Pick the algorithm and the variant + +Example filenames encode both: `hf__.yaml`, where mode is `offline` +(dump hidden states first, then train on them), `online` (forward the base model at +training time), or `streaming`. + +```bash +ls tools/launcher/examples/*/*/hf_*_.yaml +``` + +Offline is the default choice when the target model is too large to forward +alongside training. Task count follows from the variant, not from the algorithm — do +not assume a fixed number of tasks; copy the reference's layout. + +## Step 2 — Fill in the algorithm-specific values + +From the algorithm sheet (`../algorithms/.md`): + +- **Pipeline tasks** — which script each task runs, and the artifact paths they pass + between each other. +- **Recipe and training knobs** — the recipe path for the training task, and which + overrides this model needs. +- **Per-model adjustments** — the non-obvious knobs that vary by target model. + +For offline variants, the hidden-state dump task usually offers more than one +backend (vLLM / HF / TRT-LLM). The sheet's *Pipeline tasks* section says how to pick. + +## Step 3 — Size the job + +Copy node/GPU counts from the reference example, then sanity-check against the +target: the base model's BF16 weights must fit in the allocated GPU memory for the +serving and dump tasks, so scale `tensor-parallel-size`, `gpus_per_node`, or `nodes` +if the target is larger than the reference. + +## Step 4 — Preview + +```bash +cd tools/launcher +uv run launch.py --yaml examples///.yaml --dryrun +``` + +Check the resolved scripts, paths, and containers before submitting for real. diff --git a/plugins/modelopt/skills/eagle3-review-logs/SKILL.md b/plugins/modelopt/skills/speculative-decoding/references/stages/review-logs.md similarity index 50% rename from plugins/modelopt/skills/eagle3-review-logs/SKILL.md rename to plugins/modelopt/skills/speculative-decoding/references/stages/review-logs.md index 8c9478d4e6e..eebdfeb3a68 100644 --- a/plugins/modelopt/skills/eagle3-review-logs/SKILL.md +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/review-logs.md @@ -1,21 +1,13 @@ ---- -name: eagle3-review-logs -description: > - Review EAGLE3 pipeline experiment logs from the launcher's experiments/ directory. - Summarizes pass/fail status for all 4 tasks, diagnoses failures with root causes - and fixes, and flags warnings. Use when the user asks to review job logs, - check experiment results, or diagnose why a specific task failed. -user_invocable: true ---- +# Stage 2 — Review experiment logs -# Review EAGLE3 Experiment Logs +Analyze output logs from a pipeline run launched via `launch.py` or `slurm.py`, and +produce a pass/fail summary across all tasks. For a deep dive into one failing task, +go to `triage.md` instead. -Analyze output logs from an EAGLE3 pipeline run launched via `launch.py` or `slurm.py`. +## Step 0 — Find the experiment -## Step 0 — Find experiment logs - -Locate the experiment directory. The default is `experiments/` relative to the launcher root, -or wherever `--job-dir` was pointed. +The default job directory is `experiments/` relative to the launcher root, or +wherever `--job-dir` was pointed. ```bash ls -td experiments/cicd/cicd_* | head -10 @@ -25,9 +17,9 @@ If no experiments exist, ask the user for the directory. ## Step 1 — Read all task logs -Each experiment has one subdirectory per task (0–3). Log filenames vary by launch mode -(Slurm writes `sbatch_*.out`, local Docker writes `*.log`), so match log files generally and -read the tail of each in a single Bash call — errors surface at the end: +Each experiment has one subdirectory per task. Log filenames vary by launch mode +(Slurm writes `sbatch_*.out`, local Docker writes `*.log`), so match log files +generally and read the tail of each in a single Bash call — errors surface at the end: ```bash find experiments// -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do @@ -39,11 +31,12 @@ done For each task log, check: -- **Exit / cancellation**: `DUE TO TIME LIMIT`, `FAILED`, signal (e.g., `signal 15`) -- **Python exceptions / tracebacks**: last exception is usually the root cause +- **Exit / cancellation**: `DUE TO TIME LIMIT`, `FAILED`, signal (e.g. `signal 15`) +- **Python exceptions / tracebacks**: the last exception is usually the root cause - **CUDA errors**: OOM, NCCL timeout - **Slurm state**: COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY -- **Success indicators**: "Saved N samples", "Successfully processed N conversations", training loss line, AR output +- **Success indicators**: see *Success markers* in `../algorithms/.md` — + each task has a specific log line that proves it worked ## Step 3 — Produce report @@ -52,14 +45,15 @@ Output a structured markdown report: ### Summary - Overall status: PASSED / FAILED / MIXED / PARTIAL -- Task breakdown: e.g., task_0 TIMEOUT, task_1 FAIL, task_2 skipped, task_3 skipped +- Task breakdown: e.g. task_0 TIMEOUT, task_1 FAIL, task_2 skipped, task_3 skipped ### Task Results -For each task (0–3): +For each task: **Task N — \: PASS / FAIL / TIMEOUT** -- Key output: (e.g., "3277/3295 samples generated" or "Script not found") + +- Key output: (e.g. "3277/3295 samples generated" or "Script not found") - Error (if failed): quoted error message, max 10 lines - Root cause: one-line diagnosis - Suggested fix: actionable step @@ -70,26 +64,24 @@ Non-fatal issues worth noting (near-OOM, tokenizer warnings, slow throughput). ## Step 4 — Suggest next steps -Based on results: - -- If a task failed due to a known issue, suggest the fix and how to re-run from that task: +- If a task failed due to a known issue, suggest the fix and how to re-run from that + task: ```bash - uv run launch.py --yaml examples///hf_offline_eagle3.yaml \ + uv run launch.py --yaml examples///.yaml \ pipeline.task_0.skip=true \ --yes ``` -- If the failure pattern looks new, suggest capturing it in the team's internal triage - tracker, and use `/eagle3-triage` for a deeper diagnosis. - -- If all tasks passed, suggest running `/eagle3-validate` to confirm AR meets threshold. +- If the failure pattern looks new, suggest capturing it in the team's internal + triage tracker, and use `triage.md` for a deeper diagnosis. +- If all tasks passed, move to `validate.md` to confirm the quality gate. ## Known benign patterns (do NOT mark as failures) | Pattern | Explanation | -|---|---| +| --- | --- | | vLLM server exit code 143 | SIGTERM — server was killed after queries completed. Expected. | -| `CANCELLED AT ... DUE TO TASK FAILURE` after `exit code: 0` | Slurm cleanup of worker nodes after main task succeeded. | +| `CANCELLED AT ... DUE TO TASK FAILURE` after `exit code: 0` | Slurm cleanup of worker nodes after the main task succeeded. | | `destroy_process_group() was not called` | Benign PyTorch shutdown warning. | | `tokenizer class ... not equal to the registered tokenizer class` | Harmless tokenizer mismatch warning. | diff --git a/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md b/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md new file mode 100644 index 00000000000..37b5cf39c45 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md @@ -0,0 +1,102 @@ +# Stage 3 — Triage a failed run + +Diagnose a failure in the draft-training pipeline: identify the failing task, find +the root cause, and give a fix plus a re-run command. + +## Step 0 — Locate the experiment + +Ask the user for one of: + +- The experiment directory (e.g. the `--job-dir` passed to `launch.py` / `slurm.py`) +- The model name / YAML they ran + +Find recent experiments under the job directory: + +```bash +ls -td experiments/cicd/cicd_* | head -10 +# or wherever --job-dir was pointed +``` + +Each experiment directory contains one subdirectory per task, each with a log file +whose name varies by launch mode (Slurm: `sbatch_*.out`, local Docker: `*.log`). + +## Step 1 — Fetch logs for the failed task + +Match the log files generally and read the tail of each — errors appear at the end: + +```bash +find experiments// -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do + echo "=== $f ==="; tail -200 "$f"; echo +done +``` + +Find the first task with a non-zero exit code or an error message. Later tasks +usually fail only because an upstream artifact is missing, so fix the first one. + +## Step 2 — Diagnose + +Work through two tables. Start here — these failures are independent of the +algorithm and account for most runs: + +| Error pattern | Root cause | Fix | +| --- | --- | --- | +| Server never becomes healthy (hangs at the health check) | Model too large for the allocated GPUs, or a server startup crash | Compare BF16 weight size against total allocated GPU memory; increase TP and/or nodes | +| `CUDA out of memory` during model load | Insufficient GPU memory | Reduce `--max-model-len`, or increase `--tensor-parallel-size` | +| `CUDA out of memory` during the hidden-state dump | Model too large for the chosen backend | Switch to a `device_map="auto"` backend, or increase TP | +| `CUDA out of memory` during training | Batch or sequence length too large | Reduce the recipe's training batch size or sequence length (see the algorithm sheet's *Recipe and training knobs*) | +| `CUDA out of memory` at benchmark | Target plus draft exceeds GPU memory | Increase TP | +| `pyxis: child terminated with signal 15` | SIGTERM — usually OOM | Increase TP or switch backends | +| `NCCL timeout` / `NCCL error` | Multi-node communication failure | Retry; reduce EP | +| `CANCELLED ... DUE TO TIME LIMIT` | Slurm wall-clock limit too short | Increase `--time`. Note that `afterany` dependencies let the next task start anyway. | +| `trust_remote_code` error | Model needs custom code but the flag isn't set | Add the flag to the serving task args (before the `--` separator) **and** to the benchmark task args | +| Vocab / tokenizer error | Missing tokenizer cache (e.g. a tiktoken cache) | Point the relevant cache env var at a pre-populated path | +| Architecture not supported by the serving engine | Engine version too old for this model | Try a newer container image | + +Then check *Known failures* in `../algorithms/.md` for failures specific +to this algorithm — wrong script paths, missing scratchspace artifacts, export +failures, draft-config incompatibilities. + +## Step 3 — Check for new-model issues + +If the user is adding support for a new model, re-read *Per-model adjustments* in +`../algorithms/.md` and confirm each applicable knob was set — attention +type, MoE dimensions, custom tokenizer, and `trust_remote_code` are the usual +offenders. + +If the architecture isn't recognized by the training code at all, that needs changes +in `modelopt/torch/speculative/` and a separate ModelOpt PR — no YAML change fixes it. + +## Step 4 — Suggest fix and next steps + +Provide: + +1. **Root cause** — one-line summary +2. **Fix** — the specific config change, code edit, or command +3. **How to re-run** — skip earlier successful tasks by pointing at the existing + scratchspace artifacts + +To skip the first two tasks and re-run from the third: + +```bash +uv run launch.py --yaml examples///.yaml \ + pipeline.task_0.skip=true \ + pipeline.task_1.skip=true \ + --yes +``` + +To run a single task standalone, skip every other one: + +```bash +uv run launch.py --yaml examples///.yaml \ + pipeline.task_0.skip=true \ + pipeline.task_2.skip=true \ + pipeline.task_3.skip=true \ + --yes +``` + +## Step 5 — Record the failure pattern + +If you hit a failure pattern not seen before, capture it in the team's internal +triage tracker — symptom, root cause, and fix — so the next engineer benefits. If +it's algorithm-specific, add a row to *Known failures* in the algorithm sheet; if it +applies to every algorithm, add it to Step 2 above. diff --git a/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md b/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md new file mode 100644 index 00000000000..783a790550c --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md @@ -0,0 +1,107 @@ +# Stage 4 — Validate a completed run + +Verify that a pipeline run completed successfully end-to-end and meets its quality +gate. + +## Step 0 — Identify the experiment + +Find the most recent experiment directory (or ask the user for the path): + +```bash +ls -td experiments/cicd/cicd_* | head -5 +``` + +Each experiment directory has one subdirectory per task, each containing a log file +whose name varies by launch mode (Slurm: `sbatch_*.out`, local Docker: `*.log`). + +## Step 1 — Check task outcomes + +Match the log files generally and read the tail of each: + +```bash +find experiments// -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do + echo "=== $f ==="; tail -50 "$f"; echo +done +``` + +Every task must complete without error. Look for: + +- `exit code: 0` or no error — success +- `DUE TO TIME LIMIT` — timeout +- `FAILED` / `signal` / exception traceback — failure + +If any task failed, go to `triage.md` instead. + +## Step 2 — Verify artifacts exist + +Check each task produced its expected output. Artifacts live on the cluster under +`/scratchspace/`, so confirm via log messages. The per-task log evidence and artifact +paths are in *Success markers* in `../algorithms/.md`. + +## Step 3 — Check the quality gate + +*Quality gate* in `../algorithms/.md` gives the metric, the log line it +appears on, and the pass threshold. Extract the value from the benchmark task's log +and compare. + +If the log already reports the metric below its lower bound, the run tripped the +threshold check itself and exited non-zero. + +## Step 4 — Check training quality + +In the training task's log look for: + +- **Final training loss** — should be decreasing, not NaN +- **Metric validation during training** — if the recipe enabled periodic validation +- **Number of training steps** — confirms full training duration + +## Step 5 — Produce validation report + +```markdown +## Speculative Decoding Pipeline Validation Report + +**Experiment:** +**Model:** +**Algorithm:** +**Date:** +**Pipeline config:** + +### Task Status +| Task | Name | Status | Notes | +|------|------|--------|-------| +| 0 | Data synthesis | PASS/FAIL/TIMEOUT | N samples generated | +| 1 | Hidden state dump | PASS/FAIL | N .pt files | +| 2 | Training + export | PASS/FAIL | Final loss: X.XX | +| 3 | Benchmark | PASS/FAIL | AR: X.XX | + +### Quality Gate +- : X.XX (threshold: ) — PASS/FAIL + +### Training Summary +- Final loss: X.XX +- Training steps: N +- Metric during training: X.XX (if validated) + +### Overall: PASS / FAIL + +``` + +Adjust the task rows to the tasks this config actually ran — task count varies by +algorithm and variant. + +## Step 6 — Suggest next steps + +**If PASS:** + +- Record the verified result (and checkpoint path) in the team's internal triage + tracker +- This model is now a candidate to add as a launcher example in a dedicated PR + +**If FAIL:** + +- Identify which task or metric failed +- Go to `triage.md` for diagnosis +- For a low acceptance rate, diagnose the specific cause from the run (training loss + curve, data volume/quality, draft capacity, hyperparameters) and suggest fixes + targeted to that scenario — a low rate can have many causes, so avoid a generic + checklist. diff --git a/tools/launcher/docs/claude_code.md b/tools/launcher/docs/claude_code.md index 18511b6d697..9d581401e6d 100644 --- a/tools/launcher/docs/claude_code.md +++ b/tools/launcher/docs/claude_code.md @@ -80,7 +80,7 @@ Available skills: |---|---|---| | `/review-logs` | After job completion/failure | Analyze logs, diagnose failures, JUnit XML | | `/wait-for-jobs` | After detached submission | Poll experiment status | -| `/eagle3-new-model` | Adding a new EAGLE3 model | Generate pipeline YAML | +| `/speculative-decoding` | Draft-model pipeline work (EAGLE3, DFlash, …) | Configure a new model, review logs, triage, validate | ## CI Integration From 5aeed10f923a3256c5195250e1d5f7b5335e08f3 Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Mon, 17 Aug 2026 03:38:25 -0700 Subject: [PATCH 2/7] skills: add DFlash algorithm sheet Second sheet under the stage/algorithm split, sourced from the launcher examples, dflash.yaml, dflash_online_training.sh, and hf_dflash.py. DFlash exercises parts of the contract EAGLE3 alone did not: three variants with different task counts (online 3, offline 2, streaming 3), a shared hidden-state dump driven by --aux-layers, and a three-part quality gate (training regression thresholds, smoke-test acceptance length, benchmark Average_AL) rather than a single acceptance rate. Also notes that dflash.yaml's chat_templates/ comment points at a directory that does not exist; templates live per-model beside each example YAML. Signed-off-by: Ye Yu --- .../skills/speculative-decoding/SKILL.md | 2 +- .../references/algorithms/README.md | 5 +- .../references/algorithms/dflash.md | 142 ++++++++++++++++++ 3 files changed, 146 insertions(+), 3 deletions(-) create mode 100644 plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md diff --git a/plugins/modelopt/skills/speculative-decoding/SKILL.md b/plugins/modelopt/skills/speculative-decoding/SKILL.md index cfe9d7b9693..479615306cd 100644 --- a/plugins/modelopt/skills/speculative-decoding/SKILL.md +++ b/plugins/modelopt/skills/speculative-decoding/SKILL.md @@ -54,7 +54,7 @@ the user already knows which task broke. | Algorithm | Sheet | Recipe | | --- | --- | --- | | EAGLE3 | `references/algorithms/eagle3.md` | `modelopt_recipes/general/speculative_decoding/eagle3.yaml` | -| DFlash | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/dflash.yaml` | +| DFlash | `references/algorithms/dflash.md` | `modelopt_recipes/general/speculative_decoding/dflash.yaml` | | DSpark | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/dspark.yaml` | | Domino | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/domino.yaml` | diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md index a8e37f7f6b1..d9d714e6281 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md @@ -1,7 +1,8 @@ # Algorithm sheets One file per draft-model algorithm. A sheet holds only what differs between -algorithms; the procedure lives in `../stages/`. `eagle3.md` is the worked example. +algorithms; the procedure lives in `../stages/`. `eagle3.md` and `dflash.md` are the +worked examples — read one before writing a new sheet. Keep sheets short. If something is true for every algorithm, it belongs in the stage file instead. @@ -33,4 +34,4 @@ Source the facts from the repo rather than from memory: Then add a row to the algorithm table in `../../SKILL.md`. -Algorithms with launcher examples but no sheet yet: DFlash, DSpark, Domino. +Algorithms with launcher examples but no sheet yet: DSpark, Domino. diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md new file mode 100644 index 00000000000..6150d4bc33a --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md @@ -0,0 +1,142 @@ +# DFlash + +Block-diffusion draft: predicts a whole block of `block_size` tokens in one forward +pass instead of autoregressively. Design details, results, and open items are in +`examples/speculative_decoding/doc/dflash.md`; the paper is arXiv:2602.06036. + +Examples: `tools/launcher/examples/*/*/hf_online_dflash.yaml`, +`hf_offline_dflash.yaml`, `hf_streaming_dflash_multi_node.yaml`, +`specdec_bench_dflash_vllm.yaml`. + +## Pipeline tasks + +Task count depends on the variant — DFlash has three, and none of them is the 4-task +shape EAGLE3 offline uses. + +**Online** (`hf_online_dflash.yaml`) — base model forwards during training: + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/specdec/dflash_online_training.sh` | Train the draft, then export | `/checkpoint-*`, `/exported-checkpoint-*` | +| task_1 | `common/specdec/vllm_smoke_test.sh` | Serve target + draft, verify responses | Smoke-test log | +| task_2 | `common/specdec/ar_eval_mtbench.sh` | MT-Bench per-category AR evaluation (1 GPU) | AR per category | + +**Offline** (`hf_offline_dflash.yaml`) — for base models too large to forward +alongside training: + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/eagle3/dump_offline_data_vllm.sh` | Dump base hidden states via vLLM (TP-sharded) | Hidden-state dump directory | +| task_1 | `common/specdec/dflash_online_training.sh` | Train on the dump, then export | `/exported-checkpoint-*` | + +The dump script is shared with EAGLE3 — DFlash selects its own captured layers with +`--aux-layers dflash`. Two flags matter for the dump: + +- `--aux-layers dflash` — **must match the draft's `num_hidden_layers`** (recipe + default 5). `build_target_layer_ids(num_target_layers, num_draft_layers)` picks the + captured layers, so a mismatch here silently produces the wrong hidden states. +- `--answer-only-loss` and `--chat-template` — must agree with the training task's + `training.answer_only_loss` and `data.chat_template`. + +Offline training additionally needs `data.mode=offline`, +`model.use_fake_base_for_offline=true` (loads only `lm_head` + `embed_tokens` rather +than the full base), and `data.offline_data_path` pointing at the dump. + +**Streaming** (`hf_streaming_dflash_multi_node.yaml`) — same NIXL RDMA transport as +streaming EAGLE3, splitting nodes into serve replicas plus DDP trainers. See +`common/eagle3/train_eagle_streaming.sh` for dispatch and sharding. + +**Benchmark** (`specdec_bench_dflash_vllm.yaml`) — `common/specdec_bench/run.sh` with +`--speculative_algorithm DFLASH` and `--block_size`. + +## Recipe and training knobs + +`modelopt_recipes/general/speculative_decoding/dflash.yaml`, passed to +`dflash_online_training.sh` via `--config` with OmegaConf dotted overrides. Full table +in `examples/speculative_decoding/README.md#dflash-block-diffusion-for-speculative-decoding`. + +| Override | Default | Note | +| --- | --- | --- | +| `dflash.dflash_block_size` | 8 | Tokens predicted per block. `training.training_seq_len` **must** be divisible by it. | +| `dflash.dflash_num_anchors` | 512 | Random anchor positions sampled per sequence | +| `dflash.dflash_loss_decay_factor` | 4.0 | Exponential decay gamma; 0 disables | +| `dflash.dflash_self_logit_distillation` | true | Logit distillation from the target | +| `dflash.dflash_architecture_config.num_hidden_layers` | 5 | Draft decoder layers — keep in sync with `--aux-layers` | +| `dflash.dflash_mask_token_id` | auto | See *Per-model adjustments* | +| `dflash.dflash_swa_window_size` | unset | Sliding-window attention for the draft; must be >= `dflash_block_size` | +| `dflash.dflash_export_rope_scaling` | `{}` | YaRN config injected at export so a short-window draft can serve long context | +| `training.learning_rate` | 6.0e-4 | | +| `training.training_seq_len` | 4096 | | +| `data.chat_template` | — | Required when `answer_only_loss=true` | + +Export is automatic: after training, rank 0 exports every `checkpoint-` to +`exported-checkpoint-`, plus `exported-checkpoint-final` when +`modelopt_state.pth` sits directly in `output_dir`. + +## Per-model adjustments + +| Situation | What to change | +| --- | --- | +| Any model | Pin `dflash.dflash_mask_token_id` to a token that **already exists in the target's embedding** — the draft reuses the target's `embed_tokens`. Unset falls back to `tokenizer.mask_token_id`, which many tokenizers lack. MiniMax-M2.7 uses a reserved row (200054); Qwen3-8B uses 151669. | +| `answer_only_loss=true` (recipe default) | The chat template must contain `{% generation %}` / `{% endgeneration %}` tags. Most stock templates don't — supply one via `data.chat_template=.jinja`. Each model keeps its own next to its example YAML (`examples///chat_template_train.jinja`); copy the closest one. Note `dflash.yaml`'s comment points at a `chat_templates/` recipe directory that does not exist — ignore it. | +| `trust_remote_code` MoE with an older transformers pin | Set `OVERRIDE_TRANSFORMERS` in the task environment (MiniMax-M2.7 needs 4.57.1). Set `ACCELERATE_CONFIG` when the model needs FSDP2 via accelerate config rather than transformers-native `ParallelismConfig`. | +| Very large MoE base | Use the offline variant with `model.use_fake_base_for_offline=true`; plain DDP suffices, so no FSDP2 patches. Set `MIXED_PRECISION: "no"` with `training.bf16=false` if the model requires it. | +| Draft trained at short context, served long | Set `dflash.dflash_export_rope_scaling` (YaRN); factor = target context / `training_seq_len`. | +| Multi-node | Set `NUM_NODES` in the environment; `HEAD_NODE_IP` is auto-detected from Slurm. | + +## Success markers + +| Task | Log evidence | Artifact | +| --- | --- | --- | +| Hidden-state dump (offline) | vLLM extraction completes over the input data | Dump directory populated | +| Training | `Training time: N seconds`, then `=== Exporting: → ===` and `=== Regression Check (...) ===` | `/checkpoint-*/trainer_state.json`, `/exported-checkpoint-*` | +| Smoke test | `Auto-detected draft model: ...`, `Server ready after Ns` | Smoke-test log with responses | +| AR eval | Per-category MT-Bench AR output | AR results | +| Benchmark | `Average_AL` in the saved results | JSON under `--save_dir` | + +## Quality gate + +DFlash gates in three places rather than on one acceptance-rate number. + +**1. Training regression** — `common/check_regression.py` reads the latest +`trainer_state.json` and compares against env thresholds set in the YAML: + +| Env var | Meaning | +| --- | --- | +| `MAX_FINAL_LOSS` | Final loss must be below this | +| `MIN_FINAL_ACC` | Final accuracy must be above this (any log key containing `acc`) | + +Qwen3-8B online reference uses `MAX_FINAL_LOSS=5.0`, `MIN_FINAL_ACC=0.15`. Its +convergence baseline (8×B200, bs=1, seq_len=4096, 5-layer draft, block_size=16, 100K +samples, 1 epoch ≈ 12,500 steps) is in the YAML header — compare against it when +judging whether a run under-trained. + +Note: `check_regression.py` is invoked with `|| true`, and it only warns when no +`trainer_state.json` exists. A green Slurm exit is not proof the gate ran — confirm +the `=== Regression Check ===` block is present in the log. + +**2. Smoke test** — `MIN_ACCEPTANCE_LENGTH` env var (Qwen3-8B online uses 1.4) with +`NUM_SPEC_TOKENS` speculative tokens. + +**3. Benchmark** — `Average_AL` (average acceptance length) from +`common/specdec_bench/run.sh`. Acceptance length is concurrency-independent, so it is +the primary metric even when the run trades timing fidelity for wall clock. + +## Known failures + +Generic infrastructure failures are in `../stages/triage.md`. These are +DFlash-specific: + +| Error pattern | Root cause | Fix | +| --- | --- | --- | +| `seq_len (N) must be divisible by block_size (B)` | `training.training_seq_len` not a multiple of `dflash_block_size` | Adjust either value, or pad | +| `DFlash offline model cannot run eval/inference forward` | Offline conversion deletes base-model layers to save memory | Don't run eval on the offline model; reload the full base first | +| `DFlash offline model cannot run AR validation / pseudo_speculative_generate` | Same cause, hit via AR validation | Keep `training.estimate_ar=false` and `training.ar_validate_steps=0` in offline runs | +| `dflash_swa_window_size (N) must be >= dflash_block_size (B)` | Config validation | Raise the window or lower the block size | +| `The base model did not return hidden states required for DFlash training` | Base model's top-level forward ignores `output_hidden_states=True` | Usually a multimodal wrapper — needs a model-side fix | +| `ERROR: DRAFT_CKPT_DIR=... contains no exported-checkpoint-* directory` | Upstream training produced no draft | Fix training; do not chase the smoke test | +| vLLM rejects the speculative config / no DFlash method | DFlash landed in vLLM v0.22.0 (`vllm/v1/spec_decode/dflash.py`) | Use `vllm/vllm-openai:v0.22.1` or newer | +| Draft quality plateaus despite clean training | `--aux-layers` count and `num_hidden_layers` disagree, so the dump captured the wrong layers | Re-dump with matching values | +| Loss stalls high with `answer_only_loss=true` | Chat template lacks `{% generation %}` tags, so no positions contribute loss | Supply a template with generation tags | +| `dflash_dpace_alpha must be in (0, 1]` | Invalid D-PACE alpha | Correct the value | +| Qwen3-VL mRoPE / `mm_token_type_ids` errors | Qwen3-VL DFlash needs Transformers 5.3.0 or >=5.4.0 and the AutoProcessor's `mm_token_type_ids` | Match the version; don't drop processor outputs | From 12d8c61b312f00a0eeb294de626e894f72830b8a Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Mon, 17 Aug 2026 03:43:37 -0700 Subject: [PATCH 3/7] skills: add DSpark and Domino algorithm sheets Both are DFlash variants rather than separate pipelines: same recipe_type (speculative_dflash), same training script, same dflash.* config namespace, selected by dflash_architecture_config.projector_type. Their sheets document only the delta and defer to dflash.md for the shared pipeline, dump flags, and failure modes; algorithms/README.md now describes that variant shape as a supported sheet type. Both recipes deliberately pin estimate_ar=false / ar_validate_steps=0 because eval runs the DFlash backbone with the new head bypassed, so in-training acceptance rate is backbone-only and must not be read as a quality signal. Each sheet says so under its quality gate. Every recipe in modelopt_recipes/general/speculative_decoding/ now has a sheet. Signed-off-by: Ye Yu --- .../skills/speculative-decoding/SKILL.md | 17 ++- .../references/algorithms/README.md | 8 +- .../references/algorithms/domino.md | 104 ++++++++++++++++ .../references/algorithms/dspark.md | 117 ++++++++++++++++++ 4 files changed, 240 insertions(+), 6 deletions(-) create mode 100644 plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md create mode 100644 plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md diff --git a/plugins/modelopt/skills/speculative-decoding/SKILL.md b/plugins/modelopt/skills/speculative-decoding/SKILL.md index 479615306cd..456e626670d 100644 --- a/plugins/modelopt/skills/speculative-decoding/SKILL.md +++ b/plugins/modelopt/skills/speculative-decoding/SKILL.md @@ -51,12 +51,19 @@ the user already knows which task broke. ## Algorithms -| Algorithm | Sheet | Recipe | +All recipes live in `modelopt_recipes/general/speculative_decoding/.yaml`. + +| Algorithm | Sheet | Family | | --- | --- | --- | -| EAGLE3 | `references/algorithms/eagle3.md` | `modelopt_recipes/general/speculative_decoding/eagle3.yaml` | -| DFlash | `references/algorithms/dflash.md` | `modelopt_recipes/general/speculative_decoding/dflash.yaml` | -| DSpark | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/dspark.yaml` | -| Domino | not yet written — see `references/algorithms/README.md` | `modelopt_recipes/general/speculative_decoding/domino.yaml` | +| EAGLE3 | `references/algorithms/eagle3.md` | Autoregressive draft head | +| DFlash | `references/algorithms/dflash.md` | Block diffusion | +| DSpark | `references/algorithms/dspark.md` | DFlash backbone + Markov head + confidence head | +| Domino | `references/algorithms/domino.md` | DFlash backbone + GRU causal correction head | + +DSpark and Domino are **DFlash variants**, not separate pipelines: same +`recipe_type: speculative_dflash`, same training script, same `dflash.*` config +namespace, selected by `dflash_architecture_config.projector_type`. Read +`references/algorithms/dflash.md` first, then the variant's sheet for the delta. If the user's algorithm has no sheet yet, the stage procedures still apply — derive the missing values from an existing launcher example for that algorithm diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md index d9d714e6281..09678fb7995 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/README.md @@ -4,6 +4,11 @@ One file per draft-model algorithm. A sheet holds only what differs between algorithms; the procedure lives in `../stages/`. `eagle3.md` and `dflash.md` are the worked examples — read one before writing a new sheet. +When an algorithm is a **variant** of another (DSpark and Domino are both DFlash +backbones with a different head), its sheet documents only the delta and points at the +parent sheet. Don't restate the parent's pipeline, dump flags, or shared failures. +`dspark.md` and `domino.md` are the worked examples for that shape. + Keep sheets short. If something is true for every algorithm, it belongs in the stage file instead. @@ -34,4 +39,5 @@ Source the facts from the repo rather than from memory: Then add a row to the algorithm table in `../../SKILL.md`. -Algorithms with launcher examples but no sheet yet: DSpark, Domino. +Every algorithm with a recipe in +`modelopt_recipes/general/speculative_decoding/` currently has a sheet. diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md new file mode 100644 index 00000000000..4e311283d84 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md @@ -0,0 +1,104 @@ +# Domino + +**A DFlash variant, not a separate pipeline.** Domino is the DFlash draft backbone +plus a lightweight causal correction head — a GRU over the block's previously decoded +tokens producing a logit correction on the block suffix — selected with +`dflash_architecture_config.projector_type=domino`. It trains with a base/final dual +loss whose `lambda_base` weight decays from 1 to 0 over training (curriculum). + +Read `dflash.md` first — the pipeline, dump flags, export behaviour, and generic +failure modes are all shared. This sheet covers only the delta. + +Recipe: `modelopt_recipes/general/speculative_decoding/domino.yaml` (its +`metadata.recipe_type` is `speculative_dflash`, and every knob lives in the `dflash.*` +namespace). + +Example: `tools/launcher/examples/Qwen/Qwen3-8B/hf_online_domino.yaml`. Reference: +SpecForge PR #571 (z-lab); drafter format `huggingface.co/Huang2020/Qwen3-8B-Domino-b16`. + +## Pipeline tasks + +The committed example is **online**, 2 tasks: + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/eagle3/make_dataset.sh` | Build training conversations (Daring-Anteater multi-turn SFT, 50K, `--full-conversations`) | `/scratchspace/data/train.jsonl` | +| task_1 | `common/specdec/dflash_online_training.sh` | Train the draft, then export | `/exported-checkpoint-*` | + +`--full-conversations` matters: it keeps real assistant completions so +`answer_only_loss` has assistant spans to mask. + +**The inference side is intentionally not wired up yet.** The Domino correction head +is not applied in `pseudo_speculative_generate` or in the serving stack, so the +example ships no vLLM smoke test and no MT-Bench AR eval. When that path lands, add +the two steps from `hf_online_dflash.yaml` (task_1 / task_2). Do not treat their +absence as a broken config. + +## Recipe and training knobs + +Everything in `dflash.md` applies. Domino adds: + +| Override | Recipe default | Note | +| --- | --- | --- | +| `dflash.dflash_architecture_config.projector_type` | `domino` | Selects the variant | +| `dflash.dflash_architecture_config.emb_dim` | 256 | GRU head embedding dim. **Required** | +| `dflash.dflash_architecture_config.gru_hidden_dim` | 1024 | GRU hidden dim. **Required** | +| `dflash.dflash_architecture_config.pure_draft_prefix_len` | 1 | Positions at block start kept as base logits only (no causal correction). Must be in `[0, block_size-1]` | +| `dflash.dflash_architecture_config.shift_label` | true | Next-token alignment — **only `true` is supported** | +| `dflash.dflash_lambda_base_start` | 1.0 | Curriculum start weight on the base loss | +| `dflash.dflash_lambda_base_decay_ratio` | 1.0 | Fraction of training over which `lambda_base` decays to 0 | + +`dflash_self_logit_distillation` is **false** — Domino trains its own base/final CE +losses rather than distilling target logits. Recipe defaults also differ from +DFlash's: `block_size` 16, `num_anchors` 256, `num_train_epochs` 6, +`training_seq_len` 3072, `warmup_ratio` 0.04, `max_grad_norm` 1.0. + +`ddp_find_unused_parameters: true` is **required**, not incidental: while +`lambda_base == 1` the head params are absent from the backward graph and DDP would +otherwise fail. + +## Per-model adjustments + +Everything in `dflash.md`'s table applies. Additionally: + +| Situation | What to change | +| --- | --- | +| Any model | **The Domino draft does not inherit the base model's GQA/FFN dims** — a fresh `Qwen3Config` already carries defaults, so `modify()`'s inherit-if-missing guard is a no-op. Set `num_attention_heads`, `num_key_value_heads`, `head_dim`, and `intermediate_size` explicitly. The Qwen3-8B reference drafter uses `32 / 8 / 128 / 12288`. | +| Any run | **Set `training.max_steps`.** The `lambda_base` curriculum is scheduled against `state.max_steps`; if it's unset the decay window collapses to one step and the curriculum is disabled (`lambda_base` 0 from the start). This warns rather than errors. The Qwen3-8B example sets `max_steps=2000`. | + +## Success markers + +Same as `dflash.md`: `Training time: N seconds`, then the `=== Exporting: ... ===` +and `=== Regression Check (...) ===` blocks, with `exported-checkpoint-*` on disk. + +Because there is no smoke test or AR eval step, training completion plus a clean +export is the whole in-pipeline signal. + +## Quality gate + +**Do not trust in-training AR for Domino.** The recipe pins `estimate_ar: false` and +`ar_validate_steps: 0` deliberately: eval delegates to the DFlash backbone with the +correction head not applied, so reported acceptance rates are backbone-only. The code +logs this once as a warning — treat that warning as expected, not as a defect. + +The training-regression gate from `dflash.md` applies and the Qwen3-8B example sets +it: `MAX_FINAL_LOSS=5.0`, `MIN_FINAL_ACC=0.15`, checked by `check_regression.py` +against `trainer_state.json`. Since no inference metric is produced, this is currently +the only automatic gate — and per `dflash.md` it is invoked with `|| true`, so confirm +the `=== Regression Check ===` block actually appears in the log. + +## Known failures + +Generic infrastructure failures are in `../stages/triage.md`; shared block-diffusion +failures (`seq_len` divisibility, offline eval, mask token, chat template) are in +`dflash.md`. Domino-specific: + +| Error pattern | Root cause | Fix | +| --- | --- | --- | +| `Domino (projector_type='domino') requires ['emb_dim', 'gru_hidden_dim'] in dflash_architecture_config` | GRU head dims missing | Set both in `dflash_architecture_config` | +| `Domino currently supports shift_label=True (next-token alignment) only` | `shift_label=false` | Leave it at `true` | +| `pure_draft_prefix_len must be in [0, N] (block_size=B), got X` | Prefix length >= block size | Lower it below `block_size` | +| `DominoLambdaCallback: state.max_steps unset (<=0); lambda_base curriculum disabled` (warning) | `training.max_steps` not set | Set `training.max_steps`, else the curriculum never runs | +| `Domino eval uses the DFlash backbone only ...` (warning) | Correction head not applied at eval | Expected — do not chase it; evaluate after export once the inference path lands | +| DDP error about unused parameters | `ddp_find_unused_parameters` turned off | Keep it `true` | +| Draft trains but quality is poor | Draft dims left at `Qwen3Config` defaults instead of matching the base | Set the GQA/FFN dims explicitly | diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md new file mode 100644 index 00000000000..005cec2ae33 --- /dev/null +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md @@ -0,0 +1,117 @@ +# DSpark + +**A DFlash variant, not a separate pipeline.** DSpark is the DFlash draft backbone +plus a lightweight sequential (Markov) head and an optional confidence head, selected +with `dflash_architecture_config.projector_type=dspark`. The Markov head adds a +prefix-dependent transition bias to the base logits, inducing a causal block +distribution (semi-autoregressive generation). + +Read `dflash.md` first — the pipeline, dump flags, export behaviour, and generic +failure modes are all shared. This sheet covers only the delta. + +Recipe: `modelopt_recipes/general/speculative_decoding/dspark.yaml` (its +`metadata.recipe_type` is `speculative_dflash`, and every knob lives in the `dflash.*` +namespace). + +Examples: `tools/launcher/examples/moonshotai/Kimi-K2.6/hf_streaming_dspark_multi_node.yaml`, +`tools/launcher/examples/MiniMaxAI/MiniMax-M3/hf_streaming_dspark_multi_node.yaml`. + +## Pipeline tasks + +Both committed examples are **streaming, multi-node** — 2 tasks: + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/eagle3/make_dataset.sh` | Build training conversations | `/scratchspace/data/train.jsonl` | +| task_1 | `common/eagle3/train_eagle_streaming.sh` | Streaming train (serve replicas + DDP trainers over NIXL RDMA), then export | `/scratchspace/export` | + +`data.mode=streaming` with `model.use_fake_base_for_offline=true`. There is no +committed online or offline DSpark example; the recipe defaults to `data.mode=online`, +so an online run would use `common/specdec/dflash_online_training.sh` exactly as +DFlash does. + +Streaming-specific environment (set in `task_1`, see +`common/eagle3/train_eagle_streaming.sh` for dispatch and rendezvous): + +| Env var | Meaning | +| --- | --- | +| `EAGLE_CAPTURE_IDS` | Which base layers the serve side captures — the draft's target layer ids **+1**, plus the true final hidden layer | +| `SERVE_NODES` / `SERVE_TP` | How the node pool splits into serve replicas vs DDP trainers | +| `STREAMING_NUM_WORKERS` | Trainer-side streaming workers | +| `SERVE_MAX_MODEL_LEN`, `SERVE_MAX_NUM_SEQS`, `SERVE_GPU_MEM_UTIL`, `SERVE_READY_TIMEOUT` | Serve-replica limits | +| `EXPORT_EXTRA_ARGS` | Extra args at export (e.g. `--trust_remote_code`) | + +No inference path is wired into these examples — neither ships a vLLM smoke test or +AR eval step. + +## Recipe and training knobs + +Everything in `dflash.md` applies. DSpark adds: + +| Override | Recipe default | Note | +| --- | --- | --- | +| `dflash.dflash_architecture_config.projector_type` | `dspark` | Selects the variant | +| `dflash.dflash_architecture_config.markov_rank` | 256 | Markov head low-rank dimension. **Required** and must be > 0 | +| `dflash.dflash_architecture_config.markov_head_type` | `vanilla` | `vanilla` (memoryless), `gated` (hidden-gated), or `rnn` (recurrent, closest to Domino's GRU) | +| `dflash.dflash_architecture_config.use_confidence_head` | true | Builds the per-position acceptance predictor | +| `dflash.dflash_ce_loss_alpha` | 0.1 | Cross-entropy term | +| `dflash.dflash_l1_loss_alpha` | 0.9 | TVD term — the DeepSpec defaults are L1/TVD-dominant | +| `dflash.dflash_confidence_head_alpha` | 1.0 | Confidence BCE term; requires `use_confidence_head=true` when > 0 | + +Total loss is `ce_alpha*CE + l1_alpha*TVD + conf_alpha*confidence_BCE`. + +`dflash_self_logit_distillation` is **false** for DSpark — it computes the target +distribution internally for the TVD and confidence terms, so the DFlash KD path is +unused. Recipe defaults also differ from DFlash's: `block_size` 16, `num_anchors` 256, +`num_train_epochs` 6, `training_seq_len` 3072, `warmup_ratio` 0.04. + +## Per-model adjustments + +Everything in `dflash.md`'s table applies. Additionally: + +| Situation | What to change | +| --- | --- | +| Any model | **The DSpark draft does not inherit the base model's GQA/FFN dims.** Set `num_attention_heads`, `num_key_value_heads`, `head_dim`, and `intermediate_size` in `dflash_architecture_config` explicitly, or you get a silently wrong-shaped draft. Kimi-K2.6 uses `num_hidden_layers=6, num_key_value_heads=8, intermediate_size=18432`; MiniMax-M3 uses `intermediate_size=12288`. | +| Streaming | `EAGLE_CAPTURE_IDS` must be the draft's target layer ids +1 plus the final hidden layer. Kimi-K2.6: `[2,13,25,36,48,59,61]` for a 6-layer draft. Getting the final layer wrong caps acceptance length rather than erroring. | +| Sparse-attention base (e.g. MiniMax-M3 MSA) | Set `SERVE_BLOCK_SIZE` to the base's `sparse_block_size` (M3: 128) | +| Serve container lacks tensorboard | `training.report_to=none`, else trainer init crashes | +| Tokenizer can't emit assistant masks (e.g. Kimi slow tokenizer) | `training.answer_only_loss=true` still works — masks are recovered from token ids | + +## Success markers + +Same as `dflash.md`. Because the streaming examples have no smoke test or AR eval, +the only in-pipeline evidence is training progress plus the export landing in +`/scratchspace/export`. + +## Quality gate + +**Do not trust in-training AR for DSpark.** The recipe pins `estimate_ar: false` and +`ar_validate_steps: 0` deliberately: eval runs the DFlash backbone only, with the +Markov head not applied, so any reported AR reflects the backbone alone rather than +the trained model. + +`pseudo_speculative_generate` *is* overridden for DSpark (unlike Domino), so a +non-offline model can generate correctly — but the offline/streaming path deletes base +layers and refuses. Evaluate by exporting and running the offline acceptance-length +harness separately. + +Otherwise the training-regression gate from `dflash.md` (`MAX_FINAL_LOSS`, +`MIN_FINAL_ACC` via `check_regression.py`) applies; neither committed example sets +those thresholds. + +## Known failures + +Generic infrastructure failures are in `../stages/triage.md`; shared block-diffusion +failures (`seq_len` divisibility, offline eval, mask token, chat template) are in +`dflash.md`. DSpark-specific: + +| Error pattern | Root cause | Fix | +| --- | --- | --- | +| `DSpark (projector_type='dspark') requires 'markov_rank' (> 0) in dflash_architecture_config` | Markov head dimension missing | Set `dflash_architecture_config.markov_rank` | +| `DSpark requires markov_rank > 0, got N` | Non-positive value | Set a positive rank | +| `Unsupported markov_head_type: '...'. Expected 'vanilla', 'gated' or 'rnn'` | Typo or unsupported head | Use one of the three | +| `dflash_confidence_head_alpha > 0 but the confidence head was not built` | Loss term enabled without the head | Set `dflash_architecture_config.use_confidence_head=true`, or set the alpha to 0 | +| `DSpark offline model cannot run AR validation / pseudo_speculative_generate` | Offline/streaming conversion deleted base layers | Keep `estimate_ar=false` and `ar_validate_steps=0`; evaluate after export | +| Draft trains but acceptance length is poor | Draft dims left at defaults instead of matching the base | Set the GQA/FFN dims explicitly (see *Per-model adjustments*) | +| Acceptance length capped despite clean training (streaming) | `EAGLE_CAPTURE_IDS` final layer wrong, or the vLLM aux-capture fix (vllm#46788) missing | Correct the ids; use a container with the fix | +| Trainer init crash on a serve container | tensorboard absent | `training.report_to=none` | From e7dd603fe22c7a5fa35b322e860adf6b3d95bdff Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Tue, 25 Aug 2026 11:24:45 -0700 Subject: [PATCH 4/7] skills: address CodeRabbit review on the specdec skill tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - validate.md: branch the quality gate by algorithm. The stage said to extract a metric from the benchmark task's log, which contradicted the Domino and DSpark sheets — their eval path runs the DFlash backbone with the new head bypassed, and Domino ships no benchmark task at all. - validate.md: check artifacts on the filesystem when the cluster is reachable; a success log line does not prove the /scratchspace artifact survived for the next task. - triage.md: split weight-load OOM from KV-cache OOM. --max-model-len does not reduce weight memory, so it was the wrong first suggestion for a load-time OOM. - triage.md: trust_remote_code has to be set on every task that loads the model, and is spelled differently per task type — added the table. - triage.md: the re-run examples assumed 4 tasks; labelled them as EAGLE3-offline and showed how to read the real task list first. - dflash.md: state task counts per variant (online 3, offline 2, streaming 3) instead of an unqualified "three". - domino.md: future inference steps append as task_2/task_3, since task_0/task_1 are already the dataset build and training. - SKILL.md: add Domino to the trigger list; mark DSpark's confidence head optional. Not applied: the "tokenizer class not equal to the registered tokenizer class" row in review-logs.md's benign table. That table exists to stop agents chasing known-harmless noise, and the adjacent real failure — a wrong mask token id — is already documented separately in dflash.md. Signed-off-by: Ye Yu --- .../skills/speculative-decoding/SKILL.md | 4 +- .../references/algorithms/dflash.md | 5 ++- .../references/algorithms/domino.md | 11 +++-- .../references/stages/triage.md | 35 ++++++++++++++-- .../references/stages/validate.md | 40 +++++++++++++++---- 5 files changed, 76 insertions(+), 19 deletions(-) diff --git a/plugins/modelopt/skills/speculative-decoding/SKILL.md b/plugins/modelopt/skills/speculative-decoding/SKILL.md index 456e626670d..bc1ec327182 100644 --- a/plugins/modelopt/skills/speculative-decoding/SKILL.md +++ b/plugins/modelopt/skills/speculative-decoding/SKILL.md @@ -5,7 +5,7 @@ description: >- DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate - meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "draft model", + meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use ptq) or serving a checkpoint (use deployment). user_invocable: true @@ -57,7 +57,7 @@ All recipes live in `modelopt_recipes/general/speculative_decoding/.y | --- | --- | --- | | EAGLE3 | `references/algorithms/eagle3.md` | Autoregressive draft head | | DFlash | `references/algorithms/dflash.md` | Block diffusion | -| DSpark | `references/algorithms/dspark.md` | DFlash backbone + Markov head + confidence head | +| DSpark | `references/algorithms/dspark.md` | DFlash backbone + Markov head + optional confidence head | | Domino | `references/algorithms/domino.md` | DFlash backbone + GRU causal correction head | DSpark and Domino are **DFlash variants**, not separate pipelines: same diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md index 6150d4bc33a..323a308c6e2 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md @@ -10,8 +10,9 @@ Examples: `tools/launcher/examples/*/*/hf_online_dflash.yaml`, ## Pipeline tasks -Task count depends on the variant — DFlash has three, and none of them is the 4-task -shape EAGLE3 offline uses. +DFlash has three variants, and none uses EAGLE3 offline's 4-task shape: **online is 3 +tasks, offline is 2, streaming is 3.** Read the task count off the config you're +using rather than assuming one. **Online** (`hf_online_dflash.yaml`) — base model forwards during training: diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md index 4e311283d84..8ce6a85e899 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/domino.md @@ -30,9 +30,14 @@ The committed example is **online**, 2 tasks: **The inference side is intentionally not wired up yet.** The Domino correction head is not applied in `pseudo_speculative_generate` or in the serving stack, so the -example ships no vLLM smoke test and no MT-Bench AR eval. When that path lands, add -the two steps from `hf_online_dflash.yaml` (task_1 / task_2). Do not treat their -absence as a broken config. +example ships no vLLM smoke test and no MT-Bench AR eval. Do not treat their absence +as a broken config. + +When that path lands, copy the smoke-test and AR-eval steps from +`hf_online_dflash.yaml` (its `task_1` and `task_2`) and append them to the Domino +config as **`task_2` and `task_3`** — Domino's `task_0`/`task_1` are already the +dataset build and training, so keep the source file's numbering and the destination's +distinct. ## Recipe and training knobs diff --git a/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md b/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md index 37b5cf39c45..fa1a2b0c091 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/triage.md @@ -41,17 +41,31 @@ algorithm and account for most runs: | Error pattern | Root cause | Fix | | --- | --- | --- | | Server never becomes healthy (hangs at the health check) | Model too large for the allocated GPUs, or a server startup crash | Compare BF16 weight size against total allocated GPU memory; increase TP and/or nodes | -| `CUDA out of memory` during model load | Insufficient GPU memory | Reduce `--max-model-len`, or increase `--tensor-parallel-size` | +| `CUDA out of memory` **while loading weights** (before the KV cache is allocated) | The weights themselves don't fit | Increase `--tensor-parallel-size`, add nodes, or switch backend. `--max-model-len` will **not** help — it doesn't change weight memory. | +| `CUDA out of memory` **after weights load** — KV-cache allocation, or during a forward pass | Activation / KV-cache pressure | Reduce `--max-model-len`, batch size, or concurrency; raising TP also helps by splitting the cache | | `CUDA out of memory` during the hidden-state dump | Model too large for the chosen backend | Switch to a `device_map="auto"` backend, or increase TP | | `CUDA out of memory` during training | Batch or sequence length too large | Reduce the recipe's training batch size or sequence length (see the algorithm sheet's *Recipe and training knobs*) | | `CUDA out of memory` at benchmark | Target plus draft exceeds GPU memory | Increase TP | | `pyxis: child terminated with signal 15` | SIGTERM — usually OOM | Increase TP or switch backends | | `NCCL timeout` / `NCCL error` | Multi-node communication failure | Retry; reduce EP | | `CANCELLED ... DUE TO TIME LIMIT` | Slurm wall-clock limit too short | Increase `--time`. Note that `afterany` dependencies let the next task start anyway. | -| `trust_remote_code` error | Model needs custom code but the flag isn't set | Add the flag to the serving task args (before the `--` separator) **and** to the benchmark task args | +| `trust_remote_code` error | Model needs custom code but the flag isn't set for **that** task | Set it on **every** task that loads the model — see the spellings below | | Vocab / tokenizer error | Missing tokenizer cache (e.g. a tiktoken cache) | Point the relevant cache env var at a pre-populated path | | Architecture not supported by the serving engine | Engine version too old for this model | Try a newer container image | +### `trust_remote_code` spellings + +The flag is spelled differently per task type, so setting it once is not enough — a +custom-code model needs it everywhere it is loaded: + +| Task type | How to set it | +| --- | --- | +| Serving / benchmark | CLI flag before the `--` separator: `--trust-remote-code` (vLLM) or `--trust_remote_code` (trtllm-serve) | +| Hidden-state dump | `TRUST_REMOTE_CODE: "1"` in the task `environment` | +| Training | `model.trust_remote_code=true` as an OmegaConf override | +| Streaming serve replicas | `SERVE_EXTRA_ARGS: "--trust-remote-code"` | +| Export | `EXPORT_EXTRA_ARGS: "--trust_remote_code"` | + Then check *Known failures* in `../algorithms/.md` for failures specific to this algorithm — wrong script paths, missing scratchspace artifacts, export failures, draft-config incompatibilities. @@ -75,7 +89,16 @@ Provide: 3. **How to re-run** — skip earlier successful tasks by pointing at the existing scratchspace artifacts -To skip the first two tasks and re-run from the third: +Re-runs work by adding `pipeline.task_N.skip=true` for each task you want to skip. +**Read the task list out of the config first** — task count varies (EAGLE3 offline is +4, DFlash offline is 2, Domino is 2), so there is no fixed set of skip flags: + +```bash +grep -n '^ task_[0-9]*:' examples///.yaml +``` + +To resume from a failed task, skip every task before it. For a 4-task EAGLE3 offline +config whose `task_2` failed: ```bash uv run launch.py --yaml examples///.yaml \ @@ -84,7 +107,8 @@ uv run launch.py --yaml examples///.yaml \ --yes ``` -To run a single task standalone, skip every other one: +To run one task standalone, skip every other task in that config. For the same 4-task +config, running only `task_1`: ```bash uv run launch.py --yaml examples///.yaml \ @@ -94,6 +118,9 @@ uv run launch.py --yaml examples///.yaml \ --yes ``` +Both are EAGLE3-offline examples — translate the flags to the config at hand rather +than copying them verbatim. + ## Step 5 — Record the failure pattern If you hit a failure pattern not seen before, capture it in the team's internal diff --git a/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md b/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md index 783a790550c..11c5ff7fcc5 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md +++ b/plugins/modelopt/skills/speculative-decoding/references/stages/validate.md @@ -34,18 +34,42 @@ If any task failed, go to `triage.md` instead. ## Step 2 — Verify artifacts exist -Check each task produced its expected output. Artifacts live on the cluster under -`/scratchspace/`, so confirm via log messages. The per-task log evidence and artifact +Check each task produced its expected output. The per-task log evidence and artifact paths are in *Success markers* in `../algorithms/.md`. -## Step 3 — Check the quality gate +A success line in a log is not proof the artifact survived — the next task reads it +from a shared `/scratchspace`, where it may be missing, empty, or unreadable. **When +you can reach the cluster, check the filesystem directly** and treat a missing or +zero-byte artifact as a validation failure: + +```bash +test -s && echo "ok: $(du -sh )" || echo "MISSING/EMPTY" +ls -la / | head +``` + +Fall back to log evidence only when the cluster isn't reachable, and say so in the +report rather than implying the artifacts were verified. -*Quality gate* in `../algorithms/.md` gives the metric, the log line it -appears on, and the pass threshold. Extract the value from the benchmark task's log -and compare. +## Step 3 — Check the quality gate -If the log already reports the metric below its lower bound, the run tripped the -threshold check itself and exited non-zero. +Read *Quality gate* in `../algorithms/.md` first — **not every algorithm +produces an in-pipeline metric**, so what you check depends on the sheet: + +- **Sheet defines a benchmark metric** (e.g. EAGLE3's MT-Bench AR, DFlash's + `Average_AL`) — extract it from the benchmark task's log and compare against the + threshold. If the log already reports the metric below its lower bound, the run + tripped the threshold check itself and exited non-zero. +- **Sheet defines no inference metric** (currently Domino and DSpark — their eval path + runs the DFlash backbone with the new head bypassed, and Domino ships no benchmark + task at all) — do **not** go looking for a benchmark log. Report the training + regression gate instead, and state plainly that acceptance quality requires a + separate evaluation of the exported checkpoint. Never report a backbone-only + acceptance rate as the model's result. + +Where the gate is the training regression check (`check_regression.py` against +`trainer_state.json`), confirm the `=== Regression Check ===` block is actually +present in the log — it is invoked with `|| true` and only warns when no +`trainer_state.json` exists, so a green exit does not prove it ran. ## Step 4 — Check training quality From 9b3c568ed462c2fb20c565a22fa240baab4d5182 Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Mon, 31 Aug 2026 12:28:50 -0700 Subject: [PATCH 5/7] skills: fix DFlash dump draft-depth guidance and backend choice MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude review caught that the sheet attached the draft-depth constraint to the wrong flag. --aux-layers is a preset keyword accepting only 'eagle', 'dflash', or an explicit id list; it carries no count, so "--aux-layers dflash must match num_hidden_layers" gave the reader nothing to act on. The constraint is real but lives elsewhere: - vLLM dump exposes --num-draft-layers (default 5), whose help says it must match dflash.dflash_architecture_config.num_hidden_layers. - HF and TRT-LLM dumps call common.resolve_aux_layers, which hardcodes _DFLASH_DEFAULT_NUM_DRAFT_LAYERS = 5 with no override — a non-5-layer draft needs an explicit comma-separated --aux-layers id list, or the vLLM backend. Corrected in all three places that repeated the conflation (dump flags, the num_hidden_layers knob row, and the Known failures row), and noted that both offline example YAMLs carry the same misleading comment, the way the sheet already flags dflash.yaml's stale chat_templates/ path. Also adds a dump-backend subsection: configure.md sends readers to the sheet's Pipeline tasks to pick a backend, but this sheet named only the vLLM script and dead-ended. The committed offline examples use both (MiniMax-M2.7 vLLM, Qwen3-0.6B HF), and for DFlash the pick constrains draft depth, so it is not cosmetic. Signed-off-by: Ye Yu --- .../references/algorithms/dflash.md | 41 +++++++++++++++---- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md index 323a308c6e2..06960afe651 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md @@ -27,18 +27,41 @@ alongside training: | Task | Script | Purpose | Output | | --- | --- | --- | --- | -| task_0 | `common/eagle3/dump_offline_data_vllm.sh` | Dump base hidden states via vLLM (TP-sharded) | Hidden-state dump directory | +| task_0 | `common/eagle3/dump_offline_data_vllm.sh` or `dump_offline_data_hf.sh` | Dump base hidden states | Hidden-state dump directory | | task_1 | `common/specdec/dflash_online_training.sh` | Train on the dump, then export | `/exported-checkpoint-*` | -The dump script is shared with EAGLE3 — DFlash selects its own captured layers with -`--aux-layers dflash`. Two flags matter for the dump: - -- `--aux-layers dflash` — **must match the draft's `num_hidden_layers`** (recipe - default 5). `build_target_layer_ids(num_target_layers, num_draft_layers)` picks the - captured layers, so a mismatch here silently produces the wrong hidden states. +### Choosing the dump backend + +The dump script is shared with EAGLE3, so the backend choice is the same three-way +pick described in `eagle3.md` (*Choosing the task_1 dump backend*). Both committed +offline examples are in play: MiniMax-M2.7 uses `dump_offline_data_vllm.sh`, +Qwen3-0.6B uses `dump_offline_data_hf.sh`. + +For DFlash the choice is **not** cosmetic — it constrains the draft depth you can +capture. See below. + +### Dump flags + +- `--aux-layers dflash` selects DFlash's layer-selection **preset**. It is a keyword, + not a count: `--aux-layers` accepts only `eagle`, `dflash`, or an explicit + comma-separated id list (`collect_hidden_states/common.py`). +- **Draft depth is a separate flag, and only the vLLM backend exposes it.** The + captured ids come from `build_target_layer_ids(num_target_layers, num_draft_layers)`, + and `num_draft_layers` must equal the recipe's + `dflash.dflash_architecture_config.num_hidden_layers` or the dump silently captures + the wrong layers: + - **vLLM** — pass `--num-draft-layers ` (default 5). + - **HF / TRT-LLM** — no override exists; `resolve_aux_layers` hardcodes + `_DFLASH_DEFAULT_NUM_DRAFT_LAYERS = 5`. For a draft that is not 5 layers, you must + pass an explicit comma-separated id list to `--aux-layers`, or use the vLLM backend. - `--answer-only-loss` and `--chat-template` — must agree with the training task's `training.answer_only_loss` and `data.chat_template`. +> Both offline example YAMLs comment `--aux-layers dflash` with "Must match the draft +> model's num_hidden_layers (recipe default: 5)". That comment is misleading — it +> attaches the constraint to the wrong flag. The constraint is real; the knob is +> `--num-draft-layers` (vLLM) or an explicit id list (HF / TRT-LLM). + Offline training additionally needs `data.mode=offline`, `model.use_fake_base_for_offline=true` (loads only `lm_head` + `embed_tokens` rather than the full base), and `data.offline_data_path` pointing at the dump. @@ -62,7 +85,7 @@ in `examples/speculative_decoding/README.md#dflash-block-diffusion-for-speculati | `dflash.dflash_num_anchors` | 512 | Random anchor positions sampled per sequence | | `dflash.dflash_loss_decay_factor` | 4.0 | Exponential decay gamma; 0 disables | | `dflash.dflash_self_logit_distillation` | true | Logit distillation from the target | -| `dflash.dflash_architecture_config.num_hidden_layers` | 5 | Draft decoder layers — keep in sync with `--aux-layers` | +| `dflash.dflash_architecture_config.num_hidden_layers` | 5 | Draft decoder layers — must equal the dump's draft depth (`--num-draft-layers` on vLLM; hardcoded 5 on HF / TRT-LLM) | | `dflash.dflash_mask_token_id` | auto | See *Per-model adjustments* | | `dflash.dflash_swa_window_size` | unset | Sliding-window attention for the draft; must be >= `dflash_block_size` | | `dflash.dflash_export_rope_scaling` | `{}` | YaRN config injected at export so a short-window draft can serve long context | @@ -137,7 +160,7 @@ DFlash-specific: | `The base model did not return hidden states required for DFlash training` | Base model's top-level forward ignores `output_hidden_states=True` | Usually a multimodal wrapper — needs a model-side fix | | `ERROR: DRAFT_CKPT_DIR=... contains no exported-checkpoint-* directory` | Upstream training produced no draft | Fix training; do not chase the smoke test | | vLLM rejects the speculative config / no DFlash method | DFlash landed in vLLM v0.22.0 (`vllm/v1/spec_decode/dflash.py`) | Use `vllm/vllm-openai:v0.22.1` or newer | -| Draft quality plateaus despite clean training | `--aux-layers` count and `num_hidden_layers` disagree, so the dump captured the wrong layers | Re-dump with matching values | +| Draft quality plateaus despite clean training | Dump draft depth and `num_hidden_layers` disagree, so the dump captured the wrong layers. Note `--aux-layers dflash` defaults to a 5-layer draft on **every** backend, so a 6-layer draft silently mis-captures unless you override | Re-dump with `--num-draft-layers ` (vLLM), or an explicit `--aux-layers` id list (HF / TRT-LLM) | | Loss stalls high with `answer_only_loss=true` | Chat template lacks `{% generation %}` tags, so no positions contribute loss | Supply a template with generation tags | | `dflash_dpace_alpha must be in (0, 1]` | Invalid D-PACE alpha | Correct the value | | Qwen3-VL mRoPE / `mm_token_type_ids` errors | Qwen3-VL DFlash needs Transformers 5.3.0 or >=5.4.0 and the AutoProcessor's `mm_token_type_ids` | Match the version; don't drop processor outputs | From c2696b1c45f2cc94584ec72b9db8066d69b4d371 Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Mon, 31 Aug 2026 12:35:07 -0700 Subject: [PATCH 6/7] fix stale comments that misdirect DFlash offline configuration Both were found while sourcing the speculative-decoding skill sheets, and the second one had already misled this PR's first draft. - dflash.yaml pointed chat_template at a chat_templates/ directory under modelopt_recipes that does not exist. Templates live per-model beside each launcher example. - Both offline DFlash example YAMLs annotated --aux-layers dflash with "Must match the draft model's num_hidden_layers". --aux-layers is a preset keyword accepting only 'eagle', 'dflash', or an explicit id list, so it carries no count and there is nothing on it to match. The constraint is real but belongs to the draft depth the preset resolves to: --num-draft-layers on the vLLM dump, and no override at all on the HF/TRT-LLM dumps, which hardcode 5 via resolve_aux_layers. The comments now name the right knob per backend, so a non-5-layer draft no longer reads as configured when it silently mis-captures. Comment-only; no behavior change. The skill sheet's notes flagging these as stale are updated to match. Signed-off-by: Ye Yu --- .../general/speculative_decoding/dflash.yaml | 3 ++- .../references/algorithms/dflash.md | 10 +++++----- .../MiniMax/MiniMax-M2.7-DFlash/hf_offline_dflash.yaml | 4 +++- .../examples/Qwen/Qwen3-0.6B/hf_offline_dflash.yaml | 5 ++++- 4 files changed, 14 insertions(+), 8 deletions(-) diff --git a/modelopt_recipes/general/speculative_decoding/dflash.yaml b/modelopt_recipes/general/speculative_decoding/dflash.yaml index 021cccd475d..737789dc4ed 100644 --- a/modelopt_recipes/general/speculative_decoding/dflash.yaml +++ b/modelopt_recipes/general/speculative_decoding/dflash.yaml @@ -16,7 +16,8 @@ data: offline_data_path: # Jinja chat template with {% generation %} tags for answer_only_loss. # Required when answer_only_loss=true. Set in per-model launcher YAML. - # Templates are in modelopt_recipes/general/speculative_decoding/chat_templates/ + # Each model keeps its own beside its launcher example, e.g. + # tools/launcher/examples/Qwen/Qwen3-8B/chat_template_train.jinja chat_template: # maps to TrainingArguments (main.py) diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md index 06960afe651..83b5b5940de 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dflash.md @@ -57,10 +57,10 @@ capture. See below. - `--answer-only-loss` and `--chat-template` — must agree with the training task's `training.answer_only_loss` and `data.chat_template`. -> Both offline example YAMLs comment `--aux-layers dflash` with "Must match the draft -> model's num_hidden_layers (recipe default: 5)". That comment is misleading — it -> attaches the constraint to the wrong flag. The constraint is real; the knob is -> `--num-draft-layers` (vLLM) or an explicit id list (HF / TRT-LLM). +> Both offline example YAMLs used to attach this constraint to `--aux-layers`, which +> carries no count. Their comments were corrected alongside this sheet — if you find +> the old wording anywhere else, the knob is `--num-draft-layers` (vLLM) or an +> explicit id list (HF / TRT-LLM). Offline training additionally needs `data.mode=offline`, `model.use_fake_base_for_offline=true` (loads only `lm_head` + `embed_tokens` rather @@ -102,7 +102,7 @@ Export is automatic: after training, rank 0 exports every `checkpoint-` to | Situation | What to change | | --- | --- | | Any model | Pin `dflash.dflash_mask_token_id` to a token that **already exists in the target's embedding** — the draft reuses the target's `embed_tokens`. Unset falls back to `tokenizer.mask_token_id`, which many tokenizers lack. MiniMax-M2.7 uses a reserved row (200054); Qwen3-8B uses 151669. | -| `answer_only_loss=true` (recipe default) | The chat template must contain `{% generation %}` / `{% endgeneration %}` tags. Most stock templates don't — supply one via `data.chat_template=.jinja`. Each model keeps its own next to its example YAML (`examples///chat_template_train.jinja`); copy the closest one. Note `dflash.yaml`'s comment points at a `chat_templates/` recipe directory that does not exist — ignore it. | +| `answer_only_loss=true` (recipe default) | The chat template must contain `{% generation %}` / `{% endgeneration %}` tags. Most stock templates don't — supply one via `data.chat_template=.jinja`. Each model keeps its own next to its example YAML (`examples///chat_template_train.jinja`); copy the closest one. | | `trust_remote_code` MoE with an older transformers pin | Set `OVERRIDE_TRANSFORMERS` in the task environment (MiniMax-M2.7 needs 4.57.1). Set `ACCELERATE_CONFIG` when the model needs FSDP2 via accelerate config rather than transformers-native `ParallelismConfig`. | | Very large MoE base | Use the offline variant with `model.use_fake_base_for_offline=true`; plain DDP suffices, so no FSDP2 patches. Set `MIXED_PRECISION: "no"` with `training.bf16=false` if the model requires it. | | Draft trained at short context, served long | Set `dflash.dflash_export_rope_scaling` (YaRN); factor = target context / `training_seq_len`. | diff --git a/tools/launcher/examples/MiniMax/MiniMax-M2.7-DFlash/hf_offline_dflash.yaml b/tools/launcher/examples/MiniMax/MiniMax-M2.7-DFlash/hf_offline_dflash.yaml index dc3dba4a65a..84e623cb3c0 100644 --- a/tools/launcher/examples/MiniMax/MiniMax-M2.7-DFlash/hf_offline_dflash.yaml +++ b/tools/launcher/examples/MiniMax/MiniMax-M2.7-DFlash/hf_offline_dflash.yaml @@ -27,7 +27,9 @@ pipeline: args: - --input-data /hf-local/modelopt/MiniMax-M2.7-synthetic-data-clean-v2 - --output-dir /scratchspace/dflash_minimax_m2.7_hidden_states - # Must match the draft model's num_hidden_layers (recipe default: 5). + # Layer-selection preset (keyword, not a count). The draft depth it resolves + # to is --num-draft-layers (default 5), which must match the draft model's + # dflash_architecture_config.num_hidden_layers set on task_1 below. - --aux-layers dflash - --answer-only-loss - --chat-template examples/MiniMax/MiniMax-M2.7-DFlash/chat_template_train.jinja diff --git a/tools/launcher/examples/Qwen/Qwen3-0.6B/hf_offline_dflash.yaml b/tools/launcher/examples/Qwen/Qwen3-0.6B/hf_offline_dflash.yaml index 6096b4b663d..2c77d81aa17 100644 --- a/tools/launcher/examples/Qwen/Qwen3-0.6B/hf_offline_dflash.yaml +++ b/tools/launcher/examples/Qwen/Qwen3-0.6B/hf_offline_dflash.yaml @@ -24,7 +24,10 @@ pipeline: args: - --input-data modules/Model-Optimizer/examples/dataset/synthetic_conversations_1k.jsonl - --output-dir /scratchspace/dflash_qwen3_0.6b_hidden_states - # Must match the draft model's num_hidden_layers (recipe default: 5). + # Layer-selection preset (keyword, not a count). It resolves to a 5-layer + # draft, which must match the draft model's num_hidden_layers (recipe + # default: 5). The HF dump has no --num-draft-layers override, so a draft + # of a different depth needs an explicit id list here, e.g. --aux-layers 2,5,8. - --aux-layers dflash - --answer-only-loss - --chat-template examples/Qwen/Qwen3-0.6B/chat_template_train.jinja From 3d0fc458f43518fc51bdaff21dee006263ee40b0 Mon Sep 17 00:00:00 2001 From: Ye Yu Date: Thu, 3 Sep 2026 09:49:36 -0700 Subject: [PATCH 7/7] skills: cover the DSpark variants that landed on main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge brought in three DSpark capabilities the sheet contradicted: drafter PTQ (quantize_drafter.sh + hf_dspark_ptq_*.yaml), streaming warm-start from a released drafter, and per-model recipes under modelopt_recipes/models/. The sheet claimed both committed examples were streaming-from-scratch, which is no longer true. Three of the new facts are silent-failure modes, so they go in Known failures rather than prose: - The q/k/v excludes in drafter PTQ are mandatory, not tuning. DFlash family drafters build their fused context-KV projection by reading qkv_proj.weight raw, which cannot be a packed tensor. - specdec_bench reads --block_size for DSPARK/DFLASH, not --draft_length, and it must match the drafter's block size. - Warm-start drafter shape belongs in the per-model recipe, transcribed from the released checkpoint's config.json; overriding it in the launcher YAML mismatches the published weights. Quality gate updated too: the PTQ examples do produce an inference metric, unlike the streaming ones, but the gate is relative — benchmark the unquantized drafter and compare. Signed-off-by: Ye Yu --- .../references/algorithms/dspark.md | 56 ++++++++++++++++--- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md index 005cec2ae33..2547448fdea 100644 --- a/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md +++ b/plugins/modelopt/skills/speculative-decoding/references/algorithms/dspark.md @@ -13,22 +13,45 @@ Recipe: `modelopt_recipes/general/speculative_decoding/dspark.yaml` (its `metadata.recipe_type` is `speculative_dflash`, and every knob lives in the `dflash.*` namespace). -Examples: `tools/launcher/examples/moonshotai/Kimi-K2.6/hf_streaming_dspark_multi_node.yaml`, -`tools/launcher/examples/MiniMaxAI/MiniMax-M3/hf_streaming_dspark_multi_node.yaml`. +Examples: `tools/launcher/examples/*/*/hf_streaming_dspark_multi_node.yaml`, +`hf_streaming_dspark_warmstart.yaml`, `hf_dspark_ptq_*.yaml`. ## Pipeline tasks -Both committed examples are **streaming, multi-node** — 2 tasks: +Three committed shapes, all 2 tasks. No committed online or offline DSpark example; +the recipe defaults to `data.mode=online`, so an online run would use +`common/specdec/dflash_online_training.sh` exactly as DFlash does. + +**Streaming, multi-node, from scratch** (Kimi-K2.6, MiniMax-M3): | Task | Script | Purpose | Output | | --- | --- | --- | --- | | task_0 | `common/eagle3/make_dataset.sh` | Build training conversations | `/scratchspace/data/train.jsonl` | | task_1 | `common/eagle3/train_eagle_streaming.sh` | Streaming train (serve replicas + DDP trainers over NIXL RDMA), then export | `/scratchspace/export` | -`data.mode=streaming` with `model.use_fake_base_for_offline=true`. There is no -committed online or offline DSpark example; the recipe defaults to `data.mode=online`, -so an online run would use `common/specdec/dflash_online_training.sh` exactly as -DFlash does. +`data.mode=streaming` with `model.use_fake_base_for_offline=true`. + +**Streaming warm-start** (Nemotron-3.5-Lightning) — same two scripts, but continues a +**released** drafter from its published weights instead of training from scratch. The +key difference: the drafter architecture is **not** overridden in the YAML. It lives in +a per-model recipe under `modelopt_recipes/models///speculative_decoding/`, +because every field there is transcribed from the released checkpoint's own +`config.json` and must match it exactly. Keep drafter shape changes in that recipe and +cluster/serve knobs in the YAML. + +**Drafter PTQ** (Qwen3-8B nvfp4; Nemotron-3.5-Lightning fp8 and nvfp4) — quantizes an +already-exported drafter. Calibration-free: every scale comes from the weights. + +| Task | Script | Purpose | Output | +| --- | --- | --- | --- | +| task_0 | `common/specdec/quantize_drafter.sh` | Quantize the drafter (CPU-only, ~1 min for an 8B-class draft) | `/scratchspace/export_quantized` | +| task_1 | `common/specdec_bench/quick_check.sh` | Acceptance length on MT-Bench, so the cost is visible | JSON results | + +`DRAFTER_CKPT` accepts an exported checkpoint, an HF repo id, or a training +`output_dir` — for a directory it auto-selects the newest `exported-checkpoint-*`. +`w4a16_nvfp4` keeps activations in bf16; `nvfp4` does weight+activation with a fixed +`input_scale` of 1.0. To measure what quantization cost, re-run task_1 against the +unquantized drafter and compare acceptance length. Streaming-specific environment (set in `task_1`, see `common/eagle3/train_eagle_streaming.sh` for dispatch and rendezvous): @@ -46,6 +69,11 @@ AR eval step. ## Recipe and training knobs +The general recipe is `modelopt_recipes/general/speculative_decoding/dspark.yaml`. A +warm-start run instead points `--config` at a **per-model** recipe under +`modelopt_recipes/models///speculative_decoding/`, which pins the drafter +shape to the released checkpoint. + Everything in `dflash.md` applies. DSpark adds: | Override | Recipe default | Note | @@ -96,9 +124,18 @@ layers and refuses. Evaluate by exporting and running the offline acceptance-len harness separately. Otherwise the training-regression gate from `dflash.md` (`MAX_FINAL_LOSS`, -`MIN_FINAL_ACC` via `check_regression.py`) applies; neither committed example sets +`MIN_FINAL_ACC` via `check_regression.py`) applies; none of the streaming examples set those thresholds. +The **PTQ** examples are the exception — they do produce a real inference metric, via +`quick_check.sh` on MT-Bench with `--speculative_algorithm DSPARK`. There is no fixed +pass threshold; the gate is relative. Run the same benchmark against the unquantized +drafter and compare acceptance length, so the quantization cost is explicit rather than +assumed. Published reference points on Qwen3-8B: the weight+activation `nvfp4` format +costs roughly 3.9% acceptance length, which is why the examples default to +`w4a16_nvfp4` (weights only, activations in bf16). Adding `fc` to the quantized set +costs about 0.7% acceptance length (3.0186 vs 3.0392) for ~3% more size saved. + ## Known failures Generic infrastructure failures are in `../stages/triage.md`; shared block-diffusion @@ -115,3 +152,6 @@ failures (`seq_len` divisibility, offline eval, mask token, chat template) are i | Draft trains but acceptance length is poor | Draft dims left at defaults instead of matching the base | Set the GQA/FFN dims explicitly (see *Per-model adjustments*) | | Acceptance length capped despite clean training (streaming) | `EAGLE_CAPTURE_IDS` final layer wrong, or the vLLM aux-capture fix (vllm#46788) missing | Correct the ids; use a container with the fix | | Trainer init crash on a serve container | tensorboard absent | `training.report_to=none` | +| Drafter PTQ produces a broken draft, or export fails on a packed tensor | q/k/v were quantized. DFlash-family drafters build their fused context-KV projection by reading `qkv_proj.weight` raw, which cannot be a packed tensor | The exclusions are **mandatory**, not tuning: `--exclude '*q_proj*' '*k_proj*' '*v_proj*' '*qkv_proj*'`. `o_proj` and the MLP still quantize; `*fc*` is optional (~3% more size saved for ~0.7% acceptance length on Qwen3-8B) | +| Benchmark reports a poor or nonsensical acceptance length | `--draft_length` was passed. DSPARK and DFLASH read `--block_size` instead, and it must match the drafter's block size | Pass `--block_size ` matching the drafter, plus `--speculative_algorithm DSPARK` | +| Warm-start run diverges or the drafter shape mismatches the released checkpoint | Drafter architecture overridden in the launcher YAML | Leave shape fields in the per-model recipe — they are transcribed from the released `config.json` and must match exactly |