Skip to content

feat(medusa): add candidate-tree consumer and greedy acceptance evaluator - #56

Draft
sysuTobo wants to merge 4 commits into
inclusionAI:mainfrom
sysuTobo:feat/medusa-consumer
Draft

sysuTobo wants to merge 4 commits into
inclusionAI:mainfrom
sysuTobo:feat/medusa-consumer

Conversation

@sysuTobo

@sysuTobo sysuTobo commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR status: not ready for review

This is the inference-side increment for #17 (Track A).

Summary

  • Consumer: aspec/inference/medusa_consumer.py ports the reference Medusa tree buffers (indices, positions, retrieval paths, ancestor mask), candidate construction from one Target post-norm hidden state, batched teacher-forced path verification, and acceptance accounting (accept_length + 1 per verifier call). Policies: greedy, deterministic typical, and seeded nucleus sampling.
  • Evaluator: python -m tools.evaluate_medusa_acceptance measures acceptance length, greedy decode equality and repeat stability against a Hugging Face Target (--sampling greedy|typical|nucleus, temperature/top-p/posterior thresholds/seed), recording the draft manifest, Target path, tree-config hash and hardware. Decode equality is only reported for greedy.
  • Reference parity: tree buffers match the pinned FasterDecoding/Medusa revision exactly; for a random K=5 case the cartesian candidates match the reference generate_candidates.
  • Tests: tree-buffer parity, greedy/typical/nucleus path selection, HF verifier equivalence to sequential teacher forcing, speculative-vs-Target greedy equality, sampling repeat stability, EOS truncation, and default-choice filtering.
  • Docs: Medusa guide, algorithms page and support matrix describe the standalone evaluator with greedy/typical/nucleus acceptance and the absence of SGLang/production-serving claims.

This component is not a serving runtime: the caller owns the Target verifier and cache policy. No throughput claim is made.

GPU validation

The consumer was exercised end-to-end on the existing K=2 smoke draft with Qwen3-8B on one L20:

  • Greedy acceptance (4 prompts x 32 tokens): acceptance length 1.1429 (32/28, bonus included),
    decode_equal=true, repeat_stable=true.
  • Typical sampling (--temperature 0.7) and seeded nucleus sampling (--temperature 0.7 --top-p 0.8 --seed 0) on 2 prompts x 16 tokens: acceptance length 1.1852 each,
    repeat_stable=true; decode equality is not applicable to sampling.

This validates the consumer/evaluator path. K=5 training, recipe and held-out evidence are tracked
outside this PR.

Checks actually run

  • pytest -q tests/cpu tests/source — 1271 passed
  • ruff check aspec tests tools setup.py — pass
  • python -m tools.check_docs — pass
  • python -m compileall -q aspec tests tools — pass

Not in scope

  • SGLang Medusa integration and throughput/performance claims.
  • K=5 training/recipe evidence and the reference loss/acceptance comparison (tracked separately).

Part of #17.

Port the reference tree buffers, candidate construction and greedy acceptance,
and add a standalone Hugging Face Target evaluator that reports acceptance
length, greedy decode equality and repeat stability. Tree buffers match the
pinned Medusa revision exactly; the evaluator is a reference component, not a
serving runtime.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Extend the consumer beyond greedy: deterministic typical acceptance and
seeded nucleus sampling, with verifier logits, CLI flags, docs and tests.
Decode equality stays greedy-only; sampling reports acceptance and repeat
stability only.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant