[Bugfix] Fix the Eagle3 inference failure issue. #4544
Closed
+0
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request aims to fix an inference failure with Eagle3 speculative decoding. The changes involve correctly setting the attention state for Eagle3, adjusting how attention masks are generated for prefill with cache hits, and reusing pre-computed attention masks. Overall, the changes appear to correctly address the issue. However, I've found a critical bug in the calculation of max_seq_len within get_splitfuse_attn_mask which could lead to a runtime error. I've provided a specific suggestion to fix this.
Comment on lines
108
to
112
| max_seq_len = max(seq_lens, default=0) | ||
| if hasattr(max_seq_len, "item"): | ||
| max_seq_len = int(max_seq_len.item()) | ||
| else: | ||
| max_seq_len = int(max_seq_len) if max_seq_len else 0 |
Contributor
There was a problem hiding this comment.
The method for calculating max_seq_len is incorrect when seq_lens is a tensor. The max() built-in function cannot be used on a tensor to find its maximum value; seq_lens.max() should be used instead. The current implementation will raise a ValueError if seq_lens is a tensor. This logic can be simplified and corrected to properly handle a tensor input as specified by the type hint.
Suggested change
| max_seq_len = max(seq_lens, default=0) | |
| if hasattr(max_seq_len, "item"): | |
| max_seq_len = int(max_seq_len.item()) | |
| else: | |
| max_seq_len = int(max_seq_len) if max_seq_len else 0 | |
| max_seq_len = seq_lens.max().item() if seq_lens.numel() > 0 else 0 |
sunchendd
force-pushed
the
v0.11.0-dev
branch
4 times, most recently
from
bd75832 to
64e221d
Compare
sunchendd
changed the title
sunchendd
force-pushed
the
v0.11.0-dev
branch
from
64e221d to
6e4dd8e
Compare
Contributor
Qwen/Qwen3-32B eagle3 accuracy failedThe output of `python collect_env.py`command: vllm serve Qwen/Qwen3-32B --served-model-name Qwen3-32B --tensor-parallel-size 4 --trust-remote-code --speculative-config '{"method":"eagle3","model":"/root/.cache/modelscope/hub/models/AngelSlim/Qwen3-32B_eagle3","num_speculative_tokens":4,"draft_tensor_parallel_size":1}' --max-num-batched-tokens 4096 --max-model-len 4096Qwen/Qwen3-32B eagle3 accuracy failed |
Sparkheart
reviewed
Dec 3, 2025
| cann_version = getattr(torch.version, "cann", "") | ||
| target_device = device or self.device | ||
| use_chunked_mask = (seq_lens is None or position is None | ||
| or dtype is None or cann_version.startswith("8.3")) |
There was a problem hiding this comment.
Hardcoding version 8.3 for judgment is not conducive to maintenance. It is recommended to define the version condition as a constant (e.g., MIN_CANN_VERSION_FOR_OPTIMIZED_MASK = "8.3").
Sparkheart
reviewed
Dec 3, 2025
| if target_device is None: | ||
| raise ValueError( | ||
| "splitfuse_attn_mask requires device for non-chunked mask") | ||
| max_seq_len = seq_lens.max().item() if seq_lens.numel() > 0 else 0 |
There was a problem hiding this comment.
Magic number: The value 0 for max_seq_len should be defined as a constant (e.g., DEFAULT_MAX_SEQ_LEN = 0).
sunchendd
force-pushed
the
v0.11.0-dev
branch
6 times, most recently
from
12b7323 to
3388cc6
Compare
sunchendd
force-pushed
the
v0.11.0-dev
branch
from
3388cc6 to
c4a11a7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
Fix the Eagle3 inference failure issue.
error message: "EngineCore encountered an issue. See stack trace (above) for the root cause."
Fixes #4323
How was this patch tested?
vllm serve /nfs/1_AscendPackage/05_weights_public/Qwen3-32B \ --served-model-name Qwen3-32B \ -tp 4 \ --host "0.0.0.0" \ --port "8000" \ --trust-remote-code \ --speculative-config '{"method":"eagle3","model":"/home/scd/qwen3_32b_eagle3/","num_speculative_tokens":4,"draft_tensor_parallel_size":1}' \ --max-num-batched-tokens 4096 \ --max-model-len 4096vLLM version: v0.11.0
vLLM-ascend version: v0.11.0rc2