Skip to content

Preserve first-captured MoE routes for extending prompts - #890

Merged
FurtherAI merged 1 commit into
mainfrom
austin/release_route_first_wins
Sep 12, 2026
Merged

Preserve first-captured MoE routes for extending prompts#890
FurtherAI merged 1 commit into
mainfrom
austin/release_route_first_wins

Conversation

@FurtherAI

Copy link
Copy Markdown
Collaborator

Summary

  • Make overlapping MoE route assembly first-write-wins for multi-call trajectories with extending prompts.
  • Fill only route positions that have not already been covered.
  • Leave token identity, route shape, expert bounds, and downstream prefix-tree packing validation unchanged.

Why

vLLM retains routed experts for full prefix-cache blocks, but an unaligned suffix can be recomputed when a later request extends the prompt. Those tokens may move from decode execution to prefill execution and legitimately select different experts. A request that crosses an in-flight policy update can also recompute prompt positions under the new policy.

The earlier capture remains the replay authority for already-recorded tokens. Rejecting a later disagreement therefore aborts a valid trajectory instead of preserving its original routes.

Validation

  • Real captured three-call trajectory: one 4,184-token sequence, 704 intended loss tokens, exact first-capture route ownership, unchanged tokens, masks, logprobs, offsets, and serialized trajectory.
  • Focused MoE routing path: 14 passed.
  • Adjacent preprocessing and prefix-tree tests: 44 passed, 2 environment-dependent Gemma tests deselected because the local CPU environment lacks the Transformer Engine shared library.
  • Scratch overlap controls cover partial, empty, repeated, uint8, and uint16 inputs.
  • Ruff, formatting, type checking, lock sync, and git diff checks pass.

@FurtherAI
FurtherAI merged commit 879d336 into main Sep 12, 2026
7 checks passed
@FurtherAI
FurtherAI deleted the austin/release_route_first_wins branch September 12, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant