Skip to content

fix(ds4): cancel abandoned inference - #11822

Open
FiloSpaTeam wants to merge 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-request-cancellation
Open

fix(ds4): cancel abandoned inference#11822
FiloSpaTeam wants to merge 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-request-cancellation

Conversation

@FiloSpaTeam

Copy link
Copy Markdown
Contributor

Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.

Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.

Assisted-by: Codex:gpt-5.6-sol

Description

This PR fixes DS4 inference continuing after the originating gRPC request has been cancelled or disconnected.

Cancellation is propagated into prompt synchronization and checked at safe decode boundaries, between accepted MTP tokens, during distributed route waits, and after streaming write failures.

Abandoned requests do not finalize incomplete parser state or persist partial KV-cache state. Backend failures retain their existing error status rather than being incorrectly converted to cancellation.

Notes for Reviewers

  • Prompt synchronization uses a scoped cancellation callback that is cleared afterward.
  • Distributed route waits poll for cancellation instead of blocking for the complete timeout.
  • Streaming finalization is transactional: parser state and KV cache are committed only after the final write succeeds.
  • Already-streamed chunks cannot be retracted.
  • An active GPU kernel cannot be interrupted mid-execution; cancellation takes effect at the next safe boundary.
  • Generation-limit and MTP-budget changes are intentionally handled in a separate PR.
  • Both PRs modify grpc-server.cpp, so this branch may require a small rebase after the generation-boundary PR merges.
  • All 17 standalone DS4 C++ test files pass.
  • A complete CUDA/GB10 Docker end-to-end test was not available in the development workspace.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.

Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.

Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
@FiloSpaTeam
FiloSpaTeam force-pushed the fix/ds4-request-cancellation branch from 852c096 to 178157a Compare September 1, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant