Skip to content

fix(ds4): enforce generation boundaries - #11821

Open
FiloSpaTeam wants to merge 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-generation-boundaries
Open

fix(ds4): enforce generation boundaries#11821
FiloSpaTeam wants to merge 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-generation-boundaries

Conversation

@FiloSpaTeam

Copy link
Copy Markdown
Contributor

Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.

Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.

Assisted-by: Codex:gpt-5.6-sol

Description

This PR fixes DS4 generation boundary handling in both streaming and non-streaming requests.

Explicit generation limits are clamped after prompt synchronization using the remaining context capacity. Omitted or non-positive limits retain the legacy 256-token fallback.

Speculative MTP decoding is also constrained by the remaining request budget, preventing accepted tokens from advancing beyond the visible output limit.

Notes for Reviewers

  • The existing 256-token fallback remains unchanged.
  • An explicit max_tokens: 32768 is treated as an upper bound and may be reduced to fit the available model context.
  • Streaming and non-streaming paths use the same limit calculation.
  • Request cancellation is intentionally excluded and will be submitted separately.
  • All 17 standalone DS4 C++ test files pass.
  • A full CUDA/GB10 Docker end-to-end build was not available in the development workspace.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

@FiloSpaTeam
FiloSpaTeam force-pushed the fix/ds4-generation-boundaries branch from 41987c0 to 114687f Compare September 1, 2026 14:26
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.

Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.

Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
@FiloSpaTeam
FiloSpaTeam force-pushed the fix/ds4-generation-boundaries branch from 114687f to 877efad Compare September 1, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant