feat: Automatic KV/MoE Laddering for decode speed vs context-length trade off ( upto 33% faster decode )

[aswinkumar1999]

Add an automatic KV/MoE cache ladder

Summary

This adds --enable-kv-ladder, an opt-in serving mode that grows the KV cache when an
incoming request could reach its current capacity. Each growth trades GPU-resident MoE cache
slots for KV pages while staying inside the engine's existing measured cache budget.

It also adds --ladder-step-size, which defaults to 32,768 tokens. With the ladder enabled,
startup KV is at least twice the configured step size and grows by that step until reaching the
model's context limit.

ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --moe-backend offload --moe-cache-auto \
  --max-running-requests 1 --memory-ratio 0.93 \
  --enable-kv-ladder

With the default step and this model, the resulting ladder is:

65,536 -> 98,304 -> 131,072 -> 163,840 -> 196,608 -> 229,376 -> 262,144 ( or Model Maximum )

Motivation

Preallocating KV for the full model context reduces the number of MoE expert slots that can
remain resident on the GPU, even when most requests use much shorter contexts. Reserving too
little KV preserves decode throughput but prevents long sessions from reaching the model's full
context.

The ladder keeps more experts resident for common shorter requests and spends that memory on KV
only when a request actually needs the next context rung.

Implementation

• Compute the possible request length as input tokens plus the request's maximum output tokens.
• If that length reaches the current KV bound, hold the request until the scheduler is idle.
• Select the smallest configured rung that fits, always moving at least one rung when a request
  could exactly fill the current capacity.
• Derive the replacement MoE slot count from measured KV-page and expert-slot costs.
• Retain at least the model's minimum expert floor.
• Reuse the existing rollback-safe runtime cache-rebuild path.
• Admit the held request after a successful rebuild while preserving queued request order.
• Keep serving with the old cache geometry if a rebuild is rejected or fails.
• Cap growth at the checkpoint's real context limit.

The initial version is intentionally restricted to TP=1, --max-running-requests 1,
--moe-cache-auto, and an offload-family MoE backend.

Measured results

Environment:

• NVIDIA RTX 5090, 32 GiB
• RadixArk/Qwen3.8-Flash-Next-NVFP4
• offloaded MoE with automatic LRU cache sizing
• one running request
• memory ratio 0.93
• one warm-up request at each geometry

KV capacity	MoE slots	128-token decode	1,024-token decode	4,096-token decode	Long-decode gain vs fixed 256K
65,536	6,425	69.3 tok/s	81.8 tok/s	83.8 tok/s	+33.1%
98,304	6,125	69.0 tok/s	76.7 tok/s	80.9 tok/s	+28.5%
131,072	5,826	68.3 tok/s	73.5 tok/s	77.9 tok/s	+23.7%
163,840	5,526	67.8 tok/s	69.1 tok/s	74.4 tok/s	+18.1%
196,608	5,226	66.1 tok/s	65.0 tok/s	70.7 tok/s	+12.3%
229,376	4,927	64.2 tok/s	61.7 tok/s	67.1 tok/s	+6.5%
262,144	4,627	61.1 tok/s	57.1 tok/s	63.0 tok/s	baseline

Short-output results include proportionally more fixed launch and streaming overhead. The
4,096-token column is the more representative sustained-decode comparison.

Rebuild latency

• Warm/manual rebuilds between previously exercised geometries completed end to end in
  approximately 0.67-0.99 seconds.
• A first automatic visit to a new geometry required approximately 3-4 seconds, primarily
  for CUDA graph capture.

Long-context request TTFT should not be interpreted as rebuild latency. For example, the final
196,608-token cold prompt reached first token in 125.8 seconds, but almost all of that time was
prompt prefill rather than the cache rebuild.

Validation

• Exercised every automatic rung from 65,536 through 262,144 tokens on the target model.
• Every automatic rebuild returned status=ok; no capacity rejection occurred.
• Verified short, medium, and long forced decode lengths at every geometry.
• Focused scheduler/cache/parser test run: 67 passed locally.
• Python byte-compilation and git diff --check passed.

Known limitation

This implementation is a grow-only high-watermark ladder for single-decode use cases. A new, unrelated prompt with no
prefix-cache reuse does not yet shrink KV and restore MoE slots automatically. The server sees
stateless prompts and has no direct session-ended signal; shrinking immediately on idle would
also erase useful KV between consecutive turns of the same conversation.

A follow-up can inspect the next tokenized request's reusable-prefix hit and, when reuse is
negligible, rebuild to the smallest rung that fits that request. That should include hysteresis
to prevent rebuild thrashing.