[mergennachin]

Summary

When two independent requests share a long prompt, the second can repeat prefill even though the first already holds the matching KV state. This adds an experimental serving path that clones the already-computed part of an active request's prompt, then prefills only the recipient's remaining input.

A: [shared prompt: already computed][A's own work] ---> continues
                 |
                 +--- clone committed state
                 |
B: [shared prompt: reused         ][B's own input] --> generates

For each arriving anonymous text request, the server selects the active request offering the longest usable match. The clone position is the minimum of the common prompt length, the donor's committed position, and the recipient's prompt length minus one. That last token still runs through the model to obtain the recipient's next-token logits. Generated donor output is excluded from matching.

The server makes one asynchronous clone attempt using state already computed; it does not wait for the donor to reach a future position. The runner completes the clone outside its locks and returns it through the serving control loop. Both requests then own independent sessions and generation settings. If cloning fails, including when sliding-window state has been overwritten, the recipient falls back to the existing cached or full-prefill path.

--active-prefix-reuse-min-tokens sets the minimum extra prefill tokens saved over a cached match. 0 disables the feature by default; the current experiment uses 1024. An equally useful cached prefix takes precedence. The policy uses the shared serving/runner code and existing MLX/CUDA clone implementations, with no scheduler or backend storage changes. Named sessions and multimodal requests retain their existing behavior.

For review, start with candidate selection and clone completion in serving_runtime.cpp, then the callback overload in runner.cpp, followed by the focused tests and launcher plumbing. This PR is independent of #24304.

This remains a draft while assessing the limited workload benefit; CUDA-device validation is still pending. Authored with assistance from Codex.

Replay results

One paired run on the same 20-session sample, with the feature disabled versus a 1,024-token threshold:

Metric	Baseline	Active reuse
Replay time	76.57 min	75.71 min
Prefilled tokens	522,209	513,597
Median visible TTFT	2.220 s	2.225 s
P95 visible TTFT	35.28 s	35.29 s
Peak server-tree RSS	18.19 GiB	18.19 GiB

Two clone attempts succeeded, each reusing 4,306 tokens. Both were in session 7cd243f4, which finished 8.82% faster (630.26 -> 574.64 seconds). The two affected requests' visible TTFTs fell from 87.20/92.60 seconds to 43.66/43.24 seconds. All other sessions had identical prefill counts. Total replay time improved 1.12%; this single paired run supports a narrow prefill benefit, not a general speedup claim. The sample is stratified and not population-weighted. Default remains off.

Test plan

The rebased commit cf4a0cf633 was rebuilt from source, including the core runtime and MLX backend. It includes merged PRs #24302 and #24303.

• Native runner, serving, prefix-cache, and scheduler suites: 460 tests passed, including clone success/failure, cache preference, cancellation, shutdown, and automatic capture sizing.
• MLX cache suite: 141 passed, 2 existing skips; includes flat/SWA clone isolation and retention checks.
• Native Qwen3-0.6B and Muse Glimmer smoke tests on the rebased build: actual live reuse of 1,890/1,891 prompt tokens, with recipient output token IDs identical to fresh prefill.
• python -m pytest examples/llm_server/python/tests/test_serve_launcher.py examples/models/muse-glimmer/tests/test_serve_batching.py examples/models/muse-glimmer/tests/test_serve.py examples/llm_server/python/tests/test_sampling_params.py -q — 268 passed.
• Before rebasing: 18 active-prefix tests repeated 25 times; runner and serving suites also passed under ThreadSanitizer with no reports.
• Changed C++ files pass clang-format --dry-run --Werror; changed Python files pass Ruff E/F checks and formatting checks. Launcher complexity check passes after consolidating range validation.
• Paired AgentX-mini replay on the rebased build: both 20-session / 684-call passes completed. Exact prompts, native output budgets, call coverage, and dependencies passed validation. Both used automatic capture sizing (8 capture slots), root concurrency 1, and 8 inference slots. The same 23 header-only HTTP content diagnostics occurred in each pass; native generation was verified and those calls were excluded from visible-content TTFT.
• CUDA-device validation on an NVIDIA host.
• LLM-server CI, Muse MLX build CI, and lint passed. The remaining NXP job failed fetching an external firmware revision (not our ref 2553ab1d2097f0adb6e0846f054142c86045db3f) before code testing.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24471

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 1 Unrelated Failure

As of commit cf4a0cf with merge base 893d815 ():

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest / macos / macos-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / nxp-mcuxpresso-test / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.