feat(qwen4-exp): mmap PLE tables from disc

[ryseek]

closes #79 offload PLE to Disk

Adds --ple-backend mmap, demand-paging Qwen3.8-Flash-Next’s 47.7 GiB PLE table from NVMe

Usage:

set ple-backend pinned (default) or mmap

ft serve \
  --model RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --ple-backend mmap \
  --expert-load serial \
  --attention-backend qsa_sparse \
  --page-size 64 \
  --nvfp4-backend triton \
  --max-running-requests 1 \
  --max-seq-len-override 250000 \
  --kv-reserve-tokens 250000 \
  --max-prefill-length 8192 \
  --memory-ratio 0.85 \
  --enable-cache-report

Performance tests:

Fits into 128GB of total RAM + VRAM space with 250K FP16 context.

Tests under RTX 5090, 96 GB RAM:

Context	PP (tok/s)	TG (tok/s)
0K	—	44.75
32K	3,045	37.90
128K	2,981	39.02

looks sweet, but appreciate tests with pinned PLE to avoid negative regression, don't have enough RAM for that

[dejay2]

Confirmed: PR #279's SSD-backed PLE mmap path works for RadixArk/Qwen3.8-Flash-Next-NVFP4 on native Windows with a 32 GB RTX 5090.

Tested setup

• Windows 11 Pro build 26200
• RTX 5090 (32,607 MiB), driver 610.62
• Ryzen 9 9950X3D, 95.6 GiB system RAM
• Samsung 990 PRO 4 TB NVMe
• CUDA Toolkit 13.1.115
• PR commit feaeaa31c0cea385a1c9ee107d4b1053f83b35db
• 125.96 GiB checkpoint, including a 47.7 GiB PLE/n-gram lookup table mmap-backed from SSD
• Text input and one active request (--max-running-requests 1)

All 192 NVFP4 expert parts loaded, --ple-backend mmap retained decode CUDA graphs, and the OpenAI-compatible API passed sequential text, streamed reasoning/content/usage, parsed tool-call, and final-health checks.

Balanced measurement

Fixed seed 3805090, reasoning off, temperature 0, top-k 1, ignored EOS, one warm-up, then 3× short and 3× 8K requests plus one 32K context check:

KV setting	Usable context	GPU-cached experts	Startup	Short decode	8K decode	32K-check decode
50K pool	49,984	6,002	95.912 s	58.7 tok/s	63.5 tok/s	59.2 tok/s
Full pool	262,144	4,063	96.044 s	54.5 tok/s	59.1 tok/s	55.1 tok/s

The 50K pool has 50,048 total KV tokens and the full pool has 262,208; FreeToken reserves one 64-token dummy page in each. Full context was about 6.8–7.2% slower in decode because fewer experts fit in the GPU cache.

Across the measured cases, peak whole-system RAM was 91.2 GiB, peak whole-GPU VRAM was 31.1 GiB, and whole-D-drive read samples reached 3.28 GB/s. These are device/system counters rather than per-process counters.

Reproducible launcher, method, limitations, troubleshooting, and request-level JSON:

• Windows guide
• Raw benchmark data
• Fork branch

Important Windows delivery limitation: this is not a standalone Windows CLI package. The source checkout reuses the Python runtime, DLLs, CUDA sources, and libraries installed by FreeToken Desktop. Desktop itself does not need to be open or modified, but this server cannot be selected or configured in the Desktop UI. I am therefore not opening a follow-up upstream PR for the Windows bridge at this stage; the fork is an unofficial reproducibility path for users with similar hardware and Desktop already installed.

The compatibility bridge also credits and adapts relevant prior Windows approaches from closed PR #232 by MaxKerkula and contributors.

[KeithVal]

Tested on the exact configuration this PR targets — a 128 GB box that cannot run the pinned-PLE path. It works.

Machine: Ubuntu 24.04 bare metal (no WSL2), 128 GB DDR4 (125.7 GiB visible), RTX 5070 Ti 16 GB, RadixArk/Qwen3.8-Flash-Next-NVFP4, this branch rebased on 58f4b9e.

Before/after on the same machine — this is the pinned-vs-mmap comparison you asked for:

	--ple-backend pinned (main, #257)	--ple-backend mmap (this PR)
Result	OOM-killed at expert bank 191/192, in all 3 configs (parallel loader / --expert-load serial / serial + cgroup MemoryHigh reclaim + 24 G swap)	Started in 95 s
Min MemAvailable	7.6 GiB, then killed	53.7 GiB
RSS at steady state	never reached it	70 GiB
Decode	—	18.69 tok/s (avg of 3 × 160 tok, ja)
VRAM	—	15.6 / 16.3 GB

So the mmap path removes ~48 GiB of host residency, which lines up with the 47.68 GiB PLE table almost exactly. My earlier report in #214 said this checkpoint was simply out of reach at 125.7 GiB — this PR makes that report obsolete, which is the best kind of obsolete.

One practical note for 16 GB cards. My first two attempts failed fast (~76 s) with:

cache budget too small: minimum plan (moe=1024 slots, kv=128 pages)
needs 3046637568 B > budget 2205851166 B

That is a VRAM-budget failure, not a RAM one, and it happens before any of the mmap work is exercised — easy to misread as "the PR doesn't help me". Freeing the GPU (I had a ComfyUI instance holding ~560 MB) and raising --memory-ratio from 0.85 to 0.95 was enough. Working invocation:

ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --ple-backend mmap --expert-load serial \
  --attention-backend qsa_sparse --nvfp4-backend triton \
  --moe-backend offload --page-size 64 --max-running-requests 1 \
  --memory-ratio 0.95 --max-seq-len-override 4096 --kv-reserve-tokens 2048

Might be worth a line in the PR description that the minimum plan needs ~3 GB of cache budget on top of the weights, since 16 GB is exactly the class of card this feature rescues.

The 18.7 vs your 44.75 tok/s gap tracks VRAM (16 GB vs 32 GB) rather than the PLE path — on this box each ~3 GB of experts moved onto the GPU is worth roughly +1 tok/s, so the residency difference alone accounts for it. PLE lives on ext4 NVMe here. Happy to run further variants (larger contexts, --moe-cache-* sweeps, cold vs warm page cache) if any of it would help the review.

[jason-fxz]

Opened #311 — same idea, different implementation. Instead of mmap page faults it reads rows straight from the checkpoint shards with O_DIRECT + io_uring (one batched round per step), uses no cache (fixed ~21 MiB staging), and on decode the graph launches first and waits on a stream flag, so the disk round trip hides behind embedding + layer 0.