Skip to content

metal: restore streamed-expert mlock pinning (fixes #532)#578

Open
gilbert-barajas wants to merge 1 commit into
antirez:mainfrom
gilbert-barajas:fix/streaming-expert-mlock-pinning
Open

metal: restore streamed-expert mlock pinning (fixes #532)#578
gilbert-barajas wants to merge 1 commit into
antirez:mainfrom
gilbert-barajas:fix/streaming-expert-mlock-pinning

Conversation

@gilbert-barajas

Copy link
Copy Markdown

The "Add GLM 5.2 runtime support" refactor dropped all mlock pinning of streamed experts — the mlock calls, slab_lock_slot/unlock_slot, and the budget cap all went to zero. Without pinning, in the streaming regime (model ≫ RAM) the OS evicts the expert cache under memory pressure and every token re-streams from disk. That's the DeepSeek-Flash decode regression in #532 (~2–3× slower, ~6s first-token).

This ports the pinning back, adapted to the refactored slab allocator (slab_backed/slab_slot, 3 gate/up/down buffers per entry): whole-buffer mlock at alloc, per-slot lock on fill / unlock under relief, the routed-memory budget cap for graceful degradation, cap-on-failure, and a relief routine that unpins the coldest ~10% of victims under pressure. +382/−1, builds warning-free.

Bench (M5 Max 128 GB, DeepSeek-V4-Flash IQ2XXS, --simulate-used-memory 72 to force the streaming regime):

build gen t/s first-token
before (broken) 0.3–2.5 ~6.0 s
after (this PR) ~7.0 ~1.7 s
baseline 80ebbc3 ~6.1 fast

Recovers past baseline — the 427e281 kernel work pays off once experts stay resident.

Fixes #532

…efactor

The "Add GLM 5.2 runtime support" refactor removed all mlock pinning of
streamed experts -- the mlock calls, slab_lock_slot/unlock_slot, and the
budget cap all went to zero. Without pinning, in the SSD-streaming regime
(model >> RAM) the OS evicts the expert cache under memory pressure and
every token re-streams from disk. That is the DeepSeek-Flash decode
regression reported in antirez#532 (~2-3x slower, ~6s first-token).

Port the pinning back, adapted to the refactored slab allocator
(slab_backed/slab_slot, 3 gate/up/down buffers per entry): whole-buffer
mlock at alloc, per-slot lock on fill / unlock under relief, the
routed-memory budget cap for graceful degradation, cap-on-failure, and a
relief routine that unpins the coldest ~10% of victims under pressure.

Bench (M5 Max 128GB, DeepSeek-V4-Flash IQ2XXS, --simulate-used-memory 72
to force the streaming regime):
  before (broken): 0.3-2.5 gen t/s, ~6.0s first-token
  after  (this):   ~7.0 gen t/s,    ~1.7s first-token
  baseline 80ebbc3: ~6.1 gen t/s
Recovers past baseline (the 427e281 kernel work pays off once experts
stay resident).

Fixes antirez#532
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

glm5.2 branch: DeepSeek Flash decode regression (~2.8x) introduced by "Add GLM 5.2 runtime support"

1 participant