metal: restore streamed-expert mlock pinning (fixes #532)#578
Open
gilbert-barajas wants to merge 1 commit into
Open
metal: restore streamed-expert mlock pinning (fixes #532)#578gilbert-barajas wants to merge 1 commit into
gilbert-barajas wants to merge 1 commit into
Conversation
…efactor The "Add GLM 5.2 runtime support" refactor removed all mlock pinning of streamed experts -- the mlock calls, slab_lock_slot/unlock_slot, and the budget cap all went to zero. Without pinning, in the SSD-streaming regime (model >> RAM) the OS evicts the expert cache under memory pressure and every token re-streams from disk. That is the DeepSeek-Flash decode regression reported in antirez#532 (~2-3x slower, ~6s first-token). Port the pinning back, adapted to the refactored slab allocator (slab_backed/slab_slot, 3 gate/up/down buffers per entry): whole-buffer mlock at alloc, per-slot lock on fill / unlock under relief, the routed-memory budget cap for graceful degradation, cap-on-failure, and a relief routine that unpins the coldest ~10% of victims under pressure. Bench (M5 Max 128GB, DeepSeek-V4-Flash IQ2XXS, --simulate-used-memory 72 to force the streaming regime): before (broken): 0.3-2.5 gen t/s, ~6.0s first-token after (this): ~7.0 gen t/s, ~1.7s first-token baseline 80ebbc3: ~6.1 gen t/s Recovers past baseline (the 427e281 kernel work pays off once experts stay resident). Fixes antirez#532
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Add GLM 5.2 runtime support" refactor dropped all
mlockpinning of streamed experts — themlockcalls,slab_lock_slot/unlock_slot, and the budget cap all went to zero. Without pinning, in the streaming regime (model ≫ RAM) the OS evicts the expert cache under memory pressure and every token re-streams from disk. That's the DeepSeek-Flash decode regression in #532 (~2–3× slower, ~6s first-token).This ports the pinning back, adapted to the refactored slab allocator (
slab_backed/slab_slot, 3 gate/up/down buffers per entry): whole-buffermlockat alloc, per-slot lock on fill / unlock under relief, the routed-memory budget cap for graceful degradation, cap-on-failure, and a relief routine that unpins the coldest ~10% of victims under pressure. +382/−1, builds warning-free.Bench (M5 Max 128 GB, DeepSeek-V4-Flash IQ2XXS,
--simulate-used-memory 72to force the streaming regime):80ebbc3Recovers past baseline — the
427e281kernel work pays off once experts stay resident.Fixes #532