Don't reuse a Metal buffer while a command buffer is still using it#3868
Open
shubhxho wants to merge 1 commit into
Open
Don't reuse a Metal buffer while a command buffer is still using it#3868shubhxho wants to merge 1 commit into
shubhxho wants to merge 1 commit into
Conversation
shubhxho
force-pushed
the
candidate-metal-defer-buffer-recycle
branch
from
July 17, 2026 22:59
0b0fe8d to
9650550
Compare
shubhxho
changed the title
shubhxho
force-pushed
the
candidate-metal-defer-buffer-recycle
branch
from
July 17, 2026 23:01
9650550 to
8903a93
Compare
Under continuous batching with async_eval, small int32 KV-cache bookkeeping arrays (offset/left_padding) sometimes come back as garbage that looks like bf16 bits, e.g. 16256 (0x3F80) in an int32 slot -- a buffer reused while a kernel is still writing it. free() recycles buffers without knowing which command buffer last used them, which is fine in order on one stream but not once async_eval adds a second. Fix: count in-flight command buffers per buffer (marked at end_encoding, retired from the completion handler). free() holds a still-in-use buffer aside and recycles it once the work is done.
shubhxho
force-pushed
the
candidate-metal-defer-buffer-recycle
branch
from
July 17, 2026 23:05
8903a93 to
e5a86d1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under continuous batching with
async_eval, the small int32 KV-cache bookkeeping arrays (offset/left_padding) sometimes come back as garbage that looks like bf16 bits in int32 slots -- e.g.16256(0x3F80, bf161.0). A buffer is being reused while a kernel is still writing it. Full write-up and repro in the linked issue; likely related to #3346.MetalAllocator::freereturns a buffer to the reuse pool without knowing which command buffer last used it. Fine on one stream (in order), butasync_evaladds a second stream, so a buffer freed on one can be reused on the other before the first kernel finishes.Fix: count how many in-flight command buffers reference each buffer -- marked at
end_encoding, retired from the completion handler.free()holds a still-in-use buffer aside and recycles it once the work is done.