Skip to content

Free llama.cpp models before the process exits - #422

Merged
mattt merged 6 commits into
mainfrom
mattt/llama-exit-logging
Nov 18, 2026
Merged

mattt merged 6 commits into
mainfrom
mattt/llama-exit-logging

Conversation

@mattt

@mattt mattt commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

This PR works around a crash at exit in llama.cpp's Metal backend. A process that exits while a model has layers on the GPU aborts on GGML_ASSERT([rsets->data count] == 0), and an exit in the middle of a generation crashes with or without the GPU.

LlamaLanguageModel now registers an exit handler when a load finishes. The handler stops new loads and generations, waits up to three seconds for running work to stop at its next token, and then frees every loaded model. It's best effort: if work is still running after three seconds, or the exit comes from inside a load or a generation, it frees nothing and the process can still abort. If freeing the models takes more than ten seconds, the handler aborts the process itself, so an exit can't hang. Requests that start after the handler runs throw GenerationError.assetsUnavailable. The README lists the cases the handler can't cover.

The parent measures a child's time until it sees the child end, and a busy CI runner delayed that by about seven seconds, so upper bounds failed although every child ended as expected. Exact exit statuses, the lower bounds, the parent's kill timer, and the lifecycle unit tests still catch a broken handler or watchdog.
@mattt
mattt merged commit fd41342 into main Nov 18, 2026
14 checks passed
@mattt
mattt deleted the mattt/llama-exit-logging branch November 18, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant