unload idel model

[nefingar]

I might have missed something but is it possible to unload a model if it wasn't used for X minutes? Ollama has something like that, freeing up vram for other things (image generation etc.) if the llm isn't in use currently.

[nefingar]

I forgot to add, my question is about the llama-server that you can spin up to create an openapi compatible endpoint.

[kamalesh404]

Short answer: llama-server (the OpenAI-compatible endpoint that ships with llama-cpp-python) currently has no built-in idle timer like Ollama's keep_alive — the model stays loaded in the server process until you stop the process. Stopping the process immediately frees the VRAM, since all weights/KV cache live in that process's memory.

Your options, depending on your setup:

1. If you use Ollama (you mentioned it)
Ollama unloads idle models automatically. Tune it with:

# global default (default is 5m)
OLLAMA_KEEP_ALIVE=10m ollama serve

or per request:

{ "model": "qwen2.5-coder:7b", "prompt": "hi", "keep_alive": "10m" }

Setting "keep_alive": 0 (or OLLAMA_KEEP_ALIVE=0) unloads the model as soon as the request finishes. ollama stop <model> unloads it manually.

2. If you use llama-server from llama-cpp-python
There is no --idle-timeout flag, so the standard pattern is start-on-demand:

# start it only when you need it, Ctrl-C / pkill when done — VRAM is freed instantly
python -m llama_cpp.server --model model.gguf --n_gpu_layers 8

If you want it automatic, wrap it: a small shell script or a systemd service with RuntimeMaxSec= / a cron that kills the server process after your image-generation job starts. Killing the process is the unload — nothing lingers on the GPU.

3. If you use llama-cpp-python as a library

model = Llama("model.gguf", n_gpu_layers=8)
# ... use it ...
del model
import gc; gc.collect()  # weights/KV cache released here

So: Ollama = set keep_alive; llama-server/library = manage the process lifetime yourself. For your use case (LLM idle while image generation runs), I'd run the LLM server on demand rather than 24/7 on a 4GB card — I do exactly this on an RTX 2050 and it works well.

[gyanu2507]

No idle TTL like Ollama's keep_alive.

The Python Llama object holds the weights until you drop it. There is no --idle-minutes on llama_cpp.server either.

What people do: wrap the instance, reset a timer on each request, and on expiry del llm, gc.collect(). If you used CUDA, that is usually enough to give the VRAM back. Calling llama_backend_free() is process-wide and will break other models in the same process.

If you need Ollama-style unload, run llama.cpp's llama-server (router mode has model TTL on recent builds) and point clients at that, or put a tiny proxy in front that spawns/kills the process. llama-cpp-python will not do it for you.