Runtime_flags are ignored on Linux in certain cases

[janick-nl]

When running DMR on a Linux machine using runtime_flags, they are always ignored after a reboot. It also seems to revert to the default config unexpectedly when docker model config is manually run. Recreating the containers will apply them again. Any idea why this is happening and how to handle this?

Docker compose:

models:
 llm:
   model: ai/qwen3-vl:8B
   context_size: 16384
   runtime_flags:
     - "-c" 
     - "16384"
     - "-np"
     - "4"
     - "-b"  
     - "2048"
     - "--no-mmap" 
     - "--flash-attn" 
     - "on"
     - "--cache-type-k"
     - "q8_0"
     - "--cache-type-v" 
     - "q8_0"
 instant:
   model: ai/qwen3-vl:2B-UD-Q4_K_XL
   context_size: 8192
   runtime_flags:
     - "-c" 
     - "8192"
     - "-np"
     - "4"
     - "-b"  
     - "2048"
     - "--no-mmap" 
     - "--flash-attn" 
     - "on"
     - "--cache-type-k" 
     - "q8_0"
     - "--cache-type-v" 
     - "q8_0"

After reboot and executing the first request (see last line, no runtime_flags):

time="2026-03-03T10:13:24Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Listing available models" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Successfully listed models, count: 2" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Loading llama.cpp backend runner with model sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 in completion mode"
time="2026-03-03T10:13:24Z" level=info msg="Getting model by reference: sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Listing available models" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Successfully listed models, count: 2" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Listing available models" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="Successfully listed models, count: 2" component=model-manager
time="2026-03-03T10:13:24Z" level=info msg="llama.cpp args: [-ngl 999 --metrics --model /models/bundles/sha256/a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5/model/model.gguf --host inference-runner-0.sock --mmproj /models/bundles/sha256/a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5/model/model.mmproj]"

After a --force-recreate of the containers (see last line, runtime_flags applied):

time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Configuring llama.cpp runner for sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d"
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Configuring llama.cpp runner for sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5"
time="2026-03-03T10:19:40Z" level=info msg="Loading llama.cpp backend runner with model sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d in completion mode"
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Listing available models" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Successfully listed models, count: 2" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Listing available models" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Successfully listed models, count: 2" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:19:40Z" level=info msg="Loading llama.cpp backend runner with model sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 in completion mode"
time="2026-03-03T10:19:40Z" level=info msg="llama.cpp args: [-ngl 999 --metrics --model /models/bundles/sha256/50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d/model/model.gguf --host inference-runner-0.sock --ctx-size 8192 -c 8192 -np 4 -b 2048 --no-mmap --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --mmproj /models/bundles/sha256/50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d/model/model.mmproj]"

I've also tried using cronjobs to keep the models loaded with the config, but after a reboot it also ignores them and reverts to the default context size of 4096, causing issues with larger requests:

*/5 * * * * docker model configure --context-size 16384 ai/qwen3-vl:8B -- -c 16384 -np 4 -b 2048 --no-mmap --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 && docker model run --detach ai/qwen3-vl:8B 2>&1
*/5 * * * * docker model configure --context-size 16384 ai/qwen3-vl:2B-UD-Q4_K_XL -- -c 8192 -np 4 -b 2048 --no-mmap --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 && docker model run --detach ai/qwen3-vl:2B-UD-Q4_K_XL 2>&1

It also seems to reset to 4096 tokens when the cronjob is run later on, at least, that's how I interpret the n_ctx_slot = 4096:

time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Loading llama.cpp backend runner with model sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 in completion mode"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="srv  params_from_: Chat format: Hermes 2 Pro" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 0.583 (> 0.100 thold), f_keep = 0.003" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist " component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot launch_slot_: id  3 | task 4254 | processing task, is_child = 0" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot update_slots: id  3 | task 4254 | new prompt, n_ctx_slot = 4096, n_keep = 0, task.n_tokens = 12" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot update_slots: id  3 | task 4254 | n_tokens = 7, memory_seq_rm [7, end)" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot update_slots: id  3 | task 4254 | prompt processing progress, n_tokens = 12, batch.n_tokens = 5, progress = 1.000000" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot update_slots: id  3 | task 4254 | prompt done, n_tokens = 12, batch.n_tokens = 5" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="slot init_sampler: id  3 | task 4254 | init sampler, took 0.00 ms, tokens: text = 12, total = 12" component=llama.cpp
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Configuration for llama.cpp runner for modelID sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d unchanged"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Loading llama.cpp backend runner with model sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d in completion mode"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Configuration for llama.cpp runner for modelID sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 unchanged"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Loading llama.cpp backend runner with model sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 in completion mode"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:2B-UD-Q4_K_XL" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Loading llama.cpp backend runner with model sha256:50f70f7f0ca537b2ca4843bae5456bb4b6f9d9d58d4b357faf1a3ad9b574888d in completion mode"
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Getting model by reference: ai/qwen3-vl:8B" component=model-manager
time="2026-03-03T10:40:01Z" level=info msg="Loading llama.cpp backend runner with model sha256:a18971a77b8fda79c555f603c4f94ca0183cb40499f336dc2825659504e29fc5 in completion mode"

Versions:

$ docker version
Client: Docker Engine - Community
 Version:           29.2.1
 API version:       1.53
 Go version:        go1.25.6
 Git commit:        a5c7197
 Built:             Mon Feb  2 17:17:26 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.2.1
  API version:      1.53 (minimum version 1.44)
  Go version:       go1.25.6
  Git commit:       6bc6209
  Built:            Mon Feb  2 17:17:26 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker model version
Client:
 Version:    v1.1.5
 OS/Arch:    linux/amd64

Server:
 Version:    v1.1.0
 Engine:     Docker Engine

[doringeman]

Hey @janick-nl, we have a 5 minutes default timeout for unloading the model (ref.), which also removes the configuration. I believe it's not expected to have the model configuration removed when the model is simply unloaded.

To quickly fix your setup, you can use --keep-alive with docker model configure.

$ docker model configure --help | grep keep-alive
      --keep-alive string                       duration to keep model loaded (e.g., '5m', '1h', '0' to unload immediately, '-1' to never unload)

Moreover, I believe runtime flags should also be configurable via docker model package, just like context size. CC @ilopezluna

[janick-nl]

HI @doringeman , thanks for the quick reply. Does docker model configure work in conjunction with docker compose? And should it keep the context size configured as well? Because when using --keep-alive -1 after a >5 minutes it seems to reset the context size to 4096 again, though the models are still loaded.

For reference, these are the cronjobs now (removed the docker model run --detach as well, since I expected --keep-alive would make it redundant):

*/5 * * * * docker model configure --context-size 16384 --keep-alive -1 ai/qwen3-vl:8B -- -c 16384 -np 4 -b 2048 --no-mmap --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 2>&1
*/5 * * * * docker model configure --context-size 8192  --keep-alive -1 ai/qwen3-vl:2B-UD-Q4_K_XL -- -c 8192 -np 4 -b 2048 --no-mmap --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 2>&1

Still loaded:

$ nvidia-smi
Wed Mar  4 12:21:20 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   33C    P0             25W /   70W |   10043MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            4384      C   /app/bin/com.docker.llama-server       2732MiB |
|    0   N/A  N/A            4575      C   /app/bin/com.docker.llama-server       7308MiB |
+-----------------------------------------------------------------------------------------+

[doringeman]

Does docker model configure work in conjunction with docker compose? And should it keep the context size configured as well? Because when using --keep-alive -1 after a >5 minutes it seems to reset the context size to 4096 again, though the models are still loaded.

docker compose calls docker model configure, and docker model configure overwrites, doesn't append the new configuration. So, if you call docker model configure --keep-alive, and then run docker compose up which calls docker model configure --context-size, the result will be the model with the context size configure only.

I agree this is not a great UX, but we're looking into adding these configurations part of docker model package.
WDYT @ilopezluna, can we add them in docker model package?

---

FWIW this is how you can see the current configurations for your models:

$ docker model requests | jq '[.[] | {model, modelName: .records[0].model, config}]'
[
  {
    "model": "sha256:b0c5e2f704cbf9aa0824e2b6e661fa809b799399a563604dbadb07dc48c94c96",
    "modelName": "gpt-oss:32k",
    "config": {
      "context-size": 32000
    }
  },
  {
    "model": "sha256:354bf30d0aa3af413d2aa5ae4f23c66d78980072d1e07a5b0d776e9606a2f0b9",
    "modelName": "smollm2",
    "config": {
      "context-size": 1024
    }
  }
]

gpt-oss:32k's context size has been set to 32k using docker model package --from. To see this config, you can run

$ docker model inspect gpt-oss:32k | jq '{context_size: .config.context_size}'
{
  "context_size": 32000
}

smollm2's context size has been set using docker model configure. To see these configs, you can run

$ docker model configure show
[
  {
    "Backend": "llama.cpp",
    "Model": "docker.io/ai/smollm2:latest",
    "ModelID": "sha256:354bf30d0aa3af413d2aa5ae4f23c66d78980072d1e07a5b0d776e9606a2f0b9",
    "Mode": "completion",
    "Config": {
      "context-size": 1024
    }
  }
]

[janick-nl]

docker compose calls docker model configure, and docker model configure overwrites, doesn't append the new configuration. So, if you call docker model configure --keep-alive, and then run docker compose up which calls docker model configure --context-size, the result will be the model with the context size configure only.

Thanks for clarifying, it makes sense now. Would you also consider a --keep-alive in docker composes runtime_flags? The only reason I created the cronjobs was for the models to stay loaded with the given flags. I'd rather have everything in a docker compose file.

[doringeman]

Yes, we do consider it! I'll let you know when we have something you can test, which is going to be very soon! Thanks again!