diff --git a/docs/llm/reference.md b/docs/llm/reference.md index bde4cc792c..7da78e8a40 100644 --- a/docs/llm/reference.md +++ b/docs/llm/reference.md @@ -14,20 +14,23 @@ Check out the [quickstart guide](quickstart.md) for a simple example that shows ## Servable Types -Starting with 2025.1, we can highlight four servable types. Such distinction is made based on the input type and underlying GenAI pipeline. +OpenVINO Model Server facilitates multiple servable types for text generation. They reflect input type and underlying GenAI pipeline. The servable types are: - Language Model Continuous Batching, - Language Model Stateful, - Visual Language Model Continuous Batching, -- Visual Language Model Stateful. +- Visual Language Model Stateful, +- Omni Model Stateful -First part - Language Model / Visual Language Model - determines whether servable accepts only text or both text and images on the input. +First part - Language Model / Visual Language Model / Omni Model - determines whether servable accepts only text, both text and images on the input or operates on multiple modalities on both input and output. Second part - Continuous Batching / Stateful - determines what kind of GenAI pipeline is used as the engine. By default CPU and GPU devices work on Continuous Batching pipelines. NPU device works only with the Stateful servable type. +The Omni servable support is currently limited to stateful only. See the [Omni model documentation](../omni.md) for configuration and usage details. + User does not have to explicitly select servable type. It is inferred based on model directory contents and selected target device. -Model directory contents determine if model can work only with text or visual input as well. As for target device, setting it to `NPU` will always pick Stateful servable, while any other device will result in deploying Continuous Batching servable. +Model directory contents determine if model can work only with text or visual input as well. As for target device, setting it to `NPU` will always pick Stateful servable, while any other device will result in deploying Continuous Batching servable. -Stateful servables ignore most of the configuration used by Continuous Batching, but this will be mentioned later. Some servable types have additional limitations mentioned in the limitations section at the end of this document. +Stateful servables ignore most of the configuration used by Continuous Batching. Some servable types have additional limitations mentioned in the limitations section at the end of this document. Despite all the differences, all servable types share the same LLM calculator which imposes certain flow in every GenAI-based endpoint. @@ -286,12 +289,14 @@ __Tool parsers:__ - `qwen3coder` - `lfm2` - `gemma4` +- `minicpm5` __Reasoning parsers:__ - `qwen3` - `gptoss` - `lfm2` - `gemma4` +- `minicpm5` #### Automatic parser detection @@ -399,6 +404,8 @@ Loading chat template proceeds as follows: If both `chat_template.jinja` file and `chat_template` field from `tokenizer_config.json` are successfully loaded, `chat_template.jinja` takes precedence over `tokenizer_config.json`. +After loading the template, the server analyzes its source to automatically detect and apply compatibility adjustments for known template quirks — for example, some templates expect tool call `arguments` as a dict instead of a JSON string, and others use a non-standard field name for reasoning content. When these patterns are detected, the server adapts the request transparently before applying the template. No configuration is required. If a model produces unexpected tool call or reasoning output, checking server startup logs for `chatTemplateCaps` can help diagnose which adjustments were applied. + Template is not applied for calls to `/completions`, so it doesn't have to exist, if you plan to work only with `/completions`. Errors during configuration files processing (access issue, corrupted file, incorrect content) result in servable loading failure. @@ -409,7 +416,9 @@ When working with tools, `/chat/completions` API accepts `tool_choice` parameter - `none` - model server will try to push the model not to call any tool - `required` - model server will try to push the model to call at least one tool -Additionally `tool_choice` can be an object describing specific tool to be called. For more see [API reference](../model_server_rest_api_chat.md#request). +Additionally `tool_choice` can be an object describing specific tool to be called. For more see [API reference](../model_server_rest_api_chat.md#request). + +The `/chat/completions` endpoint also accepts an `add_generation_prompt` parameter (boolean, default `true`). When set to `false`, the chat template is rendered without the trailing assistant generation prompt — the standard technique for assistant prefill, where you supply a partial assistant turn and want the model to continue from it without any additional prompt injection. ## Output processing @@ -433,16 +442,19 @@ Some servable types introduce additional limitations: - sequential request processing (only one request is handled at a time), - only a single response can be returned. Parameter `n` is not supported. - prompt lookup decoding is not supported -- `usage` is not supported in streaming mode - Greedy search and multinomial sampling algorithms are supported - **beam search is not**. - **[NPU only]** models must be exported with INT4 precision and `--sym --ratio 1.0 --group-size -1` params. This is enforced in the export_model.py script when the target_device in NPU. ### Visual Language servable limitations - works only on `/chat/completions` endpoint, -- **[NPU only]** requests MUST include one and only one image in the messages context. Other request will be rejected. +- user-supplied `` placeholder strings in message content are rejected — these are reserved for internal image injection by the server, +- **[NPU only]** requests MUST include one and only one image in the messages context. Other requests will be rejected. ## References - [Chat Completions API](../model_server_rest_api_chat.md) - [Completions API](../model_server_rest_api_completions.md) +- [Responses API](../model_server_rest_api_responses.md) +- [Tokenizer API](../model_server_rest_api_tokenizer.md) - Demos on [CPU/GPU](../../demos/continuous_batching/README.md) and [NPU](../../demos/llm_npu/README.md) - VLM Demos on [CPU/GPU](../../demos/continuous_batching/vlm/README.md) and [NPU](../../demos/vlm_npu/README.md) +- [Omni Demo](../../demos/omni/README.md) diff --git a/docs/parameters.md b/docs/parameters.md index e30ba5cb0c..ac91693aa0 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -17,7 +17,7 @@ | `"model_version_policy"` | `json/string` | Optional. The model version policy lets you decide which versions of a model that the OpenVINO Model Server is to serve. By default, the server serves the latest version. One reason to use this argument is to control the server memory consumption.The accepted format is in json or string. Examples:
`{"latest": { "num_versions":2 }`
`{"specific": { "versions":[1, 3] } }`
`{"all": {} }` | | `"plugin_config"` | `json/string` | List of device plugin parameters. For full list refer to [OpenVINO documentation](https://docs.openvino.ai/2026/documentation/compatibility-and-support/supported-devices.html) and [performance tuning guide](./performance_tuning.md). Example:
`{"PERFORMANCE_HINT": "LATENCY"}` | | `"nireq"` | `integer` | The size of internal request queue. When set to 0 or no value is set value is calculated automatically based on available resources.| -| `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"` | +| `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"`. | | `"metrics_enable"` | `bool` | Flag enabling [metrics](metrics.md) endpoint on rest_port. | | `"metrics_list"` | `string` | Comma separated list of [metrics](metrics.md). If unset, only default metrics will be enabled.| @@ -75,15 +75,15 @@ Configuration options for the config management mode, which is used to manage co ## Configure mode options -Configure mode creates or updates `graph.pbtxt` for a local model without starting the server. It requires `--model_path` and `--task` parameters along with task-specific options. +Configure mode creates or updates `graph.pbtxt` for a local model without starting the server. | Option | Value format | Description | |-------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| | `--configure` | `NA` | Runs in configure mode to create or update `graph.pbtxt` for a local model. Does not start the server. | | `--model_path` | `string` | Path to the local model directory where `graph.pbtxt` will be created. | | `--model_name` | `string` | Optional. Name of the model as exposed by the server. | -| `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). | -| `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. | +| `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). If not specified, automatically inferred from model metadata. See [Automatic task detection](#automatic-task-detection). | +| `--target_device` | `string` | Device name to be used to execute inference operations. For `--task text_generation`, accepted values include `CPU`/`GPU`/`NPU`/`AUTO`, `GPU.`, and `HETERO:<...>`/`AUTO:<...>` (for example `HETERO:GPU,CPU`). If not specified, it is auto-detected using available GPU devices, with fallback to `CPU` (NPU must be selected explicitly). See [Automatic target device detection](#automatic-target-device-detection). | Task-specific options (e.g., `--max_num_seqs`, `--cache_size`, `--num_streams`) are the same as documented in the [pull mode task options](#text-generation) below. @@ -104,8 +104,8 @@ Shared configuration options for the pull, and pull & start mode. In the presenc | `--source_model` | `string` | Name of the model in the Hugging Face repository. If not set, `model_name` is used. | | `--model_repository_path` | `string` | Directory where all required model files will be saved. | | `--model_name` | `string` | Name of the model as exposed externally by the server. | -| `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"` | -| `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). | +| `--target_device` | `string` | Device name to be used to execute inference operations. For `--task text_generation`, accepted values include `CPU`/`GPU`/`NPU`/`AUTO`, `GPU.`, and `HETERO:<...>`/`AUTO:<...>` (for example `AUTO:GPU,CPU`). If not specified, it is auto-detected using available GPU devices, with fallback to `CPU` (NPU must be selected explicitly). See [Automatic target device detection](#automatic-target-device-detection). | +| `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). If not specified, automatically inferred from model metadata. See [Automatic task detection](#automatic-task-detection). | | `--overwrite_models` | `NA` | If set, an existing model with the same name will be overwritten. If not set, the server will use existing model files if available. | | `--gguf_filename` | `string` | Filename of the wanted quantization type from Hugging Face GGUF repository. | @@ -197,4 +197,28 @@ Task specific parameters for different tasks (text generation/image generation/e | `--num_streams` | `integer` | The number of parallel execution streams to use for the model. Use at least 2 on 2 socket CPU systems. Default: 1. | +## Automatic task detection +When `--task` is not provided, the server tries to infer it from model metadata. + +The inference uses metadata files available in the model directory/repository: +- `config.json` (especially the `architectures` field), +- `modules.json` (for sentence-transformers style metadata), +- `model_index.json` (Diffusers pipelines). + +The detection checks task families in priority order to resolve ambiguous architectures: `speech2text` -> `text2speech` -> `rerank` -> `image_generation` -> `embeddings` -> `text_generation`. + +If no detector matches, OVMS cannot infer the task and `--task` must be set explicitly. + +## Automatic target device detection + +The `--target_device` option defaults to auto-detected based on available GPU devices. When not specified or empty, server follows detection logic: + +- If no GPU devices are available, recommends `CPU`. +- Discrete GPUs are preferred over integrated GPUs. +- If a single discrete GPU is found, it is recommended. +- If multiple discrete GPUs are found, the one with the most free VRAM is recommended. +- If no discrete GPUs but integrated GPUs exist, the first integrated GPU is recommended. +- Falls back to `CPU` if no suitable GPU is found. + +> **Note:** Auto-detection does not select `NPU`. To use NPU, set `--target_device NPU` explicitly. \ No newline at end of file