Update docs - #4432
Open
mzegla wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates documentation for OpenVINO Model Server’s generative/model configuration, expanding guidance around automatic task/device detection and extending LLM reference docs with Omni-related and chat-template behavior details.
Changes:
- Clarifies/extends CLI documentation for
--taskand--target_device, and adds new sections describing automatic task detection and automatic target device detection. - Updates LLM reference documentation to include Omni servable type notes, additional parser names, and extra
/chat/completionsbehavior details. - Minor formatting/wording adjustments across the updated docs.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/parameters.md | Adds/updates CLI parameter descriptions and new sections for automatic task/device detection. |
| docs/llm/reference.md | Updates LLM reference to include Omni servable, parser lists, and additional chat-template/tooling notes. |
Suppressed comments (1)
docs/parameters.md:107
- This
--target_devicerow listsMULTIand omitsNPU/AUTO/GPU.<index>, but the server’s generative text generation flow validatestarget_deviceagainstCPU,GPU,NPU,AUTO(plusGPU.<index>andAUTO:/HETERO:prefixes). As written, the docs suggest some unsupported values and hide supported ones.
| `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"`. If not specified, auto-detected based on available devices. |
| | `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). | | ||
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. | | ||
| | `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). If not specified, automatically inferred from model metadata. | | ||
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. If not specified, auto-detected based on available devices. | |
przepeck
reviewed
Aug 5, 2026
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"` | | ||
| | `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). | | ||
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"`. If not specified, auto-detected based on available devices. | | ||
| | `--task` | `string` | Task type the model will support (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). If not specified, automatically inferred from model metadata. | |
Collaborator
There was a problem hiding this comment.
maybe there should be links to Automatic task detection and Automatic target device detection sections
dtrawins
reviewed
Aug 5, 2026
|
|
||
| ### Visual Language servable limitations | ||
| - works only on `/chat/completions` endpoint, | ||
| - user-supplied `<ov_genai_image_N>` placeholder strings in message content are rejected — these are reserved for internal image injection by the server, |
dtrawins
reviewed
Aug 5, 2026
| - **[NPU only]** requests MUST include one and only one image in the messages context. Other request will be rejected. | ||
|
|
||
| ## References | ||
| - [Chat Completions API](../model_server_rest_api_chat.md) |
Collaborator
There was a problem hiding this comment.
also reference to responses and tokenize?
dtrawins
reviewed
Aug 5, 2026
| | `"plugin_config"` | `json/string` | List of device plugin parameters. For full list refer to [OpenVINO documentation](https://docs.openvino.ai/2026/documentation/compatibility-and-support/supported-devices.html) and [performance tuning guide](./performance_tuning.md). Example: <br> `{"PERFORMANCE_HINT": "LATENCY"}` | | ||
| | `"nireq"` | `integer` | The size of internal request queue. When set to 0 or no value is set value is calculated automatically based on available resources.| | ||
| | `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"` | | ||
| | `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"`. | |
Collaborator
There was a problem hiding this comment.
Suggested change
| | `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"MULTI"/"HETERO"`. | | |
| | `"target_device"` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"HETERO"/"`. By default server selects the device with this priority: dGPU if present, iGPU if present, CPU. If several discrete GPUs are present, the one with most free VRAM will be selected. | |
dtrawins
reviewed
Aug 5, 2026
| | `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). | | ||
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. | | ||
| | `--task` | `string` | Task type for the model (`text_generation`, `embeddings`, `rerank`, `image_generation`, `text2speech`, `speech2text`). If not specified, automatically inferred from model metadata. | | ||
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. If not specified, auto-detected based on available devices. | |
Collaborator
There was a problem hiding this comment.
Suggested change
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"MULTI"/"HETERO"`. If not specified, auto-detected based on available devices. | | |
| | `--target_device` | `string` | Device name to be used to execute inference operations. Accepted values are: `"CPU"/"GPU"/"NPU"/"HETERO"`. If not specified, auto-detected based on available devices. | |
dtrawins
reviewed
Aug 5, 2026
| ## Automatic task detection | ||
|
|
||
| When `--task` is not provided, the server automatically infers the generative task from model metadata. Detection is performed by a chain of detectors with the following priority: | ||
|
|
Collaborator
There was a problem hiding this comment.
I think this logic is to complex to cover it in documentation. The code has this logic. Users don't need to care about those nuances in my opinion.
przepeck
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.