diff --git a/docs-site/content/kagent/supported-providers/azure-ai-foundry.md b/docs-site/content/kagent/supported-providers/azure-ai-foundry.md new file mode 100644 index 00000000..2f8edd63 --- /dev/null +++ b/docs-site/content/kagent/supported-providers/azure-ai-foundry.md @@ -0,0 +1,257 @@ +--- +title: Azure AI Foundry +description: Learn how to configure Azure AI Foundry models in kagent. +weight: 3 +author: kagent.dev +--- + +## Configuring Azure AI Foundry + +> **Note:** Foundry chat models and memory embeddings require the Go agent runtime (`runtime: go`). Chat models must be available through Foundry's OpenAI-compatible chat completions API. Claude and other models are not yet supported. + +The following steps use API key authentication. To authenticate without an API key, see [Workload Identity](#workload-identity). + +1. Create a Kubernetes Secret that contains your Foundry API key. + +```bash +export FOUNDRY_API_KEY="" + +kubectl create secret generic foundry-api-key \ + --namespace kagent \ + --from-literal=api-key="${FOUNDRY_API_KEY}" +``` + +2. Create a `ModelConfig` for your Foundry deployment. + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: foundry-chat + namespace: kagent +spec: + provider: Foundry + model: gpt-4.1-nano + apiKeySecret: foundry-api-key + apiKeySecretKey: api-key + foundry: + endpoint: https://my-foundry-account.cognitiveservices.azure.com/ + deployment: gpt-4-1-nano + apiVersion: "2024-10-21" +``` + +3. Reference the `ModelConfig` from an agent that uses the Go runtime. + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: foundry-agent + namespace: kagent +spec: + type: Declarative + declarative: + runtime: go + modelConfig: foundry-chat + systemMessage: "You are a helpful assistant." +``` + +4. Save the manifests from steps 2 and 3 as `foundry-model.yaml` and `foundry-agent.yaml`, then apply them to your cluster. + +```bash +kubectl apply -f foundry-model.yaml +kubectl apply -f foundry-agent.yaml +``` + +Alternatively, apply only `foundry-model.yaml`, then select the `ModelConfig` from the **Model** dropdown when you create or update an agent in the kagent UI. + +## ModelConfig reference + +| Field | Required | Description | +| --- | --- | --- | +| `spec.provider` | Always | Must be `Foundry`. | +| `spec.model` | Always | Model name reported to the runtime, such as `gpt-4.1-nano`. This can differ from the Azure deployment name. | +| `spec.foundry.endpoint` | Exactly one endpoint field | Account endpoint, such as `https://.cognitiveservices.azure.com/`. | +| `spec.foundry.endpointFrom` | Exactly one endpoint field | Resolve the endpoint from a ConfigMap key. See [Endpoint from a ConfigMap](#endpoint-from-a-configmap). | +| `spec.foundry.deployment` | Always | Foundry model deployment name. | +| `spec.foundry.apiVersion` | Optional | Azure AI Foundry data-plane API version. Defaults to `2024-10-21`. | +| `spec.apiKeySecret` | Optional | Secret that contains the API key. Omit both this field and `apiKeyPassthrough` to use Workload Identity. Mutually exclusive with `apiKeyPassthrough`. | +| `spec.apiKeySecretKey` | With `apiKeySecret` | Key within `apiKeySecret` that contains the API key. | +| `spec.apiKeyPassthrough` | Optional | Let each caller supply its own Foundry API key instead of using a shared Secret. Mutually exclusive with `apiKeySecret`. See [Token passthrough](#token-passthrough). | + +## Authentication + +The runtime chooses a credential based on the fields in the `ModelConfig`. + +| Configuration | Credential | +| --- | --- | +| `apiKeySecret` is set | API key from the referenced Secret. | +| `apiKeyPassthrough: true` | Foundry API key supplied by the caller's A2A request. | +| Neither field is set | Azure Workload Identity. | + +### Workload Identity + +Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime tries to authenticate using your Azure CLI login. + +> **Note:** The runtime validates Azure credentials at startup. If credentials cannot be resolved, the agent does not become ready. + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: foundry-chat + namespace: kagent +spec: + provider: Foundry + model: gpt-4.1-nano + foundry: + endpoint: https://my-foundry-account.cognitiveservices.azure.com/ + deployment: gpt-4-1-nano + apiVersion: "2024-10-21" +``` + +See [Configure the agent for Azure Workload Identity](#configure-the-agent-for-azure-workload-identity) for the required agent settings. + +### Token passthrough + +Set `apiKeyPassthrough: true`, then send the Foundry API key as the bearer token in each Agent2Agent (A2A) request. The runtime forwards that value to Foundry as the API key. Use [Workload Identity](#workload-identity), not token passthrough, for Microsoft Entra ID authentication. + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: foundry-passthrough + namespace: kagent +spec: + provider: Foundry + model: gpt-4.1-nano + apiKeyPassthrough: true + foundry: + endpoint: https://my-foundry-account.cognitiveservices.azure.com/ + deployment: gpt-4-1-nano +``` + +## Endpoint from a ConfigMap + +To use an endpoint from a ConfigMap, set `foundry.endpointFrom` to the ConfigMap name and key. For example, [Azure Service Operator](https://azure.github.io/azure-service-operator/) (ASO) can provision the account and write its endpoint to a ConfigMap. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: foundry-account + namespace: kagent +data: + endpoint: https://my-foundry-account.cognitiveservices.azure.com/ +--- +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: foundry-chat + namespace: kagent +spec: + provider: Foundry + model: gpt-4.1-nano + foundry: + endpointFrom: + name: foundry-account + key: endpoint + deployment: gpt-4-1-nano + apiVersion: "2024-10-21" +``` + +The ConfigMap must be in the same namespace as the `ModelConfig`. + +> **Note:** Updating the endpoint value triggers a rolling update of agent pods that use this `ModelConfig` as their primary model. + +## Configure the agent for Azure Workload Identity + +First, follow the [AKS Workload Identity deployment guide](https://learn.microsoft.com/azure/aks/workload-identity-deploy-cluster) to configure your cluster and managed identity. Grant the identity the `Cognitive Services User` role on the Foundry resource. + +Azure Workload Identity requires the `azure.workload.identity/use: "true"` label on the agent pod and the managed identity client ID on its ServiceAccount. Configure the agent using one of the following options. + +### Let kagent create the ServiceAccount + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: foundry-agent + namespace: kagent +spec: + type: Declarative + declarative: + runtime: go + modelConfig: foundry-chat + systemMessage: "You are a helpful assistant." + deployment: + labels: + azure.workload.identity/use: "true" + serviceAccountConfig: + annotations: + azure.workload.identity/client-id: +``` + +The ServiceAccount has the same name and namespace as the agent. Therefore, use `system:serviceaccount::` as the subject of the Azure federated identity credential. + +> **Note:** If you use Helm to configure a shared ServiceAccount with `controller.agentDeployment.serviceAccountName`, annotate that ServiceAccount and follow [Use an existing ServiceAccount](#use-an-existing-serviceaccount). + +### Use an existing ServiceAccount + +Create or reuse a pre-annotated ServiceAccount, then reference it from the agent. The pod label is still required. + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: foundry-workload-identity + namespace: kagent + annotations: + azure.workload.identity/client-id: +--- +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: foundry-agent + namespace: kagent +spec: + type: Declarative + declarative: + runtime: go + modelConfig: foundry-chat + systemMessage: "You are a helpful assistant." + deployment: + serviceAccountName: foundry-workload-identity + labels: + azure.workload.identity/use: "true" +``` + +Use `system:serviceaccount::` as the subject of the Azure federated identity credential. + +## Memory embeddings + +Configure a memory embedding `ModelConfig` for a Foundry embedding deployment. + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: foundry-embeddings + namespace: kagent +spec: + provider: Foundry + model: text-embedding-3-small + foundry: + endpoint: https://my-foundry-account.cognitiveservices.azure.com/ + deployment: text-embedding-3-small + apiVersion: "2024-10-21" + # No API key: use Azure Workload Identity. +``` + +For the complete memory and embedding configuration and model requirements, see [Agent Memory](/docs/kagent/concepts/agent-memory). + +## Troubleshooting + +- **The `ModelConfig` reports `Accepted=False`:** Check whether the required `endpointFrom` ConfigMap and key exist with `kubectl describe modelconfig MODEL_CONFIG_NAME --namespace NAMESPACE`. +- **The agent fails to become ready with a Workload Identity credential error:** Confirm the pod label and ServiceAccount annotation, and verify that the federated credential subject matches the ServiceAccount used by the pod. +- **Foundry returns `401 Unauthorized` or `403 Forbidden`:** Confirm that the managed identity has the `Cognitive Services User` role on the Foundry resource, or that the configured API key has access to the resource. diff --git a/docs-site/content/kagent/supported-providers/azure-openai.md b/docs-site/content/kagent/supported-providers/azure-openai.md index 5317818e..ff10dc02 100644 --- a/docs-site/content/kagent/supported-providers/azure-openai.md +++ b/docs-site/content/kagent/supported-providers/azure-openai.md @@ -1,41 +1,69 @@ --- title: Azure OpenAI description: Learn how to configure Azure OpenAI models in kagent. -weight: 3 +weight: 3.5 author: kagent.dev --- ## Configuring Azure OpenAI -1. Create a Kubernetes Secret that stores the API key, replace `` with an actual API key: +The following steps use API key authentication. -```shell -export AZURE_OPENAI_API_KEY= -kubectl create secret generic kagent-azureopenai -n kagent --from-literal AZURE_OPENAI_API_KEY=$AZURE_OPENAI_API_KEY +1. Create a Kubernetes Secret that contains your Azure OpenAI API key. + +```bash +export AZURE_OPENAI_API_KEY="" + +kubectl create secret generic azure-openai-api-key \ + --namespace kagent \ + --from-literal=api-key="${AZURE_OPENAI_API_KEY}" ``` -2. Create a ModelConfig resource that references the secret and key name, and specify the additional information that's required for the Azure OpenAI - that's the deployment name, version and the Azure AD token. You can get these values from Azure. +2. Create a `ModelConfig` for your Azure OpenAI deployment. ```yaml apiVersion: kagent.dev/v1alpha2 kind: ModelConfig metadata: - name: azuredefault-model-config + name: azure-openai namespace: kagent spec: - apiKeySecret: kagent-azureopenai - apiKeySecretKey: AZURE_OPENAI_API_KEY + apiKeySecret: azure-openai-api-key + apiKeySecretKey: api-key model: gpt-4o-mini provider: AzureOpenAI azureOpenAI: - azureEndpoint: "https://{yourendpointname}.openai.azure.com/" + azureEndpoint: "https://.openai.azure.com/" apiVersion: "2025-03-01-preview" - azureDeployment: "gpt-4o-mini" - azureAdToken: + azureDeployment: gpt-4o-mini ``` For Azure OpenAI's standard models, kagent automatically configures the appropriate model capabilities. -3. Apply the above resource to the cluster. +3. Save the manifest as `azure-openai.yaml`, then apply it to your cluster. + +```bash +kubectl apply -f azure-openai.yaml +``` + +After you apply the `ModelConfig`, you can select it from the **Model** dropdown when you create or update an agent in the kagent UI. + +## Authentication + +| Configuration | Credential | +| --- | --- | +| `apiKeySecret` is set | API key from the referenced Secret. | +| `apiKeyPassthrough: true` | Azure OpenAI API key supplied as the bearer token in the caller's A2A request. This is not Microsoft Entra ID authentication. | +| Neither field is set | Azure Workload Identity. | + +### Workload Identity + +> **Note:** Azure Workload Identity requires the Go agent runtime (`runtime: go`). + +Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime tries to authenticate using your Azure CLI login. + +> **Note:** The runtime validates Azure credentials at startup. If credentials cannot be resolved, the agent does not become ready. + +Follow the [AKS Workload Identity deployment guide](https://learn.microsoft.com/azure/aks/workload-identity-deploy-cluster) and grant the managed identity the `Cognitive Services User` role. -Once the resource is applied, you can select the model from the Model dropdown in the UI when creating or updating agents. +The kagent pod and ServiceAccount settings are provider-independent. Follow [Configure the agent for Azure Workload Identity](/docs/kagent/supported-providers/azure-ai-foundry#configure-the-agent-for-azure-workload-identity) and set the agent's `modelConfig` to `azure-openai`. diff --git a/docs-site/content/kagent/supported-providers/byo-openai.md b/docs-site/content/kagent/supported-providers/byo-openai.md index 52f52de4..cfa7d381 100644 --- a/docs-site/content/kagent/supported-providers/byo-openai.md +++ b/docs-site/content/kagent/supported-providers/byo-openai.md @@ -51,6 +51,45 @@ You can bring your own model from an [OpenAI API-compatible](https://platform.op Good job! You added a model to kagent. Next, you can [create or update an agent](https://kagent.dev/docs/kagent/getting-started/first-agent) to use this model. +## Self-hosted vLLM behind an OpenAI-compatible gateway + +A common self-hosted pattern places an OpenAI-compatible gateway such as [Bifrost](https://github.com/maximhq/bifrost) or [LiteLLM](https://docs.litellm.ai/) in front of a [vLLM](https://docs.vllm.ai/) server (kagent → gateway → vLLM). Configure the gateway as an OpenAI-compatible provider, exactly as shown above, with two extra things to get right. + +### Enable tool calling in vLLM + +kagent sends a `tools` array with `tool_choice: "auto"` on every request. kagent's runtime registers a built-in `ask_user` tool on every agent, so a `tools` array is always sent, even when you configure no tools yourself. The vLLM backend **must** be launched with automatic tool choice enabled, or every agent turn fails. + +```bash +vllm serve Qwen/Qwen2.5-7B-Instruct \ + --enable-auto-tool-choice \ + --tool-call-parser hermes +``` + +The correct `--tool-call-parser` depends on your model family. For example, Qwen2.5 uses `hermes` and Llama 3.1 uses `llama3_json`. Parser names change across vLLM releases, so check the [vLLM tool calling docs](https://docs.vllm.ai/en/latest/features/tool_calling.html) for your model's current parser name. + +### Use the gateway's model identifier + +Set `spec.model` to the identifier your gateway routes on (often provider-prefixed, such as `vllm/Qwen/Qwen2.5-7B-Instruct`), which can differ from the bare model name vLLM serves internally. Point `openAI.baseUrl` at the gateway (LiteLLM defaults to port `4000`, Bifrost to `8080`). + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: qwen-vllm-via-gateway + namespace: kagent +spec: + apiKeySecret: kagent-my-provider + apiKeySecretKey: ${PROVIDER_API_KEY} + provider: OpenAI + model: vllm/Qwen/Qwen2.5-7B-Instruct + openAI: + baseUrl: http://litellm.kagent.svc.cluster.local:4000/v1 +``` + +### Troubleshooting: `provider API error (status 400)` + +If every agent message fails with a generic `status 400`, the most common cause is that vLLM started without `--enable-auto-tool-choice` and a matching `--tool-call-parser`. Because kagent always sends `tool_choice: "auto"`, vLLM rejects the request until automatic tool choice is enabled. Restart vLLM with the flags above and retry. + ## TLS Configuration To secure communication to LLMs with your own custom certificates, configure the TLS CA details in the `ModelConfig`. Then, your agents communicate with the LLM with those custom certificates. This feature is useful for internal or company-managed LLM servers.