diff --git a/content/ngf/how-to/f5-ai-guardrails.md b/content/ngf/how-to/f5-ai-guardrails.md new file mode 100644 index 000000000..078cb3122 --- /dev/null +++ b/content/ngf/how-to/f5-ai-guardrails.md @@ -0,0 +1,425 @@ +--- +title: Secure LLM traffic with F5 AI Guardrails +description: Deploy F5 AI Guardrails with NGINX Gateway Fabric using PayloadProcessor to inspect and block LLM traffic +weight: 900 +toc: true +f5-content-type: how-to +f5-product: F5 NGINX Gateway Fabric +f5-keywords: NGINX Gateway Fabric, F5 AI Guardrails, AI Guardrails, PayloadProcessor, LLM, large language model, Gateway API, Kubernetes, content policy, PII, ai-guardrails module, guardrails +f5-summary: > + Deploy a large language model (LLM) behind NGINX Gateway Fabric, attach a PayloadProcessor policy + that routes request and response payloads through an external Guardrails API, and verify that + disallowed content is blocked before it reaches the model or the client. +--- + +Learn how to use NGINX Gateway Fabric with F5 AI Guardrails to inspect large language model (LLM) traffic and block disallowed content before it reaches the model or the client. + +## Overview + +F5 AI Guardrails can inspect LLM traffic on two independent paths: + +- **Prompts** — the client's *input* is inspected before it reaches the LLM. A block returns `403` with `error.type: invalid_request_error`. +- **Responses** — the model's *output* is inspected before it reaches the client. A block returns `403` with `error.type: api_error`. + +You can set up prompts and responses as described in [Prompts and scans in AI Security](https://docs.aisecurity.f5.com/api-docs/prompts-scans.html), through the F5 AI Guardrails dashboard. + +To connect NGINX Gateway Fabric with your configured F5 AI Guardrails, use the `PayloadProcessor` policy, an [inherited policy]({{< ref "/ngf/overview/custom-policies.md" >}}) that can target an HTTPRoute or a Gateway. The `PayloadProcessor` configures NGINX to offload traffic to F5 AI Guardrails to inspect. + +## Before you begin + +You need an F5 AI Guardrails API endpoint to inspect payloads. This can be an F5 hosted service or a service running inside your cluster. View the official [F5 AI Guardrails](https://docs.aisecurity.f5.com/) docs to learn more. + +## Deploy an LLM backend + +If you have an existing in-cluster LLM which can be queried you can skip this section. + +The following example uses the [vLLM simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main), which serves canned responses from a dataset rather than running a real model, making it suitable for test and development environments. The simulator loads its dataset from a ConfigMap. Download the dataset file, then create the ConfigMap from it: + +```shell +curl -sL -o inference-sim-dataset.sqlite3 \ + https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v{{< version-ngf >}}/examples/guardrails/inference-sim-dataset.sqlite3 + +kubectl create configmap inference-sim-dataset \ + --from-file=inference-sim-dataset.sqlite3=./inference-sim-dataset.sqlite3 +``` + +{{< call-out "note" >}} +The dataset file, alongside more details of the setup, can be found in the [`examples/guardrails`](https://github.com/nginx/nginx-gateway-fabric/tree/v{{< version-ngf >}}/examples/guardrails) directory of the NGINX Gateway Fabric repository. +{{< /call-out >}} + +Deploy the LLM Deployment and Service: + +```shell +kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v{{< version-ngf >}}/examples/guardrails/llm.yaml +``` + +Confirm the Pod is `Ready`: + +```shell +kubectl get deployment vllm-qwen3-32b +``` + +```text +NAME READY UP-TO-DATE AVAILABLE AGE +vllm-qwen3-32b 1/1 1 1 6m13s +``` + +## Create the authentication token Secret + +Create the Secret with your Guardrails API token under the `token` key. The Secret must live in the same namespace as the `PayloadProcessor`: + +```yaml +kubectl apply -f - <" +EOF +``` + +## Configure the Guardrails backend Service + +The Guardrails backend can live outside or inside the cluster. NGINX Gateway Fabric picks the URL scheme from the referenced Service's type: + +| Backend location | Service type | Resolved URL | +| ---------------- | ------------ | ------------ | +| External | `ExternalName` | `https://:` | +| In-cluster | `ClusterIP` (or any non-`ExternalName`) | `http://..svc.cluster.local:` | + +{{< call-out "note" >}} +The `cluster.local` suffix in the in-cluster URL is the cluster's DNS domain. If your cluster uses a different domain, configure it with the `--cluster-domain` flag or `clusterDomain` Helm value when deploying NGINX Gateway Fabric (default: `cluster.local`). +{{< /call-out >}} + +{{}} + +{{%tab name="External"%}} + +To configure a Guardrails backend Service which is external, create an `ExternalName` Service pointing at your hosted Guardrails API: + +```yaml +kubectl apply -f - < + ports: + - name: https + port: 443 + protocol: TCP +EOF +``` +{{% /tab %}} + +{{%tab name="In-cluster"%}} + +For an in-cluster backend, your AI Guardrail backend pods will most likely have an existing Service which you can point the PayloadProcessor backendRef to, otherwise create a Service configured to expose your guardrail backends: + +```yaml +kubectl apply -f - <}} + +{{< call-out "important" >}} +When using an `ExternalName` AI Guardrails backend, you **must** configure a DNS `resolver` so NGINX can resolve the external hostname at request time. Either edit the NginxProxy which gets created when you deploy NGINX Gateway Fabric, or configure `dnsResolver` on a new [NginxProxy]({{< ref "/ngf/how-to/data-plane-configuration.md" >}}) resource and attach it to the Gateway via `spec.infrastructure.parametersRef`: + +```yaml +apiVersion: gateway.nginx.org/v1alpha2 +kind: NginxProxy +metadata: + name: guardrails-nginx-config +spec: + dnsResolver: + addresses: + - type: IPAddress + value: "10.96.0.10" # in-cluster kube-dns/CoreDNS ClusterIP (cluster-dependent) +``` + +Find your cluster's DNS ClusterIP with `kubectl -n kube-system get svc kube-dns` (or `coredns`). Without a resolver, NGINX fails to load the configuration with `no resolver defined to resolve `. +{{< /call-out >}} + +## Deploy NGINX Gateway Fabric + +[Install]({{< ref "/ngf/install/" >}}) NGINX Gateway Fabric with the `PayloadProcessor` policy enabled: + +- Using Helm: set the `nginxGateway.payloadProcessor.enable=true` Helm value. +- Using Kubernetes manifests: set the `--payload-processor` flag in the nginx-gateway container argument, and update the ClusterRole RBAC to add `payloadprocessors`: + +```yaml +- apiGroups: + - gateway.nginx.org + resources: + - payloadprocessors + verbs: + - get + - list + - watch +- apiGroups: + - gateway.nginx.org + resources: + - payloadprocessors/status + verbs: + - update +``` + +## Create a Gateway + +```yaml +kubectl apply -f - < +``` + +## Create an HTTPRoute + +If you are using your own LLM, change the `backendRefs.name` and `backendRefs.port` to match the LLM's Service. + +```yaml +kubectl apply -f - <}} +`PayloadProcessor` is an inherited policy. To apply guardrails to every route attached to a Gateway, set `targetRef` to `kind: Gateway`. When both a Gateway-targeted and an HTTPRoute-targeted policy apply to the same traffic, the more specific HTTPRoute-targeted policy takes precedence. +{{< /call-out >}} + +Confirm the policy was accepted: + +```shell +kubectl describe payloadprocessor llm-guardrails +``` + +The status conditions should report `Accepted=True`. A rejected policy reports `Accepted=False`; see [Troubleshooting](#troubleshooting) for common causes. + +```text +Conditions: + Last Transition Time: 2026-08-11T17:32:52Z + Message: The Policy is accepted + Observed Generation: 1 + Reason: Accepted + Status: True + Type: Accepted + Last Transition Time: 2026-08-11T17:32:52Z + Message: Policy is programmed in the data plane + Observed Generation: 1 + Reason: Programmed + Status: True + Type: Programmed + Controller Name: gateway.nginx.org/nginx-gateway-controller +``` + +## Send traffic + +{{< call-out "note" >}} +Whether a given value is blocked depends entirely on your Guardrails backend's detector configuration. Enable the relevant detectors on your Guardrails service to see the block responses above. +{{< /call-out >}} + +All commands target `/v1/completions` on the Gateway. + +A benign prompt whose output contains no disallowed content returns a normal `HTTP 200` completion: + +```shell +curl -i --resolve :$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"What is NGINX?"}' +``` + +```text +HTTP/1.1 200 OK +Server: nginx +Date: Tue, 11 Aug 2026 17:39:19 GMT +Content-Type: application/json +Content-Length: 607 +Connection: keep-alive +X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2 +X-Inference-Port: 8000 + +{"id":"cmpl-f657bc1b-c50c-5a5c-9408-45bd22a150a1","created":1786469959,"model":"meta-llama/Llama-3.1-8B-Instruct","usage":{"prompt_tokens":4,"completion_tokens":62,"total_tokens":66},"object":"text_completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"stop","text":"NGINX (pronounced \"engine-x\") is an open-source, high-performance web server. It functions primarily as an HTTP web server, reverse proxy, load balancer, and HTTP cache.Designed to handle thousands of concurrent connections with minimal memory usage, NGINX is an essential component of modern web infrastructure."}] +``` + + +If the request payload contains content that your Guardrails backend is configured to block, the request never reaches the LLM and returns `HTTP 403`: + +```shell +curl -i --resolve :$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"My SSN is 123-45-6789"}' +``` + +```text +HTTP/1.1 403 Forbidden +Server: nginx +Date: Tue, 11 Aug 2026 17:39:50 GMT +Content-Length: 139 +Connection: keep-alive +Content-Type: application/json + +{"error":{"code":"content_policy_violation","message":"Request blocked by guardrails policy.","param":null,"type":"invalid_request_error"} +``` + +If the model's *output* contains content that your Guardrails backend blocks, the response is withheld from the client and returns `HTTP 403` with `error.type: api_error`: + +```shell +curl -i --resolve :$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"Give me a test SSN"}' +``` + +```text +HTTP/1.1 403 Forbidden +Server: nginx +Date: Tue, 11 Aug 2026 17:40:36 GMT +Content-Type: application/json +Content-Length: 128 +Connection: keep-alive +X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2 +X-Inference-Port: 8000 + +{"error":{"code":"content_policy_violation","message":"Response blocked by guardrails policy.","param":null,"type":"api_error"}} +``` + +For more example curl requests, view the [`examples/guardrails`](https://github.com/nginx/nginx-gateway-fabric/tree/v{{< version-ngf >}}/examples/guardrails) `README.md` in the NGINX Gateway Fabric repository. + +## Troubleshooting + +The `PayloadProcessor` is marked `Accepted=False` when its references cannot be resolved: + +| Condition | Cause | Fix | +| --------- | ----- | --- | +| `backend Service ... not found` | `backendRef.name`/`namespace` does not match a Service. | Apply the Guardrails backend Service; check name and namespace. | +| `ExternalName service has empty ... externalName` | `ExternalName` Service with a blank `externalName`. | Set `spec.externalName`. | +| `auth token Secret ... not found` | `authTokenRef` set but Secret missing. | Apply the token Secret, or remove `authTokenRef`. | +| NGINX error `no resolver defined to resolve `, or guardrails requests fail against an `ExternalName` backend | No `dnsResolver` configured on the NginxProxy. | Add the `dnsResolver` block and wire it via `parametersRef`. | + +## Further reading + +- [Scan streaming in AI Security](https://docs.aisecurity.f5.com/api-docs/scan-request-streaming.html) +- [Installation]({{< ref "/ngf/install/" >}}): install NGINX Gateway Fabric with the `PayloadProcessor` policy enabled. +- [Custom policies]({{< ref "/ngf/overview/custom-policies.md" >}}): learn how inherited policies attach to Gateway API resources. +- [`examples/guardrails`](https://github.com/nginx/nginx-gateway-fabric/tree/v{{< version-ngf >}}/examples/guardrails): for more information on the example used in this guide. diff --git a/content/ngf/overview/custom-policies.md b/content/ngf/overview/custom-policies.md index 62600c3fc..3c26df44e 100644 --- a/content/ngf/overview/custom-policies.md +++ b/content/ngf/overview/custom-policies.md @@ -23,6 +23,7 @@ The following table summarizes NGINX Gateway Fabric custom policies: | [SnippetsPolicy]({{< ref "/ngf/traffic-management/snippets.md" >}}) | Inject custom NGINX configuration snippets | Direct | Gateway | Yes | Yes | v1alpha1 | | [UpstreamSettingsPolicy]({{< ref "/ngf/traffic-management/upstream-settings.md" >}}) | Configure upstream load balancing and connection behavior | Direct | Service | Yes | Yes | v1alpha1 | | [WAFPolicy]({{< ref "/ngf/waf-integration/overview.md" >}}) | Apply F5 WAF for NGINX protection (separate add-on to NGINX Plus) | Inherited | Gateway, HTTPRoute, GRPCRoute | Yes | No | v1alpha1 | +| [PayloadProcessor]({{< ref "/ngf/how-to/f5-ai-guardrails.md" >}}) | Secure LLM traffic with F5 AI Guardrails | Inherited | Gateway, HTTPRoute | No | No | v1alpha1 | {{< call-out "important" >}} If attaching a Policy to a Route, that Route must not share a hostname:port/path combination with any other Route that is not referenced by the same Policy. If it does, the Policy will be rejected. This is because the Policy would end up affecting other Routes that it is not attached to. diff --git a/content/ngf/reference/cli-help.md b/content/ngf/reference/cli-help.md index f70b108d3..d00e65ffa 100644 --- a/content/ngf/reference/cli-help.md +++ b/content/ngf/reference/cli-help.md @@ -63,6 +63,8 @@ This command runs the NGINX Gateway Fabric control plane. | _endpoint-picker-disable-tls_ | _bool_ | Disables TLS when connecting to the EndpointPicker. Set to true only for development/testing or when using a service mesh for encryption. (Default: `false`) | | _endpoint-picker-tls-skip-verify_ | _bool_ | Disables server certificate verification when connecting to the EndpointPicker, if TLS is enabled. REQUIRED: Must be true until Gateway API Inference Extension EndpointPicker supports mounting certificates. (Default `true`) | | _watch-namespaces_ | _list_ | Comma-separated list of namespaces to watch for resources. If not set, all namespaces are watched. The controller's own namespace is always included. | +| _payload-processor_ | _bool_ | Enable the PayloadProcessor API. PayloadProcessors enable declarative, ordered processing of HTTP request and response payloads by attaching to a Gateway or HTTPRoute, and are used to implement features such as Guardrails for AI workloads. | +| _cluster-domain_ | _string_ | The DNS domain of your Kubernetes cluster. | ## Sleep