From 4e8a829e83b5fbaf5c58a26ec2608568404d5360 Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:56:04 +0800 Subject: [PATCH 01/18] docs: add SR-IOV KubeVirt passthrough guide --- AGENTS.md | 4 + ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 421 ++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 421 ++++++++++++++++++ 3 files changed, 846 insertions(+) create mode 100644 AGENTS.md create mode 100644 docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md create mode 100644 docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..ed9d29adf --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,4 @@ +# Project Memory + +- When adding or updating a solution document, keep the Chinese and English versions in sync. +- For files under `docs/zh/solutions/`, add or update the matching file under `docs/en/solutions/` with the same filename unless the user explicitly asks for Chinese-only work. diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..741b6ddb0 --- /dev/null +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,421 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.4.x' +--- + +# Enable SR-IOV NIC Passthrough for KubeVirt VMs on ACP + +## Issue + +Users running KubeVirt VMs on Alauda Container Platform 4.4.x may need to attach host SR-IOV VFs to the VMs as high-performance secondary NICs. The cluster primary CNI can remain kube-ovn; SR-IOV Network Operator is used to discover SR-IOV PFs, create VFs, advertise VF resources through the device plugin, and generate the corresponding `NetworkAttachmentDefinition` objects for Multus. + +This article follows the user workflow for installing the ACP 4.4.x `sriov-network-plugin`, validating the control plane in an environment without SR-IOV NICs, and completing VF and VM data-plane validation in an environment with SR-IOV hardware. + +## Environment + +This article applies to the following combination: + +| Component | Version or description | +| --- | --- | +| Alauda Container Platform | 4.4.x | +| Plugin | `sriov-network-plugin` | +| Plugin package version | `sriov-network-plugin v4.4.1` | +| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | +| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | + +The ACP package enables only the SR-IOV CNI path. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. + +## Prerequisites + +### Platform and permissions + +Prepare a kubeconfig that can manage the target cluster. The current user must be able to create the following resources: + +- Namespaces, ServiceAccounts, ClusterRoles, and ClusterRoleBindings +- CRDs +- Deployments and DaemonSets +- Custom resources in the `sriovnetwork.openshift.io` API group +- `NetworkAttachmentDefinition` objects in `k8s.cni.cncf.io/v1` + +### Nodes and hardware + +SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: + +- The node has a physical NIC PF that supports SR-IOV. +- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. +- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. +- The target node can enter a maintenance window. Creating VFs or changing VF drivers can trigger node drain or a short network interruption. + +Label the nodes that will be configured for SR-IOV. `SriovNetworkNodePolicy` will use this label to limit its scope: + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true +``` + +## Resolution + +### Install the plugin + +This capability is delivered as an ACP 4.4.x feature. The plugin package version is `sriov-network-plugin v4.4.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. + +1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. +2. Select the package whose compatible platform version is `v4.4` and whose plugin version is `v4.4.1`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.4.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.4.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.4.1.tgz`. +4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. +5. Upload the downloaded plugin package to the target ACP platform. + +If the target platform uses `violet` to upload offline packages, use the following command: + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" + +unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +After the upload succeeds, verify on the global cluster that the plugin version configuration has been generated and is ready to deploy: + +```bash +kubectl get moduleplugin sriov-network-plugin \ + -o jsonpath='{.status.latestVersion}{"\n"}' +kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ + -o jsonpath='{.status.readyForDeploy}{"\n"}' +``` + +The expected output is `v4.4.1` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. + +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: + +```bash +export SRIOV_NAMESPACE="cpaas-system" +``` + +Before installation, confirm that the cluster does not already have a manually installed SR-IOV instance. If a previous test installation exists in another namespace, such as `sriov-network-operator`, uninstall that old instance with the original installation method before installing from the marketplace. Running two operators against the same SR-IOV custom resources is not recommended: + +```bash +kubectl get pods -A | grep sriov-network +kubectl get sriovoperatorconfig -A +kubectl get sriovnetworknodestate -A +``` + +The SR-IOV node daemon requires `hostNetwork`, `hostPID`, `hostPath`, and privileged containers. Before installation, confirm that the deployment namespace allows privileged Pod Security Admission. If the labels are missing, add them: + +```bash +kubectl label namespace "$SRIOV_NAMESPACE" \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/audit=privileged \ + pod-security.kubernetes.io/warn=privileged \ + --overwrite +``` + +After installation, confirm that the operator and config-daemon are running: + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" +``` + +Expected workloads: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +Confirm that the CRDs are registered: + +```bash +kubectl get crd | grep sriovnetwork.openshift.io +``` + +Confirm that `config-daemon` has only the `sriov-cni` init container and does not include `ib-sriov-cni`, `ovs-cni`, or `rdma-cni`: + +```bash +kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' +``` + +Expected output: + +```text +sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 +``` + +### Control-plane validation without SR-IOV hardware + +If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly, CRDs are registered, the operator can synchronize node state, and no extra `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` init containers are deployed. + +Check `SriovNetworkNodeState`: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +Check a specific node: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}' +``` + +In an environment without SR-IOV hardware, `status.interfaces` can be empty. This is a hardware limitation and does not indicate a plugin deployment failure. If the operator, config-daemon, CRDs, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. + +### VF validation with SR-IOV hardware + +On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. The following example creates four VFs and advertises a device-plugin resource named `intel_sriov_netdevice`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: intel-sriov-netdevice + namespace: cpaas-system +spec: + resourceName: intel_sriov_netdevice + nodeSelector: + feature.node.kubernetes.io/sriov-capable: "true" + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: netdevice + mtu: 1500 +``` + +Apply the policy and watch node synchronization: + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +Confirm that the target node reaches `Succeeded`: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. + +### Create an SR-IOV secondary network + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: intel_sriov_netdevice + vlan: 0 + ipam: | + { + "type": "host-local", + "subnet": "192.168.100.0/24", + "rangeStart": "192.168.100.100", + "rangeEnd": "192.168.100.200", + "gateway": "192.168.100.1" + } +``` + +Apply the object and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +Inspect the effective NAD CNI configuration: + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### Validate VF allocation with a test Pod + +Create a test Pod in the application namespace. The Pod must request the SR-IOV resource and reference the NAD: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-test + namespace: kubevirt + annotations: + k8s.v1.cni.cncf.io/networks: vm-sriov-net +spec: + restartPolicy: Never + containers: + - name: test + image: + command: + - sleep + - "3600" + resources: + requests: + openshift.io/intel_sriov_netdevice: "1" + limits: + openshift.io/intel_sriov_netdevice: "1" +``` + +Confirm that the Pod is scheduled to an SR-IOV node and enters `Running`: + +```bash +kubectl get pod sriov-test -n kubevirt -o wide +``` + +Enter the Pod and inspect the secondary NIC: + +```bash +kubectl exec -n kubevirt sriov-test -- ip link +kubectl exec -n kubevirt sriov-test -- ip addr +``` + +If the underlay network is ready, continue with ping or application traffic validation: + +```bash +kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +``` + +### Use the SR-IOV network in a KubeVirt VM + +A VM can reference the same NAD through Multus. The following example shows only the network and interface-related fields: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. + +## Diagnostic steps + +### Operator or config-daemon is not running + +Check Pod status and events: + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" -o wide +kubectl describe pod -n "$SRIOV_NAMESPACE" +kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator +``` + +If `config-daemon` is stuck in init, first confirm that the init container list contains only `sriov-cni`. If `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` still appears, the installed plugin package or operator image is not the ACP-corrected version. + +### The node does not discover an SR-IOV PF + +Check `SriovNetworkNodeState`: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml +``` + +If `status.interfaces` is empty, continue checking hardware and kernel state on the node: + +```bash +lspci -nn | grep -i ethernet +ip link show +``` + +Confirm that BIOS/IOMMU is enabled and that the target PF driver supports SR-IOV. + +### Policy synchronization fails + +Check synchronization status and errors: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Common causes include: + +- `nodeSelector` does not match any node. +- `nicSelector.pfNames` is incorrect, or the PF name does not exist on the target node. +- The PF is held by another network component and cannot create VFs. +- IOMMU is not enabled, or the driver does not support the requested number of VFs. + +### Pod or VM cannot allocate a VF + +Confirm that the resource exists in node allocatable: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +Check Pod or virt-launcher events: + +```bash +kubectl describe pod -n kubevirt +``` + +If the event reports insufficient resources, verify that `SriovNetworkNodePolicy.spec.resourceName`, the Pod resource request, and `SriovNetwork.spec.resourceName` all use the same value. + +## Limitations + +- An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. +- This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. +- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..b0d1747ca --- /dev/null +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,421 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.4.x' +--- + +# 如何在 ACP 上为 KubeVirt 虚拟机启用 SR-IOV 网卡直通 + +## 问题 + +用户在 Alauda Container Platform 4.4.x 上运行 KubeVirt 虚拟机，并希望给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；SR-IOV Network Operator 只负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并为 Multus 生成对应的 `NetworkAttachmentDefinition`。 + +本文从用户使用路径说明如何安装 ACP 4.4.x 的 `sriov-network-plugin`，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 VF 和虚拟机数据面验证。 + +## 环境 + +本文适用于以下组合： + +| 组件 | 版本或说明 | +| --- | --- | +| Alauda Container Platform | 4.4.x | +| 插件 | `sriov-network-plugin` | +| 插件包版本 | `sriov-network-plugin v4.4.1` | +| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | +| 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | + +ACP 打包版本只启用 SR-IOV CNI 路径。`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 + +## 先决条件 + +### 平台和权限 + +准备一个可管理目标集群的 kubeconfig，并确保当前用户可以创建以下资源： + +- 命名空间、ServiceAccount、ClusterRole、ClusterRoleBinding +- CRD +- Deployment、DaemonSet +- `sriovnetwork.openshift.io` API 组下的自定义资源 +- `k8s.cni.cncf.io/v1` 的 `NetworkAttachmentDefinition` + +### 节点和硬件 + +如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： + +- 节点上有支持 SR-IOV 的物理网卡 PF。 +- BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 +- PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 +- 计划配置 VF 的节点可以进入维护窗口。创建 VF 或切换 VF 驱动可能触发节点 drain 或网络短暂中断。 + +给参与 SR-IOV 配置的节点打标签，后续 `SriovNetworkNodePolicy` 使用该标签限制作用范围： + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true +``` + +## 解决方案 + +### 安装插件 + +该能力作为 ACP 4.4.x 新功能交付，插件包版本为 `sriov-network-plugin v4.4.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 + +1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 +2. 选择适配平台版本为 `v4.4`、插件版本为 `v4.4.1` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.4.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.4.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.4.1.tgz`。 +4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 +5. 将下载的插件包上传到目标 ACP 平台。 + +如果目标平台使用 `violet` 上传离线包，可以参考以下命令： + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" + +unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +上传成功后，在 global 集群确认插件版本配置已经生成并可部署： + +```bash +kubectl get moduleplugin sriov-network-plugin \ + -o jsonpath='{.status.latestVersion}{"\n"}' +kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ + -o jsonpath='{.status.readyForDeploy}{"\n"}' +``` + +预期分别输出 `v4.4.1` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 + +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： + +```bash +export SRIOV_NAMESPACE="cpaas-system" +``` + +安装前确认集群中没有历史手工安装的 SR-IOV 实例。如果曾经在 `sriov-network-operator` 等其他命名空间测试安装过，应先按原安装方式卸载旧实例，避免两个 operator 同时协调同一组 SR-IOV 自定义资源： + +```bash +kubectl get pods -A | grep sriov-network +kubectl get sriovoperatorconfig -A +kubectl get sriovnetworknodestate -A +``` + +SR-IOV 节点守护进程需要使用 `hostNetwork`、`hostPID`、`hostPath` 和 privileged 容器。安装前，确认部署命名空间允许 privileged Pod Security Admission；如果标签缺失，可以补齐： + +```bash +kubectl label namespace "$SRIOV_NAMESPACE" \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/audit=privileged \ + pod-security.kubernetes.io/warn=privileged \ + --overwrite +``` + +安装后确认 operator 和 config-daemon 已运行： + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" +``` + +预期至少看到以下工作负载处于 `Running`： + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +确认 CRD 已注册： + +```bash +kubectl get crd | grep sriovnetwork.openshift.io +``` + +确认 `config-daemon` 只包含 `sriov-cni` init container，不包含 `ib-sriov-cni`、`ovs-cni`、`rdma-cni`： + +```bash +kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' +``` + +预期只出现类似输出： + +```text +sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 +``` + +### 无 SR-IOV 网卡环境的控制面验证 + +如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署、CRD 可以注册、operator 可以同步节点状态，并且不会额外部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。 + +检查 `SriovNetworkNodeState`： + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +查看具体节点状态： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}' +``` + +在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空；这是硬件条件限制，不代表插件部署失败。只要 operator、config-daemon、CRD 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 + +### 有 SR-IOV 网卡环境的 VF 验证 + +在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。以下示例创建 4 个 VF，并通过 device plugin 暴露名为 `intel_sriov_netdevice` 的资源： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: intel-sriov-netdevice + namespace: cpaas-system +spec: + resourceName: intel_sriov_netdevice + nodeSelector: + feature.node.kubernetes.io/sriov-capable: "true" + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: netdevice + mtu: 1500 +``` + +应用后观察节点同步状态： + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +确认目标节点变为 `Succeeded`： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 + +### 创建 SR-IOV 辅助网络 + +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: intel_sriov_netdevice + vlan: 0 + ipam: | + { + "type": "host-local", + "subnet": "192.168.100.0/24", + "rangeStart": "192.168.100.100", + "rangeEnd": "192.168.100.200", + "gateway": "192.168.100.1" + } +``` + +应用后确认 NAD 已生成： + +```bash +kubectl apply -f sriov-network.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +查看 NAD 的有效 CNI 配置： + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### 使用测试 Pod 验证 VF 分配 + +在业务命名空间创建测试 Pod，显式请求 SR-IOV 资源并引用 NAD： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-test + namespace: kubevirt + annotations: + k8s.v1.cni.cncf.io/networks: vm-sriov-net +spec: + restartPolicy: Never + containers: + - name: test + image: + command: + - sleep + - "3600" + resources: + requests: + openshift.io/intel_sriov_netdevice: "1" + limits: + openshift.io/intel_sriov_netdevice: "1" +``` + +确认 Pod 被调度到 SR-IOV 节点并进入 `Running`： + +```bash +kubectl get pod sriov-test -n kubevirt -o wide +``` + +进入 Pod 检查辅助网卡： + +```bash +kubectl exec -n kubevirt sriov-test -- ip link +kubectl exec -n kubevirt sriov-test -- ip addr +``` + +如果网络侧已经配置好二层或三层连通性，可以继续执行 ping 或业务流量验证： + +```bash +kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +``` + +### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 + +虚拟机可以通过 Multus 引用同一个 NAD。以下示例只展示网络和接口相关字段： + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 + +## 诊断步骤 + +### operator 或 config-daemon 未运行 + +检查 Pod 状态和事件： + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" -o wide +kubectl describe pod -n "$SRIOV_NAMESPACE" +kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator +``` + +如果 `config-daemon` 卡在 init 阶段，先确认 init container 列表是否只有 `sriov-cni`。如果仍然出现 `ovs-cni`、`rdma-cni` 或 `ib-sriov-cni`，说明安装的不是 ACP 修正后的插件包或 operator 镜像版本。 + +### 节点没有发现 SR-IOV PF + +检查 `SriovNetworkNodeState`： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml +``` + +如果 `status.interfaces` 为空，继续在节点侧确认硬件和内核状态： + +```bash +lspci -nn | grep -i ethernet +ip link show +``` + +确认 BIOS/IOMMU 已启用，并且目标 PF 驱动支持 SR-IOV。 + +### policy 同步失败 + +查看同步状态和错误： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +常见原因包括： + +- `nodeSelector` 没有匹配任何节点。 +- `nicSelector.pfNames` 写错，或者 PF 名称在目标节点上不存在。 +- PF 已被其他网络组件占用，无法创建 VF。 +- 节点未启用 IOMMU 或驱动不支持请求的 VF 数量。 + +### Pod 或 VM 无法分配 VF + +确认节点 allocatable 中存在资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +确认 Pod 或 virt-launcher 事件： + +```bash +kubectl describe pod -n kubevirt +``` + +如果事件提示资源不足，检查 `SriovNetworkNodePolicy.spec.resourceName`、Pod resource request、`SriovNetwork.spec.resourceName` 三者是否一致。 + +## 限制 + +- 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 +- 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 +- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 From 7d337e2373c7948d8ee3ddcca4aa199a21104c27 Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Mon, 6 Jul 2026 18:41:13 +0800 Subject: [PATCH 02/18] ACP-53813 document Multus SR-IOV solution --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 54 ++++++++++++++----- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 54 ++++++++++++++----- 2 files changed, 80 insertions(+), 28 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 741b6ddb0..4c058fc92 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.4.x' + - '4.3.x' --- -# Enable SR-IOV NIC Passthrough for KubeVirt VMs on ACP +# Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs on Alauda Container Platform 4.4.x may need to attach host SR-IOV VFs to the VMs as high-performance secondary NICs. The cluster primary CNI can remain kube-ovn; SR-IOV Network Operator is used to discover SR-IOV PFs, create VFs, advertise VF resources through the device plugin, and generate the corresponding `NetworkAttachmentDefinition` objects for Multus. +Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3.x may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for installing the ACP 4.4.x `sriov-network-plugin`, validating the control plane in an environment without SR-IOV NICs, and completing VF and VM data-plane validation in an environment with SR-IOV hardware. +This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3.x: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. ## Environment @@ -21,14 +21,17 @@ This article applies to the following combination: | Component | Version or description | | --- | --- | -| Alauda Container Platform | 4.4.x | +| Alauda Container Platform | 4.3.x | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.4.1` | +| Plugin package version | `sriov-network-plugin v4.3.x` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | +| Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | -The ACP package enables only the SR-IOV CNI path. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. +The ACP package enables only the SR-IOV CNI path. The `config-daemon` installs the SR-IOV CNI binary into the node CNI bin directory through the `sriov-cni` init container. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. + +This solution delivers SR-IOV VF orchestration and Multus secondary-network attachment. A DPDK userspace data plane can be built on top of SR-IOV VFs, but it still requires additional VF driver binding, HugePages, CPU isolation, NUMA planning, workload image adaptation, and performance testing. These are not provided as out-of-the-box capabilities by this plugin package. ## Prerequisites @@ -61,11 +64,11 @@ kubectl label node feature.node.kubernetes.io/sriov-capable=true ### Install the plugin -This capability is delivered as an ACP 4.4.x feature. The plugin package version is `sriov-network-plugin v4.4.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3.x feature. The plugin package version is `sriov-network-plugin v4.3.x`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.4` and whose plugin version is `v4.4.1`. -3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.4.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.4.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.4.1.tgz`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.x`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.x.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.x.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.x.tgz`. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -76,7 +79,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY @@ -93,11 +96,11 @@ After the upload succeeds, verify on the global cluster that the plugin version ```bash kubectl get moduleplugin sriov-network-plugin \ -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ +kubectl get moduleconfig sriov-network-plugin- \ -o jsonpath='{.status.readyForDeploy}{"\n"}' ``` -The expected output is `v4.4.1` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. +The expected output is `v4.3.x` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: @@ -123,6 +126,16 @@ kubectl label namespace "$SRIOV_NAMESPACE" \ --overwrite ``` +### Confirm the Multus base + +SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm that the cluster has the NAD CRD: + +```bash +kubectl get crd network-attachment-definitions.k8s.cni.cncf.io +``` + +If this CRD does not exist, the Multus base is not ready. Install or enable ACP Multus capability before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. + After installation, confirm that the operator and config-daemon are running: ```bash @@ -350,6 +363,19 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. +### Relationship with DPDK and support boundary + +The upstream `sriov-cni` can assign SR-IOV VFs to workloads. After those VFs are bound to userspace drivers such as `vfio-pci`, they can become part of a DPDK data path. Therefore, this solution provides the SR-IOV attachment base for a DPDK high-performance network path, but it is not a complete DPDK solution by itself. + +If the customer workload explicitly requires DPDK, validate the following items in addition to this solution: + +- Node BIOS settings, IOMMU, VFIO driver, HugePages, CPU isolation, and NUMA affinity. +- Whether VFs use kernel `netdevice` mode or are bound to `vfio-pci` for DPDK userspace processes. +- Whether the DPDK path is direct SR-IOV VF consumption by the workload, kube-ovn OVS-DPDK, Userspace CNI, vhostuser, or another integration pattern. +- Performance baselines for PPS, bandwidth, latency, and jitter using the customer CNF image or tools such as `testpmd` and Trex. + +Do not describe this plugin package as a complete DPDK productized delivery before the hardware, driver, resource-isolation, and performance-validation work is complete. + ## Diagnostic steps ### Operator or config-daemon is not running @@ -418,4 +444,4 @@ If the event reports insufficient resources, verify that `SriovNetworkNodePolicy - An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. - This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. -- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. +- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. \ No newline at end of file diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index b0d1747ca..4a20384d5 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.4.x' + - '4.3.x' --- -# 如何在 ACP 上为 KubeVirt 虚拟机启用 SR-IOV 网卡直通 +# 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.4.x 上运行 KubeVirt 虚拟机，并希望给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；SR-IOV Network Operator 只负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并为 Multus 生成对应的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3.x 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何安装 ACP 4.4.x 的 `sriov-network-plugin`，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 VF 和虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3.x 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 ## 环境 @@ -21,14 +21,17 @@ ProductsVersion: | 组件 | 版本或说明 | | --- | --- | -| Alauda Container Platform | 4.4.x | +| Alauda Container Platform | 4.3.x | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.4.1` | +| 插件包版本 | `sriov-network-plugin v4.3.x` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | +| 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | -ACP 打包版本只启用 SR-IOV CNI 路径。`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 +ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-cni` init container 将 SR-IOV CNI 二进制安装到节点 CNI bin 目录；`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 + +本方案交付的是 SR-IOV VF 编排和 Multus 辅助网络接入能力。DPDK 用户态数据面可以基于 SR-IOV VF 继续扩展，但还需要额外完成 VF 驱动绑定、HugePages、CPU 隔离、NUMA 规划、业务镜像适配和性能压测；这些不属于本插件包开箱即用能力。 ## 先决条件 @@ -61,11 +64,11 @@ kubectl label node feature.node.kubernetes.io/sriov-capable=true ### 安装插件 -该能力作为 ACP 4.4.x 新功能交付，插件包版本为 `sriov-network-plugin v4.4.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3.x 新功能交付，插件包版本为 `sriov-network-plugin v4.3.x`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.4`、插件版本为 `v4.4.1` 的安装包。 -3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.4.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.4.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.4.1.tgz`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.x` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.x.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.x.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.x.tgz`。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -76,7 +79,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY @@ -93,11 +96,11 @@ violet push "$PACKAGE_FILE" \ ```bash kubectl get moduleplugin sriov-network-plugin \ -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ +kubectl get moduleconfig sriov-network-plugin- \ -o jsonpath='{.status.readyForDeploy}{"\n"}' ``` -预期分别输出 `v4.4.1` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 +预期分别输出 `v4.3.x` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： @@ -123,6 +126,16 @@ kubectl label namespace "$SRIOV_NAMESPACE" \ --overwrite ``` +### 确认 Multus 基座可用 + +SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应确认集群已经具备 NAD CRD： + +```bash +kubectl get crd network-attachment-definitions.k8s.cni.cncf.io +``` + +如果该 CRD 不存在，说明 Multus 基座尚未就绪，应先安装或启用 ACP 的 Multus 能力，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 + 安装后确认 operator 和 config-daemon 已运行： ```bash @@ -350,6 +363,19 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide 进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 +### 与 DPDK 的关系和边界 + +上游 `sriov-cni` 可用于给工作负载分配 SR-IOV VF；VF 进一步绑定到 `vfio-pci` 等用户态驱动后，可以作为 DPDK 数据面的一部分使用。因此，本方案是 DPDK 高性能网络链路中的 SR-IOV 接入基础，但并不等同于完整 DPDK 方案。 + +如果客户业务明确要求 DPDK，需要在本方案之外继续确认并验证以下内容： + +- 节点 BIOS、IOMMU、VFIO 驱动、HugePages、CPU 隔离和 NUMA 亲和性。 +- VF 是使用内核 `netdevice` 模式，还是绑定 `vfio-pci` 供 DPDK 用户态进程使用。 +- DPDK 路径是业务容器直接消费 SR-IOV VF，还是使用 kube-ovn OVS-DPDK、Userspace CNI、vhostuser 等方案。 +- 使用客户真实 CNF 镜像或 `testpmd`、Trex 等工具完成 PPS、带宽、时延和抖动基线压测。 + +在没有完成上述硬件、驱动、资源隔离和压测验证前，不应把本插件包描述为完整的 DPDK 产品化交付。 + ## 诊断步骤 ### operator 或 config-daemon 未运行 @@ -418,4 +444,4 @@ kubectl describe pod -n kubevirt - 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 - 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 -- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 +- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 \ No newline at end of file From 5959b3f31a5a7899551d7a520b9a4daf4ee617ca Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Tue, 7 Jul 2026 09:36:11 +0800 Subject: [PATCH 03/18] Remove unintended AGENTS.md from PR --- AGENTS.md | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index ed9d29adf..000000000 --- a/AGENTS.md +++ /dev/null @@ -1,4 +0,0 @@ -# Project Memory - -- When adding or updating a solution document, keep the Chinese and English versions in sync. -- For files under `docs/zh/solutions/`, add or update the matching file under `docs/en/solutions/` with the same filename unless the user explicitly asks for Chinese-only work. From 42d9e373ecfc6ba351c50b53d6d3911be2830743 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 11:31:11 +0800 Subject: [PATCH 04/18] docs: refine SR-IOV KubeVirt guide --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 298 ++++-------------- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 298 ++++-------------- 2 files changed, 118 insertions(+), 478 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 4c058fc92..8aaae5250 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.3.x' + - '4.3' --- # Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3.x may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. +Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3.x: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. +This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. ## Environment @@ -21,30 +21,20 @@ This article applies to the following combination: | Component | Version or description | | --- | --- | -| Alauda Container Platform | 4.3.x | +| Alauda Container Platform | 4.3 | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.x` | +| Plugin package version | `sriov-network-plugin v4.3.1` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | | Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | -The ACP package enables only the SR-IOV CNI path. The `config-daemon` installs the SR-IOV CNI binary into the node CNI bin directory through the `sriov-cni` init container. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. +The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. -This solution delivers SR-IOV VF orchestration and Multus secondary-network attachment. A DPDK userspace data plane can be built on top of SR-IOV VFs, but it still requires additional VF driver binding, HugePages, CPU isolation, NUMA planning, workload image adaptation, and performance testing. These are not provided as out-of-the-box capabilities by this plugin package. +This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM secondary NICs. It does not cover OVS-DPDK, Userspace CNI, or DPDK application configuration inside containers. ## Prerequisites -### Platform and permissions - -Prepare a kubeconfig that can manage the target cluster. The current user must be able to create the following resources: - -- Namespaces, ServiceAccounts, ClusterRoles, and ClusterRoleBindings -- CRDs -- Deployments and DaemonSets -- Custom resources in the `sriovnetwork.openshift.io` API group -- `NetworkAttachmentDefinition` objects in `k8s.cni.cncf.io/v1` - ### Nodes and hardware SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: @@ -52,23 +42,16 @@ SR-IOV hardware is not required if you only need to validate the plugin control - The node has a physical NIC PF that supports SR-IOV. - IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. - The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. -- The target node can enter a maintenance window. Creating VFs or changing VF drivers can trigger node drain or a short network interruption. - -Label the nodes that will be configured for SR-IOV. `SriovNetworkNodePolicy` will use this label to limit its scope: - -```bash -kubectl label node feature.node.kubernetes.io/sriov-capable=true -``` ## Resolution ### Install the plugin -This capability is delivered as an ACP 4.3.x feature. The plugin package version is `sriov-network-plugin v4.3.x`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.x`. -3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.x.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.x.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.x.tgz`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.1.tgz`. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -79,9 +62,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" - -unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -91,55 +72,16 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -After the upload succeeds, verify on the global cluster that the plugin version configuration has been generated and is ready to deploy: - -```bash -kubectl get moduleplugin sriov-network-plugin \ - -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin- \ - -o jsonpath='{.status.readyForDeploy}{"\n"}' -``` - -The expected output is `v4.3.x` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. - -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: - -```bash -export SRIOV_NAMESPACE="cpaas-system" -``` - -Before installation, confirm that the cluster does not already have a manually installed SR-IOV instance. If a previous test installation exists in another namespace, such as `sriov-network-operator`, uninstall that old instance with the original installation method before installing from the marketplace. Running two operators against the same SR-IOV custom resources is not recommended: - -```bash -kubectl get pods -A | grep sriov-network -kubectl get sriovoperatorconfig -A -kubectl get sriovnetworknodestate -A -``` - -The SR-IOV node daemon requires `hostNetwork`, `hostPID`, `hostPath`, and privileged containers. Before installation, confirm that the deployment namespace allows privileged Pod Security Admission. If the labels are missing, add them: - -```bash -kubectl label namespace "$SRIOV_NAMESPACE" \ - pod-security.kubernetes.io/enforce=privileged \ - pod-security.kubernetes.io/audit=privileged \ - pod-security.kubernetes.io/warn=privileged \ - --overwrite -``` +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. ### Confirm the Multus base -SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm that the cluster has the NAD CRD: - -```bash -kubectl get crd network-attachment-definitions.k8s.cni.cncf.io -``` - -If this CRD does not exist, the Multus base is not ready. Install or enable ACP Multus capability before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. +SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the product documentation to [install Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. After installation, confirm that the operator and config-daemon are running: ```bash -kubectl get pods -n "$SRIOV_NAMESPACE" +kubectl get pods -n cpaas-system ``` Expected workloads: @@ -149,86 +91,71 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -Confirm that the CRDs are registered: - -```bash -kubectl get crd | grep sriovnetwork.openshift.io -``` - -Confirm that `config-daemon` has only the `sriov-cni` init container and does not include `ib-sriov-cni`, `ovs-cni`, or `rdma-cni`: - -```bash -kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' -``` - -Expected output: - -```text -sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 -``` - ### Control-plane validation without SR-IOV hardware -If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly, CRDs are registered, the operator can synchronize node state, and no extra `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` init containers are deployed. +If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly and the operator can synchronize node state. Check `SriovNetworkNodeState`: ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` Check a specific node: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}' ``` -In an environment without SR-IOV hardware, `status.interfaces` can be empty. This is a hardware limitation and does not indicate a plugin deployment failure. If the operator, config-daemon, CRDs, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. +In an environment without SR-IOV hardware, `status.interfaces` can be empty. If the operator, config-daemon, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. ### VF validation with SR-IOV hardware On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. The following example creates four VFs and advertises a device-plugin resource named `intel_sriov_netdevice`: +The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. + +Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: - name: intel-sriov-netdevice + name: sriov-vf-policy namespace: cpaas-system spec: - resourceName: intel_sriov_netdevice + resourceName: sriov_vf nodeSelector: - feature.node.kubernetes.io/sriov-capable: "true" + kubernetes.io/hostname: nicSelector: pfNames: - ens5f0 numVfs: 4 - deviceType: netdevice + deviceType: vfio-pci mtu: 1500 ``` +`deviceType: vfio-pci` is used for KubeVirt SR-IOV PCI passthrough. The operator configures the VF driver according to this policy and exposes the resource through the device plugin. Do not run `dpdk-devbind.py` inside the VM for host VFs. + Apply the policy and watch node synchronization: ```bash kubectl apply -f sriov-node-policy.yaml kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` Confirm that the target node reaches `Succeeded`: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` @@ -236,14 +163,14 @@ Confirm that the SR-IOV device-plugin resource appears in node allocatable resou ```bash kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' ``` If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. ### Create an SR-IOV secondary network -Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt`: +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -253,76 +180,50 @@ metadata: namespace: cpaas-system spec: networkNamespace: kubevirt - resourceName: intel_sriov_netdevice + resourceName: sriov_vf vlan: 0 ipam: | { - "type": "host-local", - "subnet": "192.168.100.0/24", - "rangeStart": "192.168.100.100", - "rangeEnd": "192.168.100.200", - "gateway": "192.168.100.1" + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" } ``` -Apply the object and confirm that the NAD is generated: - -```bash -kubectl apply -f sriov-network.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -Inspect the effective NAD CNI configuration: - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### Validate VF allocation with a test Pod +The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. -Create a test Pod in the application namespace. The Pod must request the SR-IOV resource and reference the NAD: +Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan: ```yaml -apiVersion: v1 -kind: Pod +apiVersion: kubeovn.io/v1 +kind: Subnet metadata: - name: sriov-test - namespace: kubevirt - annotations: - k8s.v1.cni.cncf.io/networks: vm-sriov-net + name: vm-sriov-net spec: - restartPolicy: Never - containers: - - name: test - image: - command: - - sleep - - "3600" - resources: - requests: - openshift.io/intel_sriov_netdevice: "1" - limits: - openshift.io/intel_sriov_netdevice: "1" + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 ``` -Confirm that the Pod is scheduled to an SR-IOV node and enters `Running`: +The KubeVirt VM default network still uses the kube-ovn primary network. The SR-IOV network is attached as a Multus secondary NIC, and its address is allocated by the Kube-OVN Subnet above. -```bash -kubectl get pod sriov-test -n kubevirt -o wide -``` - -Enter the Pod and inspect the secondary NIC: +Apply the objects and confirm that the NAD is generated: ```bash -kubectl exec -n kubevirt sriov-test -- ip link -kubectl exec -n kubevirt sriov-test -- ip addr +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net ``` -If the underlay network is ready, continue with ping or application traffic validation: +Inspect the effective NAD CNI configuration: ```bash -kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . ``` ### Use the SR-IOV network in a KubeVirt VM @@ -354,6 +255,8 @@ spec: networkName: vm-sriov-net ``` +In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. + After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: ```bash @@ -361,87 +264,4 @@ kubectl get vmi -n kubevirt sriov-vm kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` -Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. - -### Relationship with DPDK and support boundary - -The upstream `sriov-cni` can assign SR-IOV VFs to workloads. After those VFs are bound to userspace drivers such as `vfio-pci`, they can become part of a DPDK data path. Therefore, this solution provides the SR-IOV attachment base for a DPDK high-performance network path, but it is not a complete DPDK solution by itself. - -If the customer workload explicitly requires DPDK, validate the following items in addition to this solution: - -- Node BIOS settings, IOMMU, VFIO driver, HugePages, CPU isolation, and NUMA affinity. -- Whether VFs use kernel `netdevice` mode or are bound to `vfio-pci` for DPDK userspace processes. -- Whether the DPDK path is direct SR-IOV VF consumption by the workload, kube-ovn OVS-DPDK, Userspace CNI, vhostuser, or another integration pattern. -- Performance baselines for PPS, bandwidth, latency, and jitter using the customer CNF image or tools such as `testpmd` and Trex. - -Do not describe this plugin package as a complete DPDK productized delivery before the hardware, driver, resource-isolation, and performance-validation work is complete. - -## Diagnostic steps - -### Operator or config-daemon is not running - -Check Pod status and events: - -```bash -kubectl get pods -n "$SRIOV_NAMESPACE" -o wide -kubectl describe pod -n "$SRIOV_NAMESPACE" -kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator -``` - -If `config-daemon` is stuck in init, first confirm that the init container list contains only `sriov-cni`. If `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` still appears, the installed plugin package or operator image is not the ACP-corrected version. - -### The node does not discover an SR-IOV PF - -Check `SriovNetworkNodeState`: - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml -``` - -If `status.interfaces` is empty, continue checking hardware and kernel state on the node: - -```bash -lspci -nn | grep -i ethernet -ip link show -``` - -Confirm that BIOS/IOMMU is enabled and that the target PF driver supports SR-IOV. - -### Policy synchronization fails - -Check synchronization status and errors: - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -Common causes include: - -- `nodeSelector` does not match any node. -- `nicSelector.pfNames` is incorrect, or the PF name does not exist on the target node. -- The PF is held by another network component and cannot create VFs. -- IOMMU is not enabled, or the driver does not support the requested number of VFs. - -### Pod or VM cannot allocate a VF - -Confirm that the resource exists in node allocatable: - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' -``` - -Check Pod or virt-launcher events: - -```bash -kubectl describe pod -n kubevirt -``` - -If the event reports insufficient resources, verify that `SriovNetworkNodePolicy.spec.resourceName`, the Pod resource request, and `SriovNetwork.spec.resourceName` all use the same value. - -## Limitations - -- An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. -- This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. -- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. \ No newline at end of file +Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 4a20384d5..edb1e54d6 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.3.x' + - '4.3' --- # 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.3.x 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何在 ACP 4.3.x 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 ## 环境 @@ -21,30 +21,20 @@ ProductsVersion: | 组件 | 版本或说明 | | --- | --- | -| Alauda Container Platform | 4.3.x | +| Alauda Container Platform | 4.3 | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.x` | +| 插件包版本 | `sriov-network-plugin v4.3.1` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | | 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | -ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-cni` init container 将 SR-IOV CNI 二进制安装到节点 CNI bin 目录；`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 +ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 -本方案交付的是 SR-IOV VF 编排和 Multus 辅助网络接入能力。DPDK 用户态数据面可以基于 SR-IOV VF 继续扩展，但还需要额外完成 VF 驱动绑定、HugePages、CPU 隔离、NUMA 规划、业务镜像适配和性能压测；这些不属于本插件包开箱即用能力。 +本文覆盖 SR-IOV L5 插件在 KubeVirt 虚拟机辅助网卡场景中的安装与使用；不覆盖 OVS-DPDK、Userspace CNI 或容器内 DPDK 应用的配置。 ## 先决条件 -### 平台和权限 - -准备一个可管理目标集群的 kubeconfig，并确保当前用户可以创建以下资源： - -- 命名空间、ServiceAccount、ClusterRole、ClusterRoleBinding -- CRD -- Deployment、DaemonSet -- `sriovnetwork.openshift.io` API 组下的自定义资源 -- `k8s.cni.cncf.io/v1` 的 `NetworkAttachmentDefinition` - ### 节点和硬件 如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： @@ -52,23 +42,16 @@ ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-c - 节点上有支持 SR-IOV 的物理网卡 PF。 - BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 - PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 -- 计划配置 VF 的节点可以进入维护窗口。创建 VF 或切换 VF 驱动可能触发节点 drain 或网络短暂中断。 - -给参与 SR-IOV 配置的节点打标签，后续 `SriovNetworkNodePolicy` 使用该标签限制作用范围： - -```bash -kubectl label node feature.node.kubernetes.io/sriov-capable=true -``` ## 解决方案 ### 安装插件 -该能力作为 ACP 4.3.x 新功能交付，插件包版本为 `sriov-network-plugin v4.3.x`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.x` 的安装包。 -3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.x.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.x.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.x.tgz`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.1.tgz`。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -79,9 +62,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" - -unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -91,55 +72,16 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -上传成功后，在 global 集群确认插件版本配置已经生成并可部署： - -```bash -kubectl get moduleplugin sriov-network-plugin \ - -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin- \ - -o jsonpath='{.status.readyForDeploy}{"\n"}' -``` - -预期分别输出 `v4.3.x` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 - -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： - -```bash -export SRIOV_NAMESPACE="cpaas-system" -``` - -安装前确认集群中没有历史手工安装的 SR-IOV 实例。如果曾经在 `sriov-network-operator` 等其他命名空间测试安装过，应先按原安装方式卸载旧实例，避免两个 operator 同时协调同一组 SR-IOV 自定义资源： - -```bash -kubectl get pods -A | grep sriov-network -kubectl get sriovoperatorconfig -A -kubectl get sriovnetworknodestate -A -``` - -SR-IOV 节点守护进程需要使用 `hostNetwork`、`hostPID`、`hostPath` 和 privileged 容器。安装前，确认部署命名空间允许 privileged Pod Security Admission；如果标签缺失，可以补齐： - -```bash -kubectl label namespace "$SRIOV_NAMESPACE" \ - pod-security.kubernetes.io/enforce=privileged \ - pod-security.kubernetes.io/audit=privileged \ - pod-security.kubernetes.io/warn=privileged \ - --overwrite -``` +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 ### 确认 Multus 基座可用 -SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应确认集群已经具备 NAD CRD： - -```bash -kubectl get crd network-attachment-definitions.k8s.cni.cncf.io -``` - -如果该 CRD 不存在，说明 Multus 基座尚未就绪，应先安装或启用 ACP 的 Multus 能力，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 +SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[安装 Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni)，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 安装后确认 operator 和 config-daemon 已运行： ```bash -kubectl get pods -n "$SRIOV_NAMESPACE" +kubectl get pods -n cpaas-system ``` 预期至少看到以下工作负载处于 `Running`： @@ -149,86 +91,71 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -确认 CRD 已注册： - -```bash -kubectl get crd | grep sriovnetwork.openshift.io -``` - -确认 `config-daemon` 只包含 `sriov-cni` init container，不包含 `ib-sriov-cni`、`ovs-cni`、`rdma-cni`： - -```bash -kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' -``` - -预期只出现类似输出： - -```text -sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 -``` - ### 无 SR-IOV 网卡环境的控制面验证 -如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署、CRD 可以注册、operator 可以同步节点状态，并且不会额外部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。 +如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署，并且 operator 可以同步节点状态。 检查 `SriovNetworkNodeState`： ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` 查看具体节点状态： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}' ``` -在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空；这是硬件条件限制，不代表插件部署失败。只要 operator、config-daemon、CRD 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 +在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空。只要 operator、config-daemon 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 ### 有 SR-IOV 网卡环境的 VF 验证 在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。以下示例创建 4 个 VF，并通过 device plugin 暴露名为 `intel_sriov_netdevice` 的资源： +operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 + +选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源： ```yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: - name: intel-sriov-netdevice + name: sriov-vf-policy namespace: cpaas-system spec: - resourceName: intel_sriov_netdevice + resourceName: sriov_vf nodeSelector: - feature.node.kubernetes.io/sriov-capable: "true" + kubernetes.io/hostname: nicSelector: pfNames: - ens5f0 numVfs: 4 - deviceType: netdevice + deviceType: vfio-pci mtu: 1500 ``` +`deviceType: vfio-pci` 用于 KubeVirt SR-IOV PCI 直通场景。operator 会根据该策略配置 VF 驱动并通过 device plugin 暴露资源；不要在虚拟机内部对宿主机 VF 执行 `dpdk-devbind.py`。 + 应用后观察节点同步状态： ```bash kubectl apply -f sriov-node-policy.yaml kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` 确认目标节点变为 `Succeeded`： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` @@ -236,14 +163,14 @@ kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ ```bash kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' ``` 如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 ### 创建 SR-IOV 辅助网络 -创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`： +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -253,76 +180,50 @@ metadata: namespace: cpaas-system spec: networkNamespace: kubevirt - resourceName: intel_sriov_netdevice + resourceName: sriov_vf vlan: 0 ipam: | { - "type": "host-local", - "subnet": "192.168.100.0/24", - "rangeStart": "192.168.100.100", - "rangeEnd": "192.168.100.200", - "gateway": "192.168.100.1" + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" } ``` -应用后确认 NAD 已生成： - -```bash -kubectl apply -f sriov-network.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -查看 NAD 的有效 CNI 配置： - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### 使用测试 Pod 验证 VF 分配 +其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。 -在业务命名空间创建测试 Pod，显式请求 SR-IOV 资源并引用 NAD： +创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整： ```yaml -apiVersion: v1 -kind: Pod +apiVersion: kubeovn.io/v1 +kind: Subnet metadata: - name: sriov-test - namespace: kubevirt - annotations: - k8s.v1.cni.cncf.io/networks: vm-sriov-net + name: vm-sriov-net spec: - restartPolicy: Never - containers: - - name: test - image: - command: - - sleep - - "3600" - resources: - requests: - openshift.io/intel_sriov_netdevice: "1" - limits: - openshift.io/intel_sriov_netdevice: "1" + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 ``` -确认 Pod 被调度到 SR-IOV 节点并进入 `Running`： +KubeVirt 虚拟机的默认网络仍然使用 kube-ovn 主网络；SR-IOV 网络作为 Multus 辅助网卡接入，并通过上述 kube-ovn Subnet 完成辅助网卡 IPAM。 -```bash -kubectl get pod sriov-test -n kubevirt -o wide -``` - -进入 Pod 检查辅助网卡： +应用后确认 NAD 已生成： ```bash -kubectl exec -n kubevirt sriov-test -- ip link -kubectl exec -n kubevirt sriov-test -- ip addr +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net ``` -如果网络侧已经配置好二层或三层连通性，可以继续执行 ping 或业务流量验证： +查看 NAD 的有效 CNI 配置： ```bash -kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . ``` ### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 @@ -354,6 +255,8 @@ spec: networkName: vm-sriov-net ``` +上例中 VM 和 NAD 都在 `kubevirt` 命名空间，因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD，使用 `/` 格式。 + 创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： ```bash @@ -361,87 +264,4 @@ kubectl get vmi -n kubevirt sriov-vm kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` -进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 - -### 与 DPDK 的关系和边界 - -上游 `sriov-cni` 可用于给工作负载分配 SR-IOV VF；VF 进一步绑定到 `vfio-pci` 等用户态驱动后，可以作为 DPDK 数据面的一部分使用。因此，本方案是 DPDK 高性能网络链路中的 SR-IOV 接入基础，但并不等同于完整 DPDK 方案。 - -如果客户业务明确要求 DPDK，需要在本方案之外继续确认并验证以下内容： - -- 节点 BIOS、IOMMU、VFIO 驱动、HugePages、CPU 隔离和 NUMA 亲和性。 -- VF 是使用内核 `netdevice` 模式，还是绑定 `vfio-pci` 供 DPDK 用户态进程使用。 -- DPDK 路径是业务容器直接消费 SR-IOV VF，还是使用 kube-ovn OVS-DPDK、Userspace CNI、vhostuser 等方案。 -- 使用客户真实 CNF 镜像或 `testpmd`、Trex 等工具完成 PPS、带宽、时延和抖动基线压测。 - -在没有完成上述硬件、驱动、资源隔离和压测验证前，不应把本插件包描述为完整的 DPDK 产品化交付。 - -## 诊断步骤 - -### operator 或 config-daemon 未运行 - -检查 Pod 状态和事件： - -```bash -kubectl get pods -n "$SRIOV_NAMESPACE" -o wide -kubectl describe pod -n "$SRIOV_NAMESPACE" -kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator -``` - -如果 `config-daemon` 卡在 init 阶段，先确认 init container 列表是否只有 `sriov-cni`。如果仍然出现 `ovs-cni`、`rdma-cni` 或 `ib-sriov-cni`，说明安装的不是 ACP 修正后的插件包或 operator 镜像版本。 - -### 节点没有发现 SR-IOV PF - -检查 `SriovNetworkNodeState`： - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml -``` - -如果 `status.interfaces` 为空，继续在节点侧确认硬件和内核状态： - -```bash -lspci -nn | grep -i ethernet -ip link show -``` - -确认 BIOS/IOMMU 已启用，并且目标 PF 驱动支持 SR-IOV。 - -### policy 同步失败 - -查看同步状态和错误： - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -常见原因包括： - -- `nodeSelector` 没有匹配任何节点。 -- `nicSelector.pfNames` 写错，或者 PF 名称在目标节点上不存在。 -- PF 已被其他网络组件占用，无法创建 VF。 -- 节点未启用 IOMMU 或驱动不支持请求的 VF 数量。 - -### Pod 或 VM 无法分配 VF - -确认节点 allocatable 中存在资源： - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' -``` - -确认 Pod 或 virt-launcher 事件： - -```bash -kubectl describe pod -n kubevirt -``` - -如果事件提示资源不足，检查 `SriovNetworkNodePolicy.spec.resourceName`、Pod resource request、`SriovNetwork.spec.resourceName` 三者是否一致。 - -## 限制 - -- 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 -- 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 -- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 \ No newline at end of file +进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 From 06657998fea48a522a67b618f6711ffde23ea2db Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 13:29:21 +0800 Subject: [PATCH 05/18] docs: add VM DPDK binding note --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 27 +++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 27 +++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 8aaae5250..8fb36ea34 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -265,3 +265,30 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. + +### Bind the service VF to DPDK inside the VM + +If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Use the `dpdk-devbind.py` script from the DPDK package, or download it from the `dpdk-devbind.py` link in the ACP SR-IOV documentation. + +Inside the VM, identify the PCI NICs: + +```bash +lspci -Dnn | grep -i ethernet +``` + +Prepare HugePages according to the workload requirements, and load the VFIO driver: + +```bash +modprobe vfio-pci +modprobe vfio_iommu_type1 +``` + +Bind the service VF PCI address as seen inside the VM to `vfio-pci`: + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index edb1e54d6..3d366c1f5 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -265,3 +265,30 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` 进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 + +### 在虚拟机内将业务 VF 绑定给 DPDK + +如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。可以使用 DPDK 软件包自带的 `dpdk-devbind.py`，也可以从 ACP SR-IOV 文档的 `dpdk-devbind.py` 下载入口获取脚本。 + +在虚拟机内确认 PCI 网卡： + +```bash +lspci -Dnn | grep -i ethernet +``` + +按业务应用要求准备 HugePages，并加载 VFIO 驱动： + +```bash +modprobe vfio-pci +modprobe vfio_iommu_type1 +``` + +将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`： + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。 From 910241656c673512b96fa3e79b528692c07890dd Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 13:45:10 +0800 Subject: [PATCH 06/18] docs: clarify VM DPDK VF binding --- ...eploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md | 7 ++++--- ...eploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 8fb36ea34..2a625c3fa 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -268,7 +268,7 @@ Inside the guest operating system, confirm that an additional NIC appears. The K ### Bind the service VF to DPDK inside the VM -If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Use the `dpdk-devbind.py` script from the DPDK package, or download it from the `dpdk-devbind.py` link in the ACP SR-IOV documentation. +If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . Inside the VM, identify the PCI NICs: @@ -276,11 +276,12 @@ Inside the VM, identify the PCI NICs: lspci -Dnn | grep -i ethernet ``` -Prepare HugePages according to the workload requirements, and load the VFIO driver: +`dpdk-devbind.py` only binds the VF driver and does not prepare the DPDK application runtime. Configure HugePages, CPU affinity, and DPDK EAL parameters according to the workload image or application documentation. + +Load the VFIO driver: ```bash modprobe vfio-pci -modprobe vfio_iommu_type1 ``` Bind the service VF PCI address as seen inside the VM to `vfio-pci`: diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 3d366c1f5..9a4b5ec06 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -268,7 +268,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ### 在虚拟机内将业务 VF 绑定给 DPDK -如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。可以使用 DPDK 软件包自带的 `dpdk-devbind.py`，也可以从 ACP SR-IOV 文档的 `dpdk-devbind.py` 下载入口获取脚本。 +如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`；如果镜像中没有该脚本，可以从 DPDK 官方仓库获取：。 在虚拟机内确认 PCI 网卡： @@ -276,11 +276,12 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide lspci -Dnn | grep -i ethernet ``` -按业务应用要求准备 HugePages，并加载 VFIO 驱动： +`dpdk-devbind.py` 只负责 VF 驱动绑定，不负责准备 DPDK 应用运行环境。HugePages、CPU 亲和性和 DPDK EAL 参数按业务镜像或应用文档配置。 + +加载 VFIO 驱动： ```bash modprobe vfio-pci -modprobe vfio_iommu_type1 ``` 将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`： From 585bf30ea0846f73def685349ea0419538bf2c61 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 14:13:32 +0800 Subject: [PATCH 07/18] docs: improve SR-IOV VM guide clarity --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 49 +++++++++---------- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 49 +++++++++---------- 2 files changed, 44 insertions(+), 54 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 2a625c3fa..227d03641 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -7,13 +7,13 @@ ProductsVersion: - '4.3' --- -# Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP +# Provide High-Performance Secondary NICs for KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. +Users running KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to VMs, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. +This article follows the user workflow for completing an end-to-end KubeVirt VM Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use the SR-IOV secondary NIC in a KubeVirt VM. The same SR-IOV capability can also be used by application Pods, but this article uses KubeVirt VMs as the example scenario. ## Environment @@ -27,7 +27,7 @@ This article applies to the following combination: | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | -| Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | +| Multi-NIC base | ACP provides Multus capability; VMs reference the SR-IOV secondary network through NAD | The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. @@ -37,7 +37,7 @@ This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM se ### Nodes and hardware -SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: +Prepare at least one worker node that meets these requirements: - The node has a physical NIC PF that supports SR-IOV. - IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. @@ -76,7 +76,7 @@ After the upload is complete, go to **Administrator -> Marketplace -> Cluster Pl ### Confirm the Multus base -SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the product documentation to [install Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. +SR-IOV networks are attached to KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. After installation, confirm that the operator and config-daemon are running: @@ -91,29 +91,18 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -### Control-plane validation without SR-IOV hardware +### Configure VF resources -If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly and the operator can synchronize node state. +A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a VM. -Check `SriovNetworkNodeState`: +First list the node states synchronized by the operator, and select the target node from the `NAME` column: ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` -Check a specific node: - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}' -``` - -In an environment without SR-IOV hardware, `status.interfaces` can be empty. If the operator, config-daemon, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. - -### VF validation with SR-IOV hardware - -On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: +On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: ```bash kubectl get sriovnetworknodestate -n cpaas-system \ @@ -122,7 +111,9 @@ kubectl get sriovnetworknodestate -n cpaas-system \ The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. -Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`: +Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`. + +Save the following content as `sriov-node-policy.yaml`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -170,7 +161,9 @@ If the output is a positive integer, the VF resource is available to the Kuberne ### Create an SR-IOV secondary network -Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC: +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC. + +Save the following content as `sriov-network.yaml`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -190,9 +183,11 @@ spec: } ``` -The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. +The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Subnet `spec.provider`, and VM `metadata.namespace` together. + +Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan. -Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan: +Save the following content as `sriov-subnet.yaml`: ```yaml apiVersion: kubeovn.io/v1 @@ -266,7 +261,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. -### Bind the service VF to DPDK inside the VM +### Optional: Bind the service VF to DPDK inside the VM If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . @@ -292,4 +287,4 @@ python3 dpdk-devbind.py -b vfio-pci python3 dpdk-devbind.py --status ``` -`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. +`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. If the service VF is owned by the DPDK application, subsequent IP configuration and packet processing are handled by that DPDK application. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 9a4b5ec06..7fe740a3f 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -7,13 +7,13 @@ ProductsVersion: - '4.3' --- -# 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 +# 如何在 ACP 上通过 Multus 和 SR-IOV 为 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机，并希望通过 Multus 给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入虚拟机，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3 上完成 KubeVirt 虚拟机 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并在 KubeVirt 虚拟机中使用 SR-IOV 辅助网卡。同一 SR-IOV 能力也可用于业务 Pod，本文以 KubeVirt 虚拟机场景为例。 ## 环境 @@ -27,7 +27,7 @@ ProductsVersion: | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | -| 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | +| 多网卡基座 | ACP 已提供 Multus 能力，虚拟机通过 NAD 引用 SR-IOV 辅助网络 | ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 @@ -37,7 +37,7 @@ ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Mult ### 节点和硬件 -如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： +至少需要一个满足以下条件的 worker 节点： - 节点上有支持 SR-IOV 的物理网卡 PF。 - BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 @@ -76,7 +76,7 @@ violet push "$PACKAGE_FILE" \ ### 确认 Multus 基座可用 -SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[安装 Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni)，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 +SR-IOV 网络通过 Multus 作为辅助网卡接入 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 安装后确认 operator 和 config-daemon 已运行： @@ -91,29 +91,18 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -### 无 SR-IOV 网卡环境的控制面验证 +### 配置 VF 资源 -如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署，并且 operator 可以同步节点状态。 +PF 是节点上的物理网卡，VF 是从 PF 创建出来并分配给虚拟机使用的虚拟 PCI 网卡。 -检查 `SriovNetworkNodeState`： +先查看 operator 已同步的节点状态，并从输出的 `NAME` 选择目标节点： ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` -查看具体节点状态： - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}' -``` - -在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空。只要 operator、config-daemon 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 - -### 有 SR-IOV 网卡环境的 VF 验证 - -在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： +在带 SR-IOV PF 的节点上，确认 operator 能发现物理网卡： ```bash kubectl get sriovnetworknodestate -n cpaas-system \ @@ -122,7 +111,9 @@ kubectl get sriovnetworknodestate -n cpaas-system \ operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 -选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源： +从 `status.interfaces[*].name` 输出中选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源。 + +将以下内容保存为 `sriov-node-policy.yaml`： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -170,7 +161,9 @@ kubectl get node \ ### 创建 SR-IOV 辅助网络 -创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址： +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址。 + +将以下内容保存为 `sriov-network.yaml`： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -190,9 +183,11 @@ spec: } ``` -其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。 +其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。如果虚拟机运行在其他命名空间，需要同时替换 `SriovNetwork.spec.networkNamespace`、Subnet `spec.provider` 和 VM `metadata.namespace`。 + +创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整。 -创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整： +将以下内容保存为 `sriov-subnet.yaml`： ```yaml apiVersion: kubeovn.io/v1 @@ -266,7 +261,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide 进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 -### 在虚拟机内将业务 VF 绑定给 DPDK +### 在虚拟机内将业务 VF 绑定给 DPDK（可选） 如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`；如果镜像中没有该脚本，可以从 DPDK 官方仓库获取：。 @@ -292,4 +287,4 @@ python3 dpdk-devbind.py -b vfio-pci python3 dpdk-devbind.py --status ``` -`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。 +`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。如果业务 VF 被 DPDK 应用接管，后续 IP 配置和报文处理由 DPDK 应用决定。 From 67922976612a8f3d1d5277755c0b6655d84bf0b9 Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 8 Jul 2026 10:31:36 +0800 Subject: [PATCH 08/18] docs: update SR-IOV DPDK VF guide --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 372 ++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 290 -------------- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 372 ++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 290 -------------- 4 files changed, 744 insertions(+), 580 deletions(-) create mode 100644 docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md delete mode 100644 docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md create mode 100644 docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md delete mode 100644 docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md new file mode 100644 index 000000000..827d679a3 --- /dev/null +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,372 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# Attach High-Performance SR-IOV VFs to DPDK Pods and KubeVirt VMs with Multus on ACP + +## Issue + +Users running DPDK/CNF Pods or KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs as high-performance service NICs or PCI passthrough devices through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to workloads, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. + +This article follows the user workflow for attaching SR-IOV VFs with Multus on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use `vfio-pci` SR-IOV VFs from either a DPDK/CNF Pod or a KubeVirt VM. + +## Environment + +This article applies to the following combination: + +| Component | Version or description | +| --- | --- | +| Alauda Container Platform | 4.3 | +| Plugin | `sriov-network-plugin` | +| Plugin package version | `sriov-network-plugin v4.3.1` | +| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | +| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | +| Multi-NIC base | ACP provides Multus capability; Pods or VMs reference the SR-IOV VF through NAD | + +The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. + +This article covers installing and using the SR-IOV L5 plugin for DPDK/CNF Pod and KubeVirt VM VF attachment. It does not cover OVS-DPDK, Userspace CNI, or internal DPDK application parameters. + +## Prerequisites + +### Nodes and hardware + +Prepare at least one worker node that meets these requirements: + +- The node has a physical NIC PF that supports SR-IOV. +- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. +- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. +- The node kernel has VFIO support enabled. At minimum, confirm that the `vfio-pci` module is available: + +```bash +lsmod | grep vfio +modprobe vfio-pci +``` + +After VFs are created later in the procedure, record the PCI address of the target VF and confirm that its IOMMU group does not include other devices that should not be passed through to the workload: + +```bash +VF_PCI_ADDR="" +GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" +ls -l "$GROUP_PATH/devices" +``` + +If the same group contains other devices still required by the host, do not use that VF directly for `vfio-pci` or PCI passthrough. Adjust the hardware, BIOS settings, kernel IOMMU parameters, or NIC planning first. + +## Resolution + +### Install the plugin + +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. + +1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. +3. Download the `sriov-network-plugin.*.v4.3.1.tgz` package that matches the target platform architecture. +4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. +5. Upload the downloaded plugin package to the target ACP platform. + +If the target platform uses `violet` to upload offline packages, use the following command: + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. + +### Confirm the Multus base + +SR-IOV networks are attached to DPDK/CNF Pods or KubeVirt VMs by Multus as secondary networks. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. + +After installation, confirm that the operator and config-daemon are running: + +```bash +kubectl get pods -n cpaas-system +``` + +Expected workloads: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +### Confirm SR-IOV PFs on nodes + +A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a Pod or VM. + +First list the node states synchronized by the operator, and select the target node from the `NAME` column: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. + +Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. + +The following DPDK/CNF Pod and KubeVirt VM sections are two usage examples. If they share the same PF and `resourceName`, create the `SriovNetworkNodePolicy` only once. If different workloads need independent VF pools at the same time, use different PFs or plan different `resourceName` values and VF counts. + +### Configure and use an SR-IOV VF for DPDK/CNF Pods + +When a DPDK/CNF Pod needs a userspace process to own the VF directly, set `deviceType` to `vfio-pci`. In this mode, the VF is not used as a regular Linux network interface inside the Pod. Packet processing and address handling are owned by the DPDK/CNF application in the container. + +Save the following content as `sriov-node-policy-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-pod-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +Apply the policy and watch node synchronization: + +```bash +kubectl apply -f sriov-node-policy-pod.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +Confirm that the target node reaches `Succeeded`: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `default`. Because the VF is owned directly by the DPDK/CNF application, Kube-OVN IPAM is not configured here. + +Save the following content as `sriov-network-pod.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: pod-sriov-net + namespace: cpaas-system +spec: + networkNamespace: default + resourceName: sriov_vfio +``` + +In this example, the operator generates a NAD named `pod-sriov-net` in the `default` namespace. If the application Pod runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Pod `metadata.namespace`, and the Pod NAD reference together. + +If the service network must use a specific VLAN, add `vlan: ` under `spec`. If this field is omitted, the network is untagged. + +The Pod default network still uses the kube-ovn primary network. The SR-IOV VF is attached as a Multus secondary resource and consumed by the DPDK/CNF application. + +Apply the objects and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network-pod.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n default pod-sriov-net +``` + +A Pod references the NAD through the `k8s.v1.cni.cncf.io/networks` annotation and requests one SR-IOV VF through resource requests. The following example keeps the Pod default network on kube-ovn and requests one `vfio-pci` SR-IOV VF for the DPDK/CNF application. + +Save the following content as `sriov-pod.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-pod + namespace: default + annotations: + k8s.v1.cni.cncf.io/networks: default/pod-sriov-net +spec: + containers: + - name: app + image: nginx:latest + resources: + requests: + openshift.io/sriov_vfio: "1" + limits: + openshift.io/sriov_vfio: "1" +``` + +The `default/pod-sriov-net` value uses the `/` format. `openshift.io/sriov_vfio` corresponds to `SriovNetworkNodePolicy.spec.resourceName` and `SriovNetwork.spec.resourceName`. + +Apply the Pod and confirm that it is scheduled to a node with VF resources: + +```bash +kubectl apply -f sriov-pod.yaml +kubectl get pod -n default sriov-pod -o wide +``` + +If the workload uses Deployment, StatefulSet, or another controller, set `k8s.v1.cni.cncf.io/networks` in the Pod template `metadata.annotations`, and request `openshift.io/sriov_vfio` in the application container `resources.requests` and `resources.limits`. The VF resource only assigns the PCI device to the Pod. DPDK applications usually also need CPU, HugePages, and startup-parameter configuration; the exact values are determined by the business image and DPDK application documentation. A reference Pod fragment is: + +```yaml +resources: + requests: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" + limits: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" +volumeMounts: + - name: hugepage + mountPath: /dev/hugepages +volumes: + - name: hugepage + emptyDir: + medium: HugePages +``` + +### Configure and use an SR-IOV network for KubeVirt VMs + +KubeVirt SR-IOV PCI passthrough passes the VF to the VM as a PCI device, so `deviceType` is `vfio-pci`. + +If a `SriovNetworkNodePolicy` with `resourceName: sriov_vfio` already exists for the same PF, continue directly with the `SriovNetwork` for the VM. Create a separate policy only when the VM needs an independent VF pool. + +Save the following content as `sriov-node-policy-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vm-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +Apply the policy and watch node synchronization: + +```bash +kubectl apply -f sriov-node-policy-vm.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt`. If the VF is later owned by a DPDK application inside the VM, Kube-OVN IPAM is not required here. + +Save the following content as `sriov-network-vm.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vfio +``` + +In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, VM `metadata.namespace`, and the VM NAD reference together. + +If the service network must use a specific VLAN, add `vlan: ` under `spec`. If this field is omitted, the network is untagged. + +The KubeVirt VM default network still uses the kube-ovn primary network. The SR-IOV VF is attached as a Multus secondary device and passed through to the VM. + +Apply the objects and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network-vm.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +The VM secondary NIC is configured in two places in the VM template: `spec.template.spec.domain.devices.interfaces` defines the NIC type seen by the VM, and `spec.template.spec.networks` defines which Multus NAD the NIC attaches to. The `name` values in both lists must match. + +Add both `interfaces[].sriov` and `networks[].multus` to the VM template to attach the SR-IOV secondary NIC. The following example shows only the network and interface-related fields: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. + +After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +Inside the guest operating system, confirm that the passed-through SR-IOV VF appears. Whether to bind the VF to DPDK, how to allocate HugePages inside the VM for the DPDK application, and how to configure EAL parameters or the service process are determined by the guest operating system and DPDK application documentation. diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md deleted file mode 100644 index 227d03641..000000000 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -kind: - - How To -products: - - Alauda Container Platform -ProductsVersion: - - '4.3' ---- - -# Provide High-Performance Secondary NICs for KubeVirt VMs with Multus and SR-IOV on ACP - -## Issue - -Users running KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to VMs, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. - -This article follows the user workflow for completing an end-to-end KubeVirt VM Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use the SR-IOV secondary NIC in a KubeVirt VM. The same SR-IOV capability can also be used by application Pods, but this article uses KubeVirt VMs as the example scenario. - -## Environment - -This article applies to the following combination: - -| Component | Version or description | -| --- | --- | -| Alauda Container Platform | 4.3 | -| Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.1` | -| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | -| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | -| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | -| Multi-NIC base | ACP provides Multus capability; VMs reference the SR-IOV secondary network through NAD | - -The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. - -This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM secondary NICs. It does not cover OVS-DPDK, Userspace CNI, or DPDK application configuration inside containers. - -## Prerequisites - -### Nodes and hardware - -Prepare at least one worker node that meets these requirements: - -- The node has a physical NIC PF that supports SR-IOV. -- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. -- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. - -## Resolution - -### Install the plugin - -This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. - -1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. -3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.1.tgz`. -4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. -5. Upload the downloaded plugin package to the target ACP platform. - -If the target platform uses `violet` to upload offline packages, use the following command: - -```bash -export PLATFORM_URL="" -export USERNAME="" -export PASSWORD="" -export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" - -violet push "$PACKAGE_FILE" \ - --platform-address "$PLATFORM_URL" \ - --platform-username "$USERNAME" \ - --platform-password "$PASSWORD" \ - --clusters "$CLUSTER_NAME" \ - --target-catalog-source platform -``` - -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. - -### Confirm the Multus base - -SR-IOV networks are attached to KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. - -After installation, confirm that the operator and config-daemon are running: - -```bash -kubectl get pods -n cpaas-system -``` - -Expected workloads: - -```text -sriov-network-operator-xxxxx 1/1 Running -sriov-network-config-daemon-xxxxx 1/1 Running -``` - -### Configure VF resources - -A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a VM. - -First list the node states synchronized by the operator, and select the target node from the `NAME` column: - -```bash -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' -``` - -The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. - -Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`. - -Save the following content as `sriov-node-policy.yaml`: - -```yaml -apiVersion: sriovnetwork.openshift.io/v1 -kind: SriovNetworkNodePolicy -metadata: - name: sriov-vf-policy - namespace: cpaas-system -spec: - resourceName: sriov_vf - nodeSelector: - kubernetes.io/hostname: - nicSelector: - pfNames: - - ens5f0 - numVfs: 4 - deviceType: vfio-pci - mtu: 1500 -``` - -`deviceType: vfio-pci` is used for KubeVirt SR-IOV PCI passthrough. The operator configures the VF driver according to this policy and exposes the resource through the device plugin. Do not run `dpdk-devbind.py` inside the VM for host VFs. - -Apply the policy and watch node synchronization: - -```bash -kubectl apply -f sriov-node-policy.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -Confirm that the target node reaches `Succeeded`: - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' -``` - -If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. - -### Create an SR-IOV secondary network - -Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC. - -Save the following content as `sriov-network.yaml`: - -```yaml -apiVersion: sriovnetwork.openshift.io/v1 -kind: SriovNetwork -metadata: - name: vm-sriov-net - namespace: cpaas-system -spec: - networkNamespace: kubevirt - resourceName: sriov_vf - vlan: 0 - ipam: | - { - "type": "kube-ovn", - "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", - "provider": "vm-sriov-net.kubevirt.ovn" - } -``` - -The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Subnet `spec.provider`, and VM `metadata.namespace` together. - -Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan. - -Save the following content as `sriov-subnet.yaml`: - -```yaml -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: vm-sriov-net -spec: - protocol: IPv4 - enableDHCP: true - provider: vm-sriov-net.kubevirt.ovn - cidrBlock: 172.22.0.0/16 - gateway: 172.22.0.1 - excludeIps: - - 172.22.0.0..172.22.0.10 -``` - -The KubeVirt VM default network still uses the kube-ovn primary network. The SR-IOV network is attached as a Multus secondary NIC, and its address is allocated by the Kube-OVN Subnet above. - -Apply the objects and confirm that the NAD is generated: - -```bash -kubectl apply -f sriov-network.yaml -kubectl apply -f sriov-subnet.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -Inspect the effective NAD CNI configuration: - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### Use the SR-IOV network in a KubeVirt VM - -A VM can reference the same NAD through Multus. The following example shows only the network and interface-related fields: - -```yaml -apiVersion: kubevirt.io/v1 -kind: VirtualMachine -metadata: - name: sriov-vm - namespace: kubevirt -spec: - running: true - template: - spec: - domain: - devices: - interfaces: - - name: default - masquerade: {} - - name: sriov-net - sriov: {} - networks: - - name: default - pod: {} - - name: sriov-net - multus: - networkName: vm-sriov-net -``` - -In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. - -After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: - -```bash -kubectl get vmi -n kubevirt sriov-vm -kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide -``` - -Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. - -### Optional: Bind the service VF to DPDK inside the VM - -If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . - -Inside the VM, identify the PCI NICs: - -```bash -lspci -Dnn | grep -i ethernet -``` - -`dpdk-devbind.py` only binds the VF driver and does not prepare the DPDK application runtime. Configure HugePages, CPU affinity, and DPDK EAL parameters according to the workload image or application documentation. - -Load the VFIO driver: - -```bash -modprobe vfio-pci -``` - -Bind the service VF PCI address as seen inside the VM to `vfio-pci`: - -```bash -python3 dpdk-devbind.py --status -python3 dpdk-devbind.py -b vfio-pci -python3 dpdk-devbind.py --status -``` - -`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. If the service VF is owned by the DPDK application, subsequent IP configuration and packet processing are handled by that DPDK application. diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md new file mode 100644 index 000000000..920fb620d --- /dev/null +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,372 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# 如何在 ACP 上通过 Multus 和 SR-IOV 为 DPDK Pod 和 KubeVirt 虚拟机挂载高性能 VF + +## 问题 + +用户在 Alauda Container Platform 4.3 上运行 DPDK/CNF Pod 或 KubeVirt 虚拟机，并希望通过 Multus 挂载宿主机 SR-IOV VF 作为高性能业务网卡或 PCI 直通设备。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 + +本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的 VF 接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别说明 DPDK/CNF Pod 和 KubeVirt 虚拟机如何使用 `vfio-pci` 类型的 SR-IOV VF。 + +## 环境 + +本文适用于以下组合： + +| 组件 | 版本或说明 | +| --- | --- | +| Alauda Container Platform | 4.3 | +| 插件 | `sriov-network-plugin` | +| 插件包版本 | `sriov-network-plugin v4.3.1` | +| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | +| 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | +| 多网卡基座 | ACP 已提供 Multus 能力，Pod 或虚拟机通过 NAD 引用 SR-IOV VF | + +ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 + +本文覆盖 SR-IOV L5 插件在 DPDK/CNF Pod 和 KubeVirt 虚拟机 VF 接入场景中的安装与使用；不覆盖 OVS-DPDK、Userspace CNI 或 DPDK 应用内部参数配置。 + +## 先决条件 + +### 节点和硬件 + +至少需要一个满足以下条件的 worker 节点： + +- 节点上有支持 SR-IOV 的物理网卡 PF。 +- BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 +- PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 +- 节点内核已启用 VFIO 相关能力。至少确认 `vfio-pci` 模块可用： + +```bash +lsmod | grep vfio +modprobe vfio-pci +``` + +在后续创建 VF 后，记录目标 VF 的 PCI 地址，并确认该 VF 所在 IOMMU group 中没有混入不应直通给业务的其他设备： + +```bash +VF_PCI_ADDR="" +GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" +ls -l "$GROUP_PATH/devices" +``` + +如果同一个 group 内包含宿主机还需要使用的其他设备，不应直接将该 VF 用于 `vfio-pci`/PCI 直通场景，应先调整硬件、BIOS、内核 IOMMU 参数或网卡规划。 + +## 解决方案 + +### 安装插件 + +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 + +1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 +3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.1.tgz` 包。 +4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 +5. 将下载的插件包上传到目标 ACP 平台。 + +如果目标平台使用 `violet` 上传离线包，可以参考以下命令： + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 + +### 确认 Multus 基座可用 + +SR-IOV 网络通过 Multus 作为辅助网络接入 DPDK/CNF Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 + +安装后确认 operator 和 config-daemon 已运行： + +```bash +kubectl get pods -n cpaas-system +``` + +预期至少看到以下工作负载处于 `Running`： + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +### 确认节点上的 SR-IOV PF + +PF 是节点上的物理网卡，VF 是从 PF 创建出来并分配给 Pod 或虚拟机使用的虚拟 PCI 网卡。 + +先查看 operator 已同步的节点状态，并从输出的 `NAME` 选择目标节点： + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +在带 SR-IOV PF 的节点上，确认 operator 能发现物理网卡： + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 + +从 `status.interfaces[*].name` 输出中选择一个 PF，例如 `ens5f0`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。 + +后续 DPDK/CNF Pod 和 KubeVirt 虚拟机是两种使用示例。如果它们共用同一个 PF 和 `resourceName`，只需要创建一次 `SriovNetworkNodePolicy`；如果需要同时给不同业务分配独立 VF 池，应使用不同 PF，或规划不同的 `resourceName` 和 VF 数量。 + +### 为 DPDK/CNF Pod 配置并使用 SR-IOV VF + +DPDK/CNF Pod 需要由用户态进程直接接管 VF 时，`deviceType` 使用 `vfio-pci`。这种模式下 VF 不作为 Pod 内的普通 Linux 网卡使用，业务报文处理和地址配置由容器内的 DPDK/CNF 应用负责。 + +将以下内容保存为 `sriov-node-policy-pod.yaml`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-pod-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +应用后观察节点同步状态： + +```bash +kubectl apply -f sriov-node-policy-pod.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +确认目标节点变为 `Succeeded`： + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 + +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `default`。由于 VF 会被 DPDK/CNF 应用直接接管，这里不配置 kube-ovn IPAM。 + +将以下内容保存为 `sriov-network-pod.yaml`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: pod-sriov-net + namespace: cpaas-system +spec: + networkNamespace: default + resourceName: sriov_vfio +``` + +上例中 operator 会在 `default` 命名空间生成名为 `pod-sriov-net` 的 NAD。如果业务 Pod 运行在其他命名空间，需要同时替换 `SriovNetwork.spec.networkNamespace`、Pod `metadata.namespace` 和 Pod 中的 NAD 引用。 + +如果业务网络需要接入指定 VLAN，可在 `spec` 中增加 `vlan: `；不配置时使用未打标签网络。 + +Pod 的默认网络仍然使用 kube-ovn 主网络；SR-IOV VF 作为 Multus 辅助资源接入，并由 DPDK/CNF 应用使用。 + +应用后确认 NAD 已生成： + +```bash +kubectl apply -f sriov-network-pod.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n default pod-sriov-net +``` + +Pod 通过 `k8s.v1.cni.cncf.io/networks` annotation 引用 NAD，并通过 resource request 申请一个 SR-IOV VF。以下示例将 Pod 的默认网络保留为 kube-ovn，同时申请一个 `vfio-pci` 类型的 SR-IOV VF 给 DPDK/CNF 应用使用。 + +将以下内容保存为 `sriov-pod.yaml`： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-pod + namespace: default + annotations: + k8s.v1.cni.cncf.io/networks: default/pod-sriov-net +spec: + containers: + - name: app + image: nginx:latest + resources: + requests: + openshift.io/sriov_vfio: "1" + limits: + openshift.io/sriov_vfio: "1" +``` + +其中 `default/pod-sriov-net` 使用 `/` 格式；`openshift.io/sriov_vfio` 与 `SriovNetworkNodePolicy.spec.resourceName` 和 `SriovNetwork.spec.resourceName` 对应。 + +应用后确认 Pod 被调度到有 VF 资源的节点： + +```bash +kubectl apply -f sriov-pod.yaml +kubectl get pod -n default sriov-pod -o wide +``` + +如果业务使用 Deployment、StatefulSet 等控制器，在 Pod template 的 `metadata.annotations` 中设置 `k8s.v1.cni.cncf.io/networks`，并在业务容器的 `resources.requests` 和 `resources.limits` 中申请 `openshift.io/sriov_vfio`。VF 资源只负责把 PCI 设备分配给 Pod；DPDK 应用通常还需要 CPU、HugePages 和启动参数等配置，具体取值由业务镜像和 DPDK 应用文档决定。参考配置片段如下： + +```yaml +resources: + requests: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" + limits: + cpu: "4" + memory: 4Gi + hugepages-2Mi: 1Gi + openshift.io/sriov_vfio: "1" +volumeMounts: + - name: hugepage + mountPath: /dev/hugepages +volumes: + - name: hugepage + emptyDir: + medium: HugePages +``` + +### 为 KubeVirt 虚拟机配置并使用 SR-IOV 网络 + +KubeVirt SR-IOV PCI 直通场景需要把 VF 作为 PCI 设备传递给虚拟机，因此 `deviceType` 使用 `vfio-pci`。 + +如果已经为同一个 PF 创建过 `resourceName: sriov_vfio` 的 `SriovNetworkNodePolicy`，可直接继续创建虚拟机使用的 `SriovNetwork`。如果虚拟机需要使用独立 VF 池，再创建单独的 policy。 + +将以下内容保存为 `sriov-node-policy-vm.yaml`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vm-policy + namespace: cpaas-system +spec: + resourceName: sriov_vfio + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +应用后观察节点同步状态： + +```bash +kubectl apply -f sriov-node-policy-vm.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' +``` + +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`。如果该 VF 后续由虚拟机内的 DPDK 应用接管，不需要在这里配置 kube-ovn IPAM。 + +将以下内容保存为 `sriov-network-vm.yaml`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vfio +``` + +上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD。如果虚拟机运行在其他命名空间，需要同时替换 `SriovNetwork.spec.networkNamespace`、VM `metadata.namespace` 和 VM 中的 NAD 引用。 + +如果业务网络需要接入指定 VLAN，可在 `spec` 中增加 `vlan: `；不配置时使用未打标签网络。 + +KubeVirt 虚拟机的默认网络仍然使用 kube-ovn 主网络；SR-IOV VF 作为 Multus 辅助设备直通给虚拟机。 + +应用后确认 NAD 已生成： + +```bash +kubectl apply -f sriov-network-vm.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +虚拟机辅助网卡配置在 VM 模板的两处：`spec.template.spec.domain.devices.interfaces` 定义虚拟机内看到的网卡类型，`spec.template.spec.networks` 定义这张网卡连接到哪个 Multus NAD。两处的 `name` 必须一致。 + +在 VM 模板中同时增加 `interfaces[].sriov` 和 `networks[].multus`，即可为虚拟机挂载 SR-IOV 辅助网卡。以下示例只展示网络和接口相关字段： + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +上例中 VM 和 NAD 都在 `kubevirt` 命名空间，因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD，使用 `/` 格式。 + +创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +进入虚拟机操作系统后，确认出现直通的 SR-IOV VF。后续是否把该 VF 绑定给 DPDK、如何在虚拟机内部为 DPDK 应用分配 HugePages、如何配置 EAL 参数和业务进程，由虚拟机内的业务系统和 DPDK 应用文档决定。 diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md deleted file mode 100644 index 7fe740a3f..000000000 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -kind: - - How To -products: - - Alauda Container Platform -ProductsVersion: - - '4.3' ---- - -# 如何在 ACP 上通过 Multus 和 SR-IOV 为 KubeVirt 虚拟机提供高性能辅助网卡 - -## 问题 - -用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机，并希望通过 Multus 给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入虚拟机，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 - -本文从用户使用路径说明如何在 ACP 4.3 上完成 KubeVirt 虚拟机 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并在 KubeVirt 虚拟机中使用 SR-IOV 辅助网卡。同一 SR-IOV 能力也可用于业务 Pod，本文以 KubeVirt 虚拟机场景为例。 - -## 环境 - -本文适用于以下组合： - -| 组件 | 版本或说明 | -| --- | --- | -| Alauda Container Platform | 4.3 | -| 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.1` | -| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | -| 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | -| 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | -| 多网卡基座 | ACP 已提供 Multus 能力，虚拟机通过 NAD 引用 SR-IOV 辅助网络 | - -ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 - -本文覆盖 SR-IOV L5 插件在 KubeVirt 虚拟机辅助网卡场景中的安装与使用；不覆盖 OVS-DPDK、Userspace CNI 或容器内 DPDK 应用的配置。 - -## 先决条件 - -### 节点和硬件 - -至少需要一个满足以下条件的 worker 节点： - -- 节点上有支持 SR-IOV 的物理网卡 PF。 -- BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 -- PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 - -## 解决方案 - -### 安装插件 - -该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 - -1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 -3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.1.tgz`。 -4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 -5. 将下载的插件包上传到目标 ACP 平台。 - -如果目标平台使用 `violet` 上传离线包，可以参考以下命令： - -```bash -export PLATFORM_URL="" -export USERNAME="" -export PASSWORD="" -export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" - -violet push "$PACKAGE_FILE" \ - --platform-address "$PLATFORM_URL" \ - --platform-username "$USERNAME" \ - --platform-password "$PASSWORD" \ - --clusters "$CLUSTER_NAME" \ - --target-catalog-source platform -``` - -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 - -### 确认 Multus 基座可用 - -SR-IOV 网络通过 Multus 作为辅助网卡接入 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 - -安装后确认 operator 和 config-daemon 已运行： - -```bash -kubectl get pods -n cpaas-system -``` - -预期至少看到以下工作负载处于 `Running`： - -```text -sriov-network-operator-xxxxx 1/1 Running -sriov-network-config-daemon-xxxxx 1/1 Running -``` - -### 配置 VF 资源 - -PF 是节点上的物理网卡，VF 是从 PF 创建出来并分配给虚拟机使用的虚拟 PCI 网卡。 - -先查看 operator 已同步的节点状态，并从输出的 `NAME` 选择目标节点： - -```bash -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -在带 SR-IOV PF 的节点上，确认 operator 能发现物理网卡： - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' -``` - -operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 - -从 `status.interfaces[*].name` 输出中选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源。 - -将以下内容保存为 `sriov-node-policy.yaml`： - -```yaml -apiVersion: sriovnetwork.openshift.io/v1 -kind: SriovNetworkNodePolicy -metadata: - name: sriov-vf-policy - namespace: cpaas-system -spec: - resourceName: sriov_vf - nodeSelector: - kubernetes.io/hostname: - nicSelector: - pfNames: - - ens5f0 - numVfs: 4 - deviceType: vfio-pci - mtu: 1500 -``` - -`deviceType: vfio-pci` 用于 KubeVirt SR-IOV PCI 直通场景。operator 会根据该策略配置 VF 驱动并通过 device plugin 暴露资源；不要在虚拟机内部对宿主机 VF 执行 `dpdk-devbind.py`。 - -应用后观察节点同步状态： - -```bash -kubectl apply -f sriov-node-policy.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -确认目标节点变为 `Succeeded`： - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -确认节点资源中出现 SR-IOV device plugin 暴露的资源： - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' -``` - -如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 - -### 创建 SR-IOV 辅助网络 - -创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址。 - -将以下内容保存为 `sriov-network.yaml`： - -```yaml -apiVersion: sriovnetwork.openshift.io/v1 -kind: SriovNetwork -metadata: - name: vm-sriov-net - namespace: cpaas-system -spec: - networkNamespace: kubevirt - resourceName: sriov_vf - vlan: 0 - ipam: | - { - "type": "kube-ovn", - "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", - "provider": "vm-sriov-net.kubevirt.ovn" - } -``` - -其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。如果虚拟机运行在其他命名空间，需要同时替换 `SriovNetwork.spec.networkNamespace`、Subnet `spec.provider` 和 VM `metadata.namespace`。 - -创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整。 - -将以下内容保存为 `sriov-subnet.yaml`： - -```yaml -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: vm-sriov-net -spec: - protocol: IPv4 - enableDHCP: true - provider: vm-sriov-net.kubevirt.ovn - cidrBlock: 172.22.0.0/16 - gateway: 172.22.0.1 - excludeIps: - - 172.22.0.0..172.22.0.10 -``` - -KubeVirt 虚拟机的默认网络仍然使用 kube-ovn 主网络；SR-IOV 网络作为 Multus 辅助网卡接入，并通过上述 kube-ovn Subnet 完成辅助网卡 IPAM。 - -应用后确认 NAD 已生成： - -```bash -kubectl apply -f sriov-network.yaml -kubectl apply -f sriov-subnet.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -查看 NAD 的有效 CNI 配置： - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 - -虚拟机可以通过 Multus 引用同一个 NAD。以下示例只展示网络和接口相关字段： - -```yaml -apiVersion: kubevirt.io/v1 -kind: VirtualMachine -metadata: - name: sriov-vm - namespace: kubevirt -spec: - running: true - template: - spec: - domain: - devices: - interfaces: - - name: default - masquerade: {} - - name: sriov-net - sriov: {} - networks: - - name: default - pod: {} - - name: sriov-net - multus: - networkName: vm-sriov-net -``` - -上例中 VM 和 NAD 都在 `kubevirt` 命名空间，因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD，使用 `/` 格式。 - -创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： - -```bash -kubectl get vmi -n kubevirt sriov-vm -kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide -``` - -进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 - -### 在虚拟机内将业务 VF 绑定给 DPDK（可选） - -如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`；如果镜像中没有该脚本，可以从 DPDK 官方仓库获取：。 - -在虚拟机内确认 PCI 网卡： - -```bash -lspci -Dnn | grep -i ethernet -``` - -`dpdk-devbind.py` 只负责 VF 驱动绑定，不负责准备 DPDK 应用运行环境。HugePages、CPU 亲和性和 DPDK EAL 参数按业务镜像或应用文档配置。 - -加载 VFIO 驱动： - -```bash -modprobe vfio-pci -``` - -将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`： - -```bash -python3 dpdk-devbind.py --status -python3 dpdk-devbind.py -b vfio-pci -python3 dpdk-devbind.py --status -``` - -`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。如果业务 VF 被 DPDK 应用接管，后续 IP 配置和报文处理由 DPDK 应用决定。 From fbc3193498f483e414681d1577090a41094e42ed Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 8 Jul 2026 10:37:19 +0800 Subject: [PATCH 09/18] docs: refine SR-IOV VF prerequisites --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 22 ++++++++++++------- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 22 ++++++++++++------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 827d679a3..3ed99e509 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -29,8 +29,6 @@ This article applies to the following combination: | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | | Multi-NIC base | ACP provides Multus capability; Pods or VMs reference the SR-IOV VF through NAD | -The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. - This article covers installing and using the SR-IOV L5 plugin for DPDK/CNF Pod and KubeVirt VM VF attachment. It does not cover OVS-DPDK, Userspace CNI, or internal DPDK application parameters. ## Prerequisites @@ -49,16 +47,12 @@ lsmod | grep vfio modprobe vfio-pci ``` -After VFs are created later in the procedure, record the PCI address of the target VF and confirm that its IOMMU group does not include other devices that should not be passed through to the workload: +To load the module automatically after node reboot, write it to the system module-load configuration: ```bash -VF_PCI_ADDR="" -GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" -ls -l "$GROUP_PATH/devices" +echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ``` -If the same group contains other devices still required by the host, do not use that VF directly for `vfio-pci` or PCI passthrough. Adjust the hardware, BIOS settings, kernel IOMMU parameters, or NIC planning first. - ## Resolution ### Install the plugin @@ -179,6 +173,16 @@ kubectl get node \ If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. +Record the PCI address of the target VF and confirm that its IOMMU group does not include other devices that should not be passed through to the workload: + +```bash +VF_PCI_ADDR="" +GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" +ls -l "$GROUP_PATH/devices" +``` + +If the same group contains other devices still required by the host, do not use that VF directly for `vfio-pci` or PCI passthrough. Adjust the hardware, BIOS settings, kernel IOMMU parameters, or NIC planning first. + Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `default`. Because the VF is owned directly by the DPDK/CNF application, Kube-OVN IPAM is not configured here. Save the following content as `sriov-network-pod.yaml`: @@ -303,6 +307,8 @@ kubectl get node \ -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' ``` +If the target VF IOMMU group was not checked earlier, complete the same IOMMU group check before creating the `SriovNetwork` for the VM. + Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt`. If the VF is later owned by a DPDK application inside the VM, Kube-OVN IPAM is not required here. Save the following content as `sriov-network-vm.yaml`: diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 920fb620d..4149e6dca 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -29,8 +29,6 @@ ProductsVersion: | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | | 多网卡基座 | ACP 已提供 Multus 能力，Pod 或虚拟机通过 NAD 引用 SR-IOV VF | -ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 - 本文覆盖 SR-IOV L5 插件在 DPDK/CNF Pod 和 KubeVirt 虚拟机 VF 接入场景中的安装与使用；不覆盖 OVS-DPDK、Userspace CNI 或 DPDK 应用内部参数配置。 ## 先决条件 @@ -49,16 +47,12 @@ lsmod | grep vfio modprobe vfio-pci ``` -在后续创建 VF 后，记录目标 VF 的 PCI 地址，并确认该 VF 所在 IOMMU group 中没有混入不应直通给业务的其他设备： +如果需要节点重启后自动加载，可将模块写入系统模块加载配置： ```bash -VF_PCI_ADDR="" -GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" -ls -l "$GROUP_PATH/devices" +echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ``` -如果同一个 group 内包含宿主机还需要使用的其他设备，不应直接将该 VF 用于 `vfio-pci`/PCI 直通场景，应先调整硬件、BIOS、内核 IOMMU 参数或网卡规划。 - ## 解决方案 ### 安装插件 @@ -179,6 +173,16 @@ kubectl get node \ 如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 +记录目标 VF 的 PCI 地址，并确认该 VF 所在 IOMMU group 中没有混入不应直通给业务的其他设备： + +```bash +VF_PCI_ADDR="" +GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" +ls -l "$GROUP_PATH/devices" +``` + +如果同一个 group 内包含宿主机还需要使用的其他设备，不应直接将该 VF 用于 `vfio-pci`/PCI 直通场景，应先调整硬件、BIOS、内核 IOMMU 参数或网卡规划。 + 创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `default`。由于 VF 会被 DPDK/CNF 应用直接接管，这里不配置 kube-ovn IPAM。 将以下内容保存为 `sriov-network-pod.yaml`： @@ -303,6 +307,8 @@ kubectl get node \ -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' ``` +如果前面没有检查过目标 VF 的 IOMMU group，在继续创建虚拟机使用的 `SriovNetwork` 前，先完成同样的 IOMMU group 检查。 + 创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`。如果该 VF 后续由虚拟机内的 DPDK 应用接管，不需要在这里配置 kube-ovn IPAM。 将以下内容保存为 `sriov-network-vm.yaml`： From 8177416583e32fbd25fdbf22562512601b24e2ea Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 8 Jul 2026 17:20:39 +0800 Subject: [PATCH 10/18] ACP-53814 update SR-IOV plugin package version --- ..._IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 12 ++++++------ ..._IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 3ed99e509..4646cfb18 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ This article applies to the following combination: | --- | --- | | Alauda Container Platform | 4.3 | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.1` | +| Plugin package version | `sriov-network-plugin v4.3.3` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | @@ -57,11 +57,11 @@ echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ### Install the plugin -This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.3`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. -3. Download the `sriov-network-plugin.*.v4.3.1.tgz` package that matches the target platform architecture. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.3`. +3. Download the `sriov-network-plugin.*.v4.3.3.tgz` package that matches the target platform architecture. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -72,7 +72,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.3.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -82,7 +82,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.3` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. ### Confirm the Multus base diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 4149e6dca..7e51beb72 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ ProductsVersion: | --- | --- | | Alauda Container Platform | 4.3 | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.1` | +| 插件包版本 | `sriov-network-plugin v4.3.3` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | @@ -57,11 +57,11 @@ echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ### 安装插件 -该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.3`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 -3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.1.tgz` 包。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.3` 的安装包。 +3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.3.tgz` 包。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -72,7 +72,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.3.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -82,7 +82,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.3` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 ### 确认 Multus 基座可用 From eb617341d4eadc7d4f4dc1b1a27da526034025e2 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:32:39 +0800 Subject: [PATCH 11/18] ACP-53814 update SR-IOV VF usage guide --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 128 +++++++++++++++-- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 129 ++++++++++++++++-- 2 files changed, 236 insertions(+), 21 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 4646cfb18..6c96d8a43 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ This article applies to the following combination: | --- | --- | | Alauda Container Platform | 4.3 | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.3` | +| Plugin package version | `sriov-network-plugin v4.3.7` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | @@ -40,7 +40,23 @@ Prepare at least one worker node that meets these requirements: - The node has a physical NIC PF that supports SR-IOV. - IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. - The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. -- The node kernel has VFIO support enabled. At minimum, confirm that the `vfio-pci` module is available: + +On the node, use the following checks to confirm that IOMMU is enabled: + +```bash +cat /proc/cmdline +dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi' +``` + +`/proc/cmdline` usually contains a parameter such as `intel_iommu=on` or `amd_iommu=on`, and `dmesg` should include IOMMU initialization logs. After a VF is created, also confirm that the VF has an IOMMU group: + +```bash +readlink -f /sys/bus/pci/devices//iommu_group +``` + +If the path does not exist, the node usually does not have usable IOMMU isolation, and the `vfio-pci` or PCI passthrough scenario should not continue. + +The node kernel must also have VFIO support enabled. At minimum, confirm that the `vfio-pci` module is available: ```bash lsmod | grep vfio @@ -53,15 +69,25 @@ To load the module automatically after node reboot, write it to the system modul echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ``` +### Label SR-IOV nodes + +`sriov-network-config-daemon` is scheduled only to nodes labeled with `feature.node.kubernetes.io/sriov-capable=true` by default. Before installing the plugin, label the nodes that actually have SR-IOV PFs: + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true --overwrite +``` + +Apply this label only to nodes where VFs will be configured. Nodes without this label do not run `sriov-network-config-daemon` and do not get a corresponding `SriovNetworkNodeState`. + ## Resolution ### Install the plugin -This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.3`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.7`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.3`. -3. Download the `sriov-network-plugin.*.v4.3.3.tgz` package that matches the target platform architecture. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.7`. +3. Download the `sriov-network-plugin.*.v4.3.7.tgz` package that matches the target platform architecture. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -72,7 +98,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.3.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.7.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -82,7 +108,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.3` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.7` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. ### Confirm the Multus base @@ -101,6 +127,8 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` +If `sriov-network-config-daemon` is not visible, first confirm that the target node is labeled with `feature.node.kubernetes.io/sriov-capable=true`. + ### Confirm SR-IOV PFs on nodes A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a Pod or VM. @@ -112,6 +140,8 @@ kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` +If a node with an SR-IOV NIC is missing from the list, first check whether the node has the `feature.node.kubernetes.io/sriov-capable=true` label. The config-daemon collects PF status only on nodes that match this label. + On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: ```bash @@ -125,9 +155,57 @@ Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`. `node The following DPDK/CNF Pod and KubeVirt VM sections are two usage examples. If they share the same PF and `resourceName`, create the `SriovNetworkNodePolicy` only once. If different workloads need independent VF pools at the same time, use different PFs or plan different `resourceName` values and VF counts. +### Optional: Handle NICs not in the default supported list + +If `sriov-network-config-daemon` keeps logging messages like the following, the operator is running, but the NIC PCI ID is not in the default supported list. The operator skips that PF: + +```text +DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: 'hinic' ... product: 'Hi1822 Family (4*25GE)'"} +IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} +``` + +Add the NIC to the `supported-nic-ids` allowlist. The entry format is: + +```text +: " " +``` + +For example, a Huawei Hi1822 PF has PCI ID `19e5:1822`, and its VF has PCI ID `19e5:375e`. The entry is: + +```yaml +supportedExtraNICs: + - 'Huawei_Hi1822: "19e5 1822 375e"' +``` + +If the VF device ID is unknown, temporarily create one VF during a maintenance window and inspect it. In the following example, `0000:3d:00.0` is the PF PCI address: + +```bash +echo 1 > /sys/bus/pci/devices/0000:3d:00.0/sriov_numvfs +readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 +lspci -Dnn -s +``` + +For on-site validation, you can temporarily patch the ConfigMap in an installed cluster and restart the operator and config-daemon: + +```bash +kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ + '{"data":{"Huawei_Hi1822":"19e5 1822 375e"}}' + +kubectl rollout restart deployment/sriov-network-operator -n cpaas-system +kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system +``` + +This patch is only suitable for validation and can be lost after plugin upgrade or reinstall. For production delivery, persist the entry through `supportedExtraNICs` in the plugin installation parameters. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: + +```bash +echo 0 > /sys/bus/pci/devices//sriov_numvfs +``` + +Otherwise, config-daemon may report that the PF already has VFs that were not created by the sriov operator and skip part of the change flow. + ### Configure and use an SR-IOV VF for DPDK/CNF Pods -When a DPDK/CNF Pod needs a userspace process to own the VF directly, set `deviceType` to `vfio-pci`. In this mode, the VF is not used as a regular Linux network interface inside the Pod. Packet processing and address handling are owned by the DPDK/CNF application in the container. +When a DPDK/CNF Pod needs a userspace process to own the VF directly, set `deviceType` to `vfio-pci`. In this mode, the VF is not used as a regular Linux network interface inside the Pod. Packet processing and address handling are owned by the DPDK/CNF application in the container. Therefore, it is expected that `ip a` inside the Pod does not show a secondary interface such as `net1`. The validation target is a `/dev/vfio/` device inside the container. Save the following content as `sriov-node-policy-pod.yaml`: @@ -144,7 +222,7 @@ spec: nicSelector: pfNames: - ens5f0 - numVfs: 4 + numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` @@ -243,6 +321,36 @@ kubectl apply -f sriov-pod.yaml kubectl get pod -n default sriov-pod -o wide ``` +Confirm that the container has received the VFIO device: + +```bash +kubectl exec -n default sriov-pod -- ls -l /dev/vfio/ +``` + +The expected output includes the `vfio` control device and one numeric IOMMU group device, for example: + +```text +crw------- 1 root root 234, 191 ... 191 +crw-rw-rw- 1 root root 10, 196 ... vfio +``` + +The numeric device, such as `191`, is the VFIO group available to the current Pod. In this scenario, do not use the presence of a secondary interface in `ip a` as the success criterion. + +If `dpdk-devbind.py -s` is run inside the container or the Pod namespace, it may list all PF/VF PCI devices on the node. The script reads `/sys/bus/pci/devices`, and PCI sysfs is not filtered by Pod resource allocation. To determine which VF the current Pod can actually use, rely on the devices exposed under `/dev/vfio/` and the `openshift.io/sriov_vfio` resource allocated to the Pod. + +The following is a reference observation: when `numVfs: 8` creates eight VFs on the node, all eight VFs are bound to `vfio-pci`, so `dpdk-devbind.py -s` can list eight VFs. However, when a single Pod requests only one `openshift.io/sriov_vfio`, the container exposes only one numeric group device under `/dev/vfio/`. + +```text +Network devices using DPDK-compatible driver +============================================ +0000:3d:00.1 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +0000:3d:00.2 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +... +0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +``` + +This output only shows the VF driver binding status on the node. The number of VFs actually available to the Pod is still determined by `/dev/vfio/` and the resource request count. + If the workload uses Deployment, StatefulSet, or another controller, set `k8s.v1.cni.cncf.io/networks` in the Pod template `metadata.annotations`, and request `openshift.io/sriov_vfio` in the application container `resources.requests` and `resources.limits`. The VF resource only assigns the PCI device to the Pod. DPDK applications usually also need CPU, HugePages, and startup-parameter configuration; the exact values are determined by the business image and DPDK application documentation. A reference Pod fragment is: ```yaml @@ -287,7 +395,7 @@ spec: nicSelector: pfNames: - ens5f0 - numVfs: 4 + numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 7e51beb72..f03e155ac 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ ProductsVersion: | --- | --- | | Alauda Container Platform | 4.3 | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.3` | +| 插件包版本 | `sriov-network-plugin v4.3.7` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | @@ -39,8 +39,23 @@ ProductsVersion: - 节点上有支持 SR-IOV 的物理网卡 PF。 - BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 -- PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 -- 节点内核已启用 VFIO 相关能力。至少确认 `vfio-pci` 模块可用： + +在节点上可以通过以下方式确认 IOMMU 已启用： + +```bash +cat /proc/cmdline +dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi' +``` + +`/proc/cmdline` 中通常应包含 `intel_iommu=on` 或 `amd_iommu=on` 等参数；`dmesg` 中应能看到 IOMMU 初始化相关日志。创建 VF 后，还可以确认 VF 存在 IOMMU group： + +```bash +readlink -f /sys/bus/pci/devices//iommu_group +``` + +如果该路径不存在，通常表示当前节点没有可用的 IOMMU 隔离，`vfio-pci`/PCI 直通场景不应继续配置。 + +节点内核还需要启用 VFIO 相关能力。至少确认 `vfio-pci` 模块可用： ```bash lsmod | grep vfio @@ -53,15 +68,25 @@ modprobe vfio-pci echo vfio-pci > /etc/modules-load.d/vfio-pci.conf ``` +### 标记 SR-IOV 节点 + +`sriov-network-config-daemon` 默认只调度到带有 `feature.node.kubernetes.io/sriov-capable=true` 标签的节点。安装插件前，先给实际带 SR-IOV PF 的节点打标签： + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true --overwrite +``` + +只给计划配置 VF 的节点打这个标签。没有该标签的节点不会运行 `sriov-network-config-daemon`，也不会生成对应的 `SriovNetworkNodeState`。 + ## 解决方案 ### 安装插件 -该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.3`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.7`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.3` 的安装包。 -3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.3.tgz` 包。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.7` 的安装包。 +3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.7.tgz` 包。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -72,7 +97,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.3.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.7.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -82,7 +107,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.3` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.7` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 ### 确认 Multus 基座可用 @@ -101,6 +126,8 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` +如果看不到 `sriov-network-config-daemon`，先确认目标节点已经带有 `feature.node.kubernetes.io/sriov-capable=true` 标签。 + ### 确认节点上的 SR-IOV PF PF 是节点上的物理网卡，VF 是从 PF 创建出来并分配给 Pod 或虚拟机使用的虚拟 PCI 网卡。 @@ -112,6 +139,8 @@ kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` +如果带 SR-IOV 网卡的节点没有出现在列表中，先检查该节点是否已经打上 `feature.node.kubernetes.io/sriov-capable=true` 标签；config-daemon 只会在该标签匹配的节点上采集 PF 状态。 + 在带 SR-IOV PF 的节点上，确认 operator 能发现物理网卡： ```bash @@ -125,9 +154,57 @@ operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeStat 后续 DPDK/CNF Pod 和 KubeVirt 虚拟机是两种使用示例。如果它们共用同一个 PF 和 `resourceName`，只需要创建一次 `SriovNetworkNodePolicy`；如果需要同时给不同业务分配独立 VF 池，应使用不同 PF，或规划不同的 `resourceName` 和 VF 数量。 +### 处理不在默认支持列表中的网卡（可选） + +如果 `sriov-network-config-daemon` 日志持续出现类似以下信息，说明 operator 已经运行，但该网卡的 PCI ID 不在默认支持列表中，operator 会跳过该 PF： + +```text +DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: 'hinic' ... product: 'Hi1822 Family (4*25GE)'"} +IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} +``` + +这类场景需要把网卡加入 `supported-nic-ids` 白名单。白名单条目的格式为： + +```text +<名称>: " " +``` + +例如 Huawei Hi1822 PF 的 PCI ID 是 `19e5:1822`，创建出的 VF PCI ID 是 `19e5:375e`，则条目为： + +```yaml +supportedExtraNICs: + - 'Huawei_Hi1822: "19e5 1822 375e"' +``` + +如果还不知道 VF device ID，可以在维护窗口中临时创建 1 个 VF 后查看。以下示例中的 `0000:3d:00.0` 是 PF 的 PCI 地址： + +```bash +echo 1 > /sys/bus/pci/devices/0000:3d:00.0/sriov_numvfs +readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 +lspci -Dnn -s +``` + +如果只是现场验证，可以临时 patch 已安装集群中的 ConfigMap，并重启 operator 和 config-daemon： + +```bash +kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ + '{"data":{"Huawei_Hi1822":"19e5 1822 375e"}}' + +kubectl rollout restart deployment/sriov-network-operator -n cpaas-system +kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system +``` + +这种 patch 只适合验证，插件升级或重装后可能丢失；正式交付应在插件安装参数中通过 `supportedExtraNICs` 固化。若为了查看 VF ID 手动写过 `sriov_numvfs`，在让 operator 通过 `SriovNetworkNodePolicy` 接管前，先清理手工创建的 VF： + +```bash +echo 0 > /sys/bus/pci/devices//sriov_numvfs +``` + +否则 config-daemon 可能提示该 PF 已有 VF 但不是由 sriov operator 创建，并跳过部分变更流程。 + ### 为 DPDK/CNF Pod 配置并使用 SR-IOV VF -DPDK/CNF Pod 需要由用户态进程直接接管 VF 时，`deviceType` 使用 `vfio-pci`。这种模式下 VF 不作为 Pod 内的普通 Linux 网卡使用，业务报文处理和地址配置由容器内的 DPDK/CNF 应用负责。 +DPDK/CNF Pod 需要由用户态进程直接接管 VF 时，`deviceType` 使用 `vfio-pci`。这种模式下 VF 不作为 Pod 内的普通 Linux 网卡使用，业务报文处理和地址配置由容器内的 DPDK/CNF 应用负责。因此，Pod 内 `ip a` 看不到 `net1` 之类的辅助网卡是正常现象；验证目标是容器内出现 `/dev/vfio/` 设备。 将以下内容保存为 `sriov-node-policy-pod.yaml`： @@ -144,7 +221,7 @@ spec: nicSelector: pfNames: - ens5f0 - numVfs: 4 + numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` @@ -243,6 +320,36 @@ kubectl apply -f sriov-pod.yaml kubectl get pod -n default sriov-pod -o wide ``` +确认容器内已经拿到 VFIO 设备： + +```bash +kubectl exec -n default sriov-pod -- ls -l /dev/vfio/ +``` + +预期至少看到 `vfio` 控制设备和一个数字命名的 IOMMU group 设备，例如： + +```text +crw------- 1 root root 234, 191 ... 191 +crw-rw-rw- 1 root root 10, 196 ... vfio +``` + +其中数字设备（如 `191`）表示当前 Pod 可用的 VFIO group。该场景下不要使用 `ip a` 是否出现辅助网卡作为成功标准。 + +如果在容器或 Pod 命名空间中运行 `dpdk-devbind.py -s`，可能会看到节点上的所有 PF/VF PCI 设备；这是因为脚本读取 `/sys/bus/pci/devices`，PCI sysfs 不会按 Pod 资源分配过滤。判断当前 Pod 实际可用的 VF，应以 `/dev/vfio/` 下暴露的设备和 Pod 申请到的 `openshift.io/sriov_vfio` 资源为准。 + +以下是一个参考现象：节点上通过 `numVfs: 8` 创建出的 8 个 VF 都绑定到 `vfio-pci`，因此 `dpdk-devbind.py -s` 可以看到 8 个 VF；但单个 Pod 只申请 1 个 `openshift.io/sriov_vfio` 时，容器内 `/dev/vfio/` 只暴露 1 个数字 group 设备。 + +```text +Network devices using DPDK-compatible driver +============================================ +0000:3d:00.1 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +0000:3d:00.2 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +... +0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic +``` + +该输出只能说明节点上的 VF 驱动绑定状态；Pod 实际拿到几个 VF，仍以 `/dev/vfio/` 和资源申请数量为准。 + 如果业务使用 Deployment、StatefulSet 等控制器，在 Pod template 的 `metadata.annotations` 中设置 `k8s.v1.cni.cncf.io/networks`，并在业务容器的 `resources.requests` 和 `resources.limits` 中申请 `openshift.io/sriov_vfio`。VF 资源只负责把 PCI 设备分配给 Pod；DPDK 应用通常还需要 CPU、HugePages 和启动参数等配置，具体取值由业务镜像和 DPDK 应用文档决定。参考配置片段如下： ```yaml @@ -287,7 +394,7 @@ spec: nicSelector: pfNames: - ens5f0 - numVfs: 4 + numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` From 4dba17e38265089cce29ca17812f846a339fab15 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:34:57 +0800 Subject: [PATCH 12/18] ACP-53814 simplify SR-IOV policy guidance --- ...ring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 6 +----- ...ring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 6 +----- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 6c96d8a43..80b0bdc74 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -149,11 +149,7 @@ kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. - -Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. - -The following DPDK/CNF Pod and KubeVirt VM sections are two usage examples. If they share the same PF and `resourceName`, create the `SriovNetworkNodePolicy` only once. If different workloads need independent VF pools at the same time, use different PFs or plan different `resourceName` values and VF counts. +Select the target PF from the `status.interfaces[*].name` output, such as `ens5f0`, and use `SriovNetworkNodePolicy` to choose the nodes, PF, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools with different PFs, `resourceName` values, or VF counts. ### Optional: Handle NICs not in the default supported list diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index f03e155ac..b31836476 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -148,11 +148,7 @@ kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 - -从 `status.interfaces[*].name` 输出中选择一个 PF，例如 `ens5f0`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。 - -后续 DPDK/CNF Pod 和 KubeVirt 虚拟机是两种使用示例。如果它们共用同一个 PF 和 `resourceName`，只需要创建一次 `SriovNetworkNodePolicy`；如果需要同时给不同业务分配独立 VF 池，应使用不同 PF，或规划不同的 `resourceName` 和 VF 数量。 +从 `status.interfaces[*].name` 输出中选择要使用的 PF，例如 `ens5f0`，并通过 `SriovNetworkNodePolicy` 指定在哪些节点、哪个 PF 上创建多少 VF，以及使用什么 `resourceName` 和 `deviceType`。后续 DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy；只有需要独立 VF 池时，才需要规划不同 PF、`resourceName` 或 VF 数量。 ### 处理不在默认支持列表中的网卡（可选） From 285b47a9acec4c569ad6904016ee64dfce7b5c50 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:37:43 +0800 Subject: [PATCH 13/18] ACP-53814 clarify SR-IOV PF selection --- ...figuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 2 +- ...figuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 80b0bdc74..f5d38a734 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -149,7 +149,7 @@ kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -Select the target PF from the `status.interfaces[*].name` output, such as `ens5f0`, and use `SriovNetworkNodePolicy` to choose the nodes, PF, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools with different PFs, `resourceName` values, or VF counts. +Record the PF name to use, such as `ens5f0`. Later, the `SriovNetworkNodePolicy` specifies the target nodes, PF name, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools. ### Optional: Handle NICs not in the default supported list diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index b31836476..a3c02e3e3 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -148,7 +148,7 @@ kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -从 `status.interfaces[*].name` 输出中选择要使用的 PF，例如 `ens5f0`，并通过 `SriovNetworkNodePolicy` 指定在哪些节点、哪个 PF 上创建多少 VF，以及使用什么 `resourceName` 和 `deviceType`。后续 DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy；只有需要独立 VF 池时，才需要规划不同 PF、`resourceName` 或 VF 数量。 +记录要使用的 PF 名称，例如 `ens5f0`。后续创建 `SriovNetworkNodePolicy` 时，会在 policy 中指定目标节点、PF 名称、VF 数量、`resourceName` 和 `deviceType`。DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy；只有需要独立 VF 池时，才需要创建不同的 policy。 ### 处理不在默认支持列表中的网卡（可选） From 5ae31d474a77bc4c08d1d75b6474372cd8997000 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:39:30 +0800 Subject: [PATCH 14/18] ACP-53814 align SR-IOV PF output example --- ...SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 10 ++++++++-- ...SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 10 ++++++++-- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index f5d38a734..cbff947d6 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -146,10 +146,16 @@ On a node with an SR-IOV PF, confirm that the operator discovers the physical NI ```bash kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\n"}{end}' ``` -Record the PF name to use, such as `ens5f0`. Later, the `SriovNetworkNodePolicy` specifies the target nodes, PF name, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools. +The output format is ` `, for example: + +```text +p1p1 0000:3d:00.0 19e5 1822 +``` + +Record the PF name to use, such as `p1p1`. Later, the `SriovNetworkNodePolicy` specifies the target nodes, PF name, VF count, `resourceName`, and `deviceType`. The following DPDK/CNF Pod and KubeVirt VM examples can share the same policy. Create a separate policy only when the workloads need independent VF pools. ### Optional: Handle NICs not in the default supported list diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index a3c02e3e3..975d9d9bc 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -145,10 +145,16 @@ kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ ```bash kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\n"}{end}' ``` -记录要使用的 PF 名称，例如 `ens5f0`。后续创建 `SriovNetworkNodePolicy` 时，会在 policy 中指定目标节点、PF 名称、VF 数量、`resourceName` 和 `deviceType`。DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy；只有需要独立 VF 池时，才需要创建不同的 policy。 +输出格式为 ` `，例如： + +```text +p1p1 0000:3d:00.0 19e5 1822 +``` + +记录要使用的 PF 名称，例如 `p1p1`。后续创建 `SriovNetworkNodePolicy` 时，会在 policy 中指定目标节点、PF 名称、VF 数量、`resourceName` 和 `deviceType`。DPDK/CNF Pod 和 KubeVirt 虚拟机示例可以共用同一个 policy；只有需要独立 VF 池时，才需要创建不同的 policy。 ### 处理不在默认支持列表中的网卡（可选） From b5d8406bb107916da2b4ed819c5aa2f8cc3f2cc0 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:42:13 +0800 Subject: [PATCH 15/18] ACP-53814 clarify SR-IOV NIC allowlist scope --- ...figuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 2 +- ...figuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index cbff947d6..610f6a7e4 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -197,7 +197,7 @@ kubectl rollout restart deployment/sriov-network-operator -n cpaas-system kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system ``` -This patch is only suitable for validation and can be lost after plugin upgrade or reinstall. For production delivery, persist the entry through `supportedExtraNICs` in the plugin installation parameters. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: +This patch is only suitable for on-site validation and can be lost after plugin upgrade or reinstall. NIC models and VF device IDs can differ between customer environments, so this entry should not be added as a universal default. To keep it long term, configure `supportedExtraNICs` in the plugin installation or upgrade parameters with the PCI IDs confirmed in that environment. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: ```bash echo 0 > /sys/bus/pci/devices//sriov_numvfs diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 975d9d9bc..fc51f8a4a 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -196,7 +196,7 @@ kubectl rollout restart deployment/sriov-network-operator -n cpaas-system kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system ``` -这种 patch 只适合验证，插件升级或重装后可能丢失；正式交付应在插件安装参数中通过 `supportedExtraNICs` 固化。若为了查看 VF ID 手动写过 `sriov_numvfs`，在让 operator 通过 `SriovNetworkNodePolicy` 接管前，先清理手工创建的 VF： +这种 patch 只适合现场验证，插件升级或重装后可能丢失。不同客户环境的网卡型号和 VF device ID 可能不同，不应把该条目写成通用默认值；需要长期保留时，应按现场确认的 PCI ID，在该环境的插件安装或升级参数中通过 `supportedExtraNICs` 配置。若为了查看 VF ID 手动写过 `sriov_numvfs`，在让 operator 通过 `SriovNetworkNodePolicy` 接管前，先清理手工创建的 VF： ```bash echo 0 > /sys/bus/pci/devices//sriov_numvfs From b8c847763d14c836ef45bc7adabf06a8b569b5ab Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:48:16 +0800 Subject: [PATCH 16/18] ACP-53814 simplify SR-IOV policy validation --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 26 ++++--------------- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 26 ++++--------------- 2 files changed, 10 insertions(+), 42 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 610f6a7e4..2d3dd775c 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -229,19 +229,10 @@ spec: mtu: 1500 ``` -Apply the policy and watch node synchronization: +Apply the policy: ```bash kubectl apply -f sriov-node-policy-pod.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -Confirm that the target node reaches `Succeeded`: - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: @@ -253,16 +244,13 @@ kubectl get node \ If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. -Record the PCI address of the target VF and confirm that its IOMMU group does not include other devices that should not be passed through to the workload: +If the resource does not appear for a long time, check the target node synchronization status and error message: ```bash -VF_PCI_ADDR="" -GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" -ls -l "$GROUP_PATH/devices" +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` -If the same group contains other devices still required by the host, do not use that VF directly for `vfio-pci` or PCI passthrough. Adjust the hardware, BIOS settings, kernel IOMMU parameters, or NIC planning first. - Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `default`. Because the VF is owned directly by the DPDK/CNF application, Kube-OVN IPAM is not configured here. Save the following content as `sriov-network-pod.yaml`: @@ -402,12 +390,10 @@ spec: mtu: 1500 ``` -Apply the policy and watch node synchronization: +Apply the policy: ```bash kubectl apply -f sriov-node-policy-vm.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system ``` Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: @@ -417,8 +403,6 @@ kubectl get node \ -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' ``` -If the target VF IOMMU group was not checked earlier, complete the same IOMMU group check before creating the `SriovNetwork` for the VM. - Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt`. If the VF is later owned by a DPDK application inside the VM, Kube-OVN IPAM is not required here. Save the following content as `sriov-network-vm.yaml`: diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index fc51f8a4a..74e0e102c 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -228,19 +228,10 @@ spec: mtu: 1500 ``` -应用后观察节点同步状态： +应用 policy： ```bash kubectl apply -f sriov-node-policy-pod.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system -``` - -确认目标节点变为 `Succeeded`： - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` 确认节点资源中出现 SR-IOV device plugin 暴露的资源： @@ -252,16 +243,13 @@ kubectl get node \ 如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 -记录目标 VF 的 PCI 地址，并确认该 VF 所在 IOMMU group 中没有混入不应直通给业务的其他设备： +如果长时间没有资源，再查看目标节点的同步状态和错误信息： ```bash -VF_PCI_ADDR="" -GROUP_PATH="$(readlink -f /sys/bus/pci/devices/$VF_PCI_ADDR/iommu_group)" -ls -l "$GROUP_PATH/devices" +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` -如果同一个 group 内包含宿主机还需要使用的其他设备，不应直接将该 VF 用于 `vfio-pci`/PCI 直通场景，应先调整硬件、BIOS、内核 IOMMU 参数或网卡规划。 - 创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `default`。由于 VF 会被 DPDK/CNF 应用直接接管，这里不配置 kube-ovn IPAM。 将以下内容保存为 `sriov-network-pod.yaml`： @@ -401,12 +389,10 @@ spec: mtu: 1500 ``` -应用后观察节点同步状态： +应用 policy： ```bash kubectl apply -f sriov-node-policy-vm.yaml -kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n cpaas-system ``` 确认节点资源中出现 SR-IOV device plugin 暴露的资源： @@ -416,8 +402,6 @@ kubectl get node \ -o jsonpath='{.status.allocatable.openshift\.io/sriov_vfio}{"\n"}' ``` -如果前面没有检查过目标 VF 的 IOMMU group，在继续创建虚拟机使用的 `SriovNetwork` 前，先完成同样的 IOMMU group 检查。 - 创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`。如果该 VF 后续由虚拟机内的 DPDK 应用接管，不需要在这里配置 kube-ovn IPAM。 将以下内容保存为 `sriov-network-vm.yaml`： From 44bf7cc012b43d89304e39a0b3fdf724d284ed81 Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 14:54:21 +0800 Subject: [PATCH 17/18] ACP-53814 refine SR-IOV implementation guide --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 22 +++++++++++-------- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 22 +++++++++++-------- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 2d3dd775c..5d0d5a251 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -223,12 +223,14 @@ spec: kubernetes.io/hostname: nicSelector: pfNames: - - ens5f0 + - numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` +Replace `` with the PF name recorded earlier, such as `p1p1`. + Apply the policy: ```bash @@ -244,6 +246,12 @@ kubectl get node \ If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. +For example, output `8` means that the current node has eight schedulable `openshift.io/sriov_vfio` VFs: + +```text +8 +``` + If the resource does not appear for a long time, check the target node synchronization status and error message: ```bash @@ -326,21 +334,15 @@ crw-rw-rw- 1 root root 10, 196 ... vfio The numeric device, such as `191`, is the VFIO group available to the current Pod. In this scenario, do not use the presence of a secondary interface in `ip a` as the success criterion. -If `dpdk-devbind.py -s` is run inside the container or the Pod namespace, it may list all PF/VF PCI devices on the node. The script reads `/sys/bus/pci/devices`, and PCI sysfs is not filtered by Pod resource allocation. To determine which VF the current Pod can actually use, rely on the devices exposed under `/dev/vfio/` and the `openshift.io/sriov_vfio` resource allocated to the Pod. - -The following is a reference observation: when `numVfs: 8` creates eight VFs on the node, all eight VFs are bound to `vfio-pci`, so `dpdk-devbind.py -s` can list eight VFs. However, when a single Pod requests only one `openshift.io/sriov_vfio`, the container exposes only one numeric group device under `/dev/vfio/`. +To confirm the VF driver binding state, run `dpdk-devbind.py -s` on the host node. When the target VF appears under `Network devices using DPDK-compatible driver` with `drv=vfio-pci`, the VF is bound to a DPDK-compatible driver: ```text Network devices using DPDK-compatible driver ============================================ 0000:3d:00.1 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic -0000:3d:00.2 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic ... -0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic ``` -This output only shows the VF driver binding status on the node. The number of VFs actually available to the Pod is still determined by `/dev/vfio/` and the resource request count. - If the workload uses Deployment, StatefulSet, or another controller, set `k8s.v1.cni.cncf.io/networks` in the Pod template `metadata.annotations`, and request `openshift.io/sriov_vfio` in the application container `resources.requests` and `resources.limits`. The VF resource only assigns the PCI device to the Pod. DPDK applications usually also need CPU, HugePages, and startup-parameter configuration; the exact values are determined by the business image and DPDK application documentation. A reference Pod fragment is: ```yaml @@ -384,12 +386,14 @@ spec: kubernetes.io/hostname: nicSelector: pfNames: - - ens5f0 + - numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` +Replace `` with the PF name recorded earlier, such as `p1p1`. + Apply the policy: ```bash diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 74e0e102c..84b2d8201 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -222,12 +222,14 @@ spec: kubernetes.io/hostname: nicSelector: pfNames: - - ens5f0 + - numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` +将 `` 替换为前面记录的 PF 名称，例如 `p1p1`。 + 应用 policy： ```bash @@ -243,6 +245,12 @@ kubectl get node \ 如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 +例如输出为 `8`，表示当前节点可被调度使用的 `openshift.io/sriov_vfio` VF 数量为 8： + +```text +8 +``` + 如果长时间没有资源，再查看目标节点的同步状态和错误信息： ```bash @@ -325,21 +333,15 @@ crw-rw-rw- 1 root root 10, 196 ... vfio 其中数字设备（如 `191`）表示当前 Pod 可用的 VFIO group。该场景下不要使用 `ip a` 是否出现辅助网卡作为成功标准。 -如果在容器或 Pod 命名空间中运行 `dpdk-devbind.py -s`，可能会看到节点上的所有 PF/VF PCI 设备；这是因为脚本读取 `/sys/bus/pci/devices`，PCI sysfs 不会按 Pod 资源分配过滤。判断当前 Pod 实际可用的 VF，应以 `/dev/vfio/` 下暴露的设备和 Pod 申请到的 `openshift.io/sriov_vfio` 资源为准。 - -以下是一个参考现象：节点上通过 `numVfs: 8` 创建出的 8 个 VF 都绑定到 `vfio-pci`，因此 `dpdk-devbind.py -s` 可以看到 8 个 VF；但单个 Pod 只申请 1 个 `openshift.io/sriov_vfio` 时，容器内 `/dev/vfio/` 只暴露 1 个数字 group 设备。 +如需辅助确认 VF 驱动绑定状态，可在宿主机节点上运行 `dpdk-devbind.py -s`。当目标 VF 出现在 `Network devices using DPDK-compatible driver` 区域，且显示 `drv=vfio-pci` 时，表示该 VF 已绑定到 DPDK 兼容驱动： ```text Network devices using DPDK-compatible driver ============================================ 0000:3d:00.1 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic -0000:3d:00.2 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic ... -0000:3d:01.0 'Hi1822 Family Virtual Function 375e' numa_node=0 drv=vfio-pci unused=hinic ``` -该输出只能说明节点上的 VF 驱动绑定状态；Pod 实际拿到几个 VF，仍以 `/dev/vfio/` 和资源申请数量为准。 - 如果业务使用 Deployment、StatefulSet 等控制器，在 Pod template 的 `metadata.annotations` 中设置 `k8s.v1.cni.cncf.io/networks`，并在业务容器的 `resources.requests` 和 `resources.limits` 中申请 `openshift.io/sriov_vfio`。VF 资源只负责把 PCI 设备分配给 Pod；DPDK 应用通常还需要 CPU、HugePages 和启动参数等配置，具体取值由业务镜像和 DPDK 应用文档决定。参考配置片段如下： ```yaml @@ -383,12 +385,14 @@ spec: kubernetes.io/hostname: nicSelector: pfNames: - - ens5f0 + - numVfs: 8 deviceType: vfio-pci mtu: 1500 ``` +将 `` 替换为前面记录的 PF 名称，例如 `p1p1`。 + 应用 policy： ```bash From db55143284b397054d0aad7221fcaed3fd90852c Mon Sep 17 00:00:00 2001 From: clyi Date: Thu, 9 Jul 2026 16:22:20 +0800 Subject: [PATCH 18/18] ACP-53814 update SR-IOV plugin docs to v4.3.8 --- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 34 +++++++++---------- ...s_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md | 34 +++++++++---------- 2 files changed, 32 insertions(+), 36 deletions(-) diff --git a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 5d0d5a251..8b114a172 100644 --- a/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/en/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ This article applies to the following combination: | --- | --- | | Alauda Container Platform | 4.3 | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.7` | +| Plugin package version | `sriov-network-plugin v4.3.8` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network or PCI passthrough device | @@ -83,11 +83,11 @@ Apply this label only to nodes where VFs will be configured. Nodes without this ### Install the plugin -This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.7`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.8`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.7`. -3. Download the `sriov-network-plugin.*.v4.3.7.tgz` package that matches the target platform architecture. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.8`. +3. Download the `sriov-network-plugin.*.v4.3.8.tgz` package that matches the target platform architecture. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -98,7 +98,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.7.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.8.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -108,7 +108,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.7` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.8` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. ### Confirm the Multus base @@ -166,18 +166,16 @@ DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: ' IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} ``` -Add the NIC to the `supported-nic-ids` allowlist. The entry format is: +Add the NIC to the `supported-nic-ids` allowlist. Plugin `v4.3.8` supports configuring extra NICs from the installation or upgrade form, so users do not need to write chart values manually. When installing or upgrading `sriov-network-plugin` from **Administrator -> Marketplace -> Cluster Plugins**, add a row in the **Extra Supported SR-IOV NICs** table and fill in the PCI IDs confirmed on site: -```text -: " " -``` - -For example, a Huawei Hi1822 PF has PCI ID `19e5:1822`, and its VF has PCI ID `19e5:375e`. The entry is: +| Field | Description | Example | +| --- | --- | --- | +| Name | Custom NIC name, used only as the allowlist key | `Huawei_Hi1822` | +| Vendor ID | PCI vendor ID of the PF | `19e5` | +| PF Device ID | PCI device ID of the PF | `1822` | +| VF Device ID | PCI device ID of the VF | `375e` | -```yaml -supportedExtraNICs: - - 'Huawei_Hi1822: "19e5 1822 375e"' -``` +These example values only show the required field format. They do not mean that Huawei Hi1822 is built in by default. NIC models and VF device IDs can differ between customer environments, so use the on-site `lspci` output as the source of truth. If the VF device ID is unknown, temporarily create one VF during a maintenance window and inspect it. In the following example, `0000:3d:00.0` is the PF PCI address: @@ -187,7 +185,7 @@ readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 lspci -Dnn -s ``` -For on-site validation, you can temporarily patch the ConfigMap in an installed cluster and restart the operator and config-daemon: +For on-site validation, you can also temporarily patch the ConfigMap in an installed cluster and restart the operator and config-daemon: ```bash kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ @@ -197,7 +195,7 @@ kubectl rollout restart deployment/sriov-network-operator -n cpaas-system kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system ``` -This patch is only suitable for on-site validation and can be lost after plugin upgrade or reinstall. NIC models and VF device IDs can differ between customer environments, so this entry should not be added as a universal default. To keep it long term, configure `supportedExtraNICs` in the plugin installation or upgrade parameters with the PCI IDs confirmed in that environment. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: +This patch is only suitable for on-site validation and can be lost after plugin upgrade or reinstall. Use the plugin installation or upgrade form for long-term configuration. If `sriov_numvfs` was written manually to discover the VF ID, clear the manually created VFs before letting the operator manage the PF through `SriovNetworkNodePolicy`: ```bash echo 0 > /sys/bus/pci/devices//sriov_numvfs diff --git a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md index 84b2d8201..abf9953db 100644 --- a/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md +++ b/docs/zh/solutions/Configuring_SR_IOV_VFs_for_DPDK_Pods_and_KubeVirt_VMs_on_ACP.md @@ -23,7 +23,7 @@ ProductsVersion: | --- | --- | | Alauda Container Platform | 4.3 | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.7` | +| 插件包版本 | `sriov-network-plugin v4.3.8` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络或 PCI 直通设备 | @@ -82,11 +82,11 @@ kubectl label node feature.node.kubernetes.io/sriov-capable=true --o ### 安装插件 -该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.7`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.8`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.7` 的安装包。 -3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.7.tgz` 包。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.8` 的安装包。 +3. 下载与目标平台架构匹配的 `sriov-network-plugin.*.v4.3.8.tgz` 包。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -97,7 +97,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.7.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.8.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -107,7 +107,7 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.7` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.8` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 ### 确认 Multus 基座可用 @@ -165,18 +165,16 @@ DiscoverSriovDevices(): unsupported device {"device": "0000:3d:00.0 -> driver: ' IsSupportedModel(): found unsupported model {"vendorId:": "19e5", "deviceId:": "1822"} ``` -这类场景需要把网卡加入 `supported-nic-ids` 白名单。白名单条目的格式为： +这类场景需要把网卡加入 `supported-nic-ids` 白名单。插件 `v4.3.8` 支持通过安装或升级表单配置额外网卡，不需要手工编写 chart values。在 **管理员 -> 市场 -> 集群插件** 中安装或升级 `sriov-network-plugin` 时，在“额外支持的 SR-IOV 网卡”表格中新增一行，并按现场确认的 PCI ID 填写： -```text -<名称>: " " -``` - -例如 Huawei Hi1822 PF 的 PCI ID 是 `19e5:1822`，创建出的 VF PCI ID 是 `19e5:375e`，则条目为： +| 字段 | 说明 | 示例 | +| --- | --- | --- | +| 名称 | 自定义网卡名称，只用于生成白名单键名 | `Huawei_Hi1822` | +| Vendor ID | PF 的 PCI vendor ID | `19e5` | +| PF Device ID | PF 的 PCI device ID | `1822` | +| VF Device ID | VF 的 PCI device ID | `375e` | -```yaml -supportedExtraNICs: - - 'Huawei_Hi1822: "19e5 1822 375e"' -``` +以上示例只说明字段格式，不表示插件默认内置 Huawei Hi1822。不同客户环境的网卡型号和 VF device ID 可能不同，应以现场 `lspci` 输出为准。 如果还不知道 VF device ID，可以在维护窗口中临时创建 1 个 VF 后查看。以下示例中的 `0000:3d:00.0` 是 PF 的 PCI 地址： @@ -186,7 +184,7 @@ readlink -f /sys/bus/pci/devices/0000:3d:00.0/virtfn0 lspci -Dnn -s ``` -如果只是现场验证，可以临时 patch 已安装集群中的 ConfigMap，并重启 operator 和 config-daemon： +如果只是现场验证，也可以临时 patch 已安装集群中的 ConfigMap，并重启 operator 和 config-daemon： ```bash kubectl patch cm supported-nic-ids -n cpaas-system --type merge -p \ @@ -196,7 +194,7 @@ kubectl rollout restart deployment/sriov-network-operator -n cpaas-system kubectl rollout restart daemonset/sriov-network-config-daemon -n cpaas-system ``` -这种 patch 只适合现场验证，插件升级或重装后可能丢失。不同客户环境的网卡型号和 VF device ID 可能不同，不应把该条目写成通用默认值；需要长期保留时，应按现场确认的 PCI ID，在该环境的插件安装或升级参数中通过 `supportedExtraNICs` 配置。若为了查看 VF ID 手动写过 `sriov_numvfs`，在让 operator 通过 `SriovNetworkNodePolicy` 接管前，先清理手工创建的 VF： +这种 patch 只适合现场验证，插件升级或重装后可能丢失；长期配置应使用插件安装或升级表单。若为了查看 VF ID 手动写过 `sriov_numvfs`，在让 operator 通过 `SriovNetworkNodePolicy` 接管前，先清理手工创建的 VF： ```bash echo 0 > /sys/bus/pci/devices//sriov_numvfs