[Draft] feat(nvcf-compute-plane): gang scheduling with KAI MNNVL topology, KAI/Grove primitives - #603
Open
estroz wants to merge 3 commits into
Open
[Draft] feat(nvcf-compute-plane): gang scheduling with KAI MNNVL topology, KAI/Grove primitives#603estroz wants to merge 3 commits into
estroz wants to merge 3 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (104)
📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughThe compute-plane stack adds optional KAI Scheduler topology resources, feature-gate and validation-policy integration, distribution packaging, and NVLink scheduling guidance. Helm revalidation defaults now enable three skip controls. ChangesKAI Scheduler topology integration
Helm validation defaults
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant EnvironmentValues
participant Helmfile
participant TopologyChart
participant NVCAValidationPolicy
EnvironmentValues->>Helmfile: enable KAI Scheduler topology settings
Helmfile->>TopologyChart: render configured Topology resources
TopologyChart->>Helmfile: return kai.scheduler/v1alpha1 resources
Helmfile->>NVCAValidationPolicy: configure feature gates and validation types
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/helm/helm-reval/values.yaml`:
- Around line 86-88: Add regression coverage for the default skip settings
declared near skipValidateObjects, skipValidateImages, and
skipSanitizeObjectMetadata, rendering
deploy/helm/helm-reval/templates/configmap.yaml and asserting all corresponding
skip-* keys and default values are present. If chart tests cannot be added,
document the specific reason in the pull request description.
- Around line 86-88: Update the ReVal skip controls in values.yaml so
skipValidateObjects, skipValidateImages, and skipSanitizeObjectMetadata all
default to false. Add rendered ConfigMap coverage verifying these defaults, and
document the threat model if any opt-out remains supported.
In
`@deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yaml`:
- Around line 16-39: The deployment changes lack regression coverage. Add tests
for valid and invalid rendering in topology.yaml, disabled and enabled release
rendering in 01-dependencies.yaml.gotmpl, KAI/Grove type selection including
precedence and deduplication in 02-nvca.yaml.gotmpl, and Makefile distribution
behavior ensuring the local chart is included with its relative path resolved.
Cover
deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yaml
lines 16-39,
deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl lines
38-63, deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl lines
73-114, and deploy/stacks/nvcf-compute-plane/Makefile lines 133-134; if tests
cannot be added, document the reason in the pull request.
In `@docs/user/cluster-management/kai-scheduler.md`:
- Around line 133-134: Complete the kubectl apply command in the manifest
application instructions by including the manifest filename, such as
nvcf-mnnvl-topology.yaml.
- Around line 92-96: Update the “Install the Topology” section to document the
command sequence: use `make install` for the initial compute-plane deployment,
then use `make apply` to enable or update the `nvcf-kai-topology` release.
- Around line 109-122: The KAI Scheduler version prerequisite is inconsistent
with the emitted Topology API version. Update the prerequisite documentation at
docs/user/cluster-management/kai-scheduler.md:109-122 and
deploy/stacks/nvcf-compute-plane/README.md:82-89 to require KAI Scheduler
v0.13.0 or later, preserving the kai.scheduler/v1alpha1 API reference.
In `@docs/user/helm-functions.md`:
- Around line 127-130: Update the Deployment behavior note in the Warning
section to state that KAI creates one PodGroup per Deployment replica and copies
the top-owner topology annotations to each PodGroup, allowing per-replica
placement constraints without gang atomicity. Replace the existing
recommendation to use a StatefulSet with this accurate behavior.
In
`@examples/function-samples/helmchart-samples/multi-node-helm-function-test/multi-node-test/templates/statefulset.yaml`:
- Around line 24-27: Move the commented
dra.nvcf.nvidia.io/required-nvlink-domain-index annotation from the StatefulSet
metadata annotations into spec.template.metadata.annotations, keeping the
example value and surrounding comments intact so uncommenting it applies the
annotation to created Pods.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c17f1e10-4442-4969-a336-4d43a49c9ea2
⛔ Files ignored due to path filters (62)
deploy/stacks/nvcf-compute-plane/testdata/environments/local.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/component-serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/epp.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/gpu-discovery-preflight.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/gpu-discovery-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/leader-election-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/manager-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/metrics-auth-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/metrics-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/operator-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/operator-servicemonitor.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/planner.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/profiling-job-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/prometheus.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/regcred-secret.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/validate-values.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-certificates.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-configuration.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/config-map.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/headless-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/pod-disruption-budget.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/stateful-set.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/templates/kai.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/templates/validate-values.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clusterrole.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clusterrolebinding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clustertopology-validating-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/configmap-operator.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/leaderelection-role.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/leaderelection-rolebinding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/pcs-defaulting-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/pcs-validating-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-nvcf-kai-topology/nvcf-kai-topology/templates/topology.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/agent-config-merge-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/chart-defaults-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/cluster-validator-network-checks-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/crds/nvidia.io_nvcfbackends_crd.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/cronjob.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/custom-annotations-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/custom-network-policies-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/gpu-profiling-config-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/helm-managed-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/ngc-service-key.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/nvca-operator_rq.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/pre-delete-cleanup-job.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/pre-delete-cleanup-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/rbac_allowed_extra_types.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/role.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/role_binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/sa.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/self-managed-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/shutdown-sentinel.yamlis excluded by!**/testdata/**
📒 Files selected for processing (13)
deploy/helm/helm-reval/values.yamldeploy/stacks/nvcf-compute-plane/Makefiledeploy/stacks/nvcf-compute-plane/README.mddeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/Chart.yamldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/_helpers.tpldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yamldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/values.yamldeploy/stacks/nvcf-compute-plane/environments/base.yamldeploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpldeploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpldocs/user/cluster-management/kai-scheduler.mddocs/user/helm-functions.mdexamples/function-samples/helmchart-samples/multi-node-helm-function-test/multi-node-test/templates/statefulset.yaml
Comment on lines
+109
to
+122
| The add-on requires KAI Scheduler v0.12.0 or later, which is when the native | ||
| `kai.scheduler/v1alpha1` Topology CRD was introduced. For each entry under | ||
| `topologies` it creates a cluster-scoped resource such as: | ||
|
|
||
| ```yaml | ||
| apiVersion: kai.scheduler/v1alpha1 | ||
| kind: Topology | ||
| metadata: | ||
| name: nvcf-mnnvl-topology | ||
| spec: | ||
| levels: | ||
| - nodeLabel: nvidia.com/gpu.clique | ||
| - nodeLabel: kubernetes.io/hostname | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- docs/user/cluster-management/kai-scheduler.md ---'
sed -n '1,180p' docs/user/cluster-management/kai-scheduler.md
printf '%s\n' '--- deploy/stacks/nvcf-compute-plane/README.md ---'
sed -n '60,105p' deploy/stacks/nvcf-compute-plane/README.md
printf '%s\n' '--- references to Topology and KAI versions ---'
rg -n -C 3 'Topology|topolog|kai.scheduler/v1|kai.scheduler/v1alpha1|v0\.1[23]\.0' \
docs/user/cluster-management/kai-scheduler.md \
deploy/stacks/nvcf-compute-plane/README.md \
deploy tools charts 2>/dev/null || true
printf '%s\n' '--- upstream tag documentation ---'
for tag in v0.12.0 v0.13.0; do
printf '%s\n' "### $tag"
url="https://raw.githubusercontent.com/kai-scheduler/KAI-Scheduler/$tag/docs/topology/README.md"
curl -fsSL "$url" | rg -n -C 3 'apiVersion|Topology|nodeLabel' || printf '%s\n' "No matching content or tag path: $url"
doneRepository: NVIDIA/nvcf
Length of output: 50370
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- repository templates and dependency versions ---'
rg -n -C 4 'apiVersion: kai\.scheduler/|kind: Topology|kaiScheduler|clusterTopologies|kai-scheduler' \
deploy/stacks/nvcf-compute-plane \
--glob '!testdata/golden/**' \
--glob '!*.lock' | head -250
printf '%s\n' '--- KAI v0.12.0 topology document ---'
curl -fsSL \
'https://api.github.com/repos/KAI-Scheduler/KAI-Scheduler/contents/docs/topology/README.md?ref=v0.12.0' |
jq -r '.download_url // .message'
url12='https://raw.githubusercontent.com/KAI-Scheduler/KAI-Scheduler/v0.12.0/docs/topology/README.md'
curl -fsSL "$url12" |
rg -n -C 5 'apiVersion|kind: Topology|nodeLabel|v1alpha1|scheduler/v1' || true
printf '%s\n' '--- KAI v0.13.0 topology document ---'
url13='https://raw.githubusercontent.com/KAI-Scheduler/KAI-Scheduler/v0.13.0/docs/topology/README.md'
curl -fsSL "$url13" |
rg -n -C 5 'apiVersion|kind: Topology|nodeLabel|v1alpha1|scheduler/v1' || true
printf '%s\n' '--- KAI tag refs and commits ---'
for tag in v0.12.0 v0.13.0 v0.14.0; do
curl -fsSL "https://api.github.com/repos/KAI-Scheduler/KAI-Scheduler/git/ref/tags/$tag" |
jq -r '"'"$tag"' + ": type=" + .object.type + " sha=" + .object.sha'
doneRepository: NVIDIA/nvcf
Length of output: 22907
🏁 Script executed:
#!/bin/sh
set -eu
python3 - <<'PY'
import re
from pathlib import Path
from urllib.request import Request, urlopen
template = Path("deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yaml").read_text()
docs = Path("docs/user/cluster-management/kai-scheduler.md").read_text()
readme = Path("deploy/stacks/nvcf-compute-plane/README.md").read_text()
def first_api(text, label):
match = re.search(r"apiVersion:\s*([^\s]+)", text)
if not match:
raise SystemExit(f"{label}: no apiVersion found")
return match.group(1)
print("local chart Topology apiVersion:", first_api(template, "local chart"))
print("guide documented Topology apiVersion:",
re.search(r"apiVersion:\s*(kai\.scheduler/\S+)", docs).group(1))
print("guide prerequisite:", re.search(r"requires KAI Scheduler (v[0-9.]+) or later", docs).group(1))
print("README prerequisite:", re.search(r"requires KAI Scheduler (v[0-9.]+) or later", readme).group(1))
for tag in ("v0.12.0", "v0.13.0"):
url = f"https://raw.githubusercontent.com/KAI-Scheduler/KAI-Scheduler/{tag}/docs/topology/README.md"
req = Request(url, headers={"User-Agent": "CodeRabbit-review"})
with urlopen(req, timeout=20) as response:
text = response.read().decode()
print(f"{tag} upstream Topology apiVersion:", first_api(text, tag))
PYRepository: NVIDIA/nvcf
Length of output: 427
🏁 Script executed:
#!/bin/sh
set -eu
python3 - <<'PY'
import re
from pathlib import Path
from urllib.request import Request, urlopen
template = Path("deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yaml").read_text()
guide = Path("docs/user/cluster-management/kai-scheduler.md").read_text()
readme = Path("deploy/stacks/nvcf-compute-plane/README.md").read_text()
def api_version(text, label):
match = re.search(r"apiVersion:\s*([^\s]+)", text)
if not match:
raise SystemExit(f"{label}: no apiVersion found")
return match.group(1)
def prerequisite(text, label):
match = re.search(r"requires KAI Scheduler\s+(v[0-9.]+)\s+or\s+later", text)
if not match:
raise SystemExit(f"{label}: prerequisite not found")
return match.group(1)
print("local chart Topology apiVersion:", api_version(template, "local chart"))
print("guide documented Topology apiVersion:", api_version(guide, "guide"))
print("guide prerequisite:", prerequisite(guide, "guide"))
print("README prerequisite:", prerequisite(readme, "README"))
for tag in ("v0.12.0", "v0.13.0"):
url = f"https://raw.githubusercontent.com/KAI-Scheduler/KAI-Scheduler/{tag}/docs/topology/README.md"
request = Request(url, headers={"User-Agent": "CodeRabbit-review"})
with urlopen(request, timeout=20) as response:
text = response.read().decode()
print(f"{tag} upstream Topology apiVersion:", api_version(text, tag))
PYRepository: NVIDIA/nvcf
Length of output: 2610
Use KAI Scheduler v0.13.0 or later for the Topology add-on.
The chart emits kai.scheduler/v1alpha1, but KAI v0.12.0 documents kai.scheduler/v1. Update both prerequisites, or render kai.scheduler/v1 for v0.12.0.
📍 Affects 2 files
docs/user/cluster-management/kai-scheduler.md#L109-L122(this comment)deploy/stacks/nvcf-compute-plane/README.md#L82-L89
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/user/cluster-management/kai-scheduler.md` around lines 109 - 122, The
KAI Scheduler version prerequisite is inconsistent with the emitted Topology API
version. Update the prerequisite documentation at
docs/user/cluster-management/kai-scheduler.md:109-122 and
deploy/stacks/nvcf-compute-plane/README.md:82-89 to require KAI Scheduler
v0.13.0 or later, preserving the kai.scheduler/v1alpha1 API reference.
Comment on lines
+133
to
+134
| To apply it without the compute plane stack, write the manifest above to a file and run | ||
| `kubectl apply -f`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
file="docs/user/cluster-management/kai-scheduler.md"
printf '%s\n' '--- target lines ---'
sed -n '115,145p' "$file"
printf '%s\n' '--- nearby manifest and kubectl references ---'
rg -n -C 3 'kubectl apply|nvcf-mnnvl|topology|manifest' "$file"
printf '%s\n' '--- repository references to the same manifest ---'
rg -n -C 2 'nvcf-mnnvl-topology|kai-scheduler' docs tools .github 2>/dev/null | head -n 240Repository: NVIDIA/nvcf
Length of output: 29726
Complete the kubectl apply command. Add the manifest filename, for example: kubectl apply -f nvcf-mnnvl-topology.yaml.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/user/cluster-management/kai-scheduler.md` around lines 133 - 134,
Complete the kubectl apply command in the manifest application instructions by
including the manifest filename, such as nvcf-mnnvl-topology.yaml.
4 tasks
estroz
force-pushed
the
estroczynski/feat/mnnvl-scheduling
branch
from
July 31, 2026 21:15
034fb28 to
e2f4e4e
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (1)
deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl (1)
142-153: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConfirm that disabling consolidation matches the gang-placement objective.
The PR objective is atomic placement of a full worker group inside one NVLink partition.
actions.consolidation.enabled: falseremoves the defragmentation step that frees a contiguous block for a pending PodGroup. On a fragmented cluster, gang requests then stay pending longer. If this is intentional for stability, state the reason in the comment block.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl` around lines 142 - 153, Confirm the intended gang-placement behavior in the scheduler configuration around actions.consolidation.enabled. Either enable consolidation so fragmented clusters can free contiguous capacity for a full worker group, or, if disabling it is intentional for stability, add a nearby comment documenting that rationale and its tradeoff.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ai-tooling/user/skills/nvcf-self-managed-prerequisite/SKILL.md`:
- Around line 85-86: Update the uninstall section in the prerequisite workflow
to skip standalone KAI Scheduler removal when nvcf-compute-plane has
addons.kaiScheduler.enabled set to true, preserving the compute-plane-managed
kai-scheduler release and namespace. Run the existing uninstall commands only
for standalone installations, and document or invoke the separate compute-plane
teardown path when the add-on owns KAI.
- Line 38: Update the KAI Scheduler prerequisite wording to say the standalone
Helm step should be skipped only when addons.kaiScheduler.enabled is true,
matching the Helmfile condition and preserving installation when the value is
false.
In `@deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl`:
- Around line 106-196: Remove the entire numaPlacementExporter configuration
block, including its resources requests and limits, from the dependency values
while leaving the supported operator components and scheduler configuration
unchanged.
- Around line 248-252: Make the needs dependency for the nvcf-kai-topology
release conditional on addons.kaiScheduler.enabled, so it is emitted only when
Helmfile manages the KAI installation; preserve the existing
kai-scheduler/kai-scheduler ordering when enabled, and omit it for standalone or
disabled KAI deployments.
In `@deploy/stacks/nvcf-compute-plane/Makefile.dist`:
- Line 38: Update the NAMESPACES definition to include kai-resource-reservation
so the destroy target removes it explicitly. Also document the required manual
cleanup of KAI CRDs and retained cluster-scoped resources, or add a guarded
cleanup step for those resources without affecting existing destroy behavior.
In `@docs/user/helm-functions.md`:
- Around line 109-116: Update the StatefulSet example by adding
spec.template.spec.schedulerName with the value kai-scheduler, ensuring Pods use
KAI for the documented topology and gang-scheduling behavior.
---
Nitpick comments:
In `@deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl`:
- Around line 142-153: Confirm the intended gang-placement behavior in the
scheduler configuration around actions.consolidation.enabled. Either enable
consolidation so fragmented clusters can free contiguous capacity for a full
worker group, or, if disabling it is intentional for stability, add a nearby
comment documenting that rationale and its tradeoff.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e5a8a04f-dee2-450b-819d-765c2fafc567
⛔ Files ignored due to path filters (104)
deploy/stacks/nvcf-compute-plane/testdata/environments/local.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/component-serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/epp.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/gpu-discovery-preflight.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/gpu-discovery-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/leader-election-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/manager-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/metrics-auth-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/metrics-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/operator-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/operator-servicemonitor.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/planner.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/profiling-job-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/prometheus.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/regcred-secret.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/validate-values.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-certificates.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-configuration.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/dynamo-operator/templates/webhook-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/config-map.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/headless-service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/pod-disruption-budget.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/charts/nats/templates/stateful-set.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/templates/kai.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-dynamo-operator/dynamo-platform/templates/validate-values.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clusterrole.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clusterrolebinding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/clustertopology-validating-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/configmap-operator.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/leaderelection-role.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/leaderelection-rolebinding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/pcs-defaulting-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/pcs-validating-webhook-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/service.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-grove-operator/grove-charts/templates/serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/default-queue.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/default-shard.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/hooks/post/post-delete-job.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/hooks/pre/crd-upgrader.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/hooks/pre/topology-migration/configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/hooks/pre/topology-migration/job.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/hooks/pre/topology-migration/rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/kai-config.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/priorityclasses/build-preemptible.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/priorityclasses/build.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/priorityclasses/inference.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/priorityclasses/train.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/admission-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/admission.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/binder-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/binder.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/crd-manager.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/nodescaleadjuster-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/nodescaleadjuster.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/operator-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/operator.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/podgroupcontroller-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/podgroupcontroller.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/podgrouper-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/podgrouper.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/post-delete-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/post-delete-clusterrole.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/prometheus-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/prometheus-pod.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/queuecontroller-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/queuecontroller.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/resourcereservation-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/resourcereservation.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/scc.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/scheduler-binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/rbac/scheduler.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/operator-serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/operator.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/post-delete-serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/resourcereservation-namespace.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/resourcereservation-serviceaccount.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/services/scalingpod-namespace.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-nvcf-kai-topology/nvcf-kai-topology/templates/topology.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/agent-config-merge-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/chart-defaults-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/cluster-validator-network-checks-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/crds/nvidia.io_nvcfbackends_crd.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/cronjob.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/custom-annotations-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/custom-network-policies-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/deployment.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/gpu-profiling-config-configmap.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/helm-managed-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/ngc-service-key.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/nvca-operator_rq.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/pre-delete-cleanup-job.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/pre-delete-cleanup-rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/rbac.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/rbac_allowed_extra_types.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/role.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/role_binding.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/sa.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/self-managed-nvcfbackend-cm.yamlis excluded by!**/testdata/**deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/shutdown-sentinel.yamlis excluded by!**/testdata/**
📒 Files selected for processing (16)
ai-tooling/user/skills/nvcf-self-managed-prerequisite/SKILL.mdai-tooling/user/skills/nvcf-self-managed-prerequisite/references/kai-scheduler.mddeploy/helm/helm-reval/values.yamldeploy/stacks/nvcf-compute-plane/Makefiledeploy/stacks/nvcf-compute-plane/Makefile.distdeploy/stacks/nvcf-compute-plane/README.mddeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/Chart.yamldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/_helpers.tpldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/topology.yamldeploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/values.yamldeploy/stacks/nvcf-compute-plane/environments/base.yamldeploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpldeploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpldocs/user/cluster-management/kai-scheduler.mddocs/user/helm-functions.mdexamples/function-samples/helmchart-samples/multi-node-helm-function-test/multi-node-test/templates/statefulset.yaml
🚧 Files skipped from review as they are similar to previous changes (8)
- deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/Chart.yaml
- deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/values.yaml
- deploy/helm/helm-reval/values.yaml
- deploy/stacks/nvcf-compute-plane/charts/nvcf-kai-topology/templates/_helpers.tpl
- deploy/stacks/nvcf-compute-plane/environments/base.yaml
- examples/function-samples/helmchart-samples/multi-node-helm-function-test/multi-node-test/templates/statefulset.yaml
- deploy/stacks/nvcf-compute-plane/README.md
- docs/user/cluster-management/kai-scheduler.md
Comment on lines
+85
to
+86
| Skip this step when `addons.kaiScheduler.enabled` is true in `nvcf-compute-plane`. That stack installs KAI Scheduler (release and namespace `kai-scheduler`). Enable that flag whenever Grove or Dynamo is enabled. Use the standalone install below when you need KAI without the compute-plane add-on (for example the NVCA `KAIScheduler` feature gate alone). | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Protect the compute-plane-managed KAI release during uninstall.
When addons.kaiScheduler.enabled is true, the compute-plane stack owns the kai-scheduler release in the kai-scheduler namespace. The later unconditional uninstall commands can remove that managed release and namespace. Make the uninstall section conditional on standalone installation, or document the separate compute-plane teardown path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ai-tooling/user/skills/nvcf-self-managed-prerequisite/SKILL.md` around lines
85 - 86, Update the uninstall section in the prerequisite workflow to skip
standalone KAI Scheduler removal when nvcf-compute-plane has
addons.kaiScheduler.enabled set to true, preserving the compute-plane-managed
kai-scheduler release and namespace. Run the existing uninstall commands only
for standalone installations, and document or invoke the separate compute-plane
teardown path when the add-on owns KAI.
Comment on lines
+106
to
+196
| operator: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $opReq }} | ||
| memory: {{ dig "memory" "256Mi" $opReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $opLim }} | ||
| memory: {{ dig "memory" "1Gi" $opLim }} | ||
|
|
||
| podgrouper: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $pgReq }} | ||
| memory: {{ dig "memory" "512Mi" $pgReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $pgLim }} | ||
| memory: {{ dig "memory" "4Gi" $pgLim }} | ||
|
|
||
| podgroupcontroller: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $pgcReq }} | ||
| memory: {{ dig "memory" "256Mi" $pgcReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $pgcLim }} | ||
| memory: {{ dig "memory" "1Gi" $pgcLim }} | ||
|
|
||
| binder: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $bdReq }} | ||
| memory: {{ dig "memory" "2Gi" $bdReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "4000m" $bdLim }} | ||
| memory: {{ dig "memory" "4Gi" $bdLim }} | ||
|
|
||
| scheduler: | ||
| placementStrategy: binpack | ||
| plugins: | ||
| nodeplacement: | ||
| arguments: | ||
| gpu: binpack | ||
| cpu: spread | ||
| actions: | ||
| preempt: | ||
| enabled: false | ||
| consolidation: | ||
| enabled: false | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "500m" $schReq }} | ||
| memory: {{ dig "memory" "3Gi" $schReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "2000m" $schLim }} | ||
| memory: {{ dig "memory" "6Gi" $schLim }} | ||
|
|
||
| queuecontroller: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $qcReq }} | ||
| memory: {{ dig "memory" "256Mi" $qcReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $qcLim }} | ||
| memory: {{ dig "memory" "1Gi" $qcLim }} | ||
|
|
||
| admission: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $adReq }} | ||
| memory: {{ dig "memory" "256Mi" $adReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $adLim }} | ||
| memory: {{ dig "memory" "1Gi" $adLim }} | ||
|
|
||
| nodescaleadjuster: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $nsaReq }} | ||
| memory: {{ dig "memory" "256Mi" $nsaReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $nsaLim }} | ||
| memory: {{ dig "memory" "1Gi" $nsaLim }} | ||
|
|
||
| numaPlacementExporter: | ||
| resources: | ||
| requests: | ||
| cpu: {{ dig "cpu" "50m" $npeReq }} | ||
| memory: {{ dig "memory" "64Mi" $npeReq }} | ||
| limits: | ||
| cpu: {{ dig "cpu" "1000m" $npeLim }} | ||
| memory: {{ dig "memory" "1Gi" $npeLim }} |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
kai-scheduler helm chart v0.14 values.yaml keys numaPlacementExporter podgroupcontroller defaultQueue scheduler actions
💡 Result:
In the KAI-Scheduler Helm chart (v0.14.0), the values.yaml file provides specific configuration surfaces for the scheduler and its components, including custom plugins and actions. Scheduler Actions and Plugins The scheduler configuration allows for overriding or extending default behavior via the plugins and actions keys: - plugins: This section allows users to disable, reorder, or modify arguments for built-in scheduler plugins, as well as add custom plugins by specifying names not present in the default set [1]. - actions: This section allows for the management of scheduler actions. Users can disable or reorder existing built-in actions or define new ones by specifying their name and priority [1]. NumaPlacementExporter The NumaPlacementExporter is managed as part of the KAI Scheduler operator. Recent updates (introduced around v0.14 timeframe) allow for configurable paths for the podresources socket and sysfs, which are critical for testing and non-standard hardware environments [2][3]. Key configurable fields include: - podResourcesHostPath: Path to the pod resources directory (default: /var/lib/kubelet/pod-resources) [2]. - podResourcesSocket: Path to the kubelet socket (default: /var/lib/kubelet/pod-resources/kubelet.sock) [2]. - sysfsHostPath: Path to the sysfs root (default: /sys) [2]. Podgroupcontroller and DefaultQueue These components are primarily managed via the KAI Scheduler's configuration reconciliation (Config CR). - podgrouper: Managed within the pod controller (e.g., PodReconciler), which uses configs for settings like NodePoolLabelKey and SchedulingQueueLabelKey [4]. - DefaultQueue: The scheduling queue is typically configured via settings such as SchedulingQueueLabelKey, which defines how the system associates pods with specific queues in the cluster [4]. Note: KAI-Scheduler charts template values into a Config Custom Resource (Config CR) that the KAI Operator reconciles [5]. Therefore, many operational parameters (like those for the podgrouper) are defined within the structure of this Config CR, which the Helm chart populates based on the values.yaml configuration [5][6].
Citations:
- 1: https://github.com/NVIDIA/KAI-Scheduler/blob/f909aece/deployments/kai-scheduler/values.yaml
- 2: feat(operator): make numa-placement-exporter podresources socket + sysfs configurable kai-scheduler/KAI-Scheduler#1807
- 3: kai-scheduler/KAI-Scheduler@f6e5979
- 4: https://github.com/NVIDIA/KAI-Scheduler/blob/f909aece/pkg/podgrouper/pod_controller.go
- 5: https://deepwiki.com/NVIDIA/KAI-Scheduler/2.2-helm-chart-installation
- 6: https://deepwiki.com/NVIDIA/KAI-Scheduler/2-installation-and-deployment
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- local template context ---'
sed -n '1,230p' deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl
printf '%s\n' '--- local references ---'
rg -n 'numaPlacementExporter|podgroupcontroller|queuecontroller|nodescaleadjuster|defaultQueue|nodeplacement|actions:|plugins:' .
printf '%s\n' '--- upstream v0.14 chart files ---'
for path in values.yaml templates/_helpers.tpl templates/config.yaml templates/configmap.yaml; do
url="https://raw.githubusercontent.com/NVIDIA/KAI-Scheduler/v0.14.0/deployments/kai-scheduler/$path"
echo "### $url"
curl -LfsS "$url" | sed -n '1,260p' || true
doneRepository: NVIDIA/nvcf
Length of output: 50367
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- chart fixture files ---'
find deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler \
-maxdepth 3 -type f -print | sort
printf '%s\n' '--- rendered component and scheduler sections ---'
for f in \
deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/kai-config.yaml \
deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler/templates/default-shard.yaml; do
echo "### $f"
rg -n -C 8 'name: (operator|podgrouper|podgroupcontroller|binder|scheduler|queuecontroller|admission|nodescaleadjuster|numaPlacementExporter)|defaultQueue|nodeplacement|actions:|plugins:' "$f"
done
printf '%s\n' '--- upstream repository tags and v0.14 paths ---'
curl -LfsS 'https://api.github.com/repos/NVIDIA/KAI-Scheduler/tags?per_page=100' |
jq -r '.[] | .name' | rg '(^|v)0\.14'
for path in \
deployments/kai-scheduler/values.yaml \
deployments/kai-scheduler/Chart.yaml \
deployments/kai-scheduler/templates/default-shard.yaml \
deployments/kai-scheduler/templates/kai-config.yaml \
deployments/kai-scheduler/templates/operator.yaml; do
url="https://api.github.com/repos/NVIDIA/KAI-Scheduler/contents/$path?ref=v0.14.0"
echo "### $url"
curl -LsS "$url" | jq -r 'if .content then (.content | `@base64d`) else (.message // "not found") end' |
rg -n -C 4 'operator:|podgrouper:|podgroupcontroller:|binder:|scheduler:|queuecontroller:|admission:|nodescaleadjuster:|numaPlacementExporter:|defaultQueue:|nodeplacement:|actions:|plugins:' || true
doneRepository: NVIDIA/nvcf
Length of output: 4411
🏁 Script executed:
#!/bin/bash
set -eu
fixture='deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler/kai-scheduler'
printf '%s\n' '--- complete rendered kai-config component keys ---'
sed -n '1,240p' "$fixture/templates/kai-config.yaml"
printf '%s\n' '--- complete rendered scheduler and default queue resources ---'
cat "$fixture/templates/default-shard.yaml"
cat "$fixture/templates/default-queue.yaml"
printf '%s\n' '--- chart-key references in repository documentation and fixtures ---'
rg -n -C 4 \
'podGrouper|podGroupController|queueController|nodeScaleAdjuster|numaPlacementExporter|numaPlacementExporter|defaultQueue|placementStrategy' \
deploy docs ai-tooling \
-g '*.yaml' -g '*.yaml.gotmpl' -g '*.md' -g 'values.yaml' \
| head -n 500
printf '%s\n' '--- OCI chart metadata ---'
curl -LfsS -H 'Accept: application/vnd.oci.image.index.v1+json' \
'https://ghcr.io/v2/kai-scheduler/kai-scheduler/tags/list' |
jq -r '.tags[]' | rg '(^|-)v?0\.14\.0$|0\.14' || trueRepository: NVIDIA/nvcf
Length of output: 29086
🏁 Script executed:
#!/bin/bash
set -eu
for ref in f909aece v0.14.0 main; do
url="https://raw.githubusercontent.com/NVIDIA/KAI-Scheduler/$ref/deployments/kai-scheduler/values.yaml"
echo "### $url"
body="$(curl -LfsS "$url" 2>/dev/null || true)"
if [ -n "$body" ]; then
printf '%s\n' "$body" |
rg -n -C 5 \
'^(operator|podgrouper|podGrouper|podgroupcontroller|podGroupController|binder|scheduler|queuecontroller|queueController|admission|nodescaleadjuster|nodeScaleAdjuster|numaPlacementExporter|numaPlacementExporter|defaultQueue):|plugins:|actions:' \
|| true
else
echo 'not available'
fi
done
printf '%s\n' '--- all fixture references to the suspected component ---'
rg -n 'operator|podgrouper|podGrouper|podgroupcontroller|podGroupController|queuecontroller|queueController|nodescaleadjuster|nodeScaleAdjuster|numaPlacementExporter|numaPlacementExporter' \
deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-schedulerRepository: NVIDIA/nvcf
Length of output: 21693
Remove the unsupported numaPlacementExporter block. KAI-Scheduler v0.14.0 supports the other listed keys, including scheduler.plugins, scheduler.actions, and defaultQueue, but it does not define or render numaPlacementExporter. Its resource settings are ignored.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/stacks/nvcf-compute-plane/helmfile.d/01-dependencies.yaml.gotmpl`
around lines 106 - 196, Remove the entire numaPlacementExporter configuration
block, including its resources requests and limits, from the dependency values
while leaving the supported operator components and scheduler configuration
unchanged.
|
|
||
| # Namespaces owned by the compute-plane stack (used by destroy target) | ||
| NAMESPACES := nvca-operator grove-system dynamo-system | ||
| NAMESPACES := nvca-operator grove-system dynamo-system kai-scheduler |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Find KAI namespace and CRD references across the compute-plane stack to size the cleanup gap.
set -euo pipefail
fd -t f . deploy/stacks/nvcf-compute-plane -x rg -n -C3 'kai-scheduler|reservation|kai\.scheduler|scheduling\.run\.ai' {}Repository: NVIDIA/nvcf
Length of output: 32970
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- Makefile destroy logic ---'
cat -n deploy/stacks/nvcf-compute-plane/Makefile.dist | sed -n '30,75p'
printf '%s\n' '--- KAI release and cleanup-related templates ---'
rg -n -C4 'post-delete|helm.sh/resource-policy|kind: CustomResourceDefinition|resourceNames:|kai-resource-reservation|kind: (Topology|Queue|PodGroup)' \
deploy/stacks/nvcf-compute-plane/charts \
deploy/stacks/nvcf-compute-plane/releases \
deploy/stacks/nvcf-compute-plane 2>/dev/null | sed -n '1,260p'
printf '%s\n' '--- destroy target references ---'
rg -n -C6 'destroy|NAMESPACES|helmfile|kubectl.*delete' deploy/stacks/nvcf-compute-plane/Makefile.distRepository: NVIDIA/nvcf
Length of output: 44760
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- Makefile destroy target ---'
rg -n -C12 '^destroy:|NAMESPACES|kubectl.*delete|helmfile.*destroy|helmfile.*delete' \
deploy/stacks/nvcf-compute-plane/Makefile.dist
printf '%s\n' '--- KAI chart files ---'
fd -t f . deploy/stacks/nvcf-compute-plane | rg '/kai-scheduler/' | rg -v '/testdata/'
printf '%s\n' '--- KAI chart cleanup and CRD definitions ---'
fd -t f . deploy/stacks/nvcf-compute-plane | rg '/kai-scheduler/' | rg -v '/testdata/' | \
xargs -r rg -n -C5 'post-delete|helm.sh/resource-policy|kind: CustomResourceDefinition|resourceNames:|kind: (Topology|Queue|PodGroup)|kai-resource-reservation'
printf '%s\n' '--- Direct KAI chart source locations ---'
fd -t d -i 'kai-scheduler' . deploy/stacks/nvcf-compute-plane | rg -v '/testdata/'Repository: NVIDIA/nvcf
Length of output: 3058
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- KAI post-delete hook ---'
fd -t f . deploy/stacks/nvcf-compute-plane/testdata/golden | \
rg 'kai-scheduler/.*/post-delete|kai-scheduler/.*/crd|kai-scheduler/.*/default-queue|kai-scheduler/.*/default-shard' | \
while IFS= read -r f; do
printf '\n### %s\n' "$f"
cat -n "$f" | sed -n '1,180p'
done
printf '%s\n' '--- KAI CRD-related manifests in golden output ---'
rg -n -C8 'kind: CustomResourceDefinition|name: (configs|schedulingshards|bindrequests|podgroups|queues|topologies)\.' \
deploy/stacks/nvcf-compute-plane/testdata/golden/local/01-dependencies.yaml-a814432c-kai-scheduler \
| sed -n '1,260p'
printf '%s\n' '--- Stack documentation for destroy and KAI lifecycle ---'
rg -n -C5 'destroy|CRD|manual|resource reservation|kai-resource-reservation|KAI Scheduler' \
deploy/stacks/nvcf-compute-plane --glob '!testdata/**' --glob '*.md' --glob '*.yaml' --glob '*.gotmpl' --glob 'Makefile.dist' \
| sed -n '1,260p'Repository: NVIDIA/nvcf
Length of output: 9976
🌐 Web query:
KAI Scheduler Helm chart v0.14.0 CRD deletion post-delete resource-policy keep kai-resource-reservation
💡 Result:
In the KAI Scheduler Helm chart, the kai-resource-reservation namespace is configured with the helm.sh/resource-policy: keep annotation [1]. This annotation prevents the namespace from being deleted when the Helm release is uninstalled [2][3]. This behavior is intentional and part of the chart's design to preserve critical scheduler infrastructure—such as SchedulingShards and the resource reservation namespace—even if the main Helm release is removed [2][3]. Because these resources are marked with helm.sh/resource-policy: keep, they are excluded from the default Helm uninstall cleanup [2][3]. Key details regarding this configuration include: 1. Persistence of Resources: The kai-resource-reservation namespace and SchedulingShard custom resources are not managed by standard Helm deletion because they are explicitly protected by the keep policy [2][3]. 2. Uninstallation Impact: Users may encounter leftover pods or resources after running helm uninstall because these components remain active in the cluster [2]. If full cleanup is required, these resources must be removed manually [4]. 3. Helm Chart v0.14.0 Updates: Recent improvements to the chart, such as those related to post-delete cleanup hooks and resource policies, have focused on stabilizing operator-managed deployments and preventing unnecessary recreations of resources like kai-config [5][6], but they do not alter the fundamental keep policy for the core infrastructure resources identified in your query [1]. If you are performing a cleanup and these resources are persisting, you must remove them manually using kubectl delete namespace kai-resource-reservation (or the specifically configured namespace) and the associated custom resources [4].
Citations:
- 1: kai-scheduler/KAI-Scheduler@0fe3c86
- 2: bug: Helm uninstall leaves behind scheduler pods due to "kept" SchedulingShard kai-scheduler/KAI-Scheduler#661
- 3: https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gitops/README.md
- 4: https://deepwiki.com/NVIDIA/KAI-Scheduler/2.2-helm-chart-installation
- 5: https://github.com/kai-scheduler/KAI-Scheduler/blob/main/CHANGELOG.md
- 6: fix(chart): stop recreating kai-config CR on helm upgrade kai-scheduler/KAI-Scheduler#1550
Clean up KAI resources during destroy.
Add kai-resource-reservation to NAMESPACES. Helm keeps this namespace, but the destroy target can remove it explicitly. Document that KAI CRDs and kept cluster-scoped resources require manual cleanup, or add a guarded cleanup step. The post-delete hook does not remove the CRDs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/stacks/nvcf-compute-plane/Makefile.dist` at line 38, Update the
NAMESPACES definition to include kai-resource-reservation so the destroy target
removes it explicitly. Also document the required manual cleanup of KAI CRDs and
retained cluster-scoped resources, or add a guarded cleanup step for those
resources without affecting existing destroy behavior.
Comment on lines
+109
to
+116
| ```yaml | ||
| apiVersion: apps/v1 | ||
| kind: StatefulSet | ||
| metadata: | ||
| annotations: | ||
| kai.scheduler/topology: "nvcf-mnnvl-topology" | ||
| kai.scheduler/topology-required-placement: "nvidia.com/gpu.clique" | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- docs/user/helm-functions.md ---'
sed -n '80,135p' docs/user/helm-functions.md
printf '%s\n' '--- KAI scheduler references in repository ---'
rg -n -i -C 3 'schedulerName|kai-scheduler|topology-required-placement|nvcf-mnnvl-topology' docs .github charts 2>/dev/null || true
printf '%s\n' '--- KAI v0.14.0 quickstart ---'
curl -L --fail --silent --show-error \
https://raw.githubusercontent.com/kai-scheduler/KAI-Scheduler/v0.14.0/docs/quickstart/README.md \
| rg -n -i -C 4 'schedulerName|kai-scheduler|StatefulSet|topology-required-placement|gang' || trueRepository: NVIDIA/nvcf
Length of output: 36393
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
import urllib.request
doc = Path("docs/user/helm-functions.md").read_text()
match = re.search(
r"##### With KAI Scheduler.*?```yaml\n(.*?)\n```",
doc,
re.S,
)
if not match:
raise SystemExit("KAI Scheduler YAML example not found")
manifest = match.group(1)
print("--- local YAML example checks ---")
print("StatefulSet:", "kind: StatefulSet" in manifest)
print("StatefulSet metadata annotations:",
"kai.scheduler/topology:" in manifest and
"kai.scheduler/topology-required-placement:" in manifest)
print("Pod template schedulerName:",
bool(re.search(r"^\s+ schedulerName:\s+kai-scheduler\s*$", manifest, re.M)) or
"schedulerName: kai-scheduler" in manifest)
url = "https://raw.githubusercontent.com/kai-scheduler/KAI-Scheduler/v0.14.0/docs/quickstart/README.md"
upstream = urllib.request.urlopen(url, timeout=10).read().decode()
print("--- upstream v0.14.0 requirement ---")
for line in upstream.splitlines():
if "scheduler name" in line.lower() or "kai-scheduler" in line:
print(line)
PYRepository: NVIDIA/nvcf
Length of output: 2561
Set the Pod scheduler to kai-scheduler.
Add spec.template.spec.schedulerName: kai-scheduler to the StatefulSet example, or state this requirement explicitly. Without it, the default scheduler handles the Pods, so the documented KAI topology and gang-scheduling behavior does not apply.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/user/helm-functions.md` around lines 109 - 116, Update the StatefulSet
example by adding spec.template.spec.schedulerName with the value kai-scheduler,
ensuring Pods use KAI for the documented topology and gang-scheduling behavior.
…I/Grove primitives Signed-off-by: Eric Stroczynski <estroczynski@nvidia.com>
Signed-off-by: Eric Stroczynski <estroczynski@nvidia.com>
estroz
force-pushed
the
estroczynski/feat/mnnvl-scheduling
branch
from
August 1, 2026 18:28
e2f4e4e to
23ce869
Compare
estroz
commented
Aug 1, 2026
| skipValidateObjects: false | ||
| skipValidateImages: false | ||
| skipSanitizeObjectMetadata: false | ||
| skipValidateObjects: true |
Contributor
Author
There was a problem hiding this comment.
For the reviewer: these were disabled before but re-enabled during monorepo migration
Signed-off-by: Eric Stroczynski <estroczynski@nvidia.com>
estroz
force-pushed
the
estroczynski/feat/mnnvl-scheduling
branch
from
August 1, 2026 18:48
dd6fb6f to
31b96fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
This MR implements the suggested fix from #598 with documentation on how to use MNNVL features
Additional Details (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)
For the Reviewer
For QA (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)
Issues
Relates to #598
Checklist
Summary by CodeRabbit
New Features
Documentation
Configuration