feat: node-exporter into vhd build by chmill-zz · Pull Request #7704 · Azure/AgentBaker

chmill-zz · 2026-01-22T00:14:23Z

What this PR does / why we need it:

this is adding node-exporter into the vhdbuild by default. At the end of the vhdbuild we systemctl disable to allow a fresh start during cse letting the node-exporter-startup script run and gather up node specific details needed.

Which issue(s) this PR fixes:

Fixes #

cameronmeissner · 2026-01-23T23:17:07Z

parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/node-exporter.d/web-config.yml

@@ -0,0 +1,5 @@
+tls_server_config:
+  cert_file: "/etc/kubernetes/certs/kubeletserver.crt"


these paths aren't necessarily correct - they depend on whether kubelet serving certificate rotation is enabled - when it's disabled these paths are correct, however when it's enabled both cert_file and key_file should point towards: /var/lib/kubelet/pki/kubelet-server-current.pem

interesting this is copy paste from the aks-vm-extension repo and what is on my node today. I don't see any where it's touched, just a static file.

root@aks-sys-41317600-vmss000000:/etc/node-exporter.d# cat web-config.yml tls_server_config: cert_file: "/etc/kubernetes/certs/kubeletserver.crt" key_file: "/etc/kubernetes/certs/kubeletserver.key" client_auth_type: "RequireAndVerifyClientCert" client_ca_file: "/etc/kubernetes/certs/ca.crt"

i think we could address this in the node-exporter-startup.sh and check for the existence of /var/lib/kubelet/pki/kubelet-server-current.pem and if it exists use it. And if not use /etc/kubernetes/certs/kubeletserver.crt

that's tricky - kubeelet-server-currnet.pem won't exist until after kubelet requests it from the control plane after CSE exists, though we can check IMDS during CSE to see whether the feature itself will be enabled (like we currently do in configureKubeletServing)

alrighty. Adjusted the service to wait for kubelet. Startup script to check the imds cache file first. If that's not around then call imds ourselves. And at the very end check if /var/lib/kubelet/pki/kubelet-server-current.pem exists and everything else said no for some reason.

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path

Copilot

Pull request overview

Copilot reviewed 25 out of 90 changed files in this pull request and generated 5 comments.

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

parts/linux/cloud-init/artifacts/cse_config.sh

vhdbuilder/packer/packer_source.sh

vhdbuilder/packer/test/linux-vhd-content-test.sh

e2e/validators.go

Copilot

Pull request overview

Copilot reviewed 25 out of 90 changed files in this pull request and generated 3 comments.

e2e/validation.go

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

vhdbuilder/packer/install-node-exporter.sh

…cker Packer file provisioner cannot upload directories with nested subdirectories. Replace directory upload with individual file entries for all node-exporter baseline files.

… vhd-image* locations, updated test to make sure skip is missing when we want

…sable, startup args issue, imds log, unit description

…d tests

Copilot

Pull request overview

Copilot reviewed 25 out of 90 changed files in this pull request and generated 5 comments.

Copilot · 2026-02-18T06:46:00Z

vhdbuilder/packer/install-node-exporter.sh

+        version_info=$(node_exporter_extract_package_version "${package_json}" "ubuntu" "current")
+    fi
+
+    IFS=':' read -r NODE_EXPORTER_VERSION NODE_EXPORTER_REVISION NODE_EXPORTER_UBUNTU_VERSION <<< "${version_info}"
+


If jq/sed parsing ever fails (unexpected version format, missing JSON path, etc.), version_info can be empty and the subsequent IFS split will set NODE_EXPORTER_VERSION/REVISION to empty strings, leading to invalid download URLs/paths. Add a sanity check after computing version_info / after the split to fail fast with a clear error when parsing didn’t produce the expected fields.

Copilot · 2026-02-18T06:46:00Z

vhdbuilder/packer/packer_source.sh

+  # Skip for OSGuard, Flatcar, Kata, and Mariner (only AzureLinux 3.0 gets node-exporter)
+  if ! { isAzureLinuxOSGuard "$OS" "$OS_VARIANT" || isFlatcar "$OS" || grep -q "kata" <<< "$FEATURE_FLAGS" || isMariner "$OS"; }; then
+    cpAndMode $NODE_EXPORTER_STARTUP_SRC $NODE_EXPORTER_STARTUP_DEST 755


The inline comment says "only AzureLinux 3.0 gets node-exporter", but this block also runs on Ubuntu (it only skips OSGuard/Flatcar/Kata/Mariner). Please update the comment to match the actual install/copy behavior so future readers don't assume Ubuntu is excluded.

Copilot · 2026-02-18T06:46:00Z

vhdbuilder/packer/test/linux-vhd-content-test.sh

+  # Skip check for OS variants that don't have node-exporter, but verify the skip file is NOT present
+  # Mariner/CBLMariner is skipped - only AzureLinux 3.0 gets node-exporter
+  if [ "$os_sku" = "AzureLinuxOSGuard" ] || [ "$os_sku" = "Flatcar" ] || [ "$os_sku" = "CBLMariner" ] || echo "$FEATURE_FLAGS" | grep -q "kata"; then
+    local skip_file_check="/etc/node-exporter.d/skip_vhd_node_exporter"


The comment says Mariner/CBLMariner is skipped and "only AzureLinux 3.0 gets node-exporter", but this test (and the VHD build scripts) also expect node-exporter on Ubuntu. Please correct the comment to reflect the actual supported OSes to avoid confusion when adjusting the skip logic later.

Copilot · 2026-02-18T06:46:01Z

vhdbuilder/packer/install-dependencies.sh

+      # Skip for Flatcar, OSGuard, Kata, and Mariner (we only build AzureLinuxV3 now, mariner entry removed from components.json)
+      if isFlatcar "$OS" || isAzureLinuxOSGuard "$OS" "$OS_VARIANT" || [ "${IS_KATA}" = "true" ] || [ "$OS" = "$MARINER_OS_NAME" ]; then
+        echo "Skipping node-exporter installation for ${OS} ${OS_VARIANT:-default} (IS_KATA=${IS_KATA})"
+      else


This comment implies node-exporter is only for AzureLinuxV3/and that the Mariner entry was removed, but the "node-exporter" component is also defined for Ubuntu in components.json and is installed on Ubuntu builds. Please update the comment so it doesn't contradict the actual behavior.

Copilot · 2026-02-18T06:46:01Z

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

+    echo "WARNING: No kubelet serving certs found after ${WAIT_TIMEOUT}s, node-exporter will run without TLS. Restart the service after certs are available to enable TLS."
+fi
+
+# Configure TLS if we found valid cert paths
+if [ -n "$CERT_FILE" ] && [ -n "$KEY_FILE" ]; then
+    cat > "$TLS_CONFIG_PATH" <<EOF
+tls_server_config:
+  cert_file: "$CERT_FILE"
+  key_file: "$KEY_FILE"
+  client_auth_type: "RequireAndVerifyClientCert"
+  client_ca_file: "/etc/kubernetes/certs/ca.crt"
+EOF
+    TLS_CONFIG_ARG="--web.config.file=${TLS_CONFIG_PATH}"
+fi
+
+ARGS=(
+    --web.listen-address="${NODE_IP}:19100"
+    --no-collector.wifi
+    --no-collector.hwmon
+    --collector.cpu.info
+    --collector.filesystem.mount-points-exclude="^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)"
+    --collector.netclass.ignored-devices="^(azv.*|veth.*|[a-f0-9]{15})$"
+    --collector.netclass.netlink
+    --collector.netdev.device-exclude="^(azv.*|veth.*|[a-f0-9]{15})$"
+    --no-collector.arp.netlink
+)
+
+if [ -n "$TLS_CONFIG_ARG" ]; then
+    ARGS+=("$TLS_CONFIG_ARG")
+fi
+
+exec /opt/bin/node-exporter "${ARGS[@]}"


node-exporter-startup.sh falls back to starting node-exporter without any TLS or client authentication if kubelet serving certs are not found within the wait timeout. In that case, node-exporter listens on ${NODE_IP}:19100 with plaintext HTTP, exposing detailed node metrics to any client that can reach that IP/port (e.g., pods or VNet peers), which enables host reconnaissance and information disclosure. To avoid this, consider failing or delaying service startup until TLS certs are available (or binding only to localhost in the fallback) so that node-exporter is never exposed unauthenticated on the network.

chmill-zz requested review from AbelHu, Devinwong, YaoC, awesomenix, cameronmeissner, djsly, ganeshkumarashok, juan-lee, junjiezhang1997, lilypan26, phealy, r2k1, timmy-wright, yewmsft and zachary-bailey as code owners January 22, 2026 00:14

chmill-zz changed the title ~~Nodeexportershift~~ feat: node-exporter into vhd build Jan 22, 2026

chmill-zz force-pushed the nodeexportershift branch from 0d52887 to 60cc001 Compare January 22, 2026 00:40

chmill-zz temporarily deployed to test January 22, 2026 00:41 — with GitHub Actions Inactive

github-actions bot added the components This pull request updates cached components on Linux or Windows VHDs label Jan 22, 2026

chmill-zz temporarily deployed to test January 22, 2026 02:09 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch from f0f23e8 to eb8e316 Compare January 22, 2026 19:34

chmill-zz temporarily deployed to test January 22, 2026 19:34 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch 2 times, most recently from 68a2704 to d494735 Compare January 23, 2026 21:59

chmill-zz requested a review from yagmurbaydogan as a code owner January 23, 2026 21:59

chmill-zz force-pushed the nodeexportershift branch from d494735 to 277e340 Compare January 23, 2026 22:21

chmill-zz temporarily deployed to test January 23, 2026 22:21 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch from 277e340 to 1ae3542 Compare January 23, 2026 23:16

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path Outdated Show resolved Hide resolved

chmill-zz temporarily deployed to test February 6, 2026 03:28 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings February 7, 2026 00:07

chmill-zz force-pushed the nodeexportershift branch from c578ecf to d15e2e8 Compare February 7, 2026 00:07

chmill-zz temporarily deployed to test February 7, 2026 00:07 — with GitHub Actions Inactive

Copilot started reviewing on behalf of chmill-zz February 7, 2026 00:08 View session

Copilot AI reviewed Feb 7, 2026

View reviewed changes

cameronmeissner approved these changes Feb 9, 2026

View reviewed changes

chmill-zz temporarily deployed to test February 11, 2026 02:25 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings February 14, 2026 01:45

chmill-zz force-pushed the nodeexportershift branch from 4aa60c9 to 1dca0b4 Compare February 14, 2026 01:45

chmill-zz temporarily deployed to test February 14, 2026 01:45 — with GitHub Actions Inactive

Copilot started reviewing on behalf of chmill-zz February 14, 2026 01:45 View session

chmill-zz force-pushed the nodeexportershift branch from 1dca0b4 to 206ef20 Compare February 14, 2026 01:46

chmill-zz temporarily deployed to test February 14, 2026 01:47 — with GitHub Actions Inactive

Copilot AI reviewed Feb 14, 2026

View reviewed changes

e2e/validation.go Show resolved Hide resolved

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh Show resolved Hide resolved

vhdbuilder/packer/install-node-exporter.sh Show resolved Hide resolved

chmill-zz force-pushed the nodeexportershift branch from 206ef20 to 855a38c Compare February 16, 2026 03:45

chmill-zz temporarily deployed to test February 16, 2026 03:45 — with GitHub Actions Inactive

chmill-zz added 8 commits February 18, 2026 06:33

feat: install node-exporter during VHD build

f9788fb

fix: use explicit file uploads for node-exporter baseline files in pa…

b51db59

…cker Packer file provisioner cannot upload directories with nested subdirectories. Replace directory upload with individual file entries for all node-exporter baseline files.

chore: remove accidentally committed log file

2fbbbce

startup more better in cert checking with imds now

d96f7f3

Address PR review comments for node-exporter

6edd647

removed mariner components ref, changed to use packer_source, updated…

6c3c773

… vhd-image* locations, updated test to make sure skip is missing when we want

removed azlogsg ref, explicitly skip mariner just in case, readded di…

a9c4fe3

…sable, startup args issue, imds log, unit description

fixed osg ref issue, remove skip file for touch, packer used, adjuste…

f922c1c

…d tests

Copilot AI review requested due to automatic review settings February 18, 2026 06:39

chmill-zz force-pushed the nodeexportershift branch from 855a38c to f922c1c Compare February 18, 2026 06:39

chmill-zz temporarily deployed to test February 18, 2026 06:39 — with GitHub Actions Inactive

Copilot started reviewing on behalf of chmill-zz February 18, 2026 06:39 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

		@@ -0,0 +1,5 @@
		tls_server_config:
		cert_file: "/etc/kubernetes/certs/kubeletserver.crt"

Conversation

chmill-zz commented Jan 22, 2026

Uh oh!

cameronmeissner Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

chmill-zz Jan 24, 2026

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chmill-zz Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cameronmeissner Jan 27, 2026 •

edited

Loading