feat(LocalDNS): Add CPU and Memory metrics for LocalDNS systemd unit

[saewoni]

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

[Copilot]

Pull request overview

This PR introduces a comprehensive node-exporter installation and configuration system that conditionally collects CPU and memory metrics for the LocalDNS systemd unit when enabled. The implementation spans VHD build-time installation, CSE provisioning-time configuration, and E2E validation.

Changes:

• Adds node-exporter package installation during VHD build for Ubuntu and Azure Linux (skipped on Flatcar, OSGuard, Mariner, and Kata)
• Implements dynamic TLS certificate detection and configuration for secure metrics exposure
• Conditionally enables LocalDNS systemd metrics collection when localdns.service is present
• Includes comprehensive E2E validation for node-exporter installation and runtime behavior

Reviewed changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
vhdbuilder/packer/install-node-exporter.sh	New script for installing node-exporter packages from packages.microsoft.com during VHD build
parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh	Startup script with dynamic TLS cert detection, IMDS-based rotation detection, and conditional LocalDNS metrics
parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter.service	Systemd service unit for node-exporter with kubelet dependency
parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.{service,path}	Auto-restart mechanism watching certificate file changes
parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/node-exporter.d/web-config.yml	TLS configuration template (overwritten at runtime)
parts/linux/cloud-init/artifacts/cse_config.sh	Adds configureNodeExporter function with skip file detection
parts/linux/cloud-init/artifacts/cse_main.sh	Calls configureNodeExporter during nodePrep
parts/linux/cloud-init/artifacts/cse_helpers.sh	Defines ERR_NODE_EXPORTER_START_FAIL error code
parts/common/components.json	Defines node-exporter package versions for Ubuntu and Azure Linux
vhdbuilder/packer/install-dependencies.sh	Integrates node-exporter installation into dependency installation loop
vhdbuilder/packer/packer_source.sh	Copies node-exporter files to final destinations during VHD build
vhdbuilder/packer/*.json	Adds file provisioners for node-exporter artifacts across all Linux VHD builders
vhdbuilder/packer/test/linux-vhd-content-test.sh	VHD content validation test for node-exporter installation
vhdbuilder/packer/imagecustomizer/azlosguard/*	OSGuard-specific cleanup and file provisioning
e2e/validators.go	E2E validator for node-exporter service, binaries, config, and network port
e2e/validation.go	Adds ValidateNodeExporter to common Linux validation suite
pkg/agent/testdata/CustomizedImage/CustomData	Regenerated snapshot test data (via make generate)

Comments suppressed due to low confidence (3)

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh:26

• The IMDS query on line 25 lacks error handling for network timeouts or IMDS service unavailability. If IMDS is temporarily unavailable, this could block node-exporter startup for up to 60 seconds (--max-time 60) even though there's fallback logic. Consider reducing the timeout to 10-15 seconds since the cached IMDS file should be the primary source, and IMDS unavailability at boot is unusual.

    IMDS_RESPONSE=$(curl -fsSL -H "Metadata: true" --noproxy "*" --retry 20 --retry-delay 2 --retry-connrefused --connect-timeout 5 --max-time 60 "http://169.254.169.254/metadata/instance?api-version=2021-02-01" 2>/dev/null)
fi

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh:34

• The jq command on line 33 uses test() with the "i" flag for case-insensitive matching, but it will return "false" (string) instead of false (boolean) if jq fails or the tag is not found. This could cause ROTATION_DISABLED to be set to the string "false" instead of the boolean false, which would fail the string comparison on line 44. While the || echo "false" fallback handles jq errors, the logic should use proper boolean handling or string comparison consistently throughout.

    ROTATION_DISABLED=$(echo "$IMDS_RESPONSE" | jq -r '.compute.tagsList | map(select(.name | test("aks-disable-kubelet-serving-certificate-rotation"; "i")))[0].value // "false" | test("true"; "i")' 2>/dev/null || echo "false")
fi

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh:79

• The startup script waits up to 5 minutes (300 seconds) for kubelet serving certificates to appear before starting node-exporter. However, if the certs never appear, node-exporter will start without TLS (line 78). This creates a security concern: node-exporter will expose metrics over an unauthenticated HTTP endpoint on the node's IP. Consider either (1) failing to start if certs are not found after the timeout, or (2) binding to localhost only when TLS is not configured, or (3) documenting that this is an expected degraded mode that should trigger monitoring alerts.

while [ $WAIT_ELAPSED -lt $WAIT_TIMEOUT ]; do
    if [ "$ROTATION_DISABLED" = "true" ]; then
        # Rotation disabled - wait for static certs
        if [ -f "/etc/kubernetes/certs/kubeletserver.crt" ] && [ -f "/etc/kubernetes/certs/kubeletserver.key" ]; then
            break
        fi
    else
        # Rotation enabled - wait for rotation cert
        if [ -f "/var/lib/kubelet/pki/kubelet-server-current.pem" ]; then
            break
        fi
    fi
    # Also check the other cert type in case IMDS tag detection was wrong
    if [ -f "/var/lib/kubelet/pki/kubelet-server-current.pem" ] || \
       { [ -f "/etc/kubernetes/certs/kubeletserver.crt" ] && [ -f "/etc/kubernetes/certs/kubeletserver.key" ]; }; then
        break
    fi
    sleep $WAIT_INTERVAL
    WAIT_ELAPSED=$((WAIT_ELAPSED + WAIT_INTERVAL))
done

# Detect TLS cert paths
# Priority: rotation cert > static certs > skip TLS
CERT_FILE=""
KEY_FILE=""

if [ -f "/var/lib/kubelet/pki/kubelet-server-current.pem" ]; then
    CERT_FILE="/var/lib/kubelet/pki/kubelet-server-current.pem"
    KEY_FILE="/var/lib/kubelet/pki/kubelet-server-current.pem"
    echo "Using kubelet serving certificate rotation cert: $CERT_FILE"
elif [ -f "/etc/kubernetes/certs/kubeletserver.crt" ] && [ -f "/etc/kubernetes/certs/kubeletserver.key" ]; then
    CERT_FILE="/etc/kubernetes/certs/kubeletserver.crt"
    KEY_FILE="/etc/kubernetes/certs/kubeletserver.key"
    echo "Using static kubelet serving certs: $CERT_FILE, $KEY_FILE"
else
    echo "WARNING: No kubelet serving certs found after ${WAIT_TIMEOUT}s, node-exporter will run without TLS. Restart the service after certs are available to enable TLS."
fi