feat: add CustomData/protectedSettings size guards for hotfix PRs

[Abigail Liang (abigailliang-aks-sig-node)]

Summary

• Add Go-test guard (pkg/agent/customdata_size_guard_test.go) for the scriptless NBC CSE path
• Guard 1 (TestCustomDataSizeWithHotfix): asserts slim Mode B CustomData (P + H) < 87,380
• Add CI step in hotfix-generate.yml to run guard after hotfix injection, before commit
• Design doc in docs/design-customdata-size-guard.md

⚠️ Backward compatibility note: The workflow and test live on main. New official/* branches fork from main weekly, so they inherit the guard automatically. Older official/* branches lack it, but those map to VHDs that either already bake the fix (post-0729) or are approaching end of 6-month support. Backporting is not necessary.

Context

Bridge solution until #9101 (embed hotfix scripts in ANC binary) lands. See design doc for full analysis of encoding chain, Mode A/B architecture, and why only H (not R) can overflow Mode B CustomData.

Test plan

• Guard passes locally with current (no-hotfix) nodecustomdata.yml: 1,756 / 87,380 bytes
• Validate with a mock hotfix injection that approaches the limit

[github-actions]

Windows Unit Test Results

  3 files   12 suites   51s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit f6e897c.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Adds automated size-guardrails for the scriptless NBC CSE flow to prevent hotfix PRs from accidentally exceeding Azure’s VMSS CustomData and CRP protectedSettings limits, and wires those checks into the hotfix generation workflow.

Changes:

• Added Go tests to guard Mode B slim CustomData size (P+H) and protectedSettings command size (R).
• Added worst-case certificate fixtures used to model RP count-max certificate inputs.
• Added a CI step in hotfix-generate.yml to run the guards post hotfix-injection, plus a design doc capturing the size model and rationale.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/agent/customdata_size_guard_test.go	Introduces two size-guard Go tests for scriptless Mode B CustomData and protectedSettings.
pkg/agent/testdata/customdata_size_guard/worst_case_ca_trust_certs.txt	Provides worst-case Custom CA trust cert fixture data for the protectedSettings guard.
pkg/agent/testdata/customdata_size_guard/worst_case_proxy_trusted_ca.txt	Provides worst-case proxy TrustedCA bundle fixture data for the protectedSettings guard.
.github/workflows/hotfix-generate.yml	Runs the new guards after hotfix injection and before committing generated hotfix files.
docs/design-customdata-size-guard.md	Documents the two-limit model (CustomData vs protectedSettings) and why two guards are needed.

Suppressed comments (1)

pkg/agent/customdata_size_guard_test.go:166

• The protectedSettings limit is on the serialized JSON payload, not just the commandToExecute string. Since this guard is intentionally close to the limit, it should measure len(json.Marshal(map[string]any{"commandToExecute": cseCmd})) (and use that value for both the hard-limit and soft-margin checks).

	cseCmd := templateGenerator.getNodeBootstrappingCmd(config)

	if len(cseCmd) >= protectedSettingsMaxLength {
		t.Fatalf("Mode B protectedSettings CSE command is %d bytes, must be < the CRP protectedSettings "+
			"limit (%d). Worst-case customer certs (R: %d CA trust + %d-cert proxy bundle) overflow "+

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

pkg/agent/customdata_size_guard_test.go:17

• PR description and design doc describe two size guards (including TestProtectedSettingsSizeWithWorstCaseCerts for protectedSettings), but this file only implements Guard 1 (TestCustomDataSizeWithHotfix). As-is, there is no automated protection for the protectedSettings (~65,535 bytes) constraint that Mode B relies on; either add the second guard (and any required fixtures) or update the PR/doc to remove the claim that it exists.

// CustomData size guard for the scriptless NBC CSE path.
//
// Background (see docs/design-customdata-size-guard.md):
//
//	In Mode B (slim CustomData), the generated payload carries P (platform content) + H (hotfix-
//	injected scripts). R (customer certs) is moved to protectedSettings and is now capped by RP
//	(aks-rp PR 16755346: 10 KiB per field for new clusters), so only H can overflow CustomData.
//
//	This guard asserts that the slim Mode B CustomData stays under MaxCustomDataLength (87,380).

docs/design-customdata-size-guard.md:122

• This doc states that Guard 2 (TestProtectedSettingsSizeWithWorstCaseCerts) is implemented in pkg/agent/customdata_size_guard_test.go and even references a protectedSettingsMaxLength constant, but neither the test nor the constant exists in this PR/repo state. This makes the design doc misleading; please either implement Guard 2 as described (and keep the doc accurate) or revise §5 to document only the guard(s) that actually exist.

## 5. Two Guards (Implemented)

Investigation of the Mode B fallback revealed that **H and R end up in different Azure fields**, so a single CustomData check is insufficient. Two independent guards are implemented in
`pkg/agent/customdata_size_guard_test.go` (plain `Test*` funcs so they run under `go test -run`).

### Guard 1 — `TestCustomDataSizeWithHotfix` (H vs CustomData)

- Builds a scriptless config and forces Mode B (`config.ScriptlessCSEProvisionMode = true`) so
  `getScriptlessBoothook` early-returns the **slim** CustomData (`P + H`).
- Asserts `len(slim) < MaxCustomDataLength` (87,380).
- In the hotfix CI this runs **after** `hotfix_generate.py` injects the real scripts into
  `nodecustomdata.yml`; because the template is `//go:embed`'d, `go test` recompiles and measures
  the **actual injected H** — no synthetic H is needed.

### Guard 2 — `TestProtectedSettingsSizeWithWorstCaseCerts` (R vs protectedSettings)

- Builds a worst-case customer-input (`R`) config and renders the Mode B CSE command via
  `getNodeBootstrappingCmd` (the string RP places into the extension `protectedSettings`).
- Asserts `len(cseCmd) < protectedSettingsMaxLength` (65,535, the CRP hard limit), and **warns**
  (without failing) once it crosses a 60,000 soft margin.

.github/workflows/hotfix-generate.yml:92

• The PR description says CI should run two guards after hotfix injection (CustomData + protectedSettings), but this workflow only runs TestCustomDataSizeWithHotfix. If Guard 2 is intended, add it to the workflow (and ensure it exists in the code); otherwise, the PR description/design doc should be updated to match what CI actually enforces.

      # Runs AFTER hotfix injection (so the //go:embed'd nodecustomdata.yml carries the injected
      # scripts H) and BEFORE the commit step (so an oversized hotfix fails the PR before its
      # content is committed). See docs/design-customdata-size-guard.md.
      #   Guard 1 (TestCustomDataSizeWithHotfix): slim Mode B CustomData (P + H) < 87,380.
      - name: Set up Go
        uses: actions/setup-go@v7
        with:
          go-version: '1.25'
      - name: Validate CustomData size
        run: |
          go test ./pkg/agent/ -run TestCustomDataSizeWithHotfix -v -count=1

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

pkg/agent/customdata_size_guard_test.go:27

• 🔴 High Risk — 🖥️ Cross-OS: The guard renders only Ubuntu, but getScriptlessBoothook uses a different ignition wrapper for ACL/Flatcar and the injected nodecustomdata.yml blocks render different distro-specific script bodies. A hotfix whose ACL, Mariner, OSGuard, or Flatcar payload exceeds the limit can therefore pass this Ubuntu-only check. Make this table-driven over every distinct supported rendering class and enforce the limit for each payload.

	agentPoolProfile := &datamodel.AgentPoolProfile{
		Name:   "nodepool1",
		OSType: datamodel.Linux,
		Distro: datamodel.AKSUbuntuContainerd2204Gen2,
	}

docs/design-customdata-size-guard.md:129

• The PR title/summary promises CustomData and protectedSettings size guards, but this design and the diff retain only the CustomData guard. Update the PR metadata to describe a single CustomData guard, or restore the promised protectedSettings guard, so reviewers and release owners do not assume CI validates both Azure limits.

### Guard 2 — `TestProtectedSettingsSizeWithWorstCaseCerts` (R vs protectedSettings) — **removed**

Originally rendered a worst-case customer-input (`R`) config through `getNodeBootstrappingCmd` and
asserted `len(cseCmd) < protectedSettingsMaxLength` (65,535). **Removed** because the risk it covered
is no longer reachable — see §6 for the byte-size RP validation (PR 16755346) and the empirical

pkg/agent/customdata_size_guard_test.go:22

• 🟡 Medium Risk — 🧪 Test Coverage: A minimal public-cloud configuration does not reserve the largest platform-owned P. Scriptless AKS custom-cloud configurations add the compressed init-aks-cloud.sh entry (as exercised in pkg/agent/baker_test.go:1753-1818), so a hotfix near the threshold can pass here and still overflow for those nodes. Measure representative maximum-P configurations, rather than only the minimum fixture, or reserve their additional bytes explicitly.

This issue also appears on line 23 of the same file.

// newScriptlessGuardConfig builds a minimal scriptless NBC config.
func newScriptlessGuardConfig() *datamodel.NodeBootstrappingConfiguration {

docs/design-customdata-size-guard.md:36

• This formula contradicts the corrected Mode B model later in §6: R is carried in protectedSettings, not in CustomData, so P + H + R ≤ C is not the constraint this guard enforces. State the independent Mode B constraints here (CustomData: P + H < C; protectedSettings: NBC command including R below its CRP limit) to avoid basing future sizing work on the known-false model.

This issue also appears on line 125 of the same file.

Constraint:  P + H + R ≤ C

Or equivalently:  H ≤ C − max(P + R)