feat(helm): adding helm-diff target - #470
Conversation
cert-manager-prow
Bot
added
dco-signoff: yes
|
/cybr |
maelvls
added
the
cybr
| @TMP_NEW=$$(mktemp -d); | ||
| @OUTPUT_DIR=$$(mktemp -d); | ||
|
|
||
| $(HELM) template cert-manager --repo "oci://$(helm_chart_image_registry)" --version "$(helm_chart_old_version)" > $${TMP_OLD}/old.yaml; |
There was a problem hiding this comment.
helm_chart_image_registry probably also needs to be defined.
There was a problem hiding this comment.
https://github.com/hjoshi123/makefile-modules/blob/1dcfe9cc44ee5b1240b21962db3da06c64e1378a/modules/helm/helm.mk#L41 I think it is defined already here
There was a problem hiding this comment.
Is cert-manager the chart name? could you use a variable for that too?
There was a problem hiding this comment.
Oh right.. yes I can do that
|
I struggled a little with trying to run this PR from within cert-manager. I found that the # From the cert-manager project:
yq -i '(.targets["make/_shared"][] , .targets["make/_shared_new"][]) |= (.repo_ref = "refs/pull/470/head")' klone.yaml
make upgrade-klone$ make helm-diff
make: *** No rule to make target `helm-diff'. Stop.That's because we only import In trust-manager, I had the following: # From the trust-manager project:
yq -i '(.targets["make/_shared"][] , .targets["make/_shared_new"][]) |= (.repo_ref = "refs/pull/470/head")' klone.yaml
make upgrade-klone$ make helm-diff
make/_shared/helm///helm.mk:193: *** missing separator. Stop.Looks like this new target won't work in cert-manager until we are done with cert-manager/cert-manager#7718. But it will work for all of the other projects such as trust-manager, so I'm in favor of adding it once the above error is fixed. Great work! |
|
@hjoshi123 Hey, do you need any help on this? |
|
@maelvls sorry was traveling for kubecon and then the conference so got distracted. Will fix it this weekend. |
1dcfe9c to
4684693
Compare
|
@maelvls can you try running it now? I missed a |
| @TMP_OLD=$$(mktemp -d); | ||
| @TMP_NEW=$$(mktemp -d); | ||
| @OUTPUT_DIR=$$(mktemp -d); |
There was a problem hiding this comment.
In make, each command is run in its own sub-shell, so these variables won't be set in the $(HELM) template below. For example, in trust-manager, I'm seeing:
# From the trust-manager project:
$ yq -i '(.targets["make/_shared"][] , .targets["make/_shared_new"][]) |= (.repo_ref = "refs/pull/470/head")' klone.yaml
$ make upgrade-klone
$ make helm-diff
make/_shared/helm///helm.mk:196: warning: undefined variable `OUTPUT_DIR'One way is to continue the command with backslashes:
target:
@TMP_OLD=$$(mktemp -d); \
@TMP_NEW=$$(mktemp -d); \
@OUTPUT_DIR=$$(mktemp -d); \
$(HELM) template $(helm_chart_oci_name) ...Another way is to use $(eval export ...). It is a little more convenient as this method allows you to keep set that var for all subsequent commands:
target:
$(eval export TMP_OLD=$$(mktemp -d))
$(eval export TMP_NEW=$$(mktemp -d))
$(eval export OUTPUT_DIR=$$(mktemp -d))
$(HELM) template $(helm_chart_oci_name) ...
There was a problem hiding this comment.
Ah I didnt know this.. Thank you for pointing that out @maelvls.. still learning to write proper makefiles 😅
|
Also @maelvls had a thought should we somehow set the old version i.e. |
4684693 to
1a2056b
Compare
|
|
||
| $(eval export TMP_OLD=$$(mktemp -d)) | ||
| $(eval export TMP_NEW=$$(mktemp -d)) | ||
| $(eval export OUTPUT_DIR=$$(mktemp -d)) |
There was a problem hiding this comment.
Oops, i might have given you the wrong command, I'm now getting:
$ make helm-diff
make/_shared/helm///helm.mk:196: warning: undefined variable `mktemp -d'
make/_shared/helm///helm.mk:196: warning: undefined variable `mktemp -d'
make/_shared/helm///helm.mk:196: warning: undefined variable `mktemp -d'
make/_shared/helm///helm.mk:196: warning: undefined variable `mktemp -d'
Usage: make helm-diff helm_chart_old_version=<version>
make: *** [helm-diff] Error 1
I guess it could be:
| $(eval export OUTPUT_DIR=$$(mktemp -d)) | |
| $(eval export TMP_OLD=$(shell mktemp -d)) | |
| $(eval export TMP_NEW=$(shell mktemp -d)) | |
| $(eval export OUTPUT_DIR=$(shell mktemp -d)) |
I agree, having |
|
@hjoshi123 I'll unassign myself from this PR for now as I get asked "what's the progress on this" every day in our internal standups 😅 But I can continue helping you on this one 👍 |
|
@maelvls sorry for the delay. With holidays this week, I couldn't get to it. But yes I will try to make the changes and work with you to get this merged. Thank you for helping me out on this. |
Signed-off-by: hjoshi123 <mail@hjoshi.me> Signed-off-by: Hemant Joshi <mail@hjoshi.me>
1a2056b to
a8ec8c0
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@maelvls how does this look now? I did ponder on the helm old release and its not straight forward to fetch those.. I did come up with a shell script which could do that (pinged you on slack with it) but I feel that would make this PR polluted. We could do that as a follow-up. WDYT? |
helm's --repo flag only accepts classic HTTP chart repositories, so 'helm template NAME --repo oci://...' fails with 'is not a valid chart repository' on helm v3.19.0, v4.0.1 and v4.2.3. Pass the full OCI reference as the chart argument instead. The chart name is taken from the existing helm_chart_name variable rather than a new helm_chart_oci_name variable defaulting to cert-manager, which would have templated the wrong chart in every other consumer repo. The ignore pattern used BRE-style \| alternation with grep -E, where \| matches a literal pipe character, so no line was ever filtered and the diff was full of the version noise this target exists to remove. Use plain ERE alternation. Also filter helm's Pulled:/Digest: pull progress lines, which helm v4.2 prints to stdout. Drop the mktemp scratch dirs and tee: the eval'd variables didn't survive make's per-line sub-shells, and the diff.txt path was never printed so the file was unreachable. Process substitution is safe here because repository-base sets SHELL := /usr/bin/env bash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Richard Wall <richard@the-moon.net>
|
Hey @hjoshi123, thanks for turning my old release-preflight snippet into a make target — this is exactly the tool I keep wishing for at release time, and I always have to dig through old Slack threads to remember the command. I tried the target out and hit two bugs which stop it working, so rather than another slow round of review comments I hope you don't mind that I've pushed a fix commit directly to your branch, for expediency:
While I was there I dropped the How I tested — reproductions of both bugs, and the fixed target running in approver-policyBug 1 — the original recipe's helm invocation, run with approver-policy's pinned helm: $ ./_bin/tools/helm version --short
v4.0.1+g12500dd
$ ./_bin/tools/helm template cert-manager-approver-policy --repo "oci://quay.io/jetstack/charts/" --version v0.26.0
Error: looks like "oci://quay.io/jetstack/charts/" is not a valid chart repository or cannot be reached: failed to perform "FetchReference" on source: invalid referenceThe same error occurs with helm v3.19.0 and v4.2.3, and with or without the trailing slash. Bug 2 — the original $ printf 'keep: this\napp.kubernetes.io/version: v1\nhelm.sh/chart: foo\nchart: bar\n' | grep -vE 'app.kubernetes.io/version\|helm.sh/chart\|chart:\|appVersion:\|managed-by:\|meta.helm.sh/release-namespace'
keep: this
app.kubernetes.io/version: v1
helm.sh/chart: foo
chart: barThe fixed target — I copied the fixed $ make helm-diff helm_chart_old_version=v0.26.0
...
Successfully packaged chart and saved it to: _bin/scratch/helm/cert-manager-approver-policy-v0.23.0-dirty.tgz
Pulled: quay.io/jetstack/charts/cert-manager-approver-policy:v0.26.0
Digest: sha256:c33e5f5efc3dde5f1dd1f1bae62ab4270f0814dd0cd988af400c610148950b25
--- /dev/fd/63 2026-08-10 14:34:00.613130249 +0100
+++ /dev/fd/62 2026-08-10 14:34:00.614130268 +0100
@@ -971,6 +971,7 @@
maxDuration:
description: |-
MaxDuration defines the maximum duration for a certificate request.
+ for.
Values are inclusive (i.e. a value of `1h` will accept a duration of
`1h`). MinDuration and MaxDuration may be the same value.
If set, a duration _must_ be requested in the CertificateRequest.
@@ -1216,7 +1217,7 @@
resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
verbs: ["list", "watch"]
-- apiGroups: ["", "events.k8s.io"]
+- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
@@ -1353,7 +1354,7 @@
serviceAccountName: cert-manager-approver-policy
containers:
- name: cert-manager-approver-policy
- image: "quay.io/jetstack/cert-manager-approver-policy:v0.26.0"
+ image: "quay.io/jetstack/cert-manager-approver-policy:v0.23.0-dirty"
imagePullPolicy: IfNotPresent
ports:
- name: webhook
@@ -1418,7 +1419,7 @@
- CREATE
- UPDATE
resources:
- - certificaterequestpolicies
+ - "*/*"
admissionReviewVersions: ["v1", "v1beta1"]
timeoutSeconds: 5
failurePolicy: FailReal chart changes show up; the version-label noise does not. The usage guard: $ make helm-diff
Usage: make helm-diff helm_chart_old_version=<version>
make: *** [make/_shared/helm///helm.mk:195: helm-diff] Error 1One heads-up for reviewers: this target won't be reachable from cert-manager itself yet — cert-manager only includes with claude fable-5 |
There was a problem hiding this comment.
Pull request overview
Adds a new helm-diff make target to compare rendered Helm templates between the current chart artifact and a specified older chart version pulled from an OCI registry. This supports reviewing chart changes across releases (per the referenced cert-manager issue).
Changes:
- Introduces
HELM_IGNORE_FIELDSandhelm_chart_old_versionvariables to support templated diffing. - Adds a
helm-difftarget that runshelm templatefor the old OCI version and the current built chart, then diffs the outputs.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| helm_chart_archive := $(bin_dir)/scratch/helm/$(helm_chart_name)-$(helm_chart_version).tgz | ||
| helm_digest_path := $(bin_dir)/scratch/helm/$(helm_chart_name)-$(helm_chart_version).digests | ||
| helm_digest = $(shell head -1 $(helm_digest_path) 2> /dev/null) | ||
| HELM_IGNORE_FIELDS ?= ^Pulled:|^Digest:|app.kubernetes.io/version|helm.sh/chart|chart:|appVersion:|managed-by:|meta.helm.sh/release-namespace |
There was a problem hiding this comment.
Declined, deliberately. These are long literal strings, so an unescaped . matching one arbitrary character is a negligible false-positive risk for what is a noise filter over a human-read diff, and escaping every dot would make the default much harder to read and override. We also deliberately use POSIX ERE rather than fgrep/grep -F fixed strings: it keeps the ^Pulled:/^Digest: anchors, works identically with GNU and BSD grep (macOS), and GNU grep has deemed fgrep obsolescent since 3.8 — on Fedora it is now a wrapper script that prints fgrep: warning: fgrep is obsolescent; using grep -F to stderr on every run.
with claude fable-5
| .PHONY: helm-diff | ||
| helm-diff: $(helm_chart_archive) | $(NEEDS_HELM) |
There was a problem hiding this comment.
Done in f9a8ff0 — added a ## doc comment with the [shared] Helm Chart category, matching the other targets in this file. Verified in a consumer repo:
$ make help
[shared] Helm Chart
helm-diff > Diff the locally built Helm chart against a released version,
ignoring version-label noise. Set helm_chart_old_version to the
previously released chart version to compare against.
helm-chart > Create a helm chartwith claude fable-5
| @diff -u \ | ||
| <($(HELM) template "oci://$(helm_chart_image_registry)$(helm_chart_name)" --version "$(helm_chart_old_version)" | grep -vE '$(HELM_IGNORE_FIELDS)') \ | ||
| <($(HELM) template "$(helm_chart_archive)" | grep -vE '$(HELM_IGNORE_FIELDS)') \ | ||
| || true |
There was a problem hiding this comment.
All three points addressed in f9a8ff0:
|| trueis now|| [ $? -eq 1 ], so diff exit 2 (trouble) fails the target while exit 1 (charts differ) does not. Note a helm failure inside process substitution cannot be caught by diff's exit code at all — it is discarded and diff just sees empty input — so a mistypedhelm_chart_old_versionpreviously produced a whole-chart diff and exit 0. The old version is now validated up front withhelm show chart, which fails the target with helm's own error on a missing version or registry auth failure (tested:v99.99.99now exits non-zero withFetchReference ... not found).$(INSTALL_OPTIONS)is passed to bothhelm templateinvocations, consistent with the verify targets in this file.grep -vE --terminates option parsing in case the pattern is overridden with a leading dash.
with claude fable-5
Add a doc comment so helm-diff appears in the make help output, in the [shared] Helm Chart category like the other targets in this file. Pass INSTALL_OPTIONS to both helm template invocations, consistent with the verify-pod-security-standards and verify-helm-kubeconform targets, and terminate grep option parsing with -- in case HELM_IGNORE_FIELDS is overridden with a pattern starting with a dash. Replace '|| true' with '|| [ $? -eq 1 ]' so that diff exit code 2 (trouble) fails the target while exit code 1 (charts differ) does not. A helm failure inside process substitution cannot be caught that way: its exit code is discarded and diff just sees empty input, so a mistyped helm_chart_old_version previously produced a whole-chart diff and exit 0. Validate the old chart version with helm show chart first, so a missing version or registry auth failure fails the target with helm's own error message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Richard Wall <richard@the-moon.net>
Motivation: cert-manager/cert-manager#8183
CyberArk tracker: VC-46541