Skip to content

helm diff upgrade reports no changes after a failed upgrade (baseline is the failed revision) #1045

Description

@sepich

Summary

After a helm upgrade fails, helm diff upgrade reports no changes — even though the
new manifest was never applied to the cluster. The next diff compares the new render against
the failed revision, which already stores that same render, so the output is empty while the
cluster still runs the old spec.

Versions

  • helm-diff: 3.12.4 (also verified against current main)
  • helm: v3.18.4
  • Kubernetes: 1.31 (GKE)

Reproduction

  1. Deploy a release successfully:
    helm upgrade --install myapp ./chart -n myapp
    
  2. Change a value that alters a StatefulSet/Deployment pod spec, then force the upgrade to
    fail. Any failure mode works; a short atomic timeout is the quickest:
    helm upgrade --install myapp ./chart -n myapp --atomic --timeout 10s
    # Error: UPGRADE FAILED: context deadline exceeded
    
    Another common trigger is a chart containing a CR whose CRD is not installed — the upgrade
    exits non-zero, having partially applied.
  3. Confirm the cluster does not have the new value:
    kubectl -n myapp get sts myapp -o jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}'
    # old value
    
  4. Confirm helm recorded the failure:
    helm history myapp -n myapp
    # 50   failed   Upgrade "myapp" failed: context deadline exceeded
    
  5. Diff again with the same chart and values:
    helm diff upgrade myapp ./chart -n myapp
    # (no output)
    

Expected

A diff showing the pending change, since the cluster does not have it.

Actual

Empty diff. --detailed-exitcode likewise reports no changes.

Cause

cmd/helm.go:

func getRelease(release, namespace, kubeContext string) ([]byte, error) {
	args := []string{"get", "manifest", release}
	...
}

helm get manifest returns the newest revision regardless of status. Helm writes the
rendered manifest into the release record before applying (as pending-upgrade, then switch to
failed on error), so a failed revision holds the manifest that was never fully applied — and
helm-diff uses it as the baseline.

Suggested fix

I can see these options:

  1. Nice solution would be to use the newest revision with status: deployed as the baseline rather than the newest revision. But that could be considered breaking change, and might require separate cli arg
  2. Add --revision to helm diff upgrade should be small and non-breaking change

I can work on a PR, if that is ok

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions