Skip to content

ci: publish Helm charts in a dedicated job gated on image builds#7046

Draft
TheRealShek wants to merge 1 commit into
pipe-cd:masterfrom
TheRealShek:fix/publish-image-chart-race
Draft

ci: publish Helm charts in a dedicated job gated on image builds#7046
TheRealShek wants to merge 1 commit into
pipe-cd:masterfrom
TheRealShek:fix/publish-image-chart-race

Conversation

@TheRealShek

Copy link
Copy Markdown

What this does

Moves the Helm chart publish out of the artifacts matrix job into a dedicated charts job gated on needs: artifacts, mirroring the structure publish_pipedv1_exp.yaml already uses (container_image_buildimage_chart).

Fixes #6999.

Problem

In publish_image_chart.yaml, the Install helm / Login to OCI using Helm / Publish helm charts steps live inside the artifacts matrix job. That matrix runs ~10 parallel cells, so:

  • Every cell runs Publish helm charts, and each one pushes all three charts (pipecd, piped, helloworld) to the same oci://ghcr.io/pipe-cd/chart ref — up to 10 redundant, concurrent pushes per run (a source of flaky "failed, passed on re-run" CI).
  • A fast cell (e.g. helloworld) publishes the pipecd/piped charts while those images are still building in slower cells → a window where the registry serves a chart whose image tag doesn't exist yet (ImagePullBackOff).
  • If a slow cell then fails, the charts stay published while their images were never pushed — a broken release the failed run can't undo.

Evidence

From run #29642136136 (all on ghcr.io, which the default charts reference):

Matrix cell Publish helm charts done Image push done
ghcr.io/pipe-cd, launcher 11:15:45Z (pushes all 3 charts) 11:15:32Z
ghcr.io/pipe-cd, piped 11:16:27Z 11:16:14Z (+29s after launcher's chart push)
ghcr.io/pipe-cd, pipecd 11:16:33Z 11:16:19Z (+34s after launcher's chart push)

The launcher cell published the piped/pipecd charts ~30s before those images finished pushing to ghcr. (On the gcr mirror the piped image lagged by ~7m44s, affecting installs that override image.repository to gcr.)

Fix

  • New charts job with needs: artifacts — GitHub starts it only after every matrix cell succeeds, so charts publish once, only after all images exist, and never if a build fails.
  • Repoint trigger-event-watcher and release-quickstart-manifests to needs: charts (the latter runs helm template against the just-published chart, so it must wait for the chart push).

Testing

  • actionlint — clean, aside from the pre-existing oracle-vm-8cpu-32gb-x86-64 custom self-hosted runner-label warning on the unchanged artifacts job.
  • YAML validated; job graph confirmed: artifactschartstrigger-event-watcher / release-quickstart-manifests.

Opened as a draft for maintainer feedback.

Move the Helm chart publish out of the artifacts matrix into a separate
`charts` job with `needs: artifacts`, mirroring publish_pipedv1_exp.yaml.
Charts now publish exactly once, only after every image has been pushed,
and a failed image build blocks the publish. Repoint the downstream
trigger-event-watcher and release-quickstart-manifests jobs to needs: charts.

Fixes pipe-cd#6999

Signed-off-by: TheRealShek <shadow.abhi100@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi @TheRealShek, welcome to PipeCD and thanks for opening your first pull request!

We’re really happy to have you here

Before your PR gets merged, please check a few important things below.


Helpful resources


DCO Sign-off

All commits must include a Signed-off-by line to comply with the Developer Certificate of Origin (DCO).

In case you forget to sign-off your commit(s), follow these steps:

For the last commit:

git commit --amend --signoff
git push --force-with-lease

For multiple commits:

git rebase --signoff origin/master
git push --force-with-lease

Run checks locally

Before pushing updates, please run:

make check

This runs the same checks as CI and helps catch issues early.


💬 Need help?

If anything is unclear, feel free to ask in this PR or join us on the CNCF Slack in the #pipecd channel.
You can get your Slack invite from: https://communityinviter.com/apps/cloud-native/cncf

Thanks for contributing to PipeCD! ❤️

@Ayushmore1214

Ayushmore1214 commented Jul 20, 2026

Copy link
Copy Markdown
Member

@TheRealShek Please mind getting assigned before opening a PR, this is okay for now as it is your first contribution here but please from next time be careful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: publish_image_chart pushes Helm charts from every matrix cell, before all container images exist (race condition)

2 participants