improvement(packages): Migrate CLI package publishing to per-arch RPM layout and add Cloudsmith backfill pipeline#260
Conversation
…d add backfill workflow for historical packages from Cloudsmith - Updated `upload_to_s3.sh` to upload RPM packages to architecture-specific directories on S3, preventing package collapse across architectures. - Introduced a new GitHub Actions workflow `backfill-cloudsmith.yml` for backfilling historical CLI packages from Cloudsmith to the AWS package repository. - Added `backfill_from_cloudsmith.sh` script to facilitate the migration of old packages, ensuring they are signed with the same keys as current releases. - Created `validate_backfill.sh` to validate the installation of backfilled packages from the new AWS repo. - Adjusted `setup.rpm.sh` to support architecture-specific base URLs for RPM repositories.
…simplify deployment process
|
💬 Discussion in Slack: #pr-review-cli-260-improvement-packages-migrate-cli-package-publishing-to-per-arc Posted by Review Police — reviews, comments, new commits, and CI failures will stream into this channel. |
|
| Filename | Overview |
|---|---|
| .github/workflows/backfill-cloudsmith.yml | New manual-trigger workflow for backfilling Cloudsmith packages; contains shell injection risk via free-text inputs.version interpolated directly in run steps, and downloads the AWS CLI installer without signature verification. |
| scripts/backfill_from_cloudsmith.sh | Large new backfill script; well-structured and idempotent, but fetches arbitrary Cloudsmith cdn_url values without host validation (SSRF), and the legacy flat-layout migration does not clean up source objects after copying. |
| .github/workflows/release_build_infisical_cli.yml | Pins mkrepo/boto3/univers versions, replaces pip-installed AWS CLI v1 with bundled v2, and widens CloudFront invalidation to /rpm/*; the AWS CLI install lacks checksum verification, otherwise changes are correct. |
| upload_to_s3.sh | Correct migration from flat rpm/Packages/ to per-arch rpm/<arch>/Packages/ with per-arch mkrepo runs; logic is sound and mirrors the backfill script's approach. |
| scripts/setup/setup.rpm.sh | One-line change updates baseurl to use $basearch so yum/dnf resolves the correct per-arch directory; the \$basearch escaping is correct for the generated repo file. |
| scripts/validate_backfill.sh | New validation script that runs container-based installs for deb/rpm/apk to verify backfilled packages; straightforward and well-guarded, but inputs.version injection risk is inherited from the calling workflow. |
Comments Outside Diff (1)
-
scripts/backfill_from_cloudsmith.sh, line 660-694 (link)Legacy flat
rpm/Packages/objects are never cleaned up after migrationrpm_migrate_flat_layout()copies files from the old flatrpm/Packages/path to the new per-arch directories, but leaves the original objects in S3. Clients that still have the old flatbaseurlwill keep resolving fromrpm/Packages/. Consider deleting or documenting the need to delete the source objects after a successful copy.
Reviews (1): Last reviewed commit: "Remove environment protection for produc..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
…r inputs, improving readability and maintainability. Updated echo statements and condition checks to utilize new variable references.
PrestigePvP
left a comment
PrestigePvP
left a comment
There was a problem hiding this comment.
The bash scripts are really hard to review - I'm going to assume you know how they work and what they do.
I left a couple comments
| run: sudo apt-get update && sudo apt-get install -y rpm unzip | ||
| # AWS CLI v2 (pinned), kept outside the pip env: awscli (pip, v1) cannot | ||
| # coexist with mkrepo's boto3 1.17.5 (different s3transfer). v2 is bundled. | ||
| - name: Install AWS CLI v2 (pinned) |
| APK_PRIVATE_KEY_PATH: /tmp/infisical-apk.rsa | ||
| - name: Invalidate Cloudfront cache | ||
| if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && !inputs.dry_run) | ||
| run: aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths '/rpm/Packages/*' '/rpm/repodata/*' '/deb/dists/stable/*' '/apk/stable/main/*' |
There was a problem hiding this comment.
Why are we removing this path?
2 participants
Description 📣
We're migrating the Infisical CLI package repository from Cloudsmith to a self-hosted AWS setup (S3 + CloudFront) to reduce third-party dependency and gain full control over package distribution. New releases already publish to the AWS repo, but all historical versions still only exist on Cloudsmith. This PR adds the tooling to backfill those old versions into the new repo, so that existing users pinned to older CLI versions continue to resolve packages seamlessly once we sunset the Cloudsmith repo, and fixes the RPM layout to use per-architecture directories, which is required for multi-arch support with mkrepo.
upload_to_s3.shnow publishes RPM packages torpm/<arch>/Packages/(one directory per architecture) instead of a single flatrpm/Packages/. This prevents mkrepo from collapsing multi-arch versions into a single entry.setup.rpm.shis updated to usebaseurl=.../rpm/$basearchso clients resolve the correct arch directory.scripts/backfill_from_cloudsmith.sh): One-time migration tool that copies all historical deb/rpm/apk packages from the legacy Cloudsmith repo into the AWS-hosted repo. Idempotent, resumable, supports dry-run, and signs packages with the same keys the release workflow uses.scripts/validate_backfill.sh): Spins up deb/rpm/apk containers, configures them via the real setup scripts, and verifies each backfilled version installs and runs correctly through CloudFront..github/workflows/backfill-cloudsmith.yml): Manual-trigger workflow that runs the backfill with auditable CI, supports test/prod target selection, and optionally runs validation after publishing.mkrepo,boto3, anduniversversions; replaced pip-installed AWS CLI v1 with bundled v2 to avoid dependency conflicts with boto3; updated CloudFront invalidation paths to match the new per-arch layout.Type ✨