Skip to content

feat: Add S3 package pool storage backend#1607

Open
mmxxixx wants to merge 2 commits into
aptly-dev:masterfrom
mmxxixx:feature/s3-package-pool
Open

feat: Add S3 package pool storage backend#1607
mmxxixx wants to merge 2 commits into
aptly-dev:masterfrom
mmxxixx:feature/s3-package-pool

Conversation

@mmxxixx

@mmxxixx mmxxixx commented Jul 18, 2026

Copy link
Copy Markdown

Partially addresses #843

Requirements

All new code should be covered with tests, documentation should be updated. CI should pass.

Description of the Change

Adds S3 as a packagePoolStorage backend, so the package pool can live in an
S3 bucket (or any S3-compatible store such as MinIO) instead of on local disk.

This implements the pool half of #843, following the suggestion in that issue.
The implementation follows azure/package_pool.go closely: same content-addressed
pool path scheme, same checksum storage flow, and it reuses the existing
S3PublishRoot config struct and the aws-sdk-go-v2 dependency already used for
S3 publishing.

  • Unit tests run against the existing in-process mock S3 server (no credentials
    needed) and cover import/dedup, verify, open, remove, prefixed pools, and
    both checksum cache levels.
  • PackagePoolStorage marshaling was reworked into per-type wrappers: embedding
    S3PublishRoot alongside AzureEndpoint in the previous shared wrapper would
    silently drop the colliding prefix/endpoint JSON tags (and panic in
    yaml.v3).
  • Functional test S3RepoTest mirrors AzureRepoTest (t09_repo) and skips
    cleanly when AWS credentials are not present. It has been verified against
    real AWS S3, as has the full flow end-to-end (repo add, dedup, publish,
    mirror update, db cleanup, apt client install).
  • New config documented in debian/aptly.conf and the man page.

Checklist

  • allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (no new commands or flags)
  • documentation updated
  • author name in AUTHORS

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.23105% with 88 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.90%. Comparing base (f59b0d2) to head (397c225).

Files with missing lines Patch % Lines
s3/package_pool.go 69.72% 40 Missing and 26 partials ⚠️
context/context.go 0.00% 16 Missing and 1 partial ⚠️
utils/config.go 88.09% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1607      +/-   ##
==========================================
- Coverage   77.37%   76.90%   -0.48%     
==========================================
  Files         165      166       +1     
  Lines       15747    16002     +255     
==========================================
+ Hits        12185    12307     +122     
- Misses       2356     2468     +112     
- Partials     1206     1227      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant