Skip to content

deb: rename published indexes concurrently - #1633

Open
barnumbirr wants to merge 1 commit into
aptly-dev:masterfrom
barnumbirr:concurrent-index-renames
Open

deb: rename published indexes concurrently#1633
barnumbirr wants to merge 1 commit into
aptly-dev:masterfrom
barnumbirr:concurrent-index-renames

Conversation

@barnumbirr

Copy link
Copy Markdown

Fixes #1632

Requirements

All new code should be covered with tests, documentation should be updated. CI should pass.

Description of the Change

RenameFiles walks renameMap one entry at a time. Every rename is a round trip, and on an S3 backend RenameFile is a CopyObject followed by a Remove, so a publish that rewrites a dozen indexes spends most of its wall clock waiting on them in sequence. #1632 has the traces and the background.

This runs them concurrently, bounded at four in flight to match the default used for concurrent package uploads in #1616. The first error is returned rather than the last, and the remaining renames still run, so a failure does not leave part of the set staged behind a half finished switch.

Nothing else changes: the same renames happen, the published object set and its sizes are identical before and after, and the .tmp staging that makes every index and the signature switch together is untouched.

Measurements

Against MinIO behind a proxy adding 50ms each way, publishing a repository whose Contents is dominated by one package carrying 19,500 files. publish update, n=5 each:

median range
master 2703ms 2659-2760
this PR 1624ms 1622-1679

A 40% cut. For reference, removing the staging entirely gets to 1240ms, so this captures about 75% of what the rename phase could ever give back. The rest is not worth it: writing indexes straight to their final names means a failure part way through the upload leaves the archive with some indexes new and some old and no way back, which is exactly what the .tmp set prevents.

Four in flight is a constant rather than a config key, since nothing yet asks for a different value. Happy to make it configurable the way #1616 does if you would rather the two matched.

Notes for review

FinalizeAll is serial in the same way, so the upload phase has similar headroom, but it calls the signer and I have not looked at whether that is safe to run concurrently. Batching the Remove calls into one DeleteObjects was also considered; it can only reach part of what survives this change and would need a change to the PublishedStorage interface that every backend implements.

The test file is named index_files_rename_test.go rather than index_files_test.go deliberately: #1626 adds a file by the latter name, and this way the two can land in either order without an add/add conflict.

Checklist

  • allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • documentation updated
  • author name in AUTHORS

RenameFiles had no coverage at all before this. It now has four cases: every entry renamed, the concurrency limit respected, a failure reported without abandoning the rest, and an empty rename map. Locally go test ./deb/ passes with RenameFiles at 100%, golangci-lint v2.12.2 reports 0 issues, and gofmt, go vet and go build ./... are clean.

RenameFiles walks renameMap one entry at a time. Every rename is a round trip,
and on an S3 backend RenameFile is a CopyObject followed by a DeleteObject, so
a publish that rewrites a dozen indexes spends most of its wall clock waiting
on them in sequence.

Bounded at four in flight, matching the default used for concurrent package
uploads. The first error is returned rather than the last, and the remaining
renames still run, so a failure does not leave part of the set staged behind a
half-finished switch.

Measured against MinIO behind a proxy adding 50ms each way, publishing a
repository whose Contents is dominated by one package carrying 19,500 files:
publish update drops from 2703ms to 1624ms median, n=5 either side, a 40% cut.
The published object set and its sizes are identical before and after.

RenameFiles had no test coverage. It has four cases now: every entry renamed,
the concurrency limit respected, a failure reported without abandoning the
rest, and an empty rename map.
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.10%. Comparing base (f59b0d2) to head (95da3c3).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1633      +/-   ##
==========================================
- Coverage   77.37%   77.10%   -0.28%     
==========================================
  Files         165      165              
  Lines       15747    15764      +17     
==========================================
- Hits        12185    12155      -30     
- Misses       2356     2408      +52     
+ Partials     1206     1201       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3: publish update spends most of its time renaming indexes

1 participant