Skip to content

feat(media): add phased S3 layout migration - #4533

Draft
bradseiler wants to merge 8 commits into
mainfrom
seiler/media-layout-migration
Draft

feat(media): add phased S3 layout migration#4533
bradseiler wants to merge 8 commits into
mainfrom
seiler/media-layout-migration

Conversation

@bradseiler

@bradseiler bradseiler commented Aug 3, 2026

Copy link
Copy Markdown

Why

Media payload keys are currently flat, so high-volume traffic lacks hash-leading S3 prefix entropy. Existing deployments also need a safe, explicit migration that preserves old reads and rollback behavior.

What

  • moves media payloads to media/<sha[0:2]>/<sha[2:4]>/<community>/<filename> while leaving Git/CAS namespaces unchanged
  • adds BUZZ_MEDIA_MIGRATION_PHASE with three complete read/write policies:
    • dual-read-legacy-write (default and upgrade-safe)
    • dual-read-dual-write
    • sharded-only
  • adds new-first/legacy-fallback read telemetry, strict bucket classification, logical billing deduplication, duplicate-layout gauges, and authoritative payload keys in upload records
  • includes buzz-media-layout-backfill and buzz-media-layout-delete-legacy in both relay image targets, with bounded paging, checkpoints, request-rate limiting, idempotency, destination verification, dry-run, and destructive confirmation
  • documents new-install and upgrade flows and provides Kubernetes Job examples

Validation

  • cargo test -p buzz-media --all-targets: 117 passed; live-MinIO test ignored
  • cargo test -p buzz-relay --lib storage_sweep: 15 passed
  • cargo clippy -p buzz-media -p buzz-relay --all-targets -- -D warnings: clean
  • cargo fmt --all -- --check and git diff --check: clean
  • both maintenance binary --help entry points built and ran locally
  • pre-push hooks: Rust suites and desktop Tauri checks passed
  • Docker image build could not run locally because the Docker daemon is unavailable; the Dockerfile explicitly builds, strips, and copies both binaries in debug and normal runtime targets

Generated with Builderlab Expert

npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch added 6 commits August 3, 2026 10:13
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Classify sharded payload keys in storage sweeps, preserve physical totals, and deduplicate logical legacy/sharded copies. Export read resolution, fallback, and duplicate-layout metrics.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Keep self-hosted upgrades on legacy writes by default, expose the write-layout gate through Helm and Compose, and document the explicit legacy-to-dual-to-sharded migration sequence.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Consolidate read and write policy behind one upgrade-safe phase and ship guarded maintenance binaries for backfill and legacy cleanup.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch added 2 commits August 3, 2026 10:27
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant