Skip to content

fix(backups): dump the database once instead of twice - #4916

Open
joaquinleondev wants to merge 1 commit into
Dokploy:canaryfrom
joaquinleondev:fix/backup-single-dump
Open

fix(backups): dump the database once instead of twice#4916
joaquinleondev wants to merge 1 commit into
Dokploy:canaryfrom
joaquinleondev:fix/backup-single-dump

Conversation

@joaquinleondev

Copy link
Copy Markdown

Closes #4915

Problem

getBackupCommand interpolates ${backupCommand} twice: once with its stdout thrown away, purely to check that the dump works, and once more piped into rclone. Every database backup — postgres, mysql, mariadb, mongo, libsql and compose — therefore reads and compresses the entire database two times.

On a 9.4 GB Postgres this is what the log looks like today:

[Mon Jul 27 00:00:04 UTC 2026] Executing backup command...
[Mon Jul 27 00:09:17 UTC 2026] ✅ backup completed successfully      ← 9m13s, discarded
[Mon Jul 27 00:09:17 UTC 2026] Starting upload to S3...
[Mon Jul 27 00:16:35 UTC 2026] ✅ Upload to S3 completed successfully ← 7m18s, dumped again

16m31s instead of ~8m. It also doubles the CPU burned by single-threaded gzip/pg_dump -Fc on the same box that serves traffic, and — the part that actually hurts — it doubles the window in which pg_dump holds ACCESS SHARE on every table and keeps a snapshot open, which is the window where a concurrent migration's ALTER TABLE blocks and autovacuum is held back.

What changed

The dump is piped straight into rclone, so the database is read once and the stream is uploaded as it is produced.

Three details worth reviewing:

  1. Error attribution is preserved. Each side of the pipe writes to its own stderr file, and PIPESTATUS tells which side failed, so the log keeps distinguishing ❌ Error: Backup failed from ❌ Error: Upload to S3 failed. (PIPESTATUS only survives if the pipeline is not wrapped in $(...) — hence the temp files rather than a command substitution.) As a side effect the dump's stderr now actually reaches the log on the upload path; previously 2>&1 >/dev/null bound to the rclone side only.
  2. rclone is checked first. If the upload dies early, the dump is killed by SIGPIPE, so a non-zero dump status is a consequence, not the cause.
  3. Partial objects are cleaned up. With streaming, a dump that dies halfway leaves a truncated object in the bucket, so each runner now passes an rclone delete command that removes it when the dump fails. (Worth noting the discarded first run never really prevented this: the second dump could fail just as well and leave the same truncated file.)

The script still relies on bash, exactly as before (set -o pipefail is already bash-only).

One behavioural change to be aware of: the Starting upload to S3... log line is gone, since there is no longer a separate upload phase — the two lines are now emitted together at the end. I checked that nothing in the codebase parses those strings.

Testing

Added apps/dokploy/__test__/backups/backup-command.test.ts, which runs the generated script for real with docker/rclone stubs on PATH (same approach as the existing db-backup-restore-injection.test.ts) and asserts:

  • the dump runs exactly once and the uploaded bytes are the dump's output;
  • a failing dump is reported as Backup failed, still dumping only once;
  • a failing dump removes the partially uploaded object;
  • a failing rclone is reported as Upload to S3 failed, not as a dump failure;
  • a missing container still aborts before dumping.
✓ __test__/backups/backup-command.test.ts (5 tests) 139ms
Test Files  1 passed (1)
     Tests  5 passed (5)

Full suite, before and after, on the same machine (the failures are pre-existing and environmental — they need a database and a swarm manager, neither of which exists in my sandbox):

test files tests
canary 42 failed / 29 passed 4 failed / 267 passed / 1 skipped
this branch 42 failed / 30 passed 4 failed / 272 passed / 1 skipped

biome check clean on the touched files, tsc --noEmit clean in packages/server.

Happy to drop the cleanup-command part into a separate PR if you would rather keep this one to the single-dump change.

getBackupCommand ran the full dump command twice: once with its output
discarded, only to check that it works, and once more piped into rclone.
Every database backup therefore read and compressed the whole database
two times, doubling the wall clock, the CPU cost and — for postgres —
the window in which pg_dump holds ACCESS SHARE on every table.

The dump is now piped straight into rclone. Each side of the pipe keeps
its own stderr file so the log still distinguishes a failed dump from a
failed upload; rclone is checked first because an upload that dies early
kills the dump with SIGPIPE. Since the object is now written as the
stream is produced, runners pass an rclone delete command that removes
the truncated object when the dump fails halfway through.

Closes Dokploy#4915
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Every database backup dumps the database twice

1 participant