Honor keep-forever retention in the SQLite job cleaner - #1389
Merged
Merged
Conversation
The job cleaner signals a retention period of -1 ("keep forever") for a
finalized state by passing `CancelledDoDelete`, `CompletedDoDelete`, or
`DiscardedDoDelete` as false to `JobDeleteBefore`, along with a horizon
of roughly now. The Postgres query checks these flags, but the SQLite
query ignores them and filters only on the horizons. On SQLite, a
config like `DiscardedJobRetentionPeriod: -1` with a finite completed
retention therefore deletes every discarded job on the next cleaner
run.
Add the `*_do_delete` flags to the SQLite `JobDeleteBefore` query so
each state's clause is skipped when its deletion is disabled, matching
the Postgres query, and pass them through from the SQLite driver.
bgentry
force-pushed
the
bg/sqlite-keep-forever-retention
branch
from
September 25, 2026 17:03
307099a to
24be9bc
Compare
brandur
marked this pull request as ready for review
September 25, 2026 20:53
brandur
approved these changes
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On SQLite, setting a finalized state's retention period to -1 to keep its jobs forever doesn't work if any other state's retention is finite. Instead of keeping those jobs, the job cleaner deletes all of them on its next run.
For example, a client using a SQLite driver configured with
DiscardedJobRetentionPeriod: -1and the defaultCompletedJobRetentionPeriodof 24 hours should keep discarded jobs indefinitely while cleaning up completed jobs after a day. Instead, every discarded job disappears the next time the cleaner runs, regardless of how recently it was discarded. The same applies toCancelledJobRetentionPeriodandCompletedJobRetentionPeriod. Postgres drivers behave correctly. When all three retention periods are -1, the cleaner skips deletion entirely, so that configuration isn't affected either.The cleaner handles -1 by passing a per-state "do delete" flag set to false for that state, while its horizon is computed as roughly the current time. The Postgres
JobDeleteBeforequery checks those flags, but the SQLite query ignores them and filters on the horizons alone, so a -1 retention acts like a retention of zero.This change adds the flags to the SQLite query so each state's clause is skipped when its deletion is disabled, matching the Postgres query, and passes them through from the SQLite driver. A new case in the shared driver test suite runs against every driver and checks that jobs in a state with deletion disabled are kept while other states are still cleaned.
No migration or configuration change is needed. After upgrading, SQLite users who rely on -1 retention for a state stop losing those jobs, although jobs already deleted can't be recovered.