Skip to content

Reduce missing/corrupt SQLite archive-index errors caused by overlapping cache cleanup and repair - #3519

Open
syphar wants to merge 1 commit into
rust-lang:mainfrom
syphar:archive-index-errors
Open

Reduce missing/corrupt SQLite archive-index errors caused by overlapping cache cleanup and repair#3519
syphar wants to merge 1 commit into
rust-lang:mainfrom
syphar:archive-index-errors

Conversation

@syphar

@syphar syphar commented Sep 12, 2026

Copy link
Copy Markdown
Member

This is motivated by sentry errors we see regularly (error returned from database: (code: 14) unable to open database file), and similar errors leading to flaky tests. Also keep in mind: the storage engine only reports to sentry when the problem still happens after 5 retries.

The solutions here are more or less guesses about where these race conditions happen. Also, this PR is more about quickfixes, hoping these reduce the issue.

I'm thinking about a redesign separately, the whole thing feels too complex for me, of course it's also a central part of our webserver.

details

  • Await eviction cleanup instead of spawning detached deletion tasks. Explicit invalidations leave file cleanup to the caller, avoiding a second deletion that could remove a replacement.
  • Invalidate stale manager entries before repair, then reopen the file inside the coalesced initializer. This allows immediate repair while reusing files another request has already repaired,
    instead of acting on the original force_redownload decision.
  • Retry the first failed operation without purging. This gives a concurrently published replacement a chance to succeed; subsequent failures still trigger purge within the existing five-attempt
    limit.
  • Delete the index, WAL, and SHM files concurrently, preserving the existing error handling. This might slow down get/set acccess to the moka cache, when it does a maintenance run. It shouldn't happen too often, we do a background maintenance run every 30s.

Successful fast-path lookups remain without a path lock. These changes reduce avoidable deletion and repair churn but do not eliminate all filesystem races.

more tests

  • more unit-tests for old/new edge cases
  • some optional stress-tests, seeing that they failed on main and are fixed here.

@syphar syphar self-assigned this Sep 12, 2026
@github-actions github-actions Bot added the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label Sep 12, 2026
@syphar syphar changed the title Reduce archive-index cache errors while keeping the fast path without a path lock. Reduce archive-index cache errors while keeping the fast path without a path lock Sep 12, 2026
@syphar syphar changed the title Reduce archive-index cache errors while keeping the fast path without a path lock Reduce missing/corrupt SQLite archive-index errors caused by overlapping cache cleanup and repair. Sep 12, 2026
@syphar
syphar force-pushed the archive-index-errors branch from 764aee9 to 0c52b65 Compare September 12, 2026 22:32
@syphar
syphar force-pushed the archive-index-errors branch from 5fdc8f0 to 64d2d7d Compare September 12, 2026 23:11
@syphar syphar changed the title Reduce missing/corrupt SQLite archive-index errors caused by overlapping cache cleanup and repair. Reduce missing/corrupt SQLite archive-index errors caused by overlapping cache cleanup and repair Sep 13, 2026
@syphar
syphar force-pushed the archive-index-errors branch from aceaa93 to e406617 Compare September 13, 2026 01:02
@syphar
syphar marked this pull request as ready for review September 13, 2026 01:03
@syphar
syphar requested a review from a team as a code owner September 13, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant