Repository navigation
Conversation
A turso store whose `-tshm` outlived its `-wal` cannot be opened: either "I/O error: short read on WAL frame at offset N" or "Corrupt database: Invalid page type: 0". The main `.db` stays intact in both cases, so the store is recoverable by discarding the sidecars — but nothing in the plugin does that, so the whole memory subsystem stays down until someone intervenes by hand (issue tickernelz#435). On either signature, verify the main database on a COPY (opening the live file with another engine is itself a corruption source), move `-shm`, `-tshm` and `-wal` into `<storage>/recovery/<db>-<timestamp>/` — moved, never deleted — and retry the open once. Unhealthy copy or failed retry leaves the store untouched and re-raises the original error. One attempt per path per process; no loop. Also warn once per path when a foreign SQLite `-shm` appears beside a Turso `-tshm`: measured, read-only access is harmless, but a foreign writer diverges silently and loses its own next write. Config: `tursoAutoHeal` (default true, only fires when the store is already unopenable) and `tursoMultiprocessWal` (default true; false makes foreign access fail loudly with "database is locked" instead of diverging).
Both knobs change observable behaviour on shared storage: single-owner mode turns silent divergence from a foreign writer into a loud 'database is locked', and auto-heal is what recovers a stale WAL index on open. English README only; translations are maintained separately.
There was a problem hiding this comment.
🟡 Changes recommended
Recovery can exclude committed WAL transactions and race across processes, while filesystem failures can mask the original open error.
3 open issues
What changed in this PR
Adds automatic recovery for stale Turso WAL indexes that prevent the memory subsystem from starting, addressing #435 and complementing checkpoint-on-close work in #440.
Changes:
- Verifies a temporary database copy, quarantines sidecars, and retries failed opens once.
- Adds recovery and multiprocess-WAL configuration options, plus a foreign-SQLite warning.
- Adds injected-driver tests for recovery, failure handling, and configuration.
| File | Description |
|---|---|
tests/turso-wal-index-recovery.test.ts |
Tests recovery paths and configuration switches. |
src/services/turso/wal-index-recovery.ts |
Detects error signatures, checks database copies, and quarantines sidecars. |
src/services/turso/connection-manager.ts |
Integrates recovery, retry limits, and foreign-SQLite warnings. |
src/config.ts |
Defines and documents the new configuration options. |
opencode-mem.example.jsonc |
Documents recovery and single-owner settings. |
Review effort: Balanced
| const quarantineDir = quarantineDirFor(dbPath); | ||
| mkdirSync(quarantineDir, { recursive: true }); | ||
| const moved: string[] = []; | ||
| for (const suffix of WAL_SIDECAR_SUFFIXES) { |
There was a problem hiding this comment.
Fixed in 1049517: recovery is now serialised across processes with <storagePath>/recovery/<dbstem>.lock, created O_EXCL. If the lock cannot be taken, nothing is moved and the original error is rethrown — a live owner is never raced. Under the lock the open is re-attempted before any file moves, so a peer that healed first leaves nothing to do; lock release runs in finally and cannot throw past the original error. A lock older than five minutes is treated as a crashed healer and taken over (logged). Test: a second healer finds the lock held, moves nothing, and surfaces the original error.
| for (const suffix of WAL_SIDECAR_SUFFIXES) { | ||
| const sidecar = `${dbPath}${suffix}`; | ||
| if (!existsSync(sidecar)) continue; | ||
| renameSync(sidecar, join(quarantineDir, basename(sidecar))); |
There was a problem hiding this comment.
Fixed in 1049517. Recovery no longer touches a non-empty -wal: only the index sidecars -shm/-tshm are quarantined, and -wal is removed only when it is absent or 0 bytes. Committed-but-uncheckpointed rows therefore survive.
The rebuild assumption is measured, not assumed: discarding -tshm and keeping a 16,512-byte WAL with committed rows, the engine reopened and returned both rows. The regression test seeds a real DB, leaves rows only in the WAL, discards the index, reads them back, and asserts a fresh -tshm exists beside the surviving WAL. Documented limit: the evidence covers a fully written final frame; a torn WAL is the 'retry fails, original error surfaces, WAL untouched' path.
| } | ||
| this.healedPaths.add(dbPath); | ||
|
|
||
| const outcome = await quarantineWalSidecars(dbPath, this.connectFactory, opts); |
There was a problem hiding this comment.
Fixed in 1049517: lock acquisition, verification, copy, quarantine and cleanup are all wrapped. Failures are logged with the database path, the quarantine directory and what was already moved, and the original open error is rethrown — a filesystem throw can no longer replace the corruption error or skip its diagnostics. Lock release is in finally with its own guard so it cannot throw past the original error either.
Review findings on tickernelz#441. Data safety: recovery now quarantines only the index sidecars (-shm, -tshm). A -wal is removed solely when it is absent or 0 bytes; a non-empty WAL is left in place, because committed transactions can live there while the main .db still passes integrity_check. Deleting it would silently roll the store back to an older snapshot. Measured on this engine: after discarding -tshm and keeping a 16,512-byte WAL, reopen returns both committed rows — the index is rebuilt from the surviving WAL. A torn WAL is the documented "retry fails, original error surfaces, WAL untouched" case. Cross-process: healing is serialised with <storagePath>/recovery/<dbstem>.lock, created O_EXCL. A second process that cannot take the lock leaves the store alone instead of moving files a live owner is writing to; a lock older than five minutes is treated as a crashed healer and taken over. Under the lock the open is re-attempted before anything moves, so a peer that healed first leaves nothing to do. Error reporting: lock acquisition, verification, copy, quarantine and cleanup failures are caught and logged with the database path, the quarantine state and the original error, then the original open error is rethrown — a filesystem problem can no longer mask the corruption or skip its diagnostics. Lock release runs in finally and cannot throw past the original error.
Summary
Makes a turso store recoverable when its WAL index has outlived its WAL, instead of leaving the whole memory subsystem offline until someone fixes it by hand. Fixes the failure mode in #435; complements #440.
Test plan
recovery/is never pruned, and a live foreign reader during the move is a judgement call worth a rulingChecklist
main(do not open PRs from commits already onmain)bun testpasses (or N/A) — 868 pass / 4 skip / 2 pre-existingconfigfailuresbun run typecheckpasses (or N/A)bun run checkpasses (format + lint + typecheck; or N/A)CONFIG_TEMPLATE,opencode-mem.example.jsoncand an English README paragraph for both knobs (translations handled separately)Makes a turso store recoverable when its WAL index has outlived its WAL, instead of leaving the whole memory subsystem offline until someone fixes it by hand.
Fixes the failure mode reported in #435:
<db>-tshmstill claims frames the<db>-walno longer has, so every open fails with eitherand
ensureTursoReady()gates memory tools, chat capture and the web server behind it. The main.dbis intact in both cases — measured, not assumed — so the store is recoverable by discarding the sidecars.Behaviour
On either signature, once per database path per process:
.dbonly into a temp dir and runPRAGMA integrity_checkon the copy. The live file is never opened by another engine — that is itself a documented way to produce theInvalid page typeclass.renameSyncthe index sidecars-shm/-tshminto<storagePath>/recovery/<dbstem>-<timestamp>/. A-walis removed only when it is absent or 0 bytes — a non-empty WAL can hold committed transactions and is left in place for the engine to rebuild its index from. Moved, never deleted.<storagePath>/recovery/<dbstem>.lock(O_EXCL, stale after 5 min). If the lock cannot be taken, nothing moves and the original error surfaces. Under the lock the open is re-attempted before any file is touched, so a peer that healed first leaves nothing to do.Plus a one-time warn per path when
-shmappears beside-tshm, i.e. a foreign SQLite process has visited the store.-shmis never deleted — it may belong to a live reader.Why the warn says "writer" and not "anyone"
Measured against a live
multiprocess_walstore (experimental: ["encryption", "multiprocess_wal"]— without those flags stock sqlite cannot even open it):+ -shm-tshmonly,-wal=0So read-only inspection is harmless and warning users off it would be wrong; writers are the hazard. Measurement and harness are in #435.
Config
tursoAutoHeal(defaulttrue) — the guard only runs when the store is already unopenable, and it moves rather than deletes.tursoMultiprocessWal(defaulttrue, no behaviour change) —falseopens without themultiprocess_walexperimental flag, so a foreign process gets a louddatabase is lockedinstead of diverging silently. Reasonable choice for single-user stores; needs a ruling on whether that should become the default.Documented in
CONFIG_TEMPLATEandopencode-mem.example.jsonc. I deliberately left the READMEs alone to avoid i18n churn — happy to add a sentence if you want it.Verification
bun run format:check,bun run lint,bun run typecheck— clean.bun test tests/turso-wal-index-recovery.test.ts tests/turso-connection-manager.test.ts— 12 pass.bun testfull suite — 868 pass / 4 skip / 2 fail; both failures are the pre-existingconfig > CONFIG defaultsones, reproduced on a pristinemainin this worktree. (13 other failures in a fresh checkout are only the missingdist/build;bunx tscrestores them.)ConnectFactory; no native-binary gambling.Risks / open questions
recovery/is never pruned, and now also holds one small<dbstem>.lockper healed database. Each heal costs one directory of sidecar sizes. A retention sweep is a follow-up, deliberately out of scope here.-tshmfrom it. Measured here (16,512-byte WAL, committed rows recovered after discarding the index) and covered by a regression test, but it is engine behaviour rather than a documented API contract. A torn WAL is the "retry fails, original error surfaces, WAL untouched" case — no silent rollback.-shmbeing moved, POSIX keeps the inode alive but its path changes. This only triggers on an already-unopenable store, which is why I took the trade — worth a maintainer ruling.integrity_checkof that file on a cold error path, once per path per process. Gating it on sidecar mtimes would weaken the guarantee, so it is unconditional..db-only by design: if the main file were genuinely mid-checkpoint corrupt, the check fails and the store is left alone.Relates to #435. Complements #440 (checkpoint before close): prevention on the way out, recovery on the way in.