Skip to content

fix(store): acquire reindex writer gate unbounded to stop warmup crash#328

Open
zzet wants to merge 1 commit into
mainfrom
fix/reindex-writer-gate-unbounded-lock
Open

fix(store): acquire reindex writer gate unbounded to stop warmup crash#328
zzet wants to merge 1 commit into
mainfrom
fix/reindex-writer-gate-unbounded-lock

Conversation

@zzet

@zzet zzet commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Problem

The daemon panics during warmup:

panic: store_sqlite: store_sqlite: reindex writer gate: context deadline exceeded
  ...store_sqlite.(*Store).ReindexEdges
  ...resolver.(*Resolver).persistAttributionReindexes
  ...resolver.(*Resolver).bindDataflowCalleeRefsForFile
  ...resolver.(*Resolver).ResolveAll
  ...indexer.(*MultiIndexer).RunPreEnrichResolve
  ...main.warmupDaemonState

Root cause

reindexEdgesSetOriented acquired writeMu with a bounded LockContext(sqliteBusyRetryWindow) — 15s by default. A resolver reindex is a mandatory graph.Store mutation with no error channel (the interface mirrors the in-memory store's "everything succeeds" contract), so ReindexEdges routes any failure through panicOnFatal, which only swallows ErrNoRows / ErrConnDone / store-closed — not context.DeadlineExceeded.

When a sibling writer (a warmup WAL checkpoint, or a mutation from the deferred-enrichment pool started ahead of the resolve) held writeMu past the 15s window, the gate timed out and the recoverable lock-wait was escalated into a whole-daemon panic.

reindexEdgesSetOriented was the only mandatory-write path using a bounded gate; the siblings ReindexEdge, setEdgeProvenanceBatchSetOriented, and RemoveEdge all use a plain unbounded writeMu.Lock().

Fix

Acquire the gate with a plain unbounded Lock(), matching the sibling mandatory-write paths. A mandatory graph mutation must wait for the writer gate, not abandon the batch and crash.

Waiting cannot hang:

  • beginWriteContext never re-locks writeMu;
  • the only unbounded writeMu holder — Compact's VACUUM — runs sequentially before the warmup resolve in the same goroutine, never concurrent;
  • lock order is always resolver mutex → writeMu, no inversion (the siblings already prove unbounded Lock is safe from these resolver contexts).

Per-transaction SQLITE_BUSY contention remains bounded by withSQLiteBusyRetry. The checkpoint's bounded LockContext is left untouched — a checkpoint is retryable maintenance whose caller treats failure as skip-and-continue.

Verification

  • go build ./... and go vet ./internal/graph/store_sqlite/ clean.
  • store_sqlite busy / reindex / checkpoint / provenance tests pass with -race and without: TestCheckpointSerializesBehindWriterGate, TestReindexEdgesRetriesWholeTransactionAfterBusy, TestReindexEdgesPersistentBusySurfacesAndRollsBack, TestLongWALReaderDoesNotBlockReindexWriter. No test asserted the gate-timeout path, so behavior is unchanged for covered cases.

reindexEdgesSetOriented acquired writeMu with a bounded LockContext
(sqliteBusyRetryWindow, 15s default). A resolver reindex is a mandatory
graph.Store mutation with no error channel, so ReindexEdges routes any
failure through panicOnFatal, which does not swallow
context.DeadlineExceeded. When a sibling writer (a warmup checkpoint or
a deferred-enrich mutation) held writeMu past the window, the gate timed
out and crashed the whole daemon over a recoverable lock-wait.

Acquire the gate with a plain unbounded Lock, matching the sibling
mandatory-write paths (ReindexEdge, setEdgeProvenanceBatchSetOriented,
RemoveEdge). Waiting cannot hang: beginWriteContext never re-locks
writeMu, Compact's unbounded VACUUM runs before the warmup resolve in
the same goroutine, and the lock order is always resolver mutex then
writeMu. Per-transaction SQLITE_BUSY contention remains bounded by
withSQLiteBusyRetry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant