AutoPrune deletions are invisible: no oplog record, no config knob, and same-second reaping of newborn insights

[tars23364-bit]

Summary

store.(*DB).AutoPrune (write-time retention GC) soft-deletes non-immune insights with no oplog record, no CLI-visible warning on success, and no configuration knob. Combined with same-second reaping of newly-written insights, this makes silent data loss both possible and undiagnosable from mnemon's own audit surfaces.

Observed on v0.1.14 (brew, macOS); release notes for 0.1.15–0.1.17 suggest the behavior is unchanged.

What we observed

On a production store (~2,300 live insights, heavy daily write traffic):

• 25–70 unlogged soft-deletions per day over three weeks (~430 in 11 days). Detected only by joining insights.deleted_at against the oplog: every diff-replace and forget deletion has an oplog row, but AutoPrune deletions have none — deleted_at gets set with no corresponding operation record.
• Same-second create→reap: a batch writer inserting importance-3 insights had earlier items of its own batch soft-deleted by the write of later items, within the same second. Newborn insights with importance < 4 AND access_count < 3 are immediately eligible.
• remember --no-diff does not affect it (expected — it's a separate path from duplicate/conflict detection), which surprised us when hardening against diff-replace: a "fully additive" write can still delete an unrelated insight as a side effect.
• The only success-path trace is the auto_pruned counter in the write result; the identity of what was pruned is not reported anywhere.

Practical consequence: an agent audit process restored a soft-deleted insight, and a later unrelated remember re-reaped it hours later — invisible to oplog forensics. We now compensate by writing durable content at importance ≥ 4 and pairing restores with gc --keep, but that's convention, not a guarantee.

Requests (in priority order)

1. Oplog entries for auto-prune — one row per pruned insight (operation: auto-prune, insight_id, and ideally the triggering write's id). This is the big one: reversible soft-deletes are a great design, but only if the deletion is discoverable.
2. Min-age grace period — exempt insights younger than N hours/days from AutoPrune so a write burst can't eat its own batch before anything has had a chance to access them.
3. Config knob — env var or per-store setting to tune/disable AutoPrune (threshold, or off), for deployments that prefer explicit, proposal-gated retirement.
4. Report pruned ids in the remember result payload alongside the existing auto_pruned count.

Offer

Happy to contribute a PR for (1) and (2) if the direction sounds right — same author as PR #50 (index-aware fourth-path entity extraction). The oplog change looks small from the outside (the prune site already has the ids in hand); the grace period is one predicate in GetRetentionCandidates/prune-candidate selection.

Environment

• mnemon 0.1.14 (mnemon-dev/tap via Homebrew), macOS (Apple Silicon)
• Store: single SQLite store, ~2,300 live insights, ~82k edges, WAL mode
• Detection query, for reference:

SELECT substr(i.deleted_at,1,10) AS day, COUNT(*) AS unlogged
FROM insights i
LEFT JOIN oplog o ON o.insight_id = i.id AND o.operation IN ('diff-replace','forget')
WHERE i.deleted_at IS NOT NULL AND o.insight_id IS NULL
GROUP BY day ORDER BY day;