Enqueue deleted directory objects for reclamation

[kylebernhardy]

Summary

• enqueue a versioned whole-object reclaim record in the same atomic WAL batch that removes a Tantivy file binding
• replace the directory-wide mutation mutex with per-path writer retirement, per-shard enqueue locks, and amortized object-ID reservation
• keep chunk/publication writes lock-free while preventing deletion from capturing an extent before an in-flight writer finishes
• add a dependency-free KvDirectory benchmark for buffered writes, flushes, chunk publication, deletion, and distinct-file concurrency
• document the slice boundary: physical draining, reader pins, tail-only reclamation, and Harper production enablement remain follow-up work

Design

Each binding already carries the highest chunk and tail revision that may exist for its immutable object. Deletion now retires the matching writer, waits for any admitted storage operation to reach a definitive result, rereads the binding when a writer was active, and atomically writes:

1. a sequence-addressed reclaim entry containing the object ID and both high-water marks;
2. the shard's next sequence;
3. the binding deletion; and
4. the corresponding atomic-file deletion.

The queue uses 64 fixed shards and only the existing point-read and atomic-batch KvStore operations. Object IDs are durably reserved in strides of 1,024, removing a counter read and mutation from almost every open_write(). The common closed-writer delete is four point reads and one four-mutation batch; an active writer adds one binding reread.

The writer fence uses sequentially consistent retirement and in-flight counters. Normal claim/release stays mutex-free; only the final release after retirement enters the condition-variable notification path. A live writer retains the shared directory state so a reconstructed wrapper for the same canonical store identity cannot lose its fence.

The canonical KvStoreIdentity contract is load-bearing: every wrapper for one live store incarnation and derived column family must converge on one DirectoryState. This library cannot replace that Harper lifecycle guarantee with compare-and-set because KvStore intentionally exposes no CAS primitive. Harper production use remains disabled until the derived-index integration proves canonical identity and exclusive generation ownership across its worker topology.

The release-mode kv_directory harness isolates directory coordination with the deterministic in-memory Phase 0 store. It emits versioned JSON with percentiles across per-sample mean latency, throughput, Git revision and dirty state, build profile, enabled features, and host metadata. Workloads cover caller-visible buffered writes without crossing a chunk boundary, empty and dirty flushes, 256 KiB chunk publication, closed- and active-writer deletion, and distinct-file concurrency at one, two, four, and eight threads. The in-memory store serializes access, so concurrency results detect coordination regressions rather than predict RocksDB scaling. CI runs a plural-operation correctness smoke without timing thresholds; comparative performance runs use alternating revisions on one fixed quiet host.

Verification

• full npm run check
• 84 Rust tests, including the retirement-admission and lock-free-release coverage
• 35 Node tests plus packed-package verification; three optional rocksdb-js integration tests skipped without their external fixture
• npm run benchmark:kv-directory -- --smoke --revision ce26565
• default 20-sample benchmark run from clean commit ce26565; output validated as JSON
• deterministic publication/delete and chunk-write/delete races with whole-store extent audits
• real two-segment Tantivy merge and garbage collection, followed by durability-barrier crash recovery
• applied-but-reported-failed delete reconciliation, queue collision/corruption handling, reopened queue tails, multi-chunk deletion, and bounded I/O assertions
• local Claude review; valid findings were addressed with batched storage samples, isolated caller writes, an untimed concurrency start gate, explicit sample-mean metric names, reproducibility metadata, CI headroom, and bounded CPU-friendly test waits
• direct Gemini CLI final review: no actionable findings

Part of #11

Comment generated by kAIle (GPT-5)

_{Review-Coverage: authored=unknown; ran=none; rounds=1 @ ce26565}

_{Human-Review-Need: 4 @ ce26565}