Fix web UI slowness: delete drain starved the app, janitor race stranded tombstones

[HelloThisWorld]

What was wrong

Loading a project and deleting a project from the web UI were both very slow, and the delete modal could sit on "Deleting…" indefinitely.

All three symptoms trace back to the delete path — and none to the code they point at. GET /projects/{id} measures 49–273 ms on an idle process, so the load path was never the problem. The app was simply GIL-starved from boot.

Bug 1 — the batched drain starved the whole interpreter

_drop_chroma_collection yielded a fixed time.sleep(0.01) between batches that each held the GIL ~500 ms (chromadb 1.x rust calls hold it for the entire call).

Batching bounds a single stall; only the gap controls the duty cycle, and the gap has to scale with the hold. With a 10 ms gap against a 500 ms hold, every other Python thread — all HTTP handlers, the SSE streams, the job worker — got ~5% of the interpreter for the whole drain.

Bug 2 — the startup janitor raced the cleanup that owned the collection

_warm_vectorstore passed db.list_projects(), which excludes 'deleting'. So a tombstoned project's live collections were classified as orphans and drained a second time, concurrently with the _cleanup_deleted thread _recover_on_restart had spawned one line earlier in lifespan.

Whichever thread reached delete_collection first made the other's next get() raise. That was reported as "interrupted", and _cleanup_deleted returned before the rmtree and before db.delete_project. The tombstone survived and was re-drained on every boot — so the app was permanently in delete-recovery, which is why loading a project was slow even when nothing was being deleted.

The sibling function four lines up (_sweep_orphan_project_dirs) already had this right, with the correct rationale in its docstring.

Changes

file	change
openmind/vectorstore.py	Yield proportionally to each batch's measured hold. DROP_BATCH 500→200 to bound the worst-case stall. Refuse a second concurrent drain of one collection. Treat "collection already gone" as success, not interruption. Thread cancel through drop_orphan_collections.
openmind/main.py	Include 'deleting' projects in the janitor's known set. Announce pending reclaims at startup instead of failing silently.
openmind/db.py, services/workspace_service.py, ports/workspace_repository.py	count_file_index() instead of materializing every row to take len() (36 ms → <1 ms). Count collections without get_or_create — so viewing a project stops creating its cases collection as a side effect of a read.
openmind/jobs.py	Terminate's drain takes a cancel callback, so Ctrl+C during a large Terminate no longer waits out the whole drain.

Measured

On a copy of a real 2.1 GB store (79,585-chunk collection), with a 10 ms-tick thread standing in for the FastAPI threadpool:

	before	after
p95 request stall during delete	823 ms	170 ms
worst-case stall	1480 ms	359 ms
app availability	5%	38%
drain wall-clock	131 s	157 s

The full sweep behind the DROP_BATCH / DROP_YIELD_RATIO choice is in a comment table above the constants, so the next person doesn't re-derive it.

verify_delete_responsive also improves on its own existing assertion: max /projects latency during a drop went 0.21 s → 0.01 s.

Testing

• Full suite green — 25 verification suites.
• New tests/verify_delete_race.py (26 checks) pins all three defences.

On the new test: my first version passed against the buggy code, because it relied on hitting the race by chance. It was rewritten to force the interleaving deterministically, then validated by reintroducing each of the three fixes' bugs separately and confirming the matching check fails:

bug reintroduced	result
main.py known set reverted	22/26 — "the real startup janitor does not drop a 'deleting' project's collection" FAILS
in-flight guard removed	24/26 — "the second concurrent drain is refused" FAILS (took 2.313 s doing competing work)
"already gone" = interrupted	25/26 — "a drain whose collection vanished mid-way reports SUCCESS" FAILS

It calls the real main._warm_vectorstore() rather than re-deriving its known set, so it actually guards the fix rather than a copy of it.

Reviewer notes

Upgrade impact: this makes pending deletes complete. Anyone carrying a stranded 'deleting' tombstone will have that project's storage actually reclaimed on the next server start — correct, but irreversible, so it's worth knowing before upgrading. Startup now logs pending reclaims by id.

Rejected alternatives, both deliberate:

• One-shot delete_collection. It is less total work, but measured at 51.7 s of solid GIL (2 scheduler ticks in 51.7 s), and chroma's sqlite runs in rollback-journal mode, so a kill mid-way rolls the whole thing back — precisely the failure the batched drain was introduced to remove (see 0721234).
• Reclaim in a subprocess. That opens a second Chroma client on one data dir, which is the documented GIL⊗sqlite-lock deadlock this codebase already warns about in sweep_orphan_segment_dirs.

Known follow-up: this is a mitigation — a 79k-chunk delete is still ~157 s of background churn. The measured real fix is a per-project Chroma directory, making delete an rmtree: 0.03 s with zero GIL impact. I verified several per-project PersistentClients coexist fine in one process, and that the Windows file-lock teardown needs SharedSystemClient.clear_system_cache() + gc.collect() before the rmtree. Left out of this PR because it needs a data migration and touches the ingest/search paths — worth doing as a separate deliberate change.