`SupermemoryProvider.awaitIndexing` polls forever with no timeout and treats API errors as "still pending"

[rajarshidattapy]

Description

// src/providers/supermemory/index.ts:81-113
while (pending.size > 0) {
  const results = await Promise.allSettled(pendingArray.map(async (docId) => { ... }))
  for (const res of results) {
    if (res.status === "fulfilled") { ...possibly pending.delete(docId)... }
    // rejected results: no branch at all
  }
  if (pending.size > 0) { await sleep(backoffMs); backoffMs = Math.min(backoffMs * 1.2, 5000) }
}

There is no iteration cap, no wall-clock deadline, and no handling of the rejected case. A document is only removed from pending when a successful documents.get reports a terminal status.

Two ways this never terminates:

1. A document sticks in a non-terminal status (queued/processing) — because it was dropped server-side, hit a quota, or the container was deleted. The loop polls it every 5s forever.
2. documents.get or memories.get keeps throwing — auth expiry, 429, 5xx, network partition. Every promise rejects, the for loop matches nothing, pending never shrinks, and the loop hammers the API in a 5-second cycle indefinitely. Errors are swallowed by Promise.allSettled and never surface to the user or the checkpoint.

Note the backoff also does not help case 2, since it caps at 5s and the request volume is pending.size calls per cycle — for a question with hundreds of documents, that is a sustained retry storm against a service that is already failing.

Impact

The indexing phase hangs with a progress bar frozen at, say, 847/900 episodes, no error, no log line, and no way to distinguish "slow provider" from "permanently stuck". ConcurrentExecutor has no per-task timeout, so the run never fails and never completes. Stopping via the UI does not help either — shouldStop is only checked between batches (concurrent.ts:54), never inside a task, so a hung awaitIndexing ignores the stop request entirely.

Suggested fix

• Add a deadline (e.g. INDEXING_TIMEOUT_MS) and throw a descriptive error when exceeded, listing the still-pending document IDs so the failure is actionable.
• Handle res.status === "rejected": count consecutive failures per document and give up after N, recording them in failedIds rather than looping.
• Consider checking shouldStop(runId) inside long-running provider waits, or enforcing a timeout at the ConcurrentExecutor level so no single task can wedge a run (see [[issue_9]] for the related "errors vanish silently" theme).