Description
src/providers/rag/index.ts:33-68 advances its cursor by breakPoint + 1 - overlap. Nothing guarantees that this is greater than the current start. The only guard is breakPoint <= start, but with CHUNK_OVERLAP = 320 the loop fails to advance for any breakPoint in (start, start + 319]:
if (breakPoint <= start) breakPoint = end // only catches breakPoint <= start
chunks.push(text.slice(start, breakPoint + 1).trim())
start = breakPoint + 1 - overlap // can be <= the previous start
if (start < 0) start = 0
The < start + chunkSize * 0.5 sanity check is applied to the ". " and "\n" candidates but not re-checked after the final lastIndexOf(" ", end) fallback on line 55, which is exactly the branch that produces a small breakPoint.
Reproduction
// 10 chars, one space, then 5000 unbroken chars — a URL, base64 blob, or minified JSON
const text = "x".repeat(10) + " " + "a".repeat(5000)
chunkText(text) // never returns
Verified: 50,000 iterations with start still at 0 and the chunks array at 50,000 entries and growing. Each iteration pushes a fresh ~11-character slice, so the process hangs and leaks until the heap is exhausted.
The trigger is: the last space/newline/period before start + 1600 lies within 320 characters of start. Any ingested session containing a long unbroken token — a URL, a base64 image, a stack trace, a minified payload, a CJK passage with no ASCII spaces (tokenize/lastIndexOf(" ") are whitespace-based) — hits it.
Impact
chunkText runs inside RAGProvider.ingest, which runs inside ConcurrentExecutor with no timeout. A single bad session hangs the ingest phase indefinitely with no error and no progress output, and takes the whole run's memory down with it. Because ingest is checkpointed per session, the hang reproduces on every resume attempt at the same session — the run can never make progress.
Suggested fix
Guarantee forward progress regardless of where the break point lands:
- start = breakPoint + 1 - overlap
- if (start < 0) start = 0
+ const next = breakPoint + 1 - overlap
+ start = Math.max(next, start + 1)
Better still, apply the start + chunkSize * 0.5 floor to the space fallback too, so a degenerate break point falls through to breakPoint = end and produces a genuine full-size chunk.
Worth a unit test with the repro string above — this is precisely the kind of loop that needs one runnable check behind it.
Description
src/providers/rag/index.ts:33-68advances its cursor bybreakPoint + 1 - overlap. Nothing guarantees that this is greater than the currentstart. The only guard isbreakPoint <= start, but withCHUNK_OVERLAP = 320the loop fails to advance for anybreakPointin(start, start + 319]:The
< start + chunkSize * 0.5sanity check is applied to the". "and"\n"candidates but not re-checked after the finallastIndexOf(" ", end)fallback on line 55, which is exactly the branch that produces a smallbreakPoint.Reproduction
Verified: 50,000 iterations with
startstill at0and thechunksarray at 50,000 entries and growing. Each iteration pushes a fresh ~11-character slice, so the process hangs and leaks until the heap is exhausted.The trigger is: the last space/newline/period before
start + 1600lies within 320 characters ofstart. Any ingested session containing a long unbroken token — a URL, a base64 image, a stack trace, a minified payload, a CJK passage with no ASCII spaces (tokenize/lastIndexOf(" ")are whitespace-based) — hits it.Impact
chunkTextruns insideRAGProvider.ingest, which runs insideConcurrentExecutorwith no timeout. A single bad session hangs the ingest phase indefinitely with no error and no progress output, and takes the whole run's memory down with it. Because ingest is checkpointed per session, the hang reproduces on every resume attempt at the same session — the run can never make progress.Suggested fix
Guarantee forward progress regardless of where the break point lands:
Better still, apply the
start + chunkSize * 0.5floor to the space fallback too, so a degenerate break point falls through tobreakPoint = endand produces a genuine full-size chunk.Worth a unit test with the repro string above — this is precisely the kind of loop that needs one runnable check behind it.