fix(ingest): persist document ids so a mid-question failure doesn't lose them - #88
Open
Agnik47 wants to merge 1 commit into
Open
fix(ingest): persist document ids so a mid-question failure doesn't lose them#88Agnik47 wants to merge 1 commit into
Agnik47 wants to merge 1 commit into
Conversation
completedSessions was checkpointed inside the ingest loop while the accumulated ingestResult was only written after the loop finished, so a session that threw partway through a question discarded the document ids of every session before it. Resume skipped those sessions, so their ids were never regenerated and indexing waited on a short list, letting search run before those sessions were queryable. Seed combinedResult from the persisted result on entry and write it alongside completedSessions each iteration, so the two agree at every point in the loop. Merging on entry replaces the end-of-loop merge, which would otherwise double-count on resume.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #68.
Problem
runIngestPhaseingests a question's sessions one at a time.completedSessionsis checkpointed inside the loop, but the accumulatedingestResultis only written after the loop finishes:If session 7 of 10 throws, the catch block writes
status: "failed"and the document ids for sessions 1–6 are discarded with the in-memory object. On resume those sessions are skipped — they are incompletedSessions— so their ids are never regenerated. The merge againstexistingResultcannot recover them: the failed attempt never wrote one.Impact
ingestResult.documentIdsis what indexing waits on before search is allowed to run:So after a mid-question ingest failure the run searches before the earlier sessions have finished indexing. For providers with asynchronous indexing (Supermemory, Zep, Mem0) that is a race producing quietly degraded retrieval — the memories exist but were not queryable when search ran — which reads as a provider quality problem rather than a harness bug.
getSummary'sindexingEpisodesunder-reports by the same amount.Fix
Seed
combinedResultfrom the persisted result on entry instead of merging it in at the end, and write it alongsidecompletedSessionson every iteration, so the two agree at every point in the loop.Merging on entry is what keeps this from double-counting: a resumed attempt starts from the persisted ids and appends only the sessions it actually ingests, so the end-of-loop merge is no longer needed and is removed. No extra checkpoint writes are added — the per-session
updatePhasecall already existed, it just carries the ids now.Tests
src/orchestrator/phases/ingest.test.ts(new). Every test reloads the checkpoint from disk, so it exercises the resume path rather than in-memory state.documentIdsabsent)taskIdsstays absent for providers that don't report themOnly
ingest.tsand the new test file are touched. The new file is Prettier-clean;ingest.tswas already non-Prettier onmainand is left as-is to keep the diff reviewable.