feat(kb): wire dead-letter document reconciler Lambda + nightly schedule + IAM (task 16.5) - #1008
Open
DerrickF wants to merge 2 commits into
Open
feat(kb): wire dead-letter document reconciler Lambda + nightly schedule + IAM (task 16.5)#1008DerrickF wants to merge 2 commits into
DerrickF wants to merge 2 commits into
Conversation
The ingestion consumer is the only writer of DOC# status. When its event dead-letters (Lambda async retry capped at 2), a document Bedrock already indexed is left parked non-terminal forever, and the retrieval filter serves only 'complete' -- so its content sits in the KB fully retrievable and invisible to every query. Two such docs occurred in dev; both needed manual repair. Add document_reconciler.py: the missing second writer. Daily, it finds DOC# rows stuck non-terminal (uploading/chunking/embedding) past a 60-minute grace gate, probes Bedrock per document, and drives a stranded-but-retrievable doc to 'complete' (the §5.37 case). It reuses the consumer's own probes -- document_status, the equals-on-document_id retrievability search, its status-set constants, and set_document_terminal -- so §5.37/§5.38/§5.39 live in one place. FAILED -> failed; NOT_FOUND -> re-ingest from S3 (the scheduled form of task 14.4's one-click retry). Modelled on reconciler.py: ships DISARMED (MANAGED_KB_DOC_RECONCILER_ARMED, empty reads as off); per-run action limit applies in both modes so the report is trustworthy; grace gate is a pure function of the row's own updatedAt and fails closed; terminal/deleting rows are never candidates. Guards in tests/lambdas/test_kb_document_reconciler.py (61 tests), mutation-verified. Scheduling + IAM wiring + arming are a deploy-gated follow-up; the flag is exempted in the env-contract test's OPTIONAL_OVERRIDES until that lands.
…hedule and IAM (task 16.5) Adds the fifth kb-migration Lambda (document_reconciler.lambda_handler) to the shared one-image/five-functions construct, so the reconciler built in the backend PR actually runs. - Nightly EventBridge schedule: cron(0 9 * * ? *) (~02:00-03:00 America/Denver), ENABLED regardless of flags, because it ships report-only and report-only is read-only (same inverted convention as the KB reconciler). - IAM: grantDirectIngestion (GetKnowledgeBaseDocuments + IngestKnowledgeBaseDocuments) + grantRetrieval (bedrock:Retrieve) + documents-bucket read + assistants-table RW. Deliberately NOT grantProvisioning or PassRole: it reads KB_Records from DynamoDB, never ListKnowledgeBases, and never creates/deletes a knowledge base -- a strictly narrower footprint than the KB reconciler. - New flag MANAGED_KB_DOC_RECONCILER_ARMED (config.docReconcilerArmed), empty=off, forwarded to every function and threaded through load-env.sh; ships disarmed. - SSM function-name param + deploy-image-lambda-one.sh case + backend.yml deploy step so the out-of-band image swap reaches the new function. - Bootstrap stub (document_reconciler.py) + Dockerfile COPY keep the byte-stable five-handler image consistent; import-closure and env-contract tests updated. Full infra suite green (792), touched Python supply-chain tests green (21). Stacked on the backend PR (#1007); merge after it. No deploy in this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires the dead-letter document reconciler (built in #1007) into infrastructure: a fifth
kb-migrationLambda, a nightly schedule, and its IAM. Until this lands, the reconciler module exists but nothing runs it.Changes
document_reconciler.lambda_handlerjoins the existing one-image / (now) five-function construct via anImageConfig.Commandoverride — one build, one ECR push, one out-of-bandupdate-function-codeper deploy. Bootstrap stub + Dockerfile COPY keep the byte-stable image consistent.cron(0 9 * * ? *)— 09:00 UTC, ≈ 02:00–03:00 America/Denver — rather thanrate(1 day), so "nightly" means night, not "24h after each deploy". ENABLED regardless of flags, because it ships report-only (MANAGED_KB_DOC_RECONCILER_ARMEDoff) and report-only is read-only — the same inverted convention as the KB reconciler, so the audit period actually happens.grantDirectIngestion(GetKnowledgeBaseDocuments+IngestKnowledgeBaseDocuments) +grantRetrieval(bedrock:Retrieve) + documents-bucket read + assistants-table read/write. NotgrantProvisioningand notiam:PassRole: it reads KB_Records from DynamoDB (neverListKnowledgeBases) and never creates or deletes a knowledge base — a strictly narrower footprint than the KB reconciler beside it.MANAGED_KB_DOC_RECONCILER_ARMED(config.docReconcilerArmed), empty-string = off, forwarded to every function and threaded throughload-env.sh→ context →config.ts. Ships disarmed.deploy-image-lambda-one.shcase, and abackend.ymldeploy step so the image swap reaches the new function.Why nightly
A dead-lettered document is a rare event, and the reconciler is a recovery sweep, not a hot path. Nightly matches the existing KB reconciler's rhythm; the only cost, once armed, is that a stranded-but-retrievable doc could stay invisible up to ~a day — acceptable for recovery. Tighten to hourly in one line here if dead-letters ever prove common.
Tests
Full infra jest suite green (792). Updated every "exactly four" assertion to five (Lambda count, schedules, SSM slugs, flag forwarding), added the nightly-schedule + doc-reconciler-footprint assertions, and updated the retrieval/direct-ingestion holder counts in
managed-kb.test.ts. Touched Python supply-chain tests green (21): the import-closure walk gains the new entrypoint, and the env-contract test moves the flag from an exemption to a spot-pinned load-bearing variable now that the construct sets it.tsc --noEmitclean.Not in this PR
Turning the reconciler on. That is a separate, reviewed step: flip
MANAGED_KB_DOC_RECONCILER_ARMEDafter the report-only audit period, and only after a deploy is agreed.