feat(kb): dead-letter document reconciler (task 16.5, §5.37)

[DerrickF]

What

Adds kb_migration/document_reconciler.py — the missing second writer of DOC# document status — closing HANDOFF §5.37 / task 16.5.

Why

On the managed path a document's DOC# status is written by exactly one writer: the ingestion consumer, which polls Bedrock until the doc is genuinely retrievable and only then writes complete. But it runs in a Lambda whose async retry is capped at 2 attempts (a hard service limit). When an event exhausts those retries and dead-letters, the row is left non-terminal (uploading/chunking/embedding) with nothing left to revisit it — even though Bedrock often finished indexing seconds later, so the content is in the KB fully retrievable.

That is invisible and permanent: rag_service._filter_vectors_by_document_status serves only complete docs, so a stranded row's chunks are dropped from every query. The user was told their upload worked; the assistant never cites it. Two such docs occurred in dev and both needed a manual DynamoDB edit.

How

Daily, the reconciler finds DOC# rows stuck non-terminal past a 60-minute grace gate, asks Bedrock the ground truth per document, and:

• INDEXED / PARTIALLY_INDEXED + confirmed retrievable → complete (the §5.37 fix)
• FAILED / METADATA_UPDATE_FAILED → failed
• NOT_FOUND (dead-lettered before ingest was accepted) → re-ingest the bytes still in S3 — the scheduled form of task 14.4's one-click retry
• in-flight / unknown status → left alone (§5.39: the live service returns statuses the SDK enum omits)

It reuses the consumer's own probes — document_status (GetKnowledgeBaseDocuments), the equals-on-document_id retrievability search, its status-set constants, and set_document_terminal — so §5.37 (poll budget), §5.38 (unfiltered search finds the wrong doc) and §5.39 (undeclared statuses) live in one place, not four. The one thing it does not reuse is the consumer's polling: it takes a single retrievability reading per doc, because it sweeps a fleet rather than shepherding one upload.

Safety (modelled on reconciler.py)

• Ships disarmed. MANAGED_KB_DOC_RECONCILER_ARMED, empty string reads as off. Report-only logs exactly what it would do and writes nothing.
• Per-run action limit applies in both modes, so a report never claims more corrections than an armed run would make.
• Grace gate is a pure function of the row's own updatedAt, never discovery time, and fails closed when it cannot be read — so an in-flight upload is never marked from under the consumer.
• complete/failed/deleting rows are never candidates — a soft-deleted doc must not be resurrected.
• Event payload cannot arm the reconciler (flag only), matching the KB reconciler.

Tests

tests/lambdas/test_kb_document_reconciler.py — 61 tests, moto DynamoDB + a stub backend modelling Bedrock's document view (same style as test_kb_ingestion_consumer). Mutation-verified: neutering the retrievability gate fails test_indexed_but_not_retrievable_is_left_short_of_complete; widening NON_TERMINAL_STATUSES fails test_terminal_and_deleting_rows_are_never_candidates. Green: new suite + test_kb_reconciler + test_kb_backend_boundary + test_lambda_image_imports + test_kb_migration_env_contract (145 total). ruff clean; mypy profile identical to the sibling reconciler.py.

Deploy-gated follow-up (NOT in this PR)

The reconciler's own Lambda + EventBridge schedule + IAM in kb-migration-construct.ts, then setting/flipping the arming flag. The flag is exempted in the env-contract test's OPTIONAL_OVERRIDES until that infra lands. No infrastructure changed and nothing is deployed by this PR.