fix: recover malformed multiscan receipt ledgers - #147
Conversation
rajpratham1
left a comment
There was a problem hiding this comment.
This pull request significantly improves the resilience of multiscan receipt handling by replacing the previous full-file parsing approach with a streaming parser that validates individual receipt records, enforces size limits on both the ledger and individual entries, and safely quarantines corrupted ledgers before resuming execution. Valid receipts are preserved and republished when possible, while oversized or malformed ledgers are isolated to prevent unreliable resume behavior. The changes also bound stored error message sizes and add comprehensive tests covering malformed JSON, invalid receipt fields, oversized records, oversized ledgers, and recovery workflows. Based on the visible changes, the implementation is focused on robustness and recovery without introducing any apparent blocking issues.
Summary
Fixes #133.
Bulk-scan resume now treats
results.jsonlas bounded, validated durable state instead of reading the whole file and trusting every parsed value as aMultiscanReceipt.A malformed committed record no longer wedges every future resume. The loader keeps valid history, quarantines the original damaged ledger for diagnosis, publishes a repaired ledger, and continues the campaign.
What changed
Bounded streaming
Receipt loading now:
An oversized ledger is retained under a
results.corrupt-<uuid>.jsonlname and replaced with an empty active ledger, allowing the campaign to safely rescan instead of exhausting memory or failing indefinitely.Receipt validation
Every non-empty line is decoded as strict UTF-8, parsed as JSON, and structurally validated before it can affect resume state. Validation covers:
Malformed JSON, non-object JSON, missing IDs, invalid statuses or attempts, unsafe paths, invalid costs, and torn final records are excluded from the active ledger.
Quarantine and repair
When corruption is found, valid records are written to a private repair file and flushed before publication. The original ledger is retained as
results.corrupt-<uuid>.jsonl, while the repaired set becomes the newresults.jsonl.This preserves usable attempt and completion history without hiding the damaged bytes that caused recovery. A normal, valid ledger is left unchanged.
Bounded generated receipts
New receipts are rejected if they exceed the per-record ceiling. Stored failure messages are also capped at 64 KiB after credential redaction, and inventory scopes now use the same 4,096-character bound enforced by receipt validation.
Documentation
The TypeScript SDK README now documents the ledger and record limits, quarantine naming, valid-history repair behavior, and oversized-ledger rescan behavior.
Tests
Added regression coverage for committed corruption including:
The tests verify that valid history remains usable, completed repositories stay skipped when their valid receipt survives repair, the original damaged ledger is retained, and an oversized ledger causes a clean rescan.
Verification
pnpm run typespnpm run formatpnpm run buildpnpm run testFull suite: 472 passed, 6 expected platform/integration skips, 0 failed.