Skip to content

fix: recover malformed multiscan receipt ledgers - #147

Open
GautamSharma99 wants to merge 1 commit into
openai:mainfrom
GautamSharma99:fix/recover-corrupt-multiscan-receipts
Open

fix: recover malformed multiscan receipt ledgers#147
GautamSharma99 wants to merge 1 commit into
openai:mainfrom
GautamSharma99:fix/recover-corrupt-multiscan-receipts

Conversation

@GautamSharma99

Copy link
Copy Markdown
Contributor

Summary

Fixes #133.

Bulk-scan resume now treats results.jsonl as bounded, validated durable state instead of reading the whole file and trusting every parsed value as a MultiscanReceipt.

A malformed committed record no longer wedges every future resume. The loader keeps valid history, quarantines the original damaged ledger for diagnosis, publishes a repaired ledger, and continues the campaign.

What changed

Bounded streaming

Receipt loading now:

  • streams the ledger in 64 KiB chunks instead of materializing the complete file;
  • enforces a 64 MiB total ledger limit;
  • enforces a 1 MiB limit for each JSONL record;
  • uses a fixed-size pending-line buffer and discards oversized lines without retaining them;
  • checks the total again while reading so file growth cannot bypass the metadata check;
  • observes the campaign abort signal during ledger discovery and parsing.

An oversized ledger is retained under a results.corrupt-<uuid>.jsonl name and replaced with an empty active ledger, allowing the campaign to safely rescan instead of exhausting memory or failing indefinitely.

Receipt validation

Every non-empty line is decoded as strict UTF-8, parsed as JSON, and structurally validated before it can affect resume state. Validation covers:

  • safe task IDs;
  • bounded repository and scope values;
  • full immutable Git revisions;
  • supported scan modes;
  • completed or failed status values;
  • positive safe-integer attempt numbers;
  • bounded absolute output paths;
  • optional cost fields and token counts;
  • optional error values.

Malformed JSON, non-object JSON, missing IDs, invalid statuses or attempts, unsafe paths, invalid costs, and torn final records are excluded from the active ledger.

Quarantine and repair

When corruption is found, valid records are written to a private repair file and flushed before publication. The original ledger is retained as results.corrupt-<uuid>.jsonl, while the repaired set becomes the new results.jsonl.

This preserves usable attempt and completion history without hiding the damaged bytes that caused recovery. A normal, valid ledger is left unchanged.

Bounded generated receipts

New receipts are rejected if they exceed the per-record ceiling. Stored failure messages are also capped at 64 KiB after credential redaction, and inventory scopes now use the same 4,096-character bound enforced by receipt validation.

Documentation

The TypeScript SDK README now documents the ledger and record limits, quarantine naming, valid-history repair behavior, and oversized-ledger rescan behavior.

Tests

Added regression coverage for committed corruption including:

  • malformed JSON;
  • non-object JSON;
  • missing IDs;
  • invalid attempts;
  • invalid statuses;
  • invalid output paths;
  • oversized JSONL records;
  • oversized receipt ledgers.

The tests verify that valid history remains usable, completed repositories stay skipped when their valid receipt survives repair, the original damaged ledger is retained, and an oversized ledger causes a clean rescan.

Verification

  • pnpm run types
  • pnpm run format
  • pnpm run build
  • focused multiscan tests
  • pnpm run test

Full suite: 472 passed, 6 expected platform/integration skips, 0 failed.

@rajpratham1 rajpratham1 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request significantly improves the resilience of multiscan receipt handling by replacing the previous full-file parsing approach with a streaming parser that validates individual receipt records, enforces size limits on both the ledger and individual entries, and safely quarantines corrupted ledgers before resuming execution. Valid receipts are preserved and republished when possible, while oversized or malformed ledgers are isolated to prevent unreliable resume behavior. The changes also bound stored error message sizes and add comprehensive tests covering malformed JSON, invalid receipt fields, oversized records, oversized ledgers, and recovery workflows. Based on the visible changes, the implementation is focused on robustness and recovery without introducing any apparent blocking issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Malformed complete multiscan receipts permanently block campaign resume

2 participants