fix: bound bulk-scan inventory loading

[GautamSharma99]

Summary

Fixes #132.

Add explicit resource ceilings to bulk-scan inventory loading and interactive GitHub discovery, and replace full CSV materialization with incremental parsing.

Previously, a supplied inventory was read into one complete UTF-8 string, Papa Parse retained a complete row array, and the rows were then mapped into another task array. Interactive discovery similarly accumulated every qualifying GitHub repository without a campaign limit. Both paths could consume excessive memory and create unexpectedly large checkout, scan, and model-cost scope before work began.

Limits

Bulk scan now enforces shared limits of:

• 8 MiB per CSV inventory;
• 1,000 repositories per campaign.

The same repository ceiling applies to supplied inventories and interactive GitHub discovery.

Changes

Metadata validation before campaign state

runMultiscan now checks inventory metadata before opening or parsing the CSV.

It rejects:

• inputs that are not regular files;
• files larger than 8 MiB;
• inventories containing more than 1,000 repository rows.

All inventory parsing and validation still happens before the output directory, campaign lock, manifest, checkout directory, artifact directory, or security client is created.

Incremental CSV parsing

Inventory loading now streams the file through Papa Parse in 64 KiB input chunks.

The parser:

• processes the header once;
• validates and converts one repository row at a time;
• retains only the bounded task list and duplicate-ID set;
• preserves quoted fields, embedded commas, escaped quotes, CRLF input, blank-line skipping, and UTF-8 BOM handling;
• preserves existing header, field-count, ID, revision, mode, scope, and repository validation;
• checks cancellation between rows.

This removes the previous simultaneous retention of the complete source string, complete Papa row array, and mapped task array.

In-stream byte enforcement

The initial metadata check rejects known oversized files before reading. A byte-counting transform independently enforces the 8 MiB ceiling while the file is read.

This second check prevents a file replacement or growth race after metadata validation from bypassing the input limit.

The source and parser streams are destroyed immediately when parsing, cancellation, byte, or repository-count validation fails.

Bounded GitHub discovery

Interactive discovery now stops and returns an actionable error as soon as the 1,001st qualifying repository is encountered.

The limit is enforced during pagination, before repository selection, output-directory creation, or inventory writing. Empty repositories and repositories older than the existing 90-day cutoff continue to be excluded before they count toward the campaign limit.

The error recommends splitting the repository list into smaller campaigns.

Shared limits and documentation

The byte and repository constants, along with the repository-limit diagnostic, are shared by CSV loading and GitHub discovery so the two entry points cannot drift.

The README now documents both limits and advises using separate inventories and output directories for larger campaigns.

Security and resource impact

This bounds:

• bytes read from a supplied inventory;
• parser source buffering;
• retained repository tasks;
• interactive GitHub repository accumulation;
• the maximum checkout and scan scope of one campaign.

Oversized input is rejected before persistent campaign state or checkout work begins.

Tests

Added regression coverage for:

• rejecting an inventory whose metadata exceeds 8 MiB;
• rejecting a valid 1,001-row inventory;
• ensuring neither case creates a security client;
• ensuring neither case creates the output directory or campaign state;
• accepting exactly 1,000 repositories during interactive discovery;
• paginating exactly ten 100-repository GitHub pages at the boundary;
• rejecting the 1,001st discovered repository;
• stopping before an interactive inventory or output directory is created;
• preserving the existing quoted CSV, BOM, CRLF, validation, pagination, cutoff, and selection behavior.

Verification

• pnpm run types
• pnpm run format
• pnpm run build
• focused multiscan and bulk-discovery tests
• PATH="/opt/homebrew/bin:$PATH" pnpm run test

Full test result:

• 472 passed
• 6 expected platform/integration skips
• 0 failed