[TS PBT] Prepare real and artificial benchmarks for the two-phase analysis #356

Scope agreed on 2026-10-16

One concrete PBT generation phase is followed by one USVM search phase. No return to random generation is required. Concrete replay and optional shrinking validate a USVM candidate; they are not a second PBT generation phase. This replaces the previous repeated-round acceptance.

Goal

Build the two requested parts of the experiment: real programs with meaningful original properties, and separately reported artificial programs that isolate rare branches and useful property/observation signals.

Required preparation

• Freeze source revisions, licenses, original oracles, domains, class/scenario identity and the exact executable statement universe independently of which mode wins.
• Preflight concrete replay, frontend retention and the declared symbolic subset before freezing the eligible comparisons. Keep unsupported candidates and exclusion reasons visible.
• Reuse existing domains and supported joint constraints. Do not require symbolic fast-check choice tracing, a command-model framework, helper summaries or dedicated multi-execution infrastructure.
• Artificial cases should vary branch rarity and input structure, include correct/no-benefit controls, and have independently validated faults. Report them separately from real defects and validated mutants.
• For real classes, count original executable class statements; exclude wrappers, model bodies, comments and foreign classes. Credit only checked native hits. Keep the same denominator across modes.
• Use development data to settle bounds/settings; reserve separate projects/families for any held-out claims. Record adaptation effort and project concentration.

Current evidence and blockers

The local three-class coverage pilot is diagnostic evidence, not the completed property-rich corpus. Its two-phase coverage signal was measured, but two classes were absent from IR. Track frontend retention in UnitTestBot/jacodb#447, runtime-limitation reporting in #555 and sparse array support in #388. Do not alter original class text to hide a frontend failure.

Acceptance

Pinned real/artificial manifests, original properties, validation commands and capability outcomes are available to #357. Supported and unsupported populations are reproducible. The former closed-without-merge corpus PRs #406/#408 do not count as delivered main code.