tacasv2a: --slice that keeps the suite valid (reach + assert) - #68
Open
GuilhermeBn198 wants to merge 25 commits into
Open
GuilhermeBn198 wants to merge 25 commits into
GuilhermeBn198 wants to merge 25 commits into
Conversation
Found by the final whole-branch review and confirmed by the Task 7 smoke test in the rebuilt dev image. - NonDetGeneratorAFL.c: define the persistent-mode macros exactly as afl-cc 4.40c injects them. The library is compiled to bitcode by plain clang, so without them every build (including KLEE-only and CI) failed, and afl-clang-fast never re-preprocesses the linked -result.bc, so the ##SIG_AFL_PERSISTENT## marker must already be in the bitcode. - Caller: replay crashes from afl-out/default/crashes (the single instance's directory), feeding each over stdin (a standalone persistent binary ignores argv), bounded by timeout, and stop at the first confirmed one so a non-reproducing replay cannot overwrite it. - Caller: set the headless AFL_* flags on the command line rather than relying on the image ENV; record crashing seeds as crashes and stop at the first crash, as the previous fuzzer did. - Caller: use a private afl-in/ unless --seed-exchange, and copy the AFL++ queue back into seeds/ under it (afl-fuzz never writes into -i). - tools.hpp: MAP2CHECK_AFL_CC / MAP2CHECK_AFL_FUZZ overrides; AFL_CC is AFL++'s own variable and reusing it recursed or disabled instrumentation. - Regression test: --nondet-generator afl; stale comments updated. Verified in map2check-dev:aflpp: persistent + shared-memory mode detected, VERIFICATION FAILED on the plan's smoke program and on a non-trivial one, test_testcomp_regressions.sh 19/19, ctest 9/9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Cap each crash replay at max(5s, 10% of the budget): a crash that does not reproduce from a fresh process may loop, and must not eat KLEE's share. - Select crash/queue files by type (regular, not README.txt) rather than by AFL++'s "id:" naming, which AFL_SHA1_FILENAMES changes. - Warn when the AFL++ input dir or placeholder seed cannot be prepared. - Document that seeds/ does not yet survive into the next hybrid phase: each Caller recreates the scratch directory. Inherited unchanged from v15; fixing it changes what the hybrid measures, so it belongs to the smart-seeds work. - Regression test: count only the fuzzer's copied discoveries (afl-*), since the placeholder seed made the old file count pass trivially. Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19 (4 afl-* discoveries copied), ctest 9/9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v15 LibFuzzer runs with -use_value_profile=1, which solves comparison guards such as `x == 123456`. AFL++ with PCGUARD alone does not, so the tacasv1 comparison measured an unarmed AFL++ against an armed LibFuzzer. CmpLog (input-to-state) is AFL++'s counterpart. - Caller compiles a third binary, <hash>-cmplog.out, from the same -result.bc with AFL_LLVM_CMPLOG=1 and passes it to afl-fuzz with -c. Optional: if it does not build, the fuzzer runs without it and says so. - NonDetGeneratorAFL.c rewinds its read position for every __AFL_LOOP iteration. The position was function-static and carried over between inputs, so the same input read different values on every persistent run. That nondeterminism defeats CmpLog's byte-to-operand mapping. This reverses the earlier choice to preserve the index for v15 parity; without the reset CmpLog has no effect. Fuzzer-only smoke, 3 runs x 8 seeded bugs, same image: v15 LibFuzzer 17/24 | AFL++ + CmpLog, index kept 4/24 | AFL++ + CmpLog, index rewound 17/24. - Regression test: the KLEE -> fuzzer export check retries when the (now stronger) fuzzer phase solves seed.c first and KLEE never runs. - Spec: addendum §2.1 with the decision and the numbers. Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19 (twice), ctest 9/9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
afl-fuzz -V compares gettimeofday() readings. A wall clock stepped
backwards -- measured at 1.1s within 20s under WSL2 -- underflows the
elapsed time, and afl-fuzz stops after a few hundred executions
("Time limit was reached", run_time ~2^64). `timeout` uses a relative
timer, and it is how the previous fuzzer was bounded. The budget is
also clamped to at least 1s, since `timeout 0` means no limit.
Spec §2.1 updated with the final fuzzer-only smoke (3 runs x 8 bugs):
v15 LibFuzzer 20/24, AFL++ + CmpLog with the rewound index 17/24,
without it 8/24. Hybrid verdicts identical to v15 on all 11 programs.
Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19,
ctest 9/9.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…input 621db93 downgrades a KLEE violation to UNKNOWN when no input vector can be recovered, to stop FAILED verdicts that nothing reproduces. It asked readViolatingKtest for a non-empty vector, so a program that reads no nondeterministic input -- whose aborting path has a .ktest with zero objects -- was downgraded too, and its suite came out with no test case. That is tests/testcomp/programs/no_input.c, and it has failed the TestCov CI gate on develop since that commit landed (run 33523637266). hasViolatingKtest counts an .abort.err whose .ktest exists, empty or not; the downgrade now uses it. The halted-KLEE case 621db93 guards against still has no .abort.err at all, so it is still downgraded. Verified in both the current CI image (ghcr map2check-dev:latest) and map2check-dev:aflpp: run_testcov_suite.sh 6/6 (no_input.c COVERED), test_cover_branches.sh 5/5, test_benchexec_toolinfo.py 16/16, test_testcomp_regressions.sh 19/19, ctest 9/9 (3 new KtestReader tests). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… suite valid Diagnosis on 12 tasks the v15 slice arm lost, with the tacasv1 build: the slicer's cutoff inserts exit(0) without !dbg and KLEE aborts on a broken module; and nondet calls the slicer drops misalign the suite on the original program. Cutoff off + nondets as criteria: 6/12 -> 10/12. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cutoff's exit(0) carries no !dbg and KLEE rejected every sliced module with a cut path; nondet reads the slicer dropped shifted the suite on the original program. Both are integration-tested now, and the slice is logged in functions/blocks/instructions, not only bytes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Control 12/12, slice 12/12 on the 12 tasks the v15 slice arm lost. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fixed list cannot know every __VERIFIER_nondet_* a benchmark declares (int128, uint128, ...), and a missing name silently brings the shifted suite back. The slicer input is disassembled with opt -S and every nondet symbol joins the criteria; the fixed list is the fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Branch naming follows feat/** instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GENERATOR=fuzzer still passed --nondet-generator fuzzer, which the CLI no longer accepts. It now names AFL++, and the old value is refused with a pointer to the new one rather than silently measuring nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tacasv2a, the first slice of the TACAS slicing work, making slicing per property. This PR fixes
--slicefor reachability and adds--slicefor assert mode. It builds on #69 (tacasv1, AFL++; formerly #66). The base isdevelop, so the CI runs. Until #69 is merged, this diff also shows its commits; merge #69 first and they drop out. (It replaces #67, which GitHub closed when the head branch was renamed fromtacas/slicingtofeat/tacas-slicing.)docs/superpowers/specs/2026-09-26-tacasv2a-slicing-reach-assert-design.mddocs/superpowers/plans/2026-09-26-tacasv2a-slicing-reach-assert.mdThe baseline for slicing is now tacasv1, not v15, and both sides use the same engine: AFL++ is the decided evolution, so nothing is calibrated against LibFuzzer.
What was wrong (diagnosed, not assumed)
In v15 the slice arm covered 412 tasks against 470 for the control. 57 of the 58 losses were in ECA. I sampled 12 of the lost tasks and ran them on the tacasv1 build. Two independent defects came out.
--cutoff-divergingis on by default. It rewrites every path that cannot reach the criterion into adiverge:block that callsexit(0).exit(0).-g.exithas a body, and the LLVM verifier rejects the module:inlinable function call in a function with debug info must have a !dbg location→Broken module found.exit(0)also ended the LibFuzzer session.ntdrivers/floppy.i.cil-1.chas 29 reads in the program and 10 in the slice.What changed
-cutoff-diverging=false.__VERIFIER_nondet_*function is a primary slicing criterion, so the read order is kept. The names come from a fixed list plus the program's own IR (opt -S), because no fixed list knows every name a benchmark declares (int128, …).--slicenow also works with--check-asserts. The criteria are__VERIFIER_assert,__assert_fail, the two functions AssertPass instruments, and the weak stub has the(int cond)signature.--statistics, not only in bytes.modules/frontend/utils/slicer.hpp, which has unit tests.Compared with Symbiotic (spec §4, read from its source):
Results
12-task sample: the tasks v15's slice arm lost. Build
tacas/slicing, 300 s budget, TestCov 300 s, one run per arm.floppyis FAILED + COVEREDWith this PR, the slice run finished faster than the control on 5 of the 9 ECA tasks. For example,
Problem17_label55gave FAILED in 51 s with the slice and UNKNOWN in 137 s without it. This is a diagnostic check, not an evaluation. The full Cover-Error comparison (1087 tasks, control vs slice, same build) is the tacasv2a run planned after merge (spec §7).Test plan
ctest: 10/10 (the newSlicerTesthas 11 tests)tests/integration/test_testcomp_regressions.sh: 23/23. New sections:Broken module.[-2 42]. Before the fix it gave[42].ffad3e5d0: nondet names taken from the program), 5 Minor (deferred, listed below)Deferred (minor, from the review)
__VERIFIER_assertwith--check-asserts --slice. It is safe according to the dg source.testcase-1.xml.(void)weak stub. This predates the PR.Decisions taken during execution
diverge:block: the slicer turns a plain return frommainintosafe_return. I replaced it with a small bounded ECA-shaped loop, which reproducesBroken moduleon the old code and gives FAILED on the new code.-cda ntscdand a "fixed" cutoff (keeping the cut but adding!dbg) were not adopted. The spec leaves both to measurement.🤖 Generated with Claude Code