Skip to content

tacasv2a: --slice that keeps the suite valid (reach + assert) - #68

Open
GuilhermeBn198 wants to merge 25 commits into
developfrom
feat/tacas-slicing
Open

GuilhermeBn198 wants to merge 25 commits into
developfrom
feat/tacas-slicing

Conversation

@GuilhermeBn198

@GuilhermeBn198 GuilhermeBn198 commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

tacasv2a, the first slice of the TACAS slicing work, making slicing per property. This PR fixes --slice for reachability and adds --slice for assert mode. It builds on #69 (tacasv1, AFL++; formerly #66). The base is develop, so the CI runs. Until #69 is merged, this diff also shows its commits; merge #69 first and they drop out. (It replaces #67, which GitHub closed when the head branch was renamed from tacas/slicing to feat/tacas-slicing.)

  • Spec: docs/superpowers/specs/2026-09-26-tacasv2a-slicing-reach-assert-design.md
  • Plan: docs/superpowers/plans/2026-09-26-tacasv2a-slicing-reach-assert.md

The baseline for slicing is now tacasv1, not v15, and both sides use the same engine: AFL++ is the decided evolution, so nothing is calibrated against LibFuzzer.

What was wrong (diagnosed, not assumed)

In v15 the slice arm covered 412 tasks against 470 for the control. 57 of the 58 losses were in ECA. I sampled 12 of the lost tasks and ran them on the tacasv1 build. Two independent defects came out.

  1. KLEE aborts on every slice that has a cut path.
    • sbt-slicer's --cutoff-diverging is on by default. It rewrites every path that cannot reach the criterion into a diverge: block that calls exit(0).
    • The help text says "abort()", but the dg source calls exit(0).
    • That call carries no debug location, while the program is compiled with -g.
    • Once KLEE links uClibc, exit has a body, and the LLVM verifier rejects the module: inlinable function call in a function with debug info must have a !dbg location → Broken module found.
    • KLEE never ran on those tasks. In v15 the exit(0) also ended the LibFuzzer session.
  2. The suite found on the slice did not hold on the original program.
    • The slicer drops nondet reads that do not affect the target. ntdrivers/floppy.i.cil-1.c has 29 reads in the program and 10 in the slice.
    • TestCov runs the suite on the original program, where the vector shifts. The verdict is FAILED, but the suite is NOT_COVERED.

What changed

  • The slicer runs with -cutoff-diverging=false.
  • Every __VERIFIER_nondet_* function is a primary slicing criterion, so the read order is kept. The names come from a fixed list plus the program's own IR (opt -S), because no fixed list knows every name a benchmark declares (int128, …).
  • --slice now also works with --check-asserts. The criteria are __VERIFIER_assert,__assert_fail, the two functions AssertPass instruments, and the weak stub has the (int cond) signature.
  • The slice is logged in functions/blocks/instructions, from --statistics, not only in bytes.
  • The pure pieces live in a new header-only modules/frontend/utils/slicer.hpp, which has unit tests.

Compared with Symbiotic (spec §4, read from its source):

  • Symbiotic keeps the cutoff and never slices for Test-Comp properties in its main pipeline.
  • What is new here: slicing that preserves the order in which inputs are consumed, so that a suite generated on the slice stays valid on the original program.

Results

12-task sample: the tasks v15's slice arm lost. Build tacas/slicing, 300 s budget, TestCov 300 s, one run per arm.

arm covered
control (no slice) 12/12
slice, before this PR 6/12
slice, this PR 12/12; floppy is FAILED + COVERED

With this PR, the slice run finished faster than the control on 5 of the 9 ECA tasks. For example, Problem17_label55 gave FAILED in 51 s with the slice and UNKNOWN in 137 s without it. This is a diagnostic check, not an evaluation. The full Cover-Error comparison (1087 tasks, control vs slice, same build) is the tacasv2a run planned after merge (spec §7).

Test plan

  • ctest: 10/10 (the new SlicerTest has 11 tests)
  • tests/integration/test_testcomp_regressions.sh: 23/23. New sections:
    • 13: KLEE runs on a slice with a cut path. It failed before the fix with Broken module.
    • 14: the sliced suite keeps the original read order, giving [-2 42]. Before the fix it gave [42].
    • 15: assert mode slices towards the assertions and still finds the violation.
    • 16: nondet names that only the program declares are criteria too.
  • CI TestCov step reproduced locally in the dev image: 16/16, 5/5, 6/6
  • 12-task Cover-Error sample: slice 12/12
  • Whole-branch review by a fresh reviewer: 0 Critical, 1 Important (fixed in ffad3e5d0: nondet names taken from the program), 5 Minor (deferred, listed below)
  • CI

Deferred (minor, from the review)

  • sbt-slicer prints "Statistics after" before its final cleanup, so the logged after-counts understate the reduction.
  • The read-order guarantee holds for terminating executions. Standard control dependence can slice away a non-terminating loop that feeds nothing, and the comment and CHANGELOG wording should say so.
  • There is no integration test yet for an SV-COMP-defined __VERIFIER_assert with --check-asserts --slice. It is safe according to the dg source.
  • Test §14 reads only testcase-1.xml.
  • A user target with parameters still gets a (void) weak stub. This predates the PR.

Decisions taken during execution

  • The plan's test-13 program never produced a diverge: block: the slicer turns a plain return from main into safe_return. I replaced it with a small bounded ECA-shaped loop, which reproduces Broken module on the old code and gives FAILED on the new code.
  • -cda ntscd and a "fixed" cutoff (keeping the cut but adding !dbg) were not adopted. The spec leaves both to measurement.

🤖 Generated with Claude Code

GuilhermeBn198 and others added 25 commits September 25, 2026 22:34
Found by the final whole-branch review and confirmed by the Task 7 smoke
test in the rebuilt dev image.

- NonDetGeneratorAFL.c: define the persistent-mode macros exactly as
  afl-cc 4.40c injects them. The library is compiled to bitcode by plain
  clang, so without them every build (including KLEE-only and CI) failed,
  and afl-clang-fast never re-preprocesses the linked -result.bc, so the
  ##SIG_AFL_PERSISTENT## marker must already be in the bitcode.
- Caller: replay crashes from afl-out/default/crashes (the single
  instance's directory), feeding each over stdin (a standalone persistent
  binary ignores argv), bounded by timeout, and stop at the first
  confirmed one so a non-reproducing replay cannot overwrite it.
- Caller: set the headless AFL_* flags on the command line rather than
  relying on the image ENV; record crashing seeds as crashes and stop at
  the first crash, as the previous fuzzer did.
- Caller: use a private afl-in/ unless --seed-exchange, and copy the
  AFL++ queue back into seeds/ under it (afl-fuzz never writes into -i).
- tools.hpp: MAP2CHECK_AFL_CC / MAP2CHECK_AFL_FUZZ overrides; AFL_CC is
  AFL++'s own variable and reusing it recursed or disabled instrumentation.
- Regression test: --nondet-generator afl; stale comments updated.

Verified in map2check-dev:aflpp: persistent + shared-memory mode detected,
VERIFICATION FAILED on the plan's smoke program and on a non-trivial one,
test_testcomp_regressions.sh 19/19, ctest 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Cap each crash replay at max(5s, 10% of the budget): a crash that does
  not reproduce from a fresh process may loop, and must not eat KLEE's
  share.
- Select crash/queue files by type (regular, not README.txt) rather than
  by AFL++'s "id:" naming, which AFL_SHA1_FILENAMES changes.
- Warn when the AFL++ input dir or placeholder seed cannot be prepared.
- Document that seeds/ does not yet survive into the next hybrid phase:
  each Caller recreates the scratch directory. Inherited unchanged from
  v15; fixing it changes what the hybrid measures, so it belongs to the
  smart-seeds work.
- Regression test: count only the fuzzer's copied discoveries (afl-*),
  since the placeholder seed made the old file count pass trivially.

Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19
(4 afl-* discoveries copied), ctest 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v15 LibFuzzer runs with -use_value_profile=1, which solves
comparison guards such as `x == 123456`. AFL++ with PCGUARD alone does
not, so the tacasv1 comparison measured an unarmed AFL++ against an
armed LibFuzzer. CmpLog (input-to-state) is AFL++'s counterpart.

- Caller compiles a third binary, <hash>-cmplog.out, from the same
  -result.bc with AFL_LLVM_CMPLOG=1 and passes it to afl-fuzz with -c.
  Optional: if it does not build, the fuzzer runs without it and says so.
- NonDetGeneratorAFL.c rewinds its read position for every __AFL_LOOP
  iteration. The position was function-static and carried over between
  inputs, so the same input read different values on every persistent
  run. That nondeterminism defeats CmpLog's byte-to-operand mapping.
  This reverses the earlier choice to preserve the index for v15 parity;
  without the reset CmpLog has no effect.

Fuzzer-only smoke, 3 runs x 8 seeded bugs, same image:
  v15 LibFuzzer 17/24 | AFL++ + CmpLog, index kept 4/24 |
  AFL++ + CmpLog, index rewound 17/24.

- Regression test: the KLEE -> fuzzer export check retries when the
  (now stronger) fuzzer phase solves seed.c first and KLEE never runs.
- Spec: addendum §2.1 with the decision and the numbers.

Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19
(twice), ctest 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
afl-fuzz -V compares gettimeofday() readings. A wall clock stepped
backwards -- measured at 1.1s within 20s under WSL2 -- underflows the
elapsed time, and afl-fuzz stops after a few hundred executions
("Time limit was reached", run_time ~2^64). `timeout` uses a relative
timer, and it is how the previous fuzzer was bounded. The budget is
also clamped to at least 1s, since `timeout 0` means no limit.

Spec §2.1 updated with the final fuzzer-only smoke (3 runs x 8 bugs):
v15 LibFuzzer 20/24, AFL++ + CmpLog with the rewound index 17/24,
without it 8/24. Hybrid verdicts identical to v15 on all 11 programs.

Verified in map2check-dev:aflpp: test_testcomp_regressions.sh 19/19,
ctest 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…input

621db93 downgrades a KLEE violation to UNKNOWN when no input vector
can be recovered, to stop FAILED verdicts that nothing reproduces. It
asked readViolatingKtest for a non-empty vector, so a program that
reads no nondeterministic input -- whose aborting path has a .ktest
with zero objects -- was downgraded too, and its suite came out with
no test case. That is tests/testcomp/programs/no_input.c, and it has
failed the TestCov CI gate on develop since that commit landed
(run 33523637266).

hasViolatingKtest counts an .abort.err whose .ktest exists, empty or
not; the downgrade now uses it. The halted-KLEE case 621db93 guards
against still has no .abort.err at all, so it is still downgraded.

Verified in both the current CI image (ghcr map2check-dev:latest) and
map2check-dev:aflpp: run_testcov_suite.sh 6/6 (no_input.c COVERED),
test_cover_branches.sh 5/5, test_benchexec_toolinfo.py 16/16,
test_testcomp_regressions.sh 19/19, ctest 9/9 (3 new KtestReader tests).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… suite valid

Diagnosis on 12 tasks the v15 slice arm lost, with the tacasv1 build:
the slicer's cutoff inserts exit(0) without !dbg and KLEE aborts on a
broken module; and nondet calls the slicer drops misalign the suite on
the original program. Cutoff off + nondets as criteria: 6/12 -> 10/12.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cutoff's exit(0) carries no !dbg and KLEE rejected every sliced
module with a cut path; nondet reads the slicer dropped shifted the
suite on the original program. Both are integration-tested now, and the
slice is logged in functions/blocks/instructions, not only bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Control 12/12, slice 12/12 on the 12 tasks the v15 slice arm lost.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fixed list cannot know every __VERIFIER_nondet_* a benchmark
declares (int128, uint128, ...), and a missing name silently brings the
shifted suite back. The slicer input is disassembled with opt -S and
every nondet symbol joins the criteria; the fixed list is the fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Branch naming follows feat/** instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GENERATOR=fuzzer still passed --nondet-generator fuzzer, which the CLI
no longer accepts. It now names AFL++, and the old value is refused
with a pointer to the new one rather than silently measuring nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant