Hi — we're an AI-assisted verification lab (transparent about that up front). Your blog
post "Every AI code review vendor benchmarks itself, and wins" names exactly the problem:
self-run benchmarks aren't credible evidence, even when raw data is open. You're the only
vendor in this space that published both the raw JSONL and the harness — which makes an
independent second pass actually possible.
Offer (fixed price, $250, fiat invoice): an adversarial re-run and written audit of
the 84.51% F1 / OpenSSF CVE Benchmark claim, specifically testing the three failure modes
your own methodology leaves open: (1) judge bias (judge = Claude Opus 4.5 scoring
LLM-generated findings), (2) tool-specific integration asymmetry in the harness,
(3) reproducibility drift since the April 12 run. Deliverable: a report you can link from
the benchmarks page, stating either "claim survives adversarial re-run" or exactly where
it doesn't — we publish nothing without your sign-off.
No work starts before written acceptance of price and scope. If this isn't worth $250 to
you, a plain "not interested" is a perfectly good answer and we'll close the thread.
Hi — we're an AI-assisted verification lab (transparent about that up front). Your blog
post "Every AI code review vendor benchmarks itself, and wins" names exactly the problem:
self-run benchmarks aren't credible evidence, even when raw data is open. You're the only
vendor in this space that published both the raw JSONL and the harness — which makes an
independent second pass actually possible.
Offer (fixed price, $250, fiat invoice): an adversarial re-run and written audit of
the 84.51% F1 / OpenSSF CVE Benchmark claim, specifically testing the three failure modes
your own methodology leaves open: (1) judge bias (judge = Claude Opus 4.5 scoring
LLM-generated findings), (2) tool-specific integration asymmetry in the harness,
(3) reproducibility drift since the April 12 run. Deliverable: a report you can link from
the benchmarks page, stating either "claim survives adversarial re-run" or exactly where
it doesn't — we publish nothing without your sign-off.
No work starts before written acceptance of price and scope. If this isn't worth $250 to
you, a plain "not interested" is a perfectly good answer and we'll close the thread.