CI: skill benchmark gate (localaik executor + Binoculars) with PR comment#8
CI: skill benchmark gate (localaik executor + Binoculars) with PR comment#8harshaneel wants to merge 4 commits into
Conversation
Humanize skill benchmark — PASS ✅25 fixed AI-flavored inputs humanized with the PR's Scoring model: the official Binoculars zero-shot scorer (Hans et al., ICML 2024) running the How to read: each score estimates how human the text reads (higher = more human). A humanized output should score above the raw AI input it came from; the gate checks the average lift and how many outputs fell below their own input.
Per-input inspection (25 items)id 1 · Tech blog post · raw 0.874 → humanized 0.906 ✅ · 464 words · 4sOriginal (AI-flavored input):
Humanized (PR skill):
id 2 · Engineering postmortem · raw 0.940 → humanized 1.018 ✅ · 486 words · 4sOriginal (AI-flavored input):
Humanized (PR skill):
id 3 · Product launch announcement · raw 0.826 → humanized 0.953 ✅ · 179 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 4 · Academic abstract · raw 0.929 → humanized 1.017 ✅ · 392 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 5 · Business email · raw 0.848 → humanized 0.910 ✅ · 98 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 6 · Internal Slack update · raw 0.907 → humanized 1.030 ✅ · 248 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 7 · LinkedIn post · raw 0.861 → humanized 0.904 ✅ · 333 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 8 · Cover letter · raw 0.812 → humanized 0.867 ✅ · 123 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 9 · Marketing copy · raw 0.856 → humanized 1.007 ✅ · 138 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 10 · Press release · raw 0.848 → humanized 0.867 ✅ · 124 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 11 · Investor update · raw 0.802 → humanized 0.896 ✅ · 156 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 12 · Job posting · raw 0.812 → humanized 0.909 ✅ · 482 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 13 · Customer support reply · raw 0.777 → humanized 0.864 ✅ · 116 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 14 · Recipe intro · raw 0.838 → humanized 0.919 ✅ · 483 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 15 · Travel writing · raw 0.849 → humanized 0.923 ✅ · 488 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 16 · Restaurant review · raw 0.854 → humanized 0.917 ✅ · 502 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 17 · Book review · raw 0.860 → humanized 0.963 ✅ · 317 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 18 · Personal essay · raw 0.955 → humanized 0.975 ✅ · 474 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 19 · Privacy policy section · raw 0.936 → humanized 0.970 ✅ · 143 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 20 · Tutorial intro · raw 0.858 → humanized 1.052 ✅ · 125 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 21 · Comparison article · raw 0.954 → humanized 0.970 ✅ · 506 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 22 · Roadmap update · raw 0.804 → humanized 0.872 ✅ · 128 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 23 · Conference abstract · raw 0.920 → humanized 1.000 ✅ · 515 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 24 · README intro · raw 0.842 → humanized 0.929 ✅ · 179 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 25 · Career advice · raw 0.948 → humanized 0.997 ✅ · 353 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
|
On PRs touching humanize/SKILL.md: humanize 25 fixed AI-flavored inputs (the documented benchmark registers) with the PR's skill, score outputs and raw inputs with pinned official Binoculars (TinyLlama pair), and enforce a baseline gate: mean lift over the raw inputs and a cap on outputs scoring below their own raw input (thresholds in .github/benchmark/baseline.json). Results are upserted as a marker-keyed PR comment with per-input collapsible inspection (original vs humanized text, score movement, timing), auto-fitted to GitHub's comment limit. Executor: Gemini free tier (gemini-flash-lite-latest rolling alias) via the OpenAI-compatible endpoint. The GEMINI_API_KEY secret is scoped to the `benchmark` environment so it is only usable by approved runs of this workflow. Fork PRs run under pull_request_target: workflow and all scripts execute from the base branch and only the fork's SKILL.md is fetched, as prompt data — fork code never executes next to the secret. Same-repo PRs use the plain pull_request trigger; conditions are mutually exclusive. Resilience: humanize and score are separate jobs with an artifact handoff so "Re-run failed jobs" repeats only the failed half; the executor checkpoints per item and re-run attempts seed from the prior attempt's partial artifact; patient exponential backoff (20s-300s, 6 attempts) absorbs free-tier 429/503 throttling; deps and the Binoculars source are version/SHA-pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d381ef3 to
fe44bb5
Compare
…thresholds - Per-input inspection now nests all item dropdowns inside a single outer dropdown so the comment stays compact - One-line "how to read" note before the score tables; caveats footer removed - Thresholds calibrated to the measured Gemini flash-lite run (+0.076 lift, 2/25 below raw): gate now requires mean lift >= +0.05 and at most 4 outputs below their own raw input Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The comment now states explicitly that scores come from the official Binoculars detector (Hans et al., ICML 2024) with the TinyLlama-1.1B base + chat model pair. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Comment wording: "scoring model"/"scorer" instead of detector-style language - The executor records the concrete model version each response resolved to (rolling aliases hide it), and the comment now shows the distinct resolved names next to the executor label Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Superseded by #9 — same change squashed to one commit; this PR accumulated iteration noise. |
Summary
skill-benchmarkGitHub workflow: on any PR touchinghumanize/SKILL.md(or the benchmark harness itself), humanize 25 fixed AI-flavored inputs (the 25 registers from the documented benchmark) with the PR's skill and score the outputs with pinned official Binoculars (TinyLlama pair) against the raw inputs as a same-run baseline..github/benchmark/baseline.jsonand are provisional until a few CI runs calibrate the local executor.Test Plan
Enforce baselinestep reflects the comment's verdictbaseline.jsonthresholds from this first run if neededCaveats: the CI executor is a small local model, so absolute scores are not comparable to agentic/frontier runs; the gate measures the skill text's effect under a fixed executor. Binoculars is a perplexity-class detector; the learned-classifier ceiling documented in the README still applies. First-run duration is dominated by CPU inference (~1-2h); the HF model cache warms after the first run.
🤖 Generated with Claude Code