CI: skill benchmark gate (Gemini executor + Binoculars scorer) with PR comment#9
Conversation
On PRs touching humanize/SKILL.md: humanize 25 fixed AI-flavored inputs (the documented benchmark registers) with the PR's skill, score outputs and raw inputs with the pinned official Binoculars scorer (TinyLlama base+chat pair), and enforce a measured baseline gate: mean lift over the raw inputs >= +0.05 and at most 4 outputs scoring below their own raw input (calibrated to the Gemini flash-lite executor's measured +0.076 lift, 2/25 below raw). Results are upserted as a marker-keyed PR comment: verdict, score tables with a how-to-read note, the scoring model and the resolved executor model version, and a single collapsible per-input inspection section (original vs humanized text, score movement, timing), auto- fitted to GitHub's comment limit. Executor: Gemini free tier (gemini-flash-lite-latest rolling alias) via the OpenAI-compatible endpoint. The GEMINI_API_KEY secret is scoped to the `benchmark` environment so it is only usable by approved runs of this workflow. Fork PRs run under pull_request_target: workflow and all scripts execute from the base branch and only the fork's SKILL.md is fetched, as prompt data — fork code never executes next to the secret. Same-repo PRs use the plain pull_request trigger; conditions are mutually exclusive. Resilience: humanize and score are separate jobs with an artifact handoff so "Re-run failed jobs" repeats only the failed half; the executor checkpoints per item and re-run attempts seed from the prior attempt's partial artifact; patient exponential backoff (20s-300s) absorbs free-tier 429/503 throttling; deps and the Binoculars source are version/SHA-pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Humanize skill benchmark — PASS ✅25 fixed AI-flavored inputs humanized with the PR's Scoring model: the official Binoculars zero-shot scorer (Hans et al., ICML 2024) running the How to read: each score estimates how human the text reads (higher = more human). A humanized output should score above the raw AI input it came from; the gate checks the average lift and how many outputs fell below their own input.
Outputs that scored below their raw input
Per-input inspection (25 items)id 1 · Tech blog post · raw 0.874 → humanized 0.995 ✅ · 162 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 2 · Engineering postmortem · raw 0.940 → humanized 0.995 ✅ · 139 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 3 · Product launch announcement · raw 0.826 → humanized 0.857 ✅ · 139 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 4 · Academic abstract · raw 0.929 → humanized 1.006 ✅ · 440 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 5 · Business email · raw 0.848 → humanized 0.927 ✅ · 86 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 6 · Internal Slack update · raw 0.907 → humanized 0.931 ✅ · 175 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 7 · LinkedIn post · raw 0.861 → humanized 0.840 ❌ · 154 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 8 · Cover letter · raw 0.812 → humanized 0.954 ✅ · 440 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 9 · Marketing copy · raw 0.856 → humanized 0.950 ✅ · 370 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 10 · Press release · raw 0.848 → humanized 0.969 ✅ · 172 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 11 · Investor update · raw 0.802 → humanized 0.918 ✅ · 494 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 12 · Job posting · raw 0.812 → humanized 0.929 ✅ · 438 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 13 · Customer support reply · raw 0.777 → humanized 0.877 ✅ · 104 words · 1sOriginal (AI-flavored input):
Humanized (PR skill):
id 14 · Recipe intro · raw 0.838 → humanized 0.931 ✅ · 165 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 15 · Travel writing · raw 0.849 → humanized 0.916 ✅ · 231 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 16 · Restaurant review · raw 0.854 → humanized 0.863 ✅ · 541 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 17 · Book review · raw 0.860 → humanized 0.998 ✅ · 462 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 18 · Personal essay · raw 0.955 → humanized 0.984 ✅ · 369 words · 23sOriginal (AI-flavored input):
Humanized (PR skill):
id 19 · Privacy policy section · raw 0.936 → humanized 1.020 ✅ · 181 words · 22sOriginal (AI-flavored input):
Humanized (PR skill):
id 20 · Tutorial intro · raw 0.858 → humanized 0.895 ✅ · 108 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 21 · Comparison article · raw 0.954 → humanized 0.934 ❌ · 260 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 22 · Roadmap update · raw 0.804 → humanized 0.951 ✅ · 385 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 23 · Conference abstract · raw 0.920 → humanized 0.940 ✅ · 184 words · 2sOriginal (AI-flavored input):
Humanized (PR skill):
id 24 · README intro · raw 0.842 → humanized 0.998 ✅ · 447 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
id 25 · Career advice · raw 0.948 → humanized 0.975 ✅ · 530 words · 3sOriginal (AI-flavored input):
Humanized (PR skill):
|
Summary
skill-benchmarkworkflow: on any PR touchinghumanize/SKILL.md, humanize 25 fixed AI-flavored inputs (the documented benchmark registers) with the PR's skill and score everything with the pinned official Binoculars scorer (TinyLlama-1.1B base + chat pair) against the raw inputs as a same-run baseline..github/benchmark/baseline.json.gemini-flash-lite-latestrolling alias, OpenAI-compatible endpoint) — $0, minutes per run. TheGEMINI_API_KEYsecret is scoped to thebenchmarkenvironment, so it's only usable by approved runs of this workflow.pull_request_target, where the workflow and all scripts execute from the base branch and only the fork'sSKILL.mdis fetched, as prompt data — fork code never executes next to the secret. Same-repo PRs use the plainpull_requesttrigger; the conditions are mutually exclusive so nothing runs twice.Test Plan
Replaces #8 (same content, squashed; that PR accumulated iteration noise).
🤖 Generated with Claude Code