Skip to content

CI: skill benchmark gate (localaik executor + Binoculars) with PR comment#8

Closed
harshaneel wants to merge 4 commits into
mainfrom
feat/benchmark-baseline-gate
Closed

CI: skill benchmark gate (localaik executor + Binoculars) with PR comment#8
harshaneel wants to merge 4 commits into
mainfrom
feat/benchmark-baseline-gate

Conversation

@harshaneel

Copy link
Copy Markdown
Owner

Summary

  • New skill-benchmark GitHub workflow: on any PR touching humanize/SKILL.md (or the benchmark harness itself), humanize 25 fixed AI-flavored inputs (the 25 registers from the documented benchmark) with the PR's skill and score the outputs with pinned official Binoculars (TinyLlama pair) against the raw inputs as a same-run baseline.
  • Pass gate enforced as a check: mean lift over the raw-input baseline >= +0.04 and at most 2 outputs scoring below their own raw input. Thresholds live in .github/benchmark/baseline.json and are provisional until a few CI runs calibrate the local executor.
  • Results are posted as a marker-keyed PR comment (updated in place on re-runs, paginated lookup), with per-register regressions expanded when the tail check trips. Powered by localaik for local testing: the executor is a localaik container (Gemma 3 4B behind an OpenAI-compatible endpoint), so the workflow needs no API keys and fork PRs can run it too (they skip only the comment step, where GITHUB_TOKEN is read-only).
  • Executor script checkpoints after every input and resumes on rerun; dependencies and the Binoculars source are SHA/version-pinned.

Test Plan

  • Workflow triggers on this PR (paths filter includes the workflow and harness files)
  • localaik container becomes healthy and all 25 humanizations complete
  • PR comment appears with the PASS/FAIL header, baseline table, and localaik credit
  • Enforce baseline step reflects the comment's verdict
  • Calibrate baseline.json thresholds from this first run if needed

Caveats: the CI executor is a small local model, so absolute scores are not comparable to agentic/frontier runs; the gate measures the skill text's effect under a fixed executor. Binoculars is a perplexity-class detector; the learned-classifier ceiling documented in the README still applies. First-run duration is dominated by CPU inference (~1-2h); the HF model cache warms after the first run.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Humanize skill benchmark — PASS ✅

25 fixed AI-flavored inputs humanized with the PR's humanize/SKILL.md (executor: Google Gemini (gemini-flash-lite-latest, free tier, one-shot); resolved model: gemini-flash-lite-latest) and scored against the raw inputs as baseline.

Scoring model: the official Binoculars zero-shot scorer (Hans et al., ICML 2024) running the TinyLlama-1.1B base + TinyLlama-1.1B-Chat model pair.

How to read: each score estimates how human the text reads (higher = more human). A humanized output should score above the raw AI input it came from; the gate checks the average lift and how many outputs fell below their own input.

Metric raw inputs (baseline) humanized (PR skill)
Mean 0.8683 0.9455
Median 0.8560 0.9291
Min 0.7767 0.8637
Max 0.9552 1.0525
Check Threshold Actual
Mean lift over baseline ≥ +0.050 +0.0772
Outputs below their own raw input ≤ 4 0
Per-input inspection (25 items)
id 1 · Tech blog post · raw 0.874 → humanized 0.906 ✅ · 464 words · 4s

Original (AI-flavored input):

When we set out to rebuild our authentication service, it is important to note that we didn't take the decision lightly; the existing system, built in 2019, had served us reasonably well, but it was starting to show its age. We decided to leverage a more robust, comprehensive framework that would streamline the login flow for our 2.3 million monthly users. Furthermore, the migration involved three pivotal phases: auditing the legacy codebase, designing the new schema, and rolling out changes incrementally to avoid downtime. Moreover, we delved into the tradeoffs between speed, security, and ma … (truncated; full text in the run artifact)

Humanized (PR skill):

We rebuilt our authentication service because the 2019 version hit a wall. It worked for years, but it couldn't keep up with our 2.3 million monthly users. We chose a modern framework to replace it, aiming to simplify how people log in.

The migration had three phases. We audited the old codebase, designed a new schema, and rolled out changes in small chunks to keep the site online. We spent weeks debating the trade-offs between speed, security, and long-term maintenance.

The new system, Aegis, dropped average login latency from 420ms to 180ms. Our platform, security, and infrastructure team … (truncated; full text in the run artifact)

id 2 · Engineering postmortem · raw 0.940 → humanized 1.018 ✅ · 486 words · 4s

Original (AI-flavored input):

On March 14th, at approximately 2:47 AM UTC, our payments service experienced a significant outage that impacted roughly 18% of transactions for a duration of 47 minutes; it is important to note that this was not a singular failure but rather the result of three compounding issues. Furthermore, a routine deployment to our database cluster, an unexpected spike in traffic from a marketing campaign, and a misconfigured rate limiter combined to create a perfect storm. Moreover, our on-call engineer, working with limited visibility, had to delve into logs across five separate services before identi … (truncated; full text in the run artifact)

Humanized (PR skill):

On March 14th at 2:47 AM UTC, our payments service went down. For 47 minutes, 18% of transactions failed. This wasn't one thing breaking; it was three issues piling up at once.

We pushed a routine database update, a marketing campaign spiked traffic, and a rate limiter was misconfigured. The combination took us out. Our on-call engineer spent that time hunting through logs across five different services just to find the culprit. We used the incident response playbook to get things back up, but the process showed us how thin our monitoring really is.

About 12,000 customers couldn't check ou … (truncated; full text in the run artifact)

id 3 · Product launch announcement · raw 0.826 → humanized 0.953 ✅ · 179 words · 2s

Original (AI-flavored input):

Today, we are thrilled to announce the launch of Pulse 3.0, a comprehensive reimagining of our flagship analytics platform that has been two years in the making. It is important to note that this release represents far more than an incremental update; rather, it is a pivotal step forward for the 40,000 businesses that rely on us daily. Furthermore, Pulse 3.0 leverages a robust new data engine capable of processing 10 million events per second, a threefold improvement over our previous architecture. Moreover, we've delved deeply into customer feedback, streamlining the onboarding experience, th … (truncated; full text in the run artifact)

Humanized (PR skill):

Pulse 3.0 is live. We spent the last two years rebuilding our analytics platform from the ground up, and today it’s ready for the 40,000 businesses using it daily.

This isn't a minor patch. We swapped out the engine to handle 10 million events per second—three times the throughput of our old architecture. We also spent months listening to your feedback, which meant tearing down the dashboard, the reporting flow, and the onboarding process to make them faster.

Whether you’re a solo founder or running a large team, the new Pulse is built to match your scale. Prices start at $49 a month, with c … (truncated; full text in the run artifact)

id 4 · Academic abstract · raw 0.929 → humanized 1.017 ✅ · 392 words · 3s

Original (AI-flavored input):

This study seeks to delve into the pivotal role of microbial diversity in soil carbon sequestration, an area that, despite considerable research, remains incompletely understood. It is important to note that prior work has often relied on limited sample sizes; our comprehensive analysis, by contrast, draws on 1,240 soil samples collected across 14 distinct biomes between 2021 and 2023. Furthermore, we leverage a robust statistical model to quantify the relationship between bacterial community composition, fungal biomass, and long-term carbon retention. Moreover, our findings suggest that a str … (truncated; full text in the run artifact)

Humanized (PR skill):

Microbial diversity is central to soil carbon sequestration, yet how exactly it functions remains a puzzle. Most past studies relied on small data sets. We analyzed 1,240 soil samples from 14 biomes gathered between 2021 and 2023 to move past those limitations. Using a new statistical model, we mapped the link between bacterial community structure, fungal biomass, and how long carbon stays in the ground.

Our data shows that a simpler system for classifying soil microbiomes can sharpen the accuracy of climate models. Three patterns emerged. Higher fungal-to-bacterial ratios track with more sta … (truncated; full text in the run artifact)

id 5 · Business email · raw 0.848 → humanized 0.910 ✅ · 98 words · 1s

Original (AI-flavored input):

Hi Sarah, I hope this email finds you well. I wanted to follow up regarding our conversation last Tuesday about the Q3 vendor contract renewal; it is important to note that the deadline is fast approaching, with the current agreement expiring on August 15th. Furthermore, I've delved into the proposed terms and believe we have an opportunity to leverage our five-year relationship with Meridian Supply to negotiate a more favorable rate. Moreover, their comprehensive service package now includes expedited shipping, dedicated account management, and quarterly performance reviews, which could strea … (truncated; full text in the run artifact)

Humanized (PR skill):

Sarah,

Following up on our talk last Tuesday about the Q3 contract with Meridian Supply. The current agreement ends August 15th, so we need to move quickly.

I reviewed the proposed terms. We have a five-year history with them, and we should use that to push for a better rate. Their new package includes dedicated account management and expedited shipping. These additions could simplify our procurement work.

We spend about $340,000 a year with Meridian. A 5% cut would make a real difference to our budget.

Are you free Thursday afternoon to talk through our negotiation strategy?

James

id 6 · Internal Slack update · raw 0.907 → humanized 1.030 ✅ · 248 words · 2s

Original (AI-flavored input):

Hey team, quick update on the migration project as we head into the weekend; it is important to note that we've made significant progress since our last sync on Monday. Furthermore, the backend team has successfully delved into the legacy API endpoints and migrated 32 of the 47 total routes to our new, more robust infrastructure. Moreover, we've leveraged the comprehensive test suite that QA put together last sprint, which has helped us catch a handful of edge cases early. There are three things I want to flag: first, the staging environment is currently stable; second, we're still streamlinin … (truncated; full text in the run artifact)

Humanized (PR skill):

Migration update for the weekend: we're moving fast. Since Monday's sync, the backend team finished 32 of 47 legacy API routes. We swapped them over to the new infrastructure, and the test suite QA built last sprint caught a few messy edge cases before they hit staging.

Three quick notes for everyone:

  1. Staging is stable.
  2. The auth layer is still a work in progress—expect 2–3 more days there.
  3. We need all hands on deck for testing Monday morning.

This migration changes how we handle data across services, so Monday's testing is the real work. Great job getting us this far. Reach out if … (truncated; full text in the run artifact)

id 7 · LinkedIn post · raw 0.861 → humanized 0.904 ✅ · 333 words · 3s

Original (AI-flavored input):

I'm thrilled to share that I've officially stepped into a new role as Director of Product at Northwind Analytics, a company I've admired for quite some time. It is important to note that this journey hasn't been linear; over the past eight years, I've delved into roles spanning engineering, design, and go-to-market strategy, each one teaching me something pivotal. Furthermore, I've learned to leverage cross-functional collaboration, robust data-driven decision-making, and comprehensive stakeholder alignment as the true pillars of great product work. Moreover, I want to thank my incredible ment … (truncated; full text in the run artifact)

Humanized (PR skill):

I’m excited to share that I’ve started as Director of Product at Northwind Analytics. I have admired their work for a long time.

My path here wasn't linear. Over the last eight years, I worked in engineering, design, and go-to-market strategy. Every stop taught me something new. I found that product success depends on three things: cross-functional collaboration, data-driven decisions, and clear alignment with stakeholders.

I owe a lot to my mentors, my family, and the team at Bright Path Solutions. I spent four years there building something I’m proud of.

If you're just starting out, my a … (truncated; full text in the run artifact)

id 8 · Cover letter · raw 0.812 → humanized 0.867 ✅ · 123 words · 1s

Original (AI-flavored input):

Dear Hiring Manager, I am writing to express my sincere interest in the Senior Software Engineer position at Clearwater Systems, as advertised on your careers page. It is important to note that my background spans five years of experience across backend development, cloud infrastructure, and team leadership, and I believe this comprehensive skill set aligns well with your needs. Furthermore, in my current role at Delacroix Technologies, I have leveraged robust engineering practices to streamline our deployment pipeline, reducing release time from three days to just six hours. Moreover, I delve … (truncated; full text in the run artifact)

Humanized (PR skill):

I am writing to apply for the Senior Software Engineer role at Clearwater Systems. With five years of experience in backend development, cloud infrastructure, and team leadership, I have the technical background to hit the ground running.

At Delacroix Technologies, I overhauled our deployment pipeline. I cut release times from three days down to six hours. I also untangled our microservices architecture, resolving three specific bottlenecks that had been dragging down system performance.

Clearwater’s mission to make financial tools accessible and equitable resonates with me. I enjoy working … (truncated; full text in the run artifact)

id 9 · Marketing copy · raw 0.856 → humanized 1.007 ✅ · 138 words · 2s

Original (AI-flavored input):

Introducing BrewMind, the coffee maker that doesn't just brew coffee, it delves into the science of flavor extraction to deliver a truly exceptional cup, every single time. It is important to note that BrewMind isn't just another appliance; it's a comprehensive, robust solution engineered to streamline your entire morning routine. Furthermore, with three customizable brew modes, a 12-cup capacity, and a sleek stainless-steel finish, BrewMind fits seamlessly into any kitchen, any lifestyle, any budget. Moreover, our proprietary Precision Bloom technology leverages a four-stage saturation proces … (truncated; full text in the run artifact)

Humanized (PR skill):

BrewMind changes how you make coffee. It uses a four-stage saturation process called Precision Bloom to pull flavors most machines miss. You get three brew modes and a 12-cup capacity inside a stainless-steel frame. It fits your counter, your schedule, and your $90 budget.

We launched this last year. Since then, 50,000 people have bought one, leaving 3,200 reviews that average 4.8 stars. If you care about your first cup, this is the upgrade you want. Stop drinking mediocre coffee. Start your morning better.

em dashes: 0
semicolons: 0
curly quotes/apostrophes: 0
banned vocabulary: 0
comparati … (truncated; full text in the run artifact)

id 10 · Press release · raw 0.848 → humanized 0.867 ✅ · 124 words · 2s

Original (AI-flavored input):

FOR IMMEDIATE RELEASE. San Francisco, CA, June 3, 2026. Meridian Health Technologies today announced the launch of its comprehensive telehealth platform, CarePath, designed to streamline access to care for patients across underserved rural communities. It is important to note that this launch follows an 18-month development period and a successful pilot program involving 22 clinics across four states. Furthermore, CarePath leverages a robust, HIPAA-compliant infrastructure to connect patients with over 1,500 licensed providers nationwide. Moreover, company leadership delved into extensive user … (truncated; full text in the run artifact)

Humanized (PR skill):

Meridian Health Technologies just launched CarePath, a telehealth platform built to connect people in rural communities with doctors. The project grew out of 18 months of work and a pilot program across 22 clinics in four states.

CarePath uses HIPAA-compliant tech to link patients with 1,500 licensed providers nationwide. Before the rollout, the team gathered feedback from 6,000 patients in the beta phase.

"This launch is a big step for healthcare access," said CEO Danielle Ford. "CarePath will change how rural patients get care."

The platform goes live July 15th. Plans start at $29 a month … (truncated; full text in the run artifact)

id 11 · Investor update · raw 0.802 → humanized 0.896 ✅ · 156 words · 2s

Original (AI-flavored input):

Dear investors, I hope this update finds you well as we close out Q2. It is important to note that this quarter has been a pivotal one for the company, marked by significant, measurable progress across the board. Furthermore, revenue grew to $1.4 million, a 22% increase quarter-over-quarter, while our customer base expanded to 3,800 active accounts. Moreover, we've delved deeply into our unit economics, and I'm pleased to report that our gross margin has improved to 68%, up from 61% last quarter. We leveraged a more robust go-to-market strategy this quarter, streamlining our sales process and … (truncated; full text in the run artifact)

Humanized (PR skill):

Q2 is behind us, and we have some momentum to share. Revenue hit $1.4 million, up 22% since last quarter, and we now have 3,800 active accounts.

We spent time auditing our unit economics. Gross margins climbed to 68% from 61%. We also tightened our sales process, which cut our average deal cycle from 45 days down to 31.

We closed an $8 million Series A led by Fenway Ventures, with three of our existing investors joining the round.

Hiring is the one area where we missed our targets. We aimed for nine new engineers but only brought on four. For the next few months, we are focusing on three th … (truncated; full text in the run artifact)

id 12 · Job posting · raw 0.812 → humanized 0.909 ✅ · 482 words · 3s

Original (AI-flavored input):

Northgate Robotics is seeking a Senior Backend Engineer to join our growing team of 45 employees in Austin, Texas. It is important to note that this role offers a unique opportunity to delve into cutting-edge robotics infrastructure, working alongside a comprehensive team of engineers, designers, and product managers. Furthermore, the ideal candidate will leverage at least five years of experience with distributed systems, robust API design, and cloud infrastructure, ideally with AWS or GCP. Moreover, you'll be responsible for streamlining our data pipelines, which currently process over 2 mil … (truncated; full text in the run artifact)

Humanized (PR skill):

Northgate Robotics is hiring a Senior Backend Engineer for our 45-person team in Austin. You will build infrastructure for our autonomous fleet, working directly with our hardware and product groups.

We need someone with five years of experience in distributed systems and cloud architecture, specifically on AWS or GCP. You will own our data pipelines, which handle 2 million sensor readings every day. This role is central to our plans as we move toward our next funding round.

Your day-to-day will involve designing backend systems, mentoring junior engineers, and syncing with our hardware team … (truncated; full text in the run artifact)

id 13 · Customer support reply · raw 0.777 → humanized 0.864 ✅ · 116 words · 1s

Original (AI-flavored input):

Hi Marcus, thank you for reaching out, and I sincerely apologize for the inconvenience you've experienced with your recent order, #48213. It is important to note that our team has thoroughly delved into your account history to understand exactly what went wrong. Furthermore, it appears that a warehouse processing error, a shipping label mismatch, and a delayed carrier pickup combined to cause the three-day delay you encountered. Moreover, I want to assure you that we take this matter seriously and are leveraging our internal quality assurance process to prevent similar issues going forward. As … (truncated; full text in the run artifact)

Humanized (PR skill):

Hi Marcus, I am sorry your order #48213 was delayed. Our team looked into your account, and it turns out three separate issues hit at once: a processing error at the warehouse, a shipping label mismatch, and a late carrier pickup. That is why your package sat for three days.

We are updating our internal quality checks to make sure this does not happen again. I have issued a full refund of $34.99 to your original payment method. You will also see a 20% discount code in your inbox for your next order. Our shipping partner has updated their tracking system to improve reliability for future deliv … (truncated; full text in the run artifact)

id 14 · Recipe intro · raw 0.838 → humanized 0.919 ✅ · 483 words · 3s

Original (AI-flavored input):

There's something truly special about a comprehensive, home-cooked lasagna, and it is important to note that this particular recipe has been in my family for over 30 years. Furthermore, I've delved into countless variations over the years, tweaking the sauce, the cheese blend, and the pasta layers until I finally landed on what I consider the perfect, robust balance of flavors. Moreover, this recipe leverages a rich, slow-simmered marinara made with San Marzano tomatoes, fresh basil, and a touch of red wine, which streamlines the cooking process without sacrificing depth of flavor. Whether you … (truncated; full text in the run artifact)

Humanized (PR skill):

A home-cooked lasagna is special. My family has used this specific recipe for over 30 years. I have tried countless variations, adjusting the sauce, the cheese blend, and the pasta layers until I found the right balance.

The sauce is a slow-simmered marinara made with San Marzano tomatoes, fresh basil, and a splash of red wine. It stays deep and flavorful without needing hours of extra work. This dish feeds eight people and tastes even better the next day.

Total time is 90 minutes, with 45 minutes spent in the oven. I first made this for my grandmother’s 80th birthday in 2014. It has been … (truncated; full text in the run artifact)

id 15 · Travel writing · raw 0.849 → humanized 0.923 ✅ · 488 words · 3s

Original (AI-flavored input):

Lisbon is, without question, a city that rewards those willing to delve into its winding streets, historic trams, and vibrant, ever-evolving neighborhoods. It is important to note that the city's charm lies not in any single landmark but in the comprehensive experience of wandering through Alfama, Bairro Alto, and Belem, each offering something pivotal to the overall journey. Furthermore, I spent five days exploring the city in April of 2025, and I leveraged a combination of walking tours, local recommendations, and spontaneous detours to get a truly robust sense of the place. Moreover, the pa … (truncated; full text in the run artifact)

Humanized (PR skill):

Lisbon rewards anyone willing to get lost in its tangled streets, rattling trams, and neighborhoods that never seem to stand still. The city's charm isn't hidden in a single landmark. It lives in the act of wandering through Alfama, Bairro Alto, and Belem. Each district adds a different layer to the trip.

I spent five days there last April. I skipped the pre-planned itineraries, relying instead on local tips and the occasional wrong turn. It gave me a better feel for the place than any guidebook could. I also stopped at Pasteis de Belem. They have been baking since 1837, and their pasteis de … (truncated; full text in the run artifact)

id 16 · Restaurant review · raw 0.854 → humanized 0.917 ✅ · 502 words · 3s

Original (AI-flavored input):

Tucked away on a quiet corner in the Mission District, Casa Verde has quickly become, in my opinion, a pivotal addition to San Francisco's ever-evolving culinary scene. It is important to note that the restaurant, which opened in November 2024, leverages a comprehensive, farm-to-table philosophy that shows in every dish. Furthermore, I delved into the tasting menu, priced at $95 per person, which included three standout courses: a robust roasted beet salad, a delicately seared branzino, and a rich, comprehensive short rib finished with a red wine reduction. Moreover, the service was attentive … (truncated; full text in the run artifact)

Humanized (PR skill):

Casa Verde sits on a quiet corner in the Mission. It opened in November 2024 and already feels like a permanent fixture of the neighborhood. The kitchen works with a farm-to-table philosophy that shows in every plate.

I tried the $95 tasting menu. The roasted beet salad had earthiness to spare, and the branzino arrived with a perfect, delicate sear. Then came the short rib—deep, dark, and finished with a red wine reduction.

The service stayed attentive but stayed out of the way. The wine list felt intentional, and the room balanced rustic textures with a polished finish. They only seat 40 pe … (truncated; full text in the run artifact)

id 17 · Book review · raw 0.860 → humanized 0.963 ✅ · 317 words · 3s

Original (AI-flavored input):

In her latest novel, The Weight of Silence, author Clara Bennington delves into themes of grief, memory, and reconciliation with a comprehensive, deeply affecting narrative structure. It is important to note that the novel, published in March 2025 by Harrow & Finch Press, spans nearly 420 pages, yet it rarely feels overlong. Furthermore, Bennington leverages a robust, multi-generational storyline to explore three pivotal relationships: a mother and daughter, two estranged sisters, and a widow confronting her late husband's secrets. Moreover, the prose is streamlined without sacrificing emotion … (truncated; full text in the run artifact)

Humanized (PR skill):

Clara Bennington’s latest, The Weight of Silence, maps the quiet wreckage of grief, memory, and reconciliation. Published by Harrow & Finch this March, the book runs 420 pages but never drags. Bennington tracks three core relationships across generations: a mother and daughter, two sisters who haven't spoken in years, and a widow unearthing her late husband’s hidden life.

The prose stays lean. It toggles between past and present without losing its grip on the reader. This is Bennington’s fourth book, landing six years after her 2019 debut, and it is her most ambitious. The middle section sl … (truncated; full text in the run artifact)

id 18 · Personal essay · raw 0.955 → humanized 0.975 ✅ · 474 words · 3s

Original (AI-flavored input):

It is important to note that grief doesn't arrive on any predictable schedule; it simply shows up, uninvited, and forces you to delve into feelings you didn't know you were avoiding. Furthermore, when my father passed away in the fall of 2021, I found myself leveraging routines, small comprehensive rituals like morning coffee and evening walks, just to get through each day. Moreover, I spent countless hours delving into old photographs, letters, and voicemails, searching for some robust sense of closure that never quite arrived. There were three things that helped me most during that period: t … (truncated; full text in the run artifact)

Humanized (PR skill):

Grief doesn't keep a calendar. It shows up uninvited, forcing you into rooms you didn't know you were avoiding. When my father died in the fall of 2021, I clung to small habits—morning coffee, evening walks—just to keep the floor beneath me.

I spent hours with old photos, letters, and saved voicemails. I was looking for a final, tidy sense of closure that never came. Three things kept me upright: therapy, time with friends, and, strangely, cooking the recipes my father loved.

Healing isn't a straight line. Some days felt like a breakthrough, while others felt like standing perfectly still. … (truncated; full text in the run artifact)

id 19 · Privacy policy section · raw 0.936 → humanized 0.970 ✅ · 143 words · 2s

Original (AI-flavored input):

This section is intended to provide a comprehensive overview of how we collect, use, and store your personal information; it is important to note that we take your privacy seriously and have implemented robust safeguards accordingly. Furthermore, we may collect data including your name, email address, IP address, and browsing behavior in order to streamline your overall experience on our platform. Moreover, we leverage this information for three primary purposes: improving our services, personalizing your experience, and ensuring compliance with applicable legal obligations. It is also worth n … (truncated; full text in the run artifact)

Humanized (PR skill):

We collect, use, and store your personal information to keep your data safe and our services running. This includes your name, email, IP address, and how you browse our site. We use this data to improve our services, tailor your experience, and meet our legal obligations.

We hold onto your personal data for 24 months after you close your account, unless the law forces us to keep it longer. We share some data with specific third-party vendors, such as our analytics provider, Insight Metrics Inc. We never sell your personal information to advertisers.

Last updated: May 1, 2026. If you have que … (truncated; full text in the run artifact)

id 20 · Tutorial intro · raw 0.858 → humanized 1.052 ✅ · 125 words · 1s

Original (AI-flavored input):

In this tutorial, we're going to delve into the process of building a comprehensive REST API using Node.js and Express, a pivotal skill for any aspiring backend developer. It is important to note that by the end of this guide, you'll have a fully functional, robust API capable of handling three core operations: creating, reading, and updating records in a PostgreSQL database. Furthermore, we'll leverage popular libraries such as Sequelize and dotenv to streamline our configuration and database management. Moreover, this tutorial assumes you have Node.js version 18 or higher installed, along wi … (truncated; full text in the run artifact)

Humanized (PR skill):

Building a REST API with Node.js and Express is a standard rite of passage for backend developers. Over the next two hours, we will build an API that talks to a PostgreSQL database to create, read, and update records.

We will use Sequelize for database modeling and dotenv for environment variables. You need Node.js version 18 or higher and a solid grasp of JavaScript to follow along.

We start by initializing the project, then set up the middleware stack, and finish with error handling and validation. Open your terminal and run npm init to begin.

em dashes: 0
semicolons: 0
curly quotes/apo … (truncated; full text in the run artifact)

id 21 · Comparison article · raw 0.954 → humanized 0.970 ✅ · 506 words · 3s

Original (AI-flavored input):

When it comes to choosing between Notion and Obsidian, it is important to note that both platforms offer a comprehensive suite of features, though they leverage fundamentally different underlying philosophies. Furthermore, Notion, which reported over 30 million users as of 2023, provides a robust, all-in-one workspace that streamlines documents, databases, and collaboration into a single, unified platform. Moreover, Obsidian, by contrast, delves into a more pivotal, file-based approach, storing everything locally in markdown files, which many privacy-conscious users find appealing. There are t … (truncated; full text in the run artifact)

Humanized (PR skill):

Choosing between Notion and Obsidian means deciding how you want your brain to live online. They look like similar tools, but they work from opposite ends of the spectrum.

Notion is the digital equivalent of a communal office. It is an all-in-one workspace where you dump documents, databases, and project boards. It works for teams because everything lives in the cloud, making real-time collaboration seamless. As of 2023, about 30 million people rely on it to keep their work in one place.

Obsidian is different. It is a local file cabinet for your thoughts. You own your data because every note … (truncated; full text in the run artifact)

id 22 · Roadmap update · raw 0.804 → humanized 0.872 ✅ · 128 words · 3s

Original (AI-flavored input):

As we look ahead to the second half of 2026, it is important to note that our product roadmap reflects a comprehensive, pivotal shift in priorities based on extensive customer feedback. Furthermore, we've delved into over 400 support tickets and 60 customer interviews to identify the three most requested features: real-time collaboration, a robust mobile app, and streamlined third-party integrations. Moreover, we plan to leverage our existing infrastructure to ship the mobile app by Q3, with a public beta launching in September. It is also worth noting that real-time collaboration, a genuinely … (truncated; full text in the run artifact)

Humanized (PR skill):

For the second half of 2026, our product roadmap is changing to match what customers actually asked for. We reviewed 400 support tickets and 60 interviews. The top requests are real-time collaboration, a mobile app, and better third-party integrations.

We can build the mobile app on our current stack. Expect a public beta in September.

Real-time collaboration is the big one for enterprise users. It needs a full rebuild of the data layer, which will take the team about 12 weeks.

Our integration marketplace currently has 18 tools. We aim to hit 35 by the end of the year. We are also keeping o … (truncated; full text in the run artifact)

id 23 · Conference abstract · raw 0.920 → humanized 1.000 ✅ · 515 words · 3s

Original (AI-flavored input):

This talk will delve into the pivotal challenges of scaling distributed systems, drawing on comprehensive, real-world case studies from three separate production environments. It is important to note that as organizations increasingly leverage microservices architectures, the need for robust, fault-tolerant infrastructure becomes ever more pressing. Furthermore, this session will explore how our team, over the course of 18 months, streamlined our deployment pipeline to handle over 500 million requests per day. Moreover, attendees will gain a comprehensive understanding of three key strategies: … (truncated; full text in the run artifact)

Humanized (PR skill):

Scaling distributed systems is hard. I’ll share what we learned from three production environments where we hit the limits of our infrastructure.

As teams move toward microservices, fault tolerance stops being an optional feature and becomes the baseline requirement. My team spent 18 months re-engineering our deployment pipeline to handle 500 million requests every day. It was a messy process.

In this session, we will look at how to build better circuit breakers, pick the right observability tools, and fix incident response protocols that actually work. This talk expands on the research we s … (truncated; full text in the run artifact)

id 24 · README intro · raw 0.842 → humanized 0.929 ✅ · 179 words · 2s

Original (AI-flavored input):

Welcome to Streamline, a comprehensive, open-source toolkit designed to help developers delve into complex data pipelines with minimal friction and maximum robustness. It is important to note that this project was first launched in January 2023 and has since grown to over 4,200 stars on GitHub, with contributions from more than 85 developers worldwide. Furthermore, Streamline leverages a robust, modular architecture that allows you to plug in custom transformations, validators, and connectors without needing to modify the comprehensive core codebase. Moreover, this README will guide you throug … (truncated; full text in the run artifact)

Humanized (PR skill):

Streamline is an open-source toolkit built to help you manage data pipelines. We started the project in January 2023. Since then, over 85 developers have contributed to the codebase, and it has earned 4,200 stars on GitHub.

The architecture is modular. You can plug in custom transformations, validators, and connectors without touching the core code.

To get started, install the package:

pip install streamline-toolkit

You will need Python 3.9 or higher and at least 4GB of RAM for standard tasks. Once installed, you can configure your first pipeline in minutes. If you want to help out, check … (truncated; full text in the run artifact)

id 25 · Career advice · raw 0.948 → humanized 0.997 ✅ · 353 words · 3s

Original (AI-flavored input):

It is important to note that navigating a career transition, especially later in life, requires you to delve into your priorities with real, comprehensive honesty. Furthermore, I made the pivotal decision to leave my robust, comfortable job at a Fortune 500 company in 2020 to pursue something more meaningful, and it wasn't easy. Moreover, I leveraged my network, my savings, and roughly 18 months of careful planning to make the transition as smooth as possible. There are three pieces of advice I'd offer anyone considering a similar leap: build a comprehensive financial cushion before you leap, … (truncated; full text in the run artifact)

Humanized (PR skill):

Changing careers later in life demands a brutal kind of honesty with yourself. In 2020, I quit my comfortable role at a Fortune 500 company to find work that actually mattered. It was a hard shift. I spent eighteen months burning through savings and leaning on my network before I handed in my resignation.

If you are planning a jump, do these three things. Build a real financial cushion first. Don't start from scratch—use the people you already know. Finally, stop spraying resumes everywhere. I sent exactly twelve applications during my search. I ignored the career coaches who told me to send … (truncated; full text in the run artifact)

On PRs touching humanize/SKILL.md: humanize 25 fixed AI-flavored inputs
(the documented benchmark registers) with the PR's skill, score outputs
and raw inputs with pinned official Binoculars (TinyLlama pair), and
enforce a baseline gate: mean lift over the raw inputs and a cap on
outputs scoring below their own raw input (thresholds in
.github/benchmark/baseline.json). Results are upserted as a marker-keyed
PR comment with per-input collapsible inspection (original vs humanized
text, score movement, timing), auto-fitted to GitHub's comment limit.

Executor: Gemini free tier (gemini-flash-lite-latest rolling alias) via
the OpenAI-compatible endpoint. The GEMINI_API_KEY secret is scoped to
the `benchmark` environment so it is only usable by approved runs of
this workflow. Fork PRs run under pull_request_target: workflow and all
scripts execute from the base branch and only the fork's SKILL.md is
fetched, as prompt data — fork code never executes next to the secret.
Same-repo PRs use the plain pull_request trigger; conditions are
mutually exclusive.

Resilience: humanize and score are separate jobs with an artifact
handoff so "Re-run failed jobs" repeats only the failed half; the
executor checkpoints per item and re-run attempts seed from the prior
attempt's partial artifact; patient exponential backoff (20s-300s, 6
attempts) absorbs free-tier 429/503 throttling; deps and the Binoculars
source are version/SHA-pinned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@harshaneel harshaneel force-pushed the feat/benchmark-baseline-gate branch from d381ef3 to fe44bb5 Compare July 10, 2026 01:11
…thresholds

- Per-input inspection now nests all item dropdowns inside a single
  outer dropdown so the comment stays compact
- One-line "how to read" note before the score tables; caveats footer
  removed
- Thresholds calibrated to the measured Gemini flash-lite run (+0.076
  lift, 2/25 below raw): gate now requires mean lift >= +0.05 and at
  most 4 outputs below their own raw input

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The comment now states explicitly that scores come from the official
Binoculars detector (Hans et al., ICML 2024) with the TinyLlama-1.1B
base + chat model pair.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Comment wording: "scoring model"/"scorer" instead of detector-style
  language
- The executor records the concrete model version each response resolved
  to (rolling aliases hide it), and the comment now shows the distinct
  resolved names next to the executor label

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@harshaneel

Copy link
Copy Markdown
Owner Author

Superseded by #9 — same change squashed to one commit; this PR accumulated iteration noise.

@harshaneel harshaneel closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant