Repository navigation
feat(lh-137801): add ai judge for failed harness wording checks - #45
Merged
Merged
Conversation
Nightly skill evaluations reported failures on response_concepts checks even when the agent behaved correctly, because the answer phrased a required concept in words no keyword group anticipated. The observed cases are now accepted by broader keyword groups, and an opt-in AI judge can re-check any remaining wording failure during a run or against an existing results report. The judge is a Bedrock model selected with --judge-model. It re-reads only failed response_concepts assertions of gate or quality severity in agent-fail samples and grades them against a plain-language intent that every such assertion now declares. Safety, operation, secret, confirmation, and harness checks are never judged. The evaluated response is escaped and treated as data, the verdict must arrive through a record_verdict tool call, and provider errors or malformed replies keep the original failure. A preflight asks the model for one real ruling so an incompatible model stops the run before any fixture is judged. Reports keep the keyword-check message next to the judge model and rationale, count judge overrides per fixture and overall, and mark judge-accepted samples in the Markdown report and dashboard. The judge model joins the comparison fingerprint so judged and unjudged runs never share a baseline. Claude runs that authenticate to Bedrock through AWS_PROFILE now resolve the profile in the parent process and pass short-lived keys, because the disposable home hides ~/.aws and the session previously hung during the credential-isolation preflight.
huides00
requested review from
Scoombe,
afercal and
siddhuwarrier
as code owners
November 9, 2026 09:33
afercal
approved these changes
Nov 9, 2026
Review of the judge found four gaps. AWS keys resolved from a named profile for Claude existed only in the launch environment, so report redaction never saw them; redaction now also replaces every value the harness resolved itself. Rejudging an older report checked eligibility against the current fixture alone, so a reused assertion id could expose a recorded safety failure to the judge; the recorded result must now be a judgeable wording check too. The judge subcommand now refuses a report already judged by a different model instead of mixing rulings under one fingerprint, and judge-accepted counts and dashboard labels include only samples that finally passed.
The feature commit on this branch makes the next release a minor version, and release preparation requires the target version to be recorded in source. Add the 0.44.0 entry covering the optional AI judge, the accepted Sonnet 5.5 wording, and the Claude AWS profile startup fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://cisco-sbg.atlassian.net/browse/LH-137801
Description
Nightly skill evaluations were reporting failures on answer-wording checks even when the agent behaved correctly, because the answer expressed a required idea in words the check did not anticipate, such as "on your own machine" instead of "locally". The observed cases are now accepted by broader wording rules, and an optional AI judge can take a second look at any remaining wording failure, either during a run or on an existing report, without rerunning the agent.
The judge only reconsiders failed wording checks and grades them against a plain-language statement of what each check requires. Safety, executed-operation, secret, confirmation, and harness-integrity checks are never judged, so the judge can correct a wording false negative but can never excuse unsafe behavior. If the judge errors or replies in an unexpected form, the original failure stands, and a startup check stops the run when the chosen model cannot give a valid verdict. Reports and the dashboard show which samples passed only because the judge accepted them, together with the judge's reasoning, and judged runs are never compared against baselines recorded without a judge.
Claude-based local runs that sign in to Bedrock through a named AWS profile no longer hang at startup. Existing baselines will need to be regenerated once, because the evaluation fixtures changed.
Reviewed with better-call-sol: 2 rounds with gpt-6.1-sol on high. Sol verdict: approved