test(model-onboarding): probe the model as an LLM-as-judge guardrail - #1039
Open
denispetre wants to merge 2 commits into
Open
test(model-onboarding): probe the model as an LLM-as-judge guardrail#1039denispetre wants to merge 2 commits into
denispetre wants to merge 2 commits into
Conversation
Adds a third probe putting the model under test in the *judge* role: a ReAct agent runs behind a real UiPathLLMAsJudgeMiddleware guardrail whose judge model is the one being onboarded, and the guardrail decides whether the agent's answer broke a natural-language rule. Two probes, because one cannot tell a working judge from a broken one: a violating prompt the judge must block, and a compliant one it must let through. A judge that always blocks fails the second; one that never blocks fails the first. Sampled, not single-shot. The judge is a model call, so one sample is a coin toss on a borderline verdict — an earlier single-sample version reported the compliant answer blocked in 3 of 6 end-to-end runs. Each probe now runs SAMPLES times and is decided by majority, and the observed counts are always reported: judge_guardrail: ✓ judge discriminated (violating blocked 3/3, compliant allowed 3/3) so a marginal judge is visible rather than intermittently red. Threshold stays at the middleware default of 2.0, chosen by measurement rather than intuition — 20 samples per setting against gpt-5.2 on alpha: threshold violating blocked compliant allowed 2.0 10/10 10/10 4.0 10/10 7/10 Raising it only cost specificity. The table is in the source so nobody "tunes" it the wrong way later. Skipped without a PAT. The validator lives on `agentsruntime_`, which the client-credentials app cannot reach — the OAuth resource catalog has no `agentsruntime` entry, so an S2S token comes back 401. Verified: with UIPATH_PAT unset the cell records "– skipped" and does not fail the run, so the existing CI path stays green; run.sh swaps a supplied PAT into .env (it must go in the file, not the environment — the CLI loads .env with override=True, so an exported token loses to what `uipath auth` just wrote). Requires an ALPHA_TEST_PAT secret to actually run in CI. Verified end-to-end against alpha: 2 runs, both 3/3 on each probe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… detection Review findings on the judge_guardrail probe, fixed: - create_agent evaluates AGENT-scope llm_as_judge guardrails at BOTH PRE (the incoming user message) and POST (the answer); the middleware's stage=POST only affects its own middleware instances, which the probe never used. Reword the rule to judge "the text" so the compliant *request* no longer reads as a violation at PRE — the likely cause of the intermittent wrongly-blocked compliant probe — and state the dual-stage reality (and the up-to-12-judge-call cost) in the comments. - Build the guardrail on the public seam generated coded agents use (AgentBuiltInValidatorGuardrail + build_guardrails_with_actions) instead of reaching into middleware._guardrail; drops the dual BlockAction aliases and the dead stage/action arguments. - Count a block only on exact code match AND category USER: the guardrail node raises the same TERMINATION_GUARDRAIL_VIOLATION code with category DEPLOYMENT for feature-disabled/missing-entitlement outcomes, which previously registered as blocks (a false pass on the violating probe, a misdiagnosis on the compliant one). - Carry the offending answers/block reasons into the AssertionError so a red run is debuggable without re-running under a PAT. - Replace the loop-tail `continue` PAT skip with if/else so a probe appended later still runs without a PAT; match sibling cell style. - run.sh: uv run python for the heredoc; document that the PAT becomes the ambient token for the whole run (least-privilege note, also in the workflow and README); README section for the probe and its skip. Rule text and wiring changed, so the threshold table and 3/3 runs in the PR description were measured against the previous wording — re-verify end-to-end on alpha with a PAT before relying on the gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a third probe that puts the model under test in the judge role: a ReAct agent runs behind a real
UiPathLLMAsJudgeMiddlewareguardrail whose judge model is the one being onboarded, and the guardrail decides whether the agent's answer broke a natural-language rule.Important
Stacked on #1022 — base is
fix/model-onboarding-false-pass, notmain. Merge #1022 first; this diff is judge-only (+261, 5 files, no deletions).Needs an
ALPHA_TEST_PATsecret to actually run in CI. Without it the probe skips (see below), so merging this changes nothing until the secret exists.Two probes, not one
One probe cannot tell a working judge from a broken one:
A judge that always blocks fails the second. One that never blocks fails the first. Only a judge that discriminates passes both.
Sampled, because a single sample is a coin toss
The judge is itself a model call. An earlier single-shot version of this probe reported the compliant answer blocked in 3 of 6 end-to-end runs — flaky enough to be useless as a gate.
Each probe now runs
SAMPLEStimes and is decided by majority, and the observed counts are always reported, passing or failing:so a marginal judge shows up as a number rather than an intermittent red build.
Threshold chosen by measurement, not intuition
I had assumed the default threshold was too strict and planned to raise it. That was wrong, and measuring showed the opposite — 20 samples per setting against
gpt-5.2on alpha:On this scale higher is more lenient, so raising it only cost specificity. The default stands, and the table is in the source so nobody re-tunes it the wrong way.
Skipped without a PAT — the CI path stays green
The validator lives on
agentsruntime_, which the client-credentials app cannot reach: the OAuth resource catalog has noagentsruntimeentry, so an S2S token comes back401. Rather than fail every existing dispatch, the cell skips:Verified with
UIPATH_PATunset: the cell records the skip and does not fail the run.run.shswaps a supplied PAT into.envwhen one is present. It has to go in the file, not the environment — the CLI loads.envwithoverride=True, so an exportedUIPATH_ACCESS_TOKENloses to whateveruipath authjust wrote and the PAT would be silently ignored. That was verified in isolation.Verification
End-to-end against alpha, post-rebase:
Both paths exercised: with a user token (3/3 on each probe, 2 runs) and with
UIPATH_PATunset (skips, stays green).Caveats I have not resolved
X-UiPath-Guardrails-Sourceis required by the endpoint and comes from the execution context, so the guardrail only works inside auipath run. Calling the agent directly returns400. Fine here; a real constraint if this moves.SAMPLES=3means 6 agent+guardrail round-trips per flavor. Onboarding runs are on-demand, so I judged that acceptable; drop to 1 if it drags.🤖 Generated with Claude Code