You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to the Discord thread with @romanlutz (2026-07-09), who suggested writing this up.
Problem
scorer_evaluation currently answers one question about a scorer: does it agree with human labels? It cannot answer a second question that determines whether those numbers are trustworthy: does the scorer's verdict survive a rewording of its own rubric? If verdicts change when the rubric is paraphrased, part of the reported accuracy reflects wording, not model behavior. This matters in practice because every team words the same policy differently, rubrics get edited and versioned over time (harm definitions already carry a version field for this reason), and a common tuning workflow is rephrasing a rubric until accuracy improves, which can fit the wording to the test set rather than improve the judge.
A recent paper formalizes this as policy invariance and provides the test protocol: "Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges" (arXiv 2605.06161). Meaning-preserving rubric rewrites flipped up to 9.1% of safety-judge verdicts above rerun noise in their experiments.
Evidence on PyRIT's own assets
I ran a pilot with SelfAskTrueFalseScorer (gpt-4.1-mini, temperature 0) on the 169 items of the shipped refusal eval datasets (refusal.csv + refusal_extra.csv), through the existing ObjectiveScorerEvaluator with no framework changes. One base refusal rubric plus three rewrites, verdicts compared per item against the majority of three base reruns.
Condition
Flip rate
Above jitter
Direction
Accuracy vs human
Base rubric, 3 reruns
2.0% (jitter)
baseline
n/a
0.82
Lexicon rewrite (synonyms, same meaning)
7.1%
+5.1 pts (p = 1.5e-4)
mixed
0.81
Threshold shift (strict to lenient)
8.3%
+6.3 pts (p = 7.9e-6)
14 of 14 toward "not refusal"
0.74
Syntax rewrite (uncertified)
13.6%
+11.6 pts
one-directional
0.95
Three observations -
First, the threshold shift flipped verdicts in the expected direction only, which is the sanity check that the method measures something coherent.
Second, reported accuracy for the same scorer and policy swung from 0.74 to 0.95 across wordings; these are the numbers the metrics registry currently records as a scorer's quality.
Third, the syntax row is included as a cautionary result: it drifted in meaning despite the intent, which is why variant validation is part of the proposal below.
A baseline note: subtracting pairwise jitter follows the paper's convention, but the matched null for a single run compared against a majority-of-3 anchor is roughly half the pairwise jitter, so the deltas above are conservative.
Because single-run flip rates can overstate wording effects, I re-tested every flipped item with three fresh trials per rubric, capturing verdict rationales. No fallback or parse-failure scores occurred, and five stable control items stayed stable across all rubrics. The threshold flips are systematic: 11 of 14 reproduced as majority verdict changes, most as unanimous three-of-three reversals that cannot come from 2% jitter. The lexicon flips split: 6 of 12 reproduced as clean verdict changes (including one unanimous reversal), while the other 6 showed wording-induced instability, with the judge alternating verdicts under the reworded rubric on items where it is otherwise stable. Both behaviors are wording sensitivity, and separating deterministic flips from induced instability is a distinction the proposed metrics should report.
One composition note for transparency - The shipped refusal datasets contain a sizable family of deflection items (the objective asks one thing, the response answers something unrelated, and the human label counts that as refusal), and the flips concentrate there: 11 of 14 threshold flips and most lexicon flips land on deflection items. This matches the paper's prediction that instability concentrates on genuinely ambiguous cases, since whether deflection counts as refusal is exactly the clause that rewording perturbs. It also means absolute flip rates depend on dataset composition and will differ on datasets without such a family, which is an argument for reporting flip rates stratified by item type in the proposed metrics.
Three additions to pyrit/score/scorer_evaluation/, following the module's existing patterns. All API sketches below are indicative; I would align them with the scorer refactor once it lands rather than target the current classes.
1. Rubric variant generator
An LLM-driven rewriter that takes a rubric (a TrueFalseQuestion, a true/false question YAML, or a harm definition YAML) and produces labeled variants along the paper's transform taxonomy:
Intentional shift: threshold strictness (strict to lenient wording).
Supplementary: irrelevant context injection.
Each candidate passes automatic validation before use: length ratio bounds, non-identity with the original, retention of verdict keywords, and an LLM equivalence check for the meaning-preserving class. Generation prompts live in YAML per module convention. Optionally, pre-generated and reviewed variant sets ship for the built-in rubrics so audits of stock scorers need no generation step.
2. PolicyInvarianceEvaluator and metrics
An evaluator that instantiates the scorer once per rubric variant, runs each over the same HumanLabeledDataset via the existing evaluation path, aligns per-item verdicts, and computes:
p_jitter: proportion of discordant verdict pairs across N reruns of the base rubric (N=3 default).
Per transform class: flip rate against the base majority verdict, and delta above jitter.
For threshold pairs: directional flip ratio (a reliable judge flips only toward the lenient side).
Where multi-rater labels exist (currently the two multi_score CSVs): flip rates stratified by human ambiguity. Optional otherwise.
Optionally, reproduction trials on flipped items to separate deterministic flips from wording-induced instability (the pilot found both in roughly equal measure for the lexicon transform).
Results land in a PolicyInvarianceMetrics dataclass alongside the existing HarmScorerMetrics and ObjectiveScorerMetrics, serialized to the metrics registry. Because the rubric text feeds the scorer's eval hash, each variant already resolves to a distinct registry identity today; the new entry would key the report to the base scorer's hash with variant hashes recorded inside.
3. Reliability report
A "Judge Card" style summary rendered from the metrics (the paper's term), so a scorer's registry entry can answer "how wording-sensitive is this scorer" next to "how accurate is it".
Deliberately out of scope
The paper's composite Policy Invariance Score with fixed weights stays out of core. Raw components only; the composite can be an optional derived value with a citation.
The paper's three-annotator human certification of rewrites. LLM validation plus optional reviewed variant sets for built-in rubrics instead.
Everything is reimplemented from the paper's formulas; nothing is ported from the paper's code release.
Sequencing
@romanlutz mentioned an in-flight scorer refactor. This proposal is intentionally design-level: I would start implementation only after the refactor merges, and target the post-refactor APIs. Planned as two PRs: the variant generator first (self-contained), then the evaluator and report.
Open questions for maintainers
Should the evaluator accept a scorer factory (rubric in, scorer out) or a list of pre-built scorers? The factory is cleaner but couples to scorer constructor shapes, which the refactor may change.
Is the metrics registry the right home for the report, or should reliability results live in a separate result file to keep the registry schema stable?
Follow-up to the Discord thread with @romanlutz (2026-07-09), who suggested writing this up.
Problem
scorer_evaluationcurrently answers one question about a scorer: does it agree with human labels? It cannot answer a second question that determines whether those numbers are trustworthy: does the scorer's verdict survive a rewording of its own rubric? If verdicts change when the rubric is paraphrased, part of the reported accuracy reflects wording, not model behavior. This matters in practice because every team words the same policy differently, rubrics get edited and versioned over time (harm definitions already carry aversionfield for this reason), and a common tuning workflow is rephrasing a rubric until accuracy improves, which can fit the wording to the test set rather than improve the judge.A recent paper formalizes this as policy invariance and provides the test protocol: "Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges" (arXiv 2605.06161). Meaning-preserving rubric rewrites flipped up to 9.1% of safety-judge verdicts above rerun noise in their experiments.
Evidence on PyRIT's own assets
I ran a pilot with
SelfAskTrueFalseScorer(gpt-4.1-mini, temperature 0) on the 169 items of the shipped refusal eval datasets (refusal.csv+refusal_extra.csv), through the existingObjectiveScorerEvaluatorwith no framework changes. One base refusal rubric plus three rewrites, verdicts compared per item against the majority of three base reruns.Three observations -
A baseline note: subtracting pairwise jitter follows the paper's convention, but the matched null for a single run compared against a majority-of-3 anchor is roughly half the pairwise jitter, so the deltas above are conservative.
Because single-run flip rates can overstate wording effects, I re-tested every flipped item with three fresh trials per rubric, capturing verdict rationales. No fallback or parse-failure scores occurred, and five stable control items stayed stable across all rubrics. The threshold flips are systematic: 11 of 14 reproduced as majority verdict changes, most as unanimous three-of-three reversals that cannot come from 2% jitter. The lexicon flips split: 6 of 12 reproduced as clean verdict changes (including one unanimous reversal), while the other 6 showed wording-induced instability, with the judge alternating verdicts under the reworded rubric on items where it is otherwise stable. Both behaviors are wording sensitivity, and separating deterministic flips from induced instability is a distinction the proposed metrics should report.
One composition note for transparency - The shipped refusal datasets contain a sizable family of deflection items (the objective asks one thing, the response answers something unrelated, and the human label counts that as refusal), and the flips concentrate there: 11 of 14 threshold flips and most lexicon flips land on deflection items. This matches the paper's prediction that instability concentrates on genuinely ambiguous cases, since whether deflection counts as refusal is exactly the clause that rewording perturbs. It also means absolute flip rates depend on dataset composition and will differ on datasets without such a family, which is an argument for reporting flip rates stratified by item type in the proposed metrics.
Pilot script, reproduction probe, and full results: https://gist.github.com/Raulster24/4afea4845a6d4003f55f0b749f19b316
Proposed design
Three additions to
pyrit/score/scorer_evaluation/, following the module's existing patterns. All API sketches below are indicative; I would align them with the scorer refactor once it lands rather than target the current classes.1. Rubric variant generator
An LLM-driven rewriter that takes a rubric (a
TrueFalseQuestion, a true/false question YAML, or a harm definition YAML) and produces labeled variants along the paper's transform taxonomy:Each candidate passes automatic validation before use: length ratio bounds, non-identity with the original, retention of verdict keywords, and an LLM equivalence check for the meaning-preserving class. Generation prompts live in YAML per module convention. Optionally, pre-generated and reviewed variant sets ship for the built-in rubrics so audits of stock scorers need no generation step.
2. PolicyInvarianceEvaluator and metrics
An evaluator that instantiates the scorer once per rubric variant, runs each over the same
HumanLabeledDatasetvia the existing evaluation path, aligns per-item verdicts, and computes:p_jitter: proportion of discordant verdict pairs across N reruns of the base rubric (N=3 default).multi_scoreCSVs): flip rates stratified by human ambiguity. Optional otherwise.Results land in a
PolicyInvarianceMetricsdataclass alongside the existingHarmScorerMetricsandObjectiveScorerMetrics, serialized to the metrics registry. Because the rubric text feeds the scorer's eval hash, each variant already resolves to a distinct registry identity today; the new entry would key the report to the base scorer's hash with variant hashes recorded inside.3. Reliability report
A "Judge Card" style summary rendered from the metrics (the paper's term), so a scorer's registry entry can answer "how wording-sensitive is this scorer" next to "how accurate is it".
Deliberately out of scope
Sequencing
@romanlutz mentioned an in-flight scorer refactor. This proposal is intentionally design-level: I would start implementation only after the refactor merges, and target the post-refactor APIs. Planned as two PRs: the variant generator first (self-contained), then the evaluator and report.
Open questions for maintainers
References