fix(adjudication): tag-grounding guard for fabricated loaded-at-runtime (JEF-451)#251
Merged
Merged
Conversation
…t fabricates a loaded-at-runtime tag (JEF-451)
The recurring false exploitable on protector/argocd cites a REAL CVE id (so
guard_fabricated_cve passes) but attributes a [reachability: loaded-at-runtime]
tag to it that NO evidence line carries — every CVE is not-observed. The fable
architect audit isolated the root cause: `loaded-at-runtime` is the most-primed
phrase in the prompt (~10x in instructions, 0x in evidence), so a 1.7B judge
copy-completes it; the reasons are self-contradictory ("[reachability:
loaded-at-runtime] tags ... despite not being observed running").
Adds `guard_fabricated_reachability_tag`, chained after guard_fabricated_cve: an
Exploitable whose reason asserts loaded-at-runtime while no evidence CVE carries
that tag → downgrade to the skeptic Uncertain (re-judged next pass), never
Refuted. It reads the same rendered CVE strings the prompt shows.
This is a GROUNDING/integrity check — a string-membership test over the closed
three-value reachability vocabulary the engine itself renders — not a
breach-decision gate: it weighs no severity, re-derives no breach, and can never
fire toward a breach. It is the same class as guard_fabricated_cve, one token
deeper. Recorded as an amendment to ADR-0029 (which preserves that guard class);
the forbidden classes (verdict-override on a judgement basis, evidence capping)
stand unchanged. The primary remedy remains the model/prompt-layer restructuring
being planned; this is the deterministic backstop the prompt fixes shrink but
can't guarantee.
Tests: both live flip replies downgrade to Uncertain; a genuine loaded-at-runtime
citation and a non-tag exploitable pass through; non-Exploitable verdicts
untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes JEF-451. The deterministic backstop for the recurring false
exploitableon protector/argocd, adopted per the amended ADR-0029.The failure
The judge cites a real CVE id (so
guard_fabricated_cvepasses) but fabricates its[reachability: loaded-at-runtime]tag — a tag no evidence line carries (every CVE isnot-observed). Two live replies on protector/protector:The fable architect audit (R1):
loaded-at-runtimeis the most-primed n-gram in the prompt (~10× in instructions, 0× in evidence), so a 1.7B judge copy-completes it.The guard
guard_fabricated_reachability_tag, chained afterguard_fabricated_cve: anExploitablewhose reason asserts loaded-at-runtime while no evidence CVE carries that tag → downgrade toUncertain(re-judged next pass), neverRefuted. Reads the same rendered CVE strings the prompt shows.Why this is ADR-0029-compliant (amended)
It's a grounding/integrity check — a string-membership test over the closed three-value reachability vocabulary the engine itself renders (
Reachability::label) — the same class asguard_fabricated_cve, one token deeper. It weighs no severity, re-derives no breach, and can never fire toward a breach. ADR-0029 is amended with a scope-note recording tag-grounding as grounding, not a verdict gate; the forbidden classes (judgement-basis verdict override, evidence capping) stand unchanged.The primary remedy remains the model/prompt-layer restructuring (being planned separately, bakeoff-validated A/B); this is the deterministic backstop the prompt fixes shrink but can't guarantee.
Tests
Both live flip replies → Uncertain; a genuine loaded-at-runtime citation and a non-tag exploitable pass through; non-Exploitable verdicts untouched. Full suite + clippy + fmt green.
Ships in v0.3.105 alongside JEF-445 (unfreeze), so the recurring false breach both stops being produced (G1) and self-heals if it slips (JEF-445).
🤖 Generated with Claude Code