Conversation
JAORMX
force-pushed
the
spike/jev-guardrail-poc
branch
from
September 27, 2026 07:46
0f8f0de to
18c248f
Compare
Co-Authored-By: mecatl <noreply@mecatl.dev>
Co-Authored-By: mecatl <noreply@mecatl.dev>
Replace the standalone classifier probe with a Build-selected native ToolReviewer and real action/inbound run proofs. Co-Authored-By: mecatl <noreply@mecatl.dev>
Treat incomplete and over-limit context as unresolved, reject response-model drift, and verify headless result withholding in the real run path. Co-Authored-By: mecatl <noreply@mecatl.dev>
Fence untrusted checker state, carry additive operator policy, validate complete evidence chains and failure codes, and pin end-to-end execution/result assertions. Co-Authored-By: mecatl <noreply@mecatl.dev>
Assert action side effects, withheld inbound content, model-visible history, checker-down and advisory dispositions, and worker review. Co-Authored-By: mecatl <noreply@mecatl.dev>
Document native Jev as an alternative to the LLM model without making the fully uncommented skeleton invalid. Co-Authored-By: mecatl <noreply@mecatl.dev>
Keep the LLM skeleton valid when fully uncommented, place the Jev alternative in its worked example, and render its nested field in the generated reference. Co-Authored-By: mecatl <noreply@mecatl.dev>
JAORMX
force-pushed
the
spike/jev-guardrail-poc
branch
from
September 27, 2026 07:55
18c248f to
3165126
Compare
Select the pinned experimental Jev checker with guardrails.backend alone. Leave router confidence independent, reject the retired nested key with migration guidance, and regenerate operator documentation. Co-Authored-By: mecatl <noreply@mecatl.dev>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft experimental spike. Do not merge this as a production-ready guardrail checker.
This branch tries Jev in Mecatl's existing action and inbound review paths.
guardrails.backend: jevopts in to a pinnedjev-1.13.0TypeSafe System One driver. The existing LLM guardrails remain the default. Jev model routing is independent and does not select this checker.The operator must supply
TYPESAFE_API_KEY.models.routerremains exclusively about delegated model routing; its Jev confidence setting does not govern guardrails. The earlier draft-onlyguardrails.jevblock is removed, and configuration validation tells operators to delete it. Opting in sends effective tool arguments, inbound result text, authenticated task facts, trajectory, and authorized bounded evidence to TypeSafe. Incomplete, oversized, unsupported, or low-confidence decisions do not become clean. The fixed0.8confidence threshold is uncalibrated; the synthetic tests and live smoke do not establish security efficacy. The driver has a separate 10-second request bound and no SDK retries, not the approved contextual reviewer's 90-second recovery contract.Offline
app.Build/HTTP/agent-run tests exercise action denial, inbound withholding, advisory and checker-down outcomes, and worker inheritance. The opt-in billable live test passed with synthetic Shell input and a token file. A second, synthetic HTTP user-path run used OpenRouter for the working model and TypeSafe for Jev: the Shell result matched the expected harmless phrase, Jev completed action and inbound reviews, and the session ended withend_turn. No key value was printed or added to the repository.We rebased onto current main and fixed the CI failure in
TestSkeletonRoundTripsThroughLiveSchemaat the config generator source. The generated skeleton now keeps its LLM example parseable and showsbackend: jevas a separate worked example. The generated reference has no redundantguardrails.jev.modelknob. Locally,task docs,task site:build,task test,task test:race,task lint, and the offline demo passed. CI independently checks the pushed branch.Still open before production approval: this backend adds an external review-data boundary and an additional checker-selection surface outside the approved
models.slots.guardrailcontract. Empirical adversarial detection quality, fallback/retry behavior, and canonical Jev usage attribution need a reviewed runtime contract and evidence. Draft evaluation plan #1968 explicitly does not grant runtime authority. Keep this PR draft; a green CI run alone does not resolve those decisions. Humans alone merge.