Skip to content

spike(guardrails): wire native Jev into Mecatl action and inbound review - #1969

Draft
JAORMX wants to merge 9 commits into
mainfrom
spike/jev-guardrail-poc
Draft

JAORMX wants to merge 9 commits into
mainfrom
spike/jev-guardrail-poc

Conversation

@JAORMX

@JAORMX JAORMX commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Draft experimental spike. Do not merge this as a production-ready guardrail checker.

This branch tries Jev in Mecatl's existing action and inbound review paths. guardrails.backend: jev opts in to a pinned jev-1.13.0 TypeSafe System One driver. The existing LLM guardrails remain the default. Jev model routing is independent and does not select this checker.

guardrails:
  backend: jev

The operator must supply TYPESAFE_API_KEY. models.router remains exclusively about delegated model routing; its Jev confidence setting does not govern guardrails. The earlier draft-only guardrails.jev block is removed, and configuration validation tells operators to delete it. Opting in sends effective tool arguments, inbound result text, authenticated task facts, trajectory, and authorized bounded evidence to TypeSafe. Incomplete, oversized, unsupported, or low-confidence decisions do not become clean. The fixed 0.8 confidence threshold is uncalibrated; the synthetic tests and live smoke do not establish security efficacy. The driver has a separate 10-second request bound and no SDK retries, not the approved contextual reviewer's 90-second recovery contract.

Offline app.Build/HTTP/agent-run tests exercise action denial, inbound withholding, advisory and checker-down outcomes, and worker inheritance. The opt-in billable live test passed with synthetic Shell input and a token file. A second, synthetic HTTP user-path run used OpenRouter for the working model and TypeSafe for Jev: the Shell result matched the expected harmless phrase, Jev completed action and inbound reviews, and the session ended with end_turn. No key value was printed or added to the repository.

We rebased onto current main and fixed the CI failure in TestSkeletonRoundTripsThroughLiveSchema at the config generator source. The generated skeleton now keeps its LLM example parseable and shows backend: jev as a separate worked example. The generated reference has no redundant guardrails.jev.model knob. Locally, task docs, task site:build, task test, task test:race, task lint, and the offline demo passed. CI independently checks the pushed branch.

Still open before production approval: this backend adds an external review-data boundary and an additional checker-selection surface outside the approved models.slots.guardrail contract. Empirical adversarial detection quality, fallback/retry behavior, and canonical Jev usage attribution need a reviewed runtime contract and evidence. Draft evaluation plan #1968 explicitly does not grant runtime authority. Keep this PR draft; a green CI run alone does not resolve those decisions. Humans alone merge.

@JAORMX JAORMX changed the title spike(guardrails): compare Jev signals on synthetic review cases spike(guardrails): wire native Jev into Mecatl action and inbound review Sep 26, 2026
@JAORMX
JAORMX force-pushed the spike/jev-guardrail-poc branch from 0f8f0de to 18c248f Compare September 27, 2026 07:46
JAORMX and others added 8 commits September 27, 2026 10:54
Co-Authored-By: mecatl <noreply@mecatl.dev>
Co-Authored-By: mecatl <noreply@mecatl.dev>
Replace the standalone classifier probe with a Build-selected native ToolReviewer and real action/inbound run proofs.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Treat incomplete and over-limit context as unresolved, reject response-model drift, and verify headless result withholding in the real run path.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Fence untrusted checker state, carry additive operator policy, validate complete evidence chains and failure codes, and pin end-to-end execution/result assertions.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Assert action side effects, withheld inbound content, model-visible history, checker-down and advisory dispositions, and worker review.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Document native Jev as an alternative to the LLM model without making the fully uncommented skeleton invalid.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Keep the LLM skeleton valid when fully uncommented, place the Jev alternative in its worked example, and render its nested field in the generated reference.

Co-Authored-By: mecatl <noreply@mecatl.dev>
Select the pinned experimental Jev checker with guardrails.backend alone. Leave router confidence independent, reject the retired nested key with migration guidance, and regenerate operator documentation.

Co-Authored-By: mecatl <noreply@mecatl.dev>

This branch was successfully deployed

1 active deployment
Preview — 1b3314ed Deployed Sep 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant