Fix #5979: Enforce guardrail re-validation on retry and reject negative max_retries#5980
Open
devin-ai-integration[bot] wants to merge 2 commits into
Open
Fix #5979: Enforce guardrail re-validation on retry and reject negative max_retries#5980devin-ai-integration[bot] wants to merge 2 commits into
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
…_retries Fixes #5979 — guardrails are now enforced as hard constraints. Two bugs fixed: 1. Multiple guardrails skipped re-validation of earlier guardrails after retry. When guardrail N failed and the agent retried, the new output was only checked against guardrail N onward. Earlier guardrails (0..N-1) were not re-evaluated, allowing retry outputs to silently violate them. Fix: refactored _invoke_guardrail_function into _run_guardrails which runs ALL guardrails from the beginning on each retry attempt. 2. Negative guardrail_max_retries bypassed guardrails entirely. Setting guardrail_max_retries to a negative value made the guardrail loop execute zero iterations, letting output pass without validation. Fix: added ge=0 constraint to the guardrail_max_retries field. Co-Authored-By: João <joao@crewai.com>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: João <joao@crewai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two bugs in the task guardrail enforcement system where guardrails could be silently bypassed, treating them as suggestions rather than hard constraints.
Bug 1: Multiple guardrails skip re-validation of earlier guardrails after retry.
When guardrail N failed and the agent retried, the new output was only checked against guardrail N onward — guardrails 0..N-1 were never re-evaluated. A retry output that violated an earlier guardrail would pass through unchecked.
Refactored
_invoke_guardrail_function/_ainvoke_guardrail_functioninto_run_guardrails/_arun_guardrailswhich run all guardrails in a single loop — on any failure the task is re-executed and validation restarts from the first guardrail.Bug 2: Negative
guardrail_max_retriesbypassed guardrails entirely.Setting
guardrail_max_retries = -1mademax_attempts = 0, causing the validation loop to never execute. Addedge=0constraint to the field.Tests added:
test_retry_revalidates_earlier_guardrails— core scenario where retry output violates an earlier guardrailtest_retry_restart_catches_earlier_guardrail_violation— regression test with interleaved failurestest_negative_guardrail_max_retries_rejected— validation of negative valuestest_zero_guardrail_max_retries_no_retry— zero retries fails immediatelytest_single_guardrail_retry_still_works— backward compat for single guardrailLink to Devin session: https://app.devin.ai/sessions/8b6d443fc3e4401cb126c03c8cf037e9