fix(handoffs): enforce strict Pydantic validation when strict_json_schema=True#3724
fix(handoffs): enforce strict Pydantic validation when strict_json_schema=True#3724Om-Borse26 wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0edfac8098
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Great catch! You are entirely correct—by passing I have updated the I've pushed the fix and updated the regression tests to verify that omitting the parameter still preserves Pydantic's default lenient coercion behavior for a model without strict config. Thanks for flagging this subtle regression! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d31bc497dc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
d31bc49 to
711d7ea
Compare
|
Thanks for the sharp eyes on both of these. Fixed in the latest push. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 711d7ea250
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
711d7ea to
9679c26
Compare
|
Another great catch! Passing I have updated the logic in Latest push incorporates this fix! |
Summary
Fixes #3723
The
Handoffclass explicitly setsstrict_json_schema=True(its default behavior) which instructs the LLM to adhere to a strict schema for the output type. However, during runtime validation in_invoke_handoff, thevalidate_jsonutility is called withoutstrict=True. As a result, Pydantic defaults to its lenient mode and silently coerces invalid types, entirely ignoring the strictness intent of the handoff definition.This PR passes
strict=Trueto the validation logic when handling handoff inputs, ensuring that the runtime validation strictness matches the schema provided to the LLM.Demonstration
Before (Silent Coercion)
When an LLM incorrectly passed
"25"(string) instead of25(int) for an age field requiring an integer, the lenient parsing silently accepted and coerced it:After (ValidationError)
With
strict=True, Pydantic correctly rejects the mismatch:Verification
You can verify this fix directly via the new test rather than a raw script:
(A secondary test
test_handoff_lenient_json_allows_type_coercionwas also added to ensure backward compatibility for other callers using lenient mode).(Note: AI-assisted contribution)