Repository navigation
Python: fix(core): deep-restore nested registered values in the default Pydantic state decoder - #9250
Yufeng He (he-yufeng) wants to merge 16 commits into
Conversation
…lt Pydantic state decoder The default encoder hands nested values to _serialize_value, which tags them for the registry, but the default Pydantic decoder validated the raw payload, so a nested Message arrived as a tagged dict and restoration of a registered state model failed on AgentSession.from_dict and on a fresh FileSessionStore read. Fixes microsoft#9246
There was a problem hiding this comment.
🟢 Approval recommended
The focused decoder change preserves existing codec contracts and is adequately covered by regression tests.
0 open findings
What changed in this PR
Fixes nested registered-value restoration in default Pydantic session-state decoding.
Changes:
- Deep-deserializes Pydantic payload fields before validation.
- Adds direct and file-backed regression coverage.
| File | Description |
|---|---|
python/packages/core/agent_framework/_sessions.py |
Restores nested registered values before Pydantic validation. |
python/packages/core/tests/core/test_sessions.py |
Tests nested Message round trips. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
|
Yufeng He (@he-yufeng) Please update the PR body to use the repository's current PR template at |
…lues Deep-restoring every payload field up front hijacks a plain dict field whose "type" happens to name a registered state type: the tagged shape is converted before model_validate and then rejected. Validate the raw payload first and only retry with restored values when that fails, and cover a tag-shaped dict field staying raw. Also formats the two earlier nested-message cases that missed ruff format.
|
Eduard van Valkenburg (@eavanvalkenburg) done, the body now follows the current template including the AI Assistance section with the assisted areas identified. |
Eduard van Valkenburg (eavanvalkenburg)
left a comment
There was a problem hiding this comment.
Yufeng He (@he-yufeng) Please address the inline alias-restoration finding before merge.
…lt Pydantic state decoder The retry keyed on the first segment of the validation-error location, so a failure pydantic reported at `child.message` restored the whole `child`. `_deserialize_value` then walked the siblings too, and a sibling that legitimately holds a tag-shaped dict, an ordinary `dict[str, Any]` whose `type` merely happens to name a registered type, was handed to that type's codec. It either came back as a framework object or raised `Failed to deserialize registered state type`, which failed the whole restore because of one broken field. The complete error location is now walked instead. Each segment is resolved onto the raw payload through the field name and its aliases and through container indexes, so a failure two levels down behind `Field(alias=...)` inside a list or a keyed mapping repairs exactly that value and leaves its siblings byte-identical. The class governing each level comes from the annotations: a nested model's own fields resolve the keys beneath it, while `dict[str, Item]` and `list[Item]` treat their keys or positions as application data and hand the item class down to the values. Where a location would reach inside a payload owned by a registered codec, that codec still restores the value as a whole, since only it knows the shape. A deeper location supersedes a shallower one on the same path, so blaming both `child` and `child.message` repairs `child.message` alone. Three regressions pin it, each pairing a `Message` field with raw tag-shaped application data: a nested model, an aliased field behind a list index, and an aliased field behind a keyed container. All three fail without the fix, on `Failed to deserialize registered state type 'message' at ...data`, which is the sibling being swept up. 200 tests pass in packages/core/tests/core/test_sessions.py and 8983 in packages/core/tests. `poe pyright` reports the same errors as the base on the file changed, and ruff check and format are clean.
|
Yufeng He (@he-yufeng) The deep-restoration walker still misses two supported nested shapes. First, Pydantic can include synthetic union-branch labels in error locations—for example |
|
Yufeng He (@he-yufeng), I found two material union-restore gaps on the current head:
|
|
Eduard van Valkenburg (@eavanvalkenburg) Implemented both follow-ups in Union recovery no longer depends only on class-name labels. For Added JSON and MessagePack round trips for an arbitrary-type union, an aliased nested On the merged head, the complete core suite passes with 9471 passed, 22 skipped, and 2 xfailed. Ruff, formatting, Pyright, mypy, pyrefly, ty, and zuban all pass for core. |
|
Eduard van Valkenburg (@eavanvalkenburg) Current main is merged in On this merged head, the complete core suite is 9546 passed, 21 skipped, and 2 xfailed. Test Pyright and the pre-commit hooks pass. Could you take another look at the union-restoration follow-ups from |
BlueRaddish
left a comment
There was a problem hiding this comment.
The union retry still selects the wrong branch when two distinct Pydantic model classes share the same class name. This can occur when application state combines result types from two libraries that both call their model Result.
At c7dbb108910a7288ccf226339caef692032ad81b, I reproduced it through the public register_state_type and AgentSession.to_dict / AgentSession.from_dict APIs, with a real JSON serialization step between them:
provider_a.Result: kind = Literal["a"], item = registered opaque Item
provider_b.Result: kind = Literal["b"], item = registered opaque Item
Outer.choice: provider_a.Result | provider_b.Result
original choice = provider_b.Result(kind="b", item=Item("saved-value"))
AgentSession.from_dict(JSON-round-tripped session.to_dict())
→ ValueError: Failed to deserialize registered state type ... at state.value
Pydantic reports both branches using the same Result segment, including duplicate ("choice", "Result", "item") locations. _restore_union resolves that segment with next(arg for arg ... if _union_branch_label(arg) == segment), which selects only provider_a.Result. Its kind="a" validation correctly rejects the stored kind="b", but the second matching class never gets a restoration attempt. The fallback over every candidate only runs when no name matches, so it does not cover a non-unique matching label.
Could a label that matches multiple branches try each matching candidate independently, instead of committing to the first class name? Please add a regression where the saved value belongs to the second same-named branch. Two controls confirm the distinction: the first Result branch round-trips successfully, and the second branch round-trips successfully when the class names are distinct. Both controls preserve the expected branch class and restore the nested registered Item object.
Verification used Python 3.11.17 and Pydantic 2.14.1. It executed the unchanged complete current registry, serializer, decoder and restoration definitions plus the complete public AgentSession class, isolated from unrelated SDK imports. The fixture creates two distinct real Pydantic classes named Result with different provider-module names, and uses an ordinary registered opaque class implementing to_dict / from_dict. A dataclass version also round-tripped successfully because Pydantic could validate its raw dictionary directly; it did not exercise this repair path. No full SDK import, FileSessionStore, MessagePack, repository owner suite, external provider or live model execution is claimed. Analysis, reproduction preparation and this review were AI-assisted.
| continue | ||
| config = ( | ||
| None | ||
| if isinstance(candidate, type) and issubclass(candidate, BaseModel) |
There was a problem hiding this comment.
Yufeng He (@he-yufeng) This guard recognizes only bare BaseModel classes. Annotated[...] model branches, dataclasses, and TypedDict branches fall through to ConfigDict(arbitrary_types_allowed=True), but Pydantic rejects adapter configuration for those types. A valid tagged union can therefore serialize successfully and then fail during restoration. Please unwrap Annotated and avoid passing adapter config to types that own their validation configuration, with regressions for tagged model, dataclass, and TypedDict union branches.
There was a problem hiding this comment.
Implemented in caa8b5312. Annotated is unwrapped for the configuration decision, and BaseModel, dataclass, and TypedDict candidates now use their own validation configuration instead of receiving an adapter override.
All three tagged-union shapes have JSON and MessagePack round-trip regressions. The complete core run collected 9,581 tests with 0 failures, 0 errors, and 27 skipped. Source Pyright reports 0 errors, all five test-typing tasks pass, and Ruff check/format pass. Pytest also emitted post-test OpenTelemetry export I/O messages on this machine after completing successfully; those are not being omitted from the local evidence.
There was a problem hiding this comment.
I rechecked the current head fb9f20cb29c77257e003c085f5a889cdb0a7987b. The union decoder unwraps Annotated before classification and leaves adapter configuration to types that own their validation config, including BaseModel, dataclasses and TypedDict. The same-named union control is also covered. The four focused session regressions are present, and all 50 current-head checks are green. Could you take another look at this thread and the older change request when you have a chance?
|
BlueRaddish The regression covers your second same-named |
|
Eduard van Valkenburg (@eavanvalkenburg) Current head The JSON and MessagePack round trips cover both new cases. The complete core run is 9,581 collected with 0 failures, 0 errors, and 27 skipped; source Pyright and all five test-typing tasks pass. Could you take another look at the latest follow-up? |
|
The pre-commit failure is addressed in Pyupgrade moved The full |
|
Re-verified the current head The one-level shift is gone. The regression covers all three reported shapes plus a deeper composition, in both serialization formats: → 8 passed, 230 deselected. The full The earlier Nothing further is pending on my side. The formal |
Motivation & Context
Registered Pydantic application state can persist a nested
Messagesuccessfully but fail to restore it because validation receives the tagged dictionary instead of the registered value. Error locations must be followed through the actual schema, including aliases and multiple container levels, without interpreting unrelated application dictionaries as codecs.Description & Review Guide
The reported nested shapes failed in six JSON/MessagePack cases before the fix, with two deeper-composition controls already passing. The final session file has 238 passed with three existing framework warnings. Scoped strict Pyright, scoped zuban, Ruff and formatting pass. The all-core strict Pyright attempt timed out after 120 seconds; the earlier full-core counts in this PR's history have not been rerun on this head. Hosted checks remain required.
Related Issue
Fixes #9246
AI Assistance
AI assistance covered implementation, regression tests and this description. The current verification was executed locally by the agent against the reported source checkout and existing project environment; scoped results are not presented as a full workspace check.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix). A workflow keeps the label and title prefix in sync automatically.