Skip to content

Python: [Bug]: Default registered Pydantic state cannot restore a Message field #9246

Description

@ktz03

Observed Behavior

A Pydantic application-state model with arbitrary_types_allowed=True and a Message field can be explicitly registered with register_state_type using the supplied default codecs. FileSessionStore.set successfully persists this state, but a freshly constructed store cannot restore it. The behavior occurs with both JSON and MessagePack and with the same source, model definition, and explicit registration. The deepest error is a Pydantic validation error because the Message field receives a tagged dictionary instead of a Message instance.

The same registered model also fails in AgentSession.from_dict(session.to_dict()). In-memory SessionStore works. A Message directly inside a normal state mapping, an ordinary scalar Pydantic model, and a nested Pydantic-model field all round-trip. Explicit paired codecs on the application model, reconstructing the child with Message.from_dict, also work.

The persisted snapshot is unchanged after the failed read. This is a state-decoder failure, not syntactically corrupt storage, and the existing non-destructive recovery behavior is correct.

Expected Behavior

Please clarify the intended default-codec contract for a registered Pydantic model containing a separately registered framework/application type. If this combination is supported, a successfully persisted snapshot should restore the typed parent and child with the same module-level registrations. If an explicit parent codec is required, the unsupported default-codec combination should be documented or rejected before creating an unreadable snapshot.

Steps to Reproduce

  1. Define the Pydantic model below and explicitly register it at module level. Message is already a registered framework state type.
  2. Create an AgentSession containing an instance of this model and persist it with FileSessionStore.set.
  3. Construct a new FileSessionStore and call get using the same key. Saving succeeds; restoring fails for both formats.

Minimal Reproduction

import asyncio
import tempfile

from agent_framework import AgentSession, FileSessionStore, Message, register_state_type
from pydantic import BaseModel, ConfigDict


class ApplicationState(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)
    last_message: Message


register_state_type(ApplicationState, type_id="example.application_state")


async def main():
    for fmt in ("json", "msgpack"):
        with tempfile.TemporaryDirectory() as directory:
            session = AgentSession(session_id="application-session")
            session.state["application"] = ApplicationState(
                last_message=Message("assistant", ["Saved reply"])
            )
            await FileSessionStore(directory, serialization_format=fmt).set("key", session)
            print(fmt, "save succeeded")
            try:
                await FileSessionStore(directory, serialization_format=fmt).get("key")
            except ValueError as error:
                while error.__cause__ is not None:
                    error = error.__cause__
                print(fmt, type(error).__name__, str(error))


asyncio.run(main())

Error Messages and Stack Traces

The outer store error is ValueError: Failed to restore session state from '.../key.json' (or .msgpack). Its cause is Failed to deserialize registered state type 'example.application_state' at state.application. The deepest error reports that last_message must be an instance of Message, with input_type=dict.

Package Versions

Tested executable core source: official main fd52de71579162a888fb2dd7510c57c6014dcdfc, core pyproject.toml version 1.21.0. The cached editable environment still reports agent-framework-core==1.19.0 metadata; all 109 tracked core source/metadata files were checked against current official Git blobs. pydantic==2.13.5, msgspec==0.21.1. This is source-scoped validation with cached dependencies, not a fresh dependency installation or release-wheel claim.

Python Version

Python 3.14.3.

Operating System

Windows.

Regression

Unknown; no earlier version comparison was run.

Additional Context

The public same-process matrix has 24 overlapping observations: 18 normal and 6 restore failures. Separate writer and reader processes confirm JSON and MessagePack persistence with module-level registrations: 8 normal reader observations and 2 failures. These matrices overlap and are not added. There is no hosted service, model, deployment or non-Windows validation.

Source inspection shows the registered encoder recursively translates the Message field into its tagged mapping, while the default Pydantic decoder calls model_validate before that field is reconstructed. ADR-0034 requires dynamic nested state restoration and preserves existing tagged representations. Any resolution must also preserve the current test test_registered_child_tag_does_not_hijack_message_payload: globally decoding every child tag before invoking all registered codecs would interfere with existing Message content payloads. Unknown type IDs and explicit raw-payload callback contracts must remain compatible. An explicit parent codec is already a working workaround.

The bounded ownership search and complete selected records cover the session/codec implementation in merged #7306, the closed additional-test proposal #7457, and vNext design #8522. #8522 concerns a future workflow architecture and does not authorize a current decoder overhaul. No open fix for this particular default-Pydantic parent/registered-child combination was found in the bounded search; this is not a claim of repository-wide exhaustive ownership coverage.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

AI Assistance

AI-assisted: Codex; analysis, reproduction, and writing.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 10, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 10, 2026
  3. github-actions commented on Oct 10, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    The failure occurs in python/packages/core/agent_framework/_sessions.py::_deserialize_value at line 432, which invokes the registered Pydantic parent decoder before restoring its tagged Message child; _default_state_decoder at line 274 then passes that dictionary to model_validate. It is triggered by an explicitly registered Pydantic model with arbitrary_types_allowed=True and a Message field. Minimal repro: persist that model in AgentSession.state with FileSessionStore and read it using a fresh JSON or MessagePack store.

    • Failing test: python/packages/core/tests/core/test_sessions.py::TestFileSessionStore::test_registered_pydantic_state_with_message_round_trips
    • Files examined: python/packages/core/agent_framework/_sessions.py, python/packages/core/tests/core/test_sessions.py, python/packages/core/pyproject.toml
    • Tests run: test_registered_pydantic_state_with_message_round_trips, test_explicit_pydantic_registration_round_trips, test_registered_child_tag_does_not_hijack_message_payload
    • Reported version: 1.21.0
    • Current version: 1.21.0
  4. added
    agentsUsage: [Issues, PRs], Target: Single agent
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 10, 2026
  5. he-yufeng commented on Oct 10, 2026

    @he-yufeng
    Contributor

    Verified the mechanism on current main (fd52de7). The default codecs are asymmetric:

    • Encode: model_dump() leaves a nested Message as an instance in the payload, and _serialize_value recurses into it, so it lands on disk as a tagged dict (type: ...).
    • Decode: the default Pydantic decoder calls cls.model_validate(...) on the raw payload, so the nested field receives the tagged dict where the field expects a Message instance. Nothing in between deep-restores it.

    So the stored snapshot is fine; the restore side is missing the mirror of the encoder's recursion. My proposal: make the default Pydantic decoder deep-restore payload items through _deserialize_value before model_validate, leaving custom codecs' contract (raw tagged payloads) untouched. Unregistered models keep the existing implicit-registration path, and unknown type tags still pass through as raw dicts.

    Opening a draft PR with that change plus round-trip regressions on both AgentSession.from_dict and a fresh FileSessionStore. Happy to adjust if you want the deep-restore somewhere else in the codec chain.

  6. eavanvalkenburg commented on Nov 3, 2026

    @eavanvalkenburg
    Member

    /plan Scope the current-main registered default-Pydantic nested-state restoration defect through existing #9250 and its latest review, not competing implementation. ADR 0034 requires explicitly registered nested state restoration and compatible tagged mappings; main default Pydantic decoding validates raw child tags before restoring a Message field, while encoding recursively tags the child. Preserve the equally intentional raw-payload/registered-codec boundary and test_registered_child_tag_does_not_hijack_message_payload. Current PR is now substantially larger than its original retry: the author reports complete-location alias/container restoration in f6078b7. Verify that candidate rather than repeat the older top-level/alias findings as unfixed. Decide the smallest maintainable restoration strategy without blindly decoding raw-valid dict/Any siblings or accumulating unsupported Pydantic internals. Cover nested model/list/mapping paths, aliases and validation aliases, registered-codec ownership, raw-valid siblings, custom validators and non-destructive failure/unknown tags. Validate AgentSession dictionary and fresh/cold-start JSON/MessagePack round trips with explicit registrations and custom-codec controls; preserve existing implicit-registration compatibility without claiming unregistered cold-start guarantees. Agree default-codec scope, validator/retry behavior and production size before readiness or assignment; a recursive one-liner is not sufficient. No runtime reproduction or PR-test execution is claimed by this triage plan.

  7. eavanvalkenburg commented on Nov 12, 2026

    @eavanvalkenburg
    Member

    /plan Scope the current-main registered default-Pydantic nested-state restoration defect through existing #9250 and its latest review, not competing implementation. ADR 0034 requires explicitly registered nested state restoration and compatible tagged mappings; main default Pydantic decoding validates raw child tags before restoring a Message field, while encoding recursively tags the child. Preserve the equally intentional raw-payload/registered-codec boundary and test_registered_child_tag_does_not_hijack_message_payload. Current PR is now substantially larger than its original retry: the author reports complete-location alias/container restoration in f6078b7. Verify that candidate rather than repeat the older top-level/alias findings as unfixed. Decide the smallest maintainable restoration strategy without blindly decoding raw-valid dict/Any siblings or accumulating unsupported Pydantic internals. Cover nested model/list/mapping paths, aliases and validation aliases, registered-codec ownership, raw-valid siblings, custom validators and non-destructive failure/unknown tags. Validate AgentSession dictionary and fresh/cold-start JSON/MessagePack round trips with explicit registrations and custom-codec controls; preserve existing implicit-registration compatibility without claiming unregistered cold-start guarantees. Agree default-codec scope, validator/retry behavior and production size before readiness or assignment; a recursive one-liner is not sufficient. No runtime reproduction or PR-test execution is claimed by this triage plan.

  8. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Summary

    A default-registered Pydantic model with a Message field saves fine but can't be restored. The encoder turns the child into a tagged dict, and the default decoder passes that dict to model_validate before the child is rebuilt. I read the current main code in _sessions.py. I did not run anything. I also did not read #9250's diff or reviews, per this workflow's rules, so I can't confirm the f6078b7 claims. They need the author's tests plus maintainer review.

    Findings from main

    • Encoding: _serialize_value recurses into registered payloads, so nested children get their own type tag (_sessions.py).
    • Decoding: _deserialize_value finds the parent's tag and calls its decoder with the raw mapping. It only recurses into children when no registered decoder claims the mapping. The default Pydantic decoder only strips type and calls model_validate. Nothing restores the nested child.
    • Raw-payload boundary: custom decoders receive raw tagged payloads on purpose. test_registered_child_tag_does_not_hijack_message_payload guards this. A blanket recursive pre-decode would break it for default-codec models too.
    • ADR reference: the nested-state restoration contract is in ADR 0038, not 0034. 0034 is the hosted-platform ADR. Please use 0038 in the PR and review notes.

    Recommendation

    Code change, narrowly scoped to the default Pydantic decoder only. The bug is real and general, and the current behavior is unreadable-after-save with no warning. The explicit parent codec workaround works but is easy to miss. Rejecting the combination at registration isn't practical, since the field types are arbitrary.

    Preferred strategy: error-guided restore.

    1. Validate the raw payload first, as today.
    2. If validation fails, read the public ValidationError.errors() (loc, input, and ctx). For each failing location whose input is a mapping with a registered tag, restore only that value with _deserialize_value.
    3. Retry, with a small fixed bound (for example 2 rounds).
    4. If it still fails, raise the original error. Unknown tags stay raw.

    Why this fits the constraints:

    • Raw-valid siblings are untouched. dict/Any fields that validate as raw dicts are never decoded, so nothing is silently reinterpreted.
    • No Pydantic internals. It doesn't walk annotations or core schemas, and uses no private APIs.
    • Containers and aliases come for free. loc gives paths through lists and mappings, and Pydantic reports input-side names, so validation aliases work. Tests must prove this rather than assume it.
    • Custom codecs unchanged. Only the default decoder changes.

    Alternatives I'd reject:

    • A recursive pre-decode (the "one-liner"): it breaks the raw-valid sibling boundary and Any semantics.
    • Annotation walking: it needs Pydantic type-introspection internals and grows with every typing construct.

    Decisions to agree before readiness or assignment

    1. Any/dict[str, Any] fields holding a registered object. The strategy leaves them as raw tagged dicts. That matches the raw-payload boundary but isn't a typed round trip. Is documenting this limit acceptable? I think yes.
    2. Validator and retry behavior. Validators may run more than once, and model_validators that fail with an empty loc can't be guided. Should that case simply fail non-destructively, with the original error preserved? I think yes, with the bound documented.
    3. Production size. I expect roughly 40–60 lines in one helper next to decode_pydantic, with no new public API. Any larger candidate should justify the extra code against this baseline.
    4. Implicit-registration compatibility. Implicitly registered (unregistered) Pydantic models keep today's path. No cold-start guarantee should be claimed for them, since the registration won't exist in a fresh process.

    Proposed plan (not implemented)

    1. Add the bounded error-guided restore to decode_pydantic in _sessions.py. Restore only mappings whose type is registered, using the existing _deserialize_value error paths. Keep unknown tags raw, and raise the original error if retries are exhausted.

    2. Add tests in test_sessions.py for:

      • A direct child, a list[Message], and a dict[str, Message].
      • A nested registered model inside a registered model.
      • Field and validation aliases.
      • Raw-valid siblings left untouched: Any, dict[str, Any], and a dict that merely looks tagged.
      • A custom validator that rejects the input, with a non-destructive failure.
      • An unknown tag passed through as a raw dict.
      • AgentSession.from_dict(to_dict()).
      • Fresh FileSessionStore reads in JSON and MessagePack, with module-level explicit registrations.
      • Custom-codec controls, still receiving raw payloads.
      • The existing no-hijack test, unchanged.
      • An implicit-registration compatibility test (same process only).
    3. Add a short note to the registration docs: default Pydantic decoding restores registered children at fields that reject raw dicts, and Any-typed fields stay raw. Mention the explicit-codec option.

    4. Done when: the issue's reproduction round-trips in both formats, the new tests pass, and all existing session tests pass unchanged.

    This is a proposal only. No changes have been made.

    Generated by Issue Planning for #9246 · copilot · auto · ~22 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions