Skip to content

Python: [Bug]: todo mutations race when two providers share one todo store #9868

Open

Description

Observed Behavior

TodoProvider serializes its read-modify-write todo operations with a lock that belongs to the provider instance. When two agents each use their own provider over one shared todo store, the concurrent todos_add calls take different locks, both load the same state and the same next_id, and the last save silently drops the other write while both tool calls report success for the same item id.

The lock is also keyed by session only, not by event loop, so a session reused across asyncio.run boundaries can raise RuntimeError from the loop bound lock, the hazard the file access layer documents and guards against with per loop lock keys.

Expected Behavior

Concurrent todo mutations over one shared store serialize exactly like mutations from a single provider, with distinct ids and no dropped items.

Steps to Reproduce

  1. Create one TodoFileStore and two TodoProvider instances over it, sharing one AgentSession.
  2. Run both providers' todos_add tool concurrently, with the store's load_state gated so both loads complete before either saves.
  3. Read the todos back: one of the two items is missing, or two items share id 1.

Minimal Reproduction

import asyncio, json, tempfile
from pathlib import Path
from agent_framework import AgentSession, TodoFileStore, TodoProvider

class _Ctx:  # minimal context capturing the registered tools
    def __init__(self): self.tools = []
    def extend_instructions(self, *a, **k): pass
    def extend_tools(self, source_id, tools): self.tools.extend(tools)
    def extend_messages(self, *a, **k): pass

async def main() -> None:
    with tempfile.TemporaryDirectory() as td:
        store = TodoFileStore(Path(td))
        session = AgentSession(session_id="session-1")
        toolsets = []
        for _ in range(2):
            ctx = _Ctx()
            await TodoProvider(store=store).before_run(agent=None, session=session, context=ctx, state={})
            toolsets.append({t.name: t for t in ctx.tools if hasattr(t, "name")})

        gate = asyncio.Event()   # arm after tool capture: before_run loads state itself
        original_load = store.load_state
        async def gated_load(session, *, source_id):
            await gate.wait()
            return await original_load(session, source_id=source_id)
        store.load_state = gated_load

        async def add(tools, title):
            await tools["todos_add"].invoke(arguments={"todos": [{"title": title}]})

        first = asyncio.create_task(add(toolsets[0], "From provider A"))
        second = asyncio.create_task(add(toolsets[1], "From provider B"))
        await asyncio.sleep(0)
        gate.set()
        await asyncio.wait_for(asyncio.gather(first, second), timeout=10)

        result = await toolsets[0]["todos_get_all"].invoke()
        titles = sorted(item["title"] for item in json.loads(result[0].text))
        assert len(titles) == 1  # one write silently dropped, both reported success

asyncio.run(main())

Error Messages and Stack Traces

No exception; the failure is silent data loss (a dropped item or a duplicate id reported as two successes).

Package Versions

agent-framework-core: 1.23.0 (main @ 4d28d0a)

Python Version

Python 3.14.8

Operating System

macOS

Regression

Unknown

Additional Context

The fix direction is to make the lock belong to the store and session rather than to one provider, keyed per event loop, mirroring _store_write_lock in the file access layer. I have that fix ready once the team agrees with the direction.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 30, 2026
  2. eavanvalkenburg commented on Oct 30, 2026

    @eavanvalkenburg
    Member

    please use the defined issue templates and wait for the team to review before opening a PR Manohar Paturi (@ManoharPaturi)

  3. changed the title [-]Python: todo mutations race when two providers share one todo store[/-] [+]Python: [Bug]: todo mutations race when two providers share one todo store[/+] on Oct 31, 2026
  4. eavanvalkenburg commented on Nov 2, 2026

    @eavanvalkenburg
    Member

    /plan Plan the confirmed TodoProvider lost-update/duplicate-ID race across multiple providers over shared state on current main, coordinating active #9920 and the issue discussion rather than creating another implementation. Current mutation locks are provider-owned, so equivalent backing state can be mutated under different locks. Establish the narrow backing-state/session/source identity contract for TodoSessionStore, TodoFileStore, equivalent store/session objects, and custom stores. Separate same-loop serialization and sequential event-loop reuse from any claimed concurrent cross-loop/process guarantees. Check cancellation while a threaded save is in flight so mutation ownership cannot end before the write completes. Evaluate the existing PR identity hook, weak lock lifetime, and cancellation approach for the simplest correct compatible design; do not assume public extension points are required. Use event-gated deterministic tests for overlapping adds, distinct IDs/no loss, equivalent file paths, independent sessions, loop reuse, and cancelled saves. Retain triage, do not add ready or assign while this concurrency contract is unresolved. The reporter original PR is already closed and the active PR has a different author; coordinate without reopening the earlier duplicate work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions