Skip to content

Python: .NET: [Bug]: Failed supersteps leak pending state and prevent checkpoint restore #9922

Description

Observed Behavior

After a .NET workflow executor queues a state write and then fails or is cancelled, the pending write remains in StateManager. On the same lockstep Run:

  • A later input observes the failed step's value and can publish it in a later successful step.
  • RestoreCheckpointAsync throws InvalidOperationException: Cannot import state while there are queued updates. Call PublishUpdatesAsync() first.

Expected Behavior

A failed or cancelled superstep should discard its unpublished workflow-state writes. Earlier committed state should remain usable, and restoring a saved checkpoint should not be blocked by the failed step's pending writes. This concerns framework-managed queued state, not external side effects.

Steps to Reproduce

Create a .NET 10 console application referencing Microsoft.Agents.AI.Workflows 1.27.0. Run the code below. Input 0 succeeds and creates a checkpoint. Input 1 queues a state update and throws. Restore fails, and input 2 reads 1 instead of 0.

Minimal Reproduction

using Microsoft.Agents.AI.Workflows;
using Microsoft.Agents.AI.Workflows.Checkpointing;

Workflow workflow = new WorkflowBuilder(new Worker()).Build();
var manager = CheckpointManager.CreateInMemory();
await using Run run = await InProcessExecution.Lockstep
    .WithCheckpointing(manager).RunAsync(workflow, 0);
CheckpointInfo saved = run.Checkpoints.Last();

await run.ResumeAsync<int>(default, 1);
try
{
    await run.RestoreCheckpointAsync(saved);
}
catch (InvalidOperationException error)
{
    Console.WriteLine(error.Message);
}
await run.ResumeAsync<int>(default, 2);

sealed class Worker() : Executor<int>("worker")
{
    public override async ValueTask HandleAsync(
        int message, IWorkflowContext context,
        CancellationToken cancellationToken = default)
    {
        int? previous = await context.ReadStateAsync<int?>("value");
        Console.WriteLine($"input={message}, previous={previous}");
        await context.QueueStateUpdateAsync("value", message);
        if (message == 1)
        {
            throw new InvalidOperationException("step failed");
        }
    }
}

Error Messages and Stack Traces

Observed locally with the equivalent reproduction:

[worker] input=0 sees k=null
[worker] input=1 sees k=0
restore to cp1 THREW InvalidOperationException: Cannot import state while there are queued updates. Call PublishUpdatesAsync() first.
[worker] input=2 sees k=1

Package Versions

Microsoft.Agents.AI.Workflows 1.27.0. Also reproduced with six new unit-test cases against repository source at b7884ad88.

.NET Version

SDK 10.0.401, target net10.0.

Operating System

macOS arm64.

Regression

Unknown.

Additional Context

InProcessRunner.RunSuperStepAsync catches failure/cancellation without discarding queued state. Successful steps publish it in CheckpointAsync. StateManager.DiscardQueuedUpdates() already exists and is used for failed checkpoint restoration.

I have a minimal fix calling that helper in the two superstep catch branches. Six regression cases all fail on unfixed main and pass with the fix: state retention with/without checkpointing, and restore after failure/cancellation. Cancellation tests cancel synchronously inside the executor, without timing sleeps.

Related but distinct: #8627/#8628 concern concurrent dictionary access; this defect occurs with a single executor. Python has analogous failed-step cleanup in #7859/#8819.

I intend to contribute the fix and welcome maintainer feedback.

AI assistance: investigation, reproduction, implementation, tests, and issue description.

Activity

  1. added
    .NETUsage: [Issues, PRs], Target: .Net
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 31, 2026
  2. changed the title [-].NET: [Bug]: Failed supersteps leak pending state and prevent checkpoint restore[/-] [+]Python: .NET: [Bug]: Failed supersteps leak pending state and prevent checkpoint restore[/+] on Oct 31, 2026
  3. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 31, 2026
  4. github-actions commented on Oct 31, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    dotnet/src/Microsoft.Agents.AI.Workflows/InProc/InProcessRunner.cs::ISuperStepRunner.RunSuperStepAsync leaves queued state after its exception and cancellation branches. Minimal repro: commit input 0, queue input 1 and throw, then restore the prior checkpoint or process input 2; restore throws at StateManager.ImportStateAsync, and input 2 observes 1 instead of 0.

    • Failing test: dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/InProcessStateTests.cs
    • Files examined: dotnet/src/Microsoft.Agents.AI.Workflows/InProcessRunner.cs, dotnet/src/Microsoft.Agents.AI.Workflows/Execution/StateManager.cs, dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/InProcessStateTests.cs, dotnet/nuget/nuget-package.props
    • Tests run: InProcessRun_FailedSuperstepShouldDiscardQueuedStateAsync, InProcessRun_RestoreAfterFailedSuperstepShouldSucceedAsync
    • Reported version: 1.27.0
    • Current version: 1.27.0
  5. added
    workflowsUsage: [Issues, PRs], Target: Workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 31, 2026
  6. sumedh-aerram commented on Nov 11, 2026

    @sumedh-aerram
    Author

    Peter Ibekwe (@peibekwe) could you confirm whether the direction in draft #9926 is acceptable before it goes further? In short: when a superstep fails (executor exception or hook cancellation), discard the pending state updates from that superstep instead of committing them, so a later checkpoint or resume doesn't observe partially applied state. Published state, edge buffers, external requests and executor fields are not rolled back; checkpoint restore remains the recovery boundary.

    If you'd prefer a different scope (or a /plan first), I'm happy to align #9926 to it or close it and open a fresh PR once the issue is marked ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

.NETUsage: [Issues, PRs], Target: .NetpythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions