Skip to content

Python: resolve checkpoint timestamp ties in the approval resume test - #10369

Open
DZH166 wants to merge 1 commit into
microsoft:mainfrom
DZH166:codex/af-10293-checkpoint-tie-20261112
Open

DZH166 wants to merge 1 commit into
microsoft:mainfrom
DZH166:codex/af-10293-checkpoint-tie-20261112

Conversation

@DZH166

@DZH166 DZH166 commented Nov 14, 2026

Copy link
Copy Markdown

Motivation & Context

TestApprovalFlow.test_auth_context_binding_survives_workflow_resume[True-True]
failed once in the Python Tests (3.12, windows-latest) CI job on an unrelated
README PR (#10293): at python/packages/declarative/tests/test_invoke_mcp_tool_executor.py:471
the test saw handler.call_count == 1 where the changed authorization context
requires 0 and a fresh approval request.

Diagnosis: the test selects the checkpoint to restore with

checkpoint = max(checkpoints, key=lambda item: item.timestamp)

WorkflowCheckpoint.timestamp is an ISO 8601 string produced by
datetime.now(timezone.utc).isoformat() at save time. Several superstep
checkpoints of one run are routinely taken within the same host clock tick, so
their timestamp strings are identical (reproduced locally on Windows: the three
checkpoints saved by one workflow run shared 21:12:25.956597). Python's max
returns the first maximal element, and InMemoryCheckpointStorage.list_checkpoints
returns checkpoints in save order, so on a tie the expression returns the
earliest tied checkpoint instead of the newest one. Restoring from that
stale checkpoint resumes the workflow with the pre-response state
(Local.selector == "first"), the approval then matches the stale
context-first header binding, and the MCP tool is invoked although the
authorization context had changed — exactly the observed CI failure. The
specific [3.12, windows-latest] job failed because slow host clock ticks make
same-tick checkpoint saves far more likely there.

The framework already solves this problem: _select_latest_checkpoint (used by
CheckpointStorage.get_latest) resolves timestamp ties through the
previous_checkpoint_id lineage chain and is what the runner itself relies on
when resuming from the latest checkpoint.

Description & Review Guide

  • What are the major changes? The test now selects the restore checkpoint
    with await storage.get_latest(workflow_name=workflow.name) (lineage-aware
    tie-breaking) instead of a plain max over timestamp strings, plus a comment
    explaining why. Test-only change; no product code is touched.
  • What is the impact of these changes? Removes a real flake that cancelled
    unrelated PRs in the Windows/Python 3.12 CI job. No behavior change outside
    the test.
  • What do you want reviewers to focus on? Whether get_latest is the
    right selection semantics for the test's intent (the newest checkpoint, which
    is what the test asserts against) — it is the same selection the runner uses
    for implicit resume.

Related Issue

Fixes #10293

The issue's Desired Outcome is to determine whether the failure reproduces and
make the test reliable so it does not block unrelated PRs. This PR is the test
diagnosis and reliability fix for that. (Gate note: the issue currently carries
the python/triage labels without ready-for-implementation; a claim
comment with the diagnosis accompanies this PR.)

Observation for a possible follow-up (deliberately NOT part of this PR): the
same naive max(..., key=timestamp) pattern also exists in
python/packages/declarative/tests/test_graph_executors.py:2273,
python/packages/core/tests/workflow/test_sub_workflow.py:845,
python/packages/core/tests/workflow/test_runner.py:1411, and
python/packages/core/tests/workflow/test_workflow_kwargs.py:916.

Validation

Reproduced on Windows 11 / Python 3.12.1 with a clock-scripting harness
(recorded in the evidence log af-10293-force-ties-plugin.py.txt) that pins
run-1's last checkpoint and run-2's checkpoints to the same instant — the
situation that occurs naturally when their saves land in one host clock tick:

  • Red (pre-fix, harness forcing the tie):
    tests/test_invoke_mcp_tool_executor.py:471: AssertionError — assert 1 == 0,
    byte-for-byte the CI failure signature from Python: [Issue]: MCP workflow resume test failed in Python 3.12 Windows CI #10293.
  • Green (post-fix): the same forced-tie run passes; the unmodified test file
    passes 46/46 (46 passed); the target test passes 10/10 repeated runs; all
    four parametrizations pass under the forced-tie harness.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: AI tools assisted with the failure diagnosis (checkpoint
tie mechanism), the reproduction harness, drafting this PR description, and the
test fix. All steps, the diagnosis, the reproduction evidence, and the submitted
diff were reviewed and verified by the human contributor.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-only update to lineage-aware checkpoint selection.

No open issues 🎉

What changed in this PR

This test-only change addresses the checkpoint-selection flake reported in #10293 by using lineage-aware selection when resuming the approval workflow. Product code is unchanged.

Changes:

  • Replace timestamp-only max selection with storage.get_latest to resolve checkpoint timestamp ties.
  • Assert that a checkpoint exists and document why lineage-aware selection is needed.
File Description
python/​packages/​declarative/​tests/​test_invoke_mcp_tool_executor.py Use lineage-aware checkpoint selection in the authorization-context resume test.

Review effort: Balanced

@surlymochan Nick (surlymochan) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff (non-binding review, no approve).

Verified: the failure mode in the comment is plausible and the fix uses the right API — several superstep checkpoints can share one host clock tick (notably on Windows), and a plain max(checkpoints, key=timestamp) over ISO strings returns the first-saved checkpoint in storage order, so a restore can start from stale state. storage.get_latest(workflow_name=...) is the lineage-aware selector that resolves ties through previous_checkpoint_id (present in the core checkpoint storage), and the added assert checkpoint is not None makes the previously implicit non-empty assumption explicit.

This is test-only, so no production behaviour changes — it makes the test exercise the same selection path production restores should use, instead of re-implementing max-timestamp selection inline. No concerns.

This branch was successfully deployed

1 active deployment
github-app-auth — 022aba2f Deployed Nov 14, 2026 by DZH166 via team_check #9492
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Issue]: MCP workflow resume test failed in Python 3.12 Windows CI

3 participants