Repository navigation
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the test-only update to lineage-aware checkpoint selection.
No open issues 🎉
What changed in this PR
This test-only change addresses the checkpoint-selection flake reported in #10293 by using lineage-aware selection when resuming the approval workflow. Product code is unchanged.
Changes:
- Replace timestamp-only
maxselection withstorage.get_latestto resolve checkpoint timestamp ties. - Assert that a checkpoint exists and document why lineage-aware selection is needed.
| File | Description |
|---|---|
python/packages/declarative/tests/test_invoke_mcp_tool_executor.py |
Use lineage-aware checkpoint selection in the authorization-context resume test. |
Review effort: Balanced
Nick (surlymochan)
left a comment
There was a problem hiding this comment.
Reviewed the diff (non-binding review, no approve).
Verified: the failure mode in the comment is plausible and the fix uses the right API — several superstep checkpoints can share one host clock tick (notably on Windows), and a plain max(checkpoints, key=timestamp) over ISO strings returns the first-saved checkpoint in storage order, so a restore can start from stale state. storage.get_latest(workflow_name=...) is the lineage-aware selector that resolves ties through previous_checkpoint_id (present in the core checkpoint storage), and the added assert checkpoint is not None makes the previously implicit non-empty assumption explicit.
This is test-only, so no production behaviour changes — it makes the test exercise the same selection path production restores should use, instead of re-implementing max-timestamp selection inline. No concerns.
Motivation & Context
TestApprovalFlow.test_auth_context_binding_survives_workflow_resume[True-True]failed once in the
Python Tests (3.12, windows-latest)CI job on an unrelatedREADME PR (#10293): at
python/packages/declarative/tests/test_invoke_mcp_tool_executor.py:471the test saw
handler.call_count == 1where the changed authorization contextrequires
0and a fresh approval request.Diagnosis: the test selects the checkpoint to restore with
WorkflowCheckpoint.timestampis an ISO 8601 string produced bydatetime.now(timezone.utc).isoformat()at save time. Several superstepcheckpoints of one run are routinely taken within the same host clock tick, so
their timestamp strings are identical (reproduced locally on Windows: the three
checkpoints saved by one workflow run shared
21:12:25.956597). Python'smaxreturns the first maximal element, and
InMemoryCheckpointStorage.list_checkpointsreturns checkpoints in save order, so on a tie the expression returns the
earliest tied checkpoint instead of the newest one. Restoring from that
stale checkpoint resumes the workflow with the pre-response state
(
Local.selector == "first"), the approval then matches the stalecontext-firstheader binding, and the MCP tool is invoked although theauthorization context had changed — exactly the observed CI failure. The
specific
[3.12, windows-latest]job failed because slow host clock ticks makesame-tick checkpoint saves far more likely there.
The framework already solves this problem:
_select_latest_checkpoint(used byCheckpointStorage.get_latest) resolves timestamp ties through theprevious_checkpoint_idlineage chain and is what the runner itself relies onwhen resuming from the latest checkpoint.
Description & Review Guide
with
await storage.get_latest(workflow_name=workflow.name)(lineage-awaretie-breaking) instead of a plain
maxover timestamp strings, plus a commentexplaining why. Test-only change; no product code is touched.
unrelated PRs in the Windows/Python 3.12 CI job. No behavior change outside
the test.
get_latestis theright selection semantics for the test's intent (the newest checkpoint, which
is what the test asserts against) — it is the same selection the runner uses
for implicit resume.
Related Issue
Fixes #10293
The issue's Desired Outcome is to determine whether the failure reproduces and
make the test reliable so it does not block unrelated PRs. This PR is the test
diagnosis and reliability fix for that. (Gate note: the issue currently carries
the
python/triagelabels withoutready-for-implementation; a claimcomment with the diagnosis accompanies this PR.)
Observation for a possible follow-up (deliberately NOT part of this PR): the
same naive
max(..., key=timestamp)pattern also exists inpython/packages/declarative/tests/test_graph_executors.py:2273,python/packages/core/tests/workflow/test_sub_workflow.py:845,python/packages/core/tests/workflow/test_runner.py:1411, andpython/packages/core/tests/workflow/test_workflow_kwargs.py:916.Validation
Reproduced on Windows 11 / Python 3.12.1 with a clock-scripting harness
(recorded in the evidence log
af-10293-force-ties-plugin.py.txt) that pinsrun-1's last checkpoint and run-2's checkpoints to the same instant — the
situation that occurs naturally when their saves land in one host clock tick:
tests/test_invoke_mcp_tool_executor.py:471: AssertionError — assert 1 == 0,byte-for-byte the CI failure signature from Python: [Issue]: MCP workflow resume test failed in Python 3.12 Windows CI #10293.
passes 46/46 (
46 passed); the target test passes 10/10 repeated runs; allfour parametrizations pass under the forced-tie harness.
AI Assistance
AI assistance details: AI tools assisted with the failure diagnosis (checkpoint
tie mechanism), the reproduction harness, drafting this PR description, and the
test fix. All steps, the diagnosis, the reproduction evidence, and the submitted
diff were reviewed and verified by the human contributor.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.