Skip to content

Stop (coreml, models) hanging at the 180 minute CI cap - #24482

Open
psiddh wants to merge 1 commit into
pytorch:mainfrom
psiddh:export-D128941936
Open

psiddh wants to merge 1 commit into
pytorch:mainfrom
psiddh:export-D128941936

Conversation

@psiddh

@psiddh psiddh commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary:
(coreml, models) has been wedged at the 180 min CI cap since Oct 2. It ran in 13 min before that. 82.9% of runs get cancelled with no verdict, which is 1,540 of the repo's 10,383 cancelled macOS hours.

Bisected to #23326 "Preserve bounded dynamic input shapes in PTN" (last good b49d823, first hung 3ebf896). That change stops the ShapeEnv clamping a declared min=1 up to 2, so test_torchvision.py's Dim("height", min=1, max=16) * 16 started handing backends a 16x16 input to a stride-32 net. Only Core ML chokes on it - the other 12 backends drift by 1-3 min.

Setting min=2 declares the bound that was actually in force for the whole life of these tests, so the exercised range [32, 256] is unchanged. #23326 itself is correct and untouched.

Also adding pytest's faulthandler_timeout so the next wedge leaves a traceback. Today a hung case leaves nothing at all - the job is killed at the cap before the JSON report is written - which is why this burned a month of runner time without anyone naming the stuck model.

Why Core ML specifically, and why coreml_static_int8 is unaffected, is still open.. couldn't repro without macOS/coremltools.

Revert: restore min=1 and drop the -o flag.

Differential Revision: D128941936

Summary:
`(coreml, models)` has been wedged at the **180 min** CI cap since Oct 2. It ran in **13 min** before that. **82.9%** of runs get cancelled with no verdict, which is **1,540 of the repo's 10,383** cancelled macOS hours.

Bisected to pytorch#23326 "Preserve bounded dynamic input shapes in PTN" (last good b49d823, first hung 3ebf896). That change stops the ShapeEnv clamping a declared `min=1` up to 2, so `test_torchvision.py`'s `Dim("height", min=1, max=16) * 16` started handing backends a 16x16 input to a stride-32 net. Only Core ML chokes on it - the other 12 backends drift by 1-3 min.

Setting `min=2` declares the bound that was actually in force for the whole life of these tests, so the exercised range `[32, 256]` is unchanged. pytorch#23326 itself is correct and untouched.

Also adding pytest's `faulthandler_timeout` so the next wedge leaves a traceback. Today a hung case leaves nothing at all - the job is killed at the cap before the JSON report is written - which is why this burned a month of runner time without anyone naming the stuck model.

Why Core ML specifically, and why `coreml_static_int8` is unaffected, is still open.. couldn't repro without macOS/coremltools.

Revert: restore `min=1` and drop the `-o` flag.

Differential Revision: D128941936
Copilot AI balanced review requested due to automatic review settings November 3, 2026 04:27
@pytorch-bot

pytorch-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24482

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 5 Pending, 1 Unrelated Failure

As of commit d7ded84 with merge base 0f8e096 (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 3, 2026
@meta-codesync

meta-codesync Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128941936.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Core ML hang and its root cause remain unidentified, and macOS CI validation is needed to confirm the proposed bound resolves it and exposes timeout diagnostics.

No open issues 🎉

What changed in this PR

This PR aims to prevent Core ML model tests from reaching the 180-minute CI limit by restoring the intended dynamic input bounds and adding diagnostics for stalled tests.

Changes:

  • Raise torchvision height and width minima from 16 to 32 pixels across backends, preserving the 256-pixel maximum.
  • Enable pytest traceback dumps after 15 minutes in a test, without terminating it.
File Description
backends/​test/​suite/​models/​test_torchvision.py Adjusts dynamic shape minima and documents the historical bounds.
.ci/​scripts/​test_backend.sh Enables faulthandler_timeout=900 to diagnose stalled tests.

Review effort: Balanced

This branch was successfully deployed

1 active deployment
cadence — d7ded845 Deployed Nov 3, 2026 by pytorch-bot[bot] via hifi-op-test / hifi4 #36191
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/nightly CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants