Repository navigation
Conversation
Summary: `(coreml, models)` has been wedged at the **180 min** CI cap since Oct 2. It ran in **13 min** before that. **82.9%** of runs get cancelled with no verdict, which is **1,540 of the repo's 10,383** cancelled macOS hours. Bisected to pytorch#23326 "Preserve bounded dynamic input shapes in PTN" (last good b49d823, first hung 3ebf896). That change stops the ShapeEnv clamping a declared `min=1` up to 2, so `test_torchvision.py`'s `Dim("height", min=1, max=16) * 16` started handing backends a 16x16 input to a stride-32 net. Only Core ML chokes on it - the other 12 backends drift by 1-3 min. Setting `min=2` declares the bound that was actually in force for the whole life of these tests, so the exercised range `[32, 256]` is unchanged. pytorch#23326 itself is correct and untouched. Also adding pytest's `faulthandler_timeout` so the next wedge leaves a traceback. Today a hung case leaves nothing at all - the job is killed at the cap before the JSON report is written - which is why this burned a month of runner time without anyone naming the stuck model. Why Core ML specifically, and why `coreml_static_int8` is unaffected, is still open.. couldn't repro without macOS/coremltools. Revert: restore `min=1` and drop the `-o` flag. Differential Revision: D128941936
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24482
Note: Links to docs will display an error until the docs builds have been completed. ⏳ 5 Pending, 1 Unrelated FailureAs of commit d7ded84 with merge base 0f8e096 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128941936. |
This PR needs a
|
There was a problem hiding this comment.
🔵 Needs a closer look
The Core ML hang and its root cause remain unidentified, and macOS CI validation is needed to confirm the proposed bound resolves it and exposes timeout diagnostics.
No open issues 🎉
What changed in this PR
This PR aims to prevent Core ML model tests from reaching the 180-minute CI limit by restoring the intended dynamic input bounds and adding diagnostics for stalled tests.
Changes:
- Raise torchvision height and width minima from 16 to 32 pixels across backends, preserving the 256-pixel maximum.
- Enable pytest traceback dumps after 15 minutes in a test, without terminating it.
| File | Description |
|---|---|
backends/test/suite/models/test_torchvision.py |
Adjusts dynamic shape minima and documents the historical bounds. |
.ci/scripts/test_backend.sh |
Enables faulthandler_timeout=900 to diagnose stalled tests. |
Review effort: Balanced
Summary:
(coreml, models)has been wedged at the 180 min CI cap since Oct 2. It ran in 13 min before that. 82.9% of runs get cancelled with no verdict, which is 1,540 of the repo's 10,383 cancelled macOS hours.Bisected to #23326 "Preserve bounded dynamic input shapes in PTN" (last good b49d823, first hung 3ebf896). That change stops the ShapeEnv clamping a declared
min=1up to 2, sotest_torchvision.py'sDim("height", min=1, max=16) * 16started handing backends a 16x16 input to a stride-32 net. Only Core ML chokes on it - the other 12 backends drift by 1-3 min.Setting
min=2declares the bound that was actually in force for the whole life of these tests, so the exercised range[32, 256]is unchanged. #23326 itself is correct and untouched.Also adding pytest's
faulthandler_timeoutso the next wedge leaves a traceback. Today a hung case leaves nothing at all - the job is killed at the cap before the JSON report is written - which is why this burned a month of runner time without anyone naming the stuck model.Why Core ML specifically, and why
coreml_static_int8is unaffected, is still open.. couldn't repro without macOS/coremltools.Revert: restore
min=1and drop the-oflag.Differential Revision: D128941936