From d7ded845e9fe2c525f62d9eb3df1ee194d27ed08 Mon Sep 17 00:00:00 2001 From: Siddartha Pothapragada Date: Mon, 2 Nov 2026 20:27:36 -0800 Subject: [PATCH] Stop (coreml, models) hanging at the 180 minute CI cap Summary: `(coreml, models)` has been wedged at the **180 min** CI cap since Oct 2. It ran in **13 min** before that. **82.9%** of runs get cancelled with no verdict, which is **1,540 of the repo's 10,383** cancelled macOS hours. Bisected to #23326 "Preserve bounded dynamic input shapes in PTN" (last good b49d8232a9, first hung 3ebf896c38). That change stops the ShapeEnv clamping a declared `min=1` up to 2, so `test_torchvision.py`'s `Dim("height", min=1, max=16) * 16` started handing backends a 16x16 input to a stride-32 net. Only Core ML chokes on it - the other 12 backends drift by 1-3 min. Setting `min=2` declares the bound that was actually in force for the whole life of these tests, so the exercised range `[32, 256]` is unchanged. #23326 itself is correct and untouched. Also adding pytest's `faulthandler_timeout` so the next wedge leaves a traceback. Today a hung case leaves nothing at all - the job is killed at the cap before the JSON report is written - which is why this burned a month of runner time without anyone naming the stuck model. Why Core ML specifically, and why `coreml_static_int8` is unaffected, is still open.. couldn't repro without macOS/coremltools. Revert: restore `min=1` and drop the `-o` flag. Differential Revision: D128941936 --- .ci/scripts/test_backend.sh | 10 +++++++++- backends/test/suite/models/test_torchvision.py | 13 +++++++++++-- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/.ci/scripts/test_backend.sh b/.ci/scripts/test_backend.sh index a6dbeef9b12..d39ec3194f1 100755 --- a/.ci/scripts/test_backend.sh +++ b/.ci/scripts/test_backend.sh @@ -157,7 +157,15 @@ EXIT_CODE=0 # a step that size, taking the whole job down before pytest can report. The # reason for each failure is in its exception message and traceback, which are # unaffected. -PYTEST_ARGS=(-c /dev/null -n auto --show-capture=no) +# faulthandler_timeout is pytest's built-in stack dumper, so this adds no +# dependency and fails nothing: after 15 minutes in one test case it prints +# every thread's traceback and the case keeps running. A case that wedges +# otherwise leaves no trace at all - the job is cancelled at the job cap, the +# JSON report below is never written, and the artifact upload fails - which is +# how (coreml, models) spent a month at the 180 minute cap with no one able to +# say which model was stuck or where. Raise the value if a legitimate case ever +# runs this long. +PYTEST_ARGS=(-c /dev/null -o faulthandler_timeout=900 -n auto --show-capture=no) if [[ ${#PYTEST_RETRY_ARGS[@]} -gt 0 ]]; then PYTEST_ARGS+=("${PYTEST_RETRY_ARGS[@]}") fi diff --git a/backends/test/suite/models/test_torchvision.py b/backends/test/suite/models/test_torchvision.py index 58cf6a990d4..5fcd3d54287 100644 --- a/backends/test/suite/models/test_torchvision.py +++ b/backends/test/suite/models/test_torchvision.py @@ -40,11 +40,20 @@ def _test_cv_model( # Test a CV model that follows the standard conventions. inputs = (torch.randn(1, 3, 224, 224, dtype=dtype),) + # The declared minimum is 2, not 1. Until #23326 the exported program's + # range constraints were recomputed from the ShapeEnv after every pass, and + # the ShapeEnv raises a backed symbol's declared minimum of 0 or 1 to 2, so + # a declared min=1 never reached a backend: these models have always been + # lowered over height/width in [32, 256]. #23326 made ExecuTorch keep the + # declared range instead, which handed backends a lower bound of 16 for the + # first time and let a 16x16 input - degenerate for a network with a total + # stride of 32 - into backend shape reasoning. Declaring 2 states the bound + # that was always in force. dynamic_shapes = ( ( { - 2: Dim("height", min=1, max=16) * 16, - 3: Dim("width", min=1, max=16) * 16, + 2: Dim("height", min=2, max=16) * 16, + 3: Dim("width", min=2, max=16) * 16, }, ) if use_dynamic_shapes