Skip to content

fix(testing): bound every smoke run with a timeout and decode output as UTF-8 - #416

Merged
TMHSDigital merged 1 commit into
mainfrom
fix/smoke-timeout
Oct 5, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
fix/smoke-timeout

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Closes #402. Refs #403: this fixes the encoding crash and the stale expect_file. Status-file truncation is deliberately not done, see below.

Problem

  • About 284 sequential Blender runs shared one 75-minute job timeout, with no per-run limit, so one hang ate the whole job.
  • The status record was only written after wait(), so the summary never named the example that hung.
  • On Windows, Blender's UTF-8 output was decoded with cp1252. A byte such as 0x9D raised UnicodeDecodeError in the read loop, which crashed the runner and left Blender running.

Change

  • run_example.py:
    • new stream(): a reader thread plus proc.wait(timeout=…);
    • --timeout (default $BDT_SMOKE_TIMEOUT or 900 s);
    • on expiry, kills the whole process group (start_new_session + killpg; taskkill /T on Windows) and records FAIL "timeout after Ns (process group killed)";
    • output decoded as UTF-8 with replacement.
  • run_catalog.py:
    • forwards a row's optional timeout;
    • deletes a row's expect_file before running it, so a render left by an earlier local run can't satisfy the check.
  • check_smoke_catalog.py: accepts timeout and requires it to be a positive number.
  • check_falsifier_targets.py --run (the weekly sweep): a 900 s bound per run; a timeout counts as a mismatch and the sweep continues.
  • Not done: truncating --status in run_catalog. CI's canary and template steps append to the same JSONL before the catalog runs, so truncating it would erase their records. CI is unaffected by stale records because RUNNER_TEMP is fresh.

Evidence

Unit tests (tests/smoke/test_harness.py, Python 3.13, Windows): 40 tests OK, 1 skipped (the POSIX grandchild-kill test, which runs on Linux CI). New tests:

  • exit code returned;
  • a hang is killed and returns None in under 30 s, with output captured;
  • grandchild killed (POSIX);
  • b'\xe2\x80\x9d' decodes to U+201D;
  • a row's timeout is forwarded;
  • a stale expect_file no longer satisfies a row.

Falsified:

  • With encoding="cp1252", errors="strict" restored, the UTF-8 test fails with UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d.
  • With the old untimed proc.wait() restored, the hang test hangs until killed externally, which is the bug.

Live run with real Blender (.scratch binary reporting 5.2.1 LTS):

Run Result
bmesh-gear via run_example.py [PASS], exit 0
a script that prints then time.sleep(120), --timeout 5 [FAIL] hang-probe: timeout after 5s (process group killed) after 5 s. Get-Process blender count afterwards: 0. JSONL record proc_exit: null. summarize.py tables it as FAIL with the reason

check_smoke_catalog.py and check_falsifier_targets.py pass.

🤖 Generated with Claude Code

…as UTF-8

About 284 sequential Blender runs shared one 75-minute job timeout and
no per-run limit, so a single hang (Cycles, xvfb, a stuck handler) ate
the whole job, and because the status record was written only after
wait(), the summary never named the example that hung.

run_example.py now streams through a reader thread and waits with a
timeout (--timeout, default $BDT_SMOKE_TIMEOUT or 900 s). On expiry it
kills the whole process group (xvfb-run and Blender; taskkill /T on
Windows), records FAIL "timeout after Ns", and the catalog moves on. A
catalog row may set "timeout" for a legitimately slow example; the
catalog lint validates it. The weekly falsifier sweep gets the same bound.

Output is now decoded as UTF-8 with replacement. Blender writes UTF-8,
and the locale codec (cp1252 on Windows) raised UnicodeDecodeError on
bytes such as 0x9D, crashing the runner with Blender still alive.
run_catalog also deletes a row's expect_file before running it, so a
render left by an earlier local run cannot satisfy the check.

Closes #402
Refs #403

Signed-off-by: TMHSDigital <154358121+TMHSDigital@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TMHSDigital

Copy link
Copy Markdown
Owner Author

CI evidence before merge: all checks pass ( 13 pass ). Blender Smoke on PR head: Blender 5.2.2 LTS / Blender 4.5.14 LTS. Summaries (happy path; falsifiers) per leg: 143 passed, 1 skipped, 0 failed;142 passed, 0 skipped, 0 failed;141 passed, 3 skipped, 0 failed;136 passed, 6 skipped, 0 failed;

@TMHSDigital
TMHSDigital merged commit 192e1b1 into main Oct 5, 2026
13 checks passed
@TMHSDigital
TMHSDigital deleted the fix/smoke-timeout branch October 5, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

testing: smoke harness has no per-run timeout; one hung Blender burns the whole 75-minute job with no record of the culprit

1 participant