Repository navigation
Description
Summary
tests/test_pipeline_job_control.py::test_pipeline_stops_with_cmd2_and_returns_terminal[direct-signals-and-job-control]
timed out once on the tests (ubuntu-latest, 3.11) CI job. A second run of the same workflow on the
same commit passed on all 18 jobs, and re-running the failed job passed, so this is intermittent.
This is a different failure from #1791, which was a race on macOS free-threaded Python at the first
Ctrl-Z and was fixed by #1794. This one is on Linux, on regular CPython 3.11, and happens at the very
end of the test.
Where it was seen
- Failed job (attempt 1):
https://github.com/python-cmd2/cmd2/actions/runs/43992951605/job/133737883817 - Passing run of the same workflow on the same commit:
https://github.com/python-cmd2/cmd2/actions/runs/43992949624 - Commit
0af637b5onstage5-pager-investigation(Stage 5: use the alternate screen for the built-in pager in reserved rendering #1815), a merge ofmaininto that branch. The
merge only brought in tab-completion changes from Match glob characters literally in completion and quote them for the shell #1812, which this test does not exercise.
It has been seen once so far, and not on main. On that branch the test's application runs with
bottom_toolbar_mode=ToolbarMode.RESERVED, which main does not have. The step which timed out and
its 10 second limit are the same on main (tests/test_pipeline_job_control.py:426-428), so it is
not yet known whether this is specific to reserved rendering.
What fails
After the signal checks, the test asks the application to quit and waits for the terminal's
foreground process group to return to the outer bash:
send("help quit\n")
wait_until(lambda: "Exit this application" in transcript[start:])
send("quit\n")
if launcher != "exec":
wait_until(lambda: os.tcgetpgrp(master) == process.pid) # timed out after 10 secondsFrom the failure output:
- Everything before
quitworked: both stop and resume cycles (12x80 and 24x80), all eight
PAGER_INTERRUPTlines, and thehelp quitoutput. quitwas echoed and accepted. The transcript ends right after the prompt was accepted
(ESC[?2004l ESC[6n). The sequence which gives the reserved row back
(ESC7 ESC[24;1H ESC[0m ESC[J ESC[r ESC8), written when the earlier pipeline command was accepted,
never appears afterquit.- The application was still alone in the foreground, and its main thread was in an uninterruptible
kernel wait on a page of a file (D,folio_), not waiting on a lock or stopped by a signal:
foreground process group: 2765
PID PPID PGID STAT WCHAN COMMAND
2760 2334 2760 Ss do_wai bash --noprofile --norc -i
2765 2760 2765 Dl+ folio_ .../.venv/bin/python .../application.py
resizes: ['requested 12x80, read back (12, 80)', 'the shell saw 12x80',
'requested 24x80, read back (24, 80)', 'the shell saw 24x80']
terminal size at timeout: (24, 80)
The process table is a single snapshot taken at the timeout, so it does not show whether the
application spent all 10 seconds in that wait or only happened to be there when it was sampled.
Environment
ubuntu-latest, Python 3.11.18, pytest 9.2.0, 4 xdist workers, coverage enabled in the application
subprocess (withoutsys.monitoring, which 3.11 does not have)- Whole job:
1 failed, 3987 passed, 9 skipped in 72.46s
Open questions
- Was the application only slow to exit on a loaded runner (disk I/O while exiting under coverage), in
which case the fixed 10 second limit inwait_untilis the weak point? - Or can the application hang between accepting
quitand giving the terminal back, with the kernel
wait being incidental?
One occurrence is not enough to tell these apart. If it happens again, the useful things to compare
are whether it is again Python 3.11 on Linux, whether the application is again in state D, and
whether it happens on a branch without reserved rendering. A process table sampled more than once at
the timeout would answer the first question directly.
Activity