Skip to content

Flaky test: pipeline job control timed out waiting for the terminal after quit (ubuntu, 3.11, direct-signals-and-job-control) #1822

Open

Description

@tleonhardt

Summary

tests/test_pipeline_job_control.py::test_pipeline_stops_with_cmd2_and_returns_terminal[direct-signals-and-job-control]
timed out once on the tests (ubuntu-latest, 3.11) CI job. A second run of the same workflow on the
same commit passed on all 18 jobs, and re-running the failed job passed, so this is intermittent.

This is a different failure from #1791, which was a race on macOS free-threaded Python at the first
Ctrl-Z and was fixed by #1794. This one is on Linux, on regular CPython 3.11, and happens at the very
end of the test.

Where it was seen

It has been seen once so far, and not on main. On that branch the test's application runs with
bottom_toolbar_mode=ToolbarMode.RESERVED, which main does not have. The step which timed out and
its 10 second limit are the same on main (tests/test_pipeline_job_control.py:426-428), so it is
not yet known whether this is specific to reserved rendering.

What fails

After the signal checks, the test asks the application to quit and waits for the terminal's
foreground process group to return to the outer bash:

send("help quit\n")
wait_until(lambda: "Exit this application" in transcript[start:])
send("quit\n")
if launcher != "exec":
    wait_until(lambda: os.tcgetpgrp(master) == process.pid)   # timed out after 10 seconds

From the failure output:

  • Everything before quit worked: both stop and resume cycles (12x80 and 24x80), all eight
    PAGER_INTERRUPT lines, and the help quit output.
  • quit was echoed and accepted. The transcript ends right after the prompt was accepted
    (ESC[?2004l ESC[6n). The sequence which gives the reserved row back
    (ESC7 ESC[24;1H ESC[0m ESC[J ESC[r ESC8), written when the earlier pipeline command was accepted,
    never appears after quit.
  • The application was still alone in the foreground, and its main thread was in an uninterruptible
    kernel wait on a page of a file (D, folio_), not waiting on a lock or stopped by a signal:
foreground process group: 2765
    PID    PPID    PGID STAT WCHAN  COMMAND
   2760    2334    2760 Ss   do_wai bash --noprofile --norc -i
   2765    2760    2765 Dl+  folio_ .../.venv/bin/python .../application.py
resizes: ['requested 12x80, read back (12, 80)', 'the shell saw 12x80',
          'requested 24x80, read back (24, 80)', 'the shell saw 24x80']
terminal size at timeout: (24, 80)

The process table is a single snapshot taken at the timeout, so it does not show whether the
application spent all 10 seconds in that wait or only happened to be there when it was sampled.

Environment

  • ubuntu-latest, Python 3.11.18, pytest 9.2.0, 4 xdist workers, coverage enabled in the application
    subprocess (without sys.monitoring, which 3.11 does not have)
  • Whole job: 1 failed, 3987 passed, 9 skipped in 72.46s

Open questions

  • Was the application only slow to exit on a loaded runner (disk I/O while exiting under coverage), in
    which case the fixed 10 second limit in wait_until is the weak point?
  • Or can the application hang between accepting quit and giving the terminal back, with the kernel
    wait being incidental?

One occurrence is not enough to tell these apart. If it happens again, the useful things to compare
are whether it is again Python 3.11 on Linux, whether the application is again in state D, and
whether it happens on a branch without reserved rendering. A process table sampled more than once at
the timeout would answer the first question directly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions