Skip to content

gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths - #154837

Merged
gpshead merged 1 commit into
python:mainfrom
calvinrp:fix-subprocess-wait-timeout-overflow
Jul 29, 2026
Merged

gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths#154837
gpshead merged 1 commit into
python:mainfrom
calvinrp:fix-subprocess-wait-timeout-overflow

Conversation

@calvinrp

@calvinrp calvinrp commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Body

Fixes the 3.15 regression where subprocess.Popen.wait(timeout=...) (and
subprocess.run(..., timeout=...)) fails for very large timeout values such
as float("inf"), sys.maxsize or 1e10 — all accepted on 3.14 and
earlier. The event-driven wait introduced by gh-83069 (#144047) passes the
caller's timeout unclamped to poll() / kqueue.control(); values that do
not fit the C timestamp conversion raise OverflowError on Linux and, on
macOS/BSD, a misleading TypeError: timeout must be a real number or None, not float.

Changes

  • Lib/subprocess.py — both _wait_pidfd() and _wait_kqueue() now
    clamp each OS-level wait to a new module constant
    _MAXIMUM_WAIT_TIMEOUT = 24 * 3600 (following asyncio's
    MAXIMUM_SELECT_TIMEOUT precedent, Lib/asyncio/base_events.py) and loop
    until the caller's real deadline, so TimeoutExpired is still raised at
    the right time and huge-but-legal timeouts never reach the C conversion.
    In _wait_kqueue() the kevent changelist is only submitted on the first
    control() call (the KQ_EV_ADD | KQ_EV_ONESHOT registration persists
    across timed-out waits).
  • Modules/selectmodule.c — the kqueue control() timeout-conversion
    failure is now only rewritten into the "timeout must be a real number or
    None" TypeError when the original exception is a TypeError, guarded
    by PyErr_ExceptionMatches(PyExc_TypeError) exactly like the sibling
    select(), poll(), devpoll() and epoll() sites already do. Huge
    timeouts now surface the true OverflowError instead of a false claim
    that a float is not a real number.

Tests

  • Lib/test/test_subprocess.py (FastWaitTestCase):
    test_wait_huge_timeout (10**10, sys.maxsize, float('inf')),
    test_run_huge_timeout (the run()/communicate() path), and
    test_wait_slices_do_not_expire_early (with the slice limit patched down
    to 10 ms, a process outliving many slices is still waited for — guards
    against a clamp that fires TimeoutExpired at the slice boundary). All
    POSIX-gated: the Windows _wait has a separate, pre-existing
    int(timeout * 1000) overflow for float("inf") that predates 3.15 and
    is deliberately out of scope here.
  • Lib/test/test_kqueue.py: test_control_overflowing_timeout — an
    out-of-range timeout raises OverflowError, not TypeError; non-numbers
    still raise TypeError.

How tested

On macOS (arm64), against a 3.15.0b4 build with the patched
subprocess.py overlaid: the three new test_subprocess tests pass, the
entire pre-existing FastWaitTestCase and the POSIX wait-related suite
stay green, and wait(timeout=float("inf")) / run(..., timeout=1e10)
succeed. The Python-level clamp alone already fixes the user-visible
wait()/run() breakage (the overflowing value never reaches
kq.control()). The test_kqueue addition and the OverflowError
surfacing require the rebuilt select module (verified expected-fail
against the unpatched 3.15.0b4 C module); the Linux _wait_pidfd loop is
exercised by the new tests on Linux CI.


NEWS entry (already in the commit, blurb format)

Misc/NEWS.d/next/Library/2026-07-19-20-00-00.gh-issue-154836.kqWait.rst:

Fix :meth:subprocess.Popen.wait raising :exc:TypeError (macOS and other
BSDs) or :exc:OverflowError (Linux) for very large timeout values such
as float('inf'), a 3.15 regression in the new event-driven wait. Also
fix :meth:select.kqueue.control masking :exc:OverflowError for
out-of-range timeouts as :exc:TypeError.

…fd/kqueue wait paths

The event-driven wait introduced by pythongh-83069 passes the caller's timeout
unclamped to poll() / kqueue.control(), so values that do not fit the C
timestamp conversion (float('inf'), sys.maxsize, 1e10, ...) raise
OverflowError on Linux and, on macOS/BSD, a misleading
"TypeError: timeout must be a real number or None" -- all of which
worked on 3.14 and earlier.

- Lib/subprocess.py: clamp each wait to _MAXIMUM_WAIT_TIMEOUT (24h,
  following asyncio's MAXIMUM_SELECT_TIMEOUT precedent) and loop until
  the real deadline in both _wait_pidfd() and _wait_kqueue().
- Modules/selectmodule.c: only rewrite the kqueue.control() timeout
  conversion failure into TypeError when the original exception IS a
  TypeError, exactly like the select()/poll()/devpoll()/epoll() sites,
  so OverflowError surfaces for out-of-range values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@calvinrp
calvinrp requested a review from gpshead as a code owner July 28, 2026 18:11
@python-cla-bot

python-cla-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@BHUVANSH855 BHUVANSH855 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I noticed that the only failing required check is the Contributor License Agreement (CLA). It looks like the implementation, and the rest of the CI checks are passing.

Could you please sign the CLA? Once that's done, the required check should pass, and the PR will be ready for code review.

@gpshead gpshead self-assigned this Jul 28, 2026
@calvinrp

Copy link
Copy Markdown
Contributor Author

CLA signed, thanks

@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 29, 2026
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 9b7cdf2 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F154837%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 29, 2026
@gpshead

gpshead commented Jul 29, 2026

Copy link
Copy Markdown
Member

the changes look good, running them through the bots now to get some additional kqueue coverage as well. there'll probably be some unrelated failure noise from a few of the less reliable buildbots.

@gpshead
gpshead merged commit 11d0da5 into python:main Jul 29, 2026
128 of 131 checks passed
@gpshead gpshead added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 29, 2026
@miss-islington-app

Copy link
Copy Markdown

Thanks @calvinrp for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Jul 29, 2026

Copy link
Copy Markdown

GH-154891 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 29, 2026
gpshead pushed a commit that referenced this pull request Jul 29, 2026
…dfd/kqueue wait paths (GH-154837) (#154891)

gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths (GH-154837)

The event-driven wait introduced by gh-83069 passes the caller's timeout
unclamped to poll() / kqueue.control(), so values that do not fit the C
timestamp conversion (float('inf'), sys.maxsize, 1e10, ...) raise
OverflowError on Linux and, on macOS/BSD, a misleading
"TypeError: timeout must be a real number or None" -- all of which
worked on 3.14 and earlier.

- Lib/subprocess.py: clamp each wait to _MAXIMUM_WAIT_TIMEOUT (24h,
  following asyncio's MAXIMUM_SELECT_TIMEOUT precedent) and loop until
  the real deadline in both _wait_pidfd() and _wait_kqueue().
- Modules/selectmodule.c: only rewrite the kqueue.control() timeout
  conversion failure into TypeError when the original exception IS a
  TypeError, exactly like the select()/poll()/devpoll()/epoll() sites,
  so OverflowError surfaces for out-of-range values.
(cherry picked from commit 11d0da5)

Co-authored-by: Calvin Prewitt <calvin@setout.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants