Fix: raise soft open-file limit at serve() startup by Ankitsinghsisodya · Pull Request #83 · knative-extensions/func-python

Ankitsinghsisodya · 2026-04-16T02:20:15Z

Problem

Python functions fail under load on container platforms that set a low
soft open-file limit (e.g. 1024). The Go and Java runtimes raise the
soft limit to the hard limit automatically at startup; Python does not.

Relates to knative/func#3513

Solution

Add _raise_nofile_limit() in a new src/func_python/_ulimit.py
module and call it at the very top of serve() in both http.py and
cloudevent.py. Placing the fix in the library means every deployed
function benefits from a single version bump, rather than requiring a
rebuild of every container image.

The implementation:

For finite hard limits: raises the soft RLIMIT_NOFILE to the hard
limit exactly, matching Go and Java runtime behaviour.
For RLIM_INFINITY hard limits: caps the soft limit at _MAX_NOFILE
(65536) because the kernel rejects setting the soft limit to
RLIM_INFINITY without CAP_SYS_RESOURCE.
Silently skips on non-Unix platforms where resource is unavailable
(_resource = None at module level).
Logs a DEBUG message on success and a WARNING on failure via a
named logger (logging.getLogger(__name__)), but never raises — a
failure to adjust the limit is not a fatal error.

Changes

src/func_python/_ulimit.py — new shared helper
src/func_python/http.py — import and call _raise_nofile_limit() at top of serve()
src/func_python/cloudevent.py — same
tests/test_ulimit.py — nine unit tests using patch("func_python._ulimit._resource") for guaranteed mock isolation
pyproject.toml — version bumped to 0.8.2
CHANGELOG.md — entry added under 0.8.2

Test plan

test_raises_soft_limit_to_hard — soft < hard (finite ≤ 65536), setrlimit called with full hard value
test_raises_soft_limit_to_hard_above_65536 — hard = 131072, setrlimit called with 131072 (not capped)
test_no_change_when_soft_equals_hard — setrlimit not called
test_rlim_infinity_capped_at_max — hard is RLIM_INFINITY, target is _MAX_NOFILE
test_import_error_is_silently_skipped — _resource = None, no exception
test_os_error_logs_warning — setrlimit raises OSError, warning logged, no exception
test_value_error_logs_warning — setrlimit raises ValueError, warning logged, no exception
test_http_serve_calls_raise_nofile_limit — wire-up verified for http.serve()
test_cloudevent_serve_calls_raise_nofile_limit — wire-up verified for cloudevent.serve()

All nine tests pass (poetry run pytest tests/test_ulimit.py -v).

Python does not automatically raise the process soft open-file limit to the hard limit on startup, unlike the Go and Java runtimes. Under container environments that set a low soft limit (e.g. 1024), functions fail under load with "too many open files" errors. Add _raise_nofile_limit() in a new src/func_python/_ulimit.py module and call it at the top of serve() in both http.py and cloudevent.py. The helper caps the target at 65536 to avoid passing RLIM_INFINITY directly to setrlimit (which the kernel rejects), and silently skips on non-Unix platforms where the resource module is unavailable. Add six unit tests in tests/test_ulimit.py covering normal raise, no-op when soft equals hard, RLIM_INFINITY capping, ImportError, OSError, and ValueError cases. Relates to knative/func#3513

knative-prow · 2026-04-16T02:20:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Ankitsinghsisodya
Once this PR has been reviewed and has the lgtm label, please assign matejvasek for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

knative-prow · 2026-04-16T02:20:24Z

Welcome @Ankitsinghsisodya! It looks like this is your first PR to knative-extensions/func-python 🎉

Copilot

Pull request overview

Adds a startup helper to raise the process open-file soft limit (RLIMIT_NOFILE) so Python functions don’t fail under load on platforms with low default ulimits.

Changes:

Introduce src/func_python/_ulimit.py with _raise_nofile_limit() helper.
Invoke _raise_nofile_limit() at the top of serve() in both HTTP and CloudEvent runtimes.
Add unit tests for _raise_nofile_limit() behavior via mocks.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
`src/func_python/_ulimit.py`	New helper to adjust `RLIMIT_NOFILE` and log success/failure.
`src/func_python/http.py`	Calls the ulimit-raising helper at `serve()` startup.
`src/func_python/cloudevent.py`	Calls the ulimit-raising helper at `serve()` startup.
`tests/test_ulimit.py`	New tests validating ulimit adjustment and logging behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-16T02:22:55Z

+import logging
+import sys
+import unittest
+from unittest.mock import MagicMock, patch, call


Unused import: call is imported from unittest.mock but never used in this test module. Removing it will avoid lint noise.

Suggested change

from unittest.mock import MagicMock, patch, call

from unittest.mock import MagicMock, patch

Copilot · 2026-04-16T02:22:56Z

+            # Should complete without raising anything
+            ulimit._raise_nofile_limit()
+
+    def test_os_error_logs_warning(self, caplog=None):


test_os_error_logs_warning includes an extra caplog=None parameter (a pytest-style fixture) but the test is written using unittest.TestCase and assertLogs. Consider removing the unused parameter to keep the test signature conventional and avoid confusion.

Suggested change

def test_os_error_logs_warning(self, caplog=None):

def test_os_error_logs_warning(self):

Copilot · 2026-04-16T02:22:56Z

+            if hard == resource.RLIM_INFINITY:
+                target = _MAX_NOFILE
+            else:
+                target = min(hard, _MAX_NOFILE)
+            resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))


When hard != RLIM_INFINITY, the code currently caps target to _MAX_NOFILE via min(hard, _MAX_NOFILE). This diverges from the PR description (“raise soft to the hard limit” and only cap when hard is infinite) and can leave the soft limit below the actual hard limit on systems where hard is high (e.g. 1M). Consider setting target = hard for finite hard limits and only applying _MAX_NOFILE when hard == RLIM_INFINITY (or updating the description if the always-cap behavior is intentional).

Copilot · 2026-04-16T02:22:57Z

+            else:
+                target = min(hard, _MAX_NOFILE)
+            resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))


The code comment says RLIM_INFINITY cannot be passed directly to setrlimit, but the call still passes hard (which is RLIM_INFINITY in that branch) as the hard component: setrlimit(..., (target, hard)). If RLIMIT_NOFILE rejects RLIM_INFINITY, this will always fail and the soft limit won’t be raised. Consider also capping the hard argument (e.g. use (target, target) when hard == RLIM_INFINITY) so the call can succeed.

Suggested change

else:

target = min(hard, _MAX_NOFILE)

resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))

new_hard = target

else:

target = min(hard, _MAX_NOFILE)

new_hard = hard

resource.setrlimit(resource.RLIMIT_NOFILE, (target, new_hard))

Copilot · 2026-04-16T02:22:57Z

+            ulimit._raise_nofile_limit()
+
+        mock_resource.setrlimit.assert_called_once_with(
+            mock_resource.RLIMIT_NOFILE, (ulimit._MAX_NOFILE, RLIM_INFINITY)


This test currently asserts setrlimit is called with (ulimit._MAX_NOFILE, RLIM_INFINITY). If _raise_nofile_limit() is updated to avoid passing RLIM_INFINITY into setrlimit (so the operation can succeed), this expectation should be adjusted accordingly (e.g. hard arg capped to the same finite value).

Suggested change

mock_resource.RLIMIT_NOFILE, (ulimit._MAX_NOFILE, RLIM_INFINITY)

mock_resource.RLIMIT_NOFILE,

(ulimit._MAX_NOFILE, ulimit._MAX_NOFILE),

- Remove min(hard, _MAX_NOFILE) cap for finite hard limits; raise soft to the actual hard limit as the Go/Java runtimes do. The _MAX_NOFILE cap now applies only to the RLIM_INFINITY branch where it is needed. - Fix misleading comment: it is the soft limit, not the hard limit, that cannot be RLIM_INFINITY without CAP_SYS_RESOURCE. - Replace importlib.reload()-based test helpers with plain pytest functions that patch sys.modules only at call time, eliminating shared module-state mutation between tests. - Remove dead caplog=None default parameter from test_os_error_logs_warning; caplog is now injected as a proper pytest fixture. - Add test_raises_soft_limit_to_hard_above_65536 to guard against the truncation regression. - Add wire-up tests for http.serve() and cloudevent.serve() confirming _raise_nofile_limit() is called on every serve() invocation.

Ankitsinghsisodya · 2026-04-16T09:16:28Z

cc @matejvasek @matzew

… versioning - Move `import resource` to module level as `_resource` so tests can patch `func_python._ulimit._resource` directly with unittest.mock.patch, replacing fragile sys.modules manipulation that lacked guaranteed isolation. - Replace `logging.info` with `_logger.debug` (named logger via getLogger(__name__)): routine startup detail should not appear at INFO in production logs; named logger lets users silence func_python logs independently. - Use `resource.RLIM_INFINITY` from the real module in tests instead of a hardcoded magic integer that could silently diverge on non-Linux platforms. - Bump version to 0.8.2 and add CHANGELOG entry for the ulimit feature.

Copilot AI review requested due to automatic review settings April 16, 2026 02:20

knative-prow bot requested review from matejvasek and matzew April 16, 2026 02:20

knative-prow bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 16, 2026

Copilot started reviewing on behalf of Ankitsinghsisodya April 16, 2026 02:20 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

This was referenced Apr 16, 2026

Fix: raise soft ulimit for open files in Python runtime startup knative/func#3612

Closed

Support low-ulimit platforms in Python runtime knative/func#3513

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: raise soft open-file limit at serve() startup#83

Fix: raise soft open-file limit at serve() startup#83
Ankitsinghsisodya wants to merge 3 commits intoknative-extensions:mainfrom
Ankitsinghsisodya:issue-3513-raise-nofile-limit

Ankitsinghsisodya commented Apr 16, 2026 •

edited

Loading

Uh oh!

knative-prow bot commented Apr 16, 2026

Uh oh!

knative-prow bot commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Ankitsinghsisodya commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	from unittest.mock import MagicMock, patch, call
	from unittest.mock import MagicMock, patch

	def test_os_error_logs_warning(self, caplog=None):
	def test_os_error_logs_warning(self):

	mock_resource.RLIMIT_NOFILE, (ulimit._MAX_NOFILE, RLIM_INFINITY)
	mock_resource.RLIMIT_NOFILE,
	(ulimit._MAX_NOFILE, ulimit._MAX_NOFILE),

Conversation

Ankitsinghsisodya commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Changes

Test plan

Uh oh!

knative-prow bot commented Apr 16, 2026

Uh oh!

knative-prow bot commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Ankitsinghsisodya commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ankitsinghsisodya commented Apr 16, 2026 •

edited

Loading