Skip to content

kbuild: replace google.com probes with bounded retries - #3277

Open
adityagrewal wants to merge 1 commit into
kernelci:mainfrom
adityagrewal:fix-network-readiness-check
Open

adityagrewal wants to merge 1 commit into
kernelci:mainfrom
adityagrewal:fix-network-readiness-check

Conversation

@adityagrewal

Copy link
Copy Markdown

Replace the hard-coded google.com startup probes with bounded retries on the actual KernelCI API and storage requests (Strategy 1 in #3197).

Why

  • KBuild._verify_network() sent an un-bounded requests.get("https://google.com") without a timeout and never decremented its retry counter on non-200 HTTP responses.
  • config/runtime/base/python.jinja2 ran a socket.gethostbyname("www.google.com") loop before starting a job, creating an external dependency and risking unbounded DNS calls instead of verifying connectivity to KernelCI services.

Changes

  • kernelci/kbuild.py:
    • Removed KBuild._verify_network().
    • Added _request_with_retry() with (5.0, 15.0)s (connect, read) timeouts, a 2.0s retry interval, and a 60.0s time.monotonic() wall-clock deadline.
    • Restricted retries to transient network/DNS errors (Timeout, ConnectionError, socket.gaierror) and transient HTTP statuses (429, 5xx), letting configuration/programming errors fail fast.
    • Raised KBuildInfraError (with sanitized hostname and elapsed duration) and reported via self.submit_failure() in write_script() instead of calling sys.exit().
  • config/runtime/base/python.jinja2:
    • Removed the www.google.com DNS loop from main().
    • Wrapped initial KernelCI API lookups and source tarball fetches in BaseJob._call_with_retry() using the same timeout and time.monotonic() bounds.
    • Handled InfraReadinessError in main() by reporting result='incomplete' with error_code='infra_error' and sanitized service details.
  • tests/test_kbuild.py:
    • Added mocked unit tests covering immediate success, transient recovery, DNS/connect/read timeouts, HTTP 5xx responses, deadline exhaustion, URL query-token sanitization, programming error propagation, and Jinja2 template rendering.

Validation

  • Ran pytest tests/test_kbuild.py and ruff checks locally.
  • Staging kbuild validation on a newly started Kubernetes node is ready for an operator to trigger.

Closes #3197
Assisted-by: Gemini

Remove the hard-coded google.com connectivity and DNS probes from
KBuild and config/runtime/base/python.jinja2, which could hang
indefinitely or loop without bound when Kubernetes pod networking was
still initializing.

Apply bounded retries with explicit (5.0, 15.0)s connect/read timeouts,
a 2.0s retry interval, and a 60.0s monotonic wall-clock deadline to the
initial KernelCI API and storage requests, reporting exhausted readiness
as a sanitized infrastructure error rather than calling sys.exit().

Closes: kernelci#3197
Assisted-by: Gemini
Signed-off-by: Aditya Grewal <aditya1998yr@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Kubernetes job network readiness checks bounded and deployment-independent

1 participant