Skip to content

fix(integration): use dedicated HTTP manager with managerIdleConnectionCount = 0 for probes - #5406

Open
blackheaven wants to merge 1 commit into
gdifolco/fix-flaky-tests-leaking-metrics-1from
gdifolco/fix-flaky-tests-leaking-metrics-2
Open

fix(integration): use dedicated HTTP manager with managerIdleConnectionCount = 0 for probes#5406
blackheaven wants to merge 1 commit into
gdifolco/fix-flaky-tests-leaking-metrics-1from
gdifolco/fix-flaky-tests-leaking-metrics-2

Conversation

@blackheaven

Copy link
Copy Markdown
Contributor

This commit configures a dedicated HTTP.Manager for probe requests with managerIdleConnectionCount = 0 to prevent connection pool pollution during startup.

Background & Root Cause:
When integration tests perform service startup readiness probes (/i/status) or federation ingress warmup probes (/rpc/.../api-version), probes may time out or cancel early while unconsumed response bytes (such as from concurrent /i/metrics scrapes) remain in socket buffers.

If these probe requests share the primary HTTP.Manager connection pool, dirty sockets containing leftover response bytes can be checked out by subsequent test requests, returning unexpected status 200 text/plain metrics responses and causing assertion failures in checkFederationIngress.

Fix:

  • Add a dedicated probeManager field to GlobalEnv and Env initialized with managerIdleConnectionCount = 0 so that probe connections are never cached or reused across requests.
  • Add submitProbe helper in Testlib.HTTP using probeManager.
  • Update checkServiceIsUp and checkFederationIngress in Testlib.ModService to execute probe requests via submitProbe.

Checklist

  • Add a new entry in an appropriate subdirectory of changelog.d
  • Read and follow the PR guidelines

…onCount = 0 for probes

This commit configures a dedicated HTTP.Manager for probe requests with managerIdleConnectionCount = 0 to prevent connection pool pollution during startup.

Background & Root Cause:
When integration tests perform service startup readiness probes (/i/status) or federation ingress warmup probes (/rpc/.../api-version), probes may time out or cancel early while unconsumed response bytes (such as from concurrent /i/metrics scrapes) remain in socket buffers.

If these probe requests share the primary HTTP.Manager connection pool, dirty sockets containing leftover response bytes can be checked out by subsequent test requests, returning unexpected status 200 text/plain metrics responses and causing assertion failures in checkFederationIngress.

Fix:
- Add a dedicated probeManager field to GlobalEnv and Env initialized with managerIdleConnectionCount = 0 so that probe connections are never cached or reused across requests.
- Add submitProbe helper in Testlib.HTTP using probeManager.
- Update checkServiceIsUp and checkFederationIngress in Testlib.ModService to execute probe requests via submitProbe.
@blackheaven
blackheaven requested a review from a team as a code owner July 31, 2026 22:16
@zebot zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants