From 85da2f4c43d1b040b5ae6a884f44d76753c24b3c Mon Sep 17 00:00:00 2001 From: Gautier DI FOLCO Date: Fri, 31 Jul 2026 23:37:38 +0200 Subject: [PATCH] fix(integration): close transient probe connections and poll federatorExternal in checkServiceIsUp This commit addresses a root cause of flaky test failures where probe requests during backend startup/warmup received status 200 responses containing Prometheus metrics text payloads (Content-Type: text/plain; version=0.0.4) instead of expected JSON responses, triggering assertion failures in checkFederationIngress. Background & Root Cause: 1. Multi-listener Startup Race: FederatorInternal (port 10097) and FederatorExternal (port 10098) are run in separate asynchronous threads. Previously, checkServiceIsUp only polled FederatorInternal, allowing waitUntilServiceIsUp to unblock before FederatorExternal was bound and ready for HTTP/2 federation requests. 2. HTTP Connection Pool Desynchronization: When startup probes (e.g. /i/status) or ingress warmup probes (/rpc/.../api-version) time out or are cancelled early, unconsumed response bytes (such as from concurrent /i/metrics scrapes) remain in the TCP socket buffer. Subsequent requests reusing pooled sockets from HTTP.Manager read these leftover response headers and metrics bodies, causing non-JSON responses to be returned. Fix: - Update checkServiceIsUp to poll both FederatorInternal (port 10097) and FederatorExternal (port 10098) before reporting Federator as ready. - Add Connection: close HTTP header to status probes in checkServiceIsUp and ingress probes in checkFederationIngress so that transient probe sockets are immediately closed upon completion/cancellation instead of polluting the connection pool. --- integration/test/Testlib/ModService.hs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/integration/test/Testlib/ModService.hs b/integration/test/Testlib/ModService.hs index a02f921639..7d1eb82c21 100644 --- a/integration/test/Testlib/ModService.hs +++ b/integration/test/Testlib/ModService.hs @@ -456,6 +456,7 @@ checkFederationIngress origin target = do Unversioned ("/rpc/" <> target <> "/brig/api-version") <&> (addHeader "Wire-Origin-Domain" origin) + . (addHeader "Connection" "close") . (addJSONObject []) checkStatus <- appToIO $ do submit "POST" req `bindResponse` \res -> do @@ -515,13 +516,13 @@ waitUntilServiceIsUp mDebug domain srv = do checkServiceIsUp :: String -> Service -> App Bool checkServiceIsUp _ Nginz = pure True checkServiceIsUp domain srv = do - req <- baseRequest domain srv Unversioned "/i/status" + req <- baseRequest domain srv Unversioned "/i/status" <&> addHeader "Connection" "close" mExtReq <- case srv of FederatorInternal -> do sMap <- getServiceMap domain let extHostPort = sMap.federatorExternal extUrl = "http://" <> extHostPort.host <> ":" <> show extHostPort.port <> "/i/status" - Just <$> externalRequest extUrl + Just . addHeader "Connection" "close" <$> externalRequest extUrl _ -> pure Nothing checkStatus <- appToIO $ do res <- submit "GET" req