CI: DNS lookups on runners take ~38s instead of returning NXDOMAIN, blocking 04070_url_base_setting

[CarlosFelipeOR]

Type of problem

CI infrastructure — DNS, separate from the dataset latency in #2227.

Summary

04070_url_base_setting is our single worst-failing test: 7.9% failure rate on antalya-26.6 (42 of 530 runs, 90 days), against 0.0065% upstream.

It fails because DNS lookups on the runners take 64-96 seconds instead of returning NXDOMAIN immediately.

Why the test depends on DNS

The test verifies RFC 3986 relative-URL resolution for the url_base setting. It can't fetch anything real, so it makes each request fail on purpose and reads the resolved URL out of the error message. Every hostname uses the .invalid TLD (RFC 2606), which is reserved to never resolve.

It runs 25 such lookups in series. The design assumes each one fails instantly.

The measurement

A healthy resolver denies .invalid in ~40ms:

$ dig @1.1.1.1 base.invalid    -> status: NXDOMAIN, 0.04s
$ dig @8.8.8.8 base.invalid    -> status: NXDOMAIN, 0.05s
$ getaddrinfo('base.invalid')  -> 0.003s   (developer machine)

On the runners, from the per-command timestamps in the job log:

query 1 -> 2:  65s
query 2 -> 3:  80s
query 3 -> 4:  64s
query 4 -> 5:  96s
...            always 64-96s

25 lookups x ~80s ~= 33 minutes. The test cannot finish under any reasonable timeout.

Why the test's own settings don't protect it

The test sets http_connection_timeout = 1, http_max_tries = 1. That bounds the TCP connection, not the name resolution — and ClickHouse cannot bound the latter:

src/Common/DNSResolver.cpp:507
/// FIXME Updating may take a long time because we cannot manage timeouts of
///       getaddrinfo(...) and getnameinfo(...).

getaddrinfo is a blocking libc call with no timeout parameter. Only /etc/resolv.conf controls it.

What we already did

Added --dns-search='.' to the stateless job container, matching what the integration jobs already use. That removed the search-domain expansion and roughly halved the cost — ~86s -> ~38s per lookup.

The test now passes, but in 944-965s, where a healthy run takes 8.4s. It only fits because we also raised the per-test timeout to 1200s.

It is per-machine, not per-config

In a single run, the same test on the same commit:

job	duration
amd_debug, distributed plan, s3 storage, parallel	3.1s
arm_binary, parallel	965s
amd_debug, parallel	909.8s
amd_msan, WasmEdge 2/4	BROKEN at 600.1s (this job has no timeout increase)

One runner had a healthy resolver. The others did not.

What would help

Two commands from inside a runner would confirm the cause:

cat /etc/resolv.conf             # how many nameservers, what timeout/attempts
time getent hosts base.invalid   # should be ~40ms; we measure ~38s

If the nameservers are dropping queries for nonexistent names instead of answering NXDOMAIN, the fix is on the resolver side. Options we considered on the CI side:

1. --dns=<a resolver that answers> on the job container. We verified that no stateless, integration, stress or performance test needs private-name resolution — all endpoints are localhost/127.0.0.x or public names, and the one test that genuinely exercises DNS (test_host_regexp_multiple_ptr_records) runs its own coredns and rewrites the container's resolv.conf. Caveat: this would break Antalya 26.6 - Use new euro mirror for test datasets #2248, which points the web disk at the internal dockerhub-proxy name, so a public-only resolver is not an option there.
2. --dns-option=timeout:2 --dns-option=attempts:2 — bounds the wait, but applies to legitimate lookups too.
3. A local caching resolver with negative caching — 25 lookups hit only 5 distinct hosts, so this would cut it to 5 slow lookups.

Scope, stated honestly

Today this affects one test. Other tests that resolve .invalid names (04066_json_value_tuple_array, 04407_datalake_getcatalog_deprecated_settings_guard, 04213_base_backup_with_query_parameter) do only one or two lookups each and absorb the cost — all at 0% or ~0.1% failure. 04070 is exposed because it chains 25.

But a resolver that doesn't answer NXDOMAIN is an infrastructure property. It's currently costing ~80s wherever a nonexistent name gets resolved; only this test makes it visible.