CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease

[CarlosFelipeOR]

🤖 Generated with AI assistance — Claude Opus 5 (claude-opus-5[1m]) — from the CI logs and artifacts linked below, reviewed before posting.

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report — availability failure under read-heavy concurrency on a CAS default policy.

Describe the situation

With cas_disk as the default storage policy, a read-heavy concurrent phase drives the ClickHouse container to exhaust its local ephemeral ports against the object store. Every CAS request then fails at connect() with errno 99:

Poco::Exception. Code: 1000, e.code() = 99,
Net Exception: Cannot assign requested address: 172.18.0.2:9001

Nothing on the object-store side is down — the port pool on the ClickHouse side is empty. Because the mount-lease renewal is a conditional write over the same path, the outage takes down the lease, and the whole disk stops answering reads and writes until a self-remount.

Found in RegressionTestsRelease / CAS (selects) / cas_selects, test /selects/final/force/concurrent/SELECT WHERE parallel inserts, deletes, updates. Both retries of the job failed, at the same phase, from the same cause — this is not flaky.

This issue:

• Turns a client-side resource exhaustion into a full read+write outage of cas_disk (1330 queries refused over ~42 s in retry 2)
• Fences out the mount lease and forces a self-remount, discarding an in-flight PartWriteTxn
• Is not caused by the store: RustFS 1.0.0-beta.12 honours HTTP keep-alive (verified — 3 requests, 1 connect, no Connection: close), so the connection churn is client-side
• Is not bounded by any existing limit: disk_connections_store_limit=10000 / disk_connections_hard_limit=200000 against 28 232 ephemeral ports in the container netns

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20000.altinityantalya
• Build type: release, amd (build_amd_release)
• Package: PR #2159 7b274f0a461d67fff65a56aafb0865ba0e1af4c8
  • https://altinity-build-artifacts.s3.amazonaws.com/PRs/2159/7b274f0a461d67fff65a56aafb0865ba0e1af4c8/build_amd_release/clickhouse
• Store: RustFS 1.0.0-beta.12, single container (selects/selects_env/rustfs-service.yml)
• Disk: cas_disk overriding the default policy (selects/cas_mode.py), no filesystem cache in front of it
• Container netns: ip_local_port_range = 32768 60999 (28 232 ports), tcp_fin_timeout = 60

Steps

python3 -u selects/regression.py --clickhouse https://altinity-build-artifacts.s3.amazonaws.com/PRs/2159/7b274f0a461d67fff65a56aafb0865ba0e1af4c8/build_amd_release/clickhouse --cas --local --clickhouse-version 26.6.2.20000.altinityantalya --log log.log  --only '/selects/final/*'

While it runs, watch the port pressure inside the server container:

docker exec <clickhouse1> grep -c ' 06 ' /proc/net/tcp   # TIME_WAIT

Scope matters. Running only the failing scenario (--only '/selects/final/force/concurrent/SELECT WHERE parallel inserts, deletes, updates/*') passes — 280 s, 3837 OK, 0 fail, peak TIME_WAIT 1051 of 28 232. Exhaustion is cumulative: it needs the whole final subtree, a store with accumulated manifests (GC sweep running), and enough parts for FINAL to re-read.

---

Expected behavior

Concurrent SELECT ... FINAL load should not be able to drain the port pool the mount lease depends on. If the data plane does saturate, the lease should survive and the failure should stay scoped to the affected queries — not take the disk offline for reads.

---

Actual behavior

Same root cause, two blast radii across the two retries of the same job:

Retry 1 — lease survived; the write budget did not:

05:33:32 <Warning> CasWriteRetryLater: CAS write could not be committed
  (stageManifest: part-manifest PUT at 'data/cas/manifests/selects-cas-clickhouse1/store/6c6/
   6c641193-...@cas@/0000000000000001-0000000000000a6e/000001.zst' is UNCERTAIN
   (retry budget exhausted) — nothing conclusive was named); retrying later

Surfaces to the client as the INSERT failing:

Code: 210. DB::Exception: CAS write could not be committed (...) ; retrying later:
while pushing to view default.MergeTree_table_..._core_mview:
While executing WaitForAsyncInsert. (NETWORK_ERROR)

16 attempts / 90 s operation deadline (CasRequestControl.h) against ~2.5 min of unavailable ports.

Retry 2 — the lease was lost:

12:18:01 <Error> CasMountLeaseKeeper: background renewal failed transiently, retrying while the lease is still valid: ... e.code() = 99 ...
12:19:41 <Error> CasMountLeaseKeeper: (same)
12:19:51 <Error> CasMountLeaseKeeper: (same)
12:20:01 <Error> CasMountLeaseKeeper: background renewal failed, the mount-lease stops advancing

The pool goes TransientNotLive and every query on the disk is refused for ~42 s:

Code: 210. DB::Exception: content-addressed disk 'cas_disk' -- mount lease not held;
backing may be temporarily unreachable; ... TRANSIENT unavailability, not damage

→ 1330 occurrences in the TestFlows log, including plain SELECTs and the async-insert queue.

Then the self-remount invalidates work already in flight:

12:23:16 <Warning> CasWriteRetryLater: PartWriteTxn (writer_epoch 1) belongs to a superseded
  mount incarnation (live epoch 2) — the mount was fenced out and self-remounted; restart the build

---

Root cause analysis

The trigger is measurable in the server log. MergeTreePrefetchedReadPool activity jumps ~25× in exactly the minute the ports run out — cas_disk has no cache, so each of the 4 parallel SELECT ... FINAL variants re-reads every part from the store:

	prefetch baseline	prefetch peak	first errno 99	total errno 99
retry 1	~1 000/min	22 973/min (05:31)	05:31:24	7 362 (until 05:33:46)
retry 2	~1 000/min	25 818/min (12:17)	12:17:32	8 722 (until 12:20:06)

All 8 722 target 172.18.0.2:9001 — the object store, nothing else.

~430 GETs/s with tcp_fin_timeout=60 puts on the order of 26 000 sockets in TIME_WAIT, against 28 232 available ports. The retry loops then make it worse rather than better: errno 99 is local resource exhaustion, but it is classified as an ordinary retryable network error, so 16 CAS attempts (and the base client's s3_retry_attempts=500) keep opening more sockets. The lease renewal shares that path and has only 3 chances (mount_lease_ttl_ms=30000 / mount_renew_period=10000).

---

Suggested fix

1. Bound port usage. disk_connections_store_limit=10000 / disk_connections_hard_limit=200000 are both far above the 28 232 ports a container has. Cap disk connections for CAS disks below the ephemeral range (and/or investigate why the pool churns connections when the store honours keep-alive).
2. Do not let the data plane fence out the lease. Give the mount-lease renewal reserved capacity, or widen the margin — 30 s TTL / 10 s renew is 3 attempts.
3. Classify EADDRNOTAVAIL as local exhaustion, not as a retryable remote error: back off hard instead of spending 16 attempts opening further sockets.
4. CI mitigation (separate PR): put a filesystem cache in front of cas_disk in selects/cas_mode.py, and/or raise ip_local_port_range / lower tcp_fin_timeout in the CAS test envs.

---

Additional context

• Job: RegressionTestsRelease / CAS (selects) / cas_selects — failed on both retries

• Server logs (cas_selects-artifacts-release_zookeeper, path inside the zip: selects/_instances/clickhouse1/logs/clickhouse-server.log) — every excerpt above comes from these:

  • retry 1 → artifact 9392364351
  • retry 2 → artifact 9402740372
  • both expire 2026-11-18; the TestFlows raw.log is in the same zip

• Possibly related: #2233 (replica HTTP dies under a CAS NETWORK_ERROR storm on soak) — worth checking whether port exhaustion is the shared mechanism there too

[filimonov]

Confirmed — good report, and the diagnosis is close. Adjudicated against cas-gc-rebuild head; the mechanism is fully explainable from code, and an in-house precedent exists (2026-07-06: a 20,000-column merge exhausted ephemeral ports the same way; it was mis-scoped then as "cost only" — this issue shows the same condition takes the lease down, so it is re-rated to availability class).

The exact churn mechanism (refines the report's "investigate why the pool churns"): the S3 client's keep-alive default is 5 seconds (S3Defines.h:12), not the store's. A pooled DISK connection expires after 0.8×5 s of idleness (0.1×5 s once the group crosses disk_connections_soft_limit), mustReconnect() resets any connection whose request started >4.5 s ago (Poco stamps _lastRequest at request start, so long-streaming GETs are reset even on success), and http_keep_alive_max_requests=100 rotates the rest. With P pooled connections and R req/s, reuse survives only while P/R < 4 s; concurrent FINAL prefetch puts P in exactly the dead band, so nearly every request opens a fresh socket and the client closes it on return — TIME_WAIT at request rate, 430 × 60 ≈ 26k, matching the measurement. RustFS honoring keep-alive is beside the point: the client throws the sockets away.

On the suggested fixes:

1. Capping disk connections below the ephemeral range will not help and can make it worse. All disk_connections_* limits gate keeping a connection, not creating one (only hard_limit gates creation, by throwing); TIME_WAIT sockets are closed and invisible to every limit — that is why the failure is EADDRNOTAVAIL and never HTTP_CONNECTION_LIMIT_REACHED; and a lower soft limit flips the pool into the 0.1× (0.5 s) TTL regime sooner, i.e. more churn. Separately, s3_max_connections is dead code on the disk path — it sets an AWS-SDK field that PocoHTTPClient never reads.
2. Lease resilience: agreed, and it is worse than the issue states in one way, better in another. The renewal is a bare single-attempt conditional PUT once per 10 s through the same DISK pool — no reserved capacity, no retry inside the period (a failed renewal continues into a full wait_for(period)), so the lease has less resilience than a data write (16 attempts / 90 s). But the margin arithmetic (ttl 30 s, period 10 s, margin 2 s) allows at most two ride-out warnings per lease generation — the three warnings over ~2.5 min in retry 2 imply interleaved successful renewals, i.e. the exhaustion was intermittent rather than a continuous blackout. Queued: in-period fast retry + reserved capacity for the renewal.
3. EADDRNOTAVAIL classification: agreed. It is treated as a remote transient at every layer (Poco → CoreErrors::NETWORK_CONNECTION → retryable; the CAS controller's classifier → Unresolved, the most retry-hungry class). Each retry consumes another port, so the retry policy is the positive-feedback term. The DNS branch in PocoHTTPClient.cpp:840 is the template for a local-exhaustion sub-classification (EADDRNOTAVAIL/EMFILE/ENFILE → hard backoff). Queued.
4. CI mitigation: the effective one-liner is raising the keep-alive in the disk's S3 config — <http_keep_alive_timeout>60</http_keep_alive_timeout> (optionally <http_keep_alive_max_requests> too) — which moves the reuse threshold from P<~1700 to P<~25000 at 430 req/s. One caveat to verify on the wire: Poco mins the client value down to whatever Keep-Alive: timeout=N the server advertises, so check what RustFS sends. A filesystem cache in front also works (it removes R rather than fixing reuse).

Everything downstream of the trigger worked as designed: fence → TransientNotLive → refusal with the transient-not-damage message → self-remount → superseded-epoch PartWriteTxn refusal is the intended fail-close chain. The ~42 s outage length is constructive, not backoff: the self-remount's token-stability observation wait is ttl + ttl/20 + poll = 36.5 s with these settings, plus claim/quiesce work — i.e. the outage is ≈1.2× lease TTL by construction. Shortening that window (or holding queries in a bounded wait instead of refusing during it) is a pre-existing tracked design question that this issue gives field data for.

Diagnostic worth adding to the repro before any tuning: the pool is fully instrumented — system.events: DiskConnectionsCreated/Reused/Preserved/Expired/Reset; system.metrics: DiskConnectionsTotal/Stored. The TTL-churn mechanism predicts Created ≈ request rate, Expired ≫ Reset, Stored ≈ 0. If instead Reset ≫ Expired, the driver is the non-drained-response branch and the fix belongs in the read path — worth capturing either way.

Re #2233: different mechanism there (interserver relink refusals; adjudicated separately), but the port-exhaustion angle is worth checking in that repro too — grep -c ' 06 ' /proc/net/tcp during the soak would settle it.

[filimonov]

TLDR version: add <http_keep_alive_timeout>60</http_keep_alive_timeout> to disk config as a workaround.

UPD: need to make those settings passed through.

[CarlosFelipeOR]

🤖 AI-assisted (Claude Opus 5).

Ran the diagnostic you asked for, on --only '/selects/final/*' with --cas (26.6.2.20000, PR 2159 7b274f0a), baseline config. It reproduces outside CI — peak TIME_WAIT to the store hit 28 192 of 28 232 ports (99.9%), 3 679 Cannot assign requested address, same stageManifest ... UNCERTAIN signature as retry 1.

The counters go the other way from the TTL-churn prediction. Over 16.4 min / 617 849 DiskS3* requests:

counter	total	/s
DiskConnectionsCreated	208 047	212.1
DiskConnectionsReused	424 775	433.0
DiskConnectionsExpired	2 265	2.3
DiskConnectionsReset	206 958	211.0

Created ≈ Reset (208 047 vs 206 958) — every reset connection is replaced by a fresh socket; 33.7% of all requests open one. Reset exceeds Expired by 91×, so by your own discriminator this is the non-drained-response branch and the fix belongs in the read path.

Narrowing which of the five reset causes in atConnectionDestroy:

• mustReconnect() — ruled out: avg read 0.6–2.2 ms, avg write 12–56 ms, against a 5 s keep-alive.
• isStoreLimitReached() — ruled out: peak pooled 58, store_limit 10 000.
• !isCompleted() / buffered() — the remaining explanation.

Note Expired also covers the idle-TTL wipe in wipeExpiredImpl, not just keep_alive_max_requests, so the TTL regime would have surfaced there — it didn't. Reset is exactly 0 for minutes 0–3 and jumps to 8 961 at minute 4, when read volume roughly doubles.

On the keep-alive workaround — it can't be applied on a CAS disk at all. ContentAddressedSettings::loadFromConfig iterates every key in the disk block and forwards anything outside the 15-entry non_cas_keys allow-list to the CAS settings setter, so <http_keep_alive_timeout> throws UNKNOWN_SETTING and the server dies at startup (CI run, reverted since). No alternative route: the S3AuthSettings fallback looks for the user setting s3_http_keep_alive_timeout, which doesn't exist in Settings.cpp. Looks like the same one-line addition already made for expect_continue_min_bytes / http_client. I checked the wire caveat you raised as well, in case it's useful later: rustfs 1.0.0-beta.12 advertises neither Keep-Alive nor Connection on /health or on a real S3 path, so nothing would have negotiated the client value down.

Withdrawing suggested fix #1 from the report — agreed the disk_connections_* limits gate keeping rather than creating, and a lower soft limit would make it worse.

Confirming your intermittency read on retry 2, with the timeline: the EADDRNOTAVAIL come in two bursts — 12:17:30→12:18:19, then ~80 s with zero errors, then 12:19:40→12:20:09. The renewal warnings sit inside the bursts (12:18:01; then 12:19:41 / 12:19:51 and the give-up at 12:20:01), so the lease rode out the first burst exactly as you described.

One correction to my own report: the 1 051 TIME_WAIT figure was an unattributed total. Counting by remote port, inbound clickhouse client connections on :9000 peaked at 3 073 on their own and are excluded from every number above.

Still open from your list: I haven't run the #2233 soak repro with grep -c ' 06 ' /proc/net/tcp yet.

Caveat: Reset lumps five causes into one counter, so "non-drained response" is reached by elimination, not direct attribution — splitting that counter (or a debug build) would confirm it, and I haven't identified the read-path call site.