Retry newly provisioned Query API endpoints until ready

[sdairs]

Summary

• retry a newly provisioned Query API endpoint for up to 120 seconds with exponential backoff capped at 5 seconds
• retry only non-SQL 401, 403, and generic 404 readiness responses; stopped services, SQL errors, rate limits, transport failures, and 5xx responses remain terminal
• reuse the single key and endpoint upsert from the inherited locked provisioning flow
• add subprocess wiremock coverage for a 401 -> 403 -> 404 -> success sequence, asserting one key creation and one endpoint upsert

Closes #453

Tests

• cargo test -p clickhousectl (748 passed)
• cargo test -p clickhousectl --test cli_request_shape_test service_query_retries_new_endpoint_readiness_without_reprovisioning -- --exact (1 passed)
• cargo fmt --all --check
• cargo clippy -p clickhousectl --all-targets -- -D warnings

No live Cloud resources or credentials were used.

Stack

This is the child of issue-451-query-provision-singleflight (PR #493). It targets that parent branch; PR #493 remains unchanged and should merge first.

[macroscopeapp]

🟡 Medium cloud/services.rs:1768

A single run_basic_service_query(...).await can block past QUERY_ENDPOINT_READY_TIMEOUT indefinitely, so a newly provisioned endpoint that accepts a connection but never completes its response leaves the command hung. The deadline is checked only after each attempt returns; wrap each attempt in tokio::time::timeout using the remaining duration and return a timeout error when it expires.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @crates/clickhousectl/src/cloud/services.rs around line 1768:

A single `run_basic_service_query(...).await` can block past `QUERY_ENDPOINT_READY_TIMEOUT` indefinitely, so a newly provisioned endpoint that accepts a connection but never completes its response leaves the command hung. The deadline is checked only after each attempt returns; wrap each attempt in `tokio::time::timeout` using the remaining duration and return a timeout error when it expires.