Make Query API provisioning single-flight across processes

[sdairs]

Summary

• serialize Query API auto-provisioning with a project-local cross-process lock
• re-read and merge credentials under the lock, then atomically replace credentials.json; waiting processes reuse the persisted service credential
• delete the exact management API key when post-create binding or persistence fails
• add deterministic six-process wiremock coverage for one key creation, one endpoint binding, valid merged credentials, and atomic replacement
• document the residual cross-project lost-update risk because endpoint upsert replaces the complete openApiKeys list without conditional updates

Closes #451

Tests

• cargo test -p clickhousectl --test cli_request_shape_test service_query_auto_provisioning_is_single_flight_across_processes -- --exact (1 passed)
• cargo test -p clickhousectl --test cli_request_shape_test service_query_deletes_the_exact_key_when_atomic_persistence_fails -- --exact (1 passed)
• cargo test -p clickhousectl --bin clickhousectl service_query (21 passed)
• cargo test -p clickhousectl --test cli_request_shape_test service_query (24 passed)
• cargo test -p clickhousectl --bin clickhousectl (609 passed)
• cargo test -p clickhousectl --test cli_request_shape_test (104 passed)
• cargo fmt --all --check
• cargo clippy -p clickhousectl --all-targets -- -D warnings

No live Cloud resources or credentials were used.

Stack

This is child 3 of the Cloud-query stack. It targets issue-455-query-input-conflict (PR #490), above issue-452-service-query-selector (PR #483). Both parent PRs remain unchanged and should merge first.

[macroscopeapp]

🟠 High cloud/service_query.rs:144

save_credentials(&creds) writes the stale snapshot captured before the network awaits, so concurrent updates from remove_service_query_key or authentication flows are overwritten, potentially resurrecting removed keys or reverting newly saved credentials. Atomic replacement only prevents partial writes; make all credential writers use this lock, or re-read and merge credentials.json immediately before saving.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @crates/clickhousectl/src/cloud/service_query.rs around line 144:

`save_credentials(&creds)` writes the stale snapshot captured before the network awaits, so concurrent updates from `remove_service_query_key` or authentication flows are overwritten, potentially resurrecting removed keys or reverting newly saved credentials. Atomic replacement only prevents partial writes; make all credential writers use this lock, or re-read and merge `credentials.json` immediately before saving.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a2dd2a6. Configure here.}

[cursor]

Stale credentials overwrite concurrent writers

High Severity

ensure_service_query_setup loads credentials.json once under the provision lock, then keeps that snapshot across create_api_key and bind_query_endpoint awaits before save_credentials. Auth login, logout, and remove_service_query_key still write the same file without that lock, so a later save can silently revert new API keys or resurrect deleted service_query_keys entries. Previously set_service_query_key reloaded immediately before writing.

Additional Locations (1)

• crates/clickhousectl/src/cloud/credentials.rs#L132-L140

 

^{Reviewed by Cursor Bugbot for commit a2dd2a6. Configure here.}