Repair stale stored query keys

[sdairs]

Summary

• add explicit cloud service query --repair-query-key replacement for one stored service credential
• turn stored-key Query API 401/403 responses into actionable, non-mutating repair guidance
• require exact saved organization, management key, endpoint, and binding ownership before repair
• replace only the owned endpoint key UUID while preserving other endpoint keys, roles, origins, project API credentials, and service credentials
• retain the inherited provisioning lock, atomic credential write, rollback on persistence failure, and newly provisioned endpoint readiness retry
• refuse legacy or missing records instead of guessing or silently provisioning

Tests

• cargo test -p clickhousectl
• cargo test -p clickhousectl --test cli_request_shape_test service_query_repair_replaces_only_the_exact_owned_key_and_binding -- --exact
• cargo test -p clickhousectl cloud::services::tests::parses_service_query
• cargo fmt --all --check
• cargo clippy -p clickhousectl --all-targets -- -D warnings

Subprocess coverage pins stored-key 401 and 403 guidance with no writes, exact binding replacement when the old management key is already absent, unrelated credential preservation, and safe legacy-record refusal.

Stack

This is the child of issue-453-query-endpoint-readiness (PR #504) in gh-stack #505. It targets the parent branch; PR #504 should merge first.

Closes #454

_{Stack created with GitHub Stacks CLI • Give Feedback}

[macroscopeapp]

🟠 High cloud/service_query.rs:311

When saving the repaired credentials fails and restoring the old endpoint also fails, this code deletes the replacement API key, leaving the endpoint without a usable key while local credentials still reference the old unbound key. Only delete the replacement after rollback succeeds; otherwise retain it for recovery.

        let rollback_error = client
            .create_query_endpoint(org_id, service_id, &rollback_request)
            .await
            .err();
-        discard_api_key(client, org_id, &key.api_key_id).await;
         return match rollback_error {
             Some(rollback_error) => Err(CloudError::new(format!(
                 "{save_error}; additionally failed to restore the previous query endpoint binding: {rollback_error}"
             ))),
-            None => Err(save_error),
+            None => {
+                discard_api_key(client, org_id, &key.api_key_id).await;
+                Err(save_error)
+            }
         };

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @crates/clickhousectl/src/cloud/service_query.rs around lines 311-321:

When saving the repaired credentials fails and restoring the old endpoint also fails, this code deletes the replacement API key, leaving the endpoint without a usable key while local credentials still reference the old unbound key. Only delete the replacement after rollback succeeds; otherwise retain it for recovery.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 428ce34. Configure here.}

[cursor]

Concurrent repair deletes winner's key

Medium Severity

repair_service_query_setup takes the shared provisioning lock and always rotates the stored key, with no post-lock check that another waiter already repaired it. ensure_service_query_setup reuses the winner's credential after the same lock. A second concurrent --repair-query-key can delete the first process's new key after that process has dropped the lock and is about to query with it in memory.

Additional Locations (1)

• crates/clickhousectl/src/cloud/services.rs#L1869-L1897

 

^{Reviewed by Cursor Bugbot for commit 428ce34. Configure here.}