diff --git a/docs/en/operations/external-authenticators/tokens.md b/docs/en/operations/external-authenticators/tokens.md index aefa7e6eb549..4bc2151c2498 100644 --- a/docs/en/operations/external-authenticators/tokens.md +++ b/docs/en/operations/external-authenticators/tokens.md @@ -133,6 +133,10 @@ For JWKS-based validators (`jwt_static_jwks` and `jwt_dynamic_jwks`), RS* and ES This section covers two related kinds of processor: per-IdP convenience presets built on top of the generic JWT processors (currently `entra`), and the generic `openid` processor that talks to an arbitrary OIDC-compliant identity provider. +:::note +If the IdP issues access tokens that follow [RFC 9068](https://datatracker.ietf.org/doc/html/rfc9068) (the *JSON Web Token Profile for OAuth 2.0 Access Tokens*), the access token is itself a verifiable JWT and is best handled by one of the JWT processors above (typically `jwt_dynamic_jwks`) — no `/userinfo` or `/tokeninfo` round-trip is needed. The processors in this section exist for IdPs whose access tokens are opaque (e.g. Google), or whose JWT access tokens you prefer to validate by asking the IdP rather than locally. +::: + ### Entra (Microsoft Entra ID, pure OIDC) {#entra} `entra` is a preset for Microsoft Entra ID built on top of `jwt_dynamic_jwks`. Tokens are validated **locally** against Entra's per-tenant JWKS — no Microsoft Graph call, no userinfo round trip, no OIDC discovery fetch. `username_claim` and `groups_claim` are read directly from the JWT payload. Use this when the access token's `aud` is your own app (registered via Entra's *Expose an API* blade), not `https://graph.microsoft.com`. @@ -178,52 +182,118 @@ All remaining parameters are optional: - `expected_issuer` — Expected value of the `iss` claim. Default: `https://login.microsoftonline.com/{tenant_id}/v2.0` (derived from `tenant_id`). Override for v1.0 tokens (`https://sts.windows.net/{tenant_id}/`) or sovereign clouds. - `expected_audience` — Expected value of the `aud` claim, normally your app's Application ID URI (e.g. `api://clickhouse`) or client ID. If unset, no audience check is performed (any signature-valid token from the tenant will authenticate); a warning is logged at startup so the gap is visible. - `username_claim` — JWT claim to use as the ClickHouse username. Default: `sub`. Common Entra alternatives: `preferred_username`, `upn`, `oid`. -- `groups_claim` — JWT claim that carries the array of group identifiers. Default: `groups`. Set to `roles` if you use App Roles in Entra instead of security-group claims. +- `groups_claim` — JWT claim that carries the array of group identifiers. Default: `groups`. Set to `roles` when using App Roles. See [Mapping groups to ClickHouse roles](#entra-group-mapping) for how to get human-readable values instead of GUIDs. - `expected_typ`, `verifier_leeway`, `jwks_cache_lifetime`, `claims`, `allow_no_expiration`, `token_cache_lifetime` — Same as for `jwt_dynamic_jwks`. +#### Mapping groups to ClickHouse roles {#entra-group-mapping} + +By default the `groups` claim contains group **object IDs (GUIDs)**, not names. Three ways to surface human-readable identifiers, in order of preference: + +**Option A — App Roles** (recommended) + +Operator-chosen role strings in a separate `roles` claim. Compact even for users in many groups (no `hasgroups` overage indicator), and immune to Entra-side group renames. + +1. App registration → **App roles** → **Create app role**. Set `Value` to the string ClickHouse should receive (e.g. `ch_admin`); `Allowed member types` = `Users/Groups`. +2. Enterprise application → **Properties** → `Assignment required` = **Yes**. +3. Enterprise application → **Users and groups** → assign each user or security group to a role. Group assignment requires Entra ID P1/P2; free-tier tenants can only assign individual users here. +4. On the processor: `roles`. + +**Option B — Format the `groups` claim** + +Names emitted in the existing `groups` claim. Works on free tier; useful when group membership is already maintained in Entra and a separate role-assignment surface is not wanted. + +Prerequisites in the app registration: + +- `"groupMembershipClaims": "ApplicationGroup"` (or `"SecurityGroup"` for tenant-wide). +- `optionalClaims.accessToken` entry for `groups` with `additionalProperties` set to one or more of: + +| Value | Effect | +|---|---| +| `sam_account_name` | On-prem-synced groups emit as `sAMAccountName`. | +| `dns_domain_and_sam_account_name` | On-prem-synced groups emit as `DOMAIN\sAMAccountName`. | +| `cloud_displayname` | Cloud-only groups emit their `displayName`. | + +Entra picks per group; groups not covered by a chosen format still emit as GUIDs. Display names are mutable — a rename in Entra silently breaks the mapping until config is updated. + +Leave `groups` (the default). + +**Option C — `roles_mapping`** + +Keep GUIDs in the token and translate them in the user-directory config (see [Identity Provider as an External User Directory](#idp-external-user-directory)). Always works, including on free tier. Tedious for many groups but immune to renames. + :::note -The `groups` claim must be enabled in the app registration's manifest (`"groupMembershipClaims": "ApplicationGroup"` is recommended) and exposed in access tokens via `optionalClaims.accessToken`. Group identifiers in the token are object IDs (GUIDs) by default; map them to ClickHouse roles via the user-directory's `roles_mapping` block (see [Identity Provider as an External User Directory](#idp-external-user-directory)). +When switching from GUIDs to names, retune any `roles_filter` regex — for example `\bclickhouse-[a-zA-Z0-9]+\b` will not match strings like `ch_admin`. ::: ### OpenID + +The `openid` processor speaks the OIDC protocol surface — `/userinfo` for identity, plus (when discovered or configured) the local JWT fast-path against the IdP's JWKS and RFC 7662 token introspection. Two mutually-exclusive configuration shapes: + +- **Discovery** — point `configuration_endpoint` at `.well-known/openid-configuration`. Endpoints and the issuer are resolved from the doc. When it advertises `jwks_uri`, JWT access tokens (RFC 9068) are validated locally. When it advertises `introspection_endpoint` and you supply `introspection_client_id`/`introspection_client_secret`, RFC 7662 introspection runs on each authentication — alongside the JWT fast-path if both are available, since JWT validates signature/`exp` while introspection adds the revocation check. + +- **Manual** — `userinfo_endpoint` is mandatory. For RFC 9068 JWT access tokens prefer `jwt_dynamic_jwks`. Add `token_introspection_endpoint` + `introspection_client_id` + `introspection_client_secret` for RFC 7662 liveness, expiry, and `iss`/`aud` enforcement; without them, manual mode is `/userinfo` only. + ```xml - + openid - url/.well-known/openid-configuration - 60 - 3600 - - + https://idp.example.com/.well-known/openid-configuration + my-clickhouse-client-id + clickhouse-rs + ... + + + openid - url/userinfo - url/tokeninfo - url/.well-known/jwks.json - 60 - 3600 - + https://idp.example.com/userinfo + https://idp.example.com/introspect + clickhouse-rs + ... + https://idp.example.com + clickhouse-rs + ``` -:::note -Either `configuration_endpoint` or both `userinfo_endpoint` and `token_introspection_endpoint` (and, optionally, `jwks_uri`) shall be set. If none of them are set or all three are set, this is an invalid configuration that will not be parsed. +:::note Parser rules +- `configuration_endpoint` and `userinfo_endpoint` are mutually exclusive. +- `jwks_uri` is rejected in both shapes — use `jwt_dynamic_jwks` for an explicit JWKS URL. +- `introspection_client_id` and `introspection_client_secret` must be set together; both honor `from_env=` / `from_zk=` for secrets handling. +- In manual mode, `expected_issuer` / `expected_audience` are accepted only when introspection is wired (`/userinfo` carries neither claim and so cannot enforce them). ::: -**Parameters:** +#### Setting up the introspection client at your IdP -- `configuration_endpoint` - URI of OpenID configuration (often ends with `.well-known/openid-configuration`); -- `userinfo_endpoint` - URI of endpoint that returns user information in exchange for a valid token; -- `token_introspection_endpoint` - URI of token introspection endpoint (returns information about a valid token); -- `jwks_uri` - URI of OpenID configuration (often ends with `.well-known/jwks.json`) -- `jwks_cache_lifetime` - Period for resend request for refreshing JWKS. Optional, default: 3600. -- `verifier_leeway` - Clock skew tolerance (seconds). Useful for handling small differences in system clocks between ClickHouse and the token issuer. Optional, default: 60 -- `expected_issuer` - Expected value of the `iss` (issuer) claim in the JWT. If specified, tokens with a different issuer will be rejected. Optional. -- `expected_audience` - Expected value of the `aud` (audience) claim in the JWT. If specified, tokens with a different audience will be rejected. Optional. -- `allow_no_expiration` - If `true`, tokens without the `exp` (expiration) claim are accepted. Otherwise they are rejected. Optional, default: `false`. +Introspection needs an OAuth client representing ClickHouse-as-resource-server — separate from any user-facing client app, with no redirect URIs. + +| IdP | RFC 7662 introspection | How to create the introspection client | +|---|---|---| +| **Keycloak** | Yes | Realm → Clients → confidential client with *Service Accounts* enabled; copy `client_id` and the secret from the *Credentials* tab | +| **Okta** | Yes (Org AS + Custom AS) | Admin → Applications → Create App Integration → *API Services* | +| **Auth0** | Not for opaque user tokens | Auth0 does not provide `/introspect` for the opaque tokens issued at the `/userinfo` audience; for custom-API JWT access tokens use `jwt_dynamic_jwks` instead | +| **Google**, **GitHub**, **Microsoft Entra ID** (MS Graph) | No | No RFC 7662 endpoint — use the provider-specific processor (`google`) or JWT validation against your own API's tokens (`entra`, `jwt_dynamic_jwks`) | + +#### Parameters + +*Discovery mode:* +- `configuration_endpoint` — URI of the OIDC configuration document. Mandatory. +- `expected_issuer` — Expected `iss`. Enforced via the JWT fast-path or RFC 7662 introspection (whichever the discovery doc surfaces); also anchors the discovery doc's own `issuer` field. Optional. +- `expected_audience` — Expected `aud`. Same enforcement scope as `expected_issuer`. Optional. +- `introspection_client_id`, `introspection_client_secret` — `client_secret_basic` credentials for the introspection endpoint. Both must be set together. Optional; required only if you want introspection enabled. +- `allow_no_expiration` — Accept JWTs without `exp` on the JWT fast-path. Optional, default `false`. +- `verifier_leeway` — Clock-skew tolerance (seconds) for the JWT fast-path. Optional, default 60. +- `jwks_cache_lifetime` — JWKS refresh interval. Optional, default 3600. +- `allow_http_discovery_urls` — Allow non-HTTPS URLs returned by the discovery document. Optional, default `false`. + +*Manual mode:* +- `userinfo_endpoint` — URI of the OIDC userinfo endpoint. Mandatory. +- `token_introspection_endpoint` — URI of an RFC 7662 introspection endpoint. Optional; when set together with introspection credentials, enables liveness, `exp`, and `iss`/`aud` enforcement. +- `introspection_client_id`, `introspection_client_secret` — As above. Required iff `token_introspection_endpoint` is set. +- `expected_issuer`, `expected_audience` — Accepted only when introspection is wired; enforced against the introspection response. Optional. -Sometimes a token is a valid JWT. In that case token will be decoded and validated locally if configuration endpoint returns JWKS URI (or `jwks_uri` is specified alongside `userinfo_endpoint` and `token_introspection_endpoint`). +If the IdP issues access tokens that follow [RFC 9068](https://datatracker.ietf.org/doc/html/rfc9068), prefer `jwt_dynamic_jwks` for direct local validation. The `openid` processor is for opaque tokens (via userinfo and/or introspection) and for cases where you want to consult the IdP rather than validate locally. ### Tokens cache To reduce number of requests to IdP, tokens are cached internally for a maximum period of `token_cache_lifetime` seconds. diff --git a/src/Access/TokenProcessors.h b/src/Access/TokenProcessors.h index 35c6a218b5b1..7f2fea416980 100644 --- a/src/Access/TokenProcessors.h +++ b/src/Access/TokenProcessors.h @@ -197,19 +197,18 @@ class GoogleTokenProcessor : public ITokenProcessor class OpenIdTokenProcessor : public ITokenProcessor { public: - /// Specify endpoints manually + /// Manual mode: `/userinfo` for identity, plus RFC 7662 introspection + /// before it when an introspection endpoint and client credentials are set. OpenIdTokenProcessor(const String & processor_name_, UInt64 token_cache_lifetime_, const String & username_claim_, const String & groups_claim_, const String & expected_issuer_, const String & expected_audience_, - bool allow_no_expiration_, const String & userinfo_endpoint_, const String & token_introspection_endpoint_, - UInt64 verifier_leeway_, - const String & jwks_uri_, - UInt64 jwks_cache_lifetime_); + const String & introspection_client_id_, + const String & introspection_client_secret_); /// Obtain endpoints from openid-configuration URL OpenIdTokenProcessor(const String & processor_name_, @@ -222,15 +221,24 @@ class OpenIdTokenProcessor : public ITokenProcessor const String & openid_config_endpoint_, UInt64 verifier_leeway_, UInt64 jwks_cache_lifetime_, + const String & introspection_client_id_, + const String & introspection_client_secret_, const RemoteHostFilter & remote_host_filter_, bool allow_http_discovery_urls_); bool resolveAndValidate(TokenCredentials & credentials) const override; private: + /// True on `active=true`; populates `expires_at` from `exp` if present. + bool runIntrospection(const String & token, std::chrono::system_clock::time_point & expires_at) const; + Poco::URI userinfo_endpoint; Poco::URI token_introspection_endpoint; + String expected_issuer; + String expected_audience; + String introspection_client_id; + String introspection_client_secret; - /// Access token is often a valid JWT, so we can validate it locally to avoid unnecesary network requests. + /// Populated only by the discovery constructor when the doc advertises a `jwks_uri`. std::optional jwt_validator = std::nullopt; }; diff --git a/src/Access/TokenProcessorsOpaque.cpp b/src/Access/TokenProcessorsOpaque.cpp index 06471bf9af99..7fd8273c0c26 100644 --- a/src/Access/TokenProcessorsOpaque.cpp +++ b/src/Access/TokenProcessorsOpaque.cpp @@ -5,6 +5,8 @@ #include #include #include +#include +#include #include #include #include @@ -87,6 +89,7 @@ namespace std::ostringstream responseString; Poco::Net::HTTPRequest request{Poco::Net::HTTPRequest::HTTP_GET, uri.getPathAndQuery()}; + request.add("Accept", "application/json"); if (!token.empty()) request.add("Authorization", "Bearer " + token); @@ -118,6 +121,74 @@ namespace throw Exception(ErrorCodes::AUTHENTICATION_FAILED, "Failed to parse server response: {}", e.what()); } } + + /// RFC 7662 form-POST with `client_secret_basic` auth. Returns parsed JSON; + /// non-200 throws so callers can distinguish "inactive" (200+active:false) + /// from "client auth or transport failure". + picojson::object postFormToURI(const Poco::URI & uri, + const std::vector> & form, + const String & basic_user, + const String & basic_password) + { + Poco::Net::HTTPResponse response; + std::ostringstream responseString; + + String body; + for (const auto & [key, value] : form) + { + if (!body.empty()) + body += '&'; + String encoded_key; + String encoded_value; + Poco::URI::encode(key, "", encoded_key); + Poco::URI::encode(value, "", encoded_value); + body += encoded_key + "=" + encoded_value; + } + + Poco::Net::HTTPRequest request{Poco::Net::HTTPRequest::HTTP_POST, uri.getPathAndQuery(), + Poco::Net::HTTPMessage::HTTP_1_1}; + request.setContentType("application/x-www-form-urlencoded"); + request.setContentLength(body.size()); + request.add("Accept", "application/json"); + if (!basic_user.empty()) + { + Poco::Net::HTTPBasicCredentials creds(basic_user, basic_password); + creds.authenticate(request); + } + + auto send_and_receive = [&](Poco::Net::HTTPClientSession & session) + { + applyIdpSessionTimeouts(session); + session.sendRequest(request) << body; + Poco::StreamCopier::copyStream(session.receiveResponse(response), responseString); + }; + + if (uri.getScheme() == "https") + { + Poco::Net::HTTPSClientSession session(uri.getHost(), uri.getPort()); + send_and_receive(session); + } + else + { + Poco::Net::HTTPClientSession session(uri.getHost(), uri.getPort()); + send_and_receive(session); + } + + if (response.getStatus() != Poco::Net::HTTPResponse::HTTP_OK) + throw Exception(ErrorCodes::AUTHENTICATION_FAILED, + "POST to '{}' returned HTTP {} ({})", + uri.toString(), static_cast(response.getStatus()), response.getReason()); + + try + { + return parseJSON(responseString.str()); + } + catch (const std::runtime_error & e) + { + throw Exception(ErrorCodes::AUTHENTICATION_FAILED, + "Failed to parse JSON response from '{}': {}", uri.toString(), e.what()); + } + } } GoogleTokenProcessor::GoogleTokenProcessor(const String & processor_name_, @@ -289,50 +360,18 @@ OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, const String & groups_claim_, const String & expected_issuer_, const String & expected_audience_, - bool allow_no_expiration_, const String & userinfo_endpoint_, const String & token_introspection_endpoint_, - UInt64 verifier_leeway_, - const String & jwks_uri_, - UInt64 jwks_cache_lifetime_) + const String & introspection_client_id_, + const String & introspection_client_secret_) : ITokenProcessor(processor_name_, token_cache_lifetime_, username_claim_, groups_claim_), - userinfo_endpoint(userinfo_endpoint_), token_introspection_endpoint(token_introspection_endpoint_) + userinfo_endpoint(userinfo_endpoint_), + token_introspection_endpoint(token_introspection_endpoint_), + expected_issuer(expected_issuer_), + expected_audience(expected_audience_), + introspection_client_id(introspection_client_id_), + introspection_client_secret(introspection_client_secret_) { - /// Without `jwks_uri`, no `jwt_validator` is created and so `expected_issuer` - /// / `expected_audience` cannot be enforced anywhere on the validation path - /// -- the runtime falls straight to the userinfo endpoint, which only - /// answers "the IdP describes this user", not "the token's `iss`/`aud` - /// match what this deployment pinned". Refuse to load with that combination - /// rather than silently dropping the operator's bindings. - if (jwks_uri_.empty() && (!expected_issuer_.empty() || !expected_audience_.empty())) - throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, - "{}: 'expected_issuer' / 'expected_audience' are configured but no 'jwks_uri' is provided. " - "These bindings can only be enforced via local JWT validation against a JWKS; the userinfo " - "fallback alone cannot enforce them. Configure 'jwks_uri' (or, if you intentionally want " - "userinfo-only validation, clear 'expected_issuer'/'expected_audience').", - processor_name); - - if (!jwks_uri_.empty()) - { - LOG_TRACE(getLogger("TokenAuthentication"), "{}: JWKS URI set, local JWT processing will be attempted", processor_name_); - /// `expected_typ` is left empty here: OpenID's JWT-fastpath inherits no - /// `typ` enforcement from the operator config (the parser doesn't surface - /// `expected_typ` for the `openid` processor type yet). Operators who - /// want strict `typ` enforcement should use `jwt_static_jwks` / - /// `jwt_dynamic_jwks` directly instead of `openid`. - jwt_validator.emplace(processor_name_ + "jwks_val", - token_cache_lifetime_, - username_claim_, - groups_claim_, - expected_issuer_, - expected_audience_, - /*expected_typ=*/"", - allow_no_expiration_, - "", - verifier_leeway_, - jwks_uri_, - jwks_cache_lifetime_); - } } OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, @@ -345,9 +384,15 @@ OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, const String & openid_config_endpoint_, UInt64 verifier_leeway_, UInt64 jwks_cache_lifetime_, + const String & introspection_client_id_, + const String & introspection_client_secret_, const RemoteHostFilter & remote_host_filter_, bool allow_http_discovery_urls_) - : ITokenProcessor(processor_name_, token_cache_lifetime_, username_claim_, groups_claim_) + : ITokenProcessor(processor_name_, token_cache_lifetime_, username_claim_, groups_claim_), + expected_issuer(expected_issuer_), + expected_audience(expected_audience_), + introspection_client_id(introspection_client_id_), + introspection_client_secret(introspection_client_secret_) { /// Defense in depth: the discovery endpoint itself was already validated by /// the parser, but re-check here in case this constructor is reached via a @@ -365,11 +410,6 @@ OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, const picojson::object openid_config = getObjectFromURI(Poco::URI(openid_config_endpoint_)); - /// Only `userinfo_endpoint` is mandatory: it backs the runtime userinfo - /// fallback (and is the sole user-info source when no JWKS is configured). - /// `introspection_endpoint` is currently unused at runtime -- it's plumbed - /// for a future RFC 7662 introspection feature -- so a discovery document - /// that omits it should not block processor construction. if (!openid_config.contains("userinfo_endpoint")) throw Exception(ErrorCodes::AUTHENTICATION_FAILED, "{}: Cannot extract userinfo_endpoint from OIDC configuration at '{}'; consider manual configuration.", @@ -484,16 +524,24 @@ OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, if (openid_config.contains("introspection_endpoint")) token_introspection_endpoint = Poco::URI(getValueByKey(openid_config, "introspection_endpoint").value()); - /// See manual-constructor comment: `expected_issuer` / `expected_audience` - /// can only be enforced via local JWT validation. If the discovery document - /// does not advertise a `jwks_uri`, no `jwt_validator` will be created and - /// the userinfo fallback alone cannot enforce these bindings. Refuse the - /// configuration rather than silently dropping them. - if (!openid_config.contains("jwks_uri") && (!expected_issuer_.empty() || !expected_audience_.empty())) + const bool can_enforce_via_jwks = openid_config.contains("jwks_uri"); + const bool can_enforce_via_introspection = + openid_config.contains("introspection_endpoint") && !introspection_client_id_.empty(); + + /// Catch creds configured for a discovery doc that does not advertise an + /// introspection endpoint -- otherwise the credentials would be silently + /// ignored at runtime. + if (!introspection_client_id_.empty() && !openid_config.contains("introspection_endpoint")) throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, - "{}: OIDC discovery at '{}' did not advertise a 'jwks_uri', but 'expected_issuer' / " - "'expected_audience' are configured. These bindings can only be enforced via local JWT " - "validation against a JWKS; userinfo cannot enforce them. Refusing to load.", + "{}: 'introspection_client_id' / 'introspection_client_secret' are set but the OIDC " + "discovery at '{}' does not advertise an 'introspection_endpoint'.", + processor_name, openid_config_endpoint_); + + if (!can_enforce_via_jwks && !can_enforce_via_introspection + && (!expected_issuer_.empty() || !expected_audience_.empty())) + throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "{}: 'expected_issuer' / 'expected_audience' need either a 'jwks_uri' or an " + "'introspection_endpoint' (with operator credentials) in the discovery doc at '{}'.", processor_name, openid_config_endpoint_); if (openid_config.contains("jwks_uri")) @@ -515,6 +563,93 @@ OpenIdTokenProcessor::OpenIdTokenProcessor(const String & processor_name_, } } +bool OpenIdTokenProcessor::runIntrospection(const String & token, + std::chrono::system_clock::time_point & expires_at) const +{ + expires_at = {}; + + picojson::object response; + try + { + response = postFormToURI(token_introspection_endpoint, + {{"token", token}, {"token_type_hint", "access_token"}}, + introspection_client_id, + introspection_client_secret); + } + catch (const Exception & e) + { + /// LOG_WARNING (not TRACE): a non-200 from the introspection endpoint + /// almost always means the operator's `introspection_client_*` is + /// wrong or the IdP is unreachable -- worth surfacing by default. + LOG_WARNING(getLogger("TokenAuthentication"), + "{}: Token introspection request failed: {}", processor_name, e.message()); + return false; + } + + /// active=true is authoritative per RFC 7662 §2.2. + const auto active_opt = getValueByKey(response, "active"); + if (!active_opt.has_value() || !active_opt.value()) + { + LOG_TRACE(getLogger("TokenAuthentication"), + "{}: Token introspection reported active=false (or missing); rejecting", processor_name); + return false; + } + + if (!expected_issuer.empty()) + { + const auto iss = getValueByKey(response, "iss").value_or(""); + if (iss != expected_issuer) + { + LOG_TRACE(getLogger("TokenAuthentication"), + "{}: Token introspection 'iss' '{}' does not match expected_issuer '{}'; rejecting", + processor_name, iss, expected_issuer); + return false; + } + } + + /// `aud` may be a string or an array (RFC 7519 §4.1.3). + if (!expected_audience.empty()) + { + auto aud_it = response.find("aud"); + bool ok = false; + if (aud_it != response.end()) + { + const picojson::value & aud_val = aud_it->second; + if (aud_val.is()) + ok = (aud_val.get() == expected_audience); + else if (aud_val.is()) + for (const auto & v : aud_val.get()) + if (v.is() && v.get() == expected_audience) + ok = true; + } + if (!ok) + { + LOG_TRACE(getLogger("TokenAuthentication"), + "{}: Token introspection 'aud' does not contain expected_audience '{}'; rejecting", + processor_name, expected_audience); + return false; + } + } + + if (response.contains("exp")) + { + const auto exp_opt = getValueByKey(response, "exp"); + const double exp = exp_opt.value_or(0.0); + if (exp_opt.has_value() && std::isfinite(exp) && exp > 0.0 + && exp <= static_cast(std::numeric_limits::max())) + expires_at = std::chrono::system_clock::from_time_t(static_cast(exp)); + else + /// IdP advertised an `exp` we cannot use. Authentication still + /// succeeds (the token IS active), but the cache loses its tighter + /// upper bound; surface so operators see IdP drift. + LOG_WARNING(getLogger("TokenAuthentication"), + "{}: Token introspection returned malformed 'exp'; cache TTL falls back to token_cache_lifetime", + processor_name); + } + + return true; +} + bool OpenIdTokenProcessor::resolveAndValidate(TokenCredentials & credentials) const { const String & token = credentials.getToken(); @@ -555,7 +690,6 @@ bool OpenIdTokenProcessor::resolveAndValidate(TokenCredentials & credentials) co user_info_json = decoded_token.get_payload_json(); username = getValueByKey(user_info_json, username_claim).value(); - /// TODO: Now we work only with Keycloak -- and it provides expires_at in token itself. Need to add actual token introspection logic for other OIDC providers. if (decoded_token.has_expires_at()) credentials.setExpiresAt(decoded_token.get_expires_at()); } @@ -580,13 +714,18 @@ bool OpenIdTokenProcessor::resolveAndValidate(TokenCredentials & credentials) co } } - /// Userinfo path: only reachable when no `jwt_validator` is configured - /// (the constructor guarantees that combination is incompatible with any - /// `expected_issuer` / `expected_audience` pin), or when local JWT validation - /// passed but extracting the username/payload from the decoded token failed - /// for an unrelated reason -- in which case the bindings have already been - /// enforced by `jwt_validator` and userinfo is just being asked for the user - /// identity. + /// Run introspection whenever the operator configured it -- the JWT + /// fast-path validates signature/exp but cannot detect server-side + /// revocation, which is the whole reason to add introspection. + if (!token_introspection_endpoint.empty() && !introspection_client_id.empty()) + { + std::chrono::system_clock::time_point introspection_expires_at; + if (!runIntrospection(token, introspection_expires_at)) + return false; + if (introspection_expires_at != std::chrono::system_clock::time_point{}) + credentials.setExpiresAt(introspection_expires_at); + } + if (username.empty() || user_info_json.empty()) { try diff --git a/src/Access/TokenProcessorsParse.cpp b/src/Access/TokenProcessorsParse.cpp index f48ee3fddd07..015060cffb75 100644 --- a/src/Access/TokenProcessorsParse.cpp +++ b/src/Access/TokenProcessorsParse.cpp @@ -88,54 +88,95 @@ std::unique_ptr ITokenProcessor::parseTokenProcessor( } else if (provider_type == "openid") { - auto verifier_leeway = config.getUInt64(prefix + ".verifier_leeway", 60); - auto jwks_cache_lifetime = config.getUInt64(prefix + ".jwks_cache_lifetime", 3600); - - /// `token_introspection_endpoint` is currently unused at runtime: the - /// processor relies on JWT-local validation (when JWKS is configured) - /// or on userinfo, never on RFC 7662 introspection. Don't require it - /// for "locally configured" mode -- forcing operators to set a value - /// that does nothing is a footgun. If introspection is wired up later, - /// the field is already plumbed and can become required at that point. - bool externally_configured = config.hasProperty(prefix + ".configuration_endpoint") && !config.hasProperty(prefix + ".jwks_uri"); + bool externally_configured = config.hasProperty(prefix + ".configuration_endpoint"); bool locally_configured = config.hasProperty(prefix + ".userinfo_endpoint"); - if (externally_configured && ! locally_configured) + if (externally_configured && locally_configured) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': 'configuration_endpoint' and 'userinfo_endpoint' are mutually exclusive.", + processor_name); + + const auto introspection_client_id = config.getString(prefix + ".introspection_client_id", ""); + const auto introspection_client_secret = config.getString(prefix + ".introspection_client_secret", ""); + if (introspection_client_id.empty() != introspection_client_secret.empty()) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': 'introspection_client_id' and 'introspection_client_secret' " + "must be configured together.", + processor_name); + + auto reject_unsupported_key = [&](const char * key, const char * hint) { + if (config.hasProperty(prefix + "." + key)) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': '{}' is not supported in this mode. {}", + processor_name, key, hint); + }; + + if (externally_configured) + { + reject_unsupported_key("jwks_uri", + "In discovery mode the JWKS URL is resolved from the discovery document; " + "for an explicit JWKS URL use a 'jwt_dynamic_jwks' processor."); + + auto verifier_leeway = config.getUInt64(prefix + ".verifier_leeway", 60); + auto jwks_cache_lifetime = config.getUInt64(prefix + ".jwks_cache_lifetime", 3600); const auto configuration_endpoint = config.getString(prefix + ".configuration_endpoint"); require_allowed_url(configuration_endpoint, "configuration_endpoint"); - /// Opt-out for the HTTPS-on-discovery-returned-URLs check. False by - /// default; operators who knowingly run an IdP over plain HTTP can - /// enable it without falling back to manual trust-chain config. const auto allow_http_discovery_urls = config.getBool(prefix + ".allow_http_discovery_urls", false); return std::make_unique(processor_name, token_cache_lifetime, username_claim, groups_claim, expected_issuer, expected_audience, allow_no_expiration, configuration_endpoint, verifier_leeway, jwks_cache_lifetime, + introspection_client_id, + introspection_client_secret, remote_host_filter, allow_http_discovery_urls); } - else if (locally_configured && !externally_configured) + + if (locally_configured) { - const auto userinfo_endpoint = config.getString(prefix + ".userinfo_endpoint"); + reject_unsupported_key("jwks_uri", + "For local JWT validation against a JWKS use a 'jwt_dynamic_jwks' processor."); + reject_unsupported_key("allow_no_expiration", "It applies only to JWT validation."); + reject_unsupported_key("verifier_leeway", "It applies only to JWT validation."); + reject_unsupported_key("jwks_cache_lifetime", "It applies only to JWKS-backed processors."); + const auto token_introspection_endpoint = config.getString(prefix + ".token_introspection_endpoint", ""); - const auto jwks_uri = config.getString(prefix + ".jwks_uri", ""); + const bool has_introspection = !token_introspection_endpoint.empty(); + + if (has_introspection && introspection_client_id.empty()) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': 'token_introspection_endpoint' is set but " + "'introspection_client_id' / 'introspection_client_secret' are not.", + processor_name); + if (!has_introspection && !introspection_client_id.empty()) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': 'introspection_client_id' / 'introspection_client_secret' " + "are set but no 'token_introspection_endpoint' is configured.", + processor_name); + + if ((config.hasProperty(prefix + ".expected_issuer") || config.hasProperty(prefix + ".expected_audience")) + && !has_introspection) + throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, + "Token processor '{}': 'expected_issuer' / 'expected_audience' need either a " + "'token_introspection_endpoint' (RFC 7662) or a 'jwt_dynamic_jwks' processor.", + processor_name); + + const auto userinfo_endpoint = config.getString(prefix + ".userinfo_endpoint"); require_allowed_url(userinfo_endpoint, "userinfo_endpoint"); require_allowed_url(token_introspection_endpoint, "token_introspection_endpoint"); - require_allowed_url(jwks_uri, "jwks_uri"); return std::make_unique(processor_name, token_cache_lifetime, username_claim, groups_claim, - expected_issuer, expected_audience, allow_no_expiration, + expected_issuer, expected_audience, userinfo_endpoint, token_introspection_endpoint, - verifier_leeway, - jwks_uri, - jwks_cache_lifetime); + introspection_client_id, + introspection_client_secret); } throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, - "Either 'configuration_endpoint' or 'userinfo_endpoint' " - "(and, optionally, 'token_introspection_endpoint' / 'jwks_uri') must be specified for 'openid' processor"); + "Either 'configuration_endpoint' (discovery) or 'userinfo_endpoint' (manual) " + "must be specified for 'openid' processor"); } else if (provider_type == "entra") { @@ -256,8 +297,6 @@ std::unique_ptr ITokenProcessor::parseTokenProcessor( } else throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, "Invalid type: {}", provider_type); - - // throw DB::Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, "Failed to parse token processor: {}", processor_name); } #else diff --git a/tests/integration/compose/docker_compose_mock_oidc.yml b/tests/integration/compose/docker_compose_mock_oidc.yml new file mode 100644 index 000000000000..a17532905c0d --- /dev/null +++ b/tests/integration/compose/docker_compose_mock_oidc.yml @@ -0,0 +1,15 @@ +services: + mock-oidc: + image: nginx:alpine + volumes: + - ${MOCK_OIDC_CONFIG_FILE}:/usr/share/nginx/html/.well-known/openid-configuration:ro + ports: + - "${MOCK_OIDC_EXTERNAL_PORT:-18091}:80" + healthcheck: + test: + - CMD-SHELL + - > + wget -qO- http://localhost/.well-known/openid-configuration > /dev/null || exit 1 + interval: 5s + timeout: 3s + retries: 15 diff --git a/tests/integration/helpers/cluster.py b/tests/integration/helpers/cluster.py index aafcdd09c8e5..329ab0905c3a 100644 --- a/tests/integration/helpers/cluster.py +++ b/tests/integration/helpers/cluster.py @@ -657,6 +657,7 @@ def __init__( self.with_cassandra = False self.with_ldap = False self.with_keycloak = False + self.with_mock_oidc = False self.with_jdbc_bridge = False self.with_nginx = False self.with_hive = False @@ -764,6 +765,11 @@ def __init__( self.keycloak_port = 18080 self.base_keycloak_cmd = None + # available when with_mock_oidc == True + self.mock_oidc_host = "mock-oidc" + self.mock_oidc_port = 18091 + self.base_mock_oidc_cmd = None + # available when with_rabbitmq == True self.rabbitmq_host = "rabbitmq1" self.rabbitmq_ip = None @@ -1822,6 +1828,25 @@ def setup_keycloak_cmd(self, instance, env_variables, docker_compose_yml_dir): ) return self.base_keycloak_cmd + def setup_mock_oidc_cmd(self, instance, env_variables, docker_compose_yml_dir): + self.with_mock_oidc = True + env_variables["MOCK_OIDC_EXTERNAL_PORT"] = str(self.mock_oidc_port) + env_variables["MOCK_OIDC_CONFIG_FILE"] = p.join( + self.base_dir, + "mock_oidc", + "openid-configuration", + ) + self.base_cmd.extend( + ["--file", p.join(docker_compose_yml_dir, "docker_compose_mock_oidc.yml")] + ) + self.base_mock_oidc_cmd = self.compose_cmd( + "--env-file", + instance.env_file, + "--file", + p.join(docker_compose_yml_dir, "docker_compose_mock_oidc.yml"), + ) + return self.base_mock_oidc_cmd + def setup_jdbc_bridge_cmd(self, instance, env_variables, docker_compose_yml_dir): self.with_jdbc_bridge = True env_variables["JDBC_DRIVER_LOGS"] = self.jdbc_driver_logs_dir @@ -1988,6 +2013,7 @@ def add_instance( with_cassandra=False, with_ldap=False, with_keycloak=False, + with_mock_oidc=False, with_jdbc_bridge=False, with_hive=False, with_coredns=False, @@ -2131,6 +2157,7 @@ def add_instance( with_cassandra=with_cassandra, with_ldap=with_ldap, with_keycloak=with_keycloak, + with_mock_oidc=with_mock_oidc, with_iceberg_catalog=with_iceberg_catalog, with_glue_catalog=with_glue_catalog, with_hms_catalog=with_hms_catalog, @@ -2395,6 +2422,11 @@ def add_instance( self.setup_keycloak_cmd(instance, env_variables, docker_compose_yml_dir) ) + if with_mock_oidc and not self.with_mock_oidc: + cmds.append( + self.setup_mock_oidc_cmd(instance, env_variables, docker_compose_yml_dir) + ) + if with_jdbc_bridge and not self.with_jdbc_bridge: cmds.append( self.setup_jdbc_bridge_cmd( @@ -3388,6 +3420,26 @@ def wait_keycloak_to_start(self, timeout=120): def get_keycloak_url(self): return f"http://localhost:{self.keycloak_port}" + def wait_mock_oidc_to_start(self, timeout=60): + url = ( + f"http://localhost:{self.mock_oidc_port}" + f"/.well-known/openid-configuration" + ) + start = time.time() + while time.time() - start < timeout: + try: + resp = requests.get(url, timeout=5) + if resp.status_code == 200: + logging.info("mock-oidc is online") + return + except Exception as ex: + logging.warning("Waiting for mock-oidc: %s", ex) + time.sleep(2) + raise Exception("mock-oidc did not start in time") + + def get_mock_oidc_url(self): + return f"http://localhost:{self.mock_oidc_port}" + def wait_prometheus_to_start(self): if "writer" in self.prometheus_servers: self.prometheus_writer_ip = self.get_instance_ip(self.prometheus_writer_host) @@ -3883,6 +3935,11 @@ def logging_azurite_initialization(exception, retry_number, sleep_time): self.up_called = True self.wait_keycloak_to_start() + if self.with_mock_oidc and self.base_mock_oidc_cmd: + subprocess_check_call(self.base_mock_oidc_cmd + ["up", "-d"]) + self.up_called = True + self.wait_mock_oidc_to_start() + if self.with_jdbc_bridge and self.base_jdbc_bridge_cmd: os.makedirs(self.jdbc_driver_logs_dir) os.chmod(self.jdbc_driver_logs_dir, stat.S_IRWXU | stat.S_IRWXO) @@ -4368,6 +4425,7 @@ def __init__( with_cassandra, with_ldap, with_keycloak, + with_mock_oidc, with_iceberg_catalog, with_glue_catalog, with_hms_catalog, @@ -4492,6 +4550,7 @@ def __init__( self.with_cassandra = with_cassandra self.with_ldap = with_ldap self.with_keycloak = with_keycloak + self.with_mock_oidc = with_mock_oidc self.with_jdbc_bridge = with_jdbc_bridge self.with_hive = with_hive self.with_coredns = with_coredns @@ -5849,6 +5908,9 @@ def write_embedded_config(name, dest_dir, fix_log_level=False): if self.with_keycloak: depends_on.append("keycloak") + if self.with_mock_oidc: + depends_on.append("mock-oidc") + if self.with_rabbitmq: depends_on.append("rabbitmq1") diff --git a/tests/integration/test_keycloak_auth/configs/validators_discovery_introspect.xml b/tests/integration/test_keycloak_auth/configs/validators_discovery_introspect.xml new file mode 100644 index 000000000000..5397e9b7a6f2 --- /dev/null +++ b/tests/integration/test_keycloak_auth/configs/validators_discovery_introspect.xml @@ -0,0 +1,17 @@ + + + + + openid + http://mock-oidc/.well-known/openid-configuration + clickhouse + test-secret + preferred_username + 3 + true + + + diff --git a/tests/integration/test_keycloak_auth/configs/validators_manual_introspect.xml b/tests/integration/test_keycloak_auth/configs/validators_manual_introspect.xml new file mode 100644 index 000000000000..c14800ed9243 --- /dev/null +++ b/tests/integration/test_keycloak_auth/configs/validators_manual_introspect.xml @@ -0,0 +1,15 @@ + + + + + openid + http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/userinfo + http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/token/introspect + clickhouse + test-secret + preferred_username + 3 + + + diff --git a/tests/integration/test_keycloak_auth/configs/validators_manual_introspect_bad_secret.xml b/tests/integration/test_keycloak_auth/configs/validators_manual_introspect_bad_secret.xml new file mode 100644 index 000000000000..702761d20a83 --- /dev/null +++ b/tests/integration/test_keycloak_auth/configs/validators_manual_introspect_bad_secret.xml @@ -0,0 +1,16 @@ + + + + + openid + http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/userinfo + http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/token/introspect + clickhouse + wrong-secret + preferred_username + 3 + + + diff --git a/tests/integration/test_keycloak_auth/mock_oidc/openid-configuration b/tests/integration/test_keycloak_auth/mock_oidc/openid-configuration new file mode 100644 index 000000000000..fdd3b314b35f --- /dev/null +++ b/tests/integration/test_keycloak_auth/mock_oidc/openid-configuration @@ -0,0 +1,5 @@ +{ + "issuer": "http://keycloak:8080/realms/clickhouse-test", + "userinfo_endpoint": "http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/userinfo", + "introspection_endpoint": "http://keycloak:8080/realms/clickhouse-test/protocol/openid-connect/token/introspect" +} diff --git a/tests/integration/test_keycloak_auth/test.py b/tests/integration/test_keycloak_auth/test.py index 46a92071adb4..cb03a14e8de8 100644 --- a/tests/integration/test_keycloak_auth/test.py +++ b/tests/integration/test_keycloak_auth/test.py @@ -35,6 +35,34 @@ stay_alive=True, ) +# Each introspection scenario gets its own node so the targeted processor is the +# only one that can authenticate a token -- otherwise we cannot tell which +# processor handled a successful auth. +node_manual_introspect = cluster.add_instance( + "node_manual_introspect", + main_configs=["configs/validators_manual_introspect.xml"], + user_configs=["configs/users.xml"], + with_keycloak=True, + stay_alive=True, +) + +node_discovery_introspect = cluster.add_instance( + "node_discovery_introspect", + main_configs=["configs/validators_discovery_introspect.xml"], + user_configs=["configs/users.xml"], + with_keycloak=True, + with_mock_oidc=True, + stay_alive=True, +) + +node_manual_introspect_bad = cluster.add_instance( + "node_manual_introspect_bad", + main_configs=["configs/validators_manual_introspect_bad_secret.xml"], + user_configs=["configs/users.xml"], + with_keycloak=True, + stay_alive=True, +) + @pytest.fixture(scope="module", autouse=True) def started_cluster(): @@ -418,3 +446,119 @@ def test_device_flow_round_trip(started_cluster): # --- 4. Use the token to authenticate a ClickHouse query --- result = query_with_token(node, id_token, "SELECT 1") assert result.strip() == "1" + + +KEYCLOAK_INTROSPECT_PATH = ( + f"/realms/{KEYCLOAK_REALM}/protocol/openid-connect/token/introspect" +) +KEYCLOAK_REVOKE_PATH = ( + f"/realms/{KEYCLOAK_REALM}/protocol/openid-connect/revoke" +) + +# Pin Host header on backchannel calls so tokens have the same `iss` as the URL +# ClickHouse uses to introspect them (the existing helpers used by other tests +# keep the host-mapped URL so device-flow HTML redirects stay reachable). +KEYCLOAK_BACKCHANNEL_HOST = "keycloak:8080" + + +def _keycloak_backchannel_headers(): + return {"Host": KEYCLOAK_BACKCHANNEL_HOST} + + +def get_keycloak_access_token(started_cluster, username="alice", password="secret"): + url = f"{keycloak_url(started_cluster)}/realms/{KEYCLOAK_REALM}/protocol/openid-connect/token" + data = { + "grant_type": "password", + "client_id": KEYCLOAK_CLIENT_ID, + "client_secret": KEYCLOAK_CLIENT_SECRET, + "username": username, + "password": password, + "scope": "openid profile email", + } + resp = requests.post(url, data=data, headers=_keycloak_backchannel_headers(), timeout=30) + resp.raise_for_status() + body = resp.json() + assert "access_token" in body, f"No access_token in response: {body}" + return body["access_token"] + + +def introspect_directly(started_cluster, token): + """POST to Keycloak's introspection endpoint with the client credentials + we use in the ClickHouse config. Returns the parsed JSON body.""" + url = f"{keycloak_url(started_cluster)}{KEYCLOAK_INTROSPECT_PATH}" + resp = requests.post( + url, + data={"token": token, "token_type_hint": "access_token"}, + auth=(KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET), + headers=_keycloak_backchannel_headers(), + timeout=10, + ) + resp.raise_for_status() + return resp.json() + + +def revoke_keycloak_token(started_cluster, token): + url = f"{keycloak_url(started_cluster)}{KEYCLOAK_REVOKE_PATH}" + resp = requests.post( + url, + data={"token": token, "token_type_hint": "access_token"}, + auth=(KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET), + headers=_keycloak_backchannel_headers(), + timeout=10, + ) + resp.raise_for_status() + + +def _expect_auth_failure(node_instance, token): + """Assert that ClickHouse rejects the token with a 401/403, not some + other 5xx that would falsely satisfy a blanket-catch test.""" + try: + query_with_token(node_instance, token, "SELECT 1") + except requests.HTTPError as ex: + assert ex.response.status_code in (401, 403), \ + f"expected 401/403, got {ex.response.status_code}: {ex.response.text}" + return + pytest.fail("Expected authentication failure but query succeeded") + + +# token_cache_lifetime in the introspection validator configs. +INTROSPECT_CACHE_TTL_SECONDS = 3 + + +def _assert_revocation_detected(started_cluster, node_instance): + """Fresh token works, then is revoked, then is rejected after the cache TTL + elapses. Only passes when introspection runs on the second request.""" + token = get_keycloak_access_token(started_cluster) + + assert introspect_directly(started_cluster, token)["active"] is True + assert query_with_token(node_instance, token, "SELECT 1").strip() == "1" + + revoke_keycloak_token(started_cluster, token) + + # Wait past the cache TTL with a margin generous enough for slow CI. + time.sleep(INTROSPECT_CACHE_TTL_SECONDS + 3) + + assert introspect_directly(started_cluster, token)["active"] is False + _expect_auth_failure(node_instance, token) + + +def test_manual_introspect_detects_revocation(started_cluster): + """Manual mode: opaque-flow introspection rejects a token after revocation.""" + _assert_revocation_detected(started_cluster, node_manual_introspect) + + +def test_discovery_introspect_detects_revocation(started_cluster): + """Discovery mode against a mock OIDC doc that omits jwks_uri: the only + available validation path is RFC 7662, exercised end-to-end here.""" + _assert_revocation_detected(started_cluster, node_discovery_introspect) + + +def test_manual_introspect_rejects_on_bad_client_secret(started_cluster): + """When the resource server cannot authenticate to the introspection + endpoint (Keycloak returns 401), ClickHouse must reject the bearer token + rather than fall through to /userinfo.""" + token = get_keycloak_access_token(started_cluster) + # Sanity: the token itself is fine -- the failure must come from the + # ClickHouse-side introspection auth, not from the token being invalid. + assert introspect_directly(started_cluster, token)["active"] is True + _expect_auth_failure(node_manual_introspect_bad, token)