Skip to content

fix(connect): proxy OIDC token requests and close refresh sessions - #2223

Open
dudanogueira wants to merge 2 commits into
weaviate:mainfrom
dudanogueira:fix/oidc-session-transports-and-leak
Open

dudanogueira wants to merge 2 commits into
weaviate:mainfrom
dudanogueira:fix/oidc-session-transports-and-leak

Conversation

@dudanogueira

Copy link
Copy Markdown
Contributor

Stacked on #2222. Until that merges, this diff also shows its commit. Only the last commit, fix(connect): proxy OIDC token requests and close refresh sessions, belongs to this PR.

Motivation

While fixing #2220 in #2222, I found two more bugs in the OIDC token handling. Neither depends on httpx2. Both reproduce with authlib 1.6/1.7 as well.

  1. Token requests ignore the configured proxies. The authlib sessions are created without transports, so fetch_token and the refresh requests bypass AdditionalConfig(proxies=...), pool limits and retries. The OIDC discovery requests do use the proxy, because they go through our own client. In a network where the IdP is only reachable through the proxy, OIDC login fails.
  2. The client-credentials refresher leaks a session on every refresh. With no refresh token, each background refresh calls get_auth_session(), which builds a new session that already fetches a token. It then calls fetch_token() again and never closes the session. Each refresh costs one leaked connection pool and two token requests instead of one. Refreshes run every expires_in - 30 seconds, so a long-running client keeps accumulating open pools.

Approach

  • Transports: _make_mounts became _make_mounts_with(http, colour), which takes the HTTP library as a parameter. Our own client still uses httpx. The sessions get transports built with the library authlib builds them on: httpx2 for Authlib ≥ 1.8 when installed, otherwise httpx. That library is detected once, from the session class's bases (_session_http_module). Building them with plain httpx would bring back the error when client connect #2220 class of bug: an httpx2 client receiving httpx objects.
  • Leak: the refresher keeps the token the new session already fetched, then closes the session in a finally.

Key areas for review

  • authentication.py:_session_http_module: detects the library from the MRO. It falls back to httpx, which matches authlib's own fallback.
  • v4.py:_make_mounts_with: the same transport settings as before, built with a given library.
  • The refresher's client-credentials branch (sync and async).

Risks and mitigations

  • Behaviour change: token requests now go through the configured proxies. That's the documented meaning of proxies, and it's what every other request already does. Users with no proxies configured get an empty mounts, so nothing changes for them.
  • Token freshness after the leak fix: the session from get_auth_session() has always fetched its token explicitly before returning (see the comment in _get_session_client_credential). Reusing that token instead of fetching a second one doesn't change which token is used.

Testing

All four new tests fail on #2222 and pass here.

  • test_oidc_token_requests_use_proxies (+ async): Weaviate and the IdP sit behind *.invalid hosts that only a proxy (a second HTTPServer) can reach. Connect and an authenticated call succeed only if the token request also goes through the proxy.
  • test_client_credentials_refresh_closes_sessions (+ async): tokens expire right away, sessions are tracked over about 3 refreshes, and the test asserts every session is closed and exactly one token request is made per session.
  • The full mock_tests/ suite (81 tests) passes with authlib 1.6.12 (CI's pin), authlib 1.9.1 + httpx2, and authlib 1.9.1 without httpx2 (test_auth.py).
  • End to end against a local fake IdP behind a logging forward proxy: the token POST goes through the proxy with authlib 1.9.1 + httpx2 and with 1.7.2. Over 3.5 s of refreshes, open sessions after close() went from 3 to 0 per connection, and token requests from 7 to 4.

…te#2220)

With OIDC auth the authlib session was the HTTP client for every request.
Authlib >= 1.8 builds its sessions on httpx2 whenever httpx2 is installed
(e.g. pulled in by openai/langchain), so the httpx Timeout we pass on each
request was read as a single number and every call failed: async with
"unsupported operand type(s) for +: 'float' and 'Timeout'", sync with
"'Timeout' object cannot be interpreted as an integer".

The session now only holds and refreshes the token. Requests go through the
same httpx client API-key and anonymous connections use; we already set the
bearer header per request and refresh in the background, and authlib's
inline refresh lives in request()/stream(), which we never called. close()
closes the session too, and token-fetch failures from an httpx2-based
session raise AuthenticationFailedError like httpx ones.

As a side effect OIDC requests now honour the configured proxies, pool
limits and retries, which the session never received.
Two problems in the OIDC token handling, both independent of the httpx2
fix they build on:

- The authlib sessions were built without transports, so token requests
  (fetch and refresh) ignored the configured proxies, pool limits and
  retries. They now get the same transports as our own client, built with
  the library authlib uses (httpx2 for Authlib >= 1.8 when installed, else
  httpx), since a session must not be handed another library's transports.
- With client credentials, every background refresh built a new session
  (which already fetches a token), fetched a second token with it and never
  closed it: one leaked connection pool and two token requests per refresh.
  The refresher now keeps the new session's token and closes it.

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant