Authenticate Python downloads discovered from Simple API responses

[kbukum1]

What are you trying to accomplish?

Allow the proxy to authenticate Python package file downloads when an authenticated Simple API response points to a different stable download path than the configured index URL.

This is needed for Azure Artifacts Python indexes. Dependabot authenticates requests to the configured Simple API path, such as /pypi/simple/..., but Azure can return package links under rewritten /pypi/download/... paths that include internal project/feed identifiers. Those download URLs no longer match the original credential path, so hosted Dependabot can reach the package listing but fail to download the package file.

This affects dependabot/dependabot-core#15207.

Anything you want to highlight for special attention from reviewers?

The implementation intentionally keeps authentication scoped. It does not broaden Python index credentials to the whole host. Instead, the proxy only learns Azure Artifacts download prefixes from authenticated Python Simple API responses, then reuses the same basic or OIDC credential for later requests under those discovered prefixes.

Discovery is limited to successful Simple API HTML/JSON responses, bounded in memory, restricted to same-origin and same-organization download links, and capped so a long-running proxy cannot learn unbounded prefixes.

How will you know you've accomplished your goal?

The proxy test coverage exercises Python Simple API download-prefix discovery for both static and OIDC-backed credentials, including scoped matching, oversized responses, custom ports, IPv6 hosts, and negative cases where discovery must not broaden auth.

Validation run:

• go test ./internal/handlers ./internal/oidc

Checklist

• I have run the complete test suite to ensure all tests and linters pass.
• I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
• I have written clear and descriptive commit messages.
• I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
• I have ensured that the code is well-documented and easy to understand.

[Copilot]

Pull request overview

This PR enhances the Python index handler so it can reuse the same authentication for package file download URLs whose stable prefix is discovered from authenticated PEP 503/PEP 691 Simple API responses (instead of broadening credentials to the entire host).

Changes:

• Registers a Python index response hook in the proxy pipeline to learn download URL prefixes from Simple API responses.
• Refactors Python index auth application to reuse a shared auth representation (basic or OIDC) and persist it across request/response handling via context.
• Extends OIDC registry APIs to support “select credential” and “apply auth with known credential” flows.

Show a summary per file

File	Description
proxy.go	Wires PythonIndexHandler.HandleResponse into the response chain.
internal/oidc/oidc_registry.go	Adds CredentialForRequest and TryAuthCredential to support external matching with centralized header behavior.
internal/handlers/python_index.go	Uses discovered download-prefix auth first; records request auth in context for response learning.
internal/handlers/python_index_auth.go	Introduces auth/context plumbing and an in-memory store for discovered download-prefix auth.
internal/handlers/python_index_download_prefixes.go	Parses Simple API HTML/JSON responses to discover and store stable download prefixes.
internal/handlers/python_index_test.go	Adds tests for discovered download-prefix authentication from HTML and JSON responses.
internal/handlers/oidc_handling_test.go	Adds an OIDC-based test to ensure discovered download prefixes reuse OIDC auth.

Copilot's findings

• Files reviewed: 7/7 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 7/7 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 7/7 changed files
• Comments generated: 2

[Copilot]

Copilot's findings

• Files reviewed: 8/8 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 8/8 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 8/8 changed files
• Comments generated: 0 new

[JamieMagee]

I'd like a little bit more test coverage, but it's not a blocker, and everything else looks fine.