feat(fetch): add opt-in host allowlisting (--allowed-hosts)

[g-yixuan]

Description

Adds an opt-in host allowlist to the fetch server, addressing the SSRF surface tracked in #2317 without changing default behavior.

Fixes #2317.

• New --allowed-hosts HOST [HOST ...] CLI flag. When unset, behavior is exactly as before (all hosts allowed), so deployments that intentionally fetch localhost/internal endpoints are unaffected (per the discussion on the issue).
• Entry forms: exact host, case-insensitive (example.com); wildcard (*.example.com) matching the bare domain and any subdomain; literal IPs.
• Enforcement covers the initial request, the robots.txt pre-check, and every redirect hop: redirects are now followed manually (bounded at 20, same as httpx's default) and each hop is re-validated before connecting, so a 302 from an allowed host can no longer bounce a fetch to a disallowed host.
• Denials fail closed with a clear error that does not echo the configured allowlist.

Publishing Your Server

N/A — change to an existing server.

Server Details

• Server: fetch (Python, mcp-server-fetch)
• Changes to: tools/prompts request handling (URL validation + redirect handling), CLI args, README

Motivation and Context

#2317 asks for host allowlisting. The issue discussion raised valid concerns about breaking users who intentionally fetch internal endpoints, so this PR deliberately does not change the default posture — the default-deny/IP-range part of the issue is separately covered by #4497, and this PR composes cleanly with it. A previous allowlisting attempt (#2568) went stale and was closed with an invitation to resubmit on a current base; this is that fresh take, scoped to avoid the default-behavior controversy.

On the TOCTOU/socket-level point in the issue: with a hostname allowlist, the validated hostname is exactly what gets connected (SNI/Host), so DNS check-time/connect-time skew does not weaken the control; the realistic bypass was redirect hops, which this closes.

How Has This Been Tested?

• uv run pytest — 64 passed (existing suite untouched and green, plus new coverage: exact/wildcard/IP matching, case/trailing-dot normalization, userinfo bypass attempt, per-hop redirect enforcement parametrized over 301/302/303/307/308, redirect-loop limit, empty and malformed Location headers (matching httpx's prior behavior), relative Location resolution, robots.txt path enforcement, per-request timeout semantics, and unset-flag backwards compatibility).
• uv run pyright — 0 errors; uv run ruff check . — clean.
• Exercised end-to-end over the MCP protocol: stdio client against the real server binary started with --allowed-hosts 127.0.0.1, plus a real local HTTP server — allowlisted host fetched, http://169.254.169.254/latest/meta-data blocked, redirect to a non-allowlisted host blocked, malformed Location handled with a clean error.
• URL validation uses httpx.URL — the same parser that performs the connection — so the validated host is always the host that gets connected.

Breaking Changes

None — default behavior is unchanged when --allowed-hosts is not set.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Protocol Documentation
• My changes follows MCP security best practices
• I have updated the server's README accordingly
• I have tested this with an LLM client
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have documented all environment variables and configuration options

Additional context

• Wildcard semantics are documented in the README: *.example.com matches example.com itself and any subdomain.
• The README also states an intentional limitation: the allowlist matches hostnames, not resolved IPs, which is what makes deliberately allowlisting internal hosts possible.