feat: use sublevel llms.txt per product for MCP resources

[mattpodwysocki]

Summary

docs.mapbox.com restructured its llms.txt. Every product now has its own llms.txt (page index) and llms-full.txt (full page content) at its own URL. The root docs.mapbox.com/llms.txt is now a pure link index pointing to these sublevel files — it no longer contains any documentation content itself.

The previous resources all fetched the root llms.txt and filtered its sections by category keyword. With the new structure, those filtered sections contain only link-lists (links to other llms.txt files), not actual docs. The resources were effectively returning empty or useless content.

What changed

Resource	Before	After
resource://mapbox-api-reference	Root llms.txt, filtered for "api" sections	docs.mapbox.com/api/llms.txt — structured index of all REST APIs by service
resource://mapbox-guides	Root llms.txt, filtered for "guide/studio/design" sections	docs.mapbox.com/help/llms.txt — 39KB Help Center with actual guide content
resource://mapbox-sdk-docs	Root llms.txt, filtered for "sdk/library" sections	docs.mapbox.com/mapbox-gl-js/llms.txt — 34KB GL JS index (guides, API ref, examples)
resource://mapbox-reference	Root llms.txt, filtered for "reference" sections	Root llms.txt unfiltered — complete product catalog for discovery
resource://mapbox-examples	Root llms.txt, filtered for playground/demo sections	Same (root filtered for examples/playground/demo — still valid link lists)

Additional improvements

• fetchCachedText(url, httpRequest) — new shared helper in docFetcher.ts that fetches and caches a URL. All five resource classes now use this instead of duplicating the fetch+cache pattern (~30 lines each).

• toMarkdownUrl fix — previously returned a .txt.md URL for llms.txt URLs, causing get_document_tool to waste a request on a 404 before falling back. Now returns null for URLs already ending in .txt, .md, or .json, so they're fetched directly.

How to use llms-full.txt

Full page content for any product is available via get_document_tool:

• https://docs.mapbox.com/style-spec/llms-full.txt — 466KB, full Style Spec
• https://docs.mapbox.com/ios/navigation/llms-full.txt — 696KB, full Nav SDK iOS
• https://docs.mapbox.com/mapbox-gl-js/llms-full.txt — 1.6MB, full GL JS docs

The root resource://mapbox-reference now surfaces the complete product catalog including all llms.txt URLs, so models can discover and request any product's full docs.

Test plan

• npm test — 62 tests pass
• CI green
• Verify resource://mapbox-api-reference returns actual API endpoint list (not a link-index)
• Verify resource://mapbox-guides returns Help Center content (not empty)

"Show me the Mapbox API Reference"

"What Mapbox APls are available for navigation?"

🤖 Generated with Claude Code

[zmofei]

LGTM — clean adaptation to the upstream llms.txt restructure.

Two minor observations (neither blocking):

1. fetchDocContent Accept header removal (src/utils/docFetcher.ts:106) — the Accept: text/markdown header was also removed from the general get_document_tool fallback path, not just the new fetchCachedText. Low risk since the .md variant is tried first, but it's a slightly broader change than the PR scope suggests.

2. toMarkdownUrl fix is a nice catch — preventing the llms.txt.md 404 → fallback double-fetch is a real performance win for all .txt/.md/.json URLs going through get_document_tool.

New GetDocsIndexTool is well-designed with proper annotations and typed product enum. Good test coverage across happy path, caching, errors, and invalid input.