Improve docs agent readiness

[joshuablum]

Our docs currently don't score that high on Cloudflare's https://isitagentready.com and there is a lot of room for improvement. We already serve pages as markdown when appending .md to a URL (basic right now and some undiscoverable) and also have a llms.txt file in place (misses a few things). Those can be improved as well as a lot of other changes and improvements (e.g. much more in-depth robots.txt file). This PR implements several recommendations and best practices from https://isitagentready.com, https://blog.cloudflare.com/agent-readiness/, and https://blog.cloudflare.com/aeo/.

Discoverability

• robots.txt is now a route instead of a two line static file. It has a Sitemap: directive (there wasn't one), Content Signals declaring search=yes, ai-input=yes, ai-train=yes, and explicit groups for 12 named AI crawlers.
• Removed a dead lookup from sitemap.xml. It opened by fetching and entry ID (6aa5449b...) which doesn't exist anymore.

Content accessibility

• llms.txt went from 128 to 613 entries. It only walked one level deep, so nested pages like /getting-started/installing/laravel were dropped and the ~460 reference entries (tags, modifiers, fieldtypes, variables) weren't in there at all. An agent reading it couldn't discover any of it.
• The home page now has a Markdown version at /index.md. None before.
• Every page links to its Markdown twin via <link rel="alternate" type="text/markdown">, plus rel="canonical" and a robots tag with max-snippet:-1 so answer engines can quote a full passage. /search-results is noindex.

Meta descriptions

• 225 of 630 entries had no intro or description, like the modifiers, variables, or
  resource API pages. They were emitting empty meta descriptions, empty Open Graph tags and empty llms.txt entries.
• There's now a fallback chain (intro -> description -> first paragraph of the body) that strips code fences, ::tabs, :::tip blocks and inline markdown. It's a computed
  value so one implementation covers all templates and also the llms.txt.

Two other bugs fixed

• Link anchors were being mangled in the Markdown output. E.g. (#update-scripts) became
  (#update-scripts.md). The .md now goes before the fragment.
• An unpublished draft was listed in llms.txt. digitalocean.md is
  published: false but was included in the file. Now correctly filter the entries.