docs: page descriptions, titles, structured data and crawler files for search and answer engines

[imran-siddique]

A crawl of every page in ca2a.agentrust-io.com's sitemap on 2026-09-14 found five things that make the site harder for search and answer engines to read:

• 36 of 37 pages carried the same meta description, site_description, because Material falls back to it when a page has none. Results showed one snippet for the whole site.
• The home title read cA2A: verify who delegated what, hop by hop - cA2A.
• JSON-LD had no Organization @id and repeated an inline Organization as author on every page, 74 objects in all, and each page's TechArticle pointed at the home page rather than itself.
• /robots.txt served only Cloudflare's content-signals comment block, with no directives and no Sitemap line, because the docs build never copied a robots.txt.
• llms.txt had two sections, Getting started and Specification, with no guides and no LIMITATIONS.

Changes

• hooks/seo.py, registered under hooks: in mkdocs.yml, sets each page's description from its first paragraph of prose, capped at 155 characters. Front matter still wins. The same hook runs on every AgenTrust docs site. hooks/ is in exclude_docs.
• overrides/main.html overrides htmltitle: the home page uses its front-matter title alone, and other pages drop the - cA2A suffix when the title already names cA2A. og:title and twitter:title now equal <title>, and og:site_name is AgenTrust.
• The same template emits one JSON-LD graph per page: the Organization under https://agentrust-io.com/#organization, the id the hub and the other docs sites use; a WebSite; a WebPage on the home page or a TechArticle on inner pages, with the page's own title, description and canonical URL; a BreadcrumbList on inner pages; and on the home page a SoftwareSourceCode node with codeRepository, programmingLanguage and the MIT license.
• robots.txt at the repository root names the same ten crawlers as agentrust-io.com, disallows /cdn-cgi/, and points at https://ca2a.agentrust-io.com/sitemap.xml. docs.yml copies it into the build and deploys when it or hooks/** changes.
• The llms.txt sections follow the tabs: Get started, Specification (nine pages), Guides (five tutorials and hardware validation), Project (LIMITATIONS, ROADMAP, GOVERNANCE, SECURITY), and the description ends by linking agentrust-io.com.

Checked, on a build that replicates docs.yml (allowlist copied into .docs_build, CR stripped, docs_dir rewritten)

• mkdocs build and mkdocs build --strict exit 0.
• All 37 pages: no two share a description, none is over 160 characters or contains an em or en dash, and no title repeats the site name. Home title: cA2A: verify who delegated what, hop by hop.
• Every page has exactly one Organization node with the hub @id, no inline Organization objects, and one page node whose url equals the canonical, whose description equals the meta description, and whose headline equals the title. All 36 inner pages carry a BreadcrumbList.
• robots.txt is at the site root with the Sitemap line, hooks/seo.py is not published, and llms.txt has the four sections.
• hooks/seo.py imports cleanly under python -W error.

Generated with Claude Code

https://claude.ai/code/session_013aK3gVWzNdcM3hZ2o2awK2