docs: per-package documentation tree with Diátaxis subpages and sidebar nesting

[tony]

Summary

• Replace the workspace's flat docs/packages/<name>.md pages with per-package directories carrying Diátaxis-shaped subpages (Tutorial / How to / API Reference / Explanation / Examples), nested under each package in the sidebar.
• Add a {package-landing} directive that renders the per-package landing — meta badges, synopsis, conditional grid cards, and a hidden toctree — so adding a new package only touches pyproject.toml (or package.json).
• Add a {cluster-toctree} directive replacing six hand-edited toctree blocks in docs/index.md with five directive calls; the workspace inventory becomes the single source of truth for sidebar nesting.
• Add dual-source workspace discovery (PackageDocsRecord) so JS-only token packages (@gp-sphinx/furo-tokens) graduate alongside Python packages without separate plumbing.
• Add four optional showcase directives — {live-signature}, {package-kitchen-sink}, {surface-changelog}, {package-dependents} — opt-in per package via [tool.gp-sphinx.docs].showcase.
• Preserve every py-domain cross-reference target via a conditional pkg_docname registration that branches on env.found_docs membership; an objects.inv snapshot test mathematically proves no :py:func:/:py:class: reference was lost.
• Land showcase opt-ins for thirteen packages so the new directives are validated end-to-end against real data.

Changes by area

Sidebar IA

Before	After
Six hand-edited {toctree} blocks listing every package by name in docs/index.md	Five {cluster-toctree} directive calls; sidebar leaves derived from each package's classifier and [tool.gp-sphinx.docs] overrides
Single flat page per package, all subpages on one scroll	Per-package directory with subpages nested under the package as toctree-l2 entries
Inconsistent caption set (Domain Packages / Internal / Utils)	Consistent caption set: Autodoc / UX / Tokens / Theme & coordinator / Build & SEO

Per-package layout

docs/packages/<name>/
├── index.md          # 4-line stub: anchor + H1 + {package-landing} call
├── tutorial.md       # if the package has a tutorial-shaped surface
├── how-to.md
├── reference.md      # autodoc directives
├── explanation.md
└── examples.md       # live demos

Optional opt-in subpages (per [tool.gp-sphinx.docs]):

• errors.md, cli.md — extra keys
• signatures.md, kitchen-sink.md, surface-diff.md, dependents.md — showcase keys

docs/_ext/package_reference.py — new directives + roles

Directive / role	Purpose
{package-landing}	Renders the per-package landing markup (meta badges, synopsis, conditional grid cards, hidden toctree)
{cluster-toctree}	Emits a hidden toctree of every Shipped package in a sidebar cluster, skipping Emerging packages so the build never references a missing docname
{subpage-exists}	Conditional cross-reference role — emits a {doc} xref when the target resolves, plain text otherwise (so Tutorial → How-to "Where to next" links don't break when the sibling subpage is absent)
{live-signature}	Imports the package and renders each public function/class signature from the running interpreter; flags drift between source and docstring
{package-kitchen-sink}	One example invocation per registered directive plus a list of registered roles
{surface-changelog}	Diffs current registered directives/roles/config against a JSON snapshot at docs/_static/surface-snapshots/<name>.json
{package-dependents}	Reverse-intersphinx — lists workspace packages that declare the named package in [project].dependencies
{workspace-package-grid} :groups: by-cluster	Existing directive extended; new mode renders one grid per cluster preserving the curated framing prose in docs/packages/index.md

Workspace discovery (dual-source)

workspace_package_records() walks every directory under packages/ and probes for either manifest:

State	Predicate	Treatment
shipped-py	pyproject.toml with [project] table	Full landing + cluster toctree entry + grid card
shipped-js	package.json with name field, no pyproject.toml	Full landing (synopsis from package.json description); gp-sphinx-package-meta emits an npm badge instead of PyPI
emerging	Neither manifest exists	Skipped from cluster-toctrees so the build never references a missing docname

Cross-reference preservation (Risk 1 mitigation)

The existing _register_extension_objects() was hardcoded to register py-domain objects against the flat docname packages/<name>. After migration that docname no longer exists. The fix branches on env.found_docs membership at env-check-consistency time: prefer packages/<name>/reference once that docname is in the build, fall back to the flat packages/<name> while a package is still pre-migration. The conditional makes the registration correct both before and after each per-package commit.

CI gates added under tests/docs/

Test	What it asserts
test_no_filler.py	Denylist guard: no TBD, XXX, FIXME, Coming soon, intentionally blank, Lorem ipsum, (write me), placeholder in any shipped subpage
test_no_legacy_pages.py	No flat <name>.md co-exists with a per-package <name>/index.md
test_sidebar_density.py	Sidebar leaves stay under 19 * 6 + 30 post-migration ceiling
test_objects_inv_compat.py	Module-scoped Sphinx fixture builds the live docs/ tree to a tmp dir; asserts the live objects.inv is a superset of the committed snapshot for py: / rst: / argparse: domains
Per-directive coverage	test_subpage_exists_role.py, test_package_landing.py, test_cluster_toctree.py, test_workspace_package_grid_groups.py, test_live_signature.py, test_package_kitchen_sink.py, test_surface_changelog.py, test_package_dependents.py

Showcase opt-ins (per package)

Package	Subpages added
sphinx-ux-badges, sphinx-autodoc-typehints-gp, sphinx-ux-autodoc-layout, sphinx-fonts, sphinx-gp-theme, sphinx-gp-sitemap	Dependents
sphinx-autodoc-fastmcp, sphinx-autodoc-argparse, sphinx-autodoc-pytest-fixtures, sphinx-autodoc-sphinx, sphinx-autodoc-docutils	Kitchen sink
gp-furo-theme, sphinx-gp-opengraph	Signatures (live) + Dependents

surface-changelog is registered and tested but not yet opted into; baseline JSON snapshots can be captured at the next release tag.

Design decisions

• Conditional pkg_docname registration over flag-day rewrite. Iterative migration would have left every package's py-domain xrefs dangling for the duration. Branching on env.found_docs membership makes the new behavior safe to land before any per-package commit and self-corrects as each package migrates.
• Stub provides title; directive provides body. Sphinx's title-extraction runs at parse time, before custom directives render. Directives that emit # H1 via parse_text_to_nodes cannot retroactively set the page title — the page renders as <no title> and the parent toctree promotes the page's children to its own level. Every showcase stub is a 4-line file (anchor + blank + H1 + blank + directive call) so Sphinx finds a title.
• Cluster-toctree skips Emerging. A cluster-toctree that emits docnames Sphinx hasn't discovered would fail sphinx-build -W. Filtering at emit time keeps the build green even when the workspace has half-baked package directories.
• Body-only Group H directives. Same constraint as the landing directive — stubs supply anchor + H1, directives supply body. Makes opt-ins follow a single canonical pattern and prevents the no-title bug from recurring.

Verification

The branch passes the workspace verification gate at every commit. Reviewers can re-run from a fresh checkout:

$ rm -rf docs/_build

$ uv run ruff check . --fix --show-fixes

$ uv run ruff format .

$ uv run mypy

$ uv run py.test --reruns 0 -vvv

$ just build-docs

CI-equivalent (warnings-as-errors):

$ uv run sphinx-build -W -b dirhtml docs docs/_build/html

Inspect the rendered sidebar from the workspace landing — every package shows toctree-l1 has-children with subpages indented as toctree-l2:

$ grep -B 1 -A 8 'caption-text">Autodoc' docs/_build/html/index.html

Verify zero flat <name>.md per-package pages remain:

$ ls docs/packages/*.md | grep -v 'index.md$'

Test plan

• tests/docs/test_no_filler.py — every shipped subpage is free of denylist filler strings
• tests/docs/test_no_legacy_pages.py — no flat <name>.md shadows a per-package <name>/index.md
• tests/docs/test_sidebar_density.py — total toctree leaves stay under the post-migration ceiling
• tests/docs/test_objects_inv_compat.py::test_objects_inv_is_superset_of_baseline — live build produces a superset of the committed objects.inv snapshot for py: / rst: / argparse: domains (Risk 1 mitigation)
• tests/docs/test_package_landing.py — landing markdown helper renders meta + synopsis + conditional grid; no anchor or H1 emitted by the directive (stub-supplies-title contract)
• tests/docs/test_cluster_toctree.py — Emerging packages excluded; alphabetical leaf order; every Shipped package belongs to exactly one cluster
• tests/docs/test_subpage_exists_role.py — sibling-relative and absolute targets resolve; absent targets degrade to plain text
• tests/docs/test_workspace_package_grid_groups.py — :groups: by-cluster mode preserves curated prose; legacy single-grid output unchanged
• tests/docs/test_live_signature.py / test_package_kitchen_sink.py / test_surface_changelog.py / test_package_dependents.py — per-directive contracts (body-only output, edge cases for unknown / surface-less / unimportable packages)
• tests/test_package_reference.py::test_register_extension_objects_uses_reference_docname_after_migration — conditional pkg_docname fix branches correctly between flat and per-package docnames
• Manual sidebar audit across all five clusters confirms every package is toctree-l1 has-children (or a leaf for shipped-js packages without subpages) with subpages as toctree-l2
• uv run sphinx-build -W -b dirhtml docs docs/_build/html completes without warnings

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.86207% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.57%. Comparing base (81eb49b) to head (ad894bb).

Files with missing lines	Patch %	Lines
docs/_ext/package_reference.py	93.08%	28 Missing ⚠️
tests/docs/test_cluster_toctree.py	91.30%	6 Missing ⚠️
tests/docs/test_objects_inv_compat.py	93.22%	4 Missing ⚠️
tests/test_docs_package_pages.py	88.46%	3 Missing ⚠️
packages/gp-sphinx/src/gp_sphinx/myst_lexer.py	90.90%	2 Missing ⚠️
docs/_ext/inline_highlight.py	98.33%	1 Missing ⚠️
tests/docs/test_no_filler.py	96.29%	1 Missing ⚠️
tests/docs/test_package_dependents.py	96.15%	1 Missing ⚠️
tests/docs/test_sidebar_density.py	98.43%	1 Missing ⚠️
tests/test_package_reference.py	98.48%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
+ Coverage   89.09%   91.57%   +2.48%     
==========================================
  Files         191      205      +14     
  Lines       15675    16814    +1139     
==========================================
+ Hits        13966    15398    +1432     
+ Misses       1709     1416     -293     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tony]

Code review

Found 4 issues.

1. from html import escape violates the stdlib namespace-import rule (CLAUDE.md says "Use namespace imports for standard library modules: import enum instead of from enum import Enum. Exception: dataclasses module may use from dataclasses import dataclass, field. This rule applies to Python standard library only").

gp-sphinx/docs/_ext/inline_highlight.py

Lines 44 to 48 in aa811c6

	import re
	import typing as t
	from html import escape
	from docutils import nodes

2. subpage_exists_role lacks a doctest (CLAUDE.md says "All functions and methods MUST have working doctests"). Sibling role functions in sphinx-autodoc-argparse (e.g. cli_option_role) demonstrate the inliner=None pattern is practical here.

gp-sphinx/docs/_ext/package_reference.py

Lines 1209 to 1227 in aa811c6

	def subpage_exists_role(
	name: str,
	rawtext: str,
	text: str,
	lineno: int,
	inliner: t.Any,
	options: dict[str, t.Any] | None = None,
	content: list[str] | None = None,
	) -> tuple[list[t.Any], list[t.Any]]:
	"""Implement the ``{subpage-exists}`<target>`` MyST role.
	Renders a Sphinx ``:doc:`` cross-reference when ``<target>`` resolves
	to an existing docname (sibling-relative or absolute); otherwise
	emits plain text so the build does not fail. Used in tutorial /
	how-to "Where to next" sections so prose never refers to absent
	subpages.
	"""
	from docutils import nodes as docutils_nodes

3. The npm URL builder strips @ and percent-encodes / for scoped packages, producing https://www.npmjs.com/package/gp-sphinx%2ffuro-tokens instead of https://www.npmjs.com/package/@gp-sphinx/furo-tokens. The npm web UI 404s on the encoded slug; the test only asserts record.npm_url is not None so the bad value passes.

gp-sphinx/docs/_ext/package_reference.py

Lines 394 to 412 in aa811c6

	if package_json_path.is_file():
	manifest = json.loads(package_json_path.read_text(encoding="utf-8"))
	name = str(manifest.get("name", pkg_dir.name))
	npm_slug = name.lstrip("@").replace("/", "%2f") if name else pkg_dir.name
	return PackageDocsRecord(
	name=name,
	state="shipped-js",
	cluster=_cluster_for(name),
	package_dir=pkg_dir,
	manifest_path=package_json_path,
	src_dir=src_dir if src_dir.is_dir() else None,
	module_name="",
	description=str(manifest.get("description", "")),
	version=str(manifest.get("version", "")),
	repository_url=_repository_url_from_package_json(manifest),
	pypi_url=None,
	npm_url=f"https://www.npmjs.com/package/{npm_slug}",
	maturity="Unknown",
	)

4. test_objects_inv_is_superset_of_baseline uses the _live_objects_inv fixture which instantiates sphinx.application.Sphinx and calls app.build(), but the test function lacks @pytest.mark.integration (CLAUDE.md says "Always mark with @pytest.mark.integration" and "Any test that constructs a Sphinx app ... counts" as integration).

gp-sphinx/tests/docs/test_objects_inv_compat.py

Lines 88 to 113 in aa811c6

	status = io.StringIO()
	warning = io.StringIO()
	app = Sphinx(
	srcdir=str(src_dir),
	confdir=str(src_dir),
	outdir=str(html_dir),
	doctreedir=str(doctree_dir),
	buildername="dirhtml",
	status=status,
	warning=warning,
	freshenv=True,
	)
	app.build()
	inv_path = html_dir / "objects.inv"
	if not inv_path.is_file():
	pytest.fail(f"sphinx build produced no objects.inv at {inv_path}")
	return inv_path
	def test_objects_inv_is_superset_of_baseline(
	_live_objects_inv: pathlib.Path,
	) -> None:
	"""No baseline cross-reference target is missing from the live build."""
	live_keys = _flatten_inventory(_live_objects_inv)
	baseline = _baseline_keys()
	missing = baseline - live_keys

Generated with Claude Code