docs: comet docs design overhaul- phase 1

[pranamya123]

Key changes:

This is the first phase of a design overhaul for the Comet docs site. The goal is to bring it closer to the conventions of modern dev-doc sites (Stripe, Linear, Vercel, Anthropic) without changing any prose content — every change here is layered on top of pydata_sphinx_theme, so the existing search index, theme switcher, mobile drawer, and toctree generation all keep working unchanged.

🔗 Live preview: https://rainbow-cranachan-f70262.netlify.app/

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

The homepage is now a dedicated landing page rather than just the User Guide index. The content is borrowed from the existing Comet Overview page, happy to take suggestions on what to keep, cut, or reword.

The left sidebar got the biggest rework. It's now section-scoped: when you're on a User Guide page, the sidebar shows only User Guide content, not the four top-nav items repeated as headers. Each group toggles independently (with proper aria-expanded), and you can have multiple open at once. The sidebar also has an in-place search filter: typing in the search box hides non-matching items and highlights the matched characters live, instead of submitting the form and redirecting you to /search.html. Full-document search via the standard Sphinx index still works the same way.

A few smaller things came together with that:

A "Home" tab in the top navigation
A right-side page TOC that auto-shows on pages with 3+ H2 headings and hides on short reference pages, so it doesn't leave an empty rail
Breadcrumbs trimmed from 5 segments to 3, dropped the home icon (the logo handles it) and the redundant Comet User Guide parent, plus shortened the verbose Comet 0.16.0-SNAPSHOT User Guide label to just 0.16.0-SNAPSHOT
A quiet, structured footer used on every page (was previously the pydata default)
Project orange (#e8650a) is now the only accent color across the site, buttons, focus rings, active states, links, all consistent
There was one real build bug worth calling out: $COMET_VERSION was being substituted only inside build.sh, so anyone running sphinx-build source build/html directly (local dev, IDE preview, etc.) would see the literal placeholder in page titles, breadcrumbs, and the toctree caption. I moved that substitution into conf.py via a source-read hook, so local Sphinx builds now match what production produces.

Accessibility: every new interactive element has proper ARIA, the focus rings are now a custom orange ring instead of the browser's default pink/purple, and the terminal cursor blink in the hero respects prefers-reduced-motion.

Preview-only caveats
A few things look incomplete on the Netlify preview but render correctly
in production. They depend on extra build steps in docs/build.sh that
the preview environment doesn't run:

• Older Versions group in the User Guide sidebar is empty
  → populated by generate-versions.py, which clones the released-branch
  docs from GitHub

• 11 auto-generated reference pages are empty:

  • Configuration Settings (configs.md)
  • All 10 pages under Compatibility → Expression Compatibility
    (Aggregate, Array, Cast, Date/Time, Map, Math, Misc, String, Struct, URL)
    → populated by ./mvnw -Pgenerate-docs, which reads Spark/Comet source
    annotations and fills in the <!--BEGIN:...--> markers in the markdown

This PR doesn't touch either of those build steps; the visual styling
applies normally once they're populated by Apache CI on merge.

Phase 2 (separate PR)
A few things I deliberately scoped out for review velocity:

Content review across all pages
Re-evaluating which top-nav items earn their place
Interactive / animated graphs on the homepage

BEFORE

AFTER

With the right side page TOC

How are these changes tested?

Verified with generate-versions.py

Ran the script locally to populate the older-version directories (the
same script docs/build.sh runs in CI). All three User Guide groups
show up correctly with their nested content:

(the user guide sidebar with all groups)

Auto-generated tables (configs, expression compatibility)

The Maven ./mvnw -Pgenerate-docs step fills in the <!--BEGIN:...--> markers on the configs and expression-compatibility pages. I attempted to run it locally but hit a separate build issue unrelated to this PR (missing proto-generated classes on the spark module's classpath, which reproduces on a clean checkout of main). Apache CI doesn't hit this, so those tables will populate normally on merge. The visual styling in
this PR applies on top of whatever content the build produces.

Deployed the UI on netlify and manually tested it out- https://rainbow-cranachan-f70262.netlify.app/
In the next PRs, the plan is to add more unit tests.

[coderfender]

Thank you @pranamya123 . Could you add further screenshots with before and after changes?

[andygrove]

This is very nice!

[coderfender]

I think the mobile layout has an issue @pranamya123

[pranamya123]

I think the mobile layout has an issue @pranamya123

Thank you for this comment, I made the design responsive to all devices and redeployed, could you please check now in the same netlify link? @coderfender

[coderfender]

Thank you. It looks good now

[coderfender]

Seems like the Apache DataFusion Comet font is too less (even less than the description) here ? Can we probably highlight Apache DataFusion Comet better ?

nit : Would also be great to show the acceleration in the terminal
nit : Your existing Spark queries — now executed natively via DataFusion - probably could add comet here ?
Also on the user guide page:

(html_title)

Also, not sure if this is intended but I don't see older version docs :

current :

PR :

Missing some feature sections :

current :

PR :

[pranamya123]

Thanks @coderfender — addressed in 328f8b6:

• "Apache DataFusion Comet" is now the hero H1 (was the small eyebrow)
• Terminal demo now mentions Comet by name in the inline comment
• html_title leak fixed; reverted to standard markdown H1
• "Use Commodity Hardware" section restored with the verbatim source paragraph
• Sidebar stays consistent across all User Guide pages — clicking a Dev Snapshot child no longer reloads the sidebar with a different tree
  Re older version docs: the Netlify preview only shows Development Snapshot
  because Netlify doesn't run docs/build.sh (which calls generate-versions.py
  to clone the released-version docs from GitHub). In production, the Apache
  build runs that step, so 0.16.x (current) and Older Versions populate
  correctly when this PR merges. No source change needed for that.

[pranamya123]

Hi @andygrove, this PR is now ready for review. You can test out the new UI in https://rainbow-cranachan-f70262.netlify.app/ . Please let me know your feedback. TIA!

[coderfender]

Seeing some inconsistencies with the terminal display :

Seeing different sidebars :

PR :

Main :

Also missing important config:

PR :

Main :

Seeing issues with the expression data as well :

Could we please confirm that the text / content is not missing ?

[coderfender]

Left comments on missing content

[pranamya123]

Seeing some inconsistencies with the terminal display :

Seeing different sidebars :

PR :

Main : Also missing important config:

PR :

Main : Seeing issues with the expression data as well :

Could we please confirm that the text / content is not missing ?

Thanks @coderfender — confirming no content is missing. What you're
seeing is the Netlify preview running only sphinx-build, while
production runs the full docs/build.sh which has two extra steps that
populate this content.

1. Terminal display inconsistencies: Could you point me at the page that has this specific issue, so that I can reproduce the issue?

2. sidebar difference vs main: this is intentional.

I took a UX judgement call here: the user would expect to stay anchored on the Development Snapshot sidebar item with its child items opened under it, instead of having the whole sidebar replaced and feeling like they navigated to a different site.

That said, if the team prefers main's switching behavior over the consistent-tree approach, happy to revert.

3. Config tables on configs.md and the 10 expression compatibility pages: these use <!--BEGIN:CONFIG_TABLE[...]--> and <!--BEGIN:EXPR_COMPAT[...]--> markers that get filled in by ./mvnw -Pgenerate-docs, which reads the Spark/Comet source annotations at build time. Sphinx alone doesn't run this, so the markers stay empty on the preview.

I tried running the Maven step locally to verify it too, but hit a separate build issue (proto-generated classes not on the spark module's classpath — reproduces on a clean main checkout, not introduced by this PR). The Apache CI environment doesn't hit that, so these tables will populate normally once this PR merges.

This PR doesn't touch either build step, the visual styling layers on top of whatever content the pipeline produces.

4. same as 3, Aggregate Expressions page gets generated during build.

One option is to merge these docs changes behind a feature flag, and test them out with the real build values, and only then turn it on when we feel confident with it.

[andygrove]

Thanks @pranamya123. I plan on starting to test this out with local builds soon.

[andygrove]

The text in the Spark shell exaple is double spaced, which wastes vertical space:

[andygrove]

In http://localhost:8765/user-guide/0.16/installation.html some of the text is hidden due to colors used:

[pranamya123]

@coderfender Maven build on my local is a success now, I verified the content on Comet Configuration Settings and Aggregate Expressions pages, please find the attached screenshots.

Comet Configuration Settings:

Aggregate Expressions:

[pranamya123]

http://localhost:8765/user-guide/0.16/installation.html

Hi @andygrove, thank you for your feedback. I addressed both the issues in the latest commit (2d3e760).

Reduced the vertical spacing in the terminal block:

And fixed the font color so the text is legible in both themes:

light mode:

dark mode: