Docs: build directly using sphinx #41156

tobiasdiez · 2025-11-09T12:44:12Z

Instead of our custom docbuilder, build the documentation simply by running:

python -m sphinx ./builddir/src/doc/en/ ./builddir/src/doc/html/en/

(has to be run after an initial meson compile -C builddir doc-html which generates some of the rst files)

The old way to build the docs is not (yet) removed and can be activated via:

meson configure builddir -Dlegacy-docs=true
meson compile -C builddir doc-html

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

kiwifb · 2025-12-02T01:13:35Z

I never managed to actually start work on getting rid of our sage_autodoc.py. If this ticket works, I am going to adopt it in sage-on-gentoo faster than you can say sphinx-9. I did the last few migrations of sage_autodoc.py at sphinx-8, sphinx-8.2 and may be some earlier and it looks like it would need another migration at sphinx-9.
Let's skip the whole business if we can.

kiwifb · 2025-12-02T02:00:11Z

Alas, it dies in meson compile -C $mybuilddir doc-html - very similar message to what I got with legacy earlier. I may have accused sphinx 9.0 in vain. It looks like it is more of an issue of finding sage_docbuild. It may have been installed before or something.
I am working with a system installed sage and clean sources.

Versions
========

* Platform:         linux; (Linux-6.12.58-gentoo-dist-x86_64-Intel-R-_Core-TM-_i7-9700_CPU_@_3.00GHz-with-glibc2.41)
* Python version:   3.13.5 (CPython)
* Sphinx version:   9.0.0
* Docutils version: 0.21.2
* Jinja2 version:   3.1.6
* Pygments version: 2.19.2

Last Messages
=============

None.

Loaded Extensions
=================

None.

Traceback
=========

    Traceback (most recent call last):
      File "/usr/lib/python3.13/site-packages/sphinx/registry.py", line 543, in load_extension
        mod = import_module(extname)
      File "/usr/lib/python3.13/importlib/__init__.py", line 88, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'sage_docbuild'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/lib/python3.13/site-packages/sphinx/cmd/build.py", line 414, in build_main
        app = Sphinx(
            srcdir=args.sourcedir,
        ...<14 lines>...
            exception_on_warning=args.exception_on_warning,
        )
      File "/usr/lib/python3.13/site-packages/sphinx/application.py", line 299, in __init__
        self.setup_extension(extension)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/application.py", line 505, in setup_extension
        self.registry.load_extension(self, extname)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/registry.py", line 546, in load_extension
        raise ExtensionError(
            __('Could not import extension %s') % extname, err
        ) from err
    sphinx.errors.ExtensionError: Could not import extension sage_docbuild.ext.inventory_builder (exception: No module named 'sage_docbuild')

kiwifb · 2025-12-02T02:21:41Z

My inspection so far tells me that I need to either install sage_docbuild when I install sage (and that is probably needed for a legacy build at this stage) or I need to copy sage_docbuild in builddir/src/ or whatever its equivalent I choose. And I have no idea how to make the later happens.

kiwifb · 2025-12-03T00:36:17Z

src/doc/en/conf.py

+# ---------------------
+
+# To find sage_docbuild (from the builddir)
+sys.path.insert(0, '../../../../src/')


That's the line that's currently getting me. It is fragile in that it requires you to follow an exact workflow about the building directory location. I am wondering how to make it more foolproof.

I am making progress after replacing the line with

sys.path.insert(0, SAGE_DOC_SRC + '/..')

If we were importing SAGE_SRC, it could be used directly.

How do you invoke the docbuilding? And how do you set SAGE_SRC/DOC_SRC?

Another solution would be to copy the stuff in the sage_docbuild directory to the build directory. Would that work for you?

OK, SAGE_DOC_SRC is set in my build script https://github.com/cschwan/sage-on-gentoo/blob/4cfde2c8ca8af625ee6d84bd212c3e0fa5665a98/sci-mathematics/sagemath-doc/sagemath-doc-9999.ebuild#L101 (the file is not completely up to date compared to what I am working with). But I thought that in vanilla, the default value as sourced from env would work. I am using SAGE_DOC_SRC because it is imported earlier in conf.py so it is available.

I traditionally have setup a build_doc folder fully out of source. So one level up compared to the builddir you are using by default. So at the moment my script looks like

meson compile -C "${WORKDIR}/build_doc" doc-html ${EPYTHON} -m sphinx "${WORKDIR}/build_doc/src/doc/en" "${WORKDIR}/build_doc/src/doc/html/en/"

Where EPYTHON is the python used for building and WORKDIR is the folder which contains the sage root folder.

kiwifb · 2025-12-03T02:19:54Z

OK, so after my little fix I can go much further.
It is slow, single threaded, legacy I believe does parallel processing which makes the html much much faster.
There is very little in matter of progress output, which is slightly annoying.

It did not complete, at the end I got a lot warning like

/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:4: WARNING: undefined label: 'sage.combinat.quickref' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:5: WARNING: undefined label: 'sage.combinat.tutorial' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:10: WARNING: undefined label: 'sage.combinat.algebraic_combinatorics' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:12: WARNING: undefined label: 'sage.combinat.chas.all' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:13: WARNING: undefined label: 'sage.combinat.cluster_algebra_quiver.all' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:14: WARNING: undefined label: 'sage.combinat.crystals.all' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:15: WARNING: undefined label: 'sage.combinat.root_system.all' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:16: WARNING: undefined label: 'sage.combinat.sf.all' [ref.ref]

which do happen with the legacy build as well in the first pass. Usually it resolves itself in a second pass. But thanks to the lack of output, I do not if it is definite or just not the final pass where those things get resolved.

The build broke mysteriously

/usr/lib/python3.13/site-packages/sage/combinat/__init__.py:docstring of sage.combinat:35: WARNING: undefined label: 'sage.combinat.misc' [ref.ref]
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:192: WARNING: Could not obtain image size. :scale: option is ignored.
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:192: WARNING: Cannot scale image!
  Could not get size from "../media/complete-binary-trees-4.png":
  [Errno 2] No such file or directory: '../media/complete-binary-trees-4.png' [docutils]
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:1583: WARNING: Could not obtain image size. :scale: option is ignored.
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:1583: WARNING: Cannot scale image!
  Could not get size from "../media/prefix-tree-partitions-5.png":
  [Errno 2] No such file or directory: '../media/prefix-tree-partitions-5.png' [docutils]
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:1643: WARNING: Could not obtain image size. :scale: option is ignored.
/usr/lib/python3.13/site-packages/sage/combinat/tutorial.py:docstring of sage.combinat.tutorial:1643: WARNING: Cannot scale image!
  Could not get size from "../media/polytope.png":
  [Errno 2] No such file or directory: '../media/polytope.png' [docutils]
generating indices... genindex py-modindex done

Merging js index files...
... done (31512 js index entries)
Writing js search indexes...writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 3819 warnings.

The files exists but obviously they are not looked for from the right path

tarazed /home/portage/sci-mathematics/sagemath-doc-9999/work # find . -name complete-binary-trees-4.png
./build_doc/src/doc/en/reference/combinat/media/complete-binary-trees-4.png
./sagemath-doc-9999/src/doc/en/reference/combinat/media/complete-binary-trees-4.png
tarazed /home/portage/sci-mathematics/sagemath-doc-9999/work # find . -name prefix-tree-partitions-5.png
./build_doc/src/doc/en/reference/combinat/media/prefix-tree-partitions-5.png
./sagemath-doc-9999/src/doc/en/reference/combinat/media/prefix-tree-partitions-5.png
tarazed /home/portage/sci-mathematics/sagemath-doc-9999/work # find . -name polytope.png
./build_doc/src/doc/en/reference/combinat/media/polytope.png
./sagemath-doc-9999/src/doc/en/reference/combinat/media/polytope.png

kiwifb · 2025-12-03T02:30:37Z

And building with sphinx 9.0 fails equally in this build and the legacy build. Previous results were with sphinx-8.2.3.

tobiasdiez · 2025-12-03T09:32:21Z

OK, so after my little fix I can go much further. It is slow, single threaded, legacy I believe does parallel processing which makes the html much much faster.

Do you mean building with this PR using sphinx is slower, or with the legacy doc build? In the first case, sphinx only uses one thread by default unless you specify --jobs auto (or a number higher than 1).

It did not complete, at the end I got a lot warning like

Thanks, I'll look into it.

kiwifb · 2025-12-03T09:44:26Z

OK, so after my little fix I can go much further. It is slow, single threaded, legacy I believe does parallel processing which makes the html much much faster.

Do you mean building with this PR using sphinx is slower, or with the legacy doc build? In the first case, sphinx only uses one thread by default unless you specify --jobs auto (or a number higher than 1).

Building with this PR is very slow. That option should speed things up considerably.

Docs: build directly using sphinx

cb5907a

This was referenced Nov 11, 2025

Migrate doc/bootstrap to Python #41162

Open

Doc builds failing due to space issue #40929

Open

vincentmacri mentioned this pull request Dec 1, 2025

Try to optimize integral basis computation for global function fields over prime finite fields #41242

Open

5 tasks

kiwifb reviewed Dec 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Docs: build directly using sphinx #41156

Docs: build directly using sphinx #41156

Uh oh!

tobiasdiez commented Nov 9, 2025

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb Dec 3, 2025

Uh oh!

kiwifb Dec 3, 2025

Uh oh!

tobiasdiez Dec 3, 2025

Uh oh!

kiwifb Dec 3, 2025 •

edited

Loading

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

tobiasdiez commented Dec 3, 2025

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Docs: build directly using sphinx #41156

Are you sure you want to change the base?

Docs: build directly using sphinx #41156

Uh oh!

Conversation

tobiasdiez commented Nov 9, 2025

📝 Checklist

⌛ Dependencies

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb commented Dec 2, 2025

Uh oh!

kiwifb Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

kiwifb Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

tobiasdiez Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

kiwifb Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

tobiasdiez commented Dec 3, 2025

Uh oh!

kiwifb commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kiwifb Dec 3, 2025 •

edited

Loading