Migrate doc/bootstrap to Python

[tobiasdiez]

Migrate the shell script doc/bootstrap to Python. This improves performance and is preparation for building the docs simply by running "sphinx", see #41156.

The output should be more or less equivalent, expect for minor formatting changes and better escaping of the generated shell commands.

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation and checked the documentation preview.

⌛ Dependencies

[github-actions]

Documentation preview for this PR (built with commit dc56c2a; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[mantepse]

I have no idea how I could help in the reviewing process here. Any hints welcome! (I don't understand shell scripts)

The changes in package.py are purely cosmetic, with the exception of adding read_system_packages, right? (I admit that I almost dislike all of them - with the exception of replacing ' with ", which I don't care about. I find breaking the lines in this way makes the code less readable)

[antonio-rojas]

Doesn't build here

FAILED: [code=1] src/doc/htmlspkg 
/usr/bin/python /build/sagemath-doc-git/src/sage/src/build-docs.py --no-prune-empty-dirs --no-pdf-links reference/spkg html -o src/doc --source /build/sagemath-doc-git/src/build/src/doc
[spkg     ] inventory <https://ipywidgets.readthedocs.io/en/stable/> contains multiple definitions for std:label:examples/Widget Layout.ipynb#display
[spkg     ] building [html]: targets for 467 source files that are out of date
[spkg     ] updating environment: 0 added, 31 changed, 0 removed
[spkg     ] /build/sagemath-doc-git/src/build/src/doc/en/reference/spkg/sagemath_objects.rst:64: WARNING: undefined label: 'spkg_force' [ref.ref]
[spkg     ] The HTML pages are in src/doc/html/en/reference/spkg.
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc-git/src/sage/src/build-docs.py", line 11, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/__main__.py", line 548, in main
    build()
    ~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/builders.py", line 669, in _wrapper
    getattr(DocBuilder, build_type)(self, *args, **kwds)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/builders.py", line 142, in f
    runsphinx()
    ~~~~~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/sphinxbuild.py", line 325, in runsphinx
    sys.stderr.raise_errors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/sagemath-doc-git/src/sage/src/sage_docbuild/sphinxbuild.py", line 256, in raise_errors
    raise OSError(self._error)
OSError: /build/sagemath-doc-git/src/build/src/doc/en/reference/spkg/sagemath_objects.rst:64: WARNING: undefined label: 'spkg_force' [ref.ref]

[tobiasdiez]

I thought that was fixed already with f71e2e6. Strange... I don't have time atm, but will look at it end of next week.

[antonio-rojas]

Looks like the bootstrap.py→bootstrap-docs.py rename somehow confuses github, and it generates an invalid patch that doesn't apply correctly. I built directly from your branch and it works fine.

[tobiasdiez]

@antonio-rojas Happy to hear. So this is good to go from your side?

@mantepse

I have no idea how I could help in the reviewing process here. Any hints welcome! (I don't understand shell scripts)

The added script under tools is the major change. So I would say try to execute that on your system (directly with python) and review the code in there.

The changes in package.py are purely cosmetic, with the exception of adding read_system_packages, right?

Right

(I admit that I almost dislike all of them - with the exception of replacing ' with ", which I don't care about. I find breaking the lines in this way makes the code less readable)

That's how ruff thinks the code should be formatted. I stopped long time ago to argue with linters/code formatters... 🦊

[tobiasdiez]

Thanks!

[vbraun]

Build fails for me with

[sagemath_doc_html-none] [spkg-install] [spkg     ] The inventory file is in src/doc/inventory/en/reference/spkg.
[sagemath_doc_html-none] [spkg-install] Error building the documentation.
[sagemath_doc_html-none] [spkg-install] Traceback (most recent call last):
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/build-docs.py", line 11, in <module>
[sagemath_doc_html-none] [spkg-install]     main()
[sagemath_doc_html-none] [spkg-install]     ~~~~^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/__main__.py", line 548, in main
[sagemath_doc_html-none] [spkg-install]     build()
[sagemath_doc_html-none] [spkg-install]     ~~~~~^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 670, in _wrapper
[sagemath_doc_html-none] [spkg-install]     getattr(DocBuilder, build_type)(self, *args, **kwds)
[sagemath_doc_html-none] [spkg-install]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 143, in f
[sagemath_doc_html-none] [spkg-install]     runsphinx()
[sagemath_doc_html-none] [spkg-install]     ~~~~~~~~~^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 324, in runsphinx
[sagemath_doc_html-none] [spkg-install]     sys.stderr.raise_errors()
[sagemath_doc_html-none] [spkg-install]     ~~~~~~~~~~~~~~~~~~~~~~~^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
[sagemath_doc_html-none] [spkg-install]     raise OSError(self._error)
[sagemath_doc_html-none] [spkg-install] OSError: /home/release/Sage/build/sage-distro/src/doc/en/reference/spkg/jupyter_core.rst:37: WARNING: Literal block expected; none found. [docutils]

[vbraun]

This is when runing make -j10 doc-pdf, I did run bootstrap for the record

[tobiasdiez]

@vbraun I cannot reproduce this issue, and CI also doesn't show it. Could you please post the contents of src/doc/en/reference/spkg/jupyter_core.rst. Thanks!

[vbraun]

      1 .. _spkg_jupyter_core:
      2 
      3 jupyter_core: Jupyter core package
      4 ==================================
      5 
      6 Description
      7 -----------
      8 
      9 Jupyter core package. A base package on which Jupyter projects rely.
     10 
     11 Type
     12 ----
     13 
     14 standard
     15 
     16 
     17 Dependencies
     18 ------------
     19 
     20 - $(PYTHON)
     21 - $(PYTHON_TOOLCHAIN)
     22 - :ref:`spkg_hatchling`
     23 - :ref:`spkg_platformdirs`
     24 - :ref:`spkg_traitlets`
     25 
     26 Version Information
     27 -------------------
     28 
     29 package-version.txt::
     30 
     31     5.8.1
     32 
     33 version_requirements.txt::
     34 
     35 
     36 
     37 Equivalent System Packages
     38 --------------------------
     39 
     40 .. tab:: conda-forge
     41 
     42    .. CODE-BLOCK:: bash
     43 
     44        $ conda install jupyter_core 
     45 
     46 
     47 .. tab:: Fedora/Redhat/CentOS
     48 
     49    .. CODE-BLOCK:: bash
     50 
     51        $ sudo dnf install python3-jupyter-core 
     52 
     53 
     54 .. tab:: Gentoo Linux
     55 
     56    .. CODE-BLOCK:: bash
     57 
     58        $ sudo emerge dev-python/jupyter_core 
     59 
     60 
     61 .. tab:: MacPorts
     62 
     63    .. CODE-BLOCK:: bash
     64 
     65        $ sudo port install py-jupyter_core 
     66 
     67 
     68 .. tab:: openSUSE
     69 
     70    .. CODE-BLOCK:: bash
     71 
     72        $ sudo zypper install 'python3${PYTHON_MINOR}-jupyter-core' 
     73 
     74 
     75 .. tab:: Void Linux
     76 
     77    .. CODE-BLOCK:: bash
     78 
     79        $ sudo xbps-install python3-jupyter_core 
     80 
     81 
     82 
     83 See https://repology.org/project/jupyter-core/versions, https://repology.org/project/python:jupyter-core/versions
     84 
     85 If the system package is installed and if the (experimental) option
     86 ``--enable-system-site-packages`` is passed to ``./configure``, then ``./configure``
     87 will check if the system package can be used.
     88