Infra: bump Sphinx to 9.0+, limit docutils to <0.22

[Secrus]

Blocked by: #4908, #4906
Closes: #4087

Bump Sphinx to 9.0+ (9.2 at the moment). docutils is upper-limited because of build errors listed in #4924.

The mv in dirhtml step in the Makefile is no longer needed, since the 404 page builds as 404.html.

---

📚 Documentation preview 📚: https://pep-previews--4926.org.readthedocs.build/

[Secrus]

@hugovk, since there is only one PEP left on the #4087 list, could you please review the proposed change? I am mostly interested in knowing what the policy for older Sphinx versions is, and whether I should go with a lower base for the version range.

[hugovk]

Let's not set a higher version than needed, but if we have a good reason for bumping it, I think that's fine.

Something to note, Sphinx does drop older Python sooner than many projects. For example, Sphinx 9.1.0 requires Python 3.12 or newer.

But then people writing PEPs are targeting changes in 3.15 and soon 3.16 :)

[Secrus]

Ok, I have set the Sphinx requirement to > 8.1 to prevent the errors we were fixing. That seems like a reasonable middle ground.

[hugovk]

RTD:

Index file is not present in HTML output directory Your documentation did not generate an index.html at its root directory. This is required for documentation serving at the root URL for this version.

Try this in pep_html_builder.py:

     name = DirectoryHTMLBuilder.name
     get_target_uri = DirectoryHTMLBuilder.get_target_uri
-    get_outfilename = DirectoryHTMLBuilder.get_outfilename
+    get_output_path = DirectoryHTMLBuilder.get_output_path

Related to sphinx-doc/sphinx#13045.

[hugovk]

Only the upper bound is important here, Sphinx already sets docutils>=0.21,<0.23:

Suggested change

	docutils >= 0.19.0, < 0.22
	docutils < 0.22

And include a comment on why we need an upper bound? Something like pending sphinx-doc/sphinx#14455.

[hugovk]

I think we can do Sphinx 8.2 or greater?

Suggested change

	Sphinx > 9.0
	Sphinx >= 8.2