Skip to contentSkip to conversation

docs/python: bump minimum Sphinx version

PATCH Superseded John Snow 3 participants 10 messages Jul 2, 2024 – Jul 3, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover docs/python: bump minimum Sphinx version

With recent deprecations, we can advance our minimum sphinx version safely. This is heavily motivated by new qapidoc work which is much easier to maintain cross-version compatibility for - see difficulties in our dbus documentation which only works on sphinx >= 4.

We can only guarantee >= 3.4.3 now, but that's still vastly easier than maintaining compatibility all the way back to 1.x.

GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509

(failures appear to be unrelated to the series.)

John Snow (2):
Python: bump minimum sphinx version to 3.4.3
docs: remove Sphinx 1.x compatibility code

 docs/conf.py             |  7 +++----
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 docs/sphinx/qapidoc.py   | 29 +++--------------------------
 pythondeps.toml          |  2 +-
 6 files changed, 23 insertions(+), 102 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py
Signature · 1 line
2.45.0
author
#1 lore raw

PATCH 1/2 Python: bump minimum sphinx version to 3.4.3

With RHEL 8 support retired (It's been two years today since RHEL 9 came out), our very oldest build platform version of Sphinx is now 3.4.3; and keeping backwards compatibility for versions as old as v1.6 when using domain extensions is a lot of work we don't need to do.

This patch is motivated by my work creating a new QAPI domain, which unlike the dbus documentation, cannot be allowed to regress by creating a "dummy" doc. Easier is to improve our minimum version as far as we can push it forwards, reducing my burden in creating cross-compatibility hacks and patches.

A sampling of sphinx versions from various distributions, courtesy
https://repology.org/project/python:sphinx/versions

Alpine 3.16: v4.3.0 (QEMU support ended 2024-05-23) Alpine 3.17: v5.3.0
Alpine 3.18: v6.1.3
Alpine 3.19: v6.2.1
Ubuntu 20.04 LTS: EOL
Ubuntu 22.04 LTS: v4.3.2
Ubuntu 22.10: EOL
Ubuntu 23.04: EOL
Ubuntu 23.10: v5.3.0
Ubuntu 24.04 LTS: v7.2.6
Debian 11: v3.4.3 (QEMU support ends 2024-07-xx) Debian 12: v5.3.0
Fedora 38: EOL
Fedora 39: v6.2.1
Fedora 40: v7.2.6
CentOS Stream 8: v1.7.6 (QEMU support ended 2024-05-17) CentOS Stream 9: v3.4.3
OpenSUSE Leap 15.4: EOL
OpenSUSE Leap 15.5: 2.3.1, 4.2.0 and 7.2.6

---
 docs/conf.py           |  7 +++----
 docs/sphinx/qapidoc.py | 29 +++--------------------------
 pythondeps.toml        |  2 +-
 3 files changed, 7 insertions(+), 31 deletions(-)
docs/conf.py +3 -4
index aae0304ac6e..876f6768815 100644
@@ -53,10 +53,9 @@
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #
-# Sphinx 1.5 and earlier can't build our docs because they are too
-# picky about the syntax of the argument to the option:: directive
-# (see Sphinx bugs #646, #3366).
-needs_sphinx = '1.6'
+# 3.4.3 is the oldest version of Sphinx that ships on a platform we
+# pledge build support for.
+needs_sphinx = '3.4.3'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
docs/sphinx/qapidoc.py +3 -26
index f270b494f01..9a0cfcbce75 100644
@@ -31,6 +31,7 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 import sphinx
 from qapi.gen import QAPISchemaVisitor
@@ -38,15 +39,6 @@
 from qapi.schema import QAPISchema
 
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
-
-
 __version__ = '1.0'
 
 
@@ -517,23 +509,8 @@ def do_parse(self, rstlist, node):
         subheadings (titles) without confusing the rendering of
         anything else.
         """
-        # This is from kerneldoc.py -- it works around an API change in
-        # Sphinx between 1.6 and 1.7. Unlike kerneldoc.py, we use
-        # sphinx.util.nodes.nested_parse_with_titles() rather than the
-        # plain self.state.nested_parse(), and so we can drop the saving
-        # of title_styles and section_level that kerneldoc.py does,
-        # because nested_parse_with_titles() does that for us.
-        if Use_SSI:
-            with switch_source_input(self.state, rstlist):
-                nested_parse_with_titles(self.state, rstlist, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(
-                rstlist, self.state.memo.reporter)
-            try:
-                nested_parse_with_titles(self.state, rstlist, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, rstlist):
+            nested_parse_with_titles(self.state, rstlist, node)
 
 
 def setup(app):
pythondeps.toml +1 -1
index 9c16602d303..bc656376caa 100644
@@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=1.6", installed = "5.3.0", canary = "sphinx-build" }
+sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
 [avocado]
Signature · 1 line
2.45.0
Acked-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

Quoted 4 lines · John Snow

With RHEL 8 support retired (It's been two years today since RHEL 9 came out), our very oldest build platform version of Sphinx is now 3.4.3; and keeping backwards compatibility for versions as old as v1.6 when using domain extensions is a lot of work we don't need to do.

Technically that's unrelated: thanks to your venv work, :) builds on RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it runs under Python 3.6. Therefore the version included in RHEL 8 does not matter for picking the minimum supported Sphinx version.

Quoted 1 line

Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)

Nice. :)

pythondeps.toml meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" } · line 29

Quoted 13 lines
pythondeps.toml +1 -1
index 9c16602d303..bc656376caa 100644
@@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" }
 
  [docs]
  # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=1.6", installed = "5.3.0", canary = "sphinx-build" }
+sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
  sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
  [avocado]

Paolo

author
#3 lore raw

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

Quoted 11 lines · Paolo Bonzini
Quoted 4 lines · John Snow

With RHEL 8 support retired (It's been two years today since RHEL 9 came out), our very oldest build platform version of Sphinx is now 3.4.3; and keeping backwards compatibility for versions as old as v1.6 when using domain extensions is a lot of work we don't need to do.

Technically that's unrelated: thanks to your venv work, :) builds on RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it runs under Python 3.6. Therefore the version included in RHEL 8 does not matter for picking the minimum supported Sphinx version.

...!

I think I can't mandate 4.x because of RHEL 9 builds though, and offline requirements.

Quoted 26 lines
Quoted 1 line

Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)

Nice. :)

pythondeps.toml meson = { accepted = ">=0.63.0", installed = "1.2.3",

Quoted 5 lines
pythondeps.toml +0 -0
index 9c16602d303..bc656376caa 100644
@@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3",

canary = "meson" }

pythondeps.toml meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" }

Quoted 4 lines
pythondeps.toml +0 -1
  [docs]
  # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=1.6", installed = "5.3.0", canary =

"sphinx-build" }

pythondeps.toml meson = { accepted = ">=0.63.0", installed = "1.2.3", · line 25

Quoted 1 line
pythondeps.toml +1 -0
+sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary =

"sphinx-build" }

pythondeps.toml meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" } · line 29

Quoted 3 lines
pythondeps.toml +0 -0
  sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
  [avocado]

Paolo

#4 lore raw

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

Quoted 7 lines · John Snow
Quoted 6 lines · Paolo Bonzini
Quoted 4 lines · John Snow

With RHEL 8 support retired (It's been two years today since RHEL 9 came out), our very oldest build platform version of Sphinx is now 3.4.3; and keeping backwards compatibility for versions as old as v1.6 when using domain extensions is a lot of work we don't need to do.

Quoted 6 lines
Quoted 4 lines

Technically that's unrelated: thanks to your venv work, :) builds on RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it runs under Python 3.6. Therefore the version included in RHEL 8 does not matter for picking the minimum supported Sphinx version.

I think I can't mandate 4.x because of RHEL 9 builds though, and offline requirements.

Offline requirements are not a problem; on RHEL 8 you just have to install with pip in order to build docs offline. But yes, RHEL 9 is still using platform Python and therefore 3.4.3 remains the limit even after we stop supporting bullseye.

Paolo

author
#5 lore raw

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

Quoted 13 lines · Paolo Bonzini
Quoted 12 lines · John Snow
Quoted 10 lines · Paolo Bonzini
Quoted 4 lines · John Snow

With RHEL 8 support retired (It's been two years today since RHEL 9 came out), our very oldest build platform version of Sphinx is now 3.4.3; and keeping backwards compatibility for versions as old as v1.6 when using domain extensions is a lot of work we don't need to do.

Technically that's unrelated: thanks to your venv work, :) builds on RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it runs under Python 3.6. Therefore the version included in RHEL 8 does not matter for picking the minimum supported Sphinx version.

Quoted 8 lines
Quoted 1 line

I think I can't mandate 4.x because of RHEL 9 builds though, and offline

requirements.

Offline requirements are not a problem; on RHEL 8 you just have to install with pip in order to build docs offline. But yes, RHEL 9 is still using platform Python and therefore 3.4.3 remains the limit even after we stop supporting bullseye.

To be clear I mean offline, isolated RPM builds under RHEL9 where I don't think we can utilize PyPI at all; and vendoring Sphinx is I think not a practical option due to the number of dependencies and non-pure Python deps.

It's not a problem for developer workflow, just downstream packaging.

Luckily OpenSUSE offers newer Sphinx, but RHEL doesn't yet. Maybe that can be rectified eventually - possibly after 3.8 is EOL and there is increased demand for newer Python packages to be made available in RHEL... but not yet, today.

Quoted 4 lines

Paolo

Thanks for the ack O:-)

#6 lore raw

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

Quoted 3 lines · John Snow

To be clear I mean offline, isolated RPM builds under RHEL9 where I don't think we can utilize PyPI at all; and vendoring Sphinx is I think not a practical option due to the number of dependencies and non-pure Python deps.

It's not a problem for developer workflow, just downstream packaging.

Agreed, switching to PyPI packages is not something that should be done lightly---only after the platform Python is EOL, and only if there are benefits for QEMU to require 3.9 (like there were for 3.7).

In that case, downstream could still package QEMU but they'd have to build python39-sphinx or something like that.

Paolo

Quoted 8 lines

Luckily OpenSUSE offers newer Sphinx, but RHEL doesn't yet. Maybe that can be rectified eventually - possibly after 3.8 is EOL and there is increased demand for newer Python packages to be made available in RHEL... but not yet, today.

Quoted 4 lines

Paolo

Thanks for the ack O:-)

author
#7 lore raw

PATCH 2/2 docs: remove Sphinx 1.x compatibility code

In reply to John Snow

In general, the Use_SSI workaround is no longer needed, and neither is the pre-1.6 logging shim for kerneldoc.

---
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 3 files changed, 16 insertions(+), 71 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py
docs/sphinx/hxtool.py +4 -17
index 3729084a36c..a84723be19e 100644
@@ -24,16 +24,10 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 import sphinx
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
 
 __version__ = '1.0'
 
@@ -185,16 +179,9 @@ def run(self):
     # of title_styles and section_level that kerneldoc.py does,
     # because nested_parse_with_titles() does that for us.
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                nested_parse_with_titles(self.state, result, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            try:
-                nested_parse_with_titles(self.state, result, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            nested_parse_with_titles(self.state, result, node)
+
 
 def setup(app):
     """ Register hxtool-doc directive with Sphinx"""
docs/sphinx/kerneldoc.py +12 -26
index 72c403a7379..3aa972f2e89 100644
@@ -38,20 +38,14 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 
-#
-# AutodocReporter is only good up to Sphinx 1.7
-#
 import sphinx
+from sphinx.util import logging
+from sphinx.util.docutils import switch_source_input
 
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
-
-import kernellog
 
 __version__  = '1.0'
+logger = logging.getLogger('kerneldoc')
+
 
 class KernelDocDirective(Directive):
     """Extract kernel-doc comments from the specified file"""
@@ -111,8 +105,7 @@ def run(self):
         cmd += [filename]
 
         try:
-            kernellog.verbose(env.app,
-                              'calling kernel-doc \'%s\'' % (" ".join(cmd)))
+            logger.verbose('calling kernel-doc \'%s\'' % (" ".join(cmd)))
 
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             out, err = p.communicate()
@@ -122,8 +115,10 @@ def run(self):
             if p.returncode != 0:
                 sys.stderr.write(err)
 
-                kernellog.warn(env.app,
-                               'kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))
+                logger.warning(
+                    'kernel-doc \'%s\' failed with return code %d' %
+                    (" ".join(cmd), p.returncode)
+                )
                 return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
             elif env.config.kerneldoc_verbosity > 0:
                 sys.stderr.write(err)
@@ -149,22 +144,13 @@ def run(self):
             return node.children
 
         except Exception as e:  # pylint: disable=W0703
-            kernellog.warn(env.app, 'kernel-doc \'%s\' processing failed with: %s' %
+            logger.warning('kernel-doc \'%s\' processing failed with: %s' %
                            (" ".join(cmd), str(e)))
             return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
 
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                self.state.nested_parse(result, 0, node, match_titles=1)
-        else:
-            save = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            self.state.memo.title_styles, self.state.memo.section_level = [], 0
-            try:
-                self.state.nested_parse(result, 0, node, match_titles=1)
-            finally:
-                self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            self.state.nested_parse(result, 0, node, match_titles=1)
 
 
 def setup(app):
docs/sphinx/kernellog.py deleted +0 -28
deleted file mode 100644
index af924f51a7d..00000000000
@@ -1,28 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# Sphinx has deprecated its older logging interface, but the replacement
-# only goes back to 1.6.  So here's a wrapper layer to keep around for
-# as long as we support 1.4.
-#
-import sphinx
-
-if sphinx.__version__[:3] >= '1.6':
-    UseLogging = True
-    from sphinx.util import logging
-    logger = logging.getLogger('kerneldoc')
-else:
-    UseLogging = False
-
-def warn(app, message):
-    if UseLogging:
-        logger.warning(message)
-    else:
-        app.warn(message)
-
-def verbose(app, message):
-    if UseLogging:
-        logger.verbose(message)
-    else:
-        app.verbose(message)
-
-
Signature · 1 line
2.45.0
author
#8 lore raw
In reply to John Snow
Quoted 25 lines · John Snow

With recent deprecations, we can advance our minimum sphinx version safely. This is heavily motivated by new qapidoc work which is much easier to maintain cross-version compatibility for - see difficulties in our dbus documentation which only works on sphinx >= 4.

We can only guarantee >= 3.4.3 now, but that's still vastly easier than maintaining compatibility all the way back to 1.x.

GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509

(failures appear to be unrelated to the series.)

John Snow (2):
Python: bump minimum sphinx version to 3.4.3
docs: remove Sphinx 1.x compatibility code

 docs/conf.py             |  7 +++----
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 docs/sphinx/qapidoc.py   | 29 +++--------------------------
 pythondeps.toml          |  2 +-
 6 files changed, 23 insertions(+), 102 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py
Quoted 3 lines

--
2.45.0

Bleurgh. I meant to shuffle some of the changes in patch 1 into patch 2, I'll fix that on re-spin. If you want to review it anyway, just presume I'll fix that next go-around.

--js

#9 lore raw
Quoted 27 lines · John Snow
Quoted 25 lines · John Snow

With recent deprecations, we can advance our minimum sphinx version safely. This is heavily motivated by new qapidoc work which is much easier to maintain cross-version compatibility for - see difficulties in our dbus documentation which only works on sphinx >= 4.

We can only guarantee >= 3.4.3 now, but that's still vastly easier than maintaining compatibility all the way back to 1.x.

GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509

(failures appear to be unrelated to the series.)

John Snow (2):
Python: bump minimum sphinx version to 3.4.3
docs: remove Sphinx 1.x compatibility code

 docs/conf.py             |  7 +++----
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 docs/sphinx/qapidoc.py   | 29 +++--------------------------
 pythondeps.toml          |  2 +-
 6 files changed, 23 insertions(+), 102 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py
Quoted 5 lines
Quoted 3 lines

--
2.45.0

Bleurgh. I meant to shuffle some of the changes in patch 1 into patch 2,

I was wondering about the split :)

Quoted 2 lines

I'll fix that on re-spin. If you want to review it anyway, just presume I'll fix that next go-around.