Skip to contentSkip to conversation

python: require python3.12+

PATCH Superseded John Snow 4 participants 30 messages Aug 25 RISC-V

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

v2 cover python: require python3.12+

The stars have aligned, and we can jump all the way from Python3.9 to Python3.12.

GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2787831126

v2:
- s/types_seen/objects_seen in patch 8
- drop python3-devel from mappings.yml
- add comments explaining contentnode vs content_node in #16
- fix VM tests to use local QEMU build to support -object qmp-monitor in order to run tests (Thanks danpb)
- fix FreeBSD vm test (Pre-existing breakage)
- fix netBSD vm test (Pre-existing breakage)
- Known issue: Haiku VM test is broken, but this series makes it worse. Existing problem is that tomli has gone missing; new problem is that our Haiku image still only offers Python 3.9. This VM test has not been meaningfully updated since its introduction in 2020, and the 2023 r4 build on vagrant's registry is still the most recent version, so... I think if we want Haiku to work, it's up to someone else to show it some love.

Python 3.9 is EOL, and Python 3.10 will be EOL in about two months (October 2026). Python 3.11 is, to be blunt, perfectly serviceable - but when investigating our actual minimum dependency requirements, I realized none of our supported platforms used anything older than Python 3.12, so we may as well build and test with the version that all of our build platforms are actually using.

This reduces the Python compat test matrix by quite a lot and allows us a few nice things. From memory, the highlights are:

- Pattern matching syntax (fancy case statements). I am not dying to use these anywhere, but they're nice!
- Vastly improved error reporting from CPython
- Ability to annotate exceptions with "notes", which allows us augment the default reporting behavior for unhandled exceptions - this can be used in e.g. iotests to help give additional information on tool crashes that is sometimes hard to discover otherwise
- The ability to use standard library types for typing syntax instead of importing from Typing (Technically, valid since 3.9, but...)
- The ability to use "x | y" instead of Union[x, y] in type hint syntax. Combined with the above, it's a good time to clean up the typing in qapi to make it read much more nicely.
- ExceptionGroups - the ability to capture and report multiple exceptions simultaneously. This is extremely useful for async QMP and managing multiple VMs from a single python context. Async QMP will be able to get a few nice QOL improvements with this.

My personal favorite:

- We can move to a much newer Sphinx, which gets rid of an awful lot of very ugly compat code. This is the *actual* motivation of this series.

Anyway, here's the version chart for our supported build platforms and why this is safe to do:

distro EOS python3 pip setuptools sphinx wheel
-------------------------------------------------------------------------

centos_stream_9   2026-12-12  3.9.25   21.3.1  53.0.0      3.4.3   0.36.2
centos_stream_9+  2026-12-12  3.12.13  23.2.1  68.2.2      ---     0.41.2
freebsd           --          3.12.14  23.3.2  63.1.0      9.0.4   0.48.0
ubuntu_24_04      2028-04-23  3.12.3   24.0    68.1.2      7.2.6   0.42.0
centos_stream_10  2030-01-01  3.12.13  23.3.2  69.0.3      7.2.6   0.41.2
alpine_3_21       2026-11-01  3.12.14  24.3.1  70.3.0      8.1.3   0.43.0
debian_13         2028-08-09  3.13.5   25.1.1  78.1.1      8.1.3   0.46.1
fedora_43         2026-12-09  3.14.7   25.1.1  78.1.1      8.2.3   0.45.1
ubuntu_26_04      2031-05-29  3.14.4   25.1.1  78.1.1      8.2.3   0.46.3
fedora_44         2027-06-02  3.14.7   26.0.1  80.10.2     8.2.3   0.45.1
macports          --          3.14.7   26.0.1  80.10.2     9.1.0   0.48.0
alpine_3_22       2027-05-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
alpine_3_23       2027-11-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
openbsd           --          3.14.6   26.2.1  80.9.0      9.1.0   0.48.0
alpine_3_24       2028-06-01  3.14.7   26.1.2  82.0.1      9.1.0   0.47.0
pkgsrc_current    --          3.13.15  26.2.1  84.0.0      9.1.0   0.48.0

CentOS stream 9 note: python3.12, python3.12-pip, python3.12-setuptools and python3.12-wheel are available as optional packages in the standard repository. Sphinx is not available, but can be side-loaded from PyPI using pip3.12.

More cleanups to follow: we may have the chance to upgrade to pyproject.toml packaging once and for all now that FreeBSD has moved to Python3.12, which will get us off the legacy packaging we've been using. This isn't a problem *currently*, but it has been the cause of a lot of pain in the past, and the sooner we can ditch it the better.

toodles,
--js

John Snow (16):
tests: update vm tests to prefer locally-built QEMU
tests: update FreeBSD dependencies
tests: update netbsd dependencies
tests: drop python3-devel mapping
tests: re-add opencv2 and numpy dependencies
tests: add Python3.12 to centos9.docker
tests: update emsdk-wasm64-cross
qapi: update type hint for objects_seen
python: upgrade to Python3.12+
python: fix license configuration
python: allow mypy to run as the current python version
python: remove distutils workarounds
python: remove stale setup.cfg options
sphinx: require sphinx 7.2.6
sphinx: drop fakedbusdoc
sphinx: drop qapidoc back-compat overboiled spaghetti dinner

 docs/conf.py                                  |  11 +-
 docs/interop/dbus-display.rst                 |  14 +-
 docs/interop/dbus-vmstate.rst                 |  14 +-
 docs/interop/dbus-vnc.rst                     |  14 +-
 docs/requirements.txt                         |   2 +-
 docs/sphinx/compat.py                         | 230 ------------------
 docs/sphinx/fakedbusdoc.py                    |  30 ---
 docs/sphinx/qapi_domain.py                    |  53 ++--
 configure                                     |  11 +-
 tests/docker/dockerfiles/alpine.docker        |   1 +
 tests/docker/dockerfiles/centos9.docker       |  14 +-
 .../dockerfiles/debian-amd64-cross.docker     |   2 +
 .../dockerfiles/debian-arm64-cross.docker     |   2 +
 .../dockerfiles/debian-armhf-cross.docker     |   2 +
 .../dockerfiles/debian-i686-cross.docker      |   2 +
 .../dockerfiles/debian-ppc64el-cross.docker   |   2 +
 .../dockerfiles/debian-riscv64-cross.docker   |   2 +
 .../dockerfiles/debian-s390x-cross.docker     |   2 +
 tests/docker/dockerfiles/debian.docker        |   2 +
 .../dockerfiles/emsdk-wasm64-cross.docker     |   4 +-
 .../dockerfiles/fedora-rust-nightly.docker    |   2 +
 .../dockerfiles/fedora-win64-cross.docker     |   2 +
 tests/docker/dockerfiles/fedora.docker        |   2 +
 tests/docker/dockerfiles/opensuse-leap.docker |   2 +
 tests/docker/dockerfiles/ubuntu2404.docker    |   2 +
 .gitlab-ci.d/macos-14.vars                    |   2 +-
 python/Makefile                               |  10 +-
 python/scripts/mkvenv.py                      |  63 +----
 python/setup.cfg                              |  33 +--
 python/tests/linters.py                       |  11 -
 python/tests/minreqs.txt                      |  66 ++---
 pythondeps.toml                               |  13 +-
 .../ci/setup/debian/debian-13-ppc64le.yaml    |   2 +
 .../ci/setup/ubuntu/ubuntu-2404-aarch64.yaml  |   2 +
 .../ci/setup/ubuntu/ubuntu-2404-s390x.yaml    |   2 +
 scripts/qapi/types.py                         |   2 +-
 tests/lcitool/mappings.yml                    |  32 ++-
 tests/lcitool/projects/qemu.yml               |   2 +
 tests/lcitool/targets/centos-stream-9.yml     |   4 +-
 tests/vm/Makefile.include                     |   2 +-
 tests/vm/generated/freebsd.json               |  14 +-
 tests/vm/netbsd                               |   1 +
 42 files changed, 178 insertions(+), 507 deletions(-)
 delete mode 100644 docs/sphinx/compat.py
 delete mode 100644 docs/sphinx/fakedbusdoc.py
Signature · 1 line
2.55.0
author
#1 lore raw

PATCH v2 1/16 tests: update vm tests to prefer locally-built QEMU

In 05e4982, we updated our QMP tests to prefer -object monitor-qmp, but this requires a sufficiently recent and modern version of QEMU to run correctly.

Update the "make vm-run-xxx" targets to use the runscript that will naturally prefer the locally built version of QEMU so that these tests launch successfully once more.

---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/Makefile.include +1 -1
index 0a53dc9fdd6..0ec1b47e63e 100644
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = $(SRC_PATH)/vm-venv/bin/python3
 VM_VENV = vm-venv
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(BUILD_DIR)/run $(PYTHON)
 VM_VENV =
 endif
 
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH v2 01/16] tests: update vm tests to prefer locally-built QEMU

Quoted 3 lines · John Snow

In 05e4982, we updated our QMP tests to prefer -object monitor-qmp, but this requires a sufficiently recent and modern version of QEMU to run correctly.

So, some tests run incorrectly, and this patch fixes them. Which ones and how? What broke them? Commit 05e4982? Is it worth a Fixes: tag?

Quoted 21 lines

Update the "make vm-run-xxx" targets to use the runscript that will naturally prefer the locally built version of QEMU so that these tests launch successfully once more.

---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/Makefile.include +1 -1
index 0a53dc9fdd6..0ec1b47e63e 100644
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = $(SRC_PATH)/vm-venv/bin/python3
 VM_VENV = vm-venv
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(BUILD_DIR)/run $(PYTHON)
 VM_VENV =
 endif
#3 lore raw

Re: [PATCH v2 01/16] tests: update vm tests to prefer locally-built QEMU

Quoted 8 lines · Markus Armbruster
Quoted 3 lines · John Snow

In 05e4982, we updated our QMP tests to prefer -object monitor-qmp, but this requires a sufficiently recent and modern version of QEMU to run correctly.

So, some tests run incorrectly, and this patch fixes them. Which ones and how? What broke them? Commit 05e4982? Is it worth a Fixes: tag?

tests/vm/* because they don't use QEMU from the build tree and so can't be assumed to have 'monitor-qmp' available, unlike the iotests / functional tests. Yes commit 05e4982, and yes add a Fixes.

Quoted 10 lines
Quoted 10 lines

Update the "make vm-run-xxx" targets to use the runscript that will naturally prefer the locally built version of QEMU so that these tests launch successfully once more.

---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

tests/vm/Makefile.include ifeq ($(realpath $(SRC_PATH)),$(realpath .)) · line 10

Quoted 12 lines
tests/vm/Makefile.include +1 -1
index 0a53dc9fdd6..0ec1b47e63e 100644
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = $(SRC_PATH)/vm-venv/bin/python3
 VM_VENV = vm-venv
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(BUILD_DIR)/run $(PYTHON)
 VM_VENV =
 endif

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#4 lore raw

Re: [PATCH v2 01/16] tests: update vm tests to prefer locally-built QEMU

Quoted 7 lines · Daniel P. Berrangé
Quoted 6 lines · Markus Armbruster
Quoted 3 lines · John Snow

In 05e4982, we updated our QMP tests to prefer -object monitor-qmp, but this requires a sufficiently recent and modern version of QEMU to run correctly.

Quoted 7 lines
Quoted 2 lines

So, some tests run incorrectly, and this patch fixes them. Which ones and how? What broke them? Commit 05e4982? Is it worth a Fixes: tag?

tests/vm/* because they don't use QEMU from the build tree and so can't be assumed to have 'monitor-qmp' available, unlike the iotests / functional tests. Yes commit 05e4982, and yes add a Fixes.

Will do.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Update the "make vm-run-xxx"

* make vm-build-xxx. Thinko.

targets to use the runscript that will

Quoted 31 lines
Quoted 8 lines
Quoted 8 lines

naturally prefer the locally built version of QEMU so that these tests launch successfully once more.

---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

tests/vm/Makefile.include ifeq ($(realpath $(SRC_PATH)),$(realpath .)) · line 10

Quoted 12 lines
tests/vm/Makefile.include +1 -1
index 0a53dc9fdd6..0ec1b47e63e 100644
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = $(SRC_PATH)/vm-venv/bin/python3
 VM_VENV = vm-venv
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(BUILD_DIR)/run $(PYTHON)
 VM_VENV =
 endif

With regards,
Daniel
--

|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
Reviewed-by on v2 1/16
#5 lore raw

Re: [PATCH v2 01/16] tests: update vm tests to prefer locally-built QEMU

In reply to John Snow
Quoted 4 lines · John Snow

In 05e4982, we updated our QMP tests to prefer -object monitor-qmp, but this requires a sufficiently recent and modern version of QEMU to run correctly.

Quoted 8 lines

Update the "make vm-run-xxx" targets to use the runscript that will naturally prefer the locally built version of QEMU so that these tests launch successfully once more.

---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 18 lines
tests/vm/Makefile.include +1 -1
index 0a53dc9fdd6..0ec1b47e63e 100644
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = $(SRC_PATH)/vm-venv/bin/python3
 VM_VENV = vm-venv
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(BUILD_DIR)/run $(PYTHON)
 VM_VENV =
 endif
 
Signature · 1 line
2.55.0

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#6 lore raw

PATCH v2 2/16 tests: update FreeBSD dependencies

In reply to John Snow

FreeBSD has dropped Python3.11, now defaulting to Python3.12+; upgrade the names of the various dependencies to match.

---
 tests/lcitool/mappings.yml      |  9 +++++++++
 tests/vm/generated/freebsd.json | 12 ++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)
tests/lcitool/mappings.yml +9 -0
index 1816556dde4..afe52455235 100644
@@ -11,21 +11,29 @@ mappings:
 
   python3-pip:
     CentOSStream9: python3.11-pip
+    FreeBSD: py312-pip
 
   python3-setuptools:
     CentOSStream9: python3.11-setuptools
+    FreeBSD: py312-setuptools
 
   python3-sphinx:
     CentOSStream9:
+    FreeBSD: py312-sphinx
 
   python3-sphinx-rtd-theme:
     CentOSStream9:
+    FreeBSD: py312-sphinx_rtd_theme
 
   python3-venv:
     CentOSStream9: python3.11
 
   python3-wheel:
     CentOSStream9: python3.11-wheel
+    FreeBSD: py312-wheel
+
+  python3-numpy:
+    FreeBSD: py312-numpy
 
   meson:
     # Use Meson from PyPI wherever Rust is enabled
@@ -38,6 +46,7 @@ mappings:
     apk:
     Fedora:
     Debian12:
+    FreeBSD: py312-tomli
 
   rust:
     Debian12: rustc-web
tests/vm/generated/freebsd.json +6 -6
index dfa8b41a524..ee3402f0d2b 100644
@@ -52,12 +52,12 @@
     "pixman",
     "pkgconf",
     "png",
-    "py311-pip",
-    "py311-setuptools",
-    "py311-sphinx",
-    "py311-sphinx_rtd_theme",
-    "py311-tomli",
-    "py311-wheel",
+    "py312-pip",
+    "py312-setuptools",
+    "py312-sphinx",
+    "py312-sphinx_rtd_theme",
+    "py312-tomli",
+    "py312-wheel",
     "python3",
     "rpm2cpio",
     "rust",
Signature · 1 line
2.55.0
#7 lore raw

Re: [PATCH v2 02/16] tests: update FreeBSD dependencies

Quoted 44 lines · John Snow

FreeBSD has dropped Python3.11, now defaulting to Python3.12+; upgrade the names of the various dependencies to match.

---
 tests/lcitool/mappings.yml      |  9 +++++++++
 tests/vm/generated/freebsd.json | 12 ++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)
tests/lcitool/mappings.yml +8 -0
index 1816556dde4..afe52455235 100644
@@ -11,21 +11,29 @@ mappings:
 
   python3-pip:
     CentOSStream9: python3.11-pip
+    FreeBSD: py312-pip
 
   python3-setuptools:
     CentOSStream9: python3.11-setuptools
+    FreeBSD: py312-setuptools
 
   python3-sphinx:
     CentOSStream9:
+    FreeBSD: py312-sphinx
 
   python3-sphinx-rtd-theme:
     CentOSStream9:
+    FreeBSD: py312-sphinx_rtd_theme
 
   python3-venv:
     CentOSStream9: python3.11
 
   python3-wheel:
     CentOSStream9: python3.11-wheel
+    FreeBSD: py312-wheel
+
+  python3-numpy:
+    FreeBSD: py312-numpy
 
   meson:
     # Use Meson from PyPI wherever Rust is enabled

tests/lcitool/mappings.yml mappings: · line 52

Quoted 7 lines
tests/lcitool/mappings.yml +1 -0
@@ -38,6 +46,7 @@ mappings:
     apk:
     Fedora:
     Debian12:
+    FreeBSD: py312-tomli
 
   rust:
     Debian12: rustc-web

This shouldn't be done here. libvirt-ci.git needs to be updated to have the new names. FreeBSD does this annoying package rename every time a new python version is adopted :-(

I've proposed the required changes:

https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/582

Once that's merged just updated the tests/lcitool/libvirt-ci git submodule commit to latest HEAD and run 'make lcitool-refresh' again in this commit.

Quoted 27 lines
tests/vm/generated/freebsd.json +6 -6
index dfa8b41a524..ee3402f0d2b 100644
@@ -52,12 +52,12 @@
     "pixman",
     "pkgconf",
     "png",
-    "py311-pip",
-    "py311-setuptools",
-    "py311-sphinx",
-    "py311-sphinx_rtd_theme",
-    "py311-tomli",
-    "py311-wheel",
+    "py312-pip",
+    "py312-setuptools",
+    "py312-sphinx",
+    "py312-sphinx_rtd_theme",
+    "py312-tomli",
+    "py312-wheel",
     "python3",
     "rpm2cpio",
     "rust",
Signature · 1 line
2.55.0

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#8 lore raw

Re: [PATCH v2 02/16] tests: update FreeBSD dependencies

Quoted 50 lines · Daniel P. Berrangé
Quoted 49 lines · John Snow

FreeBSD has dropped Python3.11, now defaulting to Python3.12+; upgrade the names of the various dependencies to match.

---
 tests/lcitool/mappings.yml      |  9 +++++++++
 tests/vm/generated/freebsd.json | 12 ++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)
tests/lcitool/mappings.yml +9 -0
index 1816556dde4..afe52455235 100644
@@ -11,21 +11,29 @@ mappings:
 
   python3-pip:
     CentOSStream9: python3.11-pip
+    FreeBSD: py312-pip
 
   python3-setuptools:
     CentOSStream9: python3.11-setuptools
+    FreeBSD: py312-setuptools
 
   python3-sphinx:
     CentOSStream9:
+    FreeBSD: py312-sphinx
 
   python3-sphinx-rtd-theme:
     CentOSStream9:
+    FreeBSD: py312-sphinx_rtd_theme
 
   python3-venv:
     CentOSStream9: python3.11
 
   python3-wheel:
     CentOSStream9: python3.11-wheel
+    FreeBSD: py312-wheel
+
+  python3-numpy:
+    FreeBSD: py312-numpy
 
   meson:
     # Use Meson from PyPI wherever Rust is enabled
@@ -38,6 +46,7 @@ mappings:
     apk:
     Fedora:
     Debian12:
+    FreeBSD: py312-tomli
 
Quoted 7 lines

tests/lcitool/mappings.yml mappings: · line 52

Quoted 2 lines
tests/lcitool/mappings.yml +0 -0
   rust:
     Debian12: rustc-web

This shouldn't be done here. libvirt-ci.git needs to be updated to have the new names. FreeBSD does this annoying package rename every time a new python version is adopted :-(

I figured as much, but wanted to see it run for this series, so I thundered ahead on the stupid way.

Mea culpa.

Quoted 9 lines

I've proposed the required changes:

https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/582

Once that's merged just updated the tests/lcitool/libvirt-ci git submodule commit to latest HEAD and run 'make lcitool-refresh' again in this commit.

OK, great! Thanks!

Quoted 37 lines

Review

Quoted 1 line
(quoted diff) +0 -0

b/tests/vm/generated/freebsd.json

Quoted 23 lines

index dfa8b41a524..ee3402f0d2b 100644

tests/vm/generated/freebsd.json +6 -7
@@ -52,12 +52,12 @@
     "pixman",
     "pkgconf",
     "png",
-    "py311-pip",
-    "py311-setuptools",
-    "py311-sphinx",
-    "py311-sphinx_rtd_theme",
-    "py311-tomli",
-    "py311-wheel",
+    "py312-pip",
+    "py312-setuptools",
+    "py312-sphinx",
+    "py312-sphinx_rtd_theme",
+    "py312-tomli",
+    "py312-wheel",
     "python3",
     "rpm2cpio",
     "rust",
--
Quoted 3 lines

2.55.0

With regards,
Daniel
--

|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#9 lore raw

PATCH v2 3/16 tests: update netbsd dependencies

In reply to John Snow

NetBSD appears to need 'bison' explicitly installed now.

I am still seeing some intermittant failures in the tests for ide-test and qos-test, but those are mysteries for another day.

---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)
tests/vm/netbsd +1 -0
index b365045066c..45afe675e17 100755
@@ -39,6 +39,7 @@ class NetBSDVM(basevm.BaseVM):
         "gmake",
         "gsed",
         "gettext-tools",
+        "bison",
 
         # libs: basic
         "dtc",
Signature · 1 line
2.55.0
Reviewed-by on v2 3/16
#10 lore raw

Re: [PATCH v2 03/16] tests: update netbsd dependencies

Quoted 7 lines · John Snow

NetBSD appears to need 'bison' explicitly installed now.

I am still seeing some intermittant failures in the tests for ide-test and qos-test, but those are mysteries for another day.

Quoted 19 lines
---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)
tests/vm/netbsd +1 -1
index b365045066c..45afe675e17 100755
@@ -39,6 +39,7 @@ class NetBSDVM(basevm.BaseVM):
         "gmake",
         "gsed",
         "gettext-tools",
+        "bison",
 
         # libs: basic
         "dtc",
--

2.55.0

Reviewed-by on v2 3/16
#11 lore raw

Re: [PATCH v2 03/16] tests: update netbsd dependencies

In reply to John Snow
Quoted 9 lines · John Snow

NetBSD appears to need 'bison' explicitly installed now.

I am still seeing some intermittant failures in the tests for ide-test and qos-test, but those are mysteries for another day.

---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)

and a reminder that we'd benefit from adding NetBSD to libvirt-ci mappings too, so we can remove the manual package listing.

Quoted 17 lines
tests/vm/netbsd +1 -0
index b365045066c..45afe675e17 100755
@@ -39,6 +39,7 @@ class NetBSDVM(basevm.BaseVM):
         "gmake",
         "gsed",
         "gettext-tools",
+        "bison",
 
         # libs: basic
         "dtc",
Signature · 1 line
2.55.0

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#12 lore raw

Re: [PATCH v2 03/16] tests: update netbsd dependencies

Quoted 11 lines · Daniel P. Berrangé
Quoted 9 lines · John Snow

NetBSD appears to need 'bison' explicitly installed now.

I am still seeing some intermittant failures in the tests for ide-test and qos-test, but those are mysteries for another day.

---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)
Quoted 7 lines

and a reminder that we'd benefit from adding NetBSD to libvirt-ci mappings too, so we can remove the manual package listing.

I'll look into it. Is it possible to mock this up using the qemu local mappings.yml before I try to submit code to libvirt-ci?

I guess I can just check out the submodule and hack up what I need to and go that route.

IIRC there's a category called pkgs in libvirt-ci, and we'd just need a new one for "pkgsrc" and an os target named (checking canon capitalization...) "NetBSD" and that'll pretty much handle it.

Quoted 26 lines
Quoted 14 lines
tests/vm/netbsd +1 -1
index b365045066c..45afe675e17 100755
@@ -39,6 +39,7 @@ class NetBSDVM(basevm.BaseVM):
         "gmake",
         "gsed",
         "gettext-tools",
+        "bison",
 
         # libs: basic
         "dtc",
--
Quoted 3 lines

2.55.0

With regards,
Daniel
--

|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#13 lore raw

PATCH v2 4/16 tests: drop python3-devel mapping

In reply to John Snow

This is not being used, so drop it.

---
 tests/lcitool/mappings.yml | 3 ---
 1 file changed, 3 deletions(-)
tests/lcitool/mappings.yml +0 -3
index afe52455235..b576ded3edc 100644
@@ -6,9 +6,6 @@ mappings:
   python3:
     CentOSStream9: python3.11
 
-  python3-devel:
-    CentOSStream9: python3.11-devel
-
   python3-pip:
     CentOSStream9: python3.11-pip
     FreeBSD: py312-pip
Signature · 1 line
2.55.0
Reviewed-by on v2 4/16
#14 lore raw

Re: [PATCH v2 04/16] tests: drop python3-devel mapping

Quoted 6 lines · John Snow

This is not being used, so drop it.

---
 tests/lcitool/mappings.yml | 3 ---
 1 file changed, 3 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#15 lore raw

PATCH v2 5/16 tests: re-add opencv2 and numpy dependencies

In reply to John Snow
Fixes 2a428c8541ec5e9be54aad5f0a446ec8172226c9

Some of the dependencies identified as avocado dependencies were in fact still being used, and should be re-added to our lcitool configuration to ensure that these tests are being executed in the CI environment.

---
 tests/docker/dockerfiles/alpine.docker               |  1 +
 tests/docker/dockerfiles/centos9.docker              |  2 ++
 tests/docker/dockerfiles/debian-amd64-cross.docker   |  2 ++
 tests/docker/dockerfiles/debian-arm64-cross.docker   |  2 ++
 tests/docker/dockerfiles/debian-armhf-cross.docker   |  2 ++
 tests/docker/dockerfiles/debian-i686-cross.docker    |  2 ++
 tests/docker/dockerfiles/debian-ppc64el-cross.docker |  2 ++
 tests/docker/dockerfiles/debian-riscv64-cross.docker |  2 ++
 tests/docker/dockerfiles/debian-s390x-cross.docker   |  2 ++
 tests/docker/dockerfiles/debian.docker               |  2 ++
 tests/docker/dockerfiles/fedora-rust-nightly.docker  |  2 ++
 tests/docker/dockerfiles/fedora-win64-cross.docker   |  2 ++
 tests/docker/dockerfiles/fedora.docker               |  2 ++
 tests/docker/dockerfiles/opensuse-leap.docker        |  2 ++
 tests/docker/dockerfiles/ubuntu2404.docker           |  2 ++
 .gitlab-ci.d/macos-14.vars                           |  2 +-
 scripts/ci/setup/debian/debian-13-ppc64le.yaml       |  2 ++
 scripts/ci/setup/ubuntu/ubuntu-2404-aarch64.yaml     |  2 ++
 scripts/ci/setup/ubuntu/ubuntu-2404-s390x.yaml       |  2 ++
 tests/lcitool/mappings.yml                           | 10 ++++++++++
 tests/lcitool/projects/qemu.yml                      |  2 ++
 tests/vm/generated/freebsd.json                      |  2 ++
 22 files changed, 50 insertions(+), 1 deletion(-)
tests/docker/dockerfiles/alpine.docker +1 -0
index 8a5a355e89a..7c342e801c5 100644
@@ -83,6 +83,7 @@ RUN apk update && \
         pixman-dev \
         pkgconf \
         pulseaudio-dev \
+        py3-numpy \
         py3-pip \
         py3-setuptools \
         py3-sphinx \
tests/docker/dockerfiles/centos9.docker +2 -0
index 8101370a2db..6f5c400c2db 100644
@@ -130,6 +130,8 @@ RUN dnf --quiet distro-sync -y && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
 RUN /usr/bin/pip3.11 install \
+                     numpy \
+                     opencv-python \
                      sphinx \
                      sphinx-rtd-theme
 
tests/docker/dockerfiles/debian-amd64-cross.docker +2 -0
index 9e30457ca79..9b446a480b1 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-arm64-cross.docker +2 -0
index b9b94e5e6fa..95c757edb7c 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-armhf-cross.docker +2 -0
index aa412ff66f5..0c6287408ac 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-i686-cross.docker +2 -0
index d07eac84fe8..99447e2e79b 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-ppc64el-cross.docker +2 -0
index 7302c8236e4..534c9b55836 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-riscv64-cross.docker +2 -0
index aa5a881cab2..0df92c80c05 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian-s390x-cross.docker +2 -0
index 82eae4fc2c2..12c94a40ec7 100644
@@ -41,6 +41,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/debian.docker +2 -0
index 3a4f208ccf6..d808c945f46 100644
@@ -113,6 +113,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
tests/docker/dockerfiles/fedora-rust-nightly.docker +2 -0
index 0f3dc370a4f..0f7b4c13388 100644
@@ -104,6 +104,8 @@ exec "$@"\n' > /usr/bin/nosync && \
                        pkgconfig \
                        pulseaudio-libs-devel \
                        python3 \
+                       python3-numpy \
+                       python3-opencv \
                        python3-pip \
                        python3-setuptools \
                        python3-sphinx \
tests/docker/dockerfiles/fedora-win64-cross.docker +2 -0
index 6f972f61738..3ed3e15df33 100644
@@ -44,6 +44,8 @@ exec "$@"\n' > /usr/bin/nosync && \
                        ninja-build \
                        openssh-clients \
                        python3 \
+                       python3-numpy \
+                       python3-opencv \
                        python3-pip \
                        python3-setuptools \
                        python3-sphinx \
tests/docker/dockerfiles/fedora.docker +2 -0
index 99bb38a3665..8fa8bea1abf 100644
@@ -104,6 +104,8 @@ exec "$@"\n' > /usr/bin/nosync && \
                        pkgconfig \
                        pulseaudio-libs-devel \
                        python3 \
+                       python3-numpy \
+                       python3-opencv \
                        python3-pip \
                        python3-setuptools \
                        python3-sphinx \
tests/docker/dockerfiles/opensuse-leap.docker +2 -0
index 8d8631bbcf0..3174c2e3cc2 100644
@@ -95,6 +95,8 @@ RUN zypper update -y && \
            pkgconfig \
            python3-Sphinx \
            python3-base \
+           python3-numpy \
+           python3-opencv \
            python3-pip \
            python3-setuptools \
            python3-sphinx_rtd_theme \
tests/docker/dockerfiles/ubuntu2404.docker +2 -0
index a5c22231f98..0918622e53d 100644
@@ -113,6 +113,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       openssh-client \
                       pkgconf \
                       python3 \
+                      python3-numpy \
+                      python3-opencv \
                       python3-pip \
                       python3-setuptools \
                       python3-sphinx \
.gitlab-ci.d/macos-14.vars +1 -1
index 0c187624dd0..f96e173f663 100644
@@ -12,5 +12,5 @@ NINJA='/opt/homebrew/bin/ninja'
 PACKAGING_COMMAND='brew'
 PIP3='/opt/homebrew/bin/pip3'
 PKGS='bash bc bindgen bison bzip2 capstone ccache cmocka coreutils ctags curl dbus diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 gtk-vnc jemalloc jpeg-turbo json-c libcbor libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson mtools ncurses nettle ninja pixman pkg-config python-setuptools python3 rpm2cpio rust sdl2 sdl2_image snappy socat sparse spice-protocol swtpm tesseract usbredir vde vte3 vulkan-tools xorriso zlib zstd'
-PYPI_PKGS='sphinx sphinx-rtd-theme tomli'
+PYPI_PKGS='numpy sphinx sphinx-rtd-theme tomli'
 PYTHON='/opt/homebrew/bin/python3'
scripts/ci/setup/debian/debian-13-ppc64le.yaml +2 -0
index 9acd99a7433..918deeb6ad7 100644
@@ -105,6 +105,8 @@ packages:
   - openssh-client
   - pkgconf
   - python3
+  - python3-numpy
+  - python3-opencv
   - python3-pip
   - python3-setuptools
   - python3-sphinx
scripts/ci/setup/ubuntu/ubuntu-2404-aarch64.yaml +2 -0
index 5c948109df8..38da57519d6 100644
@@ -106,6 +106,8 @@ packages:
   - openssh-client
   - pkgconf
   - python3
+  - python3-numpy
+  - python3-opencv
   - python3-pip
   - python3-setuptools
   - python3-sphinx
scripts/ci/setup/ubuntu/ubuntu-2404-s390x.yaml +2 -0
index 010b7f86aab..bae71c1cbc6 100644
@@ -104,6 +104,8 @@ packages:
   - openssh-client
   - pkgconf
   - python3
+  - python3-numpy
+  - python3-opencv
   - python3-pip
   - python3-setuptools
   - python3-sphinx
tests/lcitool/mappings.yml +10 -0
index b576ded3edc..9bc87b4879f 100644
@@ -22,6 +22,9 @@ mappings:
     CentOSStream9:
     FreeBSD: py312-sphinx_rtd_theme
 
+  python3-opencv:
+    CentOSStream9:
+
   python3-venv:
     CentOSStream9: python3.11
 
@@ -30,6 +33,7 @@ mappings:
     FreeBSD: py312-wheel
 
   python3-numpy:
+    CentOSStream9:
     FreeBSD: py312-numpy
 
   meson:
@@ -60,3 +64,9 @@ pypi_mappings:
     apk:
     Fedora:
     Debian12:
+
+  python3-numpy:
+    CentOSStream9: numpy
+
+  python3-opencv:
+    CentOSStream9: opencv-python
tests/lcitool/projects/qemu.yml +2 -0
index b3c99936723..abbd3e3f0b8 100644
@@ -91,6 +91,8 @@ packages:
  - pkg-config
  - pulseaudio
  - python3
+ - python3-numpy
+ - python3-opencv
  - python3-pip
  - python3-setuptools
  - python3-sphinx
tests/vm/generated/freebsd.json +2 -0
index ee3402f0d2b..c35591b39d6 100644
@@ -49,9 +49,11 @@
     "ncurses",
     "nettle",
     "ninja",
+    "opencv",
     "pixman",
     "pkgconf",
     "png",
+    "py312-numpy",
     "py312-pip",
     "py312-setuptools",
     "py312-sphinx",
Signature · 1 line
2.55.0
author
#16 lore raw

PATCH v2 6/16 tests: add Python3.12 to centos9.docker

In reply to John Snow

While we recently upgraded to Python 3.11 for the CentOS 9 container, upgrade it even further to Python 3.12.

---
 tests/docker/dockerfiles/centos9.docker   | 12 ++++++------
 tests/lcitool/mappings.yml                | 10 +++++-----
 tests/lcitool/targets/centos-stream-9.yml |  4 ++--
 3 files changed, 13 insertions(+), 13 deletions(-)
tests/docker/dockerfiles/centos9.docker +6 -6
index 6f5c400c2db..8bfce395f9d 100644
@@ -96,10 +96,10 @@ RUN dnf --quiet distro-sync -y && \
                 pkgconfig \
                 pulseaudio-libs-devel \
                 python3-tomli \
-                python3.11 \
-                python3.11-pip \
-                python3.11-setuptools \
-                python3.11-wheel \
+                python3.12 \
+                python3.12-pip \
+                python3.12-setuptools \
+                python3.12-wheel \
                 rdma-core-devel \
                 rust \
                 rust-std-static \
@@ -129,7 +129,7 @@ RUN dnf --quiet distro-sync -y && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-RUN /usr/bin/pip3.11 install \
+RUN /usr/bin/pip3.12 install \
                      numpy \
                      opencv-python \
                      sphinx \
@@ -139,7 +139,7 @@ ENV CCACHE_WRAPPERSDIR="/usr/libexec/ccache-wrappers"
 ENV LANG="en_US.UTF-8"
 ENV MAKE="/usr/bin/make"
 ENV NINJA="/usr/bin/ninja"
-ENV PYTHON="/usr/bin/python3.11"
+ENV PYTHON="/usr/bin/python3.12"
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
tests/lcitool/mappings.yml +5 -5
index 9bc87b4879f..321f6bc5926 100644
@@ -4,14 +4,14 @@ mappings:
     Ubuntu2204:
 
   python3:
-    CentOSStream9: python3.11
+    CentOSStream9: python3.12
 
   python3-pip:
-    CentOSStream9: python3.11-pip
+    CentOSStream9: python3.12-pip
     FreeBSD: py312-pip
 
   python3-setuptools:
-    CentOSStream9: python3.11-setuptools
+    CentOSStream9: python3.12-setuptools
     FreeBSD: py312-setuptools
 
   python3-sphinx:
@@ -26,10 +26,10 @@ mappings:
     CentOSStream9:
 
   python3-venv:
-    CentOSStream9: python3.11
+    CentOSStream9: python3.12
 
   python3-wheel:
-    CentOSStream9: python3.11-wheel
+    CentOSStream9: python3.12-wheel
     FreeBSD: py312-wheel
 
   python3-numpy:
tests/lcitool/targets/centos-stream-9.yml +2 -2
index c2d87f6cb4d..0d6801fd9a3 100644
@@ -1,3 +1,3 @@
 paths:
-  pip3: /usr/bin/pip3.11
-  python: /usr/bin/python3.11
+  pip3: /usr/bin/pip3.12
+  python: /usr/bin/python3.12
Signature · 1 line
2.55.0
Reviewed-by on v2 6/16
#17 lore raw

Re: [PATCH v2 06/16] tests: add Python3.12 to centos9.docker

Quoted 9 lines · John Snow

While we recently upgraded to Python 3.11 for the CentOS 9 container, upgrade it even further to Python 3.12.

---
 tests/docker/dockerfiles/centos9.docker   | 12 ++++++------
 tests/lcitool/mappings.yml                | 10 +++++-----
 tests/lcitool/targets/centos-stream-9.yml |  4 ++--
 3 files changed, 13 insertions(+), 13 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#18 lore raw

PATCH v2 7/16 tests: update emsdk-wasm64-cross

In reply to John Snow

This container currently uses Ubuntu 22.04 as a base, which is no longer supported by QEMU as a build platform. Upgrade to a newer version of the image which will provide Python3.12+.

---
 tests/docker/dockerfiles/emsdk-wasm64-cross.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/docker/dockerfiles/emsdk-wasm64-cross.docker +2 -2
index c04a741b0e5..f11a9f34244 100644
@@ -1,6 +1,6 @@
 # syntax = docker/dockerfile:1.5
 
-ARG EMSDK_VERSION_QEMU=4.0.10
+ARG EMSDK_VERSION_QEMU=5.0.2
 ARG ZLIB_VERSION=1.3.2
 ARG GLIB_MINOR_VERSION=2.84
 ARG GLIB_VERSION=${GLIB_MINOR_VERSION}.0
@@ -27,7 +27,7 @@ RUN apt-get update && apt-get install -y \
     pkgconf \
     ninja-build \
     python3-pip
-RUN pip3 install meson==${MESON_VERSION} tomli
+RUN pip3 install --break-system-packages meson==${MESON_VERSION} tomli
 RUN mkdir /build
 WORKDIR /build
 RUN mkdir -p $TARGET
Signature · 1 line
2.55.0
author
#19 lore raw

PATCH v2 8/16 qapi: update type hint for objects_seen

In reply to John Snow

In preparation for upgrading Python to 3.12+, adjust the type hint on objects_seen to prevent mypy from chirping about an unknown type for this variable.

Why didn't this trigger earlier? A mystery for the ages.

---
 scripts/qapi/types.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/types.py +1 -1
index 2bf75338283..b87377c6474 100644
@@ -34,7 +34,7 @@
 
 # variants must be emitted before their container; track what has already
 # been output
-objects_seen = set()
+objects_seen: set[str] = set()
 
 
 def gen_enum_lookup(name: str,
Signature · 1 line
2.55.0
author
#20 lore raw

PATCH v2 9/16 python: upgrade to Python3.12+

In reply to John Snow

With Python3.9 EOL and Python3.10 soon to be EOL (October 2026), and with ... no other supported build platform actually using Python 3.11, require the use of Python3.12 or later.

Additionally, as part of this upgrade, remove the use of tomllib from mkvenv as it is now standard in the Python stdlib: leaving this transitional code in causes a regression with mypy which becomes confused about the overloaded use of 'tomllib'.

Our current minimum requirements as determined by standard repository packages on supported distributions is as follows:

distro EOS python3 pip setuptools sphinx wheel
-------------------------------------------------------------------------

centos_stream_9   2026-12-12  3.9.25   21.3.1  53.0.0      3.4.3   0.36.2
centos_stream_9+  2026-12-12  3.12.13  23.2.1  68.2.2      ---     0.41.2
freebsd           --          3.12.14  23.3.2  63.1.0      9.0.4   0.48.0
ubuntu_24_04      2028-04-23  3.12.3   24.0    68.1.2      7.2.6   0.42.0
centos_stream_10  2030-01-01  3.12.13  23.3.2  69.0.3      7.2.6   0.41.2
alpine_3_21       2026-11-01  3.12.14  24.3.1  70.3.0      8.1.3   0.43.0
debian_13         2028-08-09  3.13.5   25.1.1  78.1.1      8.1.3   0.46.1
fedora_43         2026-12-09  3.14.7   25.1.1  78.1.1      8.2.3   0.45.1
ubuntu_26_04      2031-05-29  3.14.4   25.1.1  78.1.1      8.2.3   0.46.3
fedora_44         2027-06-02  3.14.7   26.0.1  80.10.2     8.2.3   0.45.1
macports          --          3.14.7   26.0.1  80.10.2     9.1.0   0.48.0
alpine_3_22       2027-05-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
alpine_3_23       2027-11-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
openbsd           --          3.14.6   26.2.1  80.9.0      9.1.0   0.48.0
alpine_3_24       2028-06-01  3.14.7   26.1.2  82.0.1      9.1.0   0.47.0
pkgsrc_current    --          3.13.15  26.2.1  84.0.0      9.1.0   0.48.0

CentOS stream 9 note: python3.12, python3.12-pip, python3.12-setuptools and python3.12-wheel are available as optional packages in the standard repository. Sphinx is not available, but can be side-loaded from PyPI using pip3.12.

---
 configure                | 11 ++++---
 python/Makefile          | 10 +++----
 python/scripts/mkvenv.py | 63 ++++------------------------------------
 python/setup.cfg         | 17 ++++-------
 python/tests/minreqs.txt | 41 ++++++++++++--------------
 pythondeps.toml          |  9 ++----
 6 files changed, 43 insertions(+), 108 deletions(-)
configure +5 -6
index 56158cd95f7..46cbcc085b3 100755
@@ -503,17 +503,16 @@ if test -n "$linux_arch" && ! test -d "$source_path/linux-headers/asm-$linux_arc
 fi
 
 check_py_version() {
-    # We require python >= 3.9.
+    # We require python >= 3.12.
     # NB: a True python conditional creates a non-zero return code (Failure)
-    "$1" -c 'import sys; sys.exit(sys.version_info < (3,9))'
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,12))'
 }
 
 first_python=
 if test -z "${PYTHON}"; then
     # A bare 'python' is traditionally python 2.x, but some distros
     # have it as python 3.x, so check in both places.
-    for binary in python3 python python3.14 python3.13 python3.12 \
-                          python3.11 python3.10 python3.9 ; do
+    for binary in python3 python python3.14 python3.13 python3.12; do
         if has "$binary"; then
             python=$(command -v "$binary")
             if check_py_version "$python"; then
@@ -902,7 +901,7 @@ then
     # If first_python is set, there was a binary somewhere even though
     # it was not suitable.  Use it for the error message.
     if test -n "$first_python"; then
-        error_exit "Cannot use '$first_python', Python >= 3.9 is required." \
+        error_exit "Cannot use '$first_python', Python >= 3.12 is required." \
             "Use --python=/path/to/python to specify a supported Python."
     else
         error_exit "Python not found. Use --python=/path/to/python"
@@ -910,7 +909,7 @@ then
 fi
 
 if ! check_py_version "$python"; then
-  error_exit "Cannot use '$python', Python >= 3.9 is required." \
+  error_exit "Cannot use '$python', Python >= 3.12 is required." \
              "Use --python=/path/to/python to specify a supported Python." \
              "Maybe try:" \
              "  CentOS: dnf install python3.12"
python/Makefile +5 -5
index 42994d39618..7868a48a232 100644
@@ -9,14 +9,14 @@ help:
 	@echo "make check-minreqs:"
 	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.9"
-	@echo "    Hint (Fedora): 'sudo dnf install python3.9'"
+	@echo "    Requires: Python 3.12"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.12'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
 	@echo "    These tests use the newest dependencies."
-	@echo "    Requires: Python 3.9 - 3.11, and tox."
-	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.11'"
+	@echo "    Requires: Python 3.12 - 3.14, and tox."
+	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.14'"
 	@echo "    The variable QEMU_TOX_EXTRA_ARGS can be use to pass extra"
 	@echo "    arguments to tox".
 	@echo ""
@@ -59,7 +59,7 @@ PIP_INSTALL = pip install --disable-pip-version-check
 min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
 $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
 	@echo "VENV $(QEMU_MINVENV_DIR)"
-	@python3.9 -m venv $(QEMU_MINVENV_DIR)
+	@python3.12 -m venv $(QEMU_MINVENV_DIR)
 	@(								\
 		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
 		. $(QEMU_MINVENV_DIR)/bin/activate;			\
python/scripts/mkvenv.py +5 -58

93 lines not on this page · raw message

python/setup.cfg +6 -11
index ad4121a58d8..8f62b7c1cdb 100644
@@ -14,16 +14,13 @@ classifiers =
     Natural Language :: English
     Operating System :: OS Independent
     Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.9
-    Programming Language :: Python :: 3.10
-    Programming Language :: Python :: 3.11
     Programming Language :: Python :: 3.12
     Programming Language :: Python :: 3.13
     Programming Language :: Python :: 3.14
     Typing :: Typed
 
 [options]
-python_requires = >= 3.9
+python_requires = >= 3.12
 packages =
     qemu.machine
     qemu.utils
@@ -39,12 +36,10 @@ devel =
     distlib >= 0.3.6
     flake8 >= 5.0.4
     fusepy >= 2.0.4
-    isort >= 5.6.0
-    # Max ver is temporary until we drop 3.9
-    mypy >= 1.4.0, <2.0.0
+    isort >= 5.11.3
+    mypy >= 1.4.0
     pylint >= 2.17.3
-    pylint != 3.2.4; python_version<"3.9"
-    pytest >= 6.0.2
+    pytest >= 6.2.4
     tox >= 3.18.0
     sphinx >= 3.4.3
 
@@ -69,7 +64,7 @@ exclude = __pycache__,
 
 [mypy]
 strict = True
-python_version = 3.9
+python_version = 3.12
 namespace_packages = True
 warn_unused_ignores = False
 
@@ -164,7 +159,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py39, py310, py311, py312, py313, py314
+envlist = py312, py313, py314
 skip_missing_interpreters = true
 
 [testenv]
python/tests/minreqs.txt +19 -22

84 lines not on this page · raw message

pythondeps.toml +3 -6
index 061f0cff0c4..b1879c1a4da 100644
@@ -38,14 +38,11 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.2.2" }
 [tooling]
 "qemu.qmp" = { accepted = ">=0.0.5", installed = "0.0.6" }
 "qemu" = { path = "python/" }
-# NB: The following dependencies should be a little bit more modern than
-# the versions listed here, but we are still using Debian 11 for several
-# GitLab CI tests, so we are further restricted.
-"setuptools" = { accepted = ">=44.1.1" }
-"wheel" = { accepted = ">=0.34.2" }
+"setuptools" = { accepted = ">=63.1.0" }
+"wheel" = { accepted = ">=0.41.2" }
 # pip should be guaranteed by mkvenv, this is merely a sanity check for
 # which version we are counting on being present.
-"pip" = { accepted = ">=20.3.4" }
+"pip" = { accepted = ">=23.2.1" }
 
 # This test group is for functional tests, and can include dependencies
 # fetched from PyPI.
Signature · 1 line
2.55.0
Reviewed-by on v2 9/16
#21 lore raw

Re: [PATCH v2 09/16] python: upgrade to Python3.12+

Quoted 37 lines · John Snow

With Python3.9 EOL and Python3.10 soon to be EOL (October 2026), and with ... no other supported build platform actually using Python 3.11, require the use of Python3.12 or later.

Additionally, as part of this upgrade, remove the use of tomllib from mkvenv as it is now standard in the Python stdlib: leaving this transitional code in causes a regression with mypy which becomes confused about the overloaded use of 'tomllib'.

Our current minimum requirements as determined by standard repository packages on supported distributions is as follows:

distro EOS python3 pip setuptools sphinx wheel
-------------------------------------------------------------------------

centos_stream_9   2026-12-12  3.9.25   21.3.1  53.0.0      3.4.3   0.36.2
centos_stream_9+  2026-12-12  3.12.13  23.2.1  68.2.2      ---     0.41.2
freebsd           --          3.12.14  23.3.2  63.1.0      9.0.4   0.48.0
ubuntu_24_04      2028-04-23  3.12.3   24.0    68.1.2      7.2.6   0.42.0
centos_stream_10  2030-01-01  3.12.13  23.3.2  69.0.3      7.2.6   0.41.2
alpine_3_21       2026-11-01  3.12.14  24.3.1  70.3.0      8.1.3   0.43.0
debian_13         2028-08-09  3.13.5   25.1.1  78.1.1      8.1.3   0.46.1
fedora_43         2026-12-09  3.14.7   25.1.1  78.1.1      8.2.3   0.45.1
ubuntu_26_04      2031-05-29  3.14.4   25.1.1  78.1.1      8.2.3   0.46.3
fedora_44         2027-06-02  3.14.7   26.0.1  80.10.2     8.2.3   0.45.1
macports          --          3.14.7   26.0.1  80.10.2     9.1.0   0.48.0
alpine_3_22       2027-05-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
alpine_3_23       2027-11-01  3.12.14  25.1.1  80.9.0      8.2.3   0.46.3
openbsd           --          3.14.6   26.2.1  80.9.0      9.1.0   0.48.0
alpine_3_24       2028-06-01  3.14.7   26.1.2  82.0.1      9.1.0   0.47.0
pkgsrc_current    --          3.13.15  26.2.1  84.0.0      9.1.0   0.48.0

CentOS stream 9 note: python3.12, python3.12-pip, python3.12-setuptools and python3.12-wheel are available as optional packages in the standard repository. Sphinx is not available, but can be side-loaded from PyPI using pip3.12.

Quoted 8 lines
---
 configure                | 11 ++++---
 python/Makefile          | 10 +++----
 python/scripts/mkvenv.py | 63 ++++------------------------------------
 python/setup.cfg         | 17 ++++-------
 python/tests/minreqs.txt | 41 ++++++++++++--------------
 pythondeps.toml          |  9 ++----
 6 files changed, 43 insertions(+), 108 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author
#22 lore raw

PATCH v2 10/16 python: fix license configuration

In reply to John Snow

License trove identifiers were deprecated in favor of SPDX identifiers, correct this to stop some whining from various python tooling.

---
 python/setup.cfg | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
python/setup.cfg +1 -1
index 8f62b7c1cdb..a22abba6c36 100644
@@ -8,9 +8,9 @@ download_url = https://www.qemu.org/download/
 description = QEMU Python Build, Debug and SDK tooling.
 long_description = file:PACKAGE.rst
 long_description_content_type = text/x-rst
+license = GPL-2.0-or-later
 classifiers =
     Development Status :: 3 - Alpha
-    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
     Natural Language :: English
     Operating System :: OS Independent
     Programming Language :: Python :: 3 :: Only
Signature · 1 line
2.55.0
author
#23 lore raw

PATCH v2 11/16 python: allow mypy to run as the current python version

In reply to John Snow

Currently, we instruct mypy to run assuming a specific version of python. Lift this restriction so that under tox tests, it will always run assuming the current running version.

---
 python/setup.cfg | 1 -
 1 file changed, 1 deletion(-)
python/setup.cfg +0 -1
index a22abba6c36..f866f659574 100644
@@ -64,7 +64,6 @@ exclude = __pycache__,
 
 [mypy]
 strict = True
-python_version = 3.12
 namespace_packages = True
 warn_unused_ignores = False
 
Signature · 1 line
2.55.0
author
#24 lore raw

PATCH v2 12/16 python: remove distutils workarounds

In reply to John Snow

These are no longer necessary now that we are using Python3.12+.

---
 python/tests/linters.py | 11 -----------
 1 file changed, 11 deletions(-)
python/tests/linters.py +0 -11
index 82b88aebb20..dd89fb00ba8 100644
@@ -83,23 +83,13 @@ def test_mypy_iotests(self):
             cwd="../tests/qemu-iotests/",
         )
 
-    # Setuptools v60 introduced the SETUPTOOLS_USE_DISTUTILS=stdlib
-    # workaround; stdlib distutils was fully removed in Python
-    # 3.12+. Once we are on >=3.12+ exclusively, this workaround can be
-    # dropped safely. Until then, it is needed for some versions on
-    # Fedora/Debian distributions which relied upon distro-patched
-    # setuptools present in CPython, but not within setuptools itself.
-
     def test_pylint_pkg(self):
-        os.environ["SETUPTOOLS_USE_DISTUTILS"] = "stdlib"
         check_call([sys.executable, "-m", "pylint", "qemu/"])
 
     def test_pylint_scripts(self):
-        os.environ["SETUPTOOLS_USE_DISTUTILS"] = "stdlib"
         check_call([sys.executable, "-m", "pylint", "scripts/"])
 
     def test_pylint_qapi(self):
-        os.environ["SETUPTOOLS_USE_DISTUTILS"] = "stdlib"
         check_call(
             [
                 sys.executable,
@@ -113,7 +103,6 @@ def test_pylint_qapi(self):
         )
 
     def test_pylint_iotests(self):
-        os.environ["SETUPTOOLS_USE_DISTUTILS"] = "stdlib"
         check_call(
             [sys.executable, "-m", "linters", "--pylint"],
             cwd="../tests/qemu-iotests/",
Signature · 1 line
2.55.0
author
#25 lore raw

PATCH v2 13/16 python: remove stale setup.cfg options

In reply to John Snow

With the recent version bumps, we no longer need these directives.

---
 python/setup.cfg | 13 -------------
 1 file changed, 13 deletions(-)
python/setup.cfg +0 -13
index f866f659574..ebb4230899c 100644
@@ -65,7 +65,6 @@ exclude = __pycache__,
 [mypy]
 strict = True
 namespace_packages = True
-warn_unused_ignores = False
 
 [mypy-qemu.utils.qom_fuse]
 # fusepy has no type stubs:
@@ -76,12 +75,6 @@ allow_subclassing_any = True
 [mypy-fuse]
 ignore_missing_imports = True
 
-[mypy-tomli]
-ignore_missing_imports = True
-
-[mypy-tomllib]
-ignore_missing_imports = True
-
 [mypy-distlib]
 ignore_missing_imports = True
 
@@ -91,12 +84,6 @@ ignore_missing_imports = True
 [mypy-distlib.version]
 ignore_missing_imports = True
 
-[mypy-pip._vendor.distlib]
-ignore_missing_imports = True
-
-[mypy-pip._vendor.distlib.scripts]
-ignore_missing_imports = True
-
 [mypy-pip._vendor.distlib.version]
 ignore_missing_imports = True
 
Signature · 1 line
2.55.0
author
#26 lore raw

PATCH v2 14/16 sphinx: require sphinx 7.2.6

In reply to John Snow

Make the extraordinary leap from Sphinx 3.4.3 to Sphinx 7.2.6. Simplifications and cleanup will follow in subsequent patches.

Note that CentOS Stream 9 only ships a version of Sphinx that runs on Python 3.9, so regardless of the version of Sphinx we require, if we require Python 3.10+, CentOS Stream 9 will be unable to build QEMU documentation in an isolated offline build environment.

CentOS Stream 9 continues to be able to build documentation so long as PyPI access is permitted during configure, or if the requisite PyPI packages are installed prior to configure time.

---
 docs/conf.py             |  4 ++--
 docs/requirements.txt    |  2 +-
 python/setup.cfg         |  2 +-
 python/tests/minreqs.txt | 29 ++++++++++++++++-------------
 pythondeps.toml          |  4 ++--
 5 files changed, 22 insertions(+), 19 deletions(-)
docs/conf.py +2 -2
index 7e35d2158d3..cded047b539 100644
@@ -51,9 +51,9 @@
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #
-# 3.4.3 is the oldest version of Sphinx that ships on a platform we
+# 7.2.6 is the oldest version of Sphinx that ships on a platform we
 # pledge build support for.
-needs_sphinx = '3.4.3'
+needs_sphinx = '7.2.6'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
docs/requirements.txt +1 -1
index 87f7afcb2e7..c10c4cf7c91 100644
@@ -1,5 +1,5 @@
 # Used by readthedocs.io
 # Should be in sync with the "installed" key of pythondeps.toml
 
-sphinx==6.2.1
+sphinx==7.2.6
 sphinx_rtd_theme==1.2.2
python/setup.cfg +1 -1
index ebb4230899c..927f72a46f6 100644
@@ -41,7 +41,7 @@ devel =
     pylint >= 2.17.3
     pytest >= 6.2.4
     tox >= 3.18.0
-    sphinx >= 3.4.3
+    sphinx >= 7.2.6
 
 # Provides qom-fuse functionality
 fuse =
python/tests/minreqs.txt +16 -13
index 85cfa08cf76..7154ad40040 100644
@@ -12,7 +12,7 @@
 # on PyPI that allows the test suite to pass.
 
 # Dependencies for qapidoc/qapi_domain et al
-sphinx==3.4.3
+sphinx==7.2.6
 
 # Dependencies for qemu.machine
 qemu.qmp==0.0.6
@@ -48,23 +48,26 @@ platformdirs==2.2.0
 tomlkit==0.10.1
 
 # Transitive sphinx dependencies
-Jinja2==2.7
-MarkupSafe==1.1.0
 alabaster==0.7.1
-babel==1.3
-docutils==0.12
-imagesize==0.5.0
-packaging==14.0
-Pygments==2.0
-pytz==2014.9
-requests==2.5.0
-snowballstemmer==1.1
+babel==2.9
+certifi==2023.5.7
+charset-normalizer==2.0.0
+docutils==0.18.1
+idna==2.5
+imagesize==1.3
+Jinja2==3.0.0
+MarkupSafe==2.0
+packaging==21.0
+Pygments==2.14
+requests==2.25.0
+snowballstemmer==2.0
 sphinxcontrib-applehelp==1.0.0
 sphinxcontrib-devhelp==1.0.0
-sphinxcontrib-htmlhelp==1.0.0
+sphinxcontrib-htmlhelp==2.0.0
 sphinxcontrib-jsmath==1.0.0
 sphinxcontrib-qthelp==1.0.0
-sphinxcontrib-serializinghtml==1.0.0
+sphinxcontrib-serializinghtml==1.1.9
+urllib3==1.26
 
 # Transitive pytest dependencies
 attrs==19.2.0
pythondeps.toml +2 -2
index b1879c1a4da..f0f3af915a1 100644
@@ -28,8 +28,8 @@ meson = { accepted = ">=1.12.0", installed = "1.12.0", canary = "meson" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=3.4.3", installed = "6.2.1", canary = "sphinx-build" }
-sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.2.2" }
+sphinx = { accepted = ">=7.2.6", installed = "7.2.6", canary = "sphinx-build" }
+sphinx_rtd_theme = { accepted = ">=1.2.2", installed = "1.2.2" }
 
 # This test group is for dependencies required to run various tooling
 # and tests that should always be installed at configure time. It should
Signature · 1 line
2.55.0
author
#27 lore raw

PATCH v2 15/16 sphinx: drop fakedbusdoc

In reply to John Snow

Now that we rely on a sufficiently modern Sphinx, we no longer need a stub doc for dbus.

---
 docs/conf.py                  |  7 +------
 docs/interop/dbus-display.rst | 14 +-------------
 docs/interop/dbus-vmstate.rst | 14 +-------------
 docs/interop/dbus-vnc.rst     | 14 +-------------
 docs/sphinx/fakedbusdoc.py    | 30 ------------------------------
 5 files changed, 4 insertions(+), 75 deletions(-)
 delete mode 100644 docs/sphinx/fakedbusdoc.py
docs/conf.py +1 -6
index cded047b539..914bae67d9c 100644
@@ -59,6 +59,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'dbusdoc',
     'depfile',
     'hxtool',
     'kerneldoc',
@@ -67,12 +68,6 @@
     'qmp_lexer',
 ]
 
-if sphinx.version_info[:3] > (4, 0, 0):
-    tags.add('sphinx4')
-    extensions += ['dbusdoc']
-else:
-    extensions += ['fakedbusdoc']
-
 # Add any paths that contain templates here, relative to this directory.
 templates_path = [os.path.join(qemu_docdir, '_templates')]
 
docs/interop/dbus-display.rst +1 -13
index 87648e91dc0..c87a04c9693 100644
@@ -18,16 +18,4 @@ QEMU also implements the standard interfaces, such as
    :local:
    :depth: 1
 
-.. only:: sphinx4
-
-   .. dbus-doc:: ui/dbus-display1.xml
-
-.. only:: not sphinx4
-
-   .. warning::
-      Sphinx 4 is required to build D-Bus documentation.
-
-      This is the content of ``ui/dbus-display1.xml``:
-
-   .. literalinclude:: ../../ui/dbus-display1.xml
-      :language: xml
+.. dbus-doc:: ui/dbus-display1.xml
docs/interop/dbus-vmstate.rst +1 -13
index 5fb3f279e28..5f8c6d125de 100644
@@ -23,16 +23,4 @@ dbus-vmstate object can be configured with the expected list of
 helpers by setting its ``id-list`` property, with a comma-separated
 ``Id`` list.
 
-.. only:: sphinx4
-
-   .. dbus-doc:: backends/dbus-vmstate1.xml
-
-.. only:: not sphinx4
-
-   .. warning::
-      Sphinx 4 is required to build D-Bus documentation.
-
-      This is the content of ``backends/dbus-vmstate1.xml``:
-
-   .. literalinclude:: ../../backends/dbus-vmstate1.xml
-      :language: xml
+.. dbus-doc:: backends/dbus-vmstate1.xml
docs/interop/dbus-vnc.rst +1 -13
index d2b77978f63..5ea4fd57183 100644
@@ -11,16 +11,4 @@ Objects are exported under ``/org/qemu/Vnc1/``.
    :local:
    :depth: 1
 
-.. only:: sphinx4
-
-   .. dbus-doc:: tools/qemu-vnc/qemu-vnc1.xml
-
-.. only:: not sphinx4
-
-   .. warning::
-      Sphinx 4 is required to build D-Bus documentation.
-
-      This is the content of ``tools/qemu-vnc/qemu-vnc1.xml``:
-
-   .. literalinclude:: ../../tools/qemu-vnc/qemu-vnc1.xml
-      :language: xml
+.. dbus-doc:: tools/qemu-vnc/qemu-vnc1.xml
docs/sphinx/fakedbusdoc.py deleted +0 -30
deleted file mode 100644
index 2d2e6ef6403..00000000000
@@ -1,30 +0,0 @@
-# D-Bus XML documentation extension, compatibility gunk for <sphinx4
-#
-# Copyright (C) 2021, Red Hat Inc.
-#
-# SPDX-License-Identifier: LGPL-2.1-or-later
-#
-# Author: Marc-André Lureau <[email protected]>
-"""dbus-doc is a Sphinx extension that provides documentation from D-Bus XML."""
-
-from docutils.parsers.rst import Directive
-from sphinx.application import Sphinx
-from typing import Any, Dict
-
-
-class FakeDBusDocDirective(Directive):
-    has_content = True
-    required_arguments = 1
-
-    def run(self):
-        return []
-
-
-def setup(app: Sphinx) -> Dict[str, Any]:
-    """Register a fake dbus-doc directive with Sphinx"""
-    app.add_directive("dbus-doc", FakeDBusDocDirective)
-
-    return dict(
-        parallel_read_safe = True,
-        parallel_write_safe = True
-    )
Signature · 1 line
2.55.0
author
#28 lore raw

PATCH v2 16/16 sphinx: drop qapidoc back-compat overboiled spaghetti dinner

In reply to John Snow

Drop the disgusting amount of hacks required to keep qapidoc working on older Sphinx versions, enjoying the cutting edge technology of 2023.

This felt like an achievement to have written, and equally feels an achievement to have deleted.

A note on the content_node parameter: Sphinx has been gradually implementing typing support and renaming parameters to be consistent. As a result of losing our intermediate compatibility shim, mypy and pylint now check directly against the Sphinx version. So, alongside removing compat gunk, we need to fix up some parameter naming for the Sphinx upgrade, too.

---
 docs/sphinx/compat.py      | 230 -------------------------------------
 docs/sphinx/qapi_domain.py |  53 +++++----
 2 files changed, 28 insertions(+), 255 deletions(-)
 delete mode 100644 docs/sphinx/compat.py
docs/sphinx/compat.py deleted +0 -230
deleted file mode 100644
index 9cf7fe006e4..00000000000
@@ -1,230 +0,0 @@
-"""
-Sphinx cross-version compatibility goop
-"""
-
-import re
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Callable,
-    Optional,
-    Type,
-)
-
-from docutils import nodes
-from docutils.nodes import Element, Node, Text
-from docutils.statemachine import StringList
-
-import sphinx
-from sphinx import addnodes, util
-from sphinx.directives import ObjectDescription
-from sphinx.environment import BuildEnvironment
-from sphinx.roles import XRefRole
-from sphinx.util import docfields
-from sphinx.util.docutils import (
-    ReferenceRole,
-    SphinxDirective,
-    switch_source_input,
-)
-from sphinx.util.typing import TextlikeNode
-
-
-MAKE_XREF_WORKAROUND = sphinx.version_info[:3] < (4, 1, 0)
-
-
-SpaceNode: Callable[[str], Node]
-KeywordNode: Callable[[str, str], Node]
-
-if sphinx.version_info[:3] >= (4, 0, 0):
-    SpaceNode = addnodes.desc_sig_space
-    KeywordNode = addnodes.desc_sig_keyword
-else:
-    SpaceNode = Text
-    KeywordNode = addnodes.desc_annotation
-
-
-def nested_parse_with_titles(
-    directive: SphinxDirective, content_node: Element
-) -> None:
-    """
-    This helper preserves error parsing context across sphinx versions.
-    """
-
-    # necessary so that the child nodes get the right source/line set
-    content_node.document = directive.state.document
-
-    try:
-        # Modern sphinx (6.2.0+) supports proper offsetting for
-        # nested parse error context management
-        util.nodes.nested_parse_with_titles(
-            directive.state,
-            directive.content,
-            content_node,
-            content_offset=directive.content_offset,
-        )
-    except TypeError:
-        # No content_offset argument. Fall back to SSI method.
-        with switch_source_input(directive.state, directive.content):
-            util.nodes.nested_parse_with_titles(
-                directive.state, directive.content, content_node
-            )
-
-
-# ###########################################
-# xref compatibility hacks for Sphinx < 4.1 #
-# ###########################################
-
-# When we require >= Sphinx 4.1, the following function and the
-# subsequent 3 compatibility classes can be removed. Anywhere in
-# qapi_domain that uses one of these Compat* types can be switched to
-# using the garden-variety lib-provided classes with no trickery.
-
-
-def _compat_make_xref(  # pylint: disable=unused-argument
-    self: sphinx.util.docfields.Field,
-    rolename: str,
-    domain: str,
-    target: str,
-    innernode: Type[TextlikeNode] = addnodes.literal_emphasis,
-    contnode: Optional[Node] = None,
-    env: Optional[BuildEnvironment] = None,
-    inliner: Any = None,
-    location: Any = None,
-) -> Node:
-    """
-    Compatibility workaround for Sphinx versions prior to 4.1.0.
-
-    Older sphinx versions do not use the domain's XRefRole for parsing
-    and formatting cross-references, so we need to perform this magick
-    ourselves to avoid needing to write the parser/formatter in two
-    separate places.
-
-    This workaround isn't brick-for-brick compatible with modern Sphinx
-    versions, because we do not have access to the parent directive's
-    state during this parsing like we do in more modern versions.
-
-    It's no worse than what pre-Sphinx 4.1.0 does, so... oh well!
-    """
-
-    # Yes, this function is gross. Pre-4.1 support is a miracle.
-    # pylint: disable=too-many-locals
-
-    assert env
-    # Note: Sphinx's own code ignores the type warning here, too.
-    if not rolename:
-        return contnode or innernode(target, target)  # type: ignore[call-arg]
-
-    # Get the role instance, but don't *execute it* - we lack the
-    # correct state to do so. Instead, we'll just use its public
-    # methods to do our reference formatting, and emulate the rest.
-    role = env.get_domain(domain).roles[rolename]
-    assert isinstance(role, XRefRole)
-
-    # XRefRole features not supported by this compatibility shim;
-    # these were not supported in Sphinx 3.x either, so nothing of
-    # value is really lost.
-    assert not target.startswith("!")
-    assert not re.match(ReferenceRole.explicit_title_re, target)
-    assert not role.lowercase
-    assert not role.fix_parens
-
-    # Code below based mostly on sphinx.roles.XRefRole; run() and
-    # create_xref_node()
-    options = {
-        "refdoc": env.docname,
-        "refdomain": domain,
-        "reftype": rolename,
-        "refexplicit": False,
-        "refwarn": role.warn_dangling,
-    }
-    refnode = role.nodeclass(target, **options)
-    title, target = role.process_link(env, refnode, False, target, target)
-    refnode["reftarget"] = target
-    classes = ["xref", domain, f"{domain}-{rolename}"]
-    refnode += role.innernodeclass(target, title, classes=classes)
-
-    # This is the very gross part of the hack. Normally,
-    # result_nodes takes a document object to which we would pass
-    # self.inliner.document. Prior to Sphinx 4.1, we don't *have* an
-    # inliner to pass, so we have nothing to pass here. However, the
-    # actual implementation of role.result_nodes in this case
-    # doesn't actually use that argument, so this winds up being
-    # ... fine. Rest easy at night knowing this code only runs under
-    # old versions of Sphinx, so at least it won't change in the
-    # future on us and lead to surprising new failures.
-    # Gross, I know.
-    result_nodes, _messages = role.result_nodes(
-        None,  # type: ignore
-        env,
-        refnode,
-        is_ref=True,
-    )
-    return nodes.inline(target, "", *result_nodes)
-
-
-class CompatField(docfields.Field):
-    if MAKE_XREF_WORKAROUND:
-        make_xref = _compat_make_xref
-
-
-class CompatGroupedField(docfields.GroupedField):
-    if MAKE_XREF_WORKAROUND:
-        make_xref = _compat_make_xref
-
-
-class CompatTypedField(docfields.TypedField):
-    if MAKE_XREF_WORKAROUND:
-        make_xref = _compat_make_xref
-
-
-# ################################################################
-# Nested parsing error location fix for Sphinx 5.3.0 < x < 6.2.0 #
-# ################################################################
-
-# When we require Sphinx 4.x, the TYPE_CHECKING hack where we avoid
-# subscripting ObjectDescription at runtime can be removed in favor of
-# just always subscripting the class.
-
-# When we require Sphinx > 6.2.0, the rest of this compatibility hack
-# can be dropped and QAPIObject can just inherit directly from
-# ObjectDescription[Signature].
-
-SOURCE_LOCATION_FIX = (5, 3, 0) <= sphinx.version_info[:3] < (6, 2, 0)
-
-Signature = str
-
-
-if TYPE_CHECKING:
-    _BaseClass = ObjectDescription[Signature]
-else:
-    _BaseClass = ObjectDescription
-
-
-class ParserFix(_BaseClass):
-
-    _temp_content: StringList
-    _temp_offset: int
-    _temp_node: Optional[addnodes.desc_content]
-
-    def before_content(self) -> None:
-        # Work around a sphinx bug and parse the content ourselves.
-        self._temp_content = self.content
-        self._temp_offset = self.content_offset
-        self._temp_node = None
-
-        if SOURCE_LOCATION_FIX:
-            self._temp_node = addnodes.desc_content()
-            self.state.nested_parse(
-                self.content, self.content_offset, self._temp_node
-            )
-            # Sphinx will try to parse the content block itself,
-            # Give it nothingness to parse instead.
-            self.content = StringList()
-            self.content_offset = 0
-
-    def transform_content(self, content_node: addnodes.desc_content) -> None:
-        # Sphinx workaround: Inject our parsed content and restore state.
-        if self._temp_node:
-            content_node += self._temp_node.children
-            self.content = self._temp_content
-            self.content_offset = self._temp_offset
docs/sphinx/qapi_domain.py +28 -25

140 lines not on this page · raw message

Signature · 1 line
2.55.0
Reviewed-by on v2 16/16
#29 lore raw

Re: [PATCH v2 16/16] sphinx: drop qapidoc back-compat overboiled spaghetti dinner

Quoted 14 lines · John Snow

Drop the disgusting amount of hacks required to keep qapidoc working on older Sphinx versions, enjoying the cutting edge technology of 2023.

This felt like an achievement to have written, and equally feels an achievement to have deleted.

A note on the content_node parameter: Sphinx has been gradually implementing typing support and renaming parameters to be consistent. As a result of losing our intermediate compatibility shim, mypy and pylint now check directly against the Sphinx version. So, alongside removing compat gunk, we need to fix up some parameter naming for the Sphinx upgrade, too.

Quoted 2 lines

This repays the technical debt you had to take on to make the rewritten Sphinx QAPI doc extension work, which massively improved the rendering of QMP and QGA reference manuals. The impressive and ugly work goes, the impressive and lovely work stays. Thank you!