Skip to content

RFC configure: create a python venv and ensure meson, sphinx

John Snow qemu-devel 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 RFC Latest20 patches 25 repliesIn review
  2. v2 RFC10 patches 1 replySuperseded
  3. v1 RFC3 patches 14 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/20python: update pylint configurationpython/setup.cfg+1 -01 reply
2/20python: add mkvenv.pypython/scripts/mkvenv.py python/setup.cfg python/tests/flake8.sh +3 more+290 -00 replies
3/20mkvenv: add console script entry point generationpython/scripts/mkvenv.py+172 -70 replies
4/20mkvenv: Add better error message for missing pyexpat modulepython/scripts/mkvenv.py+17 -10 replies
5/20mkvenv: generate console entry shims from inside the venvpython/scripts/mkvenv.py+85 -140 replies
6/20mkvenv: work around broken pip installations on Debian 10python/scripts/mkvenv.py+52 -150 replies
7/20mkvenv: add nested venv workaroundpython/scripts/mkvenv.py+69 -30 replies
8/20mkvenv: add ensure subcommandpython/scripts/mkvenv.py+114 -20 replies
9/20tests/docker: add python3-venv dependencytests/docker/dockerfiles/debian-all-test-cross.docker tests/docker/dockerfiles/debian-hexagon-cross.docker tests/docker/dockerfiles/debian-riscv64-cross.docker +1 more+8 -41 reply
10/20tests/vm: Configure netbsd to use Python 3.10tests/vm/netbsd+1 -01 reply
11/20tests/vm: add py310-expat to NetBSDtests/vm/netbsd+1 -02 replies
12/20scripts/make-release: download meson==0.61.5 .whlscripts/make-release+11 -00 replies
13/20configure: create a python venv unconditionallyconfigure+31 -50 replies
14/20configure: use 'mkvenv ensure meson' to bootstrap meson.gitlab-ci.d/buildtest-template.yml configure+21 -553 replies
15/20configure: add --enable-pypi and --disable-pypiconfigure+14 -10 replies
16/20tests: Use configure-provided pyvenv for tests.gitlab-ci.d/buildtest.yml docs/devel/acpi-bits.rst docs/devel/testing.rst +4 more+26 -230 replies
17/20configure: move --enable-docs and --disable-docs back to configureconfigure+6 -00 replies
18/20mkvenv: add diagnose() method for ensure() failurespython/scripts/mkvenv.py+140 -130 replies
19/20configure: use --diagnose option with meson ensureconfigure+1 -100 replies
20/20configure: bootstrap sphinx with mkvenvconfigure+25 -20 replies

Reviews trailers per patch

4 of 20 patches have a Reviewed-by or Acked-by.

PatchDaniel P. Berrangé
1/20R-b
2/20
3/20
4/20
5/20
6/20
7/20
8/20
9/20R-b
10/20R-b
11/20R-b
12/20
13/20
14/20
15/20
16/20
17/20
18/20
19/20
20/20

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)

This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.

See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the python used to run meson is the same python used to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.

The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.

Some important points as a pre-emptive "FAQ":

- This venv is unconditionally created and lives at {build_dir}/pyvenv.

- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)

- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.

(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)

- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)

- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.

- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.

(Sphinx will not be vendored, just like it already isn't.)

- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".

- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere

- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)

- The meson git submodule is unused after this series and can be removed.

For reviewers, here's how the series is broken up:

Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.

- python: update pylint configuration

Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.

- python: add mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommand

Patches 9-11 modify our testing configuration to add new dependencies as needed.

- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSD

Patch 12 changes how we package release tarballs.

- scripts/make-release: download meson==0.61.5 .whl

Patches 13-16 wire mkvenv into configure and tests.

- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for tests

Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the diagnose() patch), but I'm keeping it separate for review for now.

- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: add diagnose() method for ensure() failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenv

That's all for now, seeya!
--js

John Snow (20):
python: update pylint configuration
python: add mkvenv.py
mkvenv: add console script entry point generation
mkvenv: Add better error message for missing pyexpat module
mkvenv: generate console entry shims from inside the venv
mkvenv: work around broken pip installations on Debian 10
mkvenv: add nested venv workaround
mkvenv: add ensure subcommand
tests/docker: add python3-venv dependency
tests/vm: Configure netbsd to use Python 3.10
tests/vm: add py310-expat to NetBSD scripts/make-release: download meson==0.61.5 .whl
configure: create a python venv unconditionally
configure: use 'mkvenv ensure meson' to bootstrap meson
configure: add --enable-pypi and --disable-pypi
tests: Use configure-provided pyvenv for tests
configure: move --enable-docs and --disable-docs back to configure
mkvenv: add diagnose() method for ensure() failures
configure: use --diagnose option with meson ensure
configure: bootstrap sphinx with mkvenv

 docs/devel/acpi-bits.rst                      |   6 +-
 docs/devel/testing.rst                        |  14 +-
 configure                                     | 139 +--
 .gitlab-ci.d/buildtest-template.yml           |   4 +-
 .gitlab-ci.d/buildtest.yml                    |   6 +-
 python/scripts/mkvenv.py                      | 871 ++++++++++++++++++
 python/setup.cfg                              |  10 +
 python/tests/flake8.sh                        |   1 +
 python/tests/isort.sh                         |   1 +
 python/tests/mypy.sh                          |   1 +
 python/tests/pylint.sh                        |   1 +
 .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
 scripts/device-crash-test                     |   2 +-
 scripts/make-release                          |  11 +
 tests/Makefile.include                        |  10 +-
 .../dockerfiles/debian-all-test-cross.docker  |   3 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
 .../dockerfiles/debian-tricore-cross.docker   |   3 +-
 tests/requirements.txt                        |   7 +-
 tests/vm/netbsd                               |   2 +
 21 files changed, 1016 insertions(+), 86 deletions(-)
 create mode 100644 python/scripts/mkvenv.py
Signature · 1 line
2.39.2