Skip to contentSkip to conversation

tests: run python tests under a venv

RFC PATCH New John Snow 1 participant 8 messages Jul 11, 2022

lore
Share & tools
lore

John Snow

author
#0 lore raw

v3 cover tests: run python tests under a venv

Hi, here's another RFC for bringing external Python dependencies to the QEMU test suite.

This is mostly a refresh of a version I sent out before, but mixes in my VM improvement test as an optional pre-requisite to improve VM test stability to ensure that the BSDs all pass with the new infrastructure. (And our oldest supported Debian and Ubuntu targets, too.)

(Note: this requires dropping support for Ubuntu 18.04, which ships with a version of setuptools that is simply too old.)

This patchset is still not without some problems that I am working on, but progress has been slow.

Problems I am aware of:

- This version of the patch series does not itself enforce any offline-only behavior for venv creation, but downstreams can modify any call to 'mkvenv' to pass '--offline'. I am working on a configure file toggle to swap the default behavior when running tests.

- iotests will now actually never run mypy or pylint tests by default anymore, because the bootstrapper won't select those packages by default, and the virtual environment won't utilize the system packages, so iotest 297 will just "skip" all of the time now.

The reason we don't want to install these packages by default is because we don't want to add dependencies on mypy and pylint for downstream builds.

With these patches, 297 would still work if you manually opened up the testing venv and installed suitable mypy/pylint packages there. I could also add a new optional dependency group, and one could theoretically invoke a once-per-build-dir command of 'make check-venv-iotests' to help make the process only semi-manual, but it's still annoying.

Ideally, the python checks in qemu.git/python/ can handle the same tests as 297 does -- but we need to give a shorthand invocation like "make check-python" that is excluded from the default "make check" to allow block developers to quickly opt-in to the same tests.

I've covered some of the problems here on-list before:
https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg03661.html

...But I haven't quite solved them yet.

- iotests can now self-bootstrap the venv when it isn't present, but this self-bootstrapping has some issues in that because it skips the Makefile magic, it cannot update the venv when changes are made to the venv configuration piece.

That's all for now. Work on the configure file integration is ongoing. I don't know if I'll beat soft freeze (It's looking unlikely given the amount of IRL issues I am juggling right now), but I'm hoping to push as much of this forward as I can to try and get some testing in for RC phase to determine what problems might exist that I haven't noticed yet.

--js

John Snow (7):
tests: create optional tests/venv dependency groups
tests: pythonize test venv creation
tests: Remove spurious pip warnings on Ubuntu20.04 tests/vm: add venv pre-requisites to VM building recipes
tests: add 'check-venv' as a dependency of 'make check'
iotests: use tests/venv for running tests
iotests: self-bootstrap testing venv

 tests/Makefile.include        |  32 +++---
 tests/mkvenv.py               | 187 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/testenv.py |  25 +++--
 tests/requirements.txt        |   6 --
 tests/setup.cfg               |  20 ++++
 tests/setup.py                |  16 +++
 tests/vm/netbsd               |   1 +
 7 files changed, 262 insertions(+), 25 deletions(-)
 create mode 100644 tests/mkvenv.py
 delete mode 100644 tests/requirements.txt
 create mode 100644 tests/setup.cfg
 create mode 100644 tests/setup.py
Signature · 1 line
2.34.3
author
#1 lore raw

RFC PATCH v3 1/7 tests: create optional tests/venv dependency groups

This patch uses a dummy package and setup.cfg/setup.py files to manage optional dependency groups for the test venv specification. Now, there's a core set of dependencies which for now includes just "qemu" (but soon, qemu.qmp) and a separate, optional 'avocado' group that includes avocado-framework and pycdlib.

Practical upshot: We install only a minimum of things for the majority of check-* targets, but allow optional add-ons to be processed when running avocado tests. This will spare downstreams from having to add more dependencies than is necessary as a build dependencies when invoking "make check".

(We also keep both sets of dependencies in one file, which is helpful for review to ensure that different option groups don't conflict with one another.)

NOTE: There is a non-fatal caveat introduced by this patch on Ubuntu 20.04 systems; see the subsequent commit "tests: Remove spurious pip warnings on Ubuntu20.04" for more information.

---
 tests/Makefile.include | 21 ++++++++++++++-------
 tests/requirements.txt |  6 ------
 tests/setup.cfg        | 20 ++++++++++++++++++++
 tests/setup.py         | 16 ++++++++++++++++
 4 files changed, 50 insertions(+), 13 deletions(-)
 delete mode 100644 tests/requirements.txt
 create mode 100644 tests/setup.cfg
 create mode 100644 tests/setup.py
tests/Makefile.include +14 -7
index 3accb83b132..82c697230e0 100644
@@ -81,13 +81,13 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 
 # Python venv for running tests
 
-.PHONY: check-venv check-avocado check-acceptance check-acceptance-deprecated-warning
+.PHONY: check-venv check-venv-avocado check-avocado check-acceptance \
+        check-acceptance-deprecated-warning
 
 # Build up our target list from the filtered list of ninja targets
 TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
-TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
 ifndef AVOCADO_TESTS
@@ -108,10 +108,16 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
     $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
     "VENVPIP","$1")
 
-$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+# Core dependencies for tests/venv
+$(TESTS_VENV_DIR): $(SRC_PATH)/tests/setup.cfg $(SRC_PATH)/python/setup.cfg
 	$(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
+	$(call quiet-venv-pip,install "$(SRC_PATH)/tests/")
+	$(call quiet-command, touch $@)
+
+# Optional avocado dependencies for tests/venv
+$(TESTS_VENV_DIR)/avocado: $(TESTS_VENV_DIR)
+	$(call quiet-venv-pip,install "$(SRC_PATH)/tests/[avocado]")
 	$(call quiet-command, touch $@)
 
 $(TESTS_RESULTS_DIR):
@@ -119,6 +125,7 @@ $(TESTS_RESULTS_DIR):
             MKDIR, $@)
 
 check-venv: $(TESTS_VENV_DIR)
+check-venv-avocado: $(TESTS_VENV_DIR)/avocado
 
 FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
 FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
@@ -126,16 +133,16 @@ FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x
 FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
 
 # download one specific Fedora 31 image
-get-vm-image-fedora-31-%: check-venv
+get-vm-image-fedora-31-%: check-venv-avocado
 	$(call quiet-command, \
              $(TESTS_PYTHON) -m avocado vmimage get \
              --distro=fedora --distro-version=31 --arch=$*, \
 	"AVOCADO", "Downloading avocado tests VM image for $*")
 
 # download all vm images, according to defined targets
-get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
+get-vm-images: check-venv-avocado $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
-check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
+check-avocado: check-venv-avocado $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_PYTHON) -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
tests/requirements.txt deleted +0 -6
deleted file mode 100644
index 0ba561b6bdf..00000000000
@@ -1,6 +0,0 @@
-# Add Python module requirements, one per line, to be installed
-# in the tests/venv Python virtual environment. For more info,
-# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-# Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
-pycdlib==1.11.0
tests/setup.cfg new file +20 -0
new file mode 100644
index 00000000000..263a5de01af
@@ -0,0 +1,20 @@
+# This file represents a "dummy" package that expresses
+# the dependencies necessary to run the various python-based
+# test suites in the qemu.git tree.
+[metadata]
+name = qemu.dummy_tests
+version = 0.0.0
+description = qemu.git testing dummy package
+
+[options]
+packages =
+python_requires = >= 3.6
+install_requires =
+    qemu  # Note: A special setup script will always install 'qemu' from
+          # qemu.git/python. It doesn't have a meaningful/real version.
+
+[options.extras_require]
+# Extra dependencies required by the avocado tests.
+avocado =
+    avocado-framework<89.0,>=88.1
+    pycdlib<2.0,>=1.11.0
tests/setup.py new file +16 -0
new file mode 100644
index 00000000000..c99fe5d1725
@@ -0,0 +1,16 @@
+# This setup file is just-enough-config to allow pip to bootstrap a
+# testing environment. It is not meant to be executed directly.
+# See also: setup.cfg
+
+import setuptools
+import pkg_resources
+
+
+def main():
+    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
+    pkg_resources.require('setuptools>=39.2')
+    setuptools.setup()
+
+
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.34.3
author
#2 lore raw

RFC PATCH v3 2/7 tests: pythonize test venv creation

In reply to John Snow

This splits the venv creation logic out of the Makefile and into a Python script.

One reason for doing this is to be able to call the venv bootstrapper from places outside of the Makefile, e.g. configure and iotests. Another reason is to be able to add "offline" logic to modify the behavior of the script in certain circumstances.

RFC: I don't like how I have an entire "enter_venv" function here, and by the end of the series, completely separate logic for entering a venv in testenv.py -- but they both operate in different ways: this patch offers a method that alters the current ENV immediately, whereas the testenv.py method alters a copied ENV that is explicitly passed to subprocesses.

Still, there's a bit more boilerplate than I'd like, but it's probably fine and it probably won't need to be updated much, but... less code is usually better.

But, even if I did write it more generically here; iotests wouldn't be able to use it anyway, because importing it would mean more sys.path hacking. So... eh?

---
 tests/Makefile.include |  16 ++--
 tests/mkvenv.py        | 186 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 192 insertions(+), 10 deletions(-)
 create mode 100644 tests/mkvenv.py
tests/Makefile.include +6 -10
index 82c697230e0..d8af6a38112 100644
@@ -104,21 +104,17 @@ else
 	AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
 endif
 
-quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
-    $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
-    "VENVPIP","$1")
-
 # Core dependencies for tests/venv
 $(TESTS_VENV_DIR): $(SRC_PATH)/tests/setup.cfg $(SRC_PATH)/python/setup.cfg
-	$(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
-	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(call quiet-venv-pip,install "$(SRC_PATH)/tests/")
-	$(call quiet-command, touch $@)
+	$(call quiet-command, \
+            $(PYTHON) "$(SRC_PATH)/tests/mkvenv.py" "$@", \
+            MKVENV, $@)
 
 # Optional avocado dependencies for tests/venv
 $(TESTS_VENV_DIR)/avocado: $(TESTS_VENV_DIR)
-	$(call quiet-venv-pip,install "$(SRC_PATH)/tests/[avocado]")
-	$(call quiet-command, touch $@)
+	$(call quiet-command, \
+            $(PYTHON) "$(SRC_PATH)/tests/mkvenv.py" --opt avocado "$<", \
+            MKVENV, $@)
 
 $(TESTS_RESULTS_DIR):
 	$(call quiet-command, mkdir -p $@, \
tests/mkvenv.py new file +186 -0

186 lines not on this page · raw message

Signature · 1 line
2.34.3
author
#3 lore raw

RFC PATCH v3 3/7 tests: Remove spurious pip warnings on Ubuntu20.04

In reply to John Snow

The version of pip ("20.0.2") that ships with Ubuntu 20.04 has a bug where it will try to attempt building a wheel even if the "wheel" python package that enables it to do so is not installed. Even though pip continues gracefully from source, The result is a lot of irrelevant failure output.

Upstream pip 20.0.2 does not have this problem, and pip 20.1 introduces a new info message that informs a user that wheel building is being skipped.

On this version, the output can be silenced by passing --no-binary to coax pip into skipping that step to begin with. Note, this error does not seem to show up for the "qemu" package because we install that package in editable mode. (I think, but did not test, that installing an empty package in editable mode caused more problems than it fixed.)

---
 tests/mkvenv.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/mkvenv.py +2 -1
index 0667106d6aa..78f8d0382e0 100644
@@ -144,7 +144,8 @@ def make_qemu_venv(
     with enter_venv(venv_path):
         if do_initialize:
             install("-e", str(pysrc_path), offline=offline)
-            install(str(test_src_path), offline=offline)
+            install("--no-binary", "qemu.dummy-tests",
+                    str(test_src_path), offline=offline)
             venv_path.touch()
 
         for option in options:
Signature · 1 line
2.34.3
author
#4 lore raw

RFC PATCH v3 4/7 tests/vm: add venv pre-requisites to VM building recipes

In reply to John Snow

NetBSD needs "py37-pip" in order to create virtual environments.

---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)
tests/vm/netbsd +1 -0
index 45aa9a7fda7..53fe508e487 100755
@@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM):
         "pkgconf",
         "xz",
         "python37",
+        "py37-pip",
         "ninja-build",
 
         # gnu tools
Signature · 1 line
2.34.3
author
#5 lore raw

RFC PATCH v3 5/7 tests: add 'check-venv' as a dependency of 'make check'

In reply to John Snow

This patch adds the 'check-venv' target as a requisite of all meson driven check-* targets. As of this commit, it will only install the "qemu" namespace package from the source tree, and nothing else.

In the future, the "qemu" namespace package in qemu.git will begin to require an external qemu.qmp package, and this would be installed into this environment as well.

The avocado test dependencies will *not* be pulled into this venv by default, but they may be added in at a later point in time by running 'make check-avocado' or, without running the tests, 'make check-venv-avocado'.

---
 tests/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)
tests/Makefile.include +3 -0
index d8af6a38112..d484a335be5 100644
@@ -155,6 +155,9 @@ check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+# The do-meson-check and do-meson-bench targets are defined in Makefile.mtest
+do-meson-check do-meson-bench: check-venv
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
Signature · 1 line
2.34.3
author
#6 lore raw

RFC PATCH v3 6/7 iotests: use tests/venv for running tests

In reply to John Snow

Essentially, the changes to testenv.py here mimic the changes that occur when you "source venv/bin/activate.fish" or similar.

(1) update iotest's internal notion of which python binary to use,
(2) export the VIRTUAL_ENV variable,
(3) front-load the venv/bin directory to PATH.

If the venv directory isn't found, raise a friendly exception that tries to give the human operator a friendly clue as to what's gone wrong. The subsequent commit attempts to address this shortcoming by teaching iotests how to invoke the venv bootstrapper in this circumstance instead.

---
 tests/qemu-iotests/testenv.py | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)
tests/qemu-iotests/testenv.py +19 -6
index a864c74b123..29404ac94be 100644
@@ -65,8 +65,9 @@ class TestEnv(ContextManager['TestEnv']):
     # lot of them. Silence pylint:
     # pylint: disable=too-many-instance-attributes
 
-    env_variables = ['PYTHONPATH', 'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR',
-                     'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
+    env_variables = ['PYTHONPATH', 'VIRTUAL_ENV', 'PYTHON', 'PATH',
+                     'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR',
+                     'QEMU_PROG', 'QEMU_IMG_PROG',
                      'QEMU_IO_PROG', 'QEMU_NBD_PROG', 'QSD_PROG',
                      'QEMU_OPTIONS', 'QEMU_IMG_OPTIONS',
                      'QEMU_IO_OPTIONS', 'QEMU_IO_OPTIONS_NO_FMT',
@@ -102,18 +103,29 @@ def get_env(self) -> Dict[str, str]:
 
     def init_directories(self) -> None:
         """Init directory variables:
+             VIRTUAL_ENV
+             PATH
              PYTHONPATH
              TEST_DIR
              SOCK_DIR
              SAMPLE_IMG_DIR
         """
+        venv_path = Path(self.build_root, 'tests/venv/')
+        if not venv_path.exists():
+            raise FileNotFoundError(
+                f"Virtual environment \"{venv_path!s}\" isn't found."
+                " (Maybe you need to run 'make check-venv'"
+                " from the build dir?)"
+            )
+        self.virtual_env: str = str(venv_path)
 
-        # Path where qemu goodies live in this source tree.
-        qemu_srctree_path = Path(__file__, '../../../python').resolve()
+        self.path = os.pathsep.join((
+            str(venv_path / 'bin'),
+            os.environ['PATH'],
+        ))
 
         self.pythonpath = os.pathsep.join(filter(None, (
             self.source_iotests,
-            str(qemu_srctree_path),
             os.getenv('PYTHONPATH'),
         )))
 
@@ -138,7 +150,7 @@ def init_binaries(self) -> None:
              PYTHON (for bash tests)
              QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG
         """
-        self.python = sys.executable
+        self.python: str = os.path.join(self.virtual_env, 'bin', 'python3')
 
         def root(*names: str) -> str:
             return os.path.join(self.build_root, *names)
@@ -300,6 +312,7 @@ def print_env(self, prefix: str = '') -> None:
 {prefix}GDB_OPTIONS   -- {GDB_OPTIONS}
 {prefix}VALGRIND_QEMU -- {VALGRIND_QEMU}
 {prefix}PRINT_QEMU_OUTPUT -- {PRINT_QEMU}
+{prefix}VIRTUAL_ENV   -- {VIRTUAL_ENV}
 {prefix}"""
 
         args = collections.defaultdict(str, self.get_env())
Signature · 1 line
2.34.3
author
#7 lore raw

RFC PATCH v3 7/7 iotests: self-bootstrap testing venv

In reply to John Snow

When iotests are invoked manually from
e.g. $build/tests/qemu-iotests/check, it is not necessarily guaranteed that we'll have run 'check-venv' yet.

With this patch, teach testenv.py how to create its own environment.

Note: this self-bootstrapping is fairly rudimentary and will miss certain triggers to refresh the venv. It will miss when new dependencies are added to either python/setup.cfg or tests/setup.cfg. It can be coaxed into updating by running 'make check', 'make check-block', 'make check-venv', etc.

---
 tests/qemu-iotests/testenv.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/qemu-iotests/testenv.py +4 -4
index 29404ac94be..e985eaf3e97 100644
@@ -112,10 +112,10 @@ def init_directories(self) -> None:
         """
         venv_path = Path(self.build_root, 'tests/venv/')
         if not venv_path.exists():
-            raise FileNotFoundError(
-                f"Virtual environment \"{venv_path!s}\" isn't found."
-                " (Maybe you need to run 'make check-venv'"
-                " from the build dir?)"
+            mkvenv = Path(self.source_iotests, '../mkvenv.py')
+            subprocess.run(
+                (sys.executable, str(mkvenv), str(venv_path)),
+                check=True,
             )
         self.virtual_env: str = str(venv_path)
 
Signature · 1 line
2.34.3