Skip to contentSkip to conversation

tests/functional: add CLI args

PATCH In review Manos Pitsidianakis 4 participants 11 messages Jul 25, 2025 – Jul 28, 2025

lore
Share & tools
lore
#0 lore raw

v3 cover tests/functional: add CLI args

This series adds extra CLI args for functional tests, useful for developers that run test files directly.

---
Changes in v3:
- Merged with other patch series
- Made argument values into class attributes, and parse_args into a
class method (staticmethod) (suggested by Thomas)
- Link to v2: https://lore.kernel.org/qemu-devel/[email protected]

Changes in v2:
- Store stdout handler in self object (thanks Daniel)
- Deduplicate handler removal code (Daniel)
- Amend commit description to mention PYTHONPATH (thanks Alex)
- Link to v1: https://lore.kernel.org/qemu-devel/[email protected]

---
Manos Pitsidianakis (4):

      tests/functional: add --debug CLI arg
      tests/functional: add --keep-scratch CLI arg
      tests/functional: add --list-tests CLI arg
      tests/functional: add -k TEST_NAME_PATTERN CLI arg
 docs/devel/testing/functional.rst      |   2 +
 tests/functional/qemu_test/testcase.py | 104 ++++++++++++++++++++++++++++++---
 2 files changed, 99 insertions(+), 7 deletions(-)

---
base-commit: 4bc8fb0135ac48cd1c8f0698976b7d73c60b7caa
change-id: 20250716-functional_tests_debug_arg-aa0a5f6b9375

--
γαῖα πυρί μιχθήτω

#1 lore raw

PATCH v3 1/4 tests/functional: add --debug CLI arg

Add argument parsing to functional tests to improve developer experience when running individual tests. All logs are printed to stdout interspersed with TAP output.

Example usage, assuming current build directory with qemu source code in the parent directory (see docs/devel/testing/functional.rst for details):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help
  usage: test_aarch64_virt [-h] [-d]

QEMU Functional test

options:
-h, --help show this help message and exit
-d, --debug Also print test and console logs on stdout. This will
make the TAP output invalid and is meant for debugging only.

---
 docs/devel/testing/functional.rst      |  2 ++
 tests/functional/qemu_test/testcase.py | 54 +++++++++++++++++++++++++++++++---
 2 files changed, 52 insertions(+), 4 deletions(-)
docs/devel/testing/functional.rst +2 -0
index 3728bab6c0c4b0cbacd00744bdb5c1462c71d7a8..2725633e09104db3912ec6167bbda652f40aa969 100644
@@ -63,6 +63,8 @@ directory should be your build folder. For example::
   $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
   $ pyvenv/bin/python3 ../tests/functional/test_file.py
 
+By default, functional tests redirect informational logs and console output to
+log files. Specify the ``--debug`` flag to also print those to standard output.
 The test framework will automatically purge any scratch files created during
 the tests. If needing to debug a failed test, it is possible to keep these
 files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env
tests/functional/qemu_test/testcase.py +50 -4

107 lines not on this page · raw message

Signature · 1 line
2.47.2
Tested-by on v3 1/4
#2 lore raw

Re: [PATCH v3 1/4] tests/functional: add --debug CLI arg

Quoted 14 lines · Manos Pitsidianakis

Add argument parsing to functional tests to improve developer experience when running individual tests. All logs are printed to stdout interspersed with TAP output.

Example usage, assuming current build directory with qemu source code in the parent directory (see docs/devel/testing/functional.rst for details):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help
  usage: test_aarch64_virt [-h] [-d]

QEMU Functional test

Quoted 7 lines

options:
-h, --help show this help message and exit
-d, --debug Also print test and console logs on stdout. This will
make the TAP output invalid and is meant for debugging only.

<snip>

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#3 lore raw

PATCH v3 2/4 tests/functional: add --keep-scratch CLI arg

In reply to Manos Pitsidianakis

Add CLI arg to keep scratch files after test execution, equivalent to setting QEMU_TEST_KEEP_SCRATCH env var.

---
 tests/functional/qemu_test/testcase.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +12 -1
index eedca7f1ad29c9e654cf56535acf9639d679f5c4..f7e306cf749e8b24a712b09dfe8673660cbb5085 100644
@@ -35,6 +35,7 @@
 
 class QemuBaseTest(unittest.TestCase):
     debug: bool = False
+    keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
 
     """
     Class method that initializes class attributes from given command-line
@@ -53,8 +54,16 @@ def parse_args():
             help="Also print test and console logs on stdout. This will make "
             "the TAP output invalid and is meant for debugging only.",
         )
+        parser.add_argument(
+            "--keep-scratch",
+            action="store_true",
+            help="Do not purge any scratch files created during the tests. "
+            "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
+            "environment.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
+        QemuBaseTest.keep_scratch |= args.keep_scratch
         return
 
     '''
@@ -262,8 +271,10 @@ def setUp(self):
             self.skipTest('One or more assets is not available')
 
     def tearDown(self):
-        if "QEMU_TEST_KEEP_SCRATCH" not in os.environ:
+        if not QemuBaseTest.keep_scratch:
             shutil.rmtree(self.workdir)
+        else:
+            self.log.info(f"Kept scratch files in {self.workdir}")
         if self.socketdir is not None:
             shutil.rmtree(self.socketdir.name)
             self.socketdir = None
Signature · 1 line
2.47.2
#4 lore raw

PATCH v3 3/4 tests/functional: add --list-tests CLI arg

In reply to Manos Pitsidianakis

Add CLI argument to list tests and exit.

Example output (current dir is build directory under root dir):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --list-tests
  test_aarch64_virt_gicv2 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv2)
  test_aarch64_virt_gicv3 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv3)
  test_alpine_virt_tcg_gic_max (test_aarch64_virt.Aarch64VirtMachine.test_alpine_virt_tcg_gic_max)
---
 tests/functional/qemu_test/testcase.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
tests/functional/qemu_test/testcase.py +14 -0
index f7e306cf749e8b24a712b09dfe8673660cbb5085..ab564f873c303bcc28c3bf7bec8c8c4569fae91c 100644
@@ -36,6 +36,7 @@
 class QemuBaseTest(unittest.TestCase):
     debug: bool = False
     keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
+    list_tests: bool = False
 
     """
     Class method that initializes class attributes from given command-line
@@ -61,9 +62,15 @@ def parse_args():
             "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
             "environment.",
         )
+        parser.add_argument(
+            "--list-tests",
+            action="store_true",
+            help="List all tests that would be executed and exit.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
         QemuBaseTest.keep_scratch |= args.keep_scratch
+        QemuBaseTest.list_tests = args.list_tests
         return
 
     '''
@@ -292,6 +299,13 @@ def main():
         path = os.path.basename(sys.argv[0])[:-3]
         QemuBaseTest.parse_args()
 
+        if QemuBaseTest.list_tests:
+            loader = unittest.TestLoader()
+            for test_suite in loader.loadTestsFromName(path):
+                for test in test_suite:
+                    print(test)
+            return
+
         cache = os.environ.get("QEMU_TEST_PRECACHE", None)
         if cache is not None:
             Asset.precache_suites(path, cache)
Signature · 1 line
2.47.2
#5 lore raw

PATCH v3 4/4 tests/functional: add -k TEST_NAME_PATTERN CLI arg

In reply to Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

---
 tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
tests/functional/qemu_test/testcase.py +21 -2
index ab564f873c303bcc28c3bf7bec8c8c4569fae91c..b045d82caa79d9d161fb868b0b0748ad7de453d9 100644
@@ -16,6 +16,7 @@
 import os
 from pathlib import Path
 import pycotap
+import itertools
 import shutil
 from subprocess import run
 import sys
@@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase):
     debug: bool = False
     keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
     list_tests: bool = False
+    test_name_patterns: list[str] = []
 
     """
     Class method that initializes class attributes from given command-line
@@ -67,10 +69,19 @@ def parse_args():
             action="store_true",
             help="List all tests that would be executed and exit.",
         )
+        parser.add_argument(
+            "-k",
+            dest="test_name_patterns",
+            action="append",
+            type=str,
+            help="Only run tests which match the given substring. "
+            "This argument is passed to unittest.main verbatim.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
         QemuBaseTest.keep_scratch |= args.keep_scratch
         QemuBaseTest.list_tests = args.list_tests
+        QemuBaseTest.test_name_patterns = args.test_name_patterns
         return
 
     '''
@@ -313,8 +324,16 @@ def main():
 
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])
+        argv = ["__dummy__", path] + (
+            list(
+                itertools.chain.from_iterable(
+                    ["-k", x] for x in QemuBaseTest.test_name_patterns
+                )
+            )
+            if QemuBaseTest.test_name_patterns
+            else []
+        )
+        res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv)
         for (test, message) in res.result.errors + res.result.failures:
 
             if hasattr(test, "log_filename"):
Signature · 1 line
2.47.2
Tested-by on v3 4/4
#6 lore raw

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

Quoted 7 lines · Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#7 lore raw

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

In reply to Manos Pitsidianakis
Quoted 57 lines · Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

---
 tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
tests/functional/qemu_test/testcase.py +11 -2
index ab564f873c303bcc28c3bf7bec8c8c4569fae91c..b045d82caa79d9d161fb868b0b0748ad7de453d9 100644
@@ -16,6 +16,7 @@
 import os
 from pathlib import Path
 import pycotap
+import itertools
 import shutil
 from subprocess import run
 import sys
@@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase):
     debug: bool = False
     keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
     list_tests: bool = False
+    test_name_patterns: list[str] = []
 
     """
     Class method that initializes class attributes from given command-line
@@ -67,10 +69,19 @@ def parse_args():
             action="store_true",
             help="List all tests that would be executed and exit.",
         )
+        parser.add_argument(
+            "-k",
+            dest="test_name_patterns",
+            action="append",
+            type=str,
+            help="Only run tests which match the given substring. "
+            "This argument is passed to unittest.main verbatim.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
         QemuBaseTest.keep_scratch |= args.keep_scratch
         QemuBaseTest.list_tests = args.list_tests
+        QemuBaseTest.test_name_patterns = args.test_name_patterns
         return
 
     '''
@@ -313,8 +324,16 @@ def main():
 
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])

tests/functional/qemu_test/testcase.py def main(): · line 336

Quoted 10 lines
tests/functional/qemu_test/testcase.py +10 -0
+        argv = ["__dummy__", path] + (
+            list(
+                itertools.chain.from_iterable(
+                    ["-k", x] for x in QemuBaseTest.test_name_patterns
+                )
+            )
+            if QemuBaseTest.test_name_patterns
+            else []
+        )
+        res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv)

unittest.main() supports a whole bunch of CLI args beyond '-k', but none of them are accessible as we're not forwarding the sys.argv that we have received. eg we're missing

$ git diff
tests/functional/qemu_test/testcase.py +1 -1
index 2a78e735f1..5caf7b13fe 100644
@@ -249,7 +249,7 @@ def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
         res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])
+                            argv=[sys.argv[0], path] + sys.argv[1:])
         for (test, message) in res.result.errors + res.result.failures:
 
             if hasattr(test, "log_filename"):

which would unlock

$ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64  PYTHONPATH=`pwd`/python ./tests/functional/test_version.py  -h
usage: test_version.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [tests ...]

positional arguments:
tests a list of any number of test modules, classes and test methods.

options:

  -h, --help           show this help message and exit
  -v, --verbose        Verbose output
  -q, --quiet          Quiet output
  --locals             Show local variables in tracebacks
  --durations N        Show the N slowest test cases (N=0 for all)
  -f, --failfast       Stop on first fail or error
  -c, --catch          Catch Ctrl-C and display results so far
  -b, --buffer         Buffer stdout and stderr during tests
  -k TESTNAMEPATTERNS  Only run tests which match the given substring

One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.

Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.

At the same time, there are some pieces in this series that do things that unittest.main() can't do on its own.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

---
 tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.

Quoted 6 lines

Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.

At the same time, there are some pieces in this series that do things that unittest.main() can't do on its own.

So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.

So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.

With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

Quoted 27 lines · Daniel P. Berrangé
Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

---
  tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
  1 file changed, 21 insertions(+), 2 deletions(-)

One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.

Quoted 6 lines

Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.

At the same time, there are some pieces in this series that do things that unittest.main() can't do on its own.

Quoted 8 lines

So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.

So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.

I basically agree, but I wonder if we could maybe also have both? First parse the sys.argv for our parameters, then pass the unknown ones to unittest.main() ?

Quoted 3 lines

With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.

With the patch applied that passes the argv to unittest.main(), there also seems to be a "discover" mode:

~/devel/qemu/tests/functional/test_vnc.py -h | grep usage:
usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f]

... however, when I try to run it, it does not work as expected (I get a "ModuleNotFoundError: No module named 'discover'" error).

Thomas

#10 lore raw

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

Quoted 33 lines · Thomas Huth
Quoted 32 lines · Daniel P. Berrangé
Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis

Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.

---
  tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
  1 file changed, 21 insertions(+), 2 deletions(-)

One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.

Quoted 6 lines

Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.

At the same time, there are some pieces in this series that do things that unittest.main() can't do on its own.

So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.

Quoted 7 lines
Quoted 3 lines

So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.

I basically agree, but I wonder if we could maybe also have both? First parse the sys.argv for our parameters, then pass the unknown ones to unittest.main() ?

Any argv that we process are subject to clashing with that defined by the test harness and it is desirable to avoid making assumptions about what test harness args exist.

To avoid that they would all have to be long options named with a fixed prefix --qemu-XXXX. Also that sould be something applied to all env vars, not just the couple of params in this series.

Quoted 7 lines
Quoted 3 lines

With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.

With the patch applied that passes the argv to unittest.main(), there also seems to be a "discover" mode:

Quoted 6 lines

~/devel/qemu/tests/functional/test_vnc.py -h | grep usage:
usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f]

... however, when I try to run it, it does not work as expected (I get a "ModuleNotFoundError: No module named 'discover'" error).

Hmm, not sure that the 'discover' option makes sense when we're asking to run a single test file.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|