tests/functional: add CLI args
v3 cover tests/functional: add CLI args
This series adds extra CLI args for functional tests, useful for developers that run test files directly.
---
Changes in v3:
- Merged with other patch series
- Made argument values into class attributes, and parse_args into a
class method (staticmethod) (suggested by Thomas)
- Link to v2: https://lore.kernel.org/qemu-devel/[email protected]
Changes in v2:
- Store stdout handler in self object (thanks Daniel)
- Deduplicate handler removal code (Daniel)
- Amend commit description to mention PYTHONPATH (thanks Alex)
- Link to v1: https://lore.kernel.org/qemu-devel/[email protected]
---
Manos Pitsidianakis (4):
tests/functional: add --debug CLI arg
tests/functional: add --keep-scratch CLI arg
tests/functional: add --list-tests CLI arg
tests/functional: add -k TEST_NAME_PATTERN CLI argdocs/devel/testing/functional.rst | 2 + tests/functional/qemu_test/testcase.py | 104 ++++++++++++++++++++++++++++++--- 2 files changed, 99 insertions(+), 7 deletions(-)
---
base-commit: 4bc8fb0135ac48cd1c8f0698976b7d73c60b7caa
change-id: 20250716-functional_tests_debug_arg-aa0a5f6b9375
--
γαῖα πυρί μιχθήτω
PATCH v3 1/4 tests/functional: add --debug CLI arg
Add argument parsing to functional tests to improve developer experience when running individual tests. All logs are printed to stdout interspersed with TAP output.
Example usage, assuming current build directory with qemu source code in the parent directory (see docs/devel/testing/functional.rst for details):
$ export PYTHONPATH=../python:../tests/functional $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64" $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help usage: test_aarch64_virt [-h] [-d]
QEMU Functional test
options:
-h, --help show this help message and exit
-d, --debug Also print test and console logs on stdout. This will
make the TAP output invalid and is meant for debugging only.
--- docs/devel/testing/functional.rst | 2 ++ tests/functional/qemu_test/testcase.py | 54 +++++++++++++++++++++++++++++++--- 2 files changed, 52 insertions(+), 4 deletions(-)
docs/devel/testing/functional.rst +2 -0
@@ -63,6 +63,8 @@ directory should be your build folder. For example:: $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64 $ pyvenv/bin/python3 ../tests/functional/test_file.py +By default, functional tests redirect informational logs and console output to +log files. Specify the ``--debug`` flag to also print those to standard output. The test framework will automatically purge any scratch files created during the tests. If needing to debug a failed test, it is possible to keep these files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env
tests/functional/qemu_test/testcase.py +50 -4
107 lines not on this page · raw message
Signature · 1 line
2.47.2
Re: [PATCH v3 1/4] tests/functional: add --debug CLI arg
Quoted 14 lines · Manos Pitsidianakis
Add argument parsing to functional tests to improve developer experience when running individual tests. All logs are printed to stdout interspersed with TAP output.
Example usage, assuming current build directory with qemu source code in the parent directory (see
docs/devel/testing/functional.rstfor details):$ export PYTHONPATH=../python:../tests/functional $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64" $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help usage: test_aarch64_virt [-h] [-d]QEMU Functional test
Quoted 7 lines
options:
-h, --help show this help message and exit
-d, --debug Also print test and console logs on stdout. This will
make the TAP output invalid and is meant for debugging only.Signed-off-by Manos Pitsidianakis <[email protected]>
<snip>
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
PATCH v3 2/4 tests/functional: add --keep-scratch CLI arg
In reply to Manos PitsidianakisAdd CLI arg to keep scratch files after test execution, equivalent to setting QEMU_TEST_KEEP_SCRATCH env var.
--- tests/functional/qemu_test/testcase.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
tests/functional/qemu_test/testcase.py +12 -1
@@ -35,6 +35,7 @@ class QemuBaseTest(unittest.TestCase): debug: bool = False + keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ """ Class method that initializes class attributes from given command-line
@@ -53,8 +54,16 @@ def parse_args(): help="Also print test and console logs on stdout. This will make " "the TAP output invalid and is meant for debugging only.", ) + parser.add_argument( + "--keep-scratch", + action="store_true", + help="Do not purge any scratch files created during the tests. " + "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the " + "environment.", + ) args = parser.parse_args() QemuBaseTest.debug = args.debug + QemuBaseTest.keep_scratch |= args.keep_scratch return '''
@@ -262,8 +271,10 @@ def setUp(self): self.skipTest('One or more assets is not available') def tearDown(self): - if "QEMU_TEST_KEEP_SCRATCH" not in os.environ: + if not QemuBaseTest.keep_scratch: shutil.rmtree(self.workdir) + else: + self.log.info(f"Kept scratch files in {self.workdir}") if self.socketdir is not None: shutil.rmtree(self.socketdir.name) self.socketdir = None
Signature · 1 line
2.47.2
PATCH v3 3/4 tests/functional: add --list-tests CLI arg
In reply to Manos PitsidianakisAdd CLI argument to list tests and exit.
Example output (current dir is build directory under root dir):
$ export PYTHONPATH=../python:../tests/functional $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64" $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --list-tests test_aarch64_virt_gicv2 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv2) test_aarch64_virt_gicv3 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv3) test_alpine_virt_tcg_gic_max (test_aarch64_virt.Aarch64VirtMachine.test_alpine_virt_tcg_gic_max)
--- tests/functional/qemu_test/testcase.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tests/functional/qemu_test/testcase.py +14 -0
@@ -36,6 +36,7 @@ class QemuBaseTest(unittest.TestCase): debug: bool = False keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ + list_tests: bool = False """ Class method that initializes class attributes from given command-line
@@ -61,9 +62,15 @@ def parse_args(): "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the " "environment.", ) + parser.add_argument( + "--list-tests", + action="store_true", + help="List all tests that would be executed and exit.", + ) args = parser.parse_args() QemuBaseTest.debug = args.debug QemuBaseTest.keep_scratch |= args.keep_scratch + QemuBaseTest.list_tests = args.list_tests return '''
@@ -292,6 +299,13 @@ def main(): path = os.path.basename(sys.argv[0])[:-3] QemuBaseTest.parse_args() + if QemuBaseTest.list_tests: + loader = unittest.TestLoader() + for test_suite in loader.loadTestsFromName(path): + for test in test_suite: + print(test) + return + cache = os.environ.get("QEMU_TEST_PRECACHE", None) if cache is not None: Asset.precache_suites(path, cache)
Signature · 1 line
2.47.2
PATCH v3 4/4 tests/functional: add -k TEST_NAME_PATTERN CLI arg
In reply to Manos PitsidianakisAdd a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
--- tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
tests/functional/qemu_test/testcase.py +21 -2
@@ -16,6 +16,7 @@ import os from pathlib import Path import pycotap +import itertools import shutil from subprocess import run import sys
@@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase): debug: bool = False keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ list_tests: bool = False + test_name_patterns: list[str] = [] """ Class method that initializes class attributes from given command-line
@@ -67,10 +69,19 @@ def parse_args(): action="store_true", help="List all tests that would be executed and exit.", ) + parser.add_argument( + "-k", + dest="test_name_patterns", + action="append", + type=str, + help="Only run tests which match the given substring. " + "This argument is passed to unittest.main verbatim.", + ) args = parser.parse_args() QemuBaseTest.debug = args.debug QemuBaseTest.keep_scratch |= args.keep_scratch QemuBaseTest.list_tests = args.list_tests + QemuBaseTest.test_name_patterns = args.test_name_patterns return '''
@@ -313,8 +324,16 @@ def main(): tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, test_output_log = pycotap.LogMode.LogToError) - res = unittest.main(module = None, testRunner = tr, exit = False, - argv=["__dummy__", path]) + argv = ["__dummy__", path] + ( + list( + itertools.chain.from_iterable( + ["-k", x] for x in QemuBaseTest.test_name_patterns + ) + ) + if QemuBaseTest.test_name_patterns + else [] + ) + res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv) for (test, message) in res.result.errors + res.result.failures: if hasattr(test, "log_filename"):
Signature · 1 line
2.47.2
Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg
Quoted 7 lines · Manos Pitsidianakis
Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
Acked-by Thomas Huth <[email protected]>Signed-off-by Manos Pitsidianakis <[email protected]>
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg
In reply to Manos PitsidianakisQuoted 57 lines · Manos Pitsidianakis
Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
Acked-by Thomas Huth <[email protected]>Signed-off-by Manos Pitsidianakis <[email protected]>--- tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)tests/functional/qemu_test/testcase.py +11
-2@@ -16,6 +16,7 @@ import os from pathlib import Path import pycotap +import itertools import shutil from subprocess import run import sys@@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase): debug: bool = False keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ list_tests: bool = False + test_name_patterns: list[str] = [] """ Class method that initializes class attributes from given command-line@@ -67,10 +69,19 @@ def parse_args(): action="store_true", help="List all tests that would be executed and exit.", ) + parser.add_argument( + "-k", + dest="test_name_patterns", + action="append", + type=str, + help="Only run tests which match the given substring. " + "This argument is passed to unittest.main verbatim.", + ) args = parser.parse_args() QemuBaseTest.debug = args.debug QemuBaseTest.keep_scratch |= args.keep_scratch QemuBaseTest.list_tests = args.list_tests + QemuBaseTest.test_name_patterns = args.test_name_patterns return '''@@ -313,8 +324,16 @@ def main(): tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, test_output_log = pycotap.LogMode.LogToError) - res = unittest.main(module = None, testRunner = tr, exit = False, - argv=["__dummy__", path])
tests/functional/qemu_test/testcase.py def main(): · line 336
Quoted 10 lines
tests/functional/qemu_test/testcase.py +10
-0+ argv = ["__dummy__", path] + ( + list( + itertools.chain.from_iterable( + ["-k", x] for x in QemuBaseTest.test_name_patterns + ) + ) + if QemuBaseTest.test_name_patterns + else [] + ) + res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv)
unittest.main() supports a whole bunch of CLI args beyond '-k', but none of them are accessible as we're not forwarding the sys.argv that we have received. eg we're missing
$ git diff
tests/functional/qemu_test/testcase.py +1 -1
@@ -249,7 +249,7 @@ def main(): tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, test_output_log = pycotap.LogMode.LogToError) res = unittest.main(module = None, testRunner = tr, exit = False, - argv=["__dummy__", path]) + argv=[sys.argv[0], path] + sys.argv[1:]) for (test, message) in res.result.errors + res.result.failures: if hasattr(test, "log_filename"):
which would unlock
$ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 PYTHONPATH=`pwd`/python ./tests/functional/test_version.py -h usage: test_version.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [tests ...]
positional arguments:
tests a list of any number of test modules, classes and test methods.
options:
-h, --help show this help message and exit -v, --verbose Verbose output -q, --quiet Quiet output --locals Show local variables in tracebacks --durations N Show the N slowest test cases (N=0 for all) -f, --failfast Stop on first fail or error -c, --catch Catch Ctrl-C and display results so far -b, --buffer Buffer stdout and stderr during tests -k TESTNAMEPATTERNS Only run tests which match the given substring
One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.
Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.
At the same time, there are some pieces in this series that do things that unittest.main() can't do on its own.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg
Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis
Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
Acked-by Thomas Huth <[email protected]>Signed-off-by Manos Pitsidianakis <[email protected]>--- tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.
Quoted 6 lines
Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.
At the same time, there are some pieces in this series that do things that
unittest.main()can't do on its own.
So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.
So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.
With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg
Quoted 27 lines · Daniel P. Berrangé
Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis
Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
Acked-by Thomas Huth <[email protected]>Signed-off-by Manos Pitsidianakis <[email protected]>--- tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.
Quoted 6 lines
Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.
At the same time, there are some pieces in this series that do things that
unittest.main()can't do on its own.
Quoted 8 lines
So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.
So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.
I basically agree, but I wonder if we could maybe also have both? First parse the sys.argv for our parameters, then pass the unknown ones to unittest.main() ?
Quoted 3 lines
With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.
With the patch applied that passes the argv to unittest.main(), there also seems to be a "discover" mode:
~/devel/qemu/tests/functional/test_vnc.py -h | grep usage:
usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f]
... however, when I try to run it, it does not work as expected (I get a "ModuleNotFoundError: No module named 'discover'" error).
Thomas
Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg
Quoted 33 lines · Thomas Huth
Quoted 32 lines · Daniel P. Berrangé
Quoted 19 lines · Daniel P. Berrangé
Quoted 10 lines · Manos Pitsidianakis
Add a CLI argument that takes fnmatch(3)-style patterns as value and can be specified many times. Only tests that match the pattern will be executed. This argument is passed to unittest.main which takes the same argument.
Acked-by Thomas Huth <[email protected]>Signed-off-by Manos Pitsidianakis <[email protected]>--- tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)One of the goals with the new functional test system was that we stop trying to (re-)invent a custom test runner harness, as was the case with Avocado, in favour of relying on the pre-existing python infrastructure to the greatest extent possible.
Quoted 6 lines
Seeing this, and all the other CLI arg handling added in this series, makes me fairly uncomfortable, as it is effectively inventing a custom test runner once again which is exactly what we wanted to get away from.
At the same time, there are some pieces in this series that do things that
unittest.main()can't do on its own.So considering the broader picture, we already have a load of tunables on the test execution that we control exclusively via envirnoment variables, as that gives us independance of the test runner, which owns sys.argv processing.
Quoted 7 lines
Quoted 3 lines
So in terms of this series, IMHO, we should just add support for QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, and not touch sys.argv at all.
I basically agree, but I wonder if we could maybe also have both? First parse the sys.argv for our parameters, then pass the unknown ones to
unittest.main()?
Any argv that we process are subject to clashing with that defined by the test harness and it is desirable to avoid making assumptions about what test harness args exist.
To avoid that they would all have to be long options named with a fixed prefix --qemu-XXXX. Also that sould be something applied to all env vars, not just the couple of params in this series.
Quoted 7 lines
Quoted 3 lines
With that, the only thing we're missing is a way to enumerate the test cases, but IMHO that's the job of the test runner and thus out of scope for QEMU to solve.
With the patch applied that passes the argv to
unittest.main(), there also seems to be a "discover" mode:
Quoted 6 lines
~/devel/qemu/tests/functional/test_vnc.py-h | grep usage:
usage:test_vnc.py[-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
usage:test_vnc.pydiscover [-h] [-v] [-q] [--locals] [--durations N] [-f]... however, when I try to run it, it does not work as expected (I get a "ModuleNotFoundError: No module named 'discover'" error).
Hmm, not sure that the 'discover' option makes sense when we're asking to run a single test file.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|