tests/functional: Allow tests to be run individually
PATCH 1/1 tests/functional: Allow tests to be run individually
The functional tests currently don't allow a single test to be selected for execution by dotted name, e.g:
../tests/functional/ppc64/test_pseries.py PseriesMachine.test_ppc64_linux_boot
^
The issue is that the testcase.py main function passes the test module's name as the second argument to unittest.main(), which makes it ignore all other positional arguments (presumably because the module is already the superset of all tests).
After commit cac08383f0 ("tests/functional: expose sys.argv to unittest.main"), the situation improves by passing the rest of the argv from the command line invocation into unittest.main(), but it still doesn't fix the issue. The short form options are now accepted, so the -k option could be used to filter for a pattern, which is useful, but not the same as listing the test names.
Fix this by passing the test module name via the "module" argument to unittest.main() and stop touching argv. The ways of invoking tests are now as per unittests documentation (-k still works):
Examples:
test_pseries.py - run default set of tests
test_pseries.py MyTestSuite - run suite 'MyTestSuite'
test_pseries.py MyTestCase.testSomething - run MyTestCase.testSomething
test_pseries.py MyTestCase - run all 'test*' test methods in MyTestCaseNote that ever since we've been programatically passing the module name to unittest.main(), the usage 'test_pseries.py test_pseries' was never valid. It used to "work" just the same as 'test_pseries.py foobar' would. After this patch, that usage results in an error.
Also note that testcase.py:main() pertains to running the test module that invoked it via QemuSystemTest.main(), i.e. module == __main__. So the 'discover' usage of unittest doesn't apply here, the module is already discovered because that's where this code was called from to begin with. This patch could just as well call unittest.main() instead of unittest.main(test_module), but the latter provides nicer error messages prefixed with the module name.
--- tests/functional/qemu_test/testcase.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
tests/functional/qemu_test/testcase.py +3 -4
@@ -249,17 +249,16 @@ def main(): warnings.simplefilter("default") os.environ["PYTHONWARNINGS"] = "default" - path = os.path.basename(sys.argv[0])[:-3] + test_module = os.path.basename(sys.argv[0])[:-3] cache = os.environ.get("QEMU_TEST_PRECACHE", None) if cache is not None: - Asset.precache_suites(path, cache) + Asset.precache_suites(test_module, cache) return tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, test_output_log = pycotap.LogMode.LogToError) - res = unittest.main(module = None, testRunner = tr, exit = False, - argv=[sys.argv[0], path] + sys.argv[1:]) + res = unittest.main(test_module, testRunner = tr, exit = False) failed = {} for (test, _message) in res.result.errors + res.result.failures: if hasattr(test, "log_filename") and not test.id() in failed:
Signature · 1 line
2.51.0
Quoted 21 lines · Fabiano Rosas
The functional tests currently don't allow a single test to be selected for execution by dotted name, e.g:
../tests/functional/ppc64/test_pseries.pyPseriesMachine.test_ppc64_linux_boot^The issue is that the
testcase.pymain function passes the test module's name as the second argument tounittest.main(), which makes it ignore all other positional arguments (presumably because the module is already the superset of all tests).After commit
cac08383f0("tests/functional: expose sys.argv to unittest.main"), the situation improves by passing the rest of the argv from the command line invocation intounittest.main(), but it still doesn't fix the issue. The short form options are now accepted, so the -k option could be used to filter for a pattern, which is useful, but not the same as listing the test names.Fix this by passing the test module name via the "module" argument to
unittest.main()and stop touching argv. The ways of invoking tests are now as per unittests documentation (-k still works):
Quoted 5 lines
Examples:
test_pseries.py - run default set of tests test_pseries.py MyTestSuite - run suite 'MyTestSuite' test_pseries.py MyTestCase.testSomething - run MyTestCase.testSomething test_pseries.py MyTestCase - run all 'test*' test methods in MyTestCase
If you've got some spare time, maybe you could also add some words about running individual sub-tests to docs/devel/testing/functional.rst ?
Thomas
Quoted 27 lines · Thomas Huth
Quoted 21 lines · Fabiano Rosas
The functional tests currently don't allow a single test to be selected for execution by dotted name, e.g:
../tests/functional/ppc64/test_pseries.pyPseriesMachine.test_ppc64_linux_boot^The issue is that the
testcase.pymain function passes the test module's name as the second argument tounittest.main(), which makes it ignore all other positional arguments (presumably because the module is already the superset of all tests).After commit
cac08383f0("tests/functional: expose sys.argv to unittest.main"), the situation improves by passing the rest of the argv from the command line invocation intounittest.main(), but it still doesn't fix the issue. The short form options are now accepted, so the -k option could be used to filter for a pattern, which is useful, but not the same as listing the test names.Fix this by passing the test module name via the "module" argument to
unittest.main()and stop touching argv. The ways of invoking tests are now as per unittests documentation (-k still works):Quoted 5 lines
Examples:
test_pseries.py - run default set of tests test_pseries.py MyTestSuite - run suite 'MyTestSuite' test_pseries.py MyTestCase.testSomething - run MyTestCase.testSomething test_pseries.py MyTestCase - run all 'test*' test methods in MyTestCase
Quoted 5 lines
Tested-by Thomas Huth <[email protected]>If you've got some spare time, maybe you could also add some words about running individual sub-tests to
docs/devel/testing/functional.rst?
Ah, thanks for the reminder, I'll do it right away.