tests/qtest: Run migration subtests via meson
cover tests/qtest: Run migration subtests via meson
We've had some issues with the migration-test failing in CI and giving no clue as to which migration sub-test has failed. I think it would be an improvement to have each subtest listed individually.
With this we can see which migration test has failed, can set timeouts individually and can run 'make check-migration' to run only the migration tests.
It does however increase the amount of lines emitted with make check, so let me know if this is an issue.
I had to include some hacking due to meson, but I think it is simple enough and worth the benefits.
CI run: https://gitlab.com/farosas/qemu/-/pipelines/919324510
Fabiano Rosas (2):
tests/qtest: Add a script to gather migration tests list tests/qtest: Pass migration tests individually to meson
.gitlab-ci.d/windows.yml | 2 +- tests/qtest/gen_migration_tests_list.py | 12 ++++++++ tests/qtest/meson.build | 40 +++++++++++++++++++------ 3 files changed, 44 insertions(+), 10 deletions(-) create mode 100644 tests/qtest/gen_migration_tests_list.py
Signature · 1 line
2.35.3
RFC PATCH 1/2 tests/qtest: Add a script to gather migration tests list
This is a workaround for limitations with meson.
We'd like to have each migration (sub)test registered with meson's test() function, but since they are all inside the single migration-test.c there's no way for meson to see the tests. We need an external way to generate the list of tests and pass it to meson.
We cannot call 'migration-test -l' because we'd need to build the migration-test binary first and while that is possible, there's no subsequent way to get the generated list into a meson variable for consumption. None of meson's routines support retrieving an arbitrary list of strings from a command at build (vs. configure) time.
We also cannot use generators to have 'migration-test' write to a file because that would happen at build time and meson does not support reading from a file in the build directory.
So the only approach left is to either have the list of tests committed into the source code or to grep the source code for the list. Committing the file would be harder to maintain and the test registration in migration-test.c is pretty static, so this patch uses the grep approach. But using python because it is more portable.
--- tests/qtest/gen_migration_tests_list.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 tests/qtest/gen_migration_tests_list.py
tests/qtest/gen_migration_tests_list.py new file +12 -0
@@ -0,0 +1,12 @@ +#!/usr/bin/env python3 + +import re +import sys + +file_path = sys.argv[1] + +with open(file_path, 'r') as f: + for line in f.readlines(): + match = re.search('\"(/migration/.*)\"', line) + if match: + print(match.groups()[0])
Signature · 1 line
2.35.3
RFC PATCH 2/2 tests/qtest: Pass migration tests individually to meson
In reply to Fabiano RosasHaving glib tests (qtest) all defined in a single test file makes it hard to know which test has failed when running CI. Create a new 'migration' test suite and move the migration tests individually to meson.
For now, use the global migration-test timeout value, but we could set a per-subtest timeout in the future.
Sample output:
$ ../configure --target-list=x86_64-softmmu,aarch64-softmmu ... $ make check-migration ... 36/469 qemu:migration / /x86_64/migration/precopy/unix/plain OK 34.25s 1 subtests passed 37/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/default-host OK 7.33s 1 subtests passed 39/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/allow-anon-client OK 7.32s 1 subtests passed 40/469 qemu:migration / /aarch64/migration/postcopy/compress/plain SKIP 0.04s 41/469 qemu:migration / /aarch64/migration/postcopy/recovery/compress/plain SKIP 0.04s 42/469 qemu:migration / /aarch64/migration/bad_dest OK 0.65s 1 subtests passed
--- .gitlab-ci.d/windows.yml | 2 +- tests/qtest/meson.build | 40 +++++++++++++++++++++++++++++++--------- 2 files changed, 32 insertions(+), 10 deletions(-)
.gitlab-ci.d/windows.yml +1 -1
@@ -84,7 +84,7 @@ msys2-64bit: - ..\msys64\usr\bin\bash -lc 'make' # qTests don't run successfully with "--without-default-devices", # so let's exclude the qtests from CI for now. - - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" || { cat meson-logs/testlog.txt; exit 1; } ;' + - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest migration\" || { cat meson-logs/testlog.txt; exit 1; } ;' msys2-32bit: extends: .shared_msys2_builder
tests/qtest/meson.build +31 -9
@@ -369,14 +369,36 @@ foreach dir : target_dirs test: executable(test, src, dependencies: deps) } endif - test('qtest-@0@/@1@'.format(target_base, test), - qtest_executables[test], - depends: [test_deps, qtest_emulator, emulator_modules], - env: qtest_env, - args: ['--tap', '-k'], - protocol: 'tap', - timeout: slow_qtests.get(test, 30), - priority: slow_qtests.get(test, 30), - suite: ['qtest', 'qtest-' + target_base]) + + migtest = 'migration-test' + if test == migtest + migtests = run_command(python, files('gen_migration_tests_list.py'), + meson.current_source_dir() / 'migration-test.c', + check: true) + + foreach item : migtests.stdout().strip().split('\n') + testname = '/@0@@1@'.format(target_base, item) + + test(testname, + qtest_executables['migration-test'], + depends: [test_deps, qtest_emulator, emulator_modules], + env: qtest_env, + args: ['-k', '-p', testname], + protocol: 'tap', + timeout: slow_qtests.get(migtest), + priority: slow_qtests.get(migtest), + suite: ['migration']) + endforeach + else + test('qtest-@0@/@1@'.format(target_base, test), + qtest_executables[test], + depends: [test_deps, qtest_emulator, emulator_modules], + env: qtest_env, + args: ['--tap', '-k'], + protocol: 'tap', + timeout: slow_qtests.get(test, 30), + priority: slow_qtests.get(test, 30), + suite: ['qtest', 'qtest-' + target_base]) + endif endforeach endforeach
Signature · 1 line
2.35.3
Quoted 3 lines · Fabiano Rosas
We've had some issues with the migration-test failing in CI and giving no clue as to which migration sub-test has failed. I think it would be an improvement to have each subtest listed individually.
If the migration-test (or any test) actually fails, the meson test log should show us exactly which test case failed already. We had some jobs which didn't publish the log as an artifact, but I think we finally have those all fixed.
If the migration-test hangs, however, then we get stuck until the GitLab CI job timeout and get no useful info. For this scenario we need to enable timeouts at the meson level, which are currently all disabled. I've got a series here to address this:
https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg00305.html
It needs
Quoted 4 lines
With this we can see which migration test has failed, can set timeouts individually and can run 'make check-migration' to run only the migration tests.
IMHO focusing just on migration-test is undesirable, as while this has been one of the least reliable ones, other tests do also exhibit hard to debug failures for much the same reasons. IOW, we need improved debugging across the board rather than specialcased solutions.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|