Skip to contentSkip to conversation

tests/qtest: Run migration subtests via meson

RFC PATCH In review Fabiano Rosas 2 participants 4 messages Jul 3, 2023 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover tests/qtest: Run migration subtests via meson

We've had some issues with the migration-test failing in CI and giving no clue as to which migration sub-test has failed. I think it would be an improvement to have each subtest listed individually.

With this we can see which migration test has failed, can set timeouts individually and can run 'make check-migration' to run only the migration tests.

It does however increase the amount of lines emitted with make check, so let me know if this is an issue.

I had to include some hacking due to meson, but I think it is simple enough and worth the benefits.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/919324510

Fabiano Rosas (2):
tests/qtest: Add a script to gather migration tests list tests/qtest: Pass migration tests individually to meson

 .gitlab-ci.d/windows.yml                |  2 +-
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++
 tests/qtest/meson.build                 | 40 +++++++++++++++++++------
 3 files changed, 44 insertions(+), 10 deletions(-)
 create mode 100644 tests/qtest/gen_migration_tests_list.py
Signature · 1 line
2.35.3
#1 lore raw

RFC PATCH 1/2 tests/qtest: Add a script to gather migration tests list

This is a workaround for limitations with meson.

We'd like to have each migration (sub)test registered with meson's test() function, but since they are all inside the single migration-test.c there's no way for meson to see the tests. We need an external way to generate the list of tests and pass it to meson.

We cannot call 'migration-test -l' because we'd need to build the migration-test binary first and while that is possible, there's no subsequent way to get the generated list into a meson variable for consumption. None of meson's routines support retrieving an arbitrary list of strings from a command at build (vs. configure) time.

We also cannot use generators to have 'migration-test' write to a file because that would happen at build time and meson does not support reading from a file in the build directory.

So the only approach left is to either have the list of tests committed into the source code or to grep the source code for the list. Committing the file would be harder to maintain and the test registration in migration-test.c is pretty static, so this patch uses the grep approach. But using python because it is more portable.

---
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 tests/qtest/gen_migration_tests_list.py
tests/qtest/gen_migration_tests_list.py new file +12 -0
new file mode 100644
index 0000000000..5127f976fa
@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+
+import re
+import sys
+
+file_path = sys.argv[1]
+
+with open(file_path, 'r') as f:
+    for line in f.readlines():
+        match = re.search('\"(/migration/.*)\"', line)
+        if match:
+            print(match.groups()[0])
Signature · 1 line
2.35.3
#2 lore raw

RFC PATCH 2/2 tests/qtest: Pass migration tests individually to meson

In reply to Fabiano Rosas

Having glib tests (qtest) all defined in a single test file makes it hard to know which test has failed when running CI. Create a new 'migration' test suite and move the migration tests individually to meson.

For now, use the global migration-test timeout value, but we could set a per-subtest timeout in the future.

Sample output:

$ ../configure --target-list=x86_64-softmmu,aarch64-softmmu ...
$ make check-migration
...
36/469 qemu:migration / /x86_64/migration/precopy/unix/plain                       OK    34.25s   1 subtests passed
37/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/default-host        OK     7.33s   1 subtests passed
39/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/allow-anon-client   OK     7.32s   1 subtests passed
40/469 qemu:migration / /aarch64/migration/postcopy/compress/plain                 SKIP   0.04s
41/469 qemu:migration / /aarch64/migration/postcopy/recovery/compress/plain        SKIP   0.04s
42/469 qemu:migration / /aarch64/migration/bad_dest                                OK     0.65s   1 subtests passed
---
 .gitlab-ci.d/windows.yml |  2 +-
 tests/qtest/meson.build  | 40 +++++++++++++++++++++++++++++++---------
 2 files changed, 32 insertions(+), 10 deletions(-)
.gitlab-ci.d/windows.yml +1 -1
index f889a468b5..04bc74a20e 100644
@@ -84,7 +84,7 @@ msys2-64bit:
   - ..\msys64\usr\bin\bash -lc 'make'
   # qTests don't run successfully with "--without-default-devices",
   # so let's exclude the qtests from CI for now.
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" || { cat meson-logs/testlog.txt; exit 1; } ;'
+  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest migration\" || { cat meson-logs/testlog.txt; exit 1; } ;'
 
 msys2-32bit:
   extends: .shared_msys2_builder
tests/qtest/meson.build +31 -9
index 5fa6833ad7..43c140921c 100644
@@ -369,14 +369,36 @@ foreach dir : target_dirs
         test: executable(test, src, dependencies: deps)
       }
     endif
-    test('qtest-@0@/@1@'.format(target_base, test),
-         qtest_executables[test],
-         depends: [test_deps, qtest_emulator, emulator_modules],
-         env: qtest_env,
-         args: ['--tap', '-k'],
-         protocol: 'tap',
-         timeout: slow_qtests.get(test, 30),
-         priority: slow_qtests.get(test, 30),
-         suite: ['qtest', 'qtest-' + target_base])
+
+    migtest = 'migration-test'
+    if test == migtest
+      migtests = run_command(python, files('gen_migration_tests_list.py'),
+                             meson.current_source_dir() / 'migration-test.c',
+                             check: true)
+
+      foreach item : migtests.stdout().strip().split('\n')
+        testname = '/@0@@1@'.format(target_base, item)
+
+        test(testname,
+             qtest_executables['migration-test'],
+             depends: [test_deps, qtest_emulator, emulator_modules],
+             env: qtest_env,
+             args: ['-k', '-p', testname],
+             protocol: 'tap',
+             timeout: slow_qtests.get(migtest),
+             priority: slow_qtests.get(migtest),
+             suite: ['migration'])
+      endforeach
+    else
+      test('qtest-@0@/@1@'.format(target_base, test),
+           qtest_executables[test],
+           depends: [test_deps, qtest_emulator, emulator_modules],
+           env: qtest_env,
+           args: ['--tap', '-k'],
+           protocol: 'tap',
+           timeout: slow_qtests.get(test, 30),
+           priority: slow_qtests.get(test, 30),
+           suite: ['qtest', 'qtest-' + target_base])
+      endif
   endforeach
 endforeach
Signature · 1 line
2.35.3
#3 lore raw
In reply to Fabiano Rosas
Quoted 3 lines · Fabiano Rosas

We've had some issues with the migration-test failing in CI and giving no clue as to which migration sub-test has failed. I think it would be an improvement to have each subtest listed individually.

If the migration-test (or any test) actually fails, the meson test log should show us exactly which test case failed already. We had some jobs which didn't publish the log as an artifact, but I think we finally have those all fixed.

If the migration-test hangs, however, then we get stuck until the GitLab CI job timeout and get no useful info. For this scenario we need to enable timeouts at the meson level, which are currently all disabled. I've got a series here to address this:

https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg00305.html

It needs

Quoted 4 lines

With this we can see which migration test has failed, can set timeouts individually and can run 'make check-migration' to run only the migration tests.

IMHO focusing just on migration-test is undesirable, as while this has been one of the least reliable ones, other tests do also exhibit hard to debug failures for much the same reasons. IOW, we need improved debugging across the board rather than specialcased solutions.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|