Skip to contentSkip to conversation

migration: Reduce migration-test time for non-KVM archs

PATCH In review Fabiano Rosas 3 participants 5 messages Jan 30, 2025 – Feb 3, 2025 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover migration: Reduce migration-test time for non-KVM archs

Hi, continuing the work from the previous[1] series to reduce the time migration-test takes during make check, here's a couple of patches to create a smaller set of tests.

The change is that from now on, ./migration-test will only run a limited set of tests (~12), while the full set (64) requires that the --full flag is passed on. Only those calling the test manually need to care about this new flag.

For everyone else, qtest/meson.build will automatically add --full in case KVM is available for the target architecture. All other architectures run the small set.

The choice of tests for the small set has been arbitrarily made in patch 2. I expect that once we close in on a list of tests, new tests will only be added under the --full flag and the small set will be kept stable.

An unwritten but maybe obvious rule is that slow or flaky tests should not be added to the small set and once detected should be moved to the full set.

More details in patch 1.

Thanks!

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1647478788
1- https://lore.kernel.org/r/[email protected]

Fabiano Rosas (2):
tests/qtest/migration: Add --full option
tests/qtest/migration: Pick smoke tests

 tests/qtest/meson.build                   | 11 +++++++++-
 tests/qtest/migration-test.c              | 25 +++++++++++++++++++++++
 tests/qtest/migration/compression-tests.c | 15 +++++++++++---
 tests/qtest/migration/cpr-tests.c         |  6 ++++++
 tests/qtest/migration/file-tests.c        | 21 +++++++++++++------
 tests/qtest/migration/framework.h         |  1 +
 tests/qtest/migration/misc-tests.c        | 16 ++++++++++++---
 tests/qtest/migration/postcopy-tests.c    | 14 ++++++++++++-
 tests/qtest/migration/precopy-tests.c     | 23 ++++++++++++++-------
 tests/qtest/migration/tls-tests.c         | 14 +++++++++++--
 10 files changed, 123 insertions(+), 23 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/2 tests/qtest/migration: Add --full option

Add a new command line option to allow selecting between running the full set of tests or a smaller set of tests. The default will be to run the small set (i.e. no comand line option provided) so we can reduce the amount of tests run by default. Only hosts which support KVM for the target architecture being tested will run the complete set of tests.

Adjust the meson.build file to pass in the --full option when appropriate.

(for now, set the option unconditionally until the next patch actually creates the small set)

Use cases:

configure --target-list=aarch64-softmmu,ppc64-softmmu,s390x-softmmu,x86_64-softmmu

                        | before - 615s/244 tests  | after - 244s/100 tests
------------------------+--------------------------+-----------------------------
make check              | full set for all archs   | full set for the KVM arch,
make check-qtest        |                          | small set for the rest
                        |                          |
qemu-system-$ARCH       | full set for $ARCH       | small set for $ARCH, KVM or
./migration-test        |                          | TCG automatically chosen
                        |                          |
qemu-system-$ARCH       | N/A                      | full set for $ARCH, KVM or
./migration-test --full |                          | TCG automatically chosen
                        |                          |
migration-compat-x86_64 | full set for x86_64      | small set for x86_64
CI job                  |                          |

------------------------+--------------------------+-----------------------------

---
 tests/qtest/meson.build                   | 11 +++++++++-
 tests/qtest/migration-test.c              | 26 +++++++++++++++++++++++
 tests/qtest/migration/compression-tests.c |  4 ++++
 tests/qtest/migration/cpr-tests.c         |  4 ++++
 tests/qtest/migration/file-tests.c        |  4 ++++
 tests/qtest/migration/framework.h         |  1 +
 tests/qtest/migration/misc-tests.c        |  4 ++++
 tests/qtest/migration/postcopy-tests.c    |  4 ++++
 tests/qtest/migration/precopy-tests.c     |  4 ++++
 tests/qtest/migration/tls-tests.c         |  4 ++++
 10 files changed, 65 insertions(+), 1 deletion(-)
tests/qtest/meson.build +10 -1
index 94b28e5a53..0219b90a70 100644
@@ -402,6 +402,8 @@ foreach dir : target_dirs
   target_base = dir.split('-')[0]
   qtest_emulator = emulators['qemu-system-' + target_base]
   target_qtests = get_variable('qtests_' + target_base, []) + qtests_generic
+  has_kvm = ('CONFIG_KVM' in config_all_accel and host_os == 'linux'
+             and cpu == target_base and fs.exists('/dev/kvm'))
 
   test_deps = roms
   qtest_env = environment()
@@ -435,11 +437,18 @@ foreach dir : target_dirs
         test: executable(test, src, dependencies: deps)
       }
     endif
+
+    test_args = ['--tap', '-k']
+
+    if test == 'migration-test' and has_kvm
+      test_args += ['--full']
+    endif
+
     test('qtest-@0@/@1@'.format(target_base, test),
          qtest_executables[test],
          depends: [test_deps, qtest_emulator, emulator_modules],
          env: qtest_env,
-         args: ['--tap', '-k'],
+         args: test_args,
          protocol: 'tap',
          timeout: slow_qtests.get(test, 60),
          priority: slow_qtests.get(test, 60),
tests/qtest/migration-test.c +26 -0
index 5cad5060b3..61180fcbd5 100644
@@ -14,13 +14,39 @@
 #include "migration/framework.h"
 #include "qemu/module.h"
 
+static void parse_args(int *argc_p, char ***argv_p, bool *full_set)
+{
+    int argc = *argc_p;
+    char **argv = *argv_p;
+    int i, j;
+
+    j = 1;
+    for (i = 1; i < argc; i++) {
+        if (g_str_equal(argv[i], "--full")) {
+            *full_set = true;
+            continue;
+        }
+        argv[j++] = argv[i];
+        if (i >= j) {
+            argv[i] = NULL;
+        }
+    }
+    *argc_p = j;
+}
+
 int main(int argc, char **argv)
 {
     MigrationTestEnv *env;
     int ret;
+    bool full_set = false;
+
+    /* strip the --full option if it's present */
+    parse_args(&argc, &argv, &full_set);
 
     g_test_init(&argc, &argv, NULL);
     env = migration_get_env();
+    env->full_set = full_set;
+    env->full_set = true; /* temporary */
     module_call_init(MODULE_INIT_QOM);
 
     migration_test_add_tls(env);
tests/qtest/migration/compression-tests.c +4 -0
index d78f1f11f1..9d0a258d51 100644
@@ -155,6 +155,10 @@ void migration_test_add_compression(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
 #ifdef CONFIG_ZSTD
     migration_test_add("/migration/multifd/tcp/plain/zstd",
                        test_multifd_tcp_zstd);
tests/qtest/migration/cpr-tests.c +4 -0
index 44ce89aa5b..3faa180dfb 100644
@@ -48,6 +48,10 @@ void migration_test_add_cpr(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
     /*
      * Our CI system has problems with shared memory.
      * Don't run this test until we find a workaround.
tests/qtest/migration/file-tests.c +4 -0
index 84225c8c33..5b190853a5 100644
@@ -304,6 +304,10 @@ void migration_test_add_file(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
     migration_test_add("/migration/precopy/file",
                        test_precopy_file);
 
tests/qtest/migration/framework.h +1 -0
index 7991ee56b6..115f887f14 100644
@@ -24,6 +24,7 @@ typedef struct MigrationTestEnv {
     bool uffd_feature_thread_id;
     bool has_dirty_ring;
     bool is_x86;
+    bool full_set;
     const char *arch;
     const char *qemu_src;
     const char *qemu_dst;
tests/qtest/migration/misc-tests.c +4 -0
index 6173430748..1a57de409a 100644
@@ -255,6 +255,10 @@ void migration_test_add_misc(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
     migration_test_add("/migration/bad_dest", test_baddest);
 #ifndef _WIN32
     migration_test_add("/migration/analyze-script", test_analyze_script);
tests/qtest/migration/postcopy-tests.c +4 -0
index daf7449f2c..b08cde3270 100644
@@ -81,6 +81,10 @@ static void test_postcopy_preempt_recovery(void)
 
 void migration_test_add_postcopy(MigrationTestEnv *env)
 {
+    if (!env->full_set) {
+        return;
+    }
+
     if (env->has_uffd) {
         migration_test_add("/migration/postcopy/plain", test_postcopy);
         migration_test_add("/migration/postcopy/recovery/plain",
tests/qtest/migration/precopy-tests.c +4 -0
index 23599b29ee..a3cf813579 100644
@@ -955,6 +955,10 @@ void migration_test_add_precopy(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
     if (env->is_x86) {
         migration_test_add("/migration/precopy/unix/suspend/live",
                            test_precopy_unix_suspend_live);
tests/qtest/migration/tls-tests.c +4 -0
index 5704a1f992..aee56930ac 100644
@@ -726,6 +726,10 @@ void migration_test_add_tls(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    if (!env->full_set) {
+        return;
+    }
+
     migration_test_add("/migration/precopy/unix/tls/psk",
                        test_precopy_unix_tls_psk);
 
Signature · 1 line
2.35.3
#2 lore raw

PATCH 2/2 tests/qtest/migration: Pick smoke tests

In reply to Fabiano Rosas

Choose a few tests per group and move them from the full set to the smoke set.

---
 tests/qtest/migration-test.c              |  1 -
 tests/qtest/migration/compression-tests.c | 11 ++++++---
 tests/qtest/migration/cpr-tests.c         |  2 ++
 tests/qtest/migration/file-tests.c        | 17 +++++++++-----
 tests/qtest/migration/misc-tests.c        | 12 +++++++---
 tests/qtest/migration/postcopy-tests.c    | 18 ++++++++++-----
 tests/qtest/migration/precopy-tests.c     | 27 ++++++++++++++---------
 tests/qtest/migration/tls-tests.c         | 10 +++++++--
 8 files changed, 67 insertions(+), 31 deletions(-)
tests/qtest/migration-test.c +0 -1
index 61180fcbd5..0893687174 100644
@@ -46,7 +46,6 @@ int main(int argc, char **argv)
     g_test_init(&argc, &argv, NULL);
     env = migration_get_env();
     env->full_set = full_set;
-    env->full_set = true; /* temporary */
     module_call_init(MODULE_INIT_QOM);
 
     migration_test_add_tls(env);
tests/qtest/migration/compression-tests.c +8 -3
index 9d0a258d51..8b58401b84 100644
@@ -151,10 +151,18 @@ static void test_multifd_tcp_zlib(void)
     test_precopy_common(&args);
 }
 
+static void migration_test_add_compression_smoke(MigrationTestEnv *env)
+{
+    migration_test_add("/migration/multifd/tcp/plain/zlib",
+                       test_multifd_tcp_zlib);
+}
+
 void migration_test_add_compression(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    migration_test_add_compression_smoke(env);
+
     if (!env->full_set) {
         return;
     }
@@ -183,7 +191,4 @@ void migration_test_add_compression(MigrationTestEnv *env)
         migration_test_add("/migration/precopy/unix/xbzrle",
                            test_precopy_unix_xbzrle);
     }
-
-    migration_test_add("/migration/multifd/tcp/plain/zlib",
-                       test_multifd_tcp_zlib);
 }
tests/qtest/migration/cpr-tests.c +2 -0
index 3faa180dfb..34e6ac79bc 100644
@@ -48,6 +48,8 @@ void migration_test_add_cpr(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    /* no tests in the smoke set for now */
+
     if (!env->full_set) {
         return;
     }
tests/qtest/migration/file-tests.c +11 -6
index 5b190853a5..f474b3590a 100644
@@ -300,17 +300,25 @@ static void test_multifd_file_mapped_ram_fdset_dio(void)
 }
 #endif /* !_WIN32 */
 
+static void migration_test_add_file_smoke(MigrationTestEnv *env)
+{
+    migration_test_add("/migration/precopy/file",
+                       test_precopy_file);
+
+    migration_test_add("/migration/multifd/file/mapped-ram/dio",
+                       test_multifd_file_mapped_ram_dio);
+}
+
 void migration_test_add_file(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    migration_test_add_file_smoke(env);
+
     if (!env->full_set) {
         return;
     }
 
-    migration_test_add("/migration/precopy/file",
-                       test_precopy_file);
-
     migration_test_add("/migration/precopy/file/offset",
                        test_precopy_file_offset);
 #ifndef _WIN32
@@ -330,9 +338,6 @@ void migration_test_add_file(MigrationTestEnv *env)
     migration_test_add("/migration/multifd/file/mapped-ram/live",
                        test_multifd_file_mapped_ram_live);
 
-    migration_test_add("/migration/multifd/file/mapped-ram/dio",
-                       test_multifd_file_mapped_ram_dio);
-
 #ifndef _WIN32
     migration_test_add("/migration/multifd/file/mapped-ram/fdset",
                        test_multifd_file_mapped_ram_fdset);
tests/qtest/migration/misc-tests.c +9 -3
index 1a57de409a..f6e0687731 100644
@@ -251,18 +251,24 @@ static void test_validate_uri_channels_none_set(void)
     do_test_validate_uri_channel(&args);
 }
 
+static void migration_test_add_misc_smoke(MigrationTestEnv *env)
+{
+#ifndef _WIN32
+    migration_test_add("/migration/analyze-script", test_analyze_script);
+#endif
+}
+
 void migration_test_add_misc(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    migration_test_add_misc_smoke(env);
+
     if (!env->full_set) {
         return;
     }
 
     migration_test_add("/migration/bad_dest", test_baddest);
-#ifndef _WIN32
-    migration_test_add("/migration/analyze-script", test_analyze_script);
-#endif
 
     /*
      * Our CI system has problems with shared memory.
tests/qtest/migration/postcopy-tests.c +13 -5
index b08cde3270..7e84e60fea 100644
@@ -79,18 +79,26 @@ static void test_postcopy_preempt_recovery(void)
     test_postcopy_recovery_common(&args);
 }
 
-void migration_test_add_postcopy(MigrationTestEnv *env)
+static void migration_test_add_postcopy_smoke(MigrationTestEnv *env)
 {
-    if (!env->full_set) {
-        return;
-    }
-
     if (env->has_uffd) {
         migration_test_add("/migration/postcopy/plain", test_postcopy);
         migration_test_add("/migration/postcopy/recovery/plain",
                            test_postcopy_recovery);
         migration_test_add("/migration/postcopy/preempt/plain",
                            test_postcopy_preempt);
+    }
+}
+
+void migration_test_add_postcopy(MigrationTestEnv *env)
+{
+    migration_test_add_postcopy_smoke(env);
+
+    if (!env->full_set) {
+        return;
+    }
+
+    if (env->has_uffd) {
         migration_test_add("/migration/postcopy/preempt/recovery/plain",
                            test_postcopy_preempt_recovery);
 
tests/qtest/migration/precopy-tests.c +16 -11
index a3cf813579..1fa114bd90 100644
@@ -951,14 +951,8 @@ static void test_dirty_limit(void)
     migrate_end(from, to, true);
 }
 
-void migration_test_add_precopy(MigrationTestEnv *env)
+static void migration_test_add_precopy_smoke(MigrationTestEnv *env)
 {
-    tmpfs = env->tmpfs;
-
-    if (!env->full_set) {
-        return;
-    }
-
     if (env->is_x86) {
         migration_test_add("/migration/precopy/unix/suspend/live",
                            test_precopy_unix_suspend_live);
@@ -970,6 +964,21 @@ void migration_test_add_precopy(MigrationTestEnv *env)
                        test_precopy_unix_plain);
 
     migration_test_add("/migration/precopy/tcp/plain", test_precopy_tcp_plain);
+    migration_test_add("/migration/multifd/tcp/uri/plain/none",
+                       test_multifd_tcp_uri_none);
+    migration_test_add("/migration/multifd/tcp/plain/cancel",
+                       test_multifd_tcp_cancel);
+}
+
+void migration_test_add_precopy(MigrationTestEnv *env)
+{
+    tmpfs = env->tmpfs;
+
+    migration_test_add_precopy_smoke(env);
+
+    if (!env->full_set) {
+        return;
+    }
 
     migration_test_add("/migration/precopy/tcp/plain/switchover-ack",
                        test_precopy_tcp_switchover_ack);
@@ -993,16 +1002,12 @@ void migration_test_add_precopy(MigrationTestEnv *env)
                                test_dirty_limit);
         }
     }
-    migration_test_add("/migration/multifd/tcp/uri/plain/none",
-                       test_multifd_tcp_uri_none);
     migration_test_add("/migration/multifd/tcp/channels/plain/none",
                        test_multifd_tcp_channels_none);
     migration_test_add("/migration/multifd/tcp/plain/zero-page/legacy",
                        test_multifd_tcp_zero_page_legacy);
     migration_test_add("/migration/multifd/tcp/plain/zero-page/none",
                        test_multifd_tcp_no_zero_page);
-    migration_test_add("/migration/multifd/tcp/plain/cancel",
-                       test_multifd_tcp_cancel);
     if (g_str_equal(env->arch, "x86_64")
         && env->has_kvm && env->has_dirty_ring) {
 
tests/qtest/migration/tls-tests.c +8 -2
index aee56930ac..2cb4a44bcd 100644
@@ -722,10 +722,18 @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
 }
 #endif /* CONFIG_TASN1 */
 
+static void migration_test_add_tls_smoke(MigrationTestEnv *env)
+{
+    migration_test_add("/migration/precopy/tcp/tls/psk/match",
+                       test_precopy_tcp_tls_psk_match);
+}
+
 void migration_test_add_tls(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
+    migration_test_add_tls_smoke(env);
+
     if (!env->full_set) {
         return;
     }
@@ -755,8 +763,6 @@ void migration_test_add_tls(MigrationTestEnv *env)
                        test_precopy_unix_tls_x509_override_host);
 #endif /* CONFIG_TASN1 */
 
-    migration_test_add("/migration/precopy/tcp/tls/psk/match",
-                       test_precopy_tcp_tls_psk_match);
     migration_test_add("/migration/precopy/tcp/tls/psk/mismatch",
                        test_precopy_tcp_tls_psk_mismatch);
 #ifdef CONFIG_TASN1
Signature · 1 line
2.35.3
Reviewed-by on the v1 series
#3 lore raw
In reply to Fabiano Rosas
Quoted 18 lines · Fabiano Rosas

Hi, continuing the work from the previous[1] series to reduce the time migration-test takes during make check, here's a couple of patches to create a smaller set of tests.

The change is that from now on, ./migration-test will only run a limited set of tests (~12), while the full set (64) requires that the --full flag is passed on. Only those calling the test manually need to care about this new flag.

For everyone else, qtest/meson.build will automatically add --full in case KVM is available for the target architecture. All other architectures run the small set.

The choice of tests for the small set has been arbitrarily made in patch 2. I expect that once we close in on a list of tests, new tests will only be added under the --full flag and the small set will be kept stable.

Quoted 5 lines

An unwritten but maybe obvious rule is that slow or flaky tests should not be added to the small set and once detected should be moved to the full set.

More details in patch 1.

Looks all good here, thanks!

Signature · 1 line
Peter Xu
#4 lore raw
In reply to Fabiano Rosas
Quoted 3 lines · Fabiano Rosas

Hi, continuing the work from the previous[1] series to reduce the time migration-test takes during make check, here's a couple of patches to create a smaller set of tests.

Queued to testing/next, thanks.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro