Skip to contentSkip to conversation

migration & CI: Add a CI job for migration compat testing

PATCH Superseded Fabiano Rosas 2 participants 7 messages Jan 4, 2024

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover migration & CI: Add a CI job for migration compat testing

Here's the second half of adding a migration compatibility test to CI.

We've already added support for running the full set of migration tests with two QEMU binaries since commit 5050ad2a380 ("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.

I included patches that solve the problem of testing older QEMUs with new test code. The old QEMU might lack features, bug fixes, etc. that the tests expect to be present. After this series we can specify a minimal QEMU version that a specific test supports.

changes since v1:
- fixed version comparison to be useful for the development tree
- s/who/qts/g

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1126977964

v1:
https://lore.kernel.org/r/[email protected]

Fabiano Rosas (4):
tests/qtest: Add a helper to query the QEMU version tests/qtest/migration: Add infrastructure to skip tests on older QEMUs tests/qtest/migration: Adapt tests to use older QEMUs
ci: Add a migration compatibility test job

 .gitlab-ci.d/buildtest.yml      | 53 +++++++++++++++++++++++++++
 tests/qtest/libqtest.c          | 24 +++++++++++++
 tests/qtest/libqtest.h          | 10 ++++++
 tests/qtest/migration-helpers.c | 11 ++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 63 +++++++++++++++++++++++++++++----
 6 files changed, 155 insertions(+), 7 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH v2 1/4 tests/qtest: Add a helper to query the QEMU version

---
 tests/qtest/libqtest.c | 24 ++++++++++++++++++++++++
 tests/qtest/libqtest.h | 10 ++++++++++
 2 files changed, 34 insertions(+)
tests/qtest/libqtest.c +24 -0
index f33a210861..af779a3248 100644
@@ -337,6 +337,30 @@ void qtest_remove_abrt_handler(void *data)
     }
 }
 
+void qtest_query_version(QTestState *qts, int *major, int *minor, int *micro)
+{
+    QDict *rsp, *version;
+
+    rsp = qtest_qmp_assert_success_ref(qts, "{ 'execute': 'query-version' }");
+    g_assert(rsp);
+
+    version = qdict_get_qdict(rsp, "qemu");
+
+    if (major) {
+        *major = qdict_get_int(version, "major");
+    }
+
+    if (minor) {
+        *minor = qdict_get_int(version, "minor");
+    }
+
+    if (micro) {
+        *micro = qdict_get_int(version, "micro");
+    }
+
+    qobject_unref(rsp);
+}
+
 static const char *qtest_qemu_binary(const char *var)
 {
     const char *qemu_bin;
tests/qtest/libqtest.h +10 -0
index 6e3d3525bf..db7a780d26 100644
@@ -1085,4 +1085,14 @@ bool have_qemu_img(void);
  */
 bool mkimg(const char *file, const char *fmt, unsigned size_mb);
 
+/**
+ * qtest_query_version:
+ * @who: QTestState instance to operate on
+ * @major: Pointer to where to store the major version number
+ * @minor: Pointer to where to store the minor version number
+ * @micro: Pointer to where to store the micro version number
+ *
+ */
+void qtest_query_version(QTestState *who, int *major, int *minor, int *micro);
+
 #endif
Signature · 1 line
2.35.3
#2 lore raw

PATCH v2 2/4 tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

In reply to Fabiano Rosas

We can run the migration tests with two different QEMU binaries to test migration compatibility between QEMU versions. This means we'll be running the tests with an older QEMU in either source or destination.

We need to avoid trying to test functionality that is unknown to the older QEMU. This could mean new features, bug fixes, error message changes, QEMU command line changes, migration API changes, etc.

Add a 'since' argument to the tests that inform when the functionality that is being test has been added to QEMU so we can skip the test on older versions.

Also add a version comparison function so we can adapt test code depending on the QEMU binary version being used.

---
 tests/qtest/migration-helpers.c | 11 +++++++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 29 +++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+)
tests/qtest/migration-helpers.c +11 -0
index 24fb7b3525..20220bfda0 100644
@@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
 
     return find_common_machine_version(machine_name, var1, var2);
 }
+
+int migration_vercmp(QTestState *who, const char *tgt_version)
+{
+    int major, minor, micro;
+    g_autofree char *version = NULL;
+
+    qtest_query_version(who, &major, &minor, &micro);
+    version = g_strdup_printf("%d.%d", major, minor + !!micro);
+
+    return strcmp(version, tgt_version);
+}
tests/qtest/migration-helpers.h +1 -0
index e31dc85cc7..7b4f8e851e 100644
@@ -47,4 +47,5 @@ char *find_common_machine_version(const char *mtype, const char *var1,
                                   const char *var2);
 char *resolve_machine_version(const char *alias, const char *var1,
                               const char *var2);
+int migration_vercmp(QTestState *who, const char *tgt_version);
 #endif /* MIGRATION_HELPERS_H */
tests/qtest/migration-test.c +29 -0
index d520c587f7..001470238b 100644
@@ -637,6 +637,12 @@ typedef struct {
     bool use_dirty_ring;
     const char *opts_source;
     const char *opts_target;
+    /*
+     * If a test checks against new functionality that might not be
+     * present in older QEMUs this needs to be set so we can skip
+     * running it when doing compatibility testing.
+     */
+    const char *since;
 } MigrateStart;
 
 /*
@@ -850,6 +856,17 @@ static int test_migrate_start(QTestState **from, QTestState **to,
         qtest_qmp_set_event_callback(*from,
                                      migrate_watch_for_stop,
                                      &got_src_stop);
+
+        if (args->since && migration_vercmp(*from, args->since) < 0) {
+            g_autofree char *msg = NULL;
+
+            msg = g_strdup_printf("Test requires at least QEMU version %s",
+                                  args->since);
+            g_test_skip(msg);
+            qtest_quit(*from);
+
+            return -1;
+        }
     }
 
     cmd_target = g_strdup_printf("-accel kvm%s -accel tcg "
@@ -872,6 +889,18 @@ static int test_migrate_start(QTestState **from, QTestState **to,
                                  migrate_watch_for_resume,
                                  &got_dst_resume);
 
+    if (args->since && migration_vercmp(*to, args->since) < 0) {
+        g_autofree char *msg = NULL;
+
+        msg = g_strdup_printf("Test requires at least QEMU version %s",
+                              args->since);
+        g_test_skip(msg);
+        qtest_quit(*to);
+        qtest_quit(*from);
+
+        return -1;
+    }
+
     /*
      * Remove shmem file immediately to avoid memory leak in test failed case.
      * It's valid because QEMU has already opened this file
Signature · 1 line
2.35.3
#3 lore raw

Re: [PATCH v2 2/4] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

Quoted 31 lines · Fabiano Rosas

We can run the migration tests with two different QEMU binaries to test migration compatibility between QEMU versions. This means we'll be running the tests with an older QEMU in either source or destination.

We need to avoid trying to test functionality that is unknown to the older QEMU. This could mean new features, bug fixes, error message changes, QEMU command line changes, migration API changes, etc.

Add a 'since' argument to the tests that inform when the functionality that is being test has been added to QEMU so we can skip the test on older versions.

Also add a version comparison function so we can adapt test code depending on the QEMU binary version being used.

---
 tests/qtest/migration-helpers.c | 11 +++++++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 29 +++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+)
tests/qtest/migration-helpers.c +0 -0
index 24fb7b3525..20220bfda0 100644
@@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
 
     return find_common_machine_version(machine_name, var1, var2);
 }

tests/qtest/migration-helpers.c char *resolve_machine_version(const char *alias, const char *var1, · line 304

Quoted 10 lines
tests/qtest/migration-helpers.c +10 -0
+
+int migration_vercmp(QTestState *who, const char *tgt_version)
+{
+    int major, minor, micro;
+    g_autofree char *version = NULL;
+
+    qtest_query_version(who, &major, &minor, &micro);
+    version = g_strdup_printf("%d.%d", major, minor + !!micro);
+
+    return strcmp(version, tgt_version);

Alphabetical version comparison will fail in 2025 when we hit QEMU 10.0, as 10.0 will compare older than 9.0

Quoted 68 lines
tests/qtest/migration-helpers.c +1 -0
+}
tests/qtest/migration-helpers.h +1 -0
index e31dc85cc7..7b4f8e851e 100644
@@ -47,4 +47,5 @@ char *find_common_machine_version(const char *mtype, const char *var1,
                                   const char *var2);
 char *resolve_machine_version(const char *alias, const char *var1,
                               const char *var2);
+int migration_vercmp(QTestState *who, const char *tgt_version);
 #endif /* MIGRATION_HELPERS_H */
tests/qtest/migration-test.c +29 -0
index d520c587f7..001470238b 100644
@@ -637,6 +637,12 @@ typedef struct {
     bool use_dirty_ring;
     const char *opts_source;
     const char *opts_target;
+    /*
+     * If a test checks against new functionality that might not be
+     * present in older QEMUs this needs to be set so we can skip
+     * running it when doing compatibility testing.
+     */
+    const char *since;
 } MigrateStart;
 
 /*
@@ -850,6 +856,17 @@ static int test_migrate_start(QTestState **from, QTestState **to,
         qtest_qmp_set_event_callback(*from,
                                      migrate_watch_for_stop,
                                      &got_src_stop);
+
+        if (args->since && migration_vercmp(*from, args->since) < 0) {
+            g_autofree char *msg = NULL;
+
+            msg = g_strdup_printf("Test requires at least QEMU version %s",
+                                  args->since);
+            g_test_skip(msg);
+            qtest_quit(*from);
+
+            return -1;
+        }
     }
 
     cmd_target = g_strdup_printf("-accel kvm%s -accel tcg "
@@ -872,6 +889,18 @@ static int test_migrate_start(QTestState **from, QTestState **to,
                                  migrate_watch_for_resume,
                                  &got_dst_resume);
 
+    if (args->since && migration_vercmp(*to, args->since) < 0) {
+        g_autofree char *msg = NULL;
+
+        msg = g_strdup_printf("Test requires at least QEMU version %s",
+                              args->since);
+        g_test_skip(msg);
+        qtest_quit(*to);
+        qtest_quit(*from);
+
+        return -1;
+    }
+
     /*
      * Remove shmem file immediately to avoid memory leak in test failed case.
      * It's valid because QEMU has already opened this file
Signature · 1 line
2.35.3

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw

Re: [PATCH v2 2/4] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

Quoted 30 lines · Daniel P. Berrangé
Quoted 29 lines · Fabiano Rosas

We can run the migration tests with two different QEMU binaries to test migration compatibility between QEMU versions. This means we'll be running the tests with an older QEMU in either source or destination.

We need to avoid trying to test functionality that is unknown to the older QEMU. This could mean new features, bug fixes, error message changes, QEMU command line changes, migration API changes, etc.

Add a 'since' argument to the tests that inform when the functionality that is being test has been added to QEMU so we can skip the test on older versions.

Also add a version comparison function so we can adapt test code depending on the QEMU binary version being used.

---
 tests/qtest/migration-helpers.c | 11 +++++++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 29 +++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+)
tests/qtest/migration-helpers.c +0 -0
index 24fb7b3525..20220bfda0 100644
@@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
 
Quoted 16 lines

tests/qtest/migration-helpers.c char *resolve_machine_version(const char *alias, const char *var1, · line 304

Quoted 12 lines
tests/qtest/migration-helpers.c +10 -0
     return find_common_machine_version(machine_name, var1, var2);
 }
+
+int migration_vercmp(QTestState *who, const char *tgt_version)
+{
+    int major, minor, micro;
+    g_autofree char *version = NULL;
+
+    qtest_query_version(who, &major, &minor, &micro);
+    version = g_strdup_printf("%d.%d", major, minor + !!micro);
+
+    return strcmp(version, tgt_version);

Alphabetical version comparison will fail in 2025 when we hit QEMU 10.0, as 10.0 will compare older than 9.0

Indeed, I'll fix it.

Thanks

#5 lore raw

PATCH v2 3/4 tests/qtest/migration: Adapt tests to use older QEMUs

In reply to Fabiano Rosas

Add the 'since' annotations to recently added tests and adapt the postcopy test to use the older "uri" API when needed.

---
 tests/qtest/migration-test.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
tests/qtest/migration-test.c +27 -7
index 001470238b..599f51f978 100644
@@ -1338,14 +1338,21 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
     migrate_ensure_non_converge(from);
 
     migrate_prepare_for_dirty_mem(from);
-    qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
-                             "  'arguments': { "
-                             "      'channels': [ { 'channel-type': 'main',"
-                             "      'addr': { 'transport': 'socket',"
-                             "                'type': 'inet',"
-                             "                'host': '127.0.0.1',"
-                             "                'port': '0' } } ] } }");
 
+    /* New syntax was introduced in 8.2 */
+    if (migration_vercmp(to, "8.2") < 0) {
+        qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
+                                 "  'arguments': { "
+                                 "      'uri': 'tcp:127.0.0.1:0' } }");
+    } else {
+        qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
+                                 "  'arguments': { "
+                                 "      'channels': [ { 'channel-type': 'main',"
+                                 "      'addr': { 'transport': 'socket',"
+                                 "                'type': 'inet',"
+                                 "                'host': '127.0.0.1',"
+                                 "                'port': '0' } } ] } }");
+    }
     /* Wait for the first serial output from the source */
     wait_for_serial("src_serial");
 
@@ -1603,6 +1610,9 @@ static void test_postcopy_recovery_double_fail(void)
 {
     MigrateCommon args = {
         .postcopy_recovery_test_fail = true,
+        .start = {
+            .since = "8.2",
+        },
     };
 
     test_postcopy_recovery_common(&args);
@@ -1665,6 +1675,7 @@ static void test_analyze_script(void)
 {
     MigrateStart args = {
         .opts_source = "-uuid 11111111-1111-1111-1111-111111111111",
+        .since = "8.2",
     };
     QTestState *from, *to;
     g_autofree char *uri = NULL;
@@ -2090,6 +2101,9 @@ static void test_precopy_file(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, true);
@@ -2134,6 +2148,9 @@ static void test_precopy_file_offset(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .finish_hook = file_offset_finish_hook,
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, false);
@@ -2148,6 +2165,9 @@ static void test_precopy_file_offset_bad(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .result = MIG_TEST_QMP_ERROR,
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, false);
Signature · 1 line
2.35.3
#6 lore raw

PATCH v2 4/4 ci: Add a migration compatibility test job

In reply to Fabiano Rosas

The migration tests have support for being passed two QEMU binaries to test migration compatibility.

Add a CI job that builds the lastest release of QEMU and another job that uses that version plus an already present build of the current version and run the migration tests with the two, both as source and destination. I.e.:

old QEMU (n-1) -> current QEMU (development tree) current QEMU (development tree) -> old QEMU (n-1)

The purpose of this CI job is to ensure the code we're about to merge will not cause a migration compatibility problem when migrating the next release (which will contain that code) to/from the previous release.

I'm leaving the jobs as manual for now because using an older QEMU in tests could hit bugs that were already fixed in the current development tree and we need to handle those case-by-case.

Note: for user forks, the version tags need to be pushed to gitlab otherwise it won't be able to checkout a different version.

---
 .gitlab-ci.d/buildtest.yml | 53 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
.gitlab-ci.d/buildtest.yml +53 -0
index 91663946de..81163a3f6a 100644
@@ -167,6 +167,59 @@ build-system-centos:
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
 
+build-previous-qemu:
+  extends: .native_build_job_template
+  artifacts:
+    when: on_success
+    expire_in: 2 days
+    paths:
+      - build-previous
+    exclude:
+      - build-previous/**/*.p
+      - build-previous/**/*.a.p
+      - build-previous/**/*.fa.p
+      - build-previous/**/*.c.o
+      - build-previous/**/*.c.o.d
+      - build-previous/**/*.fa
+  needs:
+    job: amd64-opensuse-leap-container
+  variables:
+    QEMU_JOB_OPTIONAL: 1
+    IMAGE: opensuse-leap
+    TARGETS: x86_64-softmmu aarch64-softmmu
+  before_script:
+    - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
+    - git checkout $QEMU_PREV_VERSION
+  after_script:
+    - mv build build-previous
+
+.migration-compat-common:
+  extends: .common_test_job_template
+  needs:
+    - job: build-previous-qemu
+    - job: build-system-opensuse
+  allow_failure: true
+  variables:
+    QEMU_JOB_OPTIONAL: 1
+    IMAGE: opensuse-leap
+    MAKE_CHECK_ARGS: check-build
+  script:
+    - cd build
+    - QTEST_QEMU_BINARY_SRC=../build-previous/qemu-system-${TARGET}
+          QTEST_QEMU_BINARY=./qemu-system-${TARGET} ./tests/qtest/migration-test
+    - QTEST_QEMU_BINARY_DST=../build-previous/qemu-system-${TARGET}
+          QTEST_QEMU_BINARY=./qemu-system-${TARGET} ./tests/qtest/migration-test
+
+migration-compat-aarch64:
+  extends: .migration-compat-common
+  variables:
+    TARGET: aarch64
+
+migration-compat-x86_64:
+  extends: .migration-compat-common
+  variables:
+    TARGET: x86_64
+
 check-system-centos:
   extends: .native_test_job_template
   needs:
Signature · 1 line
2.35.3