Skip to contentSkip to conversation

migration & CI: Add a CI job for migration compat testing

PATCH In review Fabiano Rosas 2 participants 10 messages Jan 18, 2024 – Jan 22, 2024

lore
Share & tools
lore
#0 lore raw

v4 cover migration & CI: Add a CI job for migration compat testing

Here's the second half of adding a migration compatibility test to CI.

We've already added support for running the full set of migration tests with two QEMU binaries since commit 5050ad2a380 ("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.

changes since v3:

- Removed all the 'since' logic and started using the n-1 version of the tests.

- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.

- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.

v3:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]

Fabiano Rosas (3):
tests/qtest/migration: Don't use -cpu max for aarch64
ci: Add a migration compatibility test job
ci: Disable migration compatibility tests for aarch64

 .gitlab-ci.d/buildtest.yml   | 64 ++++++++++++++++++++++++++++++++++++
 tests/qtest/migration-test.c |  2 +-
 2 files changed, 65 insertions(+), 1 deletion(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH v4 1/3 tests/qtest/migration: Don't use -cpu max for aarch64

The 'max' cpu is not expected to be stable in terms of features across QEMU versions, so it should not be expected to migrate.

While the tests currently all pass with -cpu max, that is only because we're not testing across QEMU versions, which is the more common use-case for migration.

We've recently introduced compatibility tests that use two different QEMU versions and the tests are now failing for aarch64. The next patch adds those tests to CI, so we cannot use the 'max' cpu anymore. Replace it with the 'neoverse-n1', which has a fixed set of features.

---
 tests/qtest/migration-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/migration-test.c +1 -1
index d3066e119f..b2390656f0 100644
@@ -820,7 +820,7 @@ static int test_migrate_start(QTestState **from, QTestState **to,
         memory_size = "150M";
         machine_alias = "virt";
         machine_opts = "gic-version=max";
-        arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath);
+        arch_opts = g_strdup_printf("-cpu neoverse-n1 -kernel %s", bootpath);
         start_address = ARM_TEST_MEM_START;
         end_address = ARM_TEST_MEM_END;
     } else {
Signature · 1 line
2.35.3
#2 lore raw

PATCH v4 2/3 ci: Add a migration compatibility test job

In reply to Fabiano Rosas

The migration tests have support for being passed two QEMU binaries to test migration compatibility.

Add a CI job that builds the lastest release of QEMU and another job that uses that version plus an already present build of the current version and run the migration tests with the two, both as source and destination. I.e.:

old QEMU (n-1) -> current QEMU (development tree) current QEMU (development tree) -> old QEMU (n-1)

The purpose of this CI job is to ensure the code we're about to merge will not cause a migration compatibility problem when migrating the next release (which will contain that code) to/from the previous release.

The version of migration-test used will be the one matching the older QEMU. That way we can avoid special-casing new tests that wouldn't be compatible with the older QEMU.

Note: for user forks, the version tags need to be pushed to gitlab otherwise it won't be able to checkout a different version.

---
 .gitlab-ci.d/buildtest.yml | 60 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
.gitlab-ci.d/buildtest.yml +60 -0
index e1c7801598..f0b0edc634 100644
@@ -167,6 +167,66 @@ build-system-centos:
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
 
+# Previous QEMU release. Used for cross-version migration tests.
+build-previous-qemu:
+  extends: .native_build_job_template
+  artifacts:
+    when: on_success
+    expire_in: 2 days
+    paths:
+      - build-previous
+    exclude:
+      - build-previous/**/*.p
+      - build-previous/**/*.a.p
+      - build-previous/**/*.fa.p
+      - build-previous/**/*.c.o
+      - build-previous/**/*.c.o.d
+      - build-previous/**/*.fa
+  needs:
+    job: amd64-opensuse-leap-container
+  variables:
+    IMAGE: opensuse-leap
+    TARGETS: x86_64-softmmu aarch64-softmmu
+  before_script:
+    - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
+    - git checkout $QEMU_PREV_VERSION
+  after_script:
+    - mv build build-previous
+
+.migration-compat-common:
+  extends: .common_test_job_template
+  needs:
+    - job: build-previous-qemu
+    - job: build-system-opensuse
+  # The old QEMU could have bugs unrelated to migration that are
+  # already fixed in the current development branch, so this test
+  # might fail.
+  allow_failure: true
+  variables:
+    IMAGE: opensuse-leap
+    MAKE_CHECK_ARGS: check-build
+  script:
+    # Use the migration-tests from the older QEMU tree. This avoids
+    # testing an old QEMU against new features/tests that it is not
+    # compatible with.
+    - cd build-previous
+    # old to new
+    - QTEST_QEMU_BINARY_SRC=./qemu-system-${TARGET}
+          QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
+    # new to old
+    - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}
+          QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
+
+migration-compat-aarch64:
+  extends: .migration-compat-common
+  variables:
+    TARGET: aarch64
+
+migration-compat-x86_64:
+  extends: .migration-compat-common
+  variables:
+    TARGET: x86_64
+
 check-system-centos:
   extends: .native_test_job_template
   needs:
Signature · 1 line
2.35.3
#3 lore raw

PATCH v4 3/3 ci: Disable migration compatibility tests for aarch64

In reply to Fabiano Rosas

Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.

We can revert this patch when 9.0 releases.

---
 .gitlab-ci.d/buildtest.yml | 4 ++++
 1 file changed, 4 insertions(+)
.gitlab-ci.d/buildtest.yml +4 -0
index f0b0edc634..b344a4685f 100644
@@ -217,10 +217,14 @@ build-previous-qemu:
     - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}
           QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
 
+# This job is disabled until we release 9.0. The existing
+# migration-test in 8.2 is broken on aarch64. The fix was already
+# commited, but it will only take effect once 9.0 is out.
 migration-compat-aarch64:
   extends: .migration-compat-common
   variables:
     TARGET: aarch64
+    QEMU_JOB_OPTIONAL: 1
 
 migration-compat-x86_64:
   extends: .migration-compat-common
Signature · 1 line
2.35.3
#4 lore raw

Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64

Quoted 18 lines · Fabiano Rosas

Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.

We can revert this patch when 9.0 releases.

---
 .gitlab-ci.d/buildtest.yml | 4 ++++
 1 file changed, 4 insertions(+)
.gitlab-ci.d/buildtest.yml +0 -0
index f0b0edc634..b344a4685f 100644
@@ -217,10 +217,14 @@ build-previous-qemu:
     - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}

.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227

Quoted 10 lines
.gitlab-ci.d/buildtest.yml +4 -0
           QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
 
+# This job is disabled until we release 9.0. The existing
+# migration-test in 8.2 is broken on aarch64. The fix was already
+# commited, but it will only take effect once 9.0 is out.
 migration-compat-aarch64:
   extends: .migration-compat-common
   variables:
     TARGET: aarch64
+    QEMU_JOB_OPTIONAL: 1

Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.

Would QEMU_JOB_SKIPPED suits more here?

https://www.qemu.org/docs/master/devel/ci.html

QEMU_JOB_OPTIONAL

The job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.

QEMU_JOB_SKIPPED

The job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.

I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.

Quoted 6 lines
.gitlab-ci.d/buildtest.yml +0 -0
 migration-compat-x86_64:
   extends: .migration-compat-common
Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
#5 lore raw

Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64

Quoted 21 lines · Peter Xu
Quoted 20 lines · Fabiano Rosas

Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.

We can revert this patch when 9.0 releases.

---
 .gitlab-ci.d/buildtest.yml | 4 ++++
 1 file changed, 4 insertions(+)
.gitlab-ci.d/buildtest.yml +0 -0
index f0b0edc634..b344a4685f 100644
@@ -217,10 +217,14 @@ build-previous-qemu:
     - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}
           QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
 
Quoted 12 lines

.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227

Quoted 8 lines
.gitlab-ci.d/buildtest.yml +4 -0
+# This job is disabled until we release 9.0. The existing
+# migration-test in 8.2 is broken on aarch64. The fix was already
+# commited, but it will only take effect once 9.0 is out.
 migration-compat-aarch64:
   extends: .migration-compat-common
   variables:
     TARGET: aarch64
+    QEMU_JOB_OPTIONAL: 1

Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.

I want to avoid having to think about this when 9.0 is out. This way it'll be just a revert of this patch.

Quoted 14 lines

Would QEMU_JOB_SKIPPED suits more here?

https://www.qemu.org/docs/master/devel/ci.html

QEMU_JOB_OPTIONAL

The job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.

QEMU_JOB_SKIPPED

Quoted 8 lines

The job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.

I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.

Ok, we can use SKIPPED.

Quoted 6 lines
Quoted 6 lines
.gitlab-ci.d/buildtest.yml +0 -0
 migration-compat-x86_64:
   extends: .migration-compat-common
Signature · 1 line
2.35.3
#6 lore raw

Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64

Quoted 36 lines · Fabiano Rosas
Quoted 21 lines · Peter Xu
Quoted 20 lines · Fabiano Rosas

Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.

We can revert this patch when 9.0 releases.

---
 .gitlab-ci.d/buildtest.yml | 4 ++++
 1 file changed, 4 insertions(+)
.gitlab-ci.d/buildtest.yml +0 -0
index f0b0edc634..b344a4685f 100644
@@ -217,10 +217,14 @@ build-previous-qemu:
     - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}
           QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
 
Quoted 12 lines

.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227

Quoted 8 lines
.gitlab-ci.d/buildtest.yml +4 -0
+# This job is disabled until we release 9.0. The existing
+# migration-test in 8.2 is broken on aarch64. The fix was already
+# commited, but it will only take effect once 9.0 is out.
 migration-compat-aarch64:
   extends: .migration-compat-common
   variables:
     TARGET: aarch64
+    QEMU_JOB_OPTIONAL: 1

Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.

Quoted 2 lines

I want to avoid having to think about this when 9.0 is out. This way it'll be just a revert of this patch.

That's ok.

Quoted 20 lines
Quoted 20 lines

Would QEMU_JOB_SKIPPED suits more here?

https://www.qemu.org/docs/master/devel/ci.html

QEMU_JOB_OPTIONAL

The job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.

QEMU_JOB_SKIPPED

The job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.

Quoted 4 lines
Quoted 3 lines

I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.

Ok, we can use SKIPPED.

No need to repost then; if this is the only thing to change I can do it when queuing. Will wait a bit more for comment on patch 2.

Signature · 1 line
Peter Xu
#7 lore raw
In reply to Fabiano Rosas

Fabiano,

Quoted 18 lines · Fabiano Rosas

Here's the second half of adding a migration compatibility test to CI.

We've already added support for running the full set of migration tests with two QEMU binaries since commit 5050ad2a380 ("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.

changes since v3:

- Removed all the 'since' logic and started using the n-1 version of the tests.

- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.

- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.

Quoted 6 lines

v3:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]

I gave it a try but build-previous-qemu somehow failed..

https://gitlab.com/peterx/qemu/-/jobs/5965634871

You normally attach a "CI run" which I wanted to reference, but it's gone in this v4 unfortunately. The error:

error: pathspec 'v8.2.0' did not match any file(s) known to git

is pretty weird, and I had a feeling it was something wrong before that. Maybe it's because I tried to run this specific test multiple times (and all failed), somehow the wrong thing was cached, so the real error (of the 1st run when there is no cache) could have got lost. Not sure.

Please have a look when you have time, thanks.

Quoted 13 lines

Fabiano Rosas (3):
tests/qtest/migration: Don't use -cpu max for aarch64
ci: Add a migration compatibility test job
ci: Disable migration compatibility tests for aarch64

 .gitlab-ci.d/buildtest.yml   | 64 ++++++++++++++++++++++++++++++++++++
 tests/qtest/migration-test.c |  2 +-
 2 files changed, 65 insertions(+), 1 deletion(-)
Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
#8 lore raw
Quoted 28 lines · Peter Xu

Fabiano,

Quoted 18 lines · Fabiano Rosas

Here's the second half of adding a migration compatibility test to CI.

We've already added support for running the full set of migration tests with two QEMU binaries since commit 5050ad2a380 ("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.

changes since v3:

- Removed all the 'since' logic and started using the n-1 version of the tests.

- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.

- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.

Quoted 6 lines

v3:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]

Quoted 8 lines

I gave it a try but build-previous-qemu somehow failed..

https://gitlab.com/peterx/qemu/-/jobs/5965634871

You normally attach a "CI run" which I wanted to reference, but it's gone in this v4 unfortunately. The error:

error: pathspec 'v8.2.0' did not match any file(s) known to git

I mentioned this at the bottom of the commit message for patch 2/3. You need to push your tags. Otherwise your fork on gitlab won't have knowledge of v8.2.0.

#9 lore raw
Quoted 3 lines · Fabiano Rosas

I mentioned this at the bottom of the commit message for patch 2/3. You need to push your tags. Otherwise your fork on gitlab won't have knowledge of v8.2.0.

Oops, my fault. Yeah it works now; I queued this into staging.

Thanks,

Signature · 1 line
Peter Xu