migration & CI: Add a CI job for migration compat testing
v4 cover migration & CI: Add a CI job for migration compat testing
Here's the second half of adding a migration compatibility test to CI.
We've already added support for running the full set of migration tests with two QEMU binaries since commit 5050ad2a380 ("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.
changes since v3:
- Removed all the 'since' logic and started using the n-1 version of the tests.
- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.
- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.
v3:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]
Fabiano Rosas (3):
tests/qtest/migration: Don't use -cpu max for aarch64
ci: Add a migration compatibility test job
ci: Disable migration compatibility tests for aarch64
.gitlab-ci.d/buildtest.yml | 64 ++++++++++++++++++++++++++++++++++++ tests/qtest/migration-test.c | 2 +- 2 files changed, 65 insertions(+), 1 deletion(-)
Signature · 1 line
2.35.3
PATCH v4 1/3 tests/qtest/migration: Don't use -cpu max for aarch64
The 'max' cpu is not expected to be stable in terms of features across QEMU versions, so it should not be expected to migrate.
While the tests currently all pass with -cpu max, that is only because we're not testing across QEMU versions, which is the more common use-case for migration.
We've recently introduced compatibility tests that use two different QEMU versions and the tests are now failing for aarch64. The next patch adds those tests to CI, so we cannot use the 'max' cpu anymore. Replace it with the 'neoverse-n1', which has a fixed set of features.
--- tests/qtest/migration-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/migration-test.c +1 -1
@@ -820,7 +820,7 @@ static int test_migrate_start(QTestState **from, QTestState **to, memory_size ="150M" ; machine_alias ="virt" ; machine_opts ="gic-version=max" ; - arch_opts =g_strdup_printf ("-cpu max -kernel %s" , bootpath); + arch_opts =g_strdup_printf ("-cpu neoverse-n1 -kernel %s" , bootpath); start_address =ARM_TEST_MEM_START ; end_address =ARM_TEST_MEM_END ; }else {
Signature · 1 line
2.35.3
PATCH v4 2/3 ci: Add a migration compatibility test job
In reply to Fabiano RosasThe migration tests have support for being passed two QEMU binaries to test migration compatibility.
Add a CI job that builds the lastest release of QEMU and another job that uses that version plus an already present build of the current version and run the migration tests with the two, both as source and destination. I.e.:
old QEMU (n-1) -> current QEMU (development tree) current QEMU (development tree) -> old QEMU (n-1)
The purpose of this CI job is to ensure the code we're about to merge will not cause a migration compatibility problem when migrating the next release (which will contain that code) to/from the previous release.
The version of migration-test used will be the one matching the older QEMU. That way we can avoid special-casing new tests that wouldn't be compatible with the older QEMU.
Note: for user forks, the version tags need to be pushed to gitlab otherwise it won't be able to checkout a different version.
--- .gitlab-ci.d/buildtest.yml | 60 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
.gitlab-ci.d/buildtest.yml +60 -0
@@ -167,6 +167,66 @@ build-system-centos: x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu MAKE_CHECK_ARGS: check-build +# Previous QEMU release. Used for cross-version migration tests. +build-previous-qemu: + extends: .native_build_job_template + artifacts: + when: on_success + expire_in: 2 days + paths: + - build-previous + exclude: + - build-previous/**/*.p + - build-previous/**/*.a.p + - build-previous/**/*.fa.p + - build-previous/**/*.c.o + - build-previous/**/*.c.o.d + - build-previous/**/*.fa + needs: + job: amd64-opensuse-leap-container + variables: + IMAGE: opensuse-leap + TARGETS: x86_64-softmmu aarch64-softmmu + before_script: + - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)" + - git checkout $QEMU_PREV_VERSION + after_script: + - mv build build-previous + +.migration-compat-common: + extends: .common_test_job_template + needs: + - job: build-previous-qemu + - job: build-system-opensuse + # The old QEMU could have bugs unrelated to migration that are + # already fixed in the current development branch, so this test + # might fail. + allow_failure: true + variables: + IMAGE: opensuse-leap + MAKE_CHECK_ARGS: check-build + script: + # Use the migration-tests from the older QEMU tree. This avoids + # testing an old QEMU against new features/tests that it is not + # compatible with. + - cd build-previous + # old to new + - QTEST_QEMU_BINARY_SRC=./qemu-system-${TARGET} + QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test + # new to old + - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET} + QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test + +migration-compat-aarch64: + extends: .migration-compat-common + variables: + TARGET: aarch64 + +migration-compat-x86_64: + extends: .migration-compat-common + variables: + TARGET: x86_64 + check-system-centos: extends: .native_test_job_template needs:
Signature · 1 line
2.35.3
PATCH v4 3/3 ci: Disable migration compatibility tests for aarch64
In reply to Fabiano RosasUntil 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.
We can revert this patch when 9.0 releases.
--- .gitlab-ci.d/buildtest.yml | 4 ++++ 1 file changed, 4 insertions(+)
.gitlab-ci.d/buildtest.yml +4 -0
@@ -217,10 +217,14 @@ build-previous-qemu: - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET} QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test +# This job is disabled until we release 9.0. The existing +# migration-test in 8.2 is broken on aarch64. The fix was already +# commited, but it will only take effect once 9.0 is out. migration-compat-aarch64: extends: .migration-compat-common variables: TARGET: aarch64 + QEMU_JOB_OPTIONAL: 1 migration-compat-x86_64: extends: .migration-compat-common
Signature · 1 line
2.35.3
Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64
Quoted 18 lines · Fabiano Rosas
Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.
We can revert this patch when 9.0 releases.
Signed-off-by Fabiano Rosas <[email protected]>--- .gitlab-ci.d/buildtest.yml | 4 ++++ 1 file changed, 4 insertions(+).gitlab-ci.d/buildtest.yml +0
-0@@ -217,10 +217,14 @@ build-previous-qemu: - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET}
.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227
Quoted 10 lines
.gitlab-ci.d/buildtest.yml +4
-0QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test +# This job is disabled until we release 9.0. The existing +# migration-test in 8.2 is broken on aarch64. The fix was already +# commited, but it will only take effect once 9.0 is out. migration-compat-aarch64: extends: .migration-compat-common variables: TARGET: aarch64 + QEMU_JOB_OPTIONAL: 1
Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.
Would QEMU_JOB_SKIPPED suits more here?
https://www.qemu.org/docs/master/devel/ci.html
QEMU_JOB_OPTIONAL
The job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.
QEMU_JOB_SKIPPED
The job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.
I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.
Quoted 6 lines
.gitlab-ci.d/buildtest.yml +0
-0migration-compat-x86_64: extends: .migration-compat-commonSignature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64
Quoted 21 lines · Peter Xu
Quoted 20 lines · Fabiano Rosas
Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.
We can revert this patch when 9.0 releases.
Signed-off-by Fabiano Rosas <[email protected]>--- .gitlab-ci.d/buildtest.yml | 4 ++++ 1 file changed, 4 insertions(+).gitlab-ci.d/buildtest.yml +0
-0@@ -217,10 +217,14 @@ build-previous-qemu: - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET} QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test
Quoted 12 lines
.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227
Quoted 8 lines
.gitlab-ci.d/buildtest.yml +4
-0+# This job is disabled until we release 9.0. The existing +# migration-test in 8.2 is broken on aarch64. The fix was already +# commited, but it will only take effect once 9.0 is out. migration-compat-aarch64: extends: .migration-compat-common variables: TARGET: aarch64 + QEMU_JOB_OPTIONAL: 1Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.
I want to avoid having to think about this when 9.0 is out. This way it'll be just a revert of this patch.
Quoted 14 lines
Would
QEMU_JOB_SKIPPEDsuits more here?https://www.qemu.org/docs/master/devel/ci.html
QEMU_JOB_OPTIONALThe job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.
QEMU_JOB_SKIPPED
Quoted 8 lines
The job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.
I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.
Ok, we can use SKIPPED.
Quoted 6 lines
Quoted 6 lines
.gitlab-ci.d/buildtest.yml +0
-0migration-compat-x86_64: extends: .migration-compat-commonSignature · 1 line
2.35.3
Re: [PATCH v4 3/3] ci: Disable migration compatibility tests for aarch64
Quoted 36 lines · Fabiano Rosas
Quoted 21 lines · Peter Xu
Quoted 20 lines · Fabiano Rosas
Until 9.0 is out, we need to keep the aarch64 job disabled because the tests always use the n-1 version of migration-test. That happens to be broken for aarch64 in 8.2. Once 9.0 is out, it will become the n-1 version and it will bring the fixed tests.
We can revert this patch when 9.0 releases.
Signed-off-by Fabiano Rosas <[email protected]>--- .gitlab-ci.d/buildtest.yml | 4 ++++ 1 file changed, 4 insertions(+).gitlab-ci.d/buildtest.yml +0
-0@@ -217,10 +217,14 @@ build-previous-qemu: - QTEST_QEMU_BINARY_DST=./qemu-system-${TARGET} QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-testQuoted 12 lines
.gitlab-ci.d/buildtest.yml build-previous-qemu: · line 227
Quoted 8 lines
.gitlab-ci.d/buildtest.yml +4
-0+# This job is disabled until we release 9.0. The existing +# migration-test in 8.2 is broken on aarch64. The fix was already +# commited, but it will only take effect once 9.0 is out. migration-compat-aarch64: extends: .migration-compat-common variables: TARGET: aarch64 + QEMU_JOB_OPTIONAL: 1Optionally you can move the whole migration-compat-aarch64 from previous patch to here, then even if someone kicks off CI for previous commit it won't fail.
Quoted 2 lines
I want to avoid having to think about this when 9.0 is out. This way it'll be just a revert of this patch.
That's ok.
Quoted 20 lines
Quoted 20 lines
Would
QEMU_JOB_SKIPPEDsuits more here?https://www.qemu.org/docs/master/devel/ci.html
QEMU_JOB_OPTIONALThe job is expected to be successful in general, but is not run by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI.
QEMU_JOB_SKIPPEDThe job is not reliably successsful in general, so is not currently suitable to be run by default. Ideally this should be a temporary marker until the problems can be addressed, or the job permanently removed.
Quoted 4 lines
Quoted 3 lines
I suppose they all fall into "manual trigger" as a result, but just in case it'll behave differently in the future.
Ok, we can use SKIPPED.
No need to repost then; if this is the only thing to change I can do it when queuing. Will wait a bit more for comment on patch 2.
Signature · 1 line
Peter Xu
Fabiano,
Quoted 18 lines · Fabiano Rosas
Here's the second half of adding a migration compatibility test to CI.
We've already added support for running the full set of migration tests with two QEMU binaries since commit
5050ad2a380("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.changes since v3:
- Removed all the 'since' logic and started using the n-1 version of the tests.
- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.
- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.
Quoted 6 lines
v3:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]
I gave it a try but build-previous-qemu somehow failed..
https://gitlab.com/peterx/qemu/-/jobs/5965634871
You normally attach a "CI run" which I wanted to reference, but it's gone in this v4 unfortunately. The error:
error: pathspec 'v8.2.0' did not match any file(s) known to git
is pretty weird, and I had a feeling it was something wrong before that. Maybe it's because I tried to run this specific test multiple times (and all failed), somehow the wrong thing was cached, so the real error (of the 1st run when there is no cache) could have got lost. Not sure.
Please have a look when you have time, thanks.
Quoted 13 lines
Fabiano Rosas (3):
tests/qtest/migration: Don't use -cpu max for aarch64
ci: Add a migration compatibility test job
ci: Disable migration compatibility tests for aarch64.gitlab-ci.d/buildtest.yml | 64 ++++++++++++++++++++++++++++++++++++ tests/qtest/migration-test.c | 2 +- 2 files changed, 65 insertions(+), 1 deletion(-)Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Quoted 28 lines · Peter Xu
Fabiano,
Quoted 18 lines · Fabiano Rosas
Here's the second half of adding a migration compatibility test to CI.
We've already added support for running the full set of migration tests with two QEMU binaries since commit
5050ad2a380("tests/qtest/migration: Support more than one QEMU binary"), now what's left is adding it to the CI.changes since v3:
- Removed all the 'since' logic and started using the n-1 version of the tests.
- New patch to fix the aarch64 cpu. We shouldn't have been using 'max'.
- New patch to disable aarch64 tests while the fix^ doesn't reach a released version.
Quoted 8 lines
I gave it a try but build-previous-qemu somehow failed..
https://gitlab.com/peterx/qemu/-/jobs/5965634871
You normally attach a "CI run" which I wanted to reference, but it's gone in this v4 unfortunately. The error:
error: pathspec 'v8.2.0' did not match any file(s) known to git
I mentioned this at the bottom of the commit message for patch 2/3. You need to push your tags. Otherwise your fork on gitlab won't have knowledge of v8.2.0.
Quoted 3 lines · Fabiano Rosas
I mentioned this at the bottom of the commit message for patch 2/3. You need to push your tags. Otherwise your fork on gitlab won't have knowledge of v8.2.0.
Oops, my fault. Yeah it works now; I queued this into staging.
Thanks,
Signature · 1 line
Peter Xu