Skip to contentSkip to conversation

migration: Fix aarch64 cpregs migration

RFC PATCH In review Fabiano Rosas 4 participants 8 messages Jul 30, 2025 – Jul 31, 2025

lore
Share & tools
lore
#0 lore raw

cover migration: Fix aarch64 cpregs migration

Commit 655659a74a ("target/arm: Correct encoding of Debug Communications Channel registers") removed one register and added two more. This breaks TCG migration:

1) to 10.1 - older versions will have one unknown register

{ .name = "DBGDTRTX", .state = ARM_CP_STATE_AA32,

    .cp = 14, .crn = 0, .crm = 5, .opc1 = 3, .opc2 = 0 }

kvmid: 0x40200000200e0298

2) from 10.1 - older versions will not accept more registers than their cpreg_vmstate_array_len (292 > 291).

I'm proposing a (RFC) stop-gap fix for (1) so 10.1 can be used as a migration target until we figure out how to add some form of versioning of the cpregs_indexes list and keep track of what has been added/removed throughout the releases.

Even with infrastructure in place to do compatibility of the cpregs list, there is still the need to avoid (or justify) guest-visible changes resulting from some registers not being migrated.

Anyway, let's discuss.

PS: the extra patches are to enable cross-version testing on aarch64, which would have flagged this early. My apologies as this is entirely my fault because enabling these tests for arm has been on my list for a long time.

Fabiano Rosas (3):
target/arm: Fix migration to QEMU 10.1
tests/qtest/migration: Only test aarch64 on TCG
tests/qtest/migration: Change cpu for aarch64

 target/arm/machine.c              | 24 +++++++++++++++++++++++-
 tests/qtest/migration/framework.c | 19 ++++++++++++++-----
 2 files changed, 37 insertions(+), 6 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

RFC PATCH 1/3 target/arm: Fix migration to QEMU 10.1

The DBGDTRTX register definition was removed from cpu->cp_regs in the 10.1 cycle. This breaks migration from QEMU versions < 10.1 due to the present of an extra, unknown (to 10.1) register in the migration stream.

Change the cpu_post_load validation code to recognise that the register has been removed and ignore it when present in the stream.

Keep a compatibility list with the registers that should be ignored when sent from versions older than 10.1. The value of the cpregs hashtable key is used because it can be derived on the destination (where this patch applies) from the cpreg_vmstate_indexes array.

Note that this solution is *not* generic for other QEMU versions moving forward, this is a stop gap to avoid machines being stuck in QEMU < 10.1 without a migration path. A proper solution would include versioning of the register list and recognizing any registers removed/changed.

Fixes 655659a74a ("target/arm: Correct encoding of Debug Communications Channel registers")

---
Will an older guest using the register have issues after migration once the register gets set to its default value?

---
 target/arm/machine.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
target/arm/machine.c +23 -1
index 6986915bee..2d4df53817 100644
@@ -8,6 +8,7 @@
 #include "cpu-features.h"
 #include "migration/qemu-file-types.h"
 #include "migration/vmstate.h"
+#include "target/arm/cpregs.h"
 #include "target/arm/gtimer.h"
 
 static bool vfp_needed(void *opaque)
@@ -868,6 +869,14 @@ static const VMStateInfo vmstate_powered_off = {
     .put = put_power,
 };
 
+static uint64_t compat_cpreg_keys_virt_10_0[] = {
+    /*
+     * { .name = "DBGDTRTX", .state = ARM_CP_STATE_AA32,
+     * .cp = 14, .crn = 0, .crm = 5, .opc1 = 3, .opc2 = 0 }
+     */
+    ENCODE_CP_REG(14, 0, 1, 0, 5, 3, 0),
+};
+
 static int cpu_pre_save(void *opaque)
 {
     ARMCPU *cpu = opaque;
@@ -951,7 +960,7 @@ static int cpu_post_load(void *opaque, int version_id)
 {
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
-    int i, v;
+    int i, j, v;
 
     /*
      * Handle migration compatibility from old QEMU which didn't
@@ -987,10 +996,23 @@ static int cpu_post_load(void *opaque, int version_id)
         }
         if (cpu->cpreg_vmstate_indexes[v] < cpu->cpreg_indexes[i]) {
             /* register in their list but not ours: fail migration */
+
+            for (j = 0; j < ARRAY_SIZE(compat_cpreg_keys_virt_10_0); j++) {
+                if (cpu->cpreg_vmstate_indexes[v] ==
+                    cpreg_to_kvm_id(compat_cpreg_keys_virt_10_0[j])) {
+                    /*
+                     * ...unless the extra register is being explicitly
+                     * ignored for migration compatibility purposes.
+                     */
+                    i--;
+                    goto next;
+                }
+            }
             return -1;
         }
         /* matching register, copy the value over */
         cpu->cpreg_values[i] = cpu->cpreg_vmstate_values[v];
+    next:
         v++;
     }
 
Signature · 1 line
2.35.3
#2 lore raw

RFC PATCH 2/3 tests/qtest/migration: Only test aarch64 on TCG

In reply to Fabiano Rosas

Currently our aarch64 tests are only being run using identical QEMU versions. When running the tests with different QEMU versions, which is a common use-case for migration, the tests are broken due to the current choice of the 'max' cpu, which is not stable and is prone to breaking migration.

This means aarch64 tests are currently only testing about the same situations as any other arch, i.e. no arm-specific testing is being done.

To make the aarch64 tests more useful, -cpu max will be changed to -cpu neoverse-n1 in the next patch. Before doing that, make sure aarch64 tests only run with TCG, since KVM testing depends on usage of the -cpu host and we currently don't have code to switch between cpus according to test runtime environment.

Also, TCG alone should allow us to catch most issues with migration, since there is no guarantee of a uniform environment as there is with KVM.

---
 tests/qtest/migration/framework.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
tests/qtest/migration/framework.c +13 -4
index 407c9023c0..f09365d122 100644
@@ -353,8 +353,17 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
         memory_backend = g_strdup_printf("-m %s ", memory_size);
     }
 
-    if (args->use_dirty_ring) {
-        kvm_opts = ",dirty-ring-size=4096";
+
+    if (g_str_equal(arch, "aarch64")) {
+        /*
+         * aarch64 is only tested with TCG because there is no single
+         * cpu that can be used for both KVM and TCG.
+         */
+        kvm_opts = NULL;
+    } else if (args->use_dirty_ring) {
+        kvm_opts = "-accel kvm,dirty-ring-size=4096";
+    } else {
+        kvm_opts = "-accel kvm";
     }
 
     if (!qtest_has_machine(machine_alias)) {
@@ -368,7 +377,7 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
 
     g_test_message("Using machine type: %s", machine);
 
-    cmd_source = g_strdup_printf("-accel kvm%s -accel tcg "
+    cmd_source = g_strdup_printf("%s -accel tcg "
                                  "-machine %s,%s "
                                  "-name source,debug-threads=on "
                                  "%s "
@@ -395,7 +404,7 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
      */
     events = args->defer_target_connect ? "-global migration.x-events=on" : "";
 
-    cmd_target = g_strdup_printf("-accel kvm%s -accel tcg "
+    cmd_target = g_strdup_printf("%s -accel tcg "
                                  "-machine %s,%s "
                                  "-name target,debug-threads=on "
                                  "%s "
Signature · 1 line
2.35.3
#3 lore raw

Re: [RFC PATCH 2/3] tests/qtest/migration: Only test aarch64 on TCG

Quoted 17 lines · Fabiano Rosas

Currently our aarch64 tests are only being run using identical QEMU versions. When running the tests with different QEMU versions, which is a common use-case for migration, the tests are broken due to the current choice of the 'max' cpu, which is not stable and is prone to breaking migration.

This means aarch64 tests are currently only testing about the same situations as any other arch, i.e. no arm-specific testing is being done.

To make the aarch64 tests more useful, -cpu max will be changed to -cpu neoverse-n1 in the next patch. Before doing that, make sure aarch64 tests only run with TCG, since KVM testing depends on usage of the -cpu host and we currently don't have code to switch between cpus according to test runtime environment.

Quoted 3 lines

Also, TCG alone should allow us to catch most issues with migration, since there is no guarantee of a uniform environment as there is with KVM.

The difficulty with only testing TCG migration is that now we're testing the setup that most cross-versions migration users don't care about. At least my assumption is that it's KVM cross-version migration that is the real use case here.

For instance, this migration bug with the DBGDTR register isn't a problem for KVM, because with KVM we use the kernel to tell us what system registers are present, and whether a register is defined with a cpreg in QEMU or not doesn't affect what we put on the wire for migration. Conversely there might be migration compat issues that show up only with KVM and not TCG (though the most obvious source of those would be host kernel changes, which is kind of out of scope for us).

Though of course with our CI jobs we're probably not doing AArch64 KVM cross-version testing anyway...

thanks
-- PMM

#4 lore raw

Re: [RFC PATCH 2/3] tests/qtest/migration: Only test aarch64 on TCG

Quoted 41 lines · Peter Maydell
Quoted 17 lines · Fabiano Rosas

Currently our aarch64 tests are only being run using identical QEMU versions. When running the tests with different QEMU versions, which is a common use-case for migration, the tests are broken due to the current choice of the 'max' cpu, which is not stable and is prone to breaking migration.

This means aarch64 tests are currently only testing about the same situations as any other arch, i.e. no arm-specific testing is being done.

To make the aarch64 tests more useful, -cpu max will be changed to -cpu neoverse-n1 in the next patch. Before doing that, make sure aarch64 tests only run with TCG, since KVM testing depends on usage of the -cpu host and we currently don't have code to switch between cpus according to test runtime environment.

Quoted 3 lines

Also, TCG alone should allow us to catch most issues with migration, since there is no guarantee of a uniform environment as there is with KVM.

The difficulty with only testing TCG migration is that now we're testing the setup that most cross-versions migration users don't care about. At least my assumption is that it's KVM cross-version migration that is the real use case here.

For instance, this migration bug with the DBGDTR register isn't a problem for KVM, because with KVM we use the kernel to tell us what system registers are present, and whether a register is defined with a cpreg in QEMU or not doesn't affect what we put on the wire for migration. Conversely there might be migration compat issues that show up only with KVM and not TCG (though the most obvious source of those would be host kernel changes, which is kind of out of scope for us).

Quoted 3 lines

Though of course with our CI jobs we're probably not doing AArch64 KVM cross-version testing anyway...

On the cloud provider side*, we do rely on having rollbacks work.

We rely on staged deployments with rolling back if things go wrong as we observe progress.

Note that the set of MSRs KVM gives (at least on AArch64) does sometimes vary between releases so for rolling back you’ll need to ignore some (new) sysregs in the vmm. With careful planning so that you deploy a VMM release with a point-fix to ignore the new registers and then the kernel update.

So not dealing with that would make the cloud use case not usable without downstream patches.

*although we don’t rely on Qemu for Nitro System VMs

Thank you,

Quoted 3 lines

thanks
-- PMM

#5 lore raw

RFC PATCH 3/3 tests/qtest/migration: Change cpu for aarch64

In reply to Fabiano Rosas

Don't use the 'max' cpu for migration testing of aarch64. That cpu does not provide a stable set of features and is expected to break migration from time to time.

---
 tests/qtest/migration/framework.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/migration/framework.c +1 -1
index f09365d122..6d980b6b51 100644
@@ -317,7 +317,7 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
         memory_size = "150M";
         machine_alias = "virt";
         machine_opts = "gic-version=3";
-        arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath);
+        arch_opts = g_strdup_printf("-cpu neoverse-n1 -kernel %s", bootpath);
         start_address = ARM_TEST_MEM_START;
         end_address = ARM_TEST_MEM_END;
     } else {
Signature · 1 line
2.35.3
#6 lore raw

Re: [RFC PATCH 3/3] tests/qtest/migration: Change cpu for aarch64

Quoted 3 lines · On Wed, 30 Jul 2025 17:52:45 -0300 Fabiano Rosas <[email protected]> wrote:

Don't use the 'max' cpu for migration testing of aarch64. That cpu does not provide a stable set of features and is expected to break migration from time to time.

Whilst I can see the motivation, doesn't this leave us with a lack of converage for new CPU features that are currently only in max?

Quoted 19 lines
---
 tests/qtest/migration/framework.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/migration/framework.c +1 -1
index f09365d122..6d980b6b51 100644
@@ -317,7 +317,7 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
         memory_size = "150M";
         machine_alias = "virt";
         machine_opts = "gic-version=3";
-        arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath);
+        arch_opts = g_strdup_printf("-cpu neoverse-n1 -kernel %s", bootpath);
         start_address = ARM_TEST_MEM_START;
         end_address = ARM_TEST_MEM_END;
     } else {
#7 lore raw

Re: [RFC PATCH 3/3] tests/qtest/migration: Change cpu for aarch64

Quoted 9 lines · Jonathan Cameron via
Quoted 3 lines · On Wed, 30 Jul 2025 17:52:45 -0300 Fabiano Rosas <[email protected]> wrote:

Don't use the 'max' cpu for migration testing of aarch64. That cpu does not provide a stable set of features and is expected to break migration from time to time.

Whilst I can see the motivation, doesn't this leave us with a lack of converage for new CPU features that are currently only in max?

It does. That's an interesting aspect. It's better to make sure the new features come with at least some basic migration support off the gate.

I'll add a patch to this series leaving one of the tests with -cpu max. That should be enough to cover this scenario.

Thanks

Quoted 19 lines
Quoted 19 lines
---
 tests/qtest/migration/framework.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/migration/framework.c +1 -1
index f09365d122..6d980b6b51 100644
@@ -317,7 +317,7 @@ int migrate_start(QTestState **from, QTestState **to, const char *uri,
         memory_size = "150M";
         machine_alias = "virt";
         machine_opts = "gic-version=3";
-        arch_opts = g_strdup_printf("-cpu max -kernel %s", bootpath);
+        arch_opts = g_strdup_printf("-cpu neoverse-n1 -kernel %s", bootpath);
         start_address = ARM_TEST_MEM_START;
         end_address = ARM_TEST_MEM_END;
     } else {