Skip to contentSkip to conversation

Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures

PATCH Applied Eric Auger 2 participants 14 messages Apr 20 – Apr 27

lore
Share & tools
lore
author
#3 lore raw

PATCH v10 2/7 target/arm/machine: Handle ToleranceNotOnBothEnds migration tolerances

In reply to Eric Auger

If there is a mismatch between the cpreg indexes found on both ends, check whether a tolerance was registered for the given kvmidx. If any, silence warning/errors.

---

v9 -> v10
- s/shall/should
- remove obsolete desc about helper in commit description
- Added Peter's R-b
- Reworded the commit message

---
 target/arm/machine.c    | 21 +++++++++++++++------
 target/arm/trace-events |  2 ++
 2 files changed, 17 insertions(+), 6 deletions(-)
target/arm/machine.c +15 -6
index b0e499515cf..8a698f601c6 100644
@@ -1065,25 +1065,34 @@ static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx
 {
     g_autofree gchar *name = print_register_name(kvmidx);
 
+    if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          0, 0, ToleranceNotOnBothEnds)) {
+        trace_tolerate_cpreg_missing_in_incoming_stream(name);
+        return;
+    }
     warn_report("%s: %s "
                 "expected by the destination but not in the incoming stream: "
                  "skip it", __func__, name);
 }
 
 /*
- * Handle the situation where @kvmidx is in the incoming stream
- * but not on destination. This currently fails the migration but
- * we plan to accomodate some exceptions, hence the boolean returned value.
+ * Handle the situation where @kvmidx is in the incoming
+ * stream but not on destination. This fails the migration if
+ * no cpreg mig tolerance is matched for this @kvmidx
+ * Return true if the migration should eventually fail
  */
 static bool handle_cpreg_only_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx)
 {
     g_autofree gchar *name = print_register_name(kvmidx);
-    bool fail = true;
 
+    if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          0, 0, ToleranceNotOnBothEnds)) {
+        trace_tolerate_cpreg_only_in_incoming_stream(name);
+        return false;
+    }
     error_report("%s: %s in the incoming stream but unknown on the "
                  "destination: fail migration", __func__, name);
-
-    return fail;
+    return true;
 }
 
 static int cpu_post_load(void *opaque, int version_id)
target/arm/trace-events +2 -0
index 2de0406f784..8502fb3265c 100644
@@ -29,3 +29,5 @@ arm_psci_call(uint64_t x0, uint64_t x1, uint64_t x2, uint64_t x3, uint32_t cpuid
 
 # machine.c
 cpu_post_load(uint32_t cpreg_vmstate_array_len, uint32_t cpreg_array_len) "cpreg_vmstate_array_len=%d cpreg_array_len=%d"
+tolerate_cpreg_missing_in_incoming_stream(char *name) "%s is missing in incoming stream but this is explicitly tolerated"
+tolerate_cpreg_only_in_incoming_stream(char *name) "%s is in incoming stream but not on destination but this is explicitly tolerated"
Signature · 1 line
2.53.0

Eric Auger

author
#0 lore raw

v10 cover Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures

This series introduces an infrastructure to explicitly tolerate some mismatches between cpu cpreg indexes/values on migration.

We especially handle failures due to an attempt to migrate more registers than exposed on the destination.

Once the infrastructure is in place we can handle the following known issues:
KVM:
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
TCG:
- We can eventually remove AArch32 DBGDTRTX register which was erroneously defined.

The series can be found at:
https://github.com/eauger/qemu/tree/x-kernel-mig-v10

History:
v9 -> v10:
- main functional change consists in the introduction of a new ToleranceOnlySrcTestValue migration tolerance type to handle KVM_REG_ARM_VENDOR_HYP_BMAP_2
- See individual patch diff logs for more details

v8 -> v9:
- took into account Seb's comments (fixed ->type check, rename one function)
- collected R-bs

v7 -> v8:
- rebase on top of v3 dep
- removal of target/arm/machine: Fix detection of unknown incoming cpregs which is integrated in prerequisite series

v6 -> v7
- complete rework after Peter's comments in
(https://lore.kernel.org/all/[email protected]/)
Especially it does not use properties at all. Do not feature hidden regs Also the initial series was split into 2.

Eric Auger (7):
target/arm/cpu: Introduce the infrastructure for cpreg migration
tolerances
target/arm/machine: Handle ToleranceNotOnBothEnds migration tolerances target/arm/machine: Handle ToleranceOnlySrcTestValue migration
tolerance
target/arm/cpu64: Mitigate migration failures due to spurious TCR_EL1,
PIRE0_EL1 and PIR_EL1
target/arm/cpu64: Define cpreg migration tolerance for
KVM_REG_ARM_VENDOR_HYP_BMAP_2
target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0 Revert "target/arm: Reinstate bogus AArch32 DBGDTRTX register for
migration compat"

 target/arm/cpu.h          |  1 +
 target/arm/internals.h    | 54 ++++++++++++++++++++++++++
 target/arm/cpu.c          | 82 +++++++++++++++++++++++++++++++++++++++
 target/arm/cpu64.c        | 39 +++++++++++++++++++
 target/arm/debug_helper.c | 29 --------------
 target/arm/helper.c       | 15 +++++++
 target/arm/machine.c      | 33 +++++++++++-----
 target/arm/trace-events   |  2 +
 8 files changed, 217 insertions(+), 38 deletions(-)
Signature · 1 line
2.53.0
author
#6 lore raw

PATCH v10 4/7 target/arm/cpu64: Mitigate migration failures due to spurious TCR_EL1, PIRE0_EL1 and PIR_EL1

Before linux v6.13 those registers were erroneously unconditionally exposed and this was fixed by commits:
- 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace when disabled for guests")
- a68cddbe47ef ("KVM: arm64: Hide S1PIE registers from userspace when disabled for guests")
in v6.13.

This means if we migrate from an old kernel host to a >= 6.13 kernel host, migration currently fails.

Declare cpreg migration tolerance for those registers.

---

v9 -> v10
- Put kvm_arm_set_cpreg_mig_tolerances definition in #if defined(CONFIG_KVM).
- Added Peter's R-b

---
 target/arm/cpu64.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
target/arm/cpu64.c +28 -0
index d6feba220e8..e7014022df4 100644
@@ -810,6 +810,33 @@ static void aarch64_a53_initfn(Object *obj)
     define_cortex_a72_a57_a53_cp_reginfo(cpu);
 }
 
+#if defined(CONFIG_KVM)
+static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
+{
+    /*
+     * Registers that may be in the incoming stream and not exposed
+     * on the destination
+     */
+
+    /*
+     * TCR_EL1 was erroneously unconditionnally exposed before linux v6.13.
+     * See commit 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace
+     * when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 2, 0, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+    /*
+     * PIRE0_EL1 and PIR_EL1 were erroneously unconditionnally exposed
+     * before linux v6.13. See commit a68cddbe47ef ("KVM: arm64: Hide
+     * S1PIE registers from userspace when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 2),
+                                     0, 0, ToleranceNotOnBothEnds);
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+}
+#endif
+
 static void aarch64_host_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
@@ -822,6 +849,7 @@ static void aarch64_host_initfn(Object *obj)
 #endif
 
 #if defined(CONFIG_KVM)
+    kvm_arm_set_cpreg_mig_tolerances(cpu);
     kvm_arm_set_cpu_features_from_host(cpu);
     aarch64_add_sve_properties(obj);
 #elif defined(CONFIG_HVF)
Signature · 1 line
2.53.0
author
#4 lore raw

PATCH v10 3/7 target/arm/machine: Handle ToleranceOnlySrcTestValue migration tolerance

In reply to Eric Auger

Pass the value of the incoming register to handle_cpreg_only_in_incoming_stream and check whether there is a matching ToleranceOnlySrcTestValue tolerance.

---
 target/arm/machine.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
target/arm/machine.c +10 -4
index 8a698f601c6..35f1e4327f8 100644
@@ -1081,12 +1081,15 @@ static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx
  * no cpreg mig tolerance is matched for this @kvmidx
  * Return true if the migration should eventually fail
  */
-static bool handle_cpreg_only_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx)
+static bool
+handle_cpreg_only_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx, uint64_t value)
 {
     g_autofree gchar *name = print_register_name(kvmidx);
 
     if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
-                                          0, 0, ToleranceNotOnBothEnds)) {
+                                          0, 0, ToleranceNotOnBothEnds) ||
+        arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          value, 0, ToleranceOnlySrcTestValue)) {
         trace_tolerate_cpreg_only_in_incoming_stream(name);
         return false;
     }
@@ -1139,7 +1142,9 @@ static int cpu_post_load(void *opaque, int version_id)
         }
         if (cpu->cpreg_vmstate_indexes[v] < cpu->cpreg_indexes[i]) {
             fail = handle_cpreg_only_in_incoming_stream(cpu,
-                                                        cpu->cpreg_vmstate_indexes[v++]);
+                                                        cpu->cpreg_vmstate_indexes[v],
+                                                        cpu->cpreg_vmstate_values[v]);
+            v++;
             continue;
         }
         /* matching register, copy the value over */
@@ -1162,7 +1167,8 @@ static int cpu_post_load(void *opaque, int version_id)
      */
     for ( ; v < cpu->cpreg_vmstate_array_len; v++) {
         fail = handle_cpreg_only_in_incoming_stream(cpu,
-                                                    cpu->cpreg_vmstate_indexes[v]);
+                                                    cpu->cpreg_vmstate_indexes[v],
+                                                    cpu->cpreg_vmstate_values[v]);
     }
     if (fail) {
         return -1;
Signature · 1 line
2.53.0
author
#7 lore raw

PATCH v10 5/7 target/arm/cpu64: Define cpreg migration tolerance for KVM_REG_ARM_VENDOR_HYP_BMAP_2

In reply to Eric Auger

KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed from v6.15 onwards. Backward migration from a >= v6.15 to an older kernel would fail without cpreg migration tolerance definition for this register. If the register is present on source but not on destination, its value must be checked to make sure it matches the reset value, ie. 0, meaning no service is exposed to the guest, hence the choice of a ToleranceOnlySrcTestValue migration tolerance.

---

v9 -> v10
- Use ToleranceOnlySrcTestValue

---
 target/arm/cpu64.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
target/arm/cpu64.c +11 -0
index e7014022df4..a93ad2da5ad 100644
@@ -834,6 +834,17 @@ static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
                                      0, 0, ToleranceNotOnBothEnds);
     arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
                                      0, 0, ToleranceNotOnBothEnds);
+
+    /*
+     * KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed
+     * from v6.15 onwards. Backward migration from a >= v6.15 to an older
+     * kernel would fail without cpreg migration tolerance definition.
+     * If the register is present on source but not on destination, make
+     * sure it has its reset value, ie. 0, meaning no service is exposed
+     * to the guest.
+     */
+    arm_register_cpreg_mig_tolerance(cpu, KVM_REG_ARM_FW_FEAT_BMAP_REG(3),
+                                     UINT64_MAX, 0, ToleranceOnlySrcTestValue);
 }
 #endif
 
Signature · 1 line
2.53.0
author
#1 lore raw

PATCH v10 1/7 target/arm/cpu: Introduce the infrastructure for cpreg migration tolerances

In reply to Eric Auger

We introduce a datatype for a tolerance with respect to a given cpreg migration issue. The tolerance applies to a given cpreg kvm index, and can be of different types:
a) mismatch in cpreg indexes
- ToleranceNotOnBothEnds (cpreg index is allowed to be only present on one end)
- ToleranceOnlySrcTestValue (cpreg index is allowed to be only present in source if its value @mask field matches @value)
b) mismatch in cpreg values
- ToleranceDiffInMask (value differences are allowed only within a mask)
- ToleranceFieldLT (incoming field value must be less than a given value)
- ToleranceFieldGT (incoming field value must be greater than a given value)

A QLIST of such tolerances can be populated using a new helper:
arm_register_cpreg_mig_tolerance() and arm_cpu_match_cpreg_mig_tolerance() allows to check whether a tolerance exists for a given kvm index and its criterion is matched.

callers for those helpers will be introduced in subsequent patches.

Only registration of migration tolerances related to cpreg index mismatch is currently allowed.

---
v9 -> v10
- s/CPUCPREG/CPReg
- move declarations and types to internals.h
- rework doc comments and detail tolerance types
- introduce find_mig_tolerance() according to Peter's suggestion
- added ToleranceOnlySrcTestValue

v8 -> v9
- fix the tolerance type checking
- rename arm_cpu_cpreg_has_mig_tolerance into arm_cpu_match_cpreg_mig_tolerance

---
 target/arm/cpu.h       |  1 +
 target/arm/internals.h | 54 ++++++++++++++++++++++++++++
 target/arm/cpu.c       | 82 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 137 insertions(+)
target/arm/cpu.h +1 -0
index 657ff4ab20b..512d18652aa 100644
@@ -1139,6 +1139,7 @@ struct ArchCPU {
 
     QLIST_HEAD(, ARMELChangeHook) pre_el_change_hooks;
     QLIST_HEAD(, ARMELChangeHook) el_change_hooks;
+    QLIST_HEAD(, ARMCPRegMigTolerance) cpreg_mig_tolerances;
 
     int32_t node_id; /* NUMA node this CPU belongs to */
 
target/arm/internals.h +54 -0

58 lines not on this page · raw message

target/arm/cpu.c +82 -0

106 lines not on this page · raw message

Signature · 1 line
2.53.0
author
#11 lore raw

PATCH v10 7/7 Revert "target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat"

In reply to Eric Auger

This reverts commit 4f2b82f60431 ("target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat). We don't need that commit anymore as the AArch32 DBGDTRTX register is declared to
be safe to ignore in the incoming migration stream.

---
 target/arm/debug_helper.c | 29 -----------------------------
 1 file changed, 29 deletions(-)
target/arm/debug_helper.c +0 -29
index 352c8e5c8e7..8477ca5def6 100644
@@ -171,13 +171,6 @@ static void dbgclaimclr_write(CPUARMState *env, const ARMCPRegInfo *ri,
     env->cp15.dbgclaim &= ~(value & 0xFF);
 }
 
-static CPAccessResult access_bogus(CPUARMState *env, const ARMCPRegInfo *ri,
-                                   bool isread)
-{
-    /* Always UNDEF, as if this cpreg didn't exist */
-    return CP_ACCESS_UNDEFINED;
-}
-
 static const ARMCPRegInfo debug_cp_reginfo[] = {
     /*
      * DBGDRAR, DBGDSAR: always RAZ since we don't implement memory mapped
@@ -240,28 +233,6 @@ static const ARMCPRegInfo debug_cp_reginfo[] = {
       .opc0 = 2, .opc1 = 3, .crn = 0, .crm = 4, .opc2 = 0,
       .access = PL0_RW, .accessfn = access_tdcc,
       .type = ARM_CP_CONST, .resetvalue = 0 },
-    /*
-     * This is not a real AArch32 register. We used to incorrectly expose
-     * this due to a QEMU bug; to avoid breaking migration compatibility we
-     * need to continue to provide it so that we don't fail the inbound
-     * migration when it tells us about a sysreg that we don't have.
-     * We set an always-fails .accessfn, which means that the guest doesn't
-     * actually see this register (it will always UNDEF, identically to if
-     * there were no cpreg definition for it other than that we won't print
-     * a LOG_UNIMP message about it), and we set the ARM_CP_NO_GDB flag so the
-     * gdbstub won't see it either.
-     * (We can't just set .access = 0, because add_cpreg_to_hashtable()
-     * helpfully ignores cpregs which aren't accessible to the highest
-     * implemented EL.)
-     *
-     * TODO: implement a system for being able to describe "this register
-     * can be ignored if it appears in the inbound stream"; then we can
-     * remove this temporary hack.
-     */
-    { .name = "BOGUS_DBGDTR_EL0", .state = ARM_CP_STATE_AA32,
-      .cp = 14, .opc1 = 3, .crn = 0, .crm = 5, .opc2 = 0,
-      .access = PL0_RW, .accessfn = access_bogus,
-      .type = ARM_CP_CONST | ARM_CP_NO_GDB, .resetvalue = 0 },
     /*
      * OSECCR_EL1 provides a mechanism for an operating system
      * to access the contents of EDECCR. EDECCR is not implemented though,
Signature · 1 line
2.53.0
author
#9 lore raw

PATCH v10 6/7 target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0

In reply to Eric Auger

We want to remove AArch32 DBGDTRTX which was erroneously exposed. This was attempted by 655659a74a36b ("target/arm: Correct encoding of Debug Communications Channel registers") but it was discovered that the removal of this debug register broke forward migration on TCG. Now we have the cpreg migration tolerance infrastructure, we can declare one for the DBGDTRTX. This allow to revert the reinstate patch.

---

v9 -> v10
- Adding a comment explaining why we want a migratoin tolerance for that cpreg

---
 target/arm/helper.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
target/arm/helper.c +15 -0
index 7389f2988c4..d80899cd096 100644
@@ -6426,9 +6426,24 @@ void register_cp_regs_for_features(ARMCPU *cpu)
             .fgt = FGT_CLIDR_EL1,
             .resetvalue = GET_IDREG(isar, CLIDR)
         };
+        uint64_t dbgtr_el0_kvmidx =
+            cpreg_to_kvm_id(ENCODE_CP_REG(14, 0, 1, 0, 5, 3, 0));
+
         define_one_arm_cp_reg(cpu, &clidr);
         define_arm_cp_regs(cpu, v7_cp_reginfo);
         define_debug_regs(cpu);
+        /**
+         * AArch32 DBGDTRTX has been introduced with a wrong encoding.
+         * This was fixed by commit 655659a74a36 ("target/arm: Correct
+         * encoding of Debug Communications Channel registers") by the
+         * introduction of correct separate cpreg definitions
+         * for AA64 and AA32 versions. However the old cpreg definition
+         * couldn't be removed without breaking the migration. From qemu 11.1
+         * onwards we can use the migration tolerance infrastructure to
+         * remove it.
+         */
+        arm_register_cpreg_mig_tolerance(cpu, dbgtr_el0_kvmidx,
+                                         0, 0, ToleranceNotOnBothEnds);
     } else {
         define_arm_cp_regs(cpu, not_v7_cp_reginfo);
     }
Signature · 1 line
2.53.0
Reviewed-by on v10 6/7
#10 lore raw

Re: [PATCH v10 6/7] target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0

Quoted 27 lines · Eric Auger

We want to remove AArch32 DBGDTRTX which was erroneously exposed. This was attempted by 655659a74a36b ("target/arm: Correct encoding of Debug Communications Channel registers") but it was discovered that the removal of this debug register broke forward migration on TCG. Now we have the cpreg migration tolerance infrastructure, we can declare one for the DBGDTRTX. This allow to revert the reinstate patch.

---

v9 -> v10
- Adding a comment explaining why we want a migratoin tolerance for that cpreg

---
 target/arm/helper.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
target/arm/helper.c +0 -0
index 7389f2988c4..d80899cd096 100644
@@ -6426,9 +6426,24 @@ void register_cp_regs_for_features(ARMCPU *cpu)
             .fgt = FGT_CLIDR_EL1,

target/arm/helper.c void register_cp_regs_for_features(ARMCPU *cpu) · line 6436

Quoted 10 lines
target/arm/helper.c +5 -0
             .resetvalue = GET_IDREG(isar, CLIDR)
         };
+        uint64_t dbgtr_el0_kvmidx =
+            cpreg_to_kvm_id(ENCODE_CP_REG(14, 0, 1, 0, 5, 3, 0));
+
         define_one_arm_cp_reg(cpu, &clidr);
         define_arm_cp_regs(cpu, v7_cp_reginfo);
         define_debug_regs(cpu);
+        /**
+         * AArch32 DBGDTRTX has been introduced with a wrong encoding.

I prefer the phrasing in the old comment here:

"We used to incorrectly expose a non-existent AArch32 "DBGDTRTX" register with this encoding. This was fixed ..."

target/arm/helper.c void register_cp_regs_for_features(ARMCPU *cpu) · line 6449

Quoted 13 lines
target/arm/helper.c +10 -0
+         * This was fixed by commit 655659a74a36 ("target/arm: Correct
+         * encoding of Debug Communications Channel registers") by the
+         * introduction of correct separate cpreg definitions
+         * for AA64 and AA32 versions. However the old cpreg definition
+         * couldn't be removed without breaking the migration. From qemu 11.1
+         * onwards we can use the migration tolerance infrastructure to
+         * remove it.
+         */
+        arm_register_cpreg_mig_tolerance(cpu, dbgtr_el0_kvmidx,
+                                         0, 0, ToleranceNotOnBothEnds);
     } else {
         define_arm_cp_regs(cpu, not_v7_cp_reginfo);
     }

Otherwise

thanks
-- PMM

Reviewed-by on v10 5/7
#8 lore raw

Re: [PATCH v10 5/7] target/arm/cpu64: Define cpreg migration tolerance for KVM_REG_ARM_VENDOR_HYP_BMAP_2

In reply to Eric Auger
Quoted 31 lines · Eric Auger

KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed from v6.15 onwards. Backward migration from a >= v6.15 to an older kernel would fail without cpreg migration tolerance definition for this register. If the register is present on source but not on destination, its value must be checked to make sure it matches the reset value, ie. 0, meaning no service is exposed to the guest, hence the choice of a ToleranceOnlySrcTestValue migration tolerance.

---

v9 -> v10
- Use ToleranceOnlySrcTestValue

---
 target/arm/cpu64.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
target/arm/cpu64.c +3 -0
index e7014022df4..a93ad2da5ad 100644
@@ -834,6 +834,17 @@ static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
                                      0, 0, ToleranceNotOnBothEnds);
     arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
                                      0, 0, ToleranceNotOnBothEnds);
+
+    /*
+     * KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed

target/arm/cpu64.c static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu) · line 849

Quoted 10 lines
target/arm/cpu64.c +8 -0
+     * from v6.15 onwards. Backward migration from a >= v6.15 to an older
+     * kernel would fail without cpreg migration tolerance definition.
+     * If the register is present on source but not on destination, make
+     * sure it has its reset value, ie. 0, meaning no service is exposed
+     * to the guest.
+     */
+    arm_register_cpreg_mig_tolerance(cpu, KVM_REG_ARM_FW_FEAT_BMAP_REG(3),
+                                     UINT64_MAX, 0, ToleranceOnlySrcTestValue);
 }
 #endif

thanks
-- PMM

Reviewed-by on v10 1/7
#2 lore raw

Re: [PATCH v10 1/7] target/arm/cpu: Introduce the infrastructure for cpreg migration tolerances

In reply to Eric Auger
Quoted 19 lines · Eric Auger

We introduce a datatype for a tolerance with respect to a given cpreg migration issue. The tolerance applies to a given cpreg kvm index, and can be of different types:
a) mismatch in cpreg indexes
- ToleranceNotOnBothEnds (cpreg index is allowed to be only present on one end)
- ToleranceOnlySrcTestValue (cpreg index is allowed to be only present in source if its value @mask field matches @value)
b) mismatch in cpreg values
- ToleranceDiffInMask (value differences are allowed only within a mask)
- ToleranceFieldLT (incoming field value must be less than a given value)
- ToleranceFieldGT (incoming field value must be greater than a given value)

A QLIST of such tolerances can be populated using a new helper:
arm_register_cpreg_mig_tolerance() and arm_cpu_match_cpreg_mig_tolerance() allows to check whether a tolerance exists for a given kvm index and its criterion is matched.

Quoted 6 lines

callers for those helpers will be introduced in subsequent patches.

Only registration of migration tolerances related to cpreg index mismatch is currently allowed.

The way that ToleranceOnlySrcTestValue allows the bits outside the field to be anything whereas the LT and GT types insist on the bits outside the field being an exact match feels a bit non-orthogonal, but I see why we do it (otherwise the EQ check would effectively be a "check all bits match" check regardless of the field mask; as it happens the only thing we use it for passes an all-bits-set mask anyway).

I guess we'll see how many of these we accumulate and whether we want to adjust the kinds of check we do in future.

-- PMM

Reviewed-by on v10 3/7
#5 lore raw

Re: [PATCH v10 3/7] target/arm/machine: Handle ToleranceOnlySrcTestValue migration tolerance

In reply to Eric Auger
Quoted 9 lines · Eric Auger

Pass the value of the incoming register to handle_cpreg_only_in_incoming_stream and check whether there is a matching ToleranceOnlySrcTestValue tolerance.

---
 target/arm/machine.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

thanks
-- PMM

#12 lore raw
In reply to Eric Auger
Quoted 7 lines · Eric Auger

This series introduces an infrastructure to explicitly tolerate some mismatches between cpu cpreg indexes/values on migration.

We especially handle failures due to an attempt to migrate more registers than exposed on the destination.

Quoted 11 lines

Once the infrastructure is in place we can handle the following known issues:
KVM:
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
TCG:
- We can eventually remove AArch32 DBGDTRTX register which was erroneously defined.

Applied to target-arm.next, thanks (with an edit to the comment in patch 6).

-- PMM

author
#13 lore raw

Hi Peter,

Quoted 18 lines · Peter Maydell
Quoted 6 lines · Eric Auger

This series introduces an infrastructure to explicitly tolerate some mismatches between cpu cpreg indexes/values on migration.

We especially handle failures due to an attempt to migrate more registers than exposed on the destination.

Quoted 11 lines

Once the infrastructure is in place we can handle the following known issues:
KVM:
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
TCG:
- We can eventually remove AArch32 DBGDTRTX register which was erroneously defined.

Quoted 2 lines

Applied to target-arm.next, thanks (with an edit to the comment in patch 6).

Ack. Many thanks!

Eric

Quoted 3 lines

-- PMM