Skip to contentSkip to conversation

Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures

PATCH Superseded Eric Auger 2 participants 16 messages Mar 6 – Apr 17

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v9 cover Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures

This series applies on top of
[PATCH v3 0/7] Improve traces on migration failure due to cpreg number mismatch

It introduces an infrastructure to explicitly tolerate some mismatches between cpu cpreg indexes/values on migration.

We especially handle failures due to an attempt to migrate more registers than exposed on the destination.

Once the infrastructure is in place we can handle the following known issues:
KVM:
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
TCG:
- We can eventually remove AArch32 DBGDTRTX register which was erroneously defined.

Both series are available at:
https://github.com/eauger/qemu/tree/cpreg_vmstate_array_len_traces_v3_mitig_v3

History:
v8 -> v9:
- took into account Seb's comments (fixed ->type check, rename one function)
- collected R-bs

v7 -> v8:
- rebase on top of v3 dep
- removal of target/arm/machine: Fix detection of unknown incoming cpregs which is integrated in prerequisite series

v6 -> v7
- complete rework after Peter's comments in
(https://lore.kernel.org/all/[email protected]/)
Especially it does not use properties at all. Do not feature hidden regs Also the initial series was split into 2.

Eric Auger (6):
target/arm/cpu: Introduce the infrastructure for cpreg migration
tolerances
target/arm/machine: Take account cpreg mig tolerances in case of
mismatch
target/arm/cpu64: Mitigate migration failures due to spurious TCR_EL1,
PIRE0_EL1 and PIR_EL1
target/arm/cpu64: Define cpreg migration tolerance for
KVM_REG_ARM_VENDOR_HYP_BMAP_2
target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0 Revert "target/arm: Reinstate bogus AArch32 DBGDTRTX register for
migration compat"

 target/arm/cpu.h          | 33 +++++++++++++++++
 target/arm/cpu.c          | 75 +++++++++++++++++++++++++++++++++++++++
 target/arm/cpu64.c        | 34 ++++++++++++++++++
 target/arm/debug_helper.c | 29 ---------------
 target/arm/helper.c       |  5 +++
 target/arm/machine.c      | 21 +++++++----
 target/arm/trace-events   |  2 ++
 7 files changed, 164 insertions(+), 35 deletions(-)
Signature · 1 line
2.53.0
author
#1 lore raw

PATCH v9 1/6 target/arm/cpu: Introduce the infrastructure for cpreg migration tolerances

We introduce a datatype for a tolerance with respect to a given cpreg migration issue. The tolerance applies to a given cpreg kvm index, and can be of different types:
- ToleranceNotOnBothEnds (cpreg index is allowed to be only present on one end)
- ToleranceDiffInMask (value differences are allowed only within a mask)
- ToleranceFieldLT (incoming field value must be less than a given value)
- ToleranceFieldGT (incoming field value must be greater than a given value)

A QLIST of such tolerances can be populated using a new helper:
arm_register_cpreg_mig_tolerance() and arm_cpu_match_cpreg_mig_tolerance() allows to check whether a tolerance exists for a given kvm index and its criterion is matched.

callers for those helpers will be introduced in subsequent patches.

---
v8 -> v9
- fix the tolerance type checking
- rename arm_cpu_cpreg_has_mig_tolerance into arm_cpu_match_cpreg_mig_tolerance

---
 target/arm/cpu.h | 33 +++++++++++++++++++++
 target/arm/cpu.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
target/arm/cpu.h +33 -0
index 657ff4ab20b..2b1d3af34f0 100644
@@ -843,6 +843,21 @@ struct ARMELChangeHook {
     QLIST_ENTRY(ARMELChangeHook) node;
 };
 
+typedef enum {
+    ToleranceNotOnBothEnds,
+    ToleranceDiffInMask,
+    ToleranceFieldLT,
+    ToleranceFieldGT,
+} ARMCPUCPREGMigToleranceType;
+
+typedef struct ARMCPUCPREGMigTolerance {
+    uint64_t kvmidx;
+    uint64_t mask;
+    uint64_t value;
+    ARMCPUCPREGMigToleranceType type;
+    QLIST_ENTRY(ARMCPUCPREGMigTolerance) node;
+} ARMCPUCPREGMigTolerance;
+
 /* These values map onto the return values for
  * QEMU_PSCI_0_2_FN_AFFINITY_INFO */
 typedef enum ARMPSCIState {
@@ -1139,6 +1154,7 @@ struct ArchCPU {
 
     QLIST_HEAD(, ARMELChangeHook) pre_el_change_hooks;
     QLIST_HEAD(, ARMELChangeHook) el_change_hooks;
+    QLIST_HEAD(, ARMCPUCPREGMigTolerance) cpreg_mig_tolerances;
 
     int32_t node_id; /* NUMA node this CPU belongs to */
 
@@ -2632,6 +2648,23 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
 void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void
         *opaque);
 
+/**
+ * arm_register_cpreg_mig_tolerance:
+ * Register a migration tolerance wrt one given cpreg identified by its
+ * @kvmidx. Only one tolerance can be registered by kvm reg idx.
+ */
+void arm_register_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                      uint64_t mask, uint64_t value,
+                                      ARMCPUCPREGMigToleranceType type);
+
+/**
+ * arm_cpu_match_cpreg_mig_tolerance:
+ * Check whether a tolerance of type @type exists for a given @kvmidx
+ * and the tolerance criterion is satified
+ */
+bool arm_cpu_match_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                       uint64_t vmstate_value, uint64_t local_value,
+                                       ARMCPUCPREGMigToleranceType type);
 /**
  * arm_rebuild_hflags:
  * Rebuild the cached TBFLAGS for arbitrary changed processor state.
target/arm/cpu.c +75 -0

99 lines not on this page · raw message

Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH v9 1/6] target/arm/cpu: Introduce the infrastructure for cpreg migration tolerances

Quoted 15 lines · Eric Auger

We introduce a datatype for a tolerance with respect to a given cpreg migration issue. The tolerance applies to a given cpreg kvm index, and can be of different types:
- ToleranceNotOnBothEnds (cpreg index is allowed to be only present on one end)
- ToleranceDiffInMask (value differences are allowed only within a mask)
- ToleranceFieldLT (incoming field value must be less than a given value)
- ToleranceFieldGT (incoming field value must be greater than a given value)

A QLIST of such tolerances can be populated using a new helper:
arm_register_cpreg_mig_tolerance() and arm_cpu_match_cpreg_mig_tolerance() allows to check whether a tolerance exists for a given kvm index and its criterion is matched.

Quoted 4 lines

callers for those helpers will be introduced in subsequent patches.

Thanks for reworking the approach to this. I like this overall, so my review comments are all minor stuff.

Quoted 7 lines

---
v8 -> v9
- fix the tolerance type checking
- rename arm_cpu_cpreg_has_mig_tolerance into arm_cpu_match_cpreg_mig_tolerance

---
 target/arm/cpu.h | 33 +++++++++++++++++++++

Could we put the new types, function prototypes, etc into internals.h or cpregs.h ? Generally we try to keep things needed only in other target/arm code in some other header than cpu.h, because cpu.h is pulled in by a lot of other files outside target/arm.

Quoted 15 lines
 target/arm/cpu.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
target/arm/cpu.h +4 -0
index 657ff4ab20b..2b1d3af34f0 100644
@@ -843,6 +843,21 @@ struct ARMELChangeHook {
     QLIST_ENTRY(ARMELChangeHook) node;
 };
 
+typedef enum {
+    ToleranceNotOnBothEnds,
+    ToleranceDiffInMask,
+    ToleranceFieldLT,

target/arm/cpu.h struct ARMELChangeHook { · line 859

Quoted 10 lines
target/arm/cpu.h +10 -0
+    ToleranceFieldGT,
+} ARMCPUCPREGMigToleranceType;
+
+typedef struct ARMCPUCPREGMigTolerance {
+    uint64_t kvmidx;
+    uint64_t mask;
+    uint64_t value;
+    ARMCPUCPREGMigToleranceType type;
+    QLIST_ENTRY(ARMCPUCPREGMigTolerance) node;
+} ARMCPUCPREGMigTolerance;

I don't think we need the "CPU" part of these type names. How about

ARMCPRegMigToleranceType
ARMCPRegMigTolerance

? That would match the prefix of the existing ARMCPRegInfo, ARMCPRegUserSpaceInfo types.

Quoted 13 lines
target/arm/cpu.h +2 -0
+
 /* These values map onto the return values for
  * QEMU_PSCI_0_2_FN_AFFINITY_INFO */
 typedef enum ARMPSCIState {
@@ -1139,6 +1154,7 @@ struct ArchCPU {
 
     QLIST_HEAD(, ARMELChangeHook) pre_el_change_hooks;
     QLIST_HEAD(, ARMELChangeHook) el_change_hooks;
+    QLIST_HEAD(, ARMCPUCPREGMigTolerance) cpreg_mig_tolerances;
 
     int32_t node_id; /* NUMA node this CPU belongs to */
 

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2654

Quoted 7 lines
target/arm/cpu.h +4 -0
@@ -2632,6 +2648,23 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
 void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void
         *opaque);
 
+/**
+ * arm_register_cpreg_mig_tolerance:
+ * Register a migration tolerance wrt one given cpreg identified by its
+ * @kvmidx. Only one tolerance can be registered by kvm reg idx.

We should say "Calling this function twice for the same @kvmidx is a programming error and will cause an assertion failure."

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2655

Quoted 1 line
target/arm/cpu.h +1 -0
+ */

Could we expand this doc comment to describe what the different kinds of tolerance we can register are? We mention them briefly in the commit message but I think that ought to be described here, including what the "mask" and "value" arguments to this function are for.

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2663

Quoted 8 lines
target/arm/cpu.h +8 -0
+void arm_register_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                      uint64_t mask, uint64_t value,
+                                      ARMCPUCPREGMigToleranceType type);
+
+/**
+ * arm_cpu_match_cpreg_mig_tolerance:
+ * Check whether a tolerance of type @type exists for a given @kvmidx
+ * and the tolerance criterion is satified

"satisfied"

Quoted 16 lines
target/arm/cpu.h +4 -0
+ */
+bool arm_cpu_match_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                       uint64_t vmstate_value, uint64_t local_value,
+                                       ARMCPUCPREGMigToleranceType type);
 /**
  * arm_rebuild_hflags:
  * Rebuild the cached TBFLAGS for arbitrary changed processor state.
target/arm/cpu.c +1 -0
index 7e3e84b4bbb..79e328c2888 100644
@@ -181,6 +181,75 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
     QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node);
 }
 
+void arm_register_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 194

Quoted 10 lines
target/arm/cpu.c +10 -0
+                                      uint64_t mask, uint64_t value,
+                                      ARMCPUCPREGMigToleranceType type)
+{
+    ARMCPUCPREGMigTolerance *t, *entry;
+
+    /* make sure the kvmidx has not tolerance already registered */
+    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
+        if (t->kvmidx == kvmidx) {
+            g_assert_not_reached();
+        }
    assert(kvmidx != t->kvm_idx);

is simpler (but see below).

Quoted 18 lines
target/arm/cpu.c +18 -0
+    }
+    entry = g_new0(ARMCPUCPREGMigTolerance, 1);
+
+    entry->kvmidx = kvmidx;
+    entry->mask = mask;
+    entry->value = value;
+    entry->type = type;
+
+    QLIST_INSERT_HEAD(&cpu->cpreg_mig_tolerances, entry, node);
+}
+
+bool arm_cpu_match_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                       uint64_t vmstate_value, uint64_t local_value,
+                                       ARMCPUCPREGMigToleranceType type)
+{
+    ARMCPUCPREGMigTolerance *t;
+    uint64_t diff, diff_outside_mask, field;
+    bool found = false;

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 222

Quoted 10 lines
target/arm/cpu.c +10 -0
+
+    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
+        if (t->kvmidx == kvmidx && t->type == type)  {
+            found = true;
+            break;
+        }
+    }
+    if (!found) {
+        return false;
+    }

How about we have a function

static ARMCPUCPRegMigTolerance *find_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx)

{
    ARMCPUCPREGMigTolerance *t;
    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
        if (t->kvmidx == kvmidx)  {
            return t;
        }
    }
    return NULL;
}

Then in arm_register_cpreg_mig_tolerance() we can write
assert(!find_mig_tolerance(cpu, kvmidx);

and in arm_cpu_match_cpreg_mig_tolerance() we can write
ARMCPUCPREGMigTolerance *t = find_mig_tolerance(cpu, kvmidx);

   if (!t || t->type != type) {
       return false;
   }

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 224

Quoted 2 lines
target/arm/cpu.c +2 -0
+
+    /* we found one tolerance woth @type associated to the @kvmidx */

"with" (but you might consider that with the naming of the function above, that makes this comment redundant)

Quoted 59 lines
target/arm/cpu.c +34 -1
+
+    if (type == ToleranceNotOnBothEnds) {
+        return true;
+    }
+
+    /* Need to check the mask */
+    diff = vmstate_value ^ local_value;
+    diff_outside_mask = diff & ~t->mask;
+
+    if (diff_outside_mask) {
+        /* there are differences outside of the mask */
+        return false;
+    }
+    if (type == ToleranceDiffInMask) {
+        /* differences only in the field, tolerance matched */
+        return true;
+    }
+    /* need to compare field value against authorized ones */
+    field = vmstate_value & t->mask;
+    if (type == ToleranceFieldLT && (field < t->value)) {
+        return true;
+    }
+    if (type == ToleranceFieldGT && (field > t->value)) {
+        return true;
+    }
+    return false;
+}
+
 static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque)
 {
     /* Reset a single ARMCPRegInfo register */
@@ -1106,6 +1175,7 @@ static void arm_cpu_initfn(Object *obj)
 
     QLIST_INIT(&cpu->pre_el_change_hooks);
     QLIST_INIT(&cpu->el_change_hooks);
+    QLIST_INIT(&cpu->cpreg_mig_tolerances);
 
 #ifdef CONFIG_USER_ONLY
 # ifdef TARGET_AARCH64
@@ -1550,6 +1620,7 @@ static void arm_cpu_finalizefn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
     ARMELChangeHook *hook, *next;
+    ARMCPUCPREGMigTolerance *t, *n;
 
     g_hash_table_destroy(cpu->cp_regs);
 
@@ -1561,6 +1632,10 @@ static void arm_cpu_finalizefn(Object *obj)
         QLIST_REMOVE(hook, node);
         g_free(hook);
     }
+    QLIST_FOREACH_SAFE(t, &cpu->cpreg_mig_tolerances, node, n) {
+        QLIST_REMOVE(t, node);
+        g_free(t);
+    }
 #ifndef CONFIG_USER_ONLY
     if (cpu->pmu_timer) {
         timer_free(cpu->pmu_timer);
--
Quoted 1 line

2.53.0

thanks
-- PMM

author
#3 lore raw

Re: [PATCH v9 1/6] target/arm/cpu: Introduce the infrastructure for cpreg migration tolerances

Hi Peter,

Quoted 188 lines · Peter Maydell
Quoted 14 lines · Eric Auger

We introduce a datatype for a tolerance with respect to a given cpreg migration issue. The tolerance applies to a given cpreg kvm index, and can be of different types:
- ToleranceNotOnBothEnds (cpreg index is allowed to be only present on one end)
- ToleranceDiffInMask (value differences are allowed only within a mask)
- ToleranceFieldLT (incoming field value must be less than a given value)
- ToleranceFieldGT (incoming field value must be greater than a given value)

A QLIST of such tolerances can be populated using a new helper:
arm_register_cpreg_mig_tolerance() and arm_cpu_match_cpreg_mig_tolerance() allows to check whether a tolerance exists for a given kvm index and its criterion is matched.

Quoted 4 lines

callers for those helpers will be introduced in subsequent patches.

Thanks for reworking the approach to this. I like this overall, so my review comments are all minor stuff.

Quoted 6 lines

---
v8 -> v9
- fix the tolerance type checking
- rename arm_cpu_cpreg_has_mig_tolerance into arm_cpu_match_cpreg_mig_tolerance

---
 target/arm/cpu.h | 33 +++++++++++++++++++++

Could we put the new types, function prototypes, etc into internals.h or cpregs.h ? Generally we try to keep things needed only in other target/arm code in some other header than cpu.h, because cpu.h is pulled in by a lot of other files outside target/arm.

Quoted 15 lines
 target/arm/cpu.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
target/arm/cpu.h +4 -0
index 657ff4ab20b..2b1d3af34f0 100644
@@ -843,6 +843,21 @@ struct ARMELChangeHook {
     QLIST_ENTRY(ARMELChangeHook) node;
 };
 
+typedef enum {
+    ToleranceNotOnBothEnds,
+    ToleranceDiffInMask,
+    ToleranceFieldLT,

target/arm/cpu.h struct ARMELChangeHook { · line 859

Quoted 10 lines
target/arm/cpu.h +10 -0
+    ToleranceFieldGT,
+} ARMCPUCPREGMigToleranceType;
+
+typedef struct ARMCPUCPREGMigTolerance {
+    uint64_t kvmidx;
+    uint64_t mask;
+    uint64_t value;
+    ARMCPUCPREGMigToleranceType type;
+    QLIST_ENTRY(ARMCPUCPREGMigTolerance) node;
+} ARMCPUCPREGMigTolerance;

I don't think we need the "CPU" part of these type names. How about

ARMCPRegMigToleranceType
ARMCPRegMigTolerance

? That would match the prefix of the existing ARMCPRegInfo, ARMCPRegUserSpaceInfo types.

Quoted 13 lines
target/arm/cpu.h +2 -0
+
 /* These values map onto the return values for
  * QEMU_PSCI_0_2_FN_AFFINITY_INFO */
 typedef enum ARMPSCIState {
@@ -1139,6 +1154,7 @@ struct ArchCPU {
 
     QLIST_HEAD(, ARMELChangeHook) pre_el_change_hooks;
     QLIST_HEAD(, ARMELChangeHook) el_change_hooks;
+    QLIST_HEAD(, ARMCPUCPREGMigTolerance) cpreg_mig_tolerances;
 
     int32_t node_id; /* NUMA node this CPU belongs to */
 

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2654

Quoted 7 lines
target/arm/cpu.h +4 -0
@@ -2632,6 +2648,23 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
 void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void
         *opaque);
 
+/**
+ * arm_register_cpreg_mig_tolerance:
+ * Register a migration tolerance wrt one given cpreg identified by its
+ * @kvmidx. Only one tolerance can be registered by kvm reg idx.

We should say "Calling this function twice for the same @kvmidx is a programming error and will cause an assertion failure."

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2655

Quoted 1 line
target/arm/cpu.h +1 -0
+ */

Could we expand this doc comment to describe what the different kinds of tolerance we can register are? We mention them briefly in the commit message but I think that ought to be described here, including what the "mask" and "value" arguments to this function are for.

target/arm/cpu.h void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 2663

Quoted 8 lines
target/arm/cpu.h +8 -0
+void arm_register_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                      uint64_t mask, uint64_t value,
+                                      ARMCPUCPREGMigToleranceType type);
+
+/**
+ * arm_cpu_match_cpreg_mig_tolerance:
+ * Check whether a tolerance of type @type exists for a given @kvmidx
+ * and the tolerance criterion is satified

"satisfied"

Quoted 16 lines
target/arm/cpu.h +4 -0
+ */
+bool arm_cpu_match_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                       uint64_t vmstate_value, uint64_t local_value,
+                                       ARMCPUCPREGMigToleranceType type);
 /**
  * arm_rebuild_hflags:
  * Rebuild the cached TBFLAGS for arbitrary changed processor state.
target/arm/cpu.c +1 -0
index 7e3e84b4bbb..79e328c2888 100644
@@ -181,6 +181,75 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
     QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node);
 }
 
+void arm_register_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 194

Quoted 10 lines
target/arm/cpu.c +10 -0
+                                      uint64_t mask, uint64_t value,
+                                      ARMCPUCPREGMigToleranceType type)
+{
+    ARMCPUCPREGMigTolerance *t, *entry;
+
+    /* make sure the kvmidx has not tolerance already registered */
+    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
+        if (t->kvmidx == kvmidx) {
+            g_assert_not_reached();
+        }
    assert(kvmidx != t->kvm_idx);

is simpler (but see below).

Quoted 18 lines
target/arm/cpu.c +18 -0
+    }
+    entry = g_new0(ARMCPUCPREGMigTolerance, 1);
+
+    entry->kvmidx = kvmidx;
+    entry->mask = mask;
+    entry->value = value;
+    entry->type = type;
+
+    QLIST_INSERT_HEAD(&cpu->cpreg_mig_tolerances, entry, node);
+}
+
+bool arm_cpu_match_cpreg_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx,
+                                       uint64_t vmstate_value, uint64_t local_value,
+                                       ARMCPUCPREGMigToleranceType type)
+{
+    ARMCPUCPREGMigTolerance *t;
+    uint64_t diff, diff_outside_mask, field;
+    bool found = false;

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 222

Quoted 10 lines
target/arm/cpu.c +10 -0
+
+    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
+        if (t->kvmidx == kvmidx && t->type == type)  {
+            found = true;
+            break;
+        }
+    }
+    if (!found) {
+        return false;
+    }

How about we have a function

static ARMCPUCPRegMigTolerance *find_mig_tolerance(ARMCPU *cpu, uint64_t kvmidx)

{
    ARMCPUCPREGMigTolerance *t;
    QLIST_FOREACH(t, &cpu->cpreg_mig_tolerances, node) {
        if (t->kvmidx == kvmidx)  {
            return t;
        }
    }
    return NULL;
}

Then in arm_register_cpreg_mig_tolerance() we can write
assert(!find_mig_tolerance(cpu, kvmidx);

and in arm_cpu_match_cpreg_mig_tolerance() we can write
ARMCPUCPREGMigTolerance *t = find_mig_tolerance(cpu, kvmidx);

   if (!t || t->type != type) {
       return false;
   }
Quoted 4 lines

target/arm/cpu.c void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, · line 224

Quoted 2 lines
target/arm/cpu.c +2 -0
+
+    /* we found one tolerance woth @type associated to the @kvmidx */

"with" (but you might consider that with the naming of the function above, that makes this comment redundant)

Took into account all your comments. Thanks!

Eric

Quoted 64 lines
Quoted 60 lines
target/arm/cpu.c +34 -1
+
+    if (type == ToleranceNotOnBothEnds) {
+        return true;
+    }
+
+    /* Need to check the mask */
+    diff = vmstate_value ^ local_value;
+    diff_outside_mask = diff & ~t->mask;
+
+    if (diff_outside_mask) {
+        /* there are differences outside of the mask */
+        return false;
+    }
+    if (type == ToleranceDiffInMask) {
+        /* differences only in the field, tolerance matched */
+        return true;
+    }
+    /* need to compare field value against authorized ones */
+    field = vmstate_value & t->mask;
+    if (type == ToleranceFieldLT && (field < t->value)) {
+        return true;
+    }
+    if (type == ToleranceFieldGT && (field > t->value)) {
+        return true;
+    }
+    return false;
+}
+
 static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque)
 {
     /* Reset a single ARMCPRegInfo register */
@@ -1106,6 +1175,7 @@ static void arm_cpu_initfn(Object *obj)
 
     QLIST_INIT(&cpu->pre_el_change_hooks);
     QLIST_INIT(&cpu->el_change_hooks);
+    QLIST_INIT(&cpu->cpreg_mig_tolerances);
 
 #ifdef CONFIG_USER_ONLY
 # ifdef TARGET_AARCH64
@@ -1550,6 +1620,7 @@ static void arm_cpu_finalizefn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
     ARMELChangeHook *hook, *next;
+    ARMCPUCPREGMigTolerance *t, *n;
 
     g_hash_table_destroy(cpu->cp_regs);
 
@@ -1561,6 +1632,10 @@ static void arm_cpu_finalizefn(Object *obj)
         QLIST_REMOVE(hook, node);
         g_free(hook);
     }
+    QLIST_FOREACH_SAFE(t, &cpu->cpreg_mig_tolerances, node, n) {
+        QLIST_REMOVE(t, node);
+        g_free(t);
+    }
 #ifndef CONFIG_USER_ONLY
     if (cpu->pmu_timer) {
         timer_free(cpu->pmu_timer);
--
Quoted 1 line

2.53.0

thanks
-- PMM

author
#4 lore raw

PATCH v9 2/6 target/arm/machine: Take account cpreg mig tolerances in case of mismatch

In reply to Eric Auger

If there is a mismatch between the cpreg indexes found on both ends, check whether a tolerance was registered for the given kvmidx. If any, silence warning/errors.

Create dedicated helper functions that print the name of the culprit reg and analyze whether a tolerance is set. According set the level of traces and analyze whether the migration must eventually fail.

---
 target/arm/machine.c    | 21 +++++++++++++++------
 target/arm/trace-events |  2 ++
 2 files changed, 17 insertions(+), 6 deletions(-)
target/arm/machine.c +15 -6
index 476dad00ee7..fa236431a8b 100644
@@ -1065,25 +1065,34 @@ static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx
 {
     g_autofree gchar *name = print_register_name(kvmidx);
 
+    if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          0, 0, ToleranceNotOnBothEnds)) {
+        trace_tolerate_cpreg_missing_in_incoming_stream(name);
+        return;
+    }
     warn_report("%s: %s "
                 "expected by the destination but not in the incoming stream: "
                  "skip it", __func__, name);
 }
 
 /*
- * Handle the situation where @kvmidx is in the incoming stream
- * but not on destination. This currently fails the migration but
- * we plan to accomodate some exceptions, hence the boolean returned value.
+ * Handle the situation where @kvmidx is in the incoming
+ * stream but not on destination. This fails the migration if
+ * no cpreg mig tolerance is matched for this @kvmidx
+ * Return true if the migration shall eventually fail
  */
 static bool handle_cpreg_only_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx)
 {
     g_autofree gchar *name = print_register_name(kvmidx);
-    bool fail = true;
 
+    if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          0, 0, ToleranceNotOnBothEnds)) {
+        trace_tolerate_cpreg_only_in_incoming_stream(name);
+        return false;
+    }
     error_report("%s: %s in the incoming stream but unknown on the "
                  "destination: fail migration", __func__, name);
-
-    return fail;
+    return true;
 }
 
 static int cpu_post_load(void *opaque, int version_id)
target/arm/trace-events +2 -0
index 2de0406f784..8502fb3265c 100644
@@ -29,3 +29,5 @@ arm_psci_call(uint64_t x0, uint64_t x1, uint64_t x2, uint64_t x3, uint32_t cpuid
 
 # machine.c
 cpu_post_load(uint32_t cpreg_vmstate_array_len, uint32_t cpreg_array_len) "cpreg_vmstate_array_len=%d cpreg_array_len=%d"
+tolerate_cpreg_missing_in_incoming_stream(char *name) "%s is missing in incoming stream but this is explicitly tolerated"
+tolerate_cpreg_only_in_incoming_stream(char *name) "%s is in incoming stream but not on destination but this is explicitly tolerated"
Signature · 1 line
2.53.0
Reviewed-by on v9 2/6
#5 lore raw

Re: [PATCH v9 2/6] target/arm/machine: Take account cpreg mig tolerances in case of mismatch

Quoted 8 lines · Eric Auger

If there is a mismatch between the cpreg indexes found on both ends, check whether a tolerance was registered for the given kvmidx. If any, silence warning/errors.

Create dedicated helper functions that print the name of the culprit reg and analyze whether a tolerance is set. According set the level of traces and analyze whether the migration must eventually fail.

This patch isn't creating any new functions -- is this a bit of commit message that is left over from a previous revision and needs rewording?

Quoted 23 lines
---
 target/arm/machine.c    | 21 +++++++++++++++------
 target/arm/trace-events |  2 ++
 2 files changed, 17 insertions(+), 6 deletions(-)
target/arm/machine.c +5 -0
index 476dad00ee7..fa236431a8b 100644
@@ -1065,25 +1065,34 @@ static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx
 {
     g_autofree gchar *name = print_register_name(kvmidx);
 
+    if (arm_cpu_match_cpreg_mig_tolerance(cpu, kvmidx,
+                                          0, 0, ToleranceNotOnBothEnds)) {
+        trace_tolerate_cpreg_missing_in_incoming_stream(name);
+        return;
+    }
     warn_report("%s: %s "
                 "expected by the destination but not in the incoming stream: "
                  "skip it", __func__, name);

target/arm/machine.c static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx · line 1082

Quoted 10 lines
target/arm/machine.c +4 -3
 }
 
 /*
- * Handle the situation where @kvmidx is in the incoming stream
- * but not on destination. This currently fails the migration but
- * we plan to accomodate some exceptions, hence the boolean returned value.
+ * Handle the situation where @kvmidx is in the incoming
+ * stream but not on destination. This fails the migration if
+ * no cpreg mig tolerance is matched for this @kvmidx
+ * Return true if the migration shall eventually fail

"should"

target/arm/machine.c static void handle_cpreg_missing_in_incoming_stream(ARMCPU *cpu, uint64_t kvmidx · line 1083

Quoted 1 line
target/arm/machine.c +0 -0
  */

Otherwise

thanks
-- PMM

author
#6 lore raw

PATCH v9 3/6 target/arm/cpu64: Mitigate migration failures due to spurious TCR_EL1, PIRE0_EL1 and PIR_EL1

In reply to Eric Auger

Before linux v6.13 those registers were erroneously unconditionally exposed and this was fixed by commits:
- 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace when disabled for guests")
- a68cddbe47ef ("KVM: arm64: Hide S1PIE registers from userspace when disabled for guests")
in v6.13.

This means if we migrate from an old kernel host to a >= 6.13 kernel host, migration currently fails.

Declare cpreg migration tolerance for those registers.

---
 target/arm/cpu64.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
target/arm/cpu64.c +26 -0
index d6feba220e8..4cfddc6e8c0 100644
@@ -810,6 +810,31 @@ static void aarch64_a53_initfn(Object *obj)
     define_cortex_a72_a57_a53_cp_reginfo(cpu);
 }
 
+static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
+{
+    /*
+     * Registers that may be in the incoming stream and not exposed
+     * on the destination
+     */
+
+    /*
+     * TCR_EL1 was erroneously unconditionnally exposed before linux v6.13.
+     * See commit 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace
+     * when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 2, 0, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+    /*
+     * PIRE0_EL1 and PIR_EL1 were erroneously unconditionnally exposed
+     * before linux v6.13. See commit a68cddbe47ef ("KVM: arm64: Hide
+     * S1PIE registers from userspace when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 2),
+                                     0, 0, ToleranceNotOnBothEnds);
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+}
+
 static void aarch64_host_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
@@ -822,6 +847,7 @@ static void aarch64_host_initfn(Object *obj)
 #endif
 
 #if defined(CONFIG_KVM)
+    kvm_arm_set_cpreg_mig_tolerances(cpu);
     kvm_arm_set_cpu_features_from_host(cpu);
     aarch64_add_sve_properties(obj);
 #elif defined(CONFIG_HVF)
Signature · 1 line
2.53.0
Reviewed-by on v9 3/6
#7 lore raw

Re: [PATCH v9 3/6] target/arm/cpu64: Mitigate migration failures due to spurious TCR_EL1, PIRE0_EL1 and PIR_EL1

Quoted 57 lines · Eric Auger

Before linux v6.13 those registers were erroneously unconditionally exposed and this was fixed by commits:
- 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace when disabled for guests")
- a68cddbe47ef ("KVM: arm64: Hide S1PIE registers from userspace when disabled for guests")
in v6.13.

This means if we migrate from an old kernel host to a >= 6.13 kernel host, migration currently fails.

Declare cpreg migration tolerance for those registers.

---
 target/arm/cpu64.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
target/arm/cpu64.c +25 -0
index d6feba220e8..4cfddc6e8c0 100644
@@ -810,6 +810,31 @@ static void aarch64_a53_initfn(Object *obj)
     define_cortex_a72_a57_a53_cp_reginfo(cpu);
 }
 
+static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
+{
+    /*
+     * Registers that may be in the incoming stream and not exposed
+     * on the destination
+     */
+
+    /*
+     * TCR_EL1 was erroneously unconditionnally exposed before linux v6.13.
+     * See commit 0fcb4eea5345 ("KVM: arm64: Hide TCR2_EL1 from userspace
+     * when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 2, 0, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+    /*
+     * PIRE0_EL1 and PIR_EL1 were erroneously unconditionnally exposed
+     * before linux v6.13. See commit a68cddbe47ef ("KVM: arm64: Hide
+     * S1PIE registers from userspace when disabled for guests")
+     */
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 2),
+                                     0, 0, ToleranceNotOnBothEnds);
+    arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
+                                     0, 0, ToleranceNotOnBothEnds);
+}
+
 static void aarch64_host_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);

target/arm/cpu64.c static void aarch64_host_initfn(Object *obj) · line 853

Quoted 7 lines
target/arm/cpu64.c +1 -0
@@ -822,6 +847,7 @@ static void aarch64_host_initfn(Object *obj)
 #endif
 
 #if defined(CONFIG_KVM)
+    kvm_arm_set_cpreg_mig_tolerances(cpu);
     kvm_arm_set_cpu_features_from_host(cpu);
     aarch64_add_sve_properties(obj);
 #elif defined(CONFIG_HVF)

If your only callsite for a static function is guarded by an ifdef then you need to also guard the function definition with the matching ifdef. Otherwise you get compilation errors (either clang complains but gcc is happy, or the other way around -- I forget which).

Otherwise

thanks
-- PMM

author
#8 lore raw

PATCH v9 4/6 target/arm/cpu64: Define cpreg migration tolerance for KVM_REG_ARM_VENDOR_HYP_BMAP_2

In reply to Eric Auger

KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed from v6.15 onwards. Backward migration from a >= v6.15 to an older kernel would fail without cpreg migration tolerance definition for this register.

---
 target/arm/cpu64.c | 8 ++++++++
 1 file changed, 8 insertions(+)
target/arm/cpu64.c +8 -0
index 4cfddc6e8c0..abc6b78ef7d 100644
@@ -833,6 +833,14 @@ static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
                                      0, 0, ToleranceNotOnBothEnds);
     arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
                                      0, 0, ToleranceNotOnBothEnds);
+
+    /*
+     * KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed
+     * from v6.15 onwards. Backward migration from a >= v6.15 to an older
+     * kernel would fail without cpreg migration tolerance definition
+     */
+    arm_register_cpreg_mig_tolerance(cpu, KVM_REG_ARM_FW_FEAT_BMAP_REG(3),
+                                     0, 0, ToleranceNotOnBothEnds);
 }
 
 static void aarch64_host_initfn(Object *obj)
Signature · 1 line
2.53.0
#9 lore raw

Re: [PATCH v9 4/6] target/arm/cpu64: Define cpreg migration tolerance for KVM_REG_ARM_VENDOR_HYP_BMAP_2

Quoted 19 lines · Eric Auger

KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed from v6.15 onwards. Backward migration from a >= v6.15 to an older kernel would fail without cpreg migration tolerance definition for this register.

---
 target/arm/cpu64.c | 8 ++++++++
 1 file changed, 8 insertions(+)
target/arm/cpu64.c +0 -0
index 4cfddc6e8c0..abc6b78ef7d 100644
@@ -833,6 +833,14 @@ static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
                                      0, 0, ToleranceNotOnBothEnds);
     arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),

target/arm/cpu64.c static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu) · line 844

Quoted 10 lines
target/arm/cpu64.c +8 -0
                                      0, 0, ToleranceNotOnBothEnds);
+
+    /*
+     * KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed
+     * from v6.15 onwards. Backward migration from a >= v6.15 to an older
+     * kernel would fail without cpreg migration tolerance definition
+     */
+    arm_register_cpreg_mig_tolerance(cpu, KVM_REG_ARM_FW_FEAT_BMAP_REG(3),
+                                     0, 0, ToleranceNotOnBothEnds);
 }

If I understand the intention of this pseudoregister correctly, it's for userspace control of what firmware features KVM exposes to the guest. In that regard it's a bit like the KVM_REG_ARM_PSCI_VERSION register, and as with that register I don't think the right approach is "disregard any differences on migration". If we exposed firmware feature X to the guest on the source then we need to also expose it to the guest on the destination, because it might be relying on it, and we shouldn't permit migration to a destination kernel that can't implement X.

But I'm also not entirely sure what the nature of the mismatch here is: https://docs.kernel.org/virt/kvm/arm/fw-pseudo-registers.html
says both
"By default, these registers are set with the upper limit of the features that are supported"
but also that for the BMAP_2 bits that
"This is reset to 0".

So does the register reset to 0 such that all we need to handle is "allow a source without the register as if it is zero, and allow a source with the register even if the destination doesn't have the register as long as the value is zero" ? Or is the "reset to 0" bit wrong, and in fact the kernel defaults to these bits being set, so we need something like the handling of kvm-psci-version where the user can suppress the features they need to give a "lowest common denominator" among the kernels they want to migrate between ?

thanks
-- PMM

author
#10 lore raw

Re: [PATCH v9 4/6] target/arm/cpu64: Define cpreg migration tolerance for KVM_REG_ARM_VENDOR_HYP_BMAP_2

Hi Peter,

Quoted 24 lines · Peter Maydell
Quoted 23 lines · Eric Auger

KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed from v6.15 onwards. Backward migration from a >= v6.15 to an older kernel would fail without cpreg migration tolerance definition for this register.

---
 target/arm/cpu64.c | 8 ++++++++
 1 file changed, 8 insertions(+)
target/arm/cpu64.c +4 -0
index 4cfddc6e8c0..abc6b78ef7d 100644
@@ -833,6 +833,14 @@ static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu)
                                      0, 0, ToleranceNotOnBothEnds);
     arm_register_cpreg_mig_tolerance(cpu, ARM64_SYS_REG(3, 0, 10, 2, 3),
                                      0, 0, ToleranceNotOnBothEnds);
+
+    /*
+     * KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register is exposed
+     * from v6.15 onwards. Backward migration from a >= v6.15 to an older
Quoted 8 lines

target/arm/cpu64.c static void kvm_arm_set_cpreg_mig_tolerances(ARMCPU *cpu) · line 844

Quoted 5 lines
target/arm/cpu64.c +4 -0
+     * kernel would fail without cpreg migration tolerance definition
+     */
+    arm_register_cpreg_mig_tolerance(cpu, KVM_REG_ARM_FW_FEAT_BMAP_REG(3),
+                                     0, 0, ToleranceNotOnBothEnds);
 }

If I understand the intention of this pseudoregister correctly, it's for userspace control of what firmware features KVM exposes to the guest. In that regard it's a bit like the KVM_REG_ARM_PSCI_VERSION

correct.

Quoted 6 lines

register, and as with that register I don't think the right approach is "disregard any differences on migration". If we exposed firmware feature X to the guest on the source then we need to also expose it to the guest on the destination, because it might be relying on it, and we shouldn't permit migration to a destination kernel that can't implement X.

Effectively if the source enabled those BMAP_2 features and we migrate to a dest that does not support those services, it looks wrong.

Quoted 8 lines

But I'm also not entirely sure what the nature of the mismatch here is: https://docs.kernel.org/virt/kvm/arm/fw-pseudo-registers.html
says both
"By default, these registers are set with the upper limit of the features that are supported"
but also that for the BMAP_2 bits that
"This is reset to 0".

Looking at kernel's arch/arm64/kvm/hypercalls.c

void kvm_arm_init_hypercalls(struct kvm *kvm)
{
        struct kvm_smccc_features *smccc_feat = &kvm->arch.smccc_feat;

        smccc_feat->std_bmap = KVM_ARM_SMCCC_STD_FEATURES; smccc_feat->std_hyp_bmap = KVM_ARM_SMCCC_STD_HYP_FEATURES; smccc_feat->vendor_hyp_bmap = KVM_ARM_SMCCC_VENDOR_HYP_FEATURES;

        mt_init(&kvm->arch.smccc_filter);
}

so effectively the reset value is 0 for smccc_feat->vendor_hyp_bmap_2 (BMAP_2) while for original BMAP it is set to
#define KVM_ARM_SMCCC_VENDOR_HYP_FEATURES                       \
        GENMASK(KVM_REG_ARM_VENDOR_HYP_BMAP_BIT_COUNT - 1, 0)

So the right version is:

"This is reset to 0"

Quoted 5 lines

So does the register reset to 0 such that all we need to handle is "allow a source without the register as if it is zero, and allow a source with the register even if the destination doesn't have the register as long as the value is zero" ? Or is the

Yes that looks the right statement:
- allow a source without the register as if it is zero
- allow a source with the register even if the destination doesn't have the register as long as the value is zero This would require another type of MigTolerance then.
My previous reasoning was that qemu currently does not mangle with BMAP_2 settings yet,
this will come with Shameer's series:

[RFC PATCH RESEND 0/4] hw/arm/virt: Add support for Target
Implementation CPUs <https://lore.kernel.org/qemu-devel/[email protected]/#r>
https://lore.kernel.org/qemu-devel/[email protected]/#r

But I agree this is not future proof.

Thank you for the detailed review!

Eric

Quoted 9 lines

"reset to 0" bit wrong, and in fact the kernel defaults to these bits being set, so we need something like the handling of kvm-psci-version where the user can suppress the features they need to give a "lowest common denominator" among the kernels they want to migrate between ?

thanks
-- PMM

author
#11 lore raw

PATCH v9 5/6 target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0

In reply to Eric Auger

We want to remove AArch32 DBGDTRTX which was erroneously exposed. This was attempted by 655659a74a36b ("target/arm: Correct encoding of Debug Communications Channel registers") but it was discovered that the removal of this debug register broke forward migration on TCG. Now we have the cpreg migration tolerance infrastructure, we can declare one for the DBGDTRTX. This allow to revert the reinstate patch.

---
 target/arm/helper.c | 5 +++++
 1 file changed, 5 insertions(+)
target/arm/helper.c +5 -0
index 7389f2988c4..3cba62705dc 100644
@@ -6426,9 +6426,14 @@ void register_cp_regs_for_features(ARMCPU *cpu)
             .fgt = FGT_CLIDR_EL1,
             .resetvalue = GET_IDREG(isar, CLIDR)
         };
+        uint64_t dbgtr_el0_kvmidx =
+            cpreg_to_kvm_id(ENCODE_CP_REG(14, 0, 1, 0, 5, 3, 0));
+
         define_one_arm_cp_reg(cpu, &clidr);
         define_arm_cp_regs(cpu, v7_cp_reginfo);
         define_debug_regs(cpu);
+        arm_register_cpreg_mig_tolerance(cpu, dbgtr_el0_kvmidx,
+                                         0, 0, ToleranceNotOnBothEnds);
     } else {
         define_arm_cp_regs(cpu, not_v7_cp_reginfo);
     }
Signature · 1 line
2.53.0
#12 lore raw

Re: [PATCH v9 5/6] target/arm/helper: Define cpreg migration tolerance for DGBDTR_EL0

Quoted 21 lines · Eric Auger

We want to remove AArch32 DBGDTRTX which was erroneously exposed. This was attempted by 655659a74a36b ("target/arm: Correct encoding of Debug Communications Channel registers") but it was discovered that the removal of this debug register broke forward migration on TCG. Now we have the cpreg migration tolerance infrastructure, we can declare one for the DBGDTRTX. This allow to revert the reinstate patch.

---
 target/arm/helper.c | 5 +++++
 1 file changed, 5 insertions(+)
target/arm/helper.c +0 -0
index 7389f2988c4..3cba62705dc 100644
@@ -6426,9 +6426,14 @@ void register_cp_regs_for_features(ARMCPU *cpu)
             .fgt = FGT_CLIDR_EL1,

target/arm/helper.c void register_cp_regs_for_features(ARMCPU *cpu) · line 6436

Quoted 10 lines
target/arm/helper.c +5 -0
             .resetvalue = GET_IDREG(isar, CLIDR)
         };
+        uint64_t dbgtr_el0_kvmidx =
+            cpreg_to_kvm_id(ENCODE_CP_REG(14, 0, 1, 0, 5, 3, 0));
+
         define_one_arm_cp_reg(cpu, &clidr);
         define_arm_cp_regs(cpu, v7_cp_reginfo);
         define_debug_regs(cpu);
+        arm_register_cpreg_mig_tolerance(cpu, dbgtr_el0_kvmidx,
+                                         0, 0, ToleranceNotOnBothEnds);

This needs a comment to explain why we're doing it.

thanks
-- PMM

author
#13 lore raw

PATCH v9 6/6 Revert "target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat"

In reply to Eric Auger

This reverts commit 4f2b82f60431 ("target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat). We don't need that commit anymore as the AArch32 DBGDTRTX register is declared to
be safe to ignore in the incoming migration stream.

---
 target/arm/debug_helper.c | 29 -----------------------------
 1 file changed, 29 deletions(-)
target/arm/debug_helper.c +0 -29
index 352c8e5c8e7..8477ca5def6 100644
@@ -171,13 +171,6 @@ static void dbgclaimclr_write(CPUARMState *env, const ARMCPRegInfo *ri,
     env->cp15.dbgclaim &= ~(value & 0xFF);
 }
 
-static CPAccessResult access_bogus(CPUARMState *env, const ARMCPRegInfo *ri,
-                                   bool isread)
-{
-    /* Always UNDEF, as if this cpreg didn't exist */
-    return CP_ACCESS_UNDEFINED;
-}
-
 static const ARMCPRegInfo debug_cp_reginfo[] = {
     /*
      * DBGDRAR, DBGDSAR: always RAZ since we don't implement memory mapped
@@ -240,28 +233,6 @@ static const ARMCPRegInfo debug_cp_reginfo[] = {
       .opc0 = 2, .opc1 = 3, .crn = 0, .crm = 4, .opc2 = 0,
       .access = PL0_RW, .accessfn = access_tdcc,
       .type = ARM_CP_CONST, .resetvalue = 0 },
-    /*
-     * This is not a real AArch32 register. We used to incorrectly expose
-     * this due to a QEMU bug; to avoid breaking migration compatibility we
-     * need to continue to provide it so that we don't fail the inbound
-     * migration when it tells us about a sysreg that we don't have.
-     * We set an always-fails .accessfn, which means that the guest doesn't
-     * actually see this register (it will always UNDEF, identically to if
-     * there were no cpreg definition for it other than that we won't print
-     * a LOG_UNIMP message about it), and we set the ARM_CP_NO_GDB flag so the
-     * gdbstub won't see it either.
-     * (We can't just set .access = 0, because add_cpreg_to_hashtable()
-     * helpfully ignores cpregs which aren't accessible to the highest
-     * implemented EL.)
-     *
-     * TODO: implement a system for being able to describe "this register
-     * can be ignored if it appears in the inbound stream"; then we can
-     * remove this temporary hack.
-     */
-    { .name = "BOGUS_DBGDTR_EL0", .state = ARM_CP_STATE_AA32,
-      .cp = 14, .opc1 = 3, .crn = 0, .crm = 5, .opc2 = 0,
-      .access = PL0_RW, .accessfn = access_bogus,
-      .type = ARM_CP_CONST | ARM_CP_NO_GDB, .resetvalue = 0 },
     /*
      * OSECCR_EL1 provides a mechanism for an operating system
      * to access the contents of EDECCR. EDECCR is not implemented though,
Signature · 1 line
2.53.0
Reviewed-by on v9 6/6
#14 lore raw

Re: [PATCH v9 6/6] Revert "target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat"

Quoted 9 lines · Eric Auger

This reverts commit 4f2b82f60431 ("target/arm: Reinstate bogus AArch32 DBGDTRTX register for migration compat). We don't need that commit anymore as the AArch32 DBGDTRTX register is declared to
be safe to ignore in the incoming migration stream.

---

thanks
-- PMM

#15 lore raw
In reply to Eric Auger
Quoted 10 lines · Eric Auger

This series applies on top of
[PATCH v3 0/7] Improve traces on migration failure due to cpreg number mismatch

It introduces an infrastructure to explicitly tolerate some mismatches between cpu cpreg indexes/values on migration.

We especially handle failures due to an attempt to migrate more registers than exposed on the destination.

Quoted 10 lines

Once the infrastructure is in place we can handle the following known issues:
KVM:
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
TCG:
- We can eventually remove AArch32 DBGDTRTX register which was erroneously defined.

Hi; I have this on my list to review, but I think this series has unfortunately missed the 11.0 freeze deadline, so we should aim to get it in for early in the 11.1 cycle.

thanks
-- PMM