Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures
cover Mitigation of "failed to load cpu:cpreg_vmstate_array_len" migration failures
When migrating ARM guests accross same machines with different host kernels we are likely to encounter failures such as:
"failed to load cpu:cpreg_vmstate_array_len"
This is due to the fact KVM exposes a different number of registers to qemu on source and destination. When trying to migrate a bigger register set to a smaller one, qemu cannot save the CPU state.
For example, recently we faced such kind of situations with:
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo register from v6.16 onwards. Causes backward migration failure.
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1, PIR_EL1 from v6.13 onwards. Causes forward migration failure.
This situation is really problematic for distributions which want to guarantee forward and backward migration of a given machine type between different releases.
This small series tries to address that issue by introducing CPU array properties that list the registers to ignore or to fake according to the situation. An example is given to illustrate how those props could be used to apply compats for machine types supposed to "see" the same register set accross various host kernels.
The first patch improves the tracing so that we can quickly detect which registers are unexpected and cause the migration failure. Missing registers are also traced. Those do not fail migration but their default value is kept on the destination.
Then we introduce the infrastructure to handle 'hidden' registers and 'fake' registers.
Eric Auger (7):
target/arm/machine: Improve traces on register mismatch during
migration
target/arm/kvm: Introduce the concept of hidden KVM regs target/arm/kvm: Introduce the concept of enforced/fake registers
kvm-all: Add the capability to blacklist some KVM regs target/arm/cpu: Implement hide_reg callback() target/arm/kvm: Expose kvm-hidden-regs and kvm-fake-regs properties hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older
kernels
include/hw/core/cpu.h | 2 ++ target/arm/cpu.h | 42 ++++++++++++++++++++++++ accel/kvm/kvm-all.c | 12 +++++++ hw/arm/virt.c | 19 +++++++++++ target/arm/cpu.c | 12 +++++++ target/arm/kvm.c | 73 ++++++++++++++++++++++++++++++++++++++++- target/arm/machine.c | 71 +++++++++++++++++++++++++++++++++++---- target/arm/trace-events | 11 +++++++ 8 files changed, 235 insertions(+), 7 deletions(-)
Signature · 1 line
2.49.0
PATCH RESEND 1/7 target/arm/machine: Improve traces on register mismatch during migration
Currently whenthe number of KVM registers exposed by the source is larger than the one exposed on the destination, the migration fails
with: "failed to load cpu:cpreg_vmstate_array_len"
This gives no information about which registers are causing the trouble.
This patches rework the target/arm/machine code so that it becomes able to handle an input stream with a larger set of registers than the destination and print useful information about which registers are causing the trouble. The migration outcome is unchanged:
- unexpected registers still will fail the migration
- missing ones are print but will not fail the migration, as done today.
The input stream can contain MAX_CPREG_VMSTATE_ANOMALIES(10) extra registers compared to what exists on the target.
If there are more registers we will still hit the previous "load cpu:cpreg_vmstate_array_len" error.
At most, MAX_CPREG_VMSTATE_ANOMALIES missing registers and MAX_CPREG_VMSTATE_ANOMALIES unexpected registers are print.
Example:
qemu-system-aarch64: kvm_arm_cpu_post_load Missing register in input stream: 0 0x6030000000160003 fw feat reg 3
qemu-system-aarch64: kvm_arm_cpu_post_load Unexpected register in input stream: 0 0x603000000013c103 op0:3 op1:0 crn:2 crm:0 op2:3
qemu-system-aarch64: kvm_arm_cpu_post_load Unexpected register in input stream: 1 0x603000000013c512 op0:3 op1:0 crn:10 crm:2 op2:2
qemu-system-aarch64: kvm_arm_cpu_post_load Unexpected register in input stream: 2 0x603000000013c513 op0:3 op1:0 crn:10 crm:2 op2:3
qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
qemu-system-aarch64: load of migration failed: Operation not permitted
--- target/arm/cpu.h | 6 +++++ target/arm/kvm.c | 23 ++++++++++++++++ target/arm/machine.c | 58 ++++++++++++++++++++++++++++++++++++----- target/arm/trace-events | 7 +++++ 4 files changed, 88 insertions(+), 6 deletions(-)
target/arm/cpu.h +6 -0
@@ -936,6 +936,12 @@ struct ArchCPU {uint64_t *cpreg_vmstate_values;int32_t cpreg_vmstate_array_len; +#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len; +DynamicGDBFeatureInfo dyn_sysreg_feature;DynamicGDBFeatureInfo dyn_svereg_feature;DynamicGDBFeatureInfo dyn_smereg_feature;
target/arm/kvm.c +23 -0
@@ -1023,6 +1023,29 @@ void kvm_arm_cpu_pre_save(ARMCPU *cpu)bool kvm_arm_cpu_post_load (ARMCPU *cpu) { +int i; + +for (i =0 ; i < cpu->cpreg_vmstate_missing_indexes_array_len; i++) { +gchar *name; + + name =kvm_print_register_name (cpu->cpreg_vmstate_missing_indexes[i]); +trace_kvm_arm_cpu_post_load_missing_reg (name); +g_free (name); + } + +for (i =0 ; i < cpu->cpreg_vmstate_unexpected_indexes_array_len; i++) { +gchar *name; + + name =kvm_print_register_name (cpu->cpreg_vmstate_unexpected_indexes[i]); +error_report ("%s Unexpected register in input stream: %i 0x%" PRIx64" %s" , + __func__, i, cpu->cpreg_vmstate_unexpected_indexes[i], name); +g_free (name); + } +/* Fail the migration if we detect unexpected registers */ +if (cpu->cpreg_vmstate_unexpected_indexes_array_len) { +return false; + } +if (!write_list_to_kvmstate (cpu,KVM_PUT_FULL_STATE )) {return false; }
target/arm/machine.c +52 -6
99 lines not on this page · raw message
target/arm/trace-events +7 -0
@@ -13,6 +13,7 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq: timer %d irqstate %d" # kvm.c kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova = 0x%"PRIx64" is translated into 0x%"PRIx64 +kvm_arm_cpu_post_load_missing_reg(char *name) "Missing register in input stream: %s" # cpu.c arm_cpu_reset(uint64_t mp_aff) "cpu %" PRIu64
@@ -23,3 +24,9 @@ arm_powerctl_set_cpu_on(uint64_t mp_aff, unsigned target_el, const char *mode, u arm_powerctl_set_cpu_on_and_reset(uint64_t mp_aff) "cpu %" PRIu64 arm_powerctl_set_cpu_off(uint64_t mp_aff) "cpu %" PRIu64 arm_powerctl_reset_cpu(uint64_t mp_aff) "cpu %" PRIu64 + +# machine.c +cpu_post_load_len(int cpreg_array_len, int cpreg_vmstate_array_len) "cpreg_array_len=%d cpreg_vmstate_array_len=%d" +cpu_post_load(int i, int v, uint64_t regidx) "i=%d v=%d regidx=0x%"PRIx64 +cpu_post_load_missing(int i, uint64_t regidx, int v) "missing register in input stream: i=%d index=0x%"PRIx64" (v=%d)" +cpu_post_load_unexpected(int v, uint64_t regidx, int i) "unexpected register in input stream: v=%d index=0x%"PRIx64" (i=%d)"
Signature · 1 line
2.49.0
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
On Thu, Oct 16 2025, Eric Auger <[email protected]> wrote:
More information is really valuable here. I have some nits :)
Quoted 1 line
Currently whenthe number of KVM registers exposed by the source is
s/whenthe/when the/
Quoted 6 lines
larger than the one exposed on the destination, the migration fails
with: "failed to load cpu:cpreg_vmstate_array_len"This gives no information about which registers are causing the trouble.
This patches rework the target/arm/machine code so that it becomes
s/patches rework/patch reworks/
Quoted 5 lines
able to handle an input stream with a larger set of registers than the destination and print useful information about which registers are causing the trouble. The migration outcome is unchanged:
- unexpected registers still will fail the migration
- missing ones are print but will not fail the migration, as done today.
s/print/printed/
Quoted 9 lines
The input stream can contain
MAX_CPREG_VMSTATE_ANOMALIES(10)extra registers compared to what exists on the target.If there are more registers we will still hit the previous "load cpu:cpreg_vmstate_array_len" error.
At most,
MAX_CPREG_VMSTATE_ANOMALIESmissing registers andMAX_CPREG_VMSTATE_ANOMALIESunexpected registers are print.
s/print/printed/
If we really get tons of register discrepancies, I'd expect the reason for that to be something more obvious, so limiting should be fine.
Quoted 23 lines
Example:
qemu-system-aarch64:
kvm_arm_cpu_post_loadMissing register in input stream: 00x6030000000160003fw feat reg 3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 00x603000000013c103op0:3 op1:0 crn:2 crm:0 op2:3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 10x603000000013c512op0:3 op1:0 crn:10 crm:2 op2:2
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 20x603000000013c513op0:3 op1:0 crn:10 crm:2 op2:3
qemu-system-aarch64: error while loading state for instance0x0of device 'cpu'
qemu-system-aarch64: load of migration failed: Operation not permittedSigned-off-by Eric Auger <[email protected]>--- target/arm/cpu.h | 6 +++++ target/arm/kvm.c | 23 ++++++++++++++++ target/arm/machine.c | 58 ++++++++++++++++++++++++++++++++++++----- target/arm/trace-events | 7 +++++ 4 files changed, 88 insertions(+), 6 deletions(-)target/arm/cpu.h +0
-0
target/arm/cpu.h struct ArchCPU { · line 943
Quoted 8 lines
target/arm/cpu.h +5
-0@@ -936,6 +936,12 @@ struct ArchCPU {uint64_t *cpreg_vmstate_values;int32_t cpreg_vmstate_array_len; +#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;
"indices"?
Quoted 4 lines
target/arm/cpu.h +1
-0+DynamicGDBFeatureInfo dyn_sysreg_feature;DynamicGDBFeatureInfo dyn_svereg_feature;DynamicGDBFeatureInfo dyn_smereg_feature;
(...)
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
Hi Connie,
Quoted 60 lines · Cornelia Huck
On Thu, Oct 16 2025, Eric Auger <[email protected]> wrote:
More information is really valuable here. I have some nits :)
Quoted 1 line
Currently whenthe number of KVM registers exposed by the source is
s/whenthe/when the/
Quoted 6 lines
larger than the one exposed on the destination, the migration fails
with: "failed to load cpu:cpreg_vmstate_array_len"This gives no information about which registers are causing the trouble.
This patches rework the target/arm/machine code so that it becomes
s/patches rework/patch reworks/
Quoted 5 lines
able to handle an input stream with a larger set of registers than the destination and print useful information about which registers are causing the trouble. The migration outcome is unchanged:
- unexpected registers still will fail the migration
- missing ones are print but will not fail the migration, as done today.s/print/printed/
Quoted 8 lines
The input stream can contain
MAX_CPREG_VMSTATE_ANOMALIES(10)extra registers compared to what exists on the target.If there are more registers we will still hit the previous "load cpu:cpreg_vmstate_array_len" error.
At most,
MAX_CPREG_VMSTATE_ANOMALIESmissing registers andMAX_CPREG_VMSTATE_ANOMALIESunexpected registers are print.s/print/printed/
If we really get tons of register discrepancies, I'd expect the reason for that to be something more obvious, so limiting should be fine.
Quoted 25 lines
Example:
qemu-system-aarch64:
kvm_arm_cpu_post_loadMissing register in input stream: 00x6030000000160003fw feat reg 3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 00x603000000013c103op0:3 op1:0 crn:2 crm:0 op2:3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 10x603000000013c512op0:3 op1:0 crn:10 crm:2 op2:2
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 20x603000000013c513op0:3 op1:0 crn:10 crm:2 op2:3
qemu-system-aarch64: error while loading state for instance0x0of device 'cpu'
qemu-system-aarch64: load of migration failed: Operation not permittedSigned-off-by Eric Auger <[email protected]>--- target/arm/cpu.h | 6 +++++ target/arm/kvm.c | 23 ++++++++++++++++ target/arm/machine.c | 58 ++++++++++++++++++++++++++++++++++++----- target/arm/trace-events | 7 +++++ 4 files changed, 88 insertions(+), 6 deletions(-)target/arm/cpu.h +0
-0@@ -936,6 +936,12 @@ struct ArchCPU {uint64_t *cpreg_vmstate_values;int32_t cpreg_vmstate_array_len;
Quoted 6 lines
target/arm/cpu.h struct ArchCPU { · line 943
Quoted 5 lines
target/arm/cpu.h +5
-0+#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;"indices"?
Thanks, all those will be fixed in next version.
Thanks
Eric
Quoted 7 lines
Quoted 5 lines
target/arm/cpu.h +1
-0+DynamicGDBFeatureInfo dyn_sysreg_feature;DynamicGDBFeatureInfo dyn_svereg_feature;DynamicGDBFeatureInfo dyn_smereg_feature;(...)
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
In reply to Cornelia HuckQuoted 60 lines · Cornelia Huck
On Thu, Oct 16 2025, Eric Auger <[email protected]> wrote:
More information is really valuable here. I have some nits :)
Quoted 1 line
Currently whenthe number of KVM registers exposed by the source is
s/whenthe/when the/
Quoted 6 lines
larger than the one exposed on the destination, the migration fails
with: "failed to load cpu:cpreg_vmstate_array_len"This gives no information about which registers are causing the trouble.
This patches rework the target/arm/machine code so that it becomes
s/patches rework/patch reworks/
Quoted 5 lines
able to handle an input stream with a larger set of registers than the destination and print useful information about which registers are causing the trouble. The migration outcome is unchanged:
- unexpected registers still will fail the migration
- missing ones are print but will not fail the migration, as done today.s/print/printed/
Quoted 8 lines
The input stream can contain
MAX_CPREG_VMSTATE_ANOMALIES(10)extra registers compared to what exists on the target.If there are more registers we will still hit the previous "load cpu:cpreg_vmstate_array_len" error.
At most,
MAX_CPREG_VMSTATE_ANOMALIESmissing registers andMAX_CPREG_VMSTATE_ANOMALIESunexpected registers are print.s/print/printed/
If we really get tons of register discrepancies, I'd expect the reason for that to be something more obvious, so limiting should be fine.
Quoted 25 lines
Example:
qemu-system-aarch64:
kvm_arm_cpu_post_loadMissing register in input stream: 00x6030000000160003fw feat reg 3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 00x603000000013c103op0:3 op1:0 crn:2 crm:0 op2:3
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 10x603000000013c512op0:3 op1:0 crn:10 crm:2 op2:2
qemu-system-aarch64:kvm_arm_cpu_post_loadUnexpected register in input stream: 20x603000000013c513op0:3 op1:0 crn:10 crm:2 op2:3
qemu-system-aarch64: error while loading state for instance0x0of device 'cpu'
qemu-system-aarch64: load of migration failed: Operation not permittedSigned-off-by Eric Auger <[email protected]>--- target/arm/cpu.h | 6 +++++ target/arm/kvm.c | 23 ++++++++++++++++ target/arm/machine.c | 58 ++++++++++++++++++++++++++++++++++++----- target/arm/trace-events | 7 +++++ 4 files changed, 88 insertions(+), 6 deletions(-)target/arm/cpu.h +0
-0@@ -936,6 +936,12 @@ struct ArchCPU {uint64_t *cpreg_vmstate_values;int32_t cpreg_vmstate_array_len;
Quoted 6 lines
target/arm/cpu.h struct ArchCPU { · line 943
Quoted 5 lines
target/arm/cpu.h +5
-0+#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;"indices"?
Originally we had
uint64_t *cpreg_vmstate_indexes;
so I reused the same terminology
As a non native english speaker I don't know if the usage is wrong. I thought some references on the net though
Eric
Quoted 7 lines
Quoted 5 lines
target/arm/cpu.h +1
-0+DynamicGDBFeatureInfo dyn_sysreg_feature;DynamicGDBFeatureInfo dyn_svereg_feature;DynamicGDBFeatureInfo dyn_smereg_feature;(...)
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
Quoted 5 lines · Eric Auger
Quoted 5 lines
Quoted 5 lines
target/arm/cpu.h +5
-0+#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;
Quoted 8 lines
Quoted 1 line
"indices"?
Originally we had
uint64_t*cpreg_vmstate_indexes;
so I reused the same terminologyAs a non native english speaker I don't know if the usage is wrong. I thought some references on the net though
Not a native English speaker, either; wiktionary says both are valid, so probably a matter of taste.
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
Quoted 14 lines · Cornelia Huck
Quoted 11 lines · Eric Auger
Quoted 6 lines
target/arm/cpu.h struct ArchCPU { · line 943
Quoted 5 lines
target/arm/cpu.h +5
-0+#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;"indices"?
Originally we had
uint64_t*cpreg_vmstate_indexes;
so I reused the same terminology
Quoted 5 lines
Quoted 2 lines
As a non native english speaker I don't know if the usage is wrong. I thought some references on the net though
Not a native English speaker, either; wiktionary says both are valid, so probably a matter of taste.
Mmm. I tend to go with "indexes" as being clearer (especially for non-native speakers) than "indices". Within QEMU sources there are twice as many "indexes" as "indices".
thanks
-- PMM
Re: [RESEND PATCH 1/7] target/arm/machine: Improve traces on register mismatch during migration
Quoted 13 lines · Peter Maydell
Quoted 12 lines · Cornelia Huck
Quoted 10 lines · Eric Auger
Quoted 6 lines
target/arm/cpu.h struct ArchCPU { · line 943
Quoted 5 lines
target/arm/cpu.h +5
-0+#define MAX_CPREG_VMSTATE_ANOMALIES 10 +uint64_t cpreg_vmstate_missing_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_missing_indexes_array_len; +uint64_t cpreg_vmstate_unexpected_indexes[MAX_CPREG_VMSTATE_ANOMALIES ]; +int32_t cpreg_vmstate_unexpected_indexes_array_len;"indices"?
Originally we had
uint64_t*cpreg_vmstate_indexes;
so I reused the same terminology
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
As a non native english speaker I don't know if the usage is wrong. I thought some references on the net though
Not a native English speaker, either; wiktionary says both are valid, so probably a matter of taste.
Mmm. I tend to go with "indexes" as being clearer (especially for non-native speakers) than "indices". Within QEMU sources there are twice as many "indexes" as "indices".
OK thanks. I will keep it.
Eric
Quoted 4 lines
thanks
-- PMM
PATCH RESEND 2/7 target/arm/kvm: Introduce the concept of hidden KVM regs
In reply to Eric AugerMore recent kernels sometimes expose new registers in an unconditionnal manner. This situation breaks backward migration as qemu notices there are more registers in the input stream than supported on the destination host. This leads to a "failed to load cpu:cpreg_vmstate_array_len" error.
A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward migration from a host kernel that features the commit to a destination host that doesn't, fail with above error.
Currently QEMU is not using that feature so ignoring this latter is not a problem. An easy way to fix the migration issue is to teach qemu we don't care about that register and we can simply ignore it when syncing its state during migration.
This patch introduces an array of such hidden registers. Soon it will be settable through an array property.
If hidden, the register is moved out of the array of cpreg which is built in kvm_arm_init_cpreg_list(). That way their state won't be synced.
---
v1 -> v2:
- Move the property in a separate patch
- improve the commit msg
- change the trace point to just print info in
kvm_arm_init_cpreg_list()
- improve comment in cpu.h (Connie)
--- target/arm/cpu.h | 11 +++++++++++ target/arm/kvm.c | 31 ++++++++++++++++++++++++++++++- target/arm/trace-events | 2 ++ 3 files changed, 43 insertions(+), 1 deletion(-)
target/arm/cpu.h +11 -0
@@ -1041,6 +1041,17 @@ struct ArchCPU {/* KVM steal time */ OnOffAuto kvm_steal_time; +/* +* KVM registers that must be ignored/hidden. While they may be +* exposed by KVM to userspace, they are not intended to be used +* by qemu and more importantly we don't want them to be migrated +* to another host which wouldn't expose them. This would break +* the migration. This may be useful to allow backward migration to +* older kernels with less features. +*/ +uint64_t *kvm_hidden_regs; +uint32_t nr_kvm_hidden_regs; +/* Uniprocessor system with MP extensions */ bool mp_is_up;
target/arm/kvm.c +30 -1
@@ -754,6 +754,25 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx) } } +/** +* kvm_vcpu_compat_hidden_reg: +* @cpu: ARMCPU +* @regidx: index of the register to check +* +* Depending on the CPU compat returns true if @regidx must be +* ignored during sync & migration +*/ +static inline bool +kvm_vcpu_compat_hidden_reg (ARMCPU *cpu,uint64_t regidx) +{ +for (int i =0 ; i < cpu->nr_kvm_hidden_regs; i++) { +if (cpu->kvm_hidden_regs[i] == regidx) { +return true; + } + } +return false; +} +/** * kvm_arm_init_cpreg_list: * @cpu: ARMCPU
@@ -788,7 +807,10 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)qsort (&rlp->reg, rlp->n,sizeof (rlp->reg[0 ]), compare_u64);for (i =0 , arraylen =0 ; i < rlp->n; i++) { -if (!kvm_arm_reg_syncs_via_cpreg_list (rlp->reg[i])) { +uint64_t regidx = rlp->reg[i]; + +if (!kvm_arm_reg_syncs_via_cpreg_list (regidx) || +kvm_vcpu_compat_hidden_reg (cpu, regidx)) {continue ; }switch (rlp->reg[i] &KVM_REG_SIZE_MASK ) {
@@ -804,6 +826,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) arraylen++; } +trace_kvm_arm_init_cpreg_list_arraylen (arraylen); + cpu->cpreg_indexes =g_renew (uint64_t, cpu->cpreg_indexes, arraylen); cpu->cpreg_values =g_renew (uint64_t, cpu->cpreg_values, arraylen); cpu->cpreg_vmstate_indexes = g_renew(uint64_t, cpu->cpreg_vmstate_indexes,
@@ -815,9 +839,14 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)for (i =0 , arraylen =0 ; i < rlp->n; i++) {uint64_t regidx = rlp->reg[i]; +if (!kvm_arm_reg_syncs_via_cpreg_list (regidx)) {continue ; } +if (kvm_vcpu_compat_hidden_reg (cpu, regidx)) { +trace_kvm_arm_init_cpreg_list_skip_hidden_reg (rlp->reg[i]); +continue ; + } cpu->cpreg_indexes[arraylen] = regidx; arraylen++; }
target/arm/trace-events +2 -0
@@ -14,6 +14,8 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq: timer %d irqstate %d" # kvm.c kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova = 0x%"PRIx64" is translated into 0x%"PRIx64 kvm_arm_cpu_post_load_missing_reg(char *name) "Missing register in input stream: %s" +kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=%d" +kvm_arm_init_cpreg_list_skip_hidden_reg(uint64_t regidx) "hidden 0x%"PRIx64" is skipped" # cpu.c arm_cpu_reset(uint64_t mp_aff) "cpu %" PRIu64
Signature · 1 line
2.49.0
PATCH RESEND 3/7 target/arm/kvm: Introduce the concept of enforced/fake registers
In reply to Eric AugerNewer kernels may revoke exposure of KVM regs to userspace. This can happen when one notices that some registers were unconditionnally exposed whether they shall be conditionnally exposed for example.
An example of such situation is: TCR2_EL1, PIRE0_EL1, PIR_EL1. Associated kernel commits were:
0fcb4eea5345 KVM: arm64: Hide TCR2_EL1 from userspace when disabled for guests a68cddbe47ef KVM: arm64: Hide S1PIE registers from userspace when disabled for guests
Those commits were actual fixes but the cons is that is breaks forward migration on some HW. Indeed when migrating from an old kernel that does not feature those commits to a more recent one, destination qemu detects there are more KVM regs in the input migration stream than exposed by the destination host and the migration fails with:
"failed to load cpu:cpreg_vmstate_array_len"
This patchs adds the capability to defined an array of enforced register indexes that teaches QEMU that some registers must exist. If they are not exposed by KVM qemu fakes their presence in the preload phase by adjusting the size of the cpreg_vmstate arrays. On postload we make sure to ignore them when analyzing potential mismatch between registers. The actual cpreg array is never altered meaning those registers are never accessed nor saved.
The array will be populated with a deficated array property.
---
v1 -> v2:
- improve comment in target/arm/cpu.h (Connie)
--- target/arm/cpu.h | 23 +++++++++++++++++++++++ target/arm/kvm.c | 24 ++++++++++++++++++++++++ target/arm/machine.c | 31 ++++++++++++++++++++++--------- target/arm/trace-events | 2 ++ 4 files changed, 71 insertions(+), 9 deletions(-)
target/arm/cpu.h +23 -0
@@ -1052,6 +1052,19 @@ struct ArchCPU {uint64_t *kvm_hidden_regs;uint32_t nr_kvm_hidden_regs; +/* +* KVM registers whose presence must be enforced +* Either they must be exposed to user space by KVM or +* they must be faked for migration sake because the source does +* feature them. +*/ +uint64_t *kvm_enforced_regs; +uint32_t nr_kvm_enforced_regs; + +/* registers that must be be faked */ +uint64_t *kvm_fake_regs; +uint32_t nr_kvm_fake_regs; +/* Uniprocessor system with MP extensions */ bool mp_is_up;
@@ -2652,6 +2665,16 @@ static inline uint64_t *aa64_vfp_qreg(CPUARMState *env, unsigned regno)/* Shared between translate-sve.c and sve_helper.c. */ extern const uint64_t pred_esz_masks[5 ]; +static inline bool is_fake_reg (ARMCPU *cpu,uint64_t regidx) +{ +for (int i =0 ; i < cpu->nr_kvm_fake_regs; i++) { +if (regidx == cpu->kvm_fake_regs[i]) { +return true; + } + } +return false; +} +/* * AArch64 usage of the PAGE_TARGET_* bits for linux-user. * Note that with the Linux kernel, PROT_MTE may not be cleared by mprotect
target/arm/kvm.c +24 -0
@@ -788,6 +788,7 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)struct kvm_reg_list rl;struct kvm_reg_list *rlp;int i, ret, arraylen; +int nr_fake_regs =0 ;CPUState *cs =CPU (cpu); rl.n =0 ;
@@ -801,6 +802,29 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)if (ret) { goto out; } + +/* +* Identify which registers belonging to the enforced list +* need to be faked because they are not exposed to userspace +* by KVM +*/ + cpu->kvm_fake_regs =g_new (uint64_t, cpu->nr_kvm_enforced_regs); +for (int j =0 ; j < cpu->nr_kvm_enforced_regs; j++) { +uint64_t v64; +int r; + + r =kvm_get_one_reg (cs, cpu->kvm_enforced_regs[j], &v64); +if (errno !=ENOENT ) { +/* enforced register exists, no need to fake */ +trace_kvm_arm_init_cpreg_exposed (cpu->kvm_enforced_regs[j], v64, r); +continue ; + } + cpu->kvm_fake_regs[nr_fake_regs] = cpu->kvm_enforced_regs[j]; +trace_kvm_arm_init_cpreg_fake (cpu->kvm_fake_regs[nr_fake_regs]); + nr_fake_regs++; + } + cpu->nr_kvm_fake_regs = nr_fake_regs; +/* Sort the list we get back from the kernel, since cpreg_tuples * must be in strictly ascending order. */
target/arm/machine.c +22 -9
@@ -990,7 +990,8 @@ static int cpu_pre_load(void *opaque) {ARMCPU *cpu = opaque;CPUARMState *env = &cpu->env; -int arraylen = cpu->cpreg_vmstate_array_len +MAX_CPREG_VMSTATE_ANOMALIES ; +int arraylen = cpu->cpreg_vmstate_array_len + + cpu->nr_kvm_fake_regs +MAX_CPREG_VMSTATE_ANOMALIES ; cpu->cpreg_vmstate_indexes =g_renew (uint64_t, cpu->cpreg_vmstate_indexes, arraylen);
@@ -1057,6 +1058,10 @@ static int cpu_post_load(void *opaque, int version_id)* entries with the right slots in our own values array. */ +/* +* at this point cpu->cpreg_vmstate_array_len was migrated with the +* actual length saved on source +*/ trace_cpu_post_load_len (cpu->cpreg_array_len, cpu->cpreg_vmstate_array_len);for (; i < cpu->cpreg_array_len && v < cpu->cpreg_vmstate_array_len;) {trace_cpu_post_load (i, v , cpu->cpreg_indexes[i]);
@@ -1071,10 +1076,14 @@ static int cpu_post_load(void *opaque, int version_id) }if (cpu->cpreg_vmstate_indexes[v] < cpu->cpreg_indexes[i]) {/* register in their list but not ours: those will fail migration */ -trace_cpu_post_load_unexpected (v, cpu->cpreg_vmstate_indexes[v], i); -if (k <MAX_CPREG_VMSTATE_ANOMALIES ) { - cpu->cpreg_vmstate_unexpected_indexes[k++] = - cpu->cpreg_vmstate_indexes[v]; +if (is_fake_reg (cpu, cpu->cpreg_vmstate_indexes[v])) { +/* this absence was expected as the reg was tagged as fake */ + }else { +trace_cpu_post_load_unexpected (v, cpu->cpreg_vmstate_indexes[v], i); +if (k <MAX_CPREG_VMSTATE_ANOMALIES ) { + cpu->cpreg_vmstate_unexpected_indexes[k++] = + cpu->cpreg_vmstate_indexes[v]; + } } v++;continue ;
@@ -1100,10 +1109,14 @@ static int cpu_post_load(void *opaque, int version_id)* still regs in the input stream, continue parsing the vmstate array */ for ( ; v < cpu->cpreg_vmstate_array_len; v++) { -if (k <MAX_CPREG_VMSTATE_ANOMALIES ) { -trace_cpu_post_load_unexpected (v, cpu->cpreg_vmstate_indexes[v], i); - cpu->cpreg_vmstate_unexpected_indexes[k++] = - cpu->cpreg_vmstate_indexes[v]; +if (is_fake_reg (cpu, cpu->cpreg_vmstate_indexes[v])) { +/* this absence was expected as the reg was tagged as fake */ + }else { +if (k <MAX_CPREG_VMSTATE_ANOMALIES ) { +trace_cpu_post_load_unexpected (v, cpu->cpreg_vmstate_indexes[v], i); + cpu->cpreg_vmstate_unexpected_indexes[k++] = + cpu->cpreg_vmstate_indexes[v]; + } } }
target/arm/trace-events +2 -0
@@ -16,6 +16,8 @@ kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova = 0x%"PRIx64" is kvm_arm_cpu_post_load_missing_reg(char *name) "Missing register in input stream: %s" kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=%d" kvm_arm_init_cpreg_list_skip_hidden_reg(uint64_t regidx) "hidden 0x%"PRIx64" is skipped" +kvm_arm_init_cpreg_exposed(uint64_t regidx, uint64_t val, int ret) "enforced reg 0x%"PRIx64" is already exposed by KVM: value=0x%"PRIx64 " ret=%d: nothing to do" +kvm_arm_init_cpreg_fake(uint64_t regidx) "register 0x%"PRIx64 " is not exposed and is faked" # cpu.c arm_cpu_reset(uint64_t mp_aff) "cpu %" PRIu64
Signature · 1 line
2.49.0
Re: [RESEND PATCH 3/7] target/arm/kvm: Introduce the concept of enforced/fake registers
Quoted 17 lines · Eric Auger
Newer kernels may revoke exposure of KVM regs to userspace. This can happen when one notices that some registers were unconditionnally exposed whether they shall be conditionnally exposed for example.
An example of such situation is:
TCR2_EL1,PIRE0_EL1,PIR_EL1. Associated kernel commits were:0fcb4eea5345KVM: arm64: HideTCR2_EL1from userspace when disabled for guestsa68cddbe47efKVM: arm64: Hide S1PIE registers from userspace when disabled for guestsThose commits were actual fixes but the cons is that is breaks forward migration on some HW. Indeed when migrating from an old kernel that does not feature those commits to a more recent one, destination qemu detects there are more KVM regs in the input migration stream than exposed by the destination host and the migration fails with:
"failed to load cpu:cpreg_vmstate_array_len"
Quoted 7 lines
This patchs adds the capability to defined an array of enforced register indexes that teaches QEMU that some registers must exist. If they are not exposed by KVM qemu fakes their presence in the preload phase by adjusting the size of the
cpreg_vmstatearrays. On postload we make sure to ignore them when analyzing potential mismatch between registers. The actual cpreg array is never altered meaning those registers are never accessed nor saved.
It's not really clear to me what we mean by "faking" these registers. It's definitely not the case that we want QEMU to think these registers must exist -- they really don't, and for migration from a new kernel to a new kernel we shouldn't include these registers. We just would like to be able to ignore them in the input migration stream so we can handle the inbound migration from an older kernel.
thanks
-- PMM
Re: [RESEND PATCH 3/7] target/arm/kvm: Introduce the concept of enforced/fake registers
Hi Peter,
Quoted 24 lines · Peter Maydell
Quoted 16 lines · Eric Auger
Newer kernels may revoke exposure of KVM regs to userspace. This can happen when one notices that some registers were unconditionnally exposed whether they shall be conditionnally exposed for example.
An example of such situation is:
TCR2_EL1,PIRE0_EL1,PIR_EL1. Associated kernel commits were:0fcb4eea5345KVM: arm64: HideTCR2_EL1from userspace when disabled for guestsa68cddbe47efKVM: arm64: Hide S1PIE registers from userspace when disabled for guestsThose commits were actual fixes but the cons is that is breaks forward migration on some HW. Indeed when migrating from an old kernel that does not feature those commits to a more recent one, destination qemu detects there are more KVM regs in the input migration stream than exposed by the destination host and the migration fails with:
"failed to load cpu:cpreg_vmstate_array_len"Quoted 7 lines
This patchs adds the capability to defined an array of enforced register indexes that teaches QEMU that some registers must exist. If they are not exposed by KVM qemu fakes their presence in the preload phase by adjusting the size of the
cpreg_vmstatearrays. On postload we make sure to ignore them when analyzing potential mismatch between registers. The actual cpreg array is never altered meaning those registers are never accessed nor saved.
Quoted 7 lines
It's not really clear to me what we mean by "faking" these registers. It's definitely not the case that we want QEMU to think these registers must exist -- they really don't, and for migration from a new kernel to a new kernel we shouldn't include these registers. We just would like to be able to ignore them in the input migration stream so we can handle the inbound migration from an older kernel.
yes maybe I should use a different terminology. Those registers are "faked" from a migration pov only.
among the provided enforced regs, we first identify those which are not actually exposed by KVM. For those only, which I named "fake regs", we tag them as "they will be missing in the input mig stream but that's expected and OK". We make sure we provision extra space for them in cpu->cpreg_vmstate_indexes/values on preload. And in post load when we compare the input stream indices with available inidices, if they are identified as missing, we just ignore the error. That's it.
Hope this helps
Eric
Quoted 4 lines
thanks
-- PMM
PATCH RESEND 4/7 kvm-all: Add the capability to blacklist some KVM regs
In reply to Eric AugerOn ARM we want to be able to blacklist registers that are exposed by KVM. To mitigate some mitigation failures that occur when a new register is exposed and does not exist on the destination, some registers are tagged "hidden" and their state won't be saved. As the state is not saved and they are expected not to be used, we want to enforce they aren't. So let's check this. The new CPUClass hide_reg() callback is optional and will be implemented on ARM in a subsequent patch.
--- include/hw/core/cpu.h | 2 ++ accel/kvm/kvm-all.c | 12 ++++++++++++ 2 files changed, 14 insertions(+)
include/hw/core/cpu.h +2 -0
@@ -142,6 +142,7 @@ struct SysemuCPUOps;* the caller will not g_free() it. * @disas_set_info: Setup architecture specific components of disassembly info * @adjust_watchpoint_address: Perform a target-specific adjustment to an +* @hide_reg: Check if a register must be hidden (optional) * address before attempting to match it against watchpoints. * @deprecation_note: If this CPUClass is deprecated, this field provides * related information.
@@ -167,6 +168,7 @@ struct CPUClass {int (*gdb_read_register)(CPUState *cpu,GByteArray *buf,int reg);int (*gdb_write_register)(CPUState *cpu,uint8_t *buf,int reg);vaddr (*gdb_adjust_breakpoint)(CPUState *cpu,vaddr addr); +bool (*hide_reg)(CPUState *cpu,uint64_t regidex);const char *gdb_core_xml_file;const char * (*gdb_arch_name)(CPUState *cpu);
accel/kvm/kvm-all.c +12 -0
@@ -3783,9 +3783,15 @@ bool kvm_device_supported(int vmfd, uint64_t type)int kvm_set_one_reg (CPUState *cs,uint64_t id,void *source) { +CPUClass *cc =CPU_GET_CLASS (cs);struct kvm_one_reg reg;int r; +if (cc->hide_reg && cc->hide_reg (cs, id)) { +error_report ("%s reg 0x%" PRIx64" is hidden and shall never been accessed" , + __func__, id); +g_assert_not_reached (); + } reg.id = id; reg.addr = (uintptr_t ) source; r =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®);
@@ -3797,9 +3803,15 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *source)int kvm_get_one_reg (CPUState *cs,uint64_t id,void *target) { +CPUClass *cc =CPU_GET_CLASS (cs);struct kvm_one_reg reg;int r; +if (cc->hide_reg && cc->hide_reg (cs, id)) { +error_report ("%s reg 0x%" PRIx64" is hidden and shall never been accessed" , + __func__, id); +g_assert_not_reached (); + } reg.id = id; reg.addr = (uintptr_t ) target; r =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®);
Signature · 1 line
2.49.0
PATCH RESEND 5/7 target/arm/cpu: Implement hide_reg callback()
In reply to Eric AugerChecks if the register is hidden.
--- target/arm/cpu.h | 2 ++ target/arm/cpu.c | 12 ++++++++++++ target/arm/kvm.c | 23 ++--------------------- 3 files changed, 16 insertions(+), 21 deletions(-)
target/arm/cpu.h +2 -0
@@ -2687,4 +2687,6 @@ static inline bool is_fake_reg(ARMCPU *cpu, uint64_t regidx)#define LOG2_TAG_GRANULE 4#define TAG_GRANULE (1 << LOG2_TAG_GRANULE) +bool arm_cpu_hide_reg (CPUState *s,uint64_t regidx); +#endif
target/arm/cpu.c +12 -0
@@ -2366,6 +2366,17 @@ static const TCGCPUOps arm_tcg_ops = { };#endif /* CONFIG_TCG */ +bool arm_cpu_hide_reg (CPUState *s,uint64_t regidx) +{ +ARMCPU *cpu =ARM_CPU (s); +for (int i =0 ; i < cpu->nr_kvm_hidden_regs; i++) { +if (cpu->kvm_hidden_regs[i] == regidx) { +return true; + } + } +return false; +} +static void arm_cpu_class_init (ObjectClass *oc,const void *data) {ARMCPUClass *acc =ARM_CPU_CLASS (oc);
@@ -2394,6 +2405,7 @@ static void arm_cpu_class_init(ObjectClass *oc, const void *data) cc->gdb_get_core_xml_file = arm_gdb_get_core_xml_file; cc->gdb_stop_before_watchpoint = true; cc->disas_set_info = arm_disas_set_info; + cc->hide_reg = arm_cpu_hide_reg;#ifdef CONFIG_TCG cc->tcg_ops = &arm_tcg_ops;
target/arm/kvm.c +2 -21
@@ -754,25 +754,6 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx) } } -/** -* kvm_vcpu_compat_hidden_reg: -* @cpu: ARMCPU -* @regidx: index of the register to check -* -* Depending on the CPU compat returns true if @regidx must be -* ignored during sync & migration -*/ -static inline bool -kvm_vcpu_compat_hidden_reg (ARMCPU *cpu,uint64_t regidx) -{ -for (int i =0 ; i < cpu->nr_kvm_hidden_regs; i++) { -if (cpu->kvm_hidden_regs[i] == regidx) { -return true; - } - } -return false; -} -/** * kvm_arm_init_cpreg_list: * @cpu: ARMCPU
@@ -834,7 +815,7 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)uint64_t regidx = rlp->reg[i];if (!kvm_arm_reg_syncs_via_cpreg_list (regidx) || -kvm_vcpu_compat_hidden_reg (cpu, regidx)) { +arm_cpu_hide_reg (&cpu->parent_obj, regidx)) {continue ; }switch (rlp->reg[i] &KVM_REG_SIZE_MASK ) {
@@ -867,7 +848,7 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)if (!kvm_arm_reg_syncs_via_cpreg_list (regidx)) {continue ; } -if (kvm_vcpu_compat_hidden_reg (cpu, regidx)) { +if (arm_cpu_hide_reg (&cpu->parent_obj, regidx)) {trace_kvm_arm_init_cpreg_list_skip_hidden_reg (rlp->reg[i]);continue ; }
Signature · 1 line
2.49.0
PATCH RESEND 6/7 target/arm/kvm: Expose kvm-hidden-regs and kvm-fake-regs properties
In reply to Eric AugerAllows to set the kvm-hidden-regs and kvm-fake-regs array properties. This will allow to define such compat machine props like:
static GlobalProperty arm_virt_kernel_compat_10_1[] = {
/* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
{ TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
{ TYPE_ARM_CPU, "kvm-enforced-regs",
/* TCR_EL1, PIRE0_EL1, PIR_EL1 */
"0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },
}--- target/arm/kvm.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
target/arm/kvm.c +14 -0
@@ -39,6 +39,8 @@#include "qemu/log.h" #include "hw/acpi/acpi.h" #include "hw/acpi/ghes.h" +#include "hw/qdev-properties.h" +#include "hw/qdev-properties-system.h" #include "target/arm/gtimer.h" #include "migration/blocker.h"
@@ -484,6 +486,15 @@ static void kvm_steal_time_set(Object *obj, bool value, Error **errp)ARM_CPU (obj)->kvm_steal_time = value ?ON_OFF_AUTO_ON :ON_OFF_AUTO_OFF ; } +static const Property arm_cpu_kvm_compat_hidden_regs_property = +DEFINE_PROP_ARRAY ("kvm-hidden-regs" ,ARMCPU , + nr_kvm_hidden_regs, kvm_hidden_regs, qdev_prop_uint64, uint64_t); + +static const Property arm_cpu_kvm_compat_enforced_regs_property = +DEFINE_PROP_ARRAY ("kvm-enforced-regs" ,ARMCPU , + nr_kvm_enforced_regs, kvm_enforced_regs, + qdev_prop_uint64, uint64_t); +/* KVM VCPU properties should be prefixed with "kvm-". */ void kvm_arm_add_vcpu_properties (ARMCPU *cpu) {
@@ -505,6 +516,9 @@ void kvm_arm_add_vcpu_properties(ARMCPU *cpu) kvm_steal_time_set);object_property_set_description (obj,"kvm-steal-time" ,"Set off to disable KVM steal time." ); + +qdev_property_add_static (DEVICE (obj), &arm_cpu_kvm_compat_hidden_regs_property); +qdev_property_add_static (DEVICE (obj), &arm_cpu_kvm_compat_enforced_regs_property); }bool kvm_arm_pmu_supported (void )
Signature · 1 line
2.49.0
PATCH RESEND 7/7 hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels
In reply to Eric AugerThis is an example on how to use the new CPU options. This catters to distributions who want machines to be migratable (forward and backward) accross different host kernel versions in case KVM registers exposed to qemu vary accross kernels. This patch is not meant to be upstreamed as it is really kernel dependent. The goal is to illustrate how this would be used.
In this example, For 10_1 machines types and older we apply the following host kernel related compats:
1) Make sure the KVM_REG_ARM_VENDOR_HYP_BMAP_2 exposed from v6.15 onwards is ignored/hidden.
2) Make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are always seen by qemu although not exposed by KVM. They were unconditionnally exposed before v6.13 while from v6.13 they are only exposed if supported by the guest.
This will allow 10_1 machines types and older machines to migrate forward and backward from old downstream kernels that do not feature those changes to newer kernels (>= v6.15).
--- hw/arm/virt.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
hw/arm/virt.c +19 -0
@@ -97,6 +97,23 @@ static GlobalProperty arm_virt_compat[] = { };static const size_t arm_virt_compat_len =G_N_ELEMENTS (arm_virt_compat); +/* +* if a 10_1 machine type or older is used: +* 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are not +* exposed by the kernel +* 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2 +*/ +static GlobalProperty arm_virt_kernel_compat_10_1[] = { +/* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ + {TYPE_ARM_CPU ,"kvm-hidden-regs" ,"0x6030000000160003" }, +/* TCR_EL1, PIRE0_EL1, PIR_EL1 */ + {TYPE_ARM_CPU ,"kvm-enforced-regs" , +"0x603000000013c103, 0x603000000013c512, 0x603000000013c513" }, +}; +static const size_t arm_virt_kernel_compat_10_1_len = +G_N_ELEMENTS (arm_virt_kernel_compat_10_1); + +/* * This cannot be called from the virt_machine_class_init() because * TYPE_VIRT_MACHINE is abstract and mc->compat_props g_ptr_array_new()
@@ -3539,6 +3556,8 @@ static void virt_machine_10_1_options(MachineClass *mc)virt_machine_10_2_options (mc); mc->smbios_memory_device_size =2047 * TiB;compat_props_add (mc->compat_props, hw_compat_10_1, hw_compat_10_1_len); +compat_props_add (mc->compat_props, + arm_virt_kernel_compat_10_1, arm_virt_kernel_compat_10_1_len); }DEFINE_VIRT_MACHINE (10 ,1 )
Signature · 1 line
2.49.0
Re: [RESEND PATCH 7/7] hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels
Quoted 36 lines · Eric Auger
This is an example on how to use the new CPU options. This catters to distributions who want machines to be migratable (forward and backward) accross different host kernel versions in case KVM registers exposed to qemu vary accross kernels. This patch is not meant to be upstreamed as it is really kernel dependent. The goal is to illustrate how this would be used.
In this example, For 10_1 machines types and older we apply the following host kernel related compats:
1) Make sure the
KVM_REG_ARM_VENDOR_HYP_BMAP_2exposed from v6.15 onwards is ignored/hidden.
2) Make sureTCR_EL1,PIRE0_EL1,PIR_EL1are always seen by qemu although not exposed by KVM. They were unconditionnally exposed before v6.13 while from v6.13 they are only exposed if supported by the guest.This will allow 10_1 machines types and older machines to migrate forward and backward from old downstream kernels that do not feature those changes to newer kernels (>= v6.15).
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)hw/arm/virt.c +2
-0@@ -97,6 +97,23 @@ static GlobalProperty arm_virt_compat[] = { };static const size_t arm_virt_compat_len =G_N_ELEMENTS (arm_virt_compat); +/* +* if a 10_1 machine type or older is used:
hw/arm/virt.c static GlobalProperty arm_virt_compat[] = { · line 111
Quoted 10 lines
hw/arm/virt.c +10
-0+* 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are not +* exposed by the kernel +* 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2 +*/ +static GlobalProperty arm_virt_kernel_compat_10_1[] = { +/* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ + {TYPE_ARM_CPU ,"kvm-hidden-regs" ,"0x6030000000160003" }, +/* TCR_EL1, PIRE0_EL1, PIR_EL1 */ + {TYPE_ARM_CPU ,"kvm-enforced-regs" , +"0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },
Strings which are lists of long hex numbers? Is there a more readable way to do this?
-- PMM
Re: [RESEND PATCH 7/7] hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels
Hi Peter,
Quoted 34 lines · Peter Maydell
Quoted 33 lines · Eric Auger
This is an example on how to use the new CPU options. This catters to distributions who want machines to be migratable (forward and backward) accross different host kernel versions in case KVM registers exposed to qemu vary accross kernels. This patch is not meant to be upstreamed as it is really kernel dependent. The goal is to illustrate how this would be used.
In this example, For 10_1 machines types and older we apply the following host kernel related compats:
1) Make sure the
KVM_REG_ARM_VENDOR_HYP_BMAP_2exposed from v6.15 onwards is ignored/hidden.
2) Make sureTCR_EL1,PIRE0_EL1,PIR_EL1are always seen by qemu although not exposed by KVM. They were unconditionnally exposed before v6.13 while from v6.13 they are only exposed if supported by the guest.This will allow 10_1 machines types and older machines to migrate forward and backward from old downstream kernels that do not feature those changes to newer kernels (>= v6.15).
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)hw/arm/virt.c +0
-0@@ -97,6 +97,23 @@ static GlobalProperty arm_virt_compat[] = { };static const size_t arm_virt_compat_len =G_N_ELEMENTS (arm_virt_compat);
Quoted 14 lines
hw/arm/virt.c static GlobalProperty arm_virt_compat[] = { · line 111
Quoted 12 lines
hw/arm/virt.c +12
-0+/* +* if a 10_1 machine type or older is used: +* 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are not +* exposed by the kernel +* 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2 +*/ +static GlobalProperty arm_virt_kernel_compat_10_1[] = { +/* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ + {TYPE_ARM_CPU ,"kvm-hidden-regs" ,"0x6030000000160003" }, +/* TCR_EL1, PIRE0_EL1, PIR_EL1 */ + {TYPE_ARM_CPU ,"kvm-enforced-regs" , +"0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },Strings which are lists of long hex numbers? Is there a more readable way to do this?
for sysregs, we could use the reg name string directly once we have script generated list of regs from json description. However for some regs it won't be possible, like pseudo FW regs.
As those props are really supposed to be used by very informed users (distro staff) who need to know the kvm indices, I thought it could be acceptable. Maybe we can rely on defines to make it more readable.
Thanks
Eric
Quoted 3 lines
-- PMM
Re: [RESEND PATCH 7/7] hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels
In reply to Peter MaydellQuoted 34 lines · Peter Maydell
Quoted 33 lines · Eric Auger
This is an example on how to use the new CPU options. This catters to distributions who want machines to be migratable (forward and backward) accross different host kernel versions in case KVM registers exposed to qemu vary accross kernels. This patch is not meant to be upstreamed as it is really kernel dependent. The goal is to illustrate how this would be used.
In this example, For 10_1 machines types and older we apply the following host kernel related compats:
1) Make sure the
KVM_REG_ARM_VENDOR_HYP_BMAP_2exposed from v6.15 onwards is ignored/hidden.
2) Make sureTCR_EL1,PIRE0_EL1,PIR_EL1are always seen by qemu although not exposed by KVM. They were unconditionnally exposed before v6.13 while from v6.13 they are only exposed if supported by the guest.This will allow 10_1 machines types and older machines to migrate forward and backward from old downstream kernels that do not feature those changes to newer kernels (>= v6.15).
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)hw/arm/virt.c +0
-0@@ -97,6 +97,23 @@ static GlobalProperty arm_virt_compat[] = { };static const size_t arm_virt_compat_len =G_N_ELEMENTS (arm_virt_compat);
Quoted 14 lines
hw/arm/virt.c static GlobalProperty arm_virt_compat[] = { · line 111
Quoted 12 lines
hw/arm/virt.c +12
-0+/* +* if a 10_1 machine type or older is used: +* 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are not +* exposed by the kernel +* 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2 +*/ +static GlobalProperty arm_virt_kernel_compat_10_1[] = { +/* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ + {TYPE_ARM_CPU ,"kvm-hidden-regs" ,"0x6030000000160003" }, +/* TCR_EL1, PIRE0_EL1, PIR_EL1 */ + {TYPE_ARM_CPU ,"kvm-enforced-regs" , +"0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },Strings which are lists of long hex numbers? Is there a more readable way to do this?
forgot to mention that we could use x- prefix for those properties to enphasize they are rather experimental and not meant to be used commonly.
Eric
Quoted 3 lines
-- PMM
Quoted 33 lines · Eric Auger
When migrating ARM guests accross same machines with different host kernels we are likely to encounter failures such as:
"failed to load cpu:cpreg_vmstate_array_len"
This is due to the fact KVM exposes a different number of registers to qemu on source and destination. When trying to migrate a bigger register set to a smaller one, qemu cannot save the CPU state.
For example, recently we faced such kind of situations with:
- unconditionnal exposure ofKVM_REG_ARM_VENDOR_HYP_BMAP_2FW pseudo register from v6.16 onwards. Causes backward migration failure.
- removal of unconditionnal exposure ofTCR2_EL1,PIRE0_EL1,PIR_EL1from v6.13 onwards. Causes forward migration failure.This situation is really problematic for distributions which want to guarantee forward and backward migration of a given machine type between different releases.
This small series tries to address that issue by introducing CPU array properties that list the registers to ignore or to fake according to the situation. An example is given to illustrate how those props could be used to apply compats for machine types supposed to "see" the same register set accross various host kernels.
The first patch improves the tracing so that we can quickly detect which registers are unexpected and cause the migration failure. Missing registers are also traced. Those do not fail migration but their default value is kept on the destination.
Then we introduce the infrastructure to handle 'hidden' registers and 'fake' registers.
Quoted 10 lines
Eric Auger (7):
target/arm/machine: Improve traces on register mismatch during
migration
target/arm/kvm: Introduce the concept of hidden KVM regs target/arm/kvm: Introduce the concept of enforced/fake registers
kvm-all: Add the capability to blacklist some KVM regs target/arm/cpu: Implementhide_regcallback()target/arm/kvm: Expose kvm-hidden-regs and kvm-fake-regs properties hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older
kernels
Gentle ping.
Any comments on the approach?
Thanks
Eric
Quoted 11 lines
include/hw/core/cpu.h | 2 ++ target/arm/cpu.h | 42 ++++++++++++++++++++++++ accel/kvm/kvm-all.c | 12 +++++++ hw/arm/virt.c | 19 +++++++++++ target/arm/cpu.c | 12 +++++++ target/arm/kvm.c | 73 ++++++++++++++++++++++++++++++++++++++++- target/arm/machine.c | 71 +++++++++++++++++++++++++++++++++++---- target/arm/trace-events | 11 +++++++ 8 files changed, 235 insertions(+), 7 deletions(-)
Quoted 19 lines · Eric Auger
Quoted 9 lines · Eric Auger
When migrating ARM guests accross same machines with different host kernels we are likely to encounter failures such as:
"failed to load cpu:cpreg_vmstate_array_len"
This is due to the fact KVM exposes a different number of registers to qemu on source and destination. When trying to migrate a bigger register set to a smaller one, qemu cannot save the CPU state.
Quoted 5 lines
For example, recently we faced such kind of situations with:
- unconditionnal exposure ofKVM_REG_ARM_VENDOR_HYP_BMAP_2FW pseudo register from v6.16 onwards. Causes backward migration failure.
- removal of unconditionnal exposure ofTCR2_EL1,PIRE0_EL1,PIR_EL1from v6.13 onwards. Causes forward migration failure.
Quoted 3 lines
Gentle ping.
Any comments on the approach?
A couple of general remarks:
(1) This isn't KVM specific -- see e.g. commit 4f2b82f60 where we had to add back a fake cpreg to un-break forward migration of TCG CPUs. So our handling of this kind of problem shouldn't be restricted to only working with KVM.
(2) essentially we're re-inventing the migration compat support that VMStateDescriptions provide. That's kind of unavoidable because of the way I implemented cpreg migration years ago, but is there anything we can learn in terms of
(a) required feature set and (b) trying to keep parallels between the two for the way things work ?
thanks
-- PMM
Hi Peter,
Quoted 20 lines · Peter Maydell
Quoted 19 lines · Eric Auger
Quoted 9 lines · Eric Auger
When migrating ARM guests accross same machines with different host kernels we are likely to encounter failures such as:
"failed to load cpu:cpreg_vmstate_array_len"
This is due to the fact KVM exposes a different number of registers to qemu on source and destination. When trying to migrate a bigger register set to a smaller one, qemu cannot save the CPU state.
Quoted 5 lines
For example, recently we faced such kind of situations with:
- unconditionnal exposure ofKVM_REG_ARM_VENDOR_HYP_BMAP_2FW pseudo register from v6.16 onwards. Causes backward migration failure.
- removal of unconditionnal exposure ofTCR2_EL1,PIRE0_EL1,PIR_EL1from v6.13 onwards. Causes forward migration failure.Gentle ping.
Quoted 7 lines
Quoted 1 line
Any comments on the approach?
A couple of general remarks:
(1) This isn't KVM specific -- see e.g. commit
4f2b82f60where we had to add back a fake cpreg to un-break forward migration of TCG CPUs. So our handling of this kind of problem shouldn't be restricted to only working with KVM.
interesting. I will see how this can be extended to TCG
Quoted 7 lines
(2) essentially we're re-inventing the migration compat support that VMStateDescriptions provide. That's kind of unavoidable because of the way I implemented cpreg migration years ago, but is there anything we can learn in terms of
(a) required feature set and (b) trying to keep parallels between the two for the way things work ?
OK I will study that further.
Thank you for your suggestions!
Eric
Quoted 4 lines
thanks
-- PMM