ARM Nested Virt Support
Subsystems (3)
v7 cover ARM Nested Virt Support
This is based on Peter's target-arm.next. This is candidate for 10.1.
For gaining virt functionality in KVM accelerated L1, The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
with: -machine virt,virtualization=on.
This series can be found at:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v7
Original version from Miguel:
[1] https://lore.kernel.org/all/[email protected]/
version from Haibo:
[2] https://lore.kernel.org/qemu-devel/[email protected]/
History:
v6 -> v7:
- rebase on Peter's target-arm.next. as a result linux header update could be dropped. Faced a small conflict in hw/arm/virt: Allow virt extensions with KVM due to recently landed hw/arm/virt: Make EL2 accelerator check an accept-list. I dared to keep Richard's R-b though.
v5 -> v6:
- linux header update against v6.16-rc2
v4 -> v5:
- rebase on top of v10.0.0
v3 -> v4:
- fix: only set maint_irq if vms->virt
v2 -> v3:
- KVM EL2 only is set if virtualization option is set
- fixes regression with virtualization=off
- Add checks against unsupported GIC configs until the kernel does
Haibo Xu (4):
hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM
include/hw/intc/arm_gicv3_common.h | 1 + target/arm/kvm_arm.h | 7 +++++++ hw/arm/virt.c | 13 ++++++++++++- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ target/arm/kvm-stub.c | 5 +++++ target/arm/kvm.c | 21 +++++++++++++++++++++ 7 files changed, 68 insertions(+), 1 deletion(-)
Signature · 1 line
2.49.0
PATCH v7 1/4 hw/arm: Allow setting KVM vGIC maintenance IRQ
From: Haibo Xu <[email protected]>
Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.
This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().
---
v3 -> v4:
- only set maint_irq if vms->virt
v2 -> v3:
- tweak the commit message and explain why we do not proceed the same way as kvm_arm_pmu_set_irq (Peter)
v1 -> v2:
- [Miguel] replaced the has_virt_extensions by the maintenance irq intid property. [Eric] restored kvm_device_check_attr and kvm_device_access standard usage and conditionally call those if the prop is set.
--- include/hw/intc/arm_gicv3_common.h | 1 + hw/arm/virt.c | 3 +++ hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ 4 files changed, 26 insertions(+)
include/hw/intc/arm_gicv3_common.h +1 -0
@@ -231,6 +231,7 @@ struct GICv3State {uint32_t num_cpu;uint32_t num_irq;uint32_t revision; +uint32_t maint_irq;bool lpi_enable;bool nmi_support;bool security_extn;
hw/arm/virt.c +3 -0
@@ -828,6 +828,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) OBJECT(mem), &error_fatal);qdev_prop_set_bit (vms->gic,"has-lpi" , true); } + }else if (vms->virt) { +qdev_prop_set_uint32 (vms->gic,"maintenance-interrupt-id" , +ARCH_GIC_MAINT_IRQ ); } }else {if (!kvm_irqchip_in_kernel ()) {
hw/intc/arm_gicv3_common.c +1 -0
@@ -612,6 +612,7 @@ static const Property arm_gicv3_common_properties[] = {DEFINE_PROP_BOOL ("has-lpi" , GICv3State, lpi_enable,0 ),DEFINE_PROP_BOOL ("has-nmi" , GICv3State, nmi_support,0 ),DEFINE_PROP_BOOL ("has-security-extensions" , GICv3State, security_extn,0 ), +DEFINE_PROP_UINT32 ("maintenance-interrupt-id" , GICv3State, maint_irq,0 ),/* * Compatibility property: force 8 bits of physical priority, even * if the CPU being emulated should have fewer.
hw/intc/arm_gicv3_kvm.c +21 -0
@@ -22,6 +22,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/arm/virt.h" #include "qemu/error-report.h" #include "qemu/module.h" #include "system/kvm.h"
@@ -825,6 +826,26 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp)return ; } +if (s->maint_irq) { +int ret; + + ret =kvm_device_check_attr (s->dev_fd, +KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ ,0 ); +if (!ret) { +error_setg_errno (errp, errno, +"VGICv3 setting maintenance IRQ is not " +"supported by this host kernel" ); +return ; + } + + ret =kvm_device_access (s->dev_fd,KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ ,0 , + &s->maint_irq, true, errp); +if (ret) { +error_setg_errno (errp, errno,"Failed to set VGIC maintenance IRQ" ); +return ; + } + } + multiple_redist_region_allowed =kvm_device_check_attr (s->dev_fd,KVM_DEV_ARM_VGIC_GRP_ADDR ,KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION );
Signature · 1 line
2.49.0
Re: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ
Quoted 27 lines · Eric Auger
From: Haibo Xu <[email protected]>
Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.
This setting must be done before the
KVM_DEV_ARM_VGIC_CTRL_INIThence the choice to perform the setting in the GICv3 realize instead of proceeding the same way askvm_arm_pmu_set_irq().Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>(quoted diff) +5
-0@@ -231,6 +231,7 @@ struct GICv3State { uint32_t num_cpu; uint32_t num_irq; uint32_t revision; + uint32_t maint_irq; bool lpi_enable; bool nmi_support; bool security_extn; + if (s->maint_irq) { + int ret; + + ret = kvm_device_check_attr(s->dev_fd,
Review struct GICv3State {
Quoted 10 lines
(quoted diff) +10
-0+ KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0); + if (!ret) { + error_setg_errno(errp, errno, + "VGICv3 setting maintenance IRQ is not " + "supported by this host kernel"); + return; + } + + ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0, + &s->maint_irq, true, errp);
This doesn't seem to line up with what the kernel documents for KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ:
https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says
# KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field of kvm_device_attr encodes the following values:
#
# bits: | 31 .... 5 | 4 .... 0 | # values: | RES0 | vINTID |
#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.
but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a uint32_t with the vINTID in it.
Looking at the kernel code in vgic_v3_attr_regs_access() it looks like maybe the kernel docs are wrong, but I'm not sure.
thanks
-- PMM
Marc Zyngier
maintainerRe: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ
Quoted 57 lines · Peter Maydell
Quoted 27 lines · Eric Auger
From: Haibo Xu <[email protected]>
Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.
This setting must be done before the
KVM_DEV_ARM_VGIC_CTRL_INIThence the choice to perform the setting in the GICv3 realize instead of proceeding the same way askvm_arm_pmu_set_irq().Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>(quoted diff) +5
-0@@ -231,6 +231,7 @@ struct GICv3State { uint32_t num_cpu; uint32_t num_irq; uint32_t revision; + uint32_t maint_irq; bool lpi_enable; bool nmi_support; bool security_extn; + if (s->maint_irq) { + int ret; + + ret = kvm_device_check_attr(s->dev_fd,Review struct GICv3State {
Quoted 10 lines
(quoted diff) +10
-0+ KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0); + if (!ret) { + error_setg_errno(errp, errno, + "VGICv3 setting maintenance IRQ is not " + "supported by this host kernel"); + return; + } + + ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0, + &s->maint_irq, true, errp);This doesn't seem to line up with what the kernel documents for
KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ:https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says#
KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field ofkvm_device_attrencodes the following values:
## bits: | 31 .... 5 | 4 .... 0 | # values: | RES0 | vINTID |#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.
Quoted 5 lines
but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a
uint32_twith the vINTID in it.Looking at the kernel code in
vgic_v3_attr_regs_access()it looks like maybe the kernel docs are wrong, but I'm not sure.
I think this is a pretty bad case of one person implementing something, and another (me) hoping (but not checking) it would be implemented in a particular way. Oh well.
Would the following match your expectations (and the code)?
Documentation/virt/kvm/devices/arm-vgic-v3.rst +1 -1
@@ -299,7 +299,7 @@ Groups: KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ Attributes: - The attr field of kvm_device_attr encodes the following values: + The attribute data pointed to by kvm_device_attr.addr is a __u32 value:: bits: | 31 .... 5 | 4 .... 0 | values: | RES0 | vINTID |
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ
Quoted 34 lines · Marc Zyngier
Quoted 14 lines · Peter Maydell
https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says#
KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field ofkvm_device_attrencodes the following values:
## bits: | 31 .... 5 | 4 .... 0 | # values: | RES0 | vINTID |#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.Quoted 5 lines
but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a
uint32_twith the vINTID in it.Looking at the kernel code in
vgic_v3_attr_regs_access()it looks like maybe the kernel docs are wrong, but I'm not sure.I think this is a pretty bad case of one person implementing something, and another (me) hoping (but not checking) it would be implemented in a particular way. Oh well.
Would the following match your expectations (and the code)?
Documentation/virt/kvm/devices/arm-vgic-v3.rst +0
-0
Documentation/virt/kvm/devices/arm-vgic-v3.rst Groups: · line 302
Quoted 5 lines
Documentation/virt/kvm/devices/arm-vgic-v3.rst +1
-1@@ -299,7 +299,7 @@ Groups: KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ Attributes: - The attr field of kvm_device_attr encodes the following values: + The attribute data pointed to by kvm_device_attr.addr is a __u32 value::
I think it's maybe better to explicitly say "Attributes: ignored" or "Attributes: none" or similar. Otherwise it looks like the text is describing attr, because it follows the "Attributes:" subheader. (Though we don't do this for KVM_DEV_ARM_VGIC_GRP_NR_IRQS either, which also ignores attrs and reads/writes a value pointed to by addr.)
Documentation/virt/kvm/devices/arm-vgic-v3.rst Groups: · line 305
Quoted 3 lines
Documentation/virt/kvm/devices/arm-vgic-v3.rst +0
-0bits: | 31 .... 5 | 4 .... 0 | values: | RES0 | vINTID |
Is the kvmattr.attr field mandatorily 0, or does the kernel ignore it? (I think the answer is "ignored".)
Side note, we seem to allow this attribute to be read via get_attr for gicv2, even though has_attr denies it exists there, because the get code is in vgic_get_common_attr(). set_attr is in the gicv3-only vgic_v3_attr_regs_access().
-- PMM
PATCH v7 2/4 target/arm/kvm: Add helper to detect EL2 when using KVM
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
Introduce query support for KVM_CAP_ARM_EL2.
--- target/arm/kvm_arm.h | 7 +++++++ target/arm/kvm-stub.c | 5 +++++ target/arm/kvm.c | 5 +++++ 3 files changed, 17 insertions(+)
target/arm/kvm_arm.h +7 -0
@@ -191,6 +191,13 @@ bool kvm_arm_sve_supported(void);*/ bool kvm_arm_mte_supported (void ); +/** +* kvm_arm_el2_supported: +* +* Returns true if KVM can enable EL2 and false otherwise. +*/ +bool kvm_arm_el2_supported (void ); +/** * kvm_arm_get_max_vm_ipa_size: * @ms: Machine state handle
target/arm/kvm-stub.c +5 -0
@@ -47,6 +47,11 @@ bool kvm_arm_mte_supported(void)return false; } +bool kvm_arm_el2_supported (void ) +{ +return false; +} +/* * These functions should never actually be called without KVM support. */
target/arm/kvm.c +5 -0
@@ -1762,6 +1762,11 @@ bool kvm_arm_aarch32_supported(void)return kvm_check_extension (kvm_state,KVM_CAP_ARM_EL1_32BIT ); } +bool kvm_arm_el2_supported (void ) +{ +return kvm_check_extension (kvm_state,KVM_CAP_ARM_EL2 ); +} +bool kvm_arm_sve_supported (void ) {return kvm_check_extension (kvm_state,KVM_CAP_ARM_SVE );
Signature · 1 line
2.49.0
PATCH v7 3/4 target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.
---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)
v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu
--- target/arm/kvm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
target/arm/kvm.c +16 -0
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)*/ int fdarray[3 ];bool sve_supported; +bool el2_supported;bool pmu_supported = false;uint64_t features =0 ;int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) init.features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; } +/* +* Ask for EL2 if supported. +*/ + el2_supported =kvm_arm_el2_supported (); +if (el2_supported) { + init.features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + } +/* * Ask for Pointer Authentication if supported, so that we get * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) features |=1ULL <<ARM_FEATURE_AARCH64 ; features |=1ULL <<ARM_FEATURE_GENERIC_TIMER ; +if (el2_supported) { + features |=1ULL <<ARM_FEATURE_EL2 ; + } + ahcf->features = features;return true;
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |= (1 <<KVM_ARM_VCPU_PTRAUTH_ADDRESS |1 <<KVM_ARM_VCPU_PTRAUTH_GENERIC ); } +if (cpu->has_el2 &&kvm_arm_el2_supported ()) { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cpu);
Signature · 1 line
2.49.0
Re: [PATCH v7 3/4] target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
Quoted 62 lines · Eric Auger
From: Haibo Xu <[email protected]>
KVM_CAP_ARM_EL2must be supported by the cpu to enableARM_FEATURE_EL2. In case the host does support NV, expose the feature.Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Richard Henderson <[email protected]>---
v2 -> v3:
- checkpmu->has_el2onkvm_arch_init_vcpu()when settingKVM_ARM_VCPU_HAS_EL2feature (Peter)v1 -> v2:
- removeisar_feature_aa64_aa32_el2modif intarget/arm/cpu.h[Richard] and useel2_supportedinkvm_arch_init_vcpu--- target/arm/kvm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)target/arm/kvm.c +13
-0@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)*/ int fdarray[3 ];bool sve_supported; +bool el2_supported;bool pmu_supported = false;uint64_t features =0 ;int err;@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) init.features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; } +/* +* Ask for EL2 if supported. +*/ + el2_supported =kvm_arm_el2_supported (); +if (el2_supported) { + init.features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + } +/* * Ask for Pointer Authentication if supported, so that we get * the unsanitized field values for AA64ISAR1_EL1. @@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) features |=1ULL <<ARM_FEATURE_AARCH64 ; features |=1ULL <<ARM_FEATURE_GENERIC_TIMER ; +if (el2_supported) { + features |=1ULL <<ARM_FEATURE_EL2 ; + } + ahcf->features = features;return true;
target/arm/kvm.c int kvm_arch_init_vcpu(CPUState *cs)
Quoted 10 lines
target/arm/kvm.c +3
-1@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |= (1 <<KVM_ARM_VCPU_PTRAUTH_ADDRESS |1 <<KVM_ARM_VCPU_PTRAUTH_GENERIC ); } +if (cpu->has_el2 &&kvm_arm_el2_supported ()) { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cpu); --
How does this interact with creating an aarch32 VM with "-cpu host,aarch64=off" ? Looking at the kernel, it will fail vcpu_init if you ask for a vcpu with both EL1_32BIT and HAS_EL2. I guess since the virt board will only request HAS_EL2 if you say virtualization=on this is fine. Do we give a sensible error message if the user tries
-accel kvm -M virt,virtualization=on -cpu host,aarch64=off ?
-- PMM
Re: [PATCH v7 3/4] target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
Hi Peter,
Quoted 69 lines · Peter Maydell
Quoted 68 lines · Eric Auger
From: Haibo Xu <[email protected]>
KVM_CAP_ARM_EL2must be supported by the cpu to enableARM_FEATURE_EL2. In case the host does support NV, expose the feature.Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Richard Henderson <[email protected]>---
v2 -> v3:
- checkpmu->has_el2onkvm_arch_init_vcpu()when settingKVM_ARM_VCPU_HAS_EL2feature (Peter)v1 -> v2:
- removeisar_feature_aa64_aa32_el2modif intarget/arm/cpu.h[Richard] and useel2_supportedinkvm_arch_init_vcpu--- target/arm/kvm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)target/arm/kvm.c +16
-0@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)*/ int fdarray[3 ];bool sve_supported; +bool el2_supported;bool pmu_supported = false;uint64_t features =0 ;int err;@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) init.features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; } +/* +* Ask for EL2 if supported. +*/ + el2_supported =kvm_arm_el2_supported (); +if (el2_supported) { + init.features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + } +/* * Ask for Pointer Authentication if supported, so that we get * the unsanitized field values for AA64ISAR1_EL1. @@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) features |=1ULL <<ARM_FEATURE_AARCH64 ; features |=1ULL <<ARM_FEATURE_GENERIC_TIMER ; +if (el2_supported) { + features |=1ULL <<ARM_FEATURE_EL2 ; + } + ahcf->features = features;return true;@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |= (1 <<KVM_ARM_VCPU_PTRAUTH_ADDRESS |1 <<KVM_ARM_VCPU_PTRAUTH_GENERIC ); } +if (cpu->has_el2 &&kvm_arm_el2_supported ()) { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + }
Quoted 10 lines
target/arm/kvm.c int kvm_arch_init_vcpu(CPUState *cs)
Quoted 3 lines
target/arm/kvm.c +0
-1/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cpu); --How does this interact with creating an aarch32 VM with "-cpu host,aarch64=off" ? Looking at the kernel, it will fail
vcpu_initif you ask for a vcpu with bothEL1_32BITandHAS_EL2. I guess since the virt board will only requestHAS_EL2if you say virtualization=on this is fine. Do we give a sensible error message if the user tries
-accel kvm -M virt,virtualization=on -cpu host,aarch64=off ?
Not yet. I will add one.
Thanks!
Eric
Quoted 3 lines
-- PMM
PATCH v7 4/4 hw/arm/virt: Allow virt extensions with KVM
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Also check only in-kernel GICv3 is used along with KVM EL2.
---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.
v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)
v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message
--- hw/arm/virt.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
hw/arm/virt.c +9 -1
@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)default :g_assert_not_reached (); } + +if (kvm_enabled () && vms->virt && + (revision !=3 || !kvm_irqchip_in_kernel ())) { +error_report ("KVM EL2 only is supported with in-kernel GICv3" ); +exit (1 ); + } + vms->gic =qdev_new (gictype);qdev_prop_set_uint32 (vms->gic,"revision" , revision);qdev_prop_set_uint32 (vms->gic,"num-cpu" , smp_cpus);
@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)exit (1 ); } -if (vms->virt && !tcg_enabled () && !qtest_enabled ()) { +if (vms->virt && !(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) {error_report ("mach-virt: %s does not support providing " "Virtualization extensions to the guest CPU" ,current_accel_name ());
Signature · 1 line
2.49.0
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 33 lines · Eric Auger
From: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Also check only in-kernel GICv3 is used along with KVM EL2.
Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Richard Henderson <[email protected]>---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message--- hw/arm/virt.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)hw/arm/virt.c +0
-0
hw/arm/virt.c static void create_gic(VirtMachineState *vms, MemoryRegion *mem) · line 798
Quoted 7 lines
hw/arm/virt.c +4
-0@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)default :g_assert_not_reached (); } + +if (kvm_enabled () && vms->virt && + (revision !=3 || !kvm_irqchip_in_kernel ())) { +error_report ("KVM EL2 only is supported with in-kernel GICv3" );
"is only supported"
Quoted 7 lines
hw/arm/virt.c +3
-0+exit (1 ); + } + vms->gic =qdev_new (gictype);qdev_prop_set_uint32 (vms->gic,"revision" , revision);qdev_prop_set_uint32 (vms->gic,"num-cpu" , smp_cpus);
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2225
Quoted 9 lines
hw/arm/virt.c +2
-1@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)exit (1 ); } -if (vms->virt && !tcg_enabled () && !qtest_enabled ()) { +if (vms->virt && !(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) {error_report ("mach-virt: %s does not support providing " "Virtualization extensions to the guest CPU" ,current_accel_name ());
Have you tested doing a VM migration of a KVM with EL2 setup? I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?
thanks
-- PMM
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Hi Peter, Marc,
Quoted 28 lines · Peter Maydell
Quoted 27 lines · Eric Auger
From: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Also check only in-kernel GICv3 is used along with KVM EL2.
Signed-off-by Haibo Xu <[email protected]>Signed-off-by Miguel Luis <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Richard Henderson <[email protected]>---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message--- hw/arm/virt.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
Quoted 13 lines
hw/arm/virt.c static void create_gic(VirtMachineState *vms, MemoryRegion *mem) · line 798
Quoted 12 lines
hw/arm/virt.c +4
-0@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)default :g_assert_not_reached (); } + +if (kvm_enabled () && vms->virt && + (revision !=3 || !kvm_irqchip_in_kernel ())) { +error_report ("KVM EL2 only is supported with in-kernel GICv3" );"is only supported"
OK
Quoted 11 lines
Quoted 11 lines
hw/arm/virt.c +3
-0+exit (1 ); + } + vms->gic =qdev_new (gictype);qdev_prop_set_uint32 (vms->gic,"revision" , revision);qdev_prop_set_uint32 (vms->gic,"num-cpu" , smp_cpus);@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)exit (1 ); }
Quoted 7 lines
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2225
Quoted 6 lines
hw/arm/virt.c +2
-1-if (vms->virt && !tcg_enabled () && !qtest_enabled ()) { +if (vms->virt && !(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) {error_report ("mach-virt: %s does not support providing " "Virtualization extensions to the guest CPU" ,current_accel_name ());Have you tested doing a VM migration of a KVM with EL2 setup?
Yes I did. Marc fixed some save/restore bugs in the past and I was able to migrate after this fix.
Now I would lie if I'd tell you it works well. It is extremely slow to converge. Usually you get rcu stalls in the L2 guest on source too.
Quoted 4 lines
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?
Do you see some specific registers/resources that would need attention?
Otherwise I can put a migration blocker until we spend more time analyzing migration slowness
What do you think?
Eric
Quoted 4 lines
thanks
-- PMM
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 9 lines · Eric Auger
Hi Peter, Marc,
Quoted 4 lines · Peter Maydell
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?Do you see some specific registers/resources that would need attention?
All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.
These all need to be exposed via KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG and hw/intc/arm_gicv3_kvm.c needs code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).
I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.
I'm ok with just putting in a migration blocker for the moment, especially if this needs kernel-side changes.
thanks
-- PMM
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Hi Peter,
Quoted 20 lines · Peter Maydell
Quoted 8 lines · Eric Auger
Hi Peter, Marc,
Quoted 4 lines · Peter Maydell
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?Do you see some specific registers/resources that would need attention?
All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2,
ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.These all need to be exposed via
KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGandhw/intc/arm_gicv3_kvm.cneeds code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).
Quoted 3 lines
I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of
KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.
OK thank you for the insights.
Quoted 3 lines
I'm ok with just putting in a migration blocker for the moment, especially if this needs kernel-side changes.
OK I will do that until we get this fixed
Thanks!
Eric
Quoted 4 lines
thanks
-- PMM
Marc Zyngier
maintainerRe: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
In reply to Peter MaydellQuoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger
Hi Peter, Marc,
Quoted 4 lines · Peter Maydell
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?
Quoted 5 lines
Quoted 1 line
Do you see some specific registers/resources that would need attention?
All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2,
ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.
It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.
Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.
Quoted 10 lines
These all need to be exposed via
KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGandhw/intc/arm_gicv3_kvm.cneeds code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of
KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.
Hmmm... This is indeed pretty inconsistent. I'll have a look at it.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 19 lines · Marc Zyngier
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger
Hi Peter, Marc,
Quoted 4 lines · Peter Maydell
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?Quoted 5 lines
Quoted 1 line
Do you see some specific registers/resources that would need attention?
All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2,
ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.
Quoted 6 lines
It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.
Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.
Good point: we don't need to expose those, only the actual state-holding registers. I was rather mindlessly going through and listing all the ICH_ registers out of the spec...
So should the list be:
ICH_AP*R<n>_EL2, ICH_HCR_EL2, ICH_HR<n>_EL2, ICH_VMCR_EL2
?
(Those are the ones the TCG GICv3 has state for in its data structures.)
How about ICH_VTR_EL2 ? I guess we should handle it the same way we do other existing ID registers (migrate it and write value back to kernel so it can validate that the destination hardware supports the same config) ?
Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1 GICD_TYPER2 register, so we don't try to send and sanity check that on migration at the moment.
thanks
-- PMM
Marc Zyngier
maintainerRe: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 32 lines · Peter Maydell
Quoted 19 lines · Marc Zyngier
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger
Hi Peter, Marc,
Quoted 4 lines · Peter Maydell
I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in
hw/intc/arm_gicv3_kvm.c?Quoted 5 lines
Quoted 1 line
Do you see some specific registers/resources that would need attention?
All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2,
ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.Quoted 6 lines
It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.
Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.
Good point: we don't need to expose those, only the actual state-holding registers. I was rather mindlessly going through and listing all the
ICH_registers out of the spec...
Quoted 5 lines
So should the list be:
ICH_AP*R<n>_EL2,ICH_HCR_EL2, ICH_HR<n>_EL2,ICH_VMCR_EL2
?
(Those are the ones the TCG GICv3 has state for in its data structures.)
Yup, this matches my own expectations.
Quoted 4 lines
How about
ICH_VTR_EL2? I guess we should handle it the same way we do other existing ID registers (migrate it and write value back to kernel so it can validate that the destination hardware supports the same config) ?
Indeed. Specially as this contains the number of LRs, which could otherwise lead to UNDEF exceptions if migrating in the wrong direction.
What of ICC_SRE_EL2? We have it as RAO/WI for all the non-RES0 fields. It could be saved/restored as well for the same purposes...
Quoted 3 lines
Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1
GICD_TYPER2register, so we don't try to send and sanity check that on migration at the moment.
KVM only exposes a virtual GICv3 to the guest, even in the host can do GICv4.0+. The only nit is the "nASSGIcap" bit, which is always advertised to the guest when the host if v4.1-capable. We have a patch series in progress to address this and make it controllable.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 11 lines · Marc Zyngier
Quoted 3 lines · Peter Maydell
Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1
GICD_TYPER2register, so we don't try to send and sanity check that on migration at the moment.KVM only exposes a virtual GICv3 to the guest, even in the host can do GICv4.0+. The only nit is the "nASSGIcap" bit, which is always advertised to the guest when the host if v4.1-capable. We have a patch series in progress to address this and make it controllable.
Isn't that out-of-spec? If the GIC the guest sees is a GICv3 then GICD_TYPER2 should be RES0...
Anyway, I have some lightly tested QEMU patches that migrate the GICD_TYPER2 value, which I'll send out in a moment.
-- PMM