ARM Nested Virt Support
v8 cover ARM Nested Virt Support
This is candidate for 10.1.
For gaining virt functionality in KVM accelerated L1, The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
with: -machine virt,virtualization=on.
This series can be found at:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v8
previous:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v7
Original version from Miguel:
[1] https://lore.kernel.org/all/[email protected]/
version from Haibo:
[2] https://lore.kernel.org/qemu-devel/[email protected]/
History:
v7 -> v8
- add migration blocker when virt is set along with kvm enabled
- test virt is not set with -cpu host,aarch64=off
- s/only is/is only
v6 -> v7:
- rebase on Peter's target-arm.next. as a result linux header update could be dropped. Faced a small conflict in hw/arm/virt: Allow virt extensions with KVM due to recently landed hw/arm/virt: Make EL2 accelerator check an accept-list. I dared to keep Richard's R-b though.
v5 -> v6:
- linux header update against v6.16-rc2
v4 -> v5:
- rebase on top of v10.0.0
v3 -> v4:
- fix: only set maint_irq if vms->virt
v2 -> v3:
- KVM EL2 only is set if virtualization option is set
- fixes regression with virtualization=off
- Add checks against unsupported GIC configs until the kernel does
Haibo Xu (4):
hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM
include/hw/intc/arm_gicv3_common.h | 1 + target/arm/kvm_arm.h | 7 ++++++ hw/arm/virt.c | 38 ++++++++++++++++++++++++++---- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++ target/arm/kvm-stub.c | 5 ++++ target/arm/kvm.c | 21 +++++++++++++++++ 7 files changed, 89 insertions(+), 5 deletions(-)
Signature · 1 line
2.49.0
PATCH v8 1/4 hw/arm: Allow setting KVM vGIC maintenance IRQ
From: Haibo Xu <[email protected]>
Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.
This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().
---
v3 -> v4:
- only set maint_irq if vms->virt
v2 -> v3:
- tweak the commit message and explain why we do not proceed the same way as kvm_arm_pmu_set_irq (Peter)
v1 -> v2:
- [Miguel] replaced the has_virt_extensions by the maintenance irq intid property. [Eric] restored kvm_device_check_attr and kvm_device_access standard usage and conditionally call those if the prop is set.
--- include/hw/intc/arm_gicv3_common.h | 1 + hw/arm/virt.c | 3 +++ hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ 4 files changed, 26 insertions(+)
include/hw/intc/arm_gicv3_common.h +1 -0
@@ -231,6 +231,7 @@ struct GICv3State {uint32_t num_cpu;uint32_t num_irq;uint32_t revision; +uint32_t maint_irq;bool lpi_enable;bool nmi_support;bool security_extn;
hw/arm/virt.c +3 -0
@@ -828,6 +828,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) OBJECT(mem), &error_fatal);qdev_prop_set_bit (vms->gic,"has-lpi" , true); } + }else if (vms->virt) { +qdev_prop_set_uint32 (vms->gic,"maintenance-interrupt-id" , +ARCH_GIC_MAINT_IRQ ); } }else {if (!kvm_irqchip_in_kernel ()) {
hw/intc/arm_gicv3_common.c +1 -0
@@ -612,6 +612,7 @@ static const Property arm_gicv3_common_properties[] = {DEFINE_PROP_BOOL ("has-lpi" , GICv3State, lpi_enable,0 ),DEFINE_PROP_BOOL ("has-nmi" , GICv3State, nmi_support,0 ),DEFINE_PROP_BOOL ("has-security-extensions" , GICv3State, security_extn,0 ), +DEFINE_PROP_UINT32 ("maintenance-interrupt-id" , GICv3State, maint_irq,0 ),/* * Compatibility property: force 8 bits of physical priority, even * if the CPU being emulated should have fewer.
hw/intc/arm_gicv3_kvm.c +21 -0
@@ -22,6 +22,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/arm/virt.h" #include "qemu/error-report.h" #include "qemu/module.h" #include "system/kvm.h"
@@ -825,6 +826,26 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp)return ; } +if (s->maint_irq) { +int ret; + + ret =kvm_device_check_attr (s->dev_fd, +KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ ,0 ); +if (!ret) { +error_setg_errno (errp, errno, +"VGICv3 setting maintenance IRQ is not " +"supported by this host kernel" ); +return ; + } + + ret =kvm_device_access (s->dev_fd,KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ ,0 , + &s->maint_irq, true, errp); +if (ret) { +error_setg_errno (errp, errno,"Failed to set VGIC maintenance IRQ" ); +return ; + } + } + multiple_redist_region_allowed =kvm_device_check_attr (s->dev_fd,KVM_DEV_ARM_VGIC_GRP_ADDR ,KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION );
Signature · 1 line
2.49.0
PATCH v8 2/4 target/arm/kvm: Add helper to detect EL2 when using KVM
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
Introduce query support for KVM_CAP_ARM_EL2.
--- target/arm/kvm_arm.h | 7 +++++++ target/arm/kvm-stub.c | 5 +++++ target/arm/kvm.c | 5 +++++ 3 files changed, 17 insertions(+)
target/arm/kvm_arm.h +7 -0
@@ -191,6 +191,13 @@ bool kvm_arm_sve_supported(void);*/ bool kvm_arm_mte_supported (void ); +/** +* kvm_arm_el2_supported: +* +* Returns true if KVM can enable EL2 and false otherwise. +*/ +bool kvm_arm_el2_supported (void ); +/** * kvm_arm_get_max_vm_ipa_size: * @ms: Machine state handle
target/arm/kvm-stub.c +5 -0
@@ -47,6 +47,11 @@ bool kvm_arm_mte_supported(void)return false; } +bool kvm_arm_el2_supported (void ) +{ +return false; +} +/* * These functions should never actually be called without KVM support. */
target/arm/kvm.c +5 -0
@@ -1762,6 +1762,11 @@ bool kvm_arm_aarch32_supported(void)return kvm_check_extension (kvm_state,KVM_CAP_ARM_EL1_32BIT ); } +bool kvm_arm_el2_supported (void ) +{ +return kvm_check_extension (kvm_state,KVM_CAP_ARM_EL2 ); +} +bool kvm_arm_sve_supported (void ) {return kvm_check_extension (kvm_state,KVM_CAP_ARM_SVE );
Signature · 1 line
2.49.0
PATCH v8 3/4 target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.
---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)
v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu
--- target/arm/kvm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
target/arm/kvm.c +16 -0
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)*/ int fdarray[3 ];bool sve_supported; +bool el2_supported;bool pmu_supported = false;uint64_t features =0 ;int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) init.features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; } +/* +* Ask for EL2 if supported. +*/ + el2_supported =kvm_arm_el2_supported (); +if (el2_supported) { + init.features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + } +/* * Ask for Pointer Authentication if supported, so that we get * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) features |=1ULL <<ARM_FEATURE_AARCH64 ; features |=1ULL <<ARM_FEATURE_GENERIC_TIMER ; +if (el2_supported) { + features |=1ULL <<ARM_FEATURE_EL2 ; + } + ahcf->features = features;return true;
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |= (1 <<KVM_ARM_VCPU_PTRAUTH_ADDRESS |1 <<KVM_ARM_VCPU_PTRAUTH_GENERIC ); } +if (cpu->has_el2 &&kvm_arm_el2_supported ()) { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_HAS_EL2 ; + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cpu);
Signature · 1 line
2.49.0
PATCH v8 4/4 hw/arm/virt: Allow virt extensions with KVM
In reply to Eric AugerFrom: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.
Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.
---
v7 -> v8:
- add the migration blocker
- s/only is/is only
- check aarch64=off
- update the commit message
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.
v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)
v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message
--- hw/arm/virt.c | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)
hw/arm/virt.c +30 -5
@@ -65,6 +65,7 @@#include "hw/intc/arm_gic.h" #include "hw/intc/arm_gicv3_common.h" #include "hw/intc/arm_gicv3_its_common.h" +#include "migration/blocker.h" #include "hw/irq.h" #include "kvm_arm.h" #include "hvf_arm.h"
@@ -792,6 +793,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)default :g_assert_not_reached (); } + +if (kvm_enabled () && vms->virt && + (revision !=3 || !kvm_irqchip_in_kernel ())) { +error_report ("KVM EL2 is only supported with in-kernel GICv3" ); +exit (1 ); + } + vms->gic =qdev_new (gictype);qdev_prop_set_uint32 (vms->gic,"revision" , revision);qdev_prop_set_uint32 (vms->gic,"num-cpu" , smp_cpus);
@@ -2066,6 +2074,10 @@ static void virt_post_cpus_gic_realized(VirtMachineState *vms,memory_region_init_ram (pvtime,NULL ,"pvtime" , pvtime_size,NULL );memory_region_add_subregion (sysmem, pvtime_reg_base, pvtime); } +if (!aarch64 && vms->virt) { +error_report ("vcpu with both EL1_32BIT and HAS_EL2 is not supported" ); +exit (1 ); + }CPU_FOREACH (cpu) {if (pmu) {
@@ -2211,11 +2223,24 @@ static void machvirt_init(MachineState *machine)exit (1 ); } -if (vms->virt && !tcg_enabled () && !qtest_enabled ()) { -error_report ("mach-virt: %s does not support providing " -"Virtualization extensions to the guest CPU" , -current_accel_name ()); -exit (1 ); +if (vms->virt) { +if (!(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) { +error_report ("mach-virt: %s does not support providing " +"Virtualization extensions to the guest CPU" , +current_accel_name ()); +exit (1 ); + } +if (kvm_enabled ()) { +Error *kvm_nv_migration_blocker =NULL ; + +error_setg (&kvm_nv_migration_blocker, +"Live migration disabled due to KVM nested virt enabled" ); +if (migrate_add_blocker (&kvm_nv_migration_blocker,NULL )) { +error_free (kvm_nv_migration_blocker); +return ; + } + } }if (vms->mte &&hvf_enabled ()) {
Signature · 1 line
2.49.0
Re: [PATCH v8 4/4] hw/arm/virt: Allow virt extensions with KVM
Quoted 24 lines · Eric Auger
From: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.
Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2,ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.hw/arm/virt.c +9
-0+if (vms->virt) { +if (!(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) { +error_report ("mach-virt: %s does not support providing " +"Virtualization extensions to the guest CPU" , +current_accel_name ()); +exit (1 ); + } +if (kvm_enabled ()) {
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2244
Quoted 10 lines
hw/arm/virt.c +9
-0+Error *kvm_nv_migration_blocker =NULL ; + +error_setg (&kvm_nv_migration_blocker, +"Live migration disabled due to KVM nested virt enabled" ); +if (migrate_add_blocker (&kvm_nv_migration_blocker,NULL )) { +error_free (kvm_nv_migration_blocker); +return ; + } + } }
I think a better place to set the migration blocker is in the GIC code itself (maybe in kvm_arm_gicv3_realize() ?) : we want to disable migration for any setup with a GICv3 + KVM + EL2, not just the virt board.
I also don't understand why we don't hit the assert in gicv3_init_cpuif() that checks that if we're not TCG or qtest accelerators then the CPU doesn't have EL2 or EL3. It looks to me from reading the code that we ought to go through that function in the KVM case.
The rest of the series looks OK to me.
thanks
-- PMM
Re: [PATCH v8 4/4] hw/arm/virt: Allow virt extensions with KVM
Hi Peter,
Quoted 33 lines · Peter Maydell
Quoted 22 lines · Eric Auger
From: Haibo Xu <[email protected]>
Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.
Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.
Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2,ICH_EISR_EL2,ICH_ELRSR_EL2,ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.hw/arm/virt.c +9
-0+if (vms->virt) { +if (!(kvm_enabled () &&kvm_arm_el2_supported ()) && + !tcg_enabled () && !qtest_enabled ()) { +error_report ("mach-virt: %s does not support providing " +"Virtualization extensions to the guest CPU" , +current_accel_name ()); +exit (1 ); + } +if (kvm_enabled ()) {hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2244
Quoted 10 lines
hw/arm/virt.c +9
-0+Error *kvm_nv_migration_blocker =NULL ; + +error_setg (&kvm_nv_migration_blocker, +"Live migration disabled due to KVM nested virt enabled" ); +if (migrate_add_blocker (&kvm_nv_migration_blocker,NULL )) { +error_free (kvm_nv_migration_blocker); +return ; + } + } }
Quoted 4 lines
I think a better place to set the migration blocker is in the GIC code itself (maybe in
kvm_arm_gicv3_realize()?) : we want to disable migration for any setup with a GICv3 + KVM + EL2, not just the virt board.
OK
Quoted 6 lines
I also don't understand why we don't hit the assert in
gicv3_init_cpuif()that checks that if we're not TCG or qtest accelerators then the CPU doesn't have EL2 or EL3. It looks to me from reading the code that we ought to go through that function in the KVM case.
with kvm gicv3 we enter
kvm_arm_gicv3_realize/arm_gicv3_common_realize
but we don't enter arm_gic_realize which calls gicv3_init_cpuif Thanks Eric
Quoted 6 lines
The rest of the series looks OK to me.
thanks
-- PMM