Skip to contentSkip to conversation

ARM Nested Virt Support

PATCH Superseded Eric Auger 2 participants 7 messages Jul 7, 2025

v8 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v8 cover ARM Nested Virt Support

This is candidate for 10.1.

For gaining virt functionality in KVM accelerated L1, The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
with: -machine virt,virtualization=on.

This series can be found at:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v8
previous:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v7

Original version from Miguel:
[1] https://lore.kernel.org/all/[email protected]/
version from Haibo:
[2] https://lore.kernel.org/qemu-devel/[email protected]/

History:
v7 -> v8
- add migration blocker when virt is set along with kvm enabled
- test virt is not set with -cpu host,aarch64=off
- s/only is/is only

v6 -> v7:
- rebase on Peter's target-arm.next. as a result linux header update could be dropped. Faced a small conflict in hw/arm/virt: Allow virt extensions with KVM due to recently landed hw/arm/virt: Make EL2 accelerator check an accept-list. I dared to keep Richard's R-b though.

v5 -> v6:
- linux header update against v6.16-rc2

v4 -> v5:
- rebase on top of v10.0.0

v3 -> v4:
- fix: only set maint_irq if vms->virt

v2 -> v3:
- KVM EL2 only is set if virtualization option is set
- fixes regression with virtualization=off
- Add checks against unsupported GIC configs until the kernel does

Haibo Xu (4):
hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM

 include/hw/intc/arm_gicv3_common.h |  1 +
 target/arm/kvm_arm.h               |  7 ++++++
 hw/arm/virt.c                      | 38 ++++++++++++++++++++++++++----
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++
 target/arm/kvm-stub.c              |  5 ++++
 target/arm/kvm.c                   | 21 +++++++++++++++++
 7 files changed, 89 insertions(+), 5 deletions(-)
Signature · 1 line
2.49.0
author
#1 lore raw

PATCH v8 1/4 hw/arm: Allow setting KVM vGIC maintenance IRQ

From: Haibo Xu <[email protected]>

Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.

This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().

---

v3 -> v4:
- only set maint_irq if vms->virt

v2 -> v3:
- tweak the commit message and explain why we do not proceed the same way as kvm_arm_pmu_set_irq (Peter)

v1 -> v2:
- [Miguel] replaced the has_virt_extensions by the maintenance irq intid property. [Eric] restored kvm_device_check_attr and kvm_device_access standard usage and conditionally call those if the prop is set.

---
 include/hw/intc/arm_gicv3_common.h |  1 +
 hw/arm/virt.c                      |  3 +++
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
 4 files changed, 26 insertions(+)
include/hw/intc/arm_gicv3_common.h +1 -0
index a3d6a0e507..c18503869f 100644
@@ -231,6 +231,7 @@ struct GICv3State {
     uint32_t num_cpu;
     uint32_t num_irq;
     uint32_t revision;
+    uint32_t maint_irq;
     bool lpi_enable;
     bool nmi_support;
     bool security_extn;
hw/arm/virt.c +3 -0
index 3bcdf92e2f..550a272fbb 100644
@@ -828,6 +828,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
                                          OBJECT(mem), &error_fatal);
                 qdev_prop_set_bit(vms->gic, "has-lpi", true);
             }
+        } else if (vms->virt) {
+            qdev_prop_set_uint32(vms->gic, "maintenance-interrupt-id",
+                                 ARCH_GIC_MAINT_IRQ);
         }
     } else {
         if (!kvm_irqchip_in_kernel()) {
hw/intc/arm_gicv3_common.c +1 -0
index 1cee68193c..e438d8c042 100644
@@ -612,6 +612,7 @@ static const Property arm_gicv3_common_properties[] = {
     DEFINE_PROP_BOOL("has-lpi", GICv3State, lpi_enable, 0),
     DEFINE_PROP_BOOL("has-nmi", GICv3State, nmi_support, 0),
     DEFINE_PROP_BOOL("has-security-extensions", GICv3State, security_extn, 0),
+    DEFINE_PROP_UINT32("maintenance-interrupt-id", GICv3State, maint_irq, 0),
     /*
      * Compatibility property: force 8 bits of physical priority, even
      * if the CPU being emulated should have fewer.
hw/intc/arm_gicv3_kvm.c +21 -0
index 3be3bf6c28..b30aac7aee 100644
@@ -22,6 +22,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/arm/virt.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
 #include "system/kvm.h"
@@ -825,6 +826,26 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp)
         return;
     }
 
+    if (s->maint_irq) {
+        int ret;
+
+        ret = kvm_device_check_attr(s->dev_fd,
+                                    KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0);
+        if (!ret) {
+            error_setg_errno(errp, errno,
+                             "VGICv3 setting maintenance IRQ is not "
+                             "supported by this host kernel");
+            return;
+        }
+
+        ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0,
+                                &s->maint_irq, true, errp);
+        if (ret) {
+            error_setg_errno(errp, errno, "Failed to set VGIC maintenance IRQ");
+            return;
+       }
+    }
+
     multiple_redist_region_allowed =
         kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
                               KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION);
Signature · 1 line
2.49.0
author
#2 lore raw

PATCH v8 2/4 target/arm/kvm: Add helper to detect EL2 when using KVM

In reply to Eric Auger

From: Haibo Xu <[email protected]>

Introduce query support for KVM_CAP_ARM_EL2.

---
 target/arm/kvm_arm.h  | 7 +++++++
 target/arm/kvm-stub.c | 5 +++++
 target/arm/kvm.c      | 5 +++++
 3 files changed, 17 insertions(+)
target/arm/kvm_arm.h +7 -0
index 7dc83caed5..b4cad05155 100644
@@ -191,6 +191,13 @@ bool kvm_arm_sve_supported(void);
  */
 bool kvm_arm_mte_supported(void);
 
+/**
+ * kvm_arm_el2_supported:
+ *
+ * Returns true if KVM can enable EL2 and false otherwise.
+ */
+bool kvm_arm_el2_supported(void);
+
 /**
  * kvm_arm_get_max_vm_ipa_size:
  * @ms: Machine state handle
target/arm/kvm-stub.c +5 -0
index 34e57fab01..c93462c5b9 100644
@@ -47,6 +47,11 @@ bool kvm_arm_mte_supported(void)
     return false;
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return false;
+}
+
 /*
  * These functions should never actually be called without KVM support.
  */
target/arm/kvm.c +5 -0
index 426f8b159e..e5708e54ae 100644
@@ -1762,6 +1762,11 @@ bool kvm_arm_aarch32_supported(void)
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL1_32BIT);
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL2);
+}
+
 bool kvm_arm_sve_supported(void)
 {
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_SVE);
Signature · 1 line
2.49.0
author
#3 lore raw

PATCH v8 3/4 target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported

In reply to Eric Auger

From: Haibo Xu <[email protected]>

KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.

---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)

v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu

---
 target/arm/kvm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
target/arm/kvm.c +16 -0
index e5708e54ae..46e5f19637 100644
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
      */
     int fdarray[3];
     bool sve_supported;
+    bool el2_supported;
     bool pmu_supported = false;
     uint64_t features = 0;
     int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         init.features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
+    /*
+     * Ask for EL2 if supported.
+     */
+    el2_supported = kvm_arm_el2_supported();
+    if (el2_supported) {
+        init.features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
+
     /*
      * Ask for Pointer Authentication if supported, so that we get
      * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     features |= 1ULL << ARM_FEATURE_AARCH64;
     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;
 
+    if (el2_supported) {
+        features |= 1ULL << ARM_FEATURE_EL2;
+    }
+
     ahcf->features = features;
 
     return true;
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= (1 << KVM_ARM_VCPU_PTRAUTH_ADDRESS |
                                       1 << KVM_ARM_VCPU_PTRAUTH_GENERIC);
     }
+    if (cpu->has_el2 && kvm_arm_el2_supported()) {
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cpu);
Signature · 1 line
2.49.0
author
#4 lore raw

PATCH v8 4/4 hw/arm/virt: Allow virt extensions with KVM

In reply to Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.

Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

---
v7 -> v8:
- add the migration blocker
- s/only is/is only
- check aarch64=off
- update the commit message

v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.

v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message

---
 hw/arm/virt.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
hw/arm/virt.c +30 -5
index 550a272fbb..54337527d0 100644
@@ -65,6 +65,7 @@
 #include "hw/intc/arm_gic.h"
 #include "hw/intc/arm_gicv3_common.h"
 #include "hw/intc/arm_gicv3_its_common.h"
+#include "migration/blocker.h"
 #include "hw/irq.h"
 #include "kvm_arm.h"
 #include "hvf_arm.h"
@@ -792,6 +793,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 is only supported with in-kernel GICv3");
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
@@ -2066,6 +2074,10 @@ static void virt_post_cpus_gic_realized(VirtMachineState *vms,
             memory_region_init_ram(pvtime, NULL, "pvtime", pvtime_size, NULL);
             memory_region_add_subregion(sysmem, pvtime_reg_base, pvtime);
         }
+        if (!aarch64 && vms->virt) {
+            error_report("vcpu with both EL1_32BIT and HAS_EL2 is not supported");
+            exit(1);
+        }
 
         CPU_FOREACH(cpu) {
             if (pmu) {
@@ -2211,11 +2223,24 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    if (vms->virt && !tcg_enabled() && !qtest_enabled()) {
-        error_report("mach-virt: %s does not support providing "
-                     "Virtualization extensions to the guest CPU",
-                     current_accel_name());
-        exit(1);
+    if (vms->virt) {
+        if (!(kvm_enabled() && kvm_arm_el2_supported()) &&
+            !tcg_enabled() && !qtest_enabled()) {
+            error_report("mach-virt: %s does not support providing "
+                         "Virtualization extensions to the guest CPU",
+                         current_accel_name());
+            exit(1);
+        }
+        if (kvm_enabled()) {
+            Error *kvm_nv_migration_blocker = NULL;
+
+            error_setg(&kvm_nv_migration_blocker,
+                       "Live migration disabled due to KVM nested virt enabled");
+            if (migrate_add_blocker(&kvm_nv_migration_blocker, NULL)) {
+                error_free(kvm_nv_migration_blocker);
+                return;
+            }
+        }
     }
 
     if (vms->mte && hvf_enabled()) {
Signature · 1 line
2.49.0
#5 lore raw

Re: [PATCH v8 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 24 lines · Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.

Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

hw/arm/virt.c +9 -0
+    if (vms->virt) {
+        if (!(kvm_enabled() && kvm_arm_el2_supported()) &&
+            !tcg_enabled() && !qtest_enabled()) {
+            error_report("mach-virt: %s does not support providing "
+                         "Virtualization extensions to the guest CPU",
+                         current_accel_name());
+            exit(1);
+        }
+        if (kvm_enabled()) {

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2244

Quoted 10 lines
hw/arm/virt.c +9 -0
+            Error *kvm_nv_migration_blocker = NULL;
+
+            error_setg(&kvm_nv_migration_blocker,
+                       "Live migration disabled due to KVM nested virt enabled");
+            if (migrate_add_blocker(&kvm_nv_migration_blocker, NULL)) {
+                error_free(kvm_nv_migration_blocker);
+                return;
+            }
+        }
     }

I think a better place to set the migration blocker is in the GIC code itself (maybe in kvm_arm_gicv3_realize() ?) : we want to disable migration for any setup with a GICv3 + KVM + EL2, not just the virt board.

I also don't understand why we don't hit the assert in gicv3_init_cpuif() that checks that if we're not TCG or qtest accelerators then the CPU doesn't have EL2 or EL3. It looks to me from reading the code that we ought to go through that function in the KVM case.

The rest of the series looks OK to me.

thanks
-- PMM

author
#6 lore raw

Re: [PATCH v8 4/4] hw/arm/virt: Allow virt extensions with KVM

Hi Peter,

Quoted 33 lines · Peter Maydell
Quoted 22 lines · Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Check neither in-kernel GICv3 nor aarch64=off is used along with KVM EL2.

Also add a migration blocker in case KVM NV is set as some GIC registers are most probably not properly saved/restored:
ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

hw/arm/virt.c +9 -0
+    if (vms->virt) {
+        if (!(kvm_enabled() && kvm_arm_el2_supported()) &&
+            !tcg_enabled() && !qtest_enabled()) {
+            error_report("mach-virt: %s does not support providing "
+                         "Virtualization extensions to the guest CPU",
+                         current_accel_name());
+            exit(1);
+        }
+        if (kvm_enabled()) {

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2244

Quoted 10 lines
hw/arm/virt.c +9 -0
+            Error *kvm_nv_migration_blocker = NULL;
+
+            error_setg(&kvm_nv_migration_blocker,
+                       "Live migration disabled due to KVM nested virt enabled");
+            if (migrate_add_blocker(&kvm_nv_migration_blocker, NULL)) {
+                error_free(kvm_nv_migration_blocker);
+                return;
+            }
+        }
     }
Quoted 4 lines

I think a better place to set the migration blocker is in the GIC code itself (maybe in kvm_arm_gicv3_realize() ?) : we want to disable migration for any setup with a GICv3 + KVM + EL2, not just the virt board.

OK

Quoted 6 lines

I also don't understand why we don't hit the assert in gicv3_init_cpuif() that checks that if we're not TCG or qtest accelerators then the CPU doesn't have EL2 or EL3. It looks to me from reading the code that we ought to go through that function in the KVM case.

with kvm gicv3 we enter
kvm_arm_gicv3_realize/arm_gicv3_common_realize

but we don't enter arm_gic_realize which calls gicv3_init_cpuif Thanks Eric

Quoted 6 lines

The rest of the series looks OK to me.

thanks
-- PMM