Skip to contentSkip to conversation

ARM Nested Virt Support

PATCH Superseded Eric Auger 3 participants 18 messages Jul 2, 2025 – Jul 7, 2025

Subsystems (3)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v7 cover ARM Nested Virt Support

This is based on Peter's target-arm.next. This is candidate for 10.1.

For gaining virt functionality in KVM accelerated L1, The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
with: -machine virt,virtualization=on.

This series can be found at:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v7

Original version from Miguel:
[1] https://lore.kernel.org/all/[email protected]/
version from Haibo:
[2] https://lore.kernel.org/qemu-devel/[email protected]/

History:

v6 -> v7:
- rebase on Peter's target-arm.next. as a result linux header update could be dropped. Faced a small conflict in hw/arm/virt: Allow virt extensions with KVM due to recently landed hw/arm/virt: Make EL2 accelerator check an accept-list. I dared to keep Richard's R-b though.

v5 -> v6:
- linux header update against v6.16-rc2

v4 -> v5:
- rebase on top of v10.0.0

v3 -> v4:
- fix: only set maint_irq if vms->virt

v2 -> v3:
- KVM EL2 only is set if virtualization option is set
- fixes regression with virtualization=off
- Add checks against unsupported GIC configs until the kernel does

Haibo Xu (4):
hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM

 include/hw/intc/arm_gicv3_common.h |  1 +
 target/arm/kvm_arm.h               |  7 +++++++
 hw/arm/virt.c                      | 13 ++++++++++++-
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
 target/arm/kvm-stub.c              |  5 +++++
 target/arm/kvm.c                   | 21 +++++++++++++++++++++
 7 files changed, 68 insertions(+), 1 deletion(-)
Signature · 1 line
2.49.0
author
#1 lore raw

PATCH v7 1/4 hw/arm: Allow setting KVM vGIC maintenance IRQ

From: Haibo Xu <[email protected]>

Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.

This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().

---

v3 -> v4:
- only set maint_irq if vms->virt

v2 -> v3:
- tweak the commit message and explain why we do not proceed the same way as kvm_arm_pmu_set_irq (Peter)

v1 -> v2:
- [Miguel] replaced the has_virt_extensions by the maintenance irq intid property. [Eric] restored kvm_device_check_attr and kvm_device_access standard usage and conditionally call those if the prop is set.

---
 include/hw/intc/arm_gicv3_common.h |  1 +
 hw/arm/virt.c                      |  3 +++
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
 4 files changed, 26 insertions(+)
include/hw/intc/arm_gicv3_common.h +1 -0
index a3d6a0e507..c18503869f 100644
@@ -231,6 +231,7 @@ struct GICv3State {
     uint32_t num_cpu;
     uint32_t num_irq;
     uint32_t revision;
+    uint32_t maint_irq;
     bool lpi_enable;
     bool nmi_support;
     bool security_extn;
hw/arm/virt.c +3 -0
index 3bcdf92e2f..550a272fbb 100644
@@ -828,6 +828,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
                                          OBJECT(mem), &error_fatal);
                 qdev_prop_set_bit(vms->gic, "has-lpi", true);
             }
+        } else if (vms->virt) {
+            qdev_prop_set_uint32(vms->gic, "maintenance-interrupt-id",
+                                 ARCH_GIC_MAINT_IRQ);
         }
     } else {
         if (!kvm_irqchip_in_kernel()) {
hw/intc/arm_gicv3_common.c +1 -0
index 1cee68193c..e438d8c042 100644
@@ -612,6 +612,7 @@ static const Property arm_gicv3_common_properties[] = {
     DEFINE_PROP_BOOL("has-lpi", GICv3State, lpi_enable, 0),
     DEFINE_PROP_BOOL("has-nmi", GICv3State, nmi_support, 0),
     DEFINE_PROP_BOOL("has-security-extensions", GICv3State, security_extn, 0),
+    DEFINE_PROP_UINT32("maintenance-interrupt-id", GICv3State, maint_irq, 0),
     /*
      * Compatibility property: force 8 bits of physical priority, even
      * if the CPU being emulated should have fewer.
hw/intc/arm_gicv3_kvm.c +21 -0
index 3be3bf6c28..b30aac7aee 100644
@@ -22,6 +22,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/arm/virt.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
 #include "system/kvm.h"
@@ -825,6 +826,26 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp)
         return;
     }
 
+    if (s->maint_irq) {
+        int ret;
+
+        ret = kvm_device_check_attr(s->dev_fd,
+                                    KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0);
+        if (!ret) {
+            error_setg_errno(errp, errno,
+                             "VGICv3 setting maintenance IRQ is not "
+                             "supported by this host kernel");
+            return;
+        }
+
+        ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0,
+                                &s->maint_irq, true, errp);
+        if (ret) {
+            error_setg_errno(errp, errno, "Failed to set VGIC maintenance IRQ");
+            return;
+       }
+    }
+
     multiple_redist_region_allowed =
         kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
                               KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION);
Signature · 1 line
2.49.0
#2 lore raw

Re: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ

Quoted 27 lines · Eric Auger

From: Haibo Xu <[email protected]>

Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.

This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().

(quoted diff) +5 -0
@@ -231,6 +231,7 @@ struct GICv3State {
     uint32_t num_cpu;
     uint32_t num_irq;
     uint32_t revision;
+    uint32_t maint_irq;
     bool lpi_enable;
     bool nmi_support;
     bool security_extn;
 
+    if (s->maint_irq) {
+        int ret;
+
+        ret = kvm_device_check_attr(s->dev_fd,

Review struct GICv3State {

Quoted 10 lines
(quoted diff) +10 -0
+                                    KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0);
+        if (!ret) {
+            error_setg_errno(errp, errno,
+                             "VGICv3 setting maintenance IRQ is not "
+                             "supported by this host kernel");
+            return;
+        }
+
+        ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0,
+                                &s->maint_irq, true, errp);

This doesn't seem to line up with what the kernel documents for KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ:

https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says

# KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field of kvm_device_attr encodes the following values:
#

#      bits:     | 31   ....    5 | 4  ....  0 |
#      values:   |      RES0      |   vINTID   |

#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.

but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a uint32_t with the vINTID in it.

Looking at the kernel code in vgic_v3_attr_regs_access() it looks like maybe the kernel docs are wrong, but I'm not sure.

thanks
-- PMM

maintainer
#3 lore raw

Re: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ

Quoted 57 lines · Peter Maydell
Quoted 27 lines · Eric Auger

From: Haibo Xu <[email protected]>

Allow virt arm machine to set the interrupt ID for the KVM GIC maintenance interrupt.

This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT hence the choice to perform the setting in the GICv3 realize instead of proceeding the same way as kvm_arm_pmu_set_irq().

(quoted diff) +5 -0
@@ -231,6 +231,7 @@ struct GICv3State {
     uint32_t num_cpu;
     uint32_t num_irq;
     uint32_t revision;
+    uint32_t maint_irq;
     bool lpi_enable;
     bool nmi_support;
     bool security_extn;
 
+    if (s->maint_irq) {
+        int ret;
+
+        ret = kvm_device_check_attr(s->dev_fd,

Review struct GICv3State {

Quoted 10 lines
(quoted diff) +10 -0
+                                    KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0);
+        if (!ret) {
+            error_setg_errno(errp, errno,
+                             "VGICv3 setting maintenance IRQ is not "
+                             "supported by this host kernel");
+            return;
+        }
+
+        ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0,
+                                &s->maint_irq, true, errp);

This doesn't seem to line up with what the kernel documents for KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ:

https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says

# KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field of kvm_device_attr encodes the following values:
#

#      bits:     | 31   ....    5 | 4  ....  0 |
#      values:   |      RES0      |   vINTID   |

#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.

Quoted 5 lines

but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a uint32_t with the vINTID in it.

Looking at the kernel code in vgic_v3_attr_regs_access() it looks like maybe the kernel docs are wrong, but I'm not sure.

I think this is a pretty bad case of one person implementing something, and another (me) hoping (but not checking) it would be implemented in a particular way. Oh well.

Would the following match your expectations (and the code)?

Documentation/virt/kvm/devices/arm-vgic-v3.rst +1 -1
index e860498b1e359..4843c91052786 100644
@@ -299,7 +299,7 @@ Groups:
   KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
    Attributes:
 
-    The attr field of kvm_device_attr encodes the following values:
+    The attribute data pointed to by kvm_device_attr.addr is a __u32 value::
 
       bits:     | 31   ....    5 | 4  ....  0 |
       values:   |      RES0      |   vINTID   |

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#4 lore raw

Re: [PATCH v7 1/4] hw/arm: Allow setting KVM vGIC maintenance IRQ

Quoted 34 lines · Marc Zyngier
Quoted 14 lines · Peter Maydell

https://www.kernel.org/doc/Documentation/virt/kvm/devices/arm-vgic-v3.rst
says

# KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
# Attributes:
#
# The attr field of kvm_device_attr encodes the following values:
#

#      bits:     | 31   ....    5 | 4  ....  0 |
#      values:   |      RES0      |   vINTID   |

#
# The vINTID specifies which interrupt is generated when the vGIC
# must generate a maintenance interrupt. This must be a PPI.

Quoted 5 lines

but this code sets kvmattr.addr to 0 and kvmaddr.addr to the address of a uint32_t with the vINTID in it.

Looking at the kernel code in vgic_v3_attr_regs_access() it looks like maybe the kernel docs are wrong, but I'm not sure.

I think this is a pretty bad case of one person implementing something, and another (me) hoping (but not checking) it would be implemented in a particular way. Oh well.

Would the following match your expectations (and the code)?

Documentation/virt/kvm/devices/arm-vgic-v3.rst +0 -0
index e860498b1e359..4843c91052786 100644

Documentation/virt/kvm/devices/arm-vgic-v3.rst Groups: · line 302

Quoted 5 lines
Documentation/virt/kvm/devices/arm-vgic-v3.rst +1 -1
@@ -299,7 +299,7 @@ Groups:
   KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ
    Attributes:
 
-    The attr field of kvm_device_attr encodes the following values:
+    The attribute data pointed to by kvm_device_attr.addr is a __u32 value::

I think it's maybe better to explicitly say "Attributes: ignored" or "Attributes: none" or similar. Otherwise it looks like the text is describing attr, because it follows the "Attributes:" subheader. (Though we don't do this for KVM_DEV_ARM_VGIC_GRP_NR_IRQS either, which also ignores attrs and reads/writes a value pointed to by addr.)

Documentation/virt/kvm/devices/arm-vgic-v3.rst Groups: · line 305

Quoted 3 lines
Documentation/virt/kvm/devices/arm-vgic-v3.rst +0 -0
       bits:     | 31   ....    5 | 4  ....  0 |
       values:   |      RES0      |   vINTID   |

Is the kvmattr.attr field mandatorily 0, or does the kernel ignore it? (I think the answer is "ignored".)

Side note, we seem to allow this attribute to be read via get_attr for gicv2, even though has_attr denies it exists there, because the get code is in vgic_get_common_attr(). set_attr is in the gicv3-only vgic_v3_attr_regs_access().

-- PMM

author
#5 lore raw

PATCH v7 2/4 target/arm/kvm: Add helper to detect EL2 when using KVM

In reply to Eric Auger

From: Haibo Xu <[email protected]>

Introduce query support for KVM_CAP_ARM_EL2.

---
 target/arm/kvm_arm.h  | 7 +++++++
 target/arm/kvm-stub.c | 5 +++++
 target/arm/kvm.c      | 5 +++++
 3 files changed, 17 insertions(+)
target/arm/kvm_arm.h +7 -0
index 7dc83caed5..b4cad05155 100644
@@ -191,6 +191,13 @@ bool kvm_arm_sve_supported(void);
  */
 bool kvm_arm_mte_supported(void);
 
+/**
+ * kvm_arm_el2_supported:
+ *
+ * Returns true if KVM can enable EL2 and false otherwise.
+ */
+bool kvm_arm_el2_supported(void);
+
 /**
  * kvm_arm_get_max_vm_ipa_size:
  * @ms: Machine state handle
target/arm/kvm-stub.c +5 -0
index 34e57fab01..c93462c5b9 100644
@@ -47,6 +47,11 @@ bool kvm_arm_mte_supported(void)
     return false;
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return false;
+}
+
 /*
  * These functions should never actually be called without KVM support.
  */
target/arm/kvm.c +5 -0
index 426f8b159e..e5708e54ae 100644
@@ -1762,6 +1762,11 @@ bool kvm_arm_aarch32_supported(void)
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL1_32BIT);
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL2);
+}
+
 bool kvm_arm_sve_supported(void)
 {
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_SVE);
Signature · 1 line
2.49.0
author
#6 lore raw

PATCH v7 3/4 target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported

In reply to Eric Auger

From: Haibo Xu <[email protected]>

KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.

---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)

v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu

---
 target/arm/kvm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
target/arm/kvm.c +16 -0
index e5708e54ae..46e5f19637 100644
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
      */
     int fdarray[3];
     bool sve_supported;
+    bool el2_supported;
     bool pmu_supported = false;
     uint64_t features = 0;
     int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         init.features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
+    /*
+     * Ask for EL2 if supported.
+     */
+    el2_supported = kvm_arm_el2_supported();
+    if (el2_supported) {
+        init.features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
+
     /*
      * Ask for Pointer Authentication if supported, so that we get
      * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     features |= 1ULL << ARM_FEATURE_AARCH64;
     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;
 
+    if (el2_supported) {
+        features |= 1ULL << ARM_FEATURE_EL2;
+    }
+
     ahcf->features = features;
 
     return true;
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= (1 << KVM_ARM_VCPU_PTRAUTH_ADDRESS |
                                       1 << KVM_ARM_VCPU_PTRAUTH_GENERIC);
     }
+    if (cpu->has_el2 && kvm_arm_el2_supported()) {
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cpu);
Signature · 1 line
2.49.0
#7 lore raw

Re: [PATCH v7 3/4] target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported

Quoted 62 lines · Eric Auger

From: Haibo Xu <[email protected]>

KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.

---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)

v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu

---
 target/arm/kvm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
target/arm/kvm.c +13 -0
index e5708e54ae..46e5f19637 100644
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
      */
     int fdarray[3];
     bool sve_supported;
+    bool el2_supported;
     bool pmu_supported = false;
     uint64_t features = 0;
     int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         init.features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
+    /*
+     * Ask for EL2 if supported.
+     */
+    el2_supported = kvm_arm_el2_supported();
+    if (el2_supported) {
+        init.features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
+
     /*
      * Ask for Pointer Authentication if supported, so that we get
      * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     features |= 1ULL << ARM_FEATURE_AARCH64;
     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;
 
+    if (el2_supported) {
+        features |= 1ULL << ARM_FEATURE_EL2;
+    }
+
     ahcf->features = features;
 
     return true;

target/arm/kvm.c int kvm_arch_init_vcpu(CPUState *cs)

Quoted 10 lines
target/arm/kvm.c +3 -1
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= (1 << KVM_ARM_VCPU_PTRAUTH_ADDRESS |
                                       1 << KVM_ARM_VCPU_PTRAUTH_GENERIC);
     }
+    if (cpu->has_el2 && kvm_arm_el2_supported()) {
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cpu);
--

How does this interact with creating an aarch32 VM with "-cpu host,aarch64=off" ? Looking at the kernel, it will fail vcpu_init if you ask for a vcpu with both EL1_32BIT and HAS_EL2. I guess since the virt board will only request HAS_EL2 if you say virtualization=on this is fine. Do we give a sensible error message if the user tries
-accel kvm -M virt,virtualization=on -cpu host,aarch64=off ?

-- PMM

author
#8 lore raw

Re: [PATCH v7 3/4] target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported

Hi Peter,

Quoted 69 lines · Peter Maydell
Quoted 68 lines · Eric Auger

From: Haibo Xu <[email protected]>

KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2. In case the host does support NV, expose the feature.

---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting KVM_ARM_VCPU_HAS_EL2 feature (Peter)

v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h [Richard] and use el2_supported in kvm_arch_init_vcpu

---
 target/arm/kvm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
target/arm/kvm.c +16 -0
index e5708e54ae..46e5f19637 100644
@@ -250,6 +250,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
      */
     int fdarray[3];
     bool sve_supported;
+    bool el2_supported;
     bool pmu_supported = false;
     uint64_t features = 0;
     int err;
@@ -269,6 +270,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         init.features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
+    /*
+     * Ask for EL2 if supported.
+     */
+    el2_supported = kvm_arm_el2_supported();
+    if (el2_supported) {
+        init.features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
+
     /*
      * Ask for Pointer Authentication if supported, so that we get
      * the unsanitized field values for AA64ISAR1_EL1.
@@ -422,6 +431,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     features |= 1ULL << ARM_FEATURE_AARCH64;
     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;
 
+    if (el2_supported) {
+        features |= 1ULL << ARM_FEATURE_EL2;
+    }
+
     ahcf->features = features;
 
     return true;
@@ -1887,6 +1900,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= (1 << KVM_ARM_VCPU_PTRAUTH_ADDRESS |
                                       1 << KVM_ARM_VCPU_PTRAUTH_GENERIC);
     }
+    if (cpu->has_el2 && kvm_arm_el2_supported()) {
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
 
Quoted 10 lines

target/arm/kvm.c int kvm_arch_init_vcpu(CPUState *cs)

Quoted 3 lines
target/arm/kvm.c +0 -1
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cpu);
--

How does this interact with creating an aarch32 VM with "-cpu host,aarch64=off" ? Looking at the kernel, it will fail vcpu_init if you ask for a vcpu with both EL1_32BIT and HAS_EL2. I guess since the virt board will only request HAS_EL2 if you say virtualization=on this is fine. Do we give a sensible error message if the user tries
-accel kvm -M virt,virtualization=on -cpu host,aarch64=off ?

Not yet. I will add one.

Thanks!

Eric

Quoted 3 lines

-- PMM

author
#9 lore raw

PATCH v7 4/4 hw/arm/virt: Allow virt extensions with KVM

In reply to Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Also check only in-kernel GICv3 is used along with KVM EL2.

---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.

v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message

---
 hw/arm/virt.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
hw/arm/virt.c +9 -1
index 550a272fbb..1c0a2c43c4 100644
@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 only is supported with in-kernel GICv3");
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    if (vms->virt && !tcg_enabled() && !qtest_enabled()) {
+    if (vms->virt && !(kvm_enabled() && kvm_arm_el2_supported()) &&
+                     !tcg_enabled() && !qtest_enabled()) {
         error_report("mach-virt: %s does not support providing "
                      "Virtualization extensions to the guest CPU",
                      current_accel_name());
Signature · 1 line
2.49.0
#10 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 33 lines · Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Also check only in-kernel GICv3 is used along with KVM EL2.

---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.

v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message

---
 hw/arm/virt.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
hw/arm/virt.c +0 -0
index 550a272fbb..1c0a2c43c4 100644

hw/arm/virt.c static void create_gic(VirtMachineState *vms, MemoryRegion *mem) · line 798

Quoted 7 lines
hw/arm/virt.c +4 -0
@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 only is supported with in-kernel GICv3");

"is only supported"

Quoted 7 lines
hw/arm/virt.c +3 -0
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2225

Quoted 9 lines
hw/arm/virt.c +2 -1
@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    if (vms->virt && !tcg_enabled() && !qtest_enabled()) {
+    if (vms->virt && !(kvm_enabled() && kvm_arm_el2_supported()) &&
+                     !tcg_enabled() && !qtest_enabled()) {
         error_report("mach-virt: %s does not support providing "
                      "Virtualization extensions to the guest CPU",
                      current_accel_name());

Have you tested doing a VM migration of a KVM with EL2 setup? I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

thanks
-- PMM

author
#11 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Hi Peter, Marc,

Quoted 28 lines · Peter Maydell
Quoted 27 lines · Eric Auger

From: Haibo Xu <[email protected]>

Up to now virt support on guest has been only supported with TCG. Now it becomes feasible to use it with KVM acceleration.

Also check only in-kernel GICv3 is used along with KVM EL2.

---
v6 -> v7:
- rebase on top of "hw/arm/virt: Make EL2 accelerator check an accept-list". I dared to keep Richard's R-b though.

v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message

---
 hw/arm/virt.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Quoted 13 lines

hw/arm/virt.c static void create_gic(VirtMachineState *vms, MemoryRegion *mem) · line 798

Quoted 12 lines
hw/arm/virt.c +4 -0
index 550a272fbb..1c0a2c43c4 100644
@@ -792,6 +792,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 only is supported with in-kernel GICv3");

"is only supported"

OK

Quoted 11 lines
Quoted 11 lines
hw/arm/virt.c +3 -0
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
@@ -2211,7 +2218,8 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
Quoted 7 lines

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 2225

Quoted 6 lines
hw/arm/virt.c +2 -1
-    if (vms->virt && !tcg_enabled() && !qtest_enabled()) {
+    if (vms->virt && !(kvm_enabled() && kvm_arm_el2_supported()) &&
+                     !tcg_enabled() && !qtest_enabled()) {
         error_report("mach-virt: %s does not support providing "
                      "Virtualization extensions to the guest CPU",
                      current_accel_name());

Have you tested doing a VM migration of a KVM with EL2 setup?

Yes I did. Marc fixed some save/restore bugs in the past and I was able to migrate after this fix.

Now I would lie if I'd tell you it works well. It is extremely slow to converge. Usually you get rcu stalls in the L2 guest on source too.

Quoted 4 lines

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Do you see some specific registers/resources that would need attention?

Otherwise I can put a migration blocker until we spend more time analyzing migration slowness

What do you think?

Eric

Quoted 4 lines

thanks
-- PMM

#12 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 9 lines · Eric Auger

Hi Peter, Marc,

Quoted 4 lines · Peter Maydell

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Do you see some specific registers/resources that would need attention?

All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

These all need to be exposed via KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG and hw/intc/arm_gicv3_kvm.c needs code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).

I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.

I'm ok with just putting in a migration blocker for the moment, especially if this needs kernel-side changes.

thanks
-- PMM

author
#13 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Hi Peter,

Quoted 20 lines · Peter Maydell
Quoted 8 lines · Eric Auger

Hi Peter, Marc,

Quoted 4 lines · Peter Maydell

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Do you see some specific registers/resources that would need attention?

All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

These all need to be exposed via KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG and hw/intc/arm_gicv3_kvm.c needs code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).

Quoted 3 lines

I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.

OK thank you for the insights.

Quoted 3 lines

I'm ok with just putting in a migration blocker for the moment, especially if this needs kernel-side changes.

OK I will do that until we get this fixed

Thanks!

Eric

Quoted 4 lines

thanks
-- PMM

maintainer
#14 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

In reply to Peter Maydell
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger

Hi Peter, Marc,

Quoted 4 lines · Peter Maydell

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Quoted 5 lines
Quoted 1 line

Do you see some specific registers/resources that would need attention?

All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.

Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.

Quoted 10 lines

These all need to be exposed via KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG and hw/intc/arm_gicv3_kvm.c needs code to be able to save and restore them into the GIC data structures (and we need to make sure the kernel isn't accidentally exposing them as CPU registers via the GET/SET_ONE_REG API, I think, in whatever way we do that for the existing EL1 GIC cpuif registers).

I don't see any of the EL2 sysregs listed in the kernel's gic_v3_icc_reg_descs[], which looks like it's what drives the handling of KVM_DEV_ARM_VGIC_GRP_CPU_SYSREG.

Hmmm... This is indeed pretty inconsistent. I'll have a look at it.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#15 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 19 lines · Marc Zyngier
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger

Hi Peter, Marc,

Quoted 4 lines · Peter Maydell

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Quoted 5 lines
Quoted 1 line

Do you see some specific registers/resources that would need attention?

All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

Quoted 6 lines

It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.

Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.

Good point: we don't need to expose those, only the actual state-holding registers. I was rather mindlessly going through and listing all the ICH_ registers out of the spec...

So should the list be:
ICH_AP*R<n>_EL2, ICH_HCR_EL2, ICH_HR<n>_EL2, ICH_VMCR_EL2
?
(Those are the ones the TCG GICv3 has state for in its data structures.)

How about ICH_VTR_EL2 ? I guess we should handle it the same way we do other existing ID registers (migrate it and write value back to kernel so it can validate that the destination hardware supports the same config) ?

Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1 GICD_TYPER2 register, so we don't try to send and sanity check that on migration at the moment.

thanks
-- PMM

maintainer
#16 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 32 lines · Peter Maydell
Quoted 19 lines · Marc Zyngier
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Eric Auger

Hi Peter, Marc,

Quoted 4 lines · Peter Maydell

I suppose the system registers probably generally Just Work via the sysreg GET/SET_ONE_REG API, but won't the in-kernel GICv3 have extra state that we need to migrate in hw/intc/arm_gicv3_kvm.c ?

Quoted 5 lines
Quoted 1 line

Do you see some specific registers/resources that would need attention?

All the EL2-only-accessible GIC registers: ICH_AP*R<n>_EL2, ICH_EISR_EL2, ICH_ELRSR_EL2, ICH_HCR_EL2, ICH_LR<n>_EL2, etc etc.

Quoted 6 lines

It isn't obvious to me why we'd expose any of the status registers to userspace. ICH_{EISR,ELRSR,MISR}_EL2 are entirely synthetic, and are directly generated by the emulation from the rest of the state.

Save/restoring this state would make things pointlessly complicated, unless I'm missing something obvious.

Good point: we don't need to expose those, only the actual state-holding registers. I was rather mindlessly going through and listing all the ICH_ registers out of the spec...

Quoted 5 lines

So should the list be:
ICH_AP*R<n>_EL2, ICH_HCR_EL2, ICH_HR<n>_EL2, ICH_VMCR_EL2
?
(Those are the ones the TCG GICv3 has state for in its data structures.)

Yup, this matches my own expectations.

Quoted 4 lines

How about ICH_VTR_EL2 ? I guess we should handle it the same way we do other existing ID registers (migrate it and write value back to kernel so it can validate that the destination hardware supports the same config) ?

Indeed. Specially as this contains the number of LRs, which could otherwise lead to UNDEF exceptions if migrating in the wrong direction.

What of ICC_SRE_EL2? We have it as RAO/WI for all the non-RES0 fields. It could be saved/restored as well for the same purposes...

Quoted 3 lines

Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1 GICD_TYPER2 register, so we don't try to send and sanity check that on migration at the moment.

KVM only exposes a virtual GICv3 to the guest, even in the host can do GICv4.0+. The only nit is the "nASSGIcap" bit, which is always advertised to the guest when the host if v4.1-capable. We have a patch series in progress to address this and make it controllable.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#17 lore raw

Re: [PATCH v7 4/4] hw/arm/virt: Allow virt extensions with KVM

Quoted 11 lines · Marc Zyngier
Quoted 3 lines · Peter Maydell

Speaking of GIC ID registers, we never updated QEMU to handle the GICv4.1 GICD_TYPER2 register, so we don't try to send and sanity check that on migration at the moment.

KVM only exposes a virtual GICv3 to the guest, even in the host can do GICv4.0+. The only nit is the "nASSGIcap" bit, which is always advertised to the guest when the host if v4.1-capable. We have a patch series in progress to address this and make it controllable.

Isn't that out-of-spec? If the GIC the guest sees is a GICv3 then GICD_TYPER2 should be RES0...

Anyway, I have some lightly tested QEMU patches that migrate the GICD_TYPER2 value, which I'll send out in a moment.

-- PMM