Skip to contentSkip to conversation

ppc/kvm: Handle CPU compatibility mode correctly for nested guests

PATCH Superseded Amit Machhiwal 2 participants 5 messages Jul 1 – Aug 4 KVM

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover ppc/kvm: Handle CPU compatibility mode correctly for nested guests

On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.

This creates issues for nested virtualization. When booting a nested KVM guest, QEMU may derive the CPU model from the raw hardware PVR and attempt to configure the guest accordingly. However, the host is constrained by the compatibility level negotiated with the hypervisor, and requests exceeding that level are rejected by KVM, leading to guest boot failures such as:

KVM-NESTEDv2: couldn't set guest wide elements

This series addresses the issue by preventing fallback to raw mode when the host itself is booted in a compatibility mode, and by querying the effective CPU compatibility modes supported by the host via KVM. The kernel interface uses copy_struct_from/to_user() for forward and backward ABI compatibility. With these changes, QEMU ensures that nested guests are configured with CPU models consistent with the host compatibility mode, allowing them to boot correctly.

Patch summary:
[1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes [2/3] target/ppc/kvm: Add support for querying host compatibility mode [3/3] target/ppc/kvm: Use host compatibility mode for nested guests

Changes in v4 (based on review from Vaibhav):
- Patch 1: Updated to match kernel v5: ioctl kept as _IO (not _IOWR) so the ioctl number remains stable if the struct grows; the size field is read first by the kernel via get_user() before copy_struct_from_user(). [Vaibhav, Amit]

- Patch 2: Replaced fprintf(stderr, ...) with error_report() for error reporting in kvmppc_get_compat_caps(). Replaced __builtin_ctzll() with portable ctz64(). Simplified kvm_ppc_host_compat_pvr() to switch directly on the capability bit value instead of a derived index, removing the KVM_PPC_COMPAT_CAP_P*_IDX defines from kvm_ppc.h. Dropped the post-ioctl size mismatch check which is no longer meaningful with the copy_struct_from/to_user() model. Added a function comment to kvm_ppc_host_compat_pvr(). [Vaibhav, Amit]
Guard the kvm_ppc_host_compat_pvr() call in cas_check_pvr() with kvm_enabled() to prevent a segfault when QEMU is built with CONFIG_KVM=y but run as a TCG guest (kvm_state is NULL in that case).

- Patch 3: Added a #ifndef CONFIG_KVM compile-time assertion inside the TARGET_PPC64 guard as a sanity check. No functional changes.

Changes in v3 (based on review from Vaibhav):
- Patch 1: Moved compatibility mode check from do_client_architecture_support() to cas_check_pvr(). Instead of error handling when KVM rejects compat mode, now prevents raw mode fallback when host is booted in compatibility mode. This ensures guests cannot exceed the host's compatibility level.

- Patch 2: Added size field to kvm_ppc_compat_caps structure for ABI versioning. Changed flags field from __u32 to __u64. Added capability bit definitions (KVM_PPC_COMPAT_CAP_POWER9/10/11) and KVM_PPC_COMPAT_BITMASK.

- Patch 3: Added size field initialization and validation in kvmppc_get_compat_caps() to ensure ABI compatibility between QEMU and kernel. Changed from H_GUEST_CAP_* constants to KVM_PPC_COMPAT_CAP_* constants. Added capability masking with KVM_PPC_COMPAT_BITMASK.

- Dropped Tested-by from Anushree due to code changes in v3.

Changes in v2:
- Patch 3: Guard compatibility mode code with #if defined(TARGET_PPC64) to fix compilation for ppc32 targets. The POWER9/10/11 PVR constants are only defined for 64-bit builds, and compatibility modes are only relevant for 64-bit systems.

Testing (with kernel v5 patches):

KVM APIv1 Testing
=================
On P10 PowerNV machine (L0)
---------------------------
- P10 L1 KVM guest -> works
- P10 nested L2 KVM guest -> works
- P9 compat nested L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 nested L2 KVM guest -> works

On Powernv11 TCG Guest (L0)
---------------------------
- P11 PowerNV TCG L0 guest -> works
- P11 L1 KVM guest -> works
- P11 L2 KVM guest -> works
- P10 compat L1 KVM guest -> works
- P10 L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 L2 KVM guest -> works

KVM APIv2 Testing
=================
On P11 PowerVM LPAR (L1)
------------------------
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without QEMU patches but Linux patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without Linux patches but QEMU patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works

On P11 LPAR in P10 compat (L1)
------------------------------
- P10 (host compat) L2 KVM guest -> works
- Without QEMU patch but Linux patches
- P10 guest fails to boot as expected (error: kvm run failed Invalid argument)
- Without Linux patch but QEMU patches
- P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea)

On P10 PowerVM LPAR (L1)
------------------------
- P10 L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected

TCG pSeries Guest
=================
- P11 (default) pSeries guest boots fine

ABI Extensibility Testing (struct size 32, extra member)
=========================================================
- Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG, QEMU retries with correct size)
- New struct on Linux kernel, older QEMU -> works (kernel zero-pads trailing fields, QEMU gets correct data)

CI test results: https://gitlab.com/amachhiw/qemu/-/pipelines/2641423050

Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates that will be synced separately once the corresponding kernel patches are merged.

The corresponding Linux kernel patches (v5) are being posted concurrently.

v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Previous kernel patch versions:
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Amit Machhiwal (3):
linux-headers: Add uapi header changes target/ppc/kvm: Add support for querying host compatibility mode target/ppc/kvm: Use host compatibility mode for nested guests

 hw/ppc/spapr_hcall.c            | 14 ++++++
 linux-headers/asm-powerpc/kvm.h | 20 ++++++++
 linux-headers/linux/kvm.h       |  4 ++
 target/ppc/kvm.c                | 87 +++++++++++++++++++++++++++++++++
 target/ppc/kvm_ppc.h            |  7 +++
 5 files changed, 132 insertions(+)

base-commit: 30e8a06b64aa58a3990ba39cb5d09531e7d265e0

Signature · 1 line
2.50.1 (Apple Git-155)
#1 lore raw

PATCH v4 1/3 linux-headers: Add uapi header changes

This is a temporary patch intended for review and testing purposes only. It syncs the QEMU linux-headers with the kernel v5 changes that introduce the KVM_PPC_GET_COMPAT_CAPS ioctl for querying host CPU compatibility capabilities. The struct kvm_ppc_compat_caps places 'size' as the first field as required by copy_struct_from/to_user() versioning, with KVM_PPC_COMPAT_CAPS_SIZE_VER0 (24) defined as the frozen version-floor constant. The capability number is KVM_CAP_PPC_COMPAT_CAPS (250) and the ioctl is defined as _IO so the ioctl number remains stable if the struct grows in future versions. Capability bit definitions for POWER9, POWER10, and POWER11 compatibility modes are also included.

The actual header sync will be done via scripts/update-linux-headers.sh once the kernel changes [1] are merged upstream.

[1] https://lore.kernel.org/all/[email protected]/

---
Changes in v4:
- Changed ioctl definition from _IOWR to _IO to match kernel v5.
- The ioctl number must remain stable if the struct grows; the size field is read first by the kernel via get_user() before any copy_struct_from_user() call.

 linux-headers/asm-powerpc/kvm.h | 20 ++++++++++++++++++++
 linux-headers/linux/kvm.h       |  4 ++++
 2 files changed, 24 insertions(+)
linux-headers/asm-powerpc/kvm.h +20 -0
index 077c5437f521..e42c86481320 100644
@@ -437,6 +437,26 @@ struct kvm_ppc_cpu_char {
 	__u64	behaviour_mask;		/* valid bits in behaviour */
 };
 
+/* For KVM_PPC_GET_COMPAT_CAPS */
+struct kvm_ppc_compat_caps {
+	__u64	size;			/* Size of this structure */
+	__u64	flags;			/* Reserved for future use */
+	__u64	compat_capabilities;	/* Capabilities supported by the host */
+};
+#define KVM_PPC_COMPAT_CAPS_SIZE_VER0	24 /* sizeof first published struct */
+
+/*
+ * Capability bits for compat_capabilities field in kvm_ppc_compat_caps.
+ * These bits indicate which processor compatibility modes are supported.
+ */
+#define KVM_PPC_COMPAT_CAP_POWER9	(1ULL << 62)
+#define KVM_PPC_COMPAT_CAP_POWER10	(1ULL << 61)
+#define KVM_PPC_COMPAT_CAP_POWER11	(1ULL << 60)
+#define KVM_PPC_COMPAT_BITMASK		(KVM_PPC_COMPAT_CAP_POWER9 | \
+					 KVM_PPC_COMPAT_CAP_POWER10 | \
+					 KVM_PPC_COMPAT_CAP_POWER11)
+
+
 /*
  * Values for character and character_mask.
  * These are identical to the values used by H_GET_CPU_CHARACTERISTICS.
linux-headers/linux/kvm.h +4 -0
index 50e87ed72c20..6aa70b8d2e61 100644
@@ -985,6 +985,8 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_USER_OPEREXEC 246
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
+#define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_PPC_COMPAT_CAPS 250
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1329,6 +1331,8 @@ struct kvm_s390_keyop {
 /* Available with KVM_CAP_COUNTER_OFFSET */
 #define KVM_ARM_SET_COUNTER_OFFSET _IOW(KVMIO,  0xb5, struct kvm_arm_counter_offset)
 #define KVM_ARM_GET_REG_WRITABLE_MASKS _IOR(KVMIO,  0xb6, struct reg_mask_range)
+/* Available with KVM_CAP_PPC_COMPAT_CAPS */
+#define KVM_PPC_GET_COMPAT_CAPS	_IO(KVMIO,  0xe4)
 
 /* ioctl for vm fd */
 #define KVM_CREATE_DEVICE	  _IOWR(KVMIO,  0xe0, struct kvm_create_device)
Signature · 1 line
2.50.1 (Apple Git-155)
#2 lore raw

PATCH v4 2/3 target/ppc/kvm: Add support for querying host compatibility mode

In reply to Amit Machhiwal

Add infrastructure to query the host CPU compatibility mode via the KVM_PPC_GET_COMPAT_CAPS ioctl. This allows QEMU to determine if the host is running in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode).

The kvmppc_get_compat_caps() function issues the ioctl and returns the compat_capabilities bitmap. The kvm_ppc_host_compat_pvr() function derives the effective PVR from the bitmap using ctz64() to find the lowest set bit (highest supported compat level in IBM MSB-0 numbering).

The struct kvm_ppc_compat_caps places 'size' first and userspace sets it to sizeof(struct kvm_ppc_compat_caps) before calling the ioctl. The kernel uses copy_struct_from/to_user() to handle forward and backward ABI
compatibility: an older userspace with a smaller struct gets trailing fields zero-padded. When newer userspace passes a larger struct to an older kernel (usize > ksize), the kernel unconditionally returns -E2BIG and writes its own ksize back into host_compat.size. QEMU detects this, validates the returned size against KVM_PPC_COMPAT_CAPS_SIZE_VER0, and retries with that size.

Additionally, cas_check_pvr() in hw/ppc/spapr_hcall.c is updated to prevent fallback to raw mode when the host is running in compatibility mode. This ensures that nested guests cannot exceed the host's compatibility level. The call is guarded with kvm_enabled() since kvm_ppc_host_compat_pvr() invokes kvm_vm_ioctl() which dereferences kvm_state; without the guard, a TCG guest on a CONFIG_KVM=y binary would segfault.

If the capability is not supported or the query fails, the functions return 0, allowing fallback to existing behavior.

---
Changes in v4:
- Replaced fprintf(stderr, ...) with error_report() for error reporting in kvmppc_get_compat_caps().
- Replaced __builtin_ctzll() with the portable ctz64() helper.
- Simplified kvm_ppc_host_compat_pvr() to switch directly on the capability bit value (e.g. KVM_PPC_COMPAT_CAP_POWER9) instead of a derived index, removing the KVM_PPC_COMPAT_CAP_P*_IDX defines from kvm_ppc.h.
- Dropped the post-ioctl size mismatch check; it is not meaningful with the copy_struct_from/to_user() model.
- Added a function comment to kvm_ppc_host_compat_pvr().
- Guarded the kvm_ppc_host_compat_pvr() call in cas_check_pvr() with kvm_enabled() to prevent a segfault when QEMU is built with CONFIG_KVM=y but run as a TCG guest (kvm_state is NULL in that case).

 hw/ppc/spapr_hcall.c | 14 +++++++++
 target/ppc/kvm.c     | 75 ++++++++++++++++++++++++++++++++++++++++++++
 target/ppc/kvm_ppc.h |  7 +++++
 3 files changed, 96 insertions(+)
hw/ppc/spapr_hcall.c +14 -0
index 23bcd788daf6..708902934cff 100644
@@ -1136,6 +1136,7 @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat,
 {
     bool explicit_match = false; /* Matched the CPU's real PVR */
     uint32_t best_compat = 0;
+    uint32_t compat_host_pvr = 0;
     int i;
 
     /*
@@ -1163,6 +1164,19 @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat,
         }
     }
 
+    if (explicit_match && kvm_enabled()) {
+        compat_host_pvr = kvm_ppc_host_compat_pvr();
+        /*
+         * If the host is booted in a compatibility mode, do not try booting in
+         * the raw mode as it may allow KVM guests to boot with a higher CPU
+         * version compared to what host was booted with; which should not be
+         * allowed.
+         */
+        if (compat_host_pvr) {
+            explicit_match = false;
+        }
+    }
+
     *raw_mode_supported = explicit_match;
 
     /* Parsing finished */
target/ppc/kvm.c +75 -0

81 lines not on this page · raw message

target/ppc/kvm_ppc.h +7 -0
index 742881231e16..195dbaac5e17 100644
@@ -81,6 +81,8 @@ bool kvmppc_supports_ail_3(void);
 int kvmppc_enable_hwrng(void);
 int kvmppc_put_books_sregs(PowerPCCPU *cpu);
 PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void);
+
+uint32_t kvm_ppc_host_compat_pvr(void);
 void kvmppc_check_papr_resize_hpt(Error **errp);
 int kvmppc_resize_hpt_prepare(PowerPCCPU *cpu, target_ulong flags, int shift);
 int kvmppc_resize_hpt_commit(PowerPCCPU *cpu, target_ulong flags, int shift);
@@ -440,6 +442,11 @@ static inline PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void)
     return NULL;
 }
 
+static inline uint32_t kvm_ppc_host_compat_pvr(void)
+{
+    return 0;
+}
+
 static inline void kvmppc_check_papr_resize_hpt(Error **errp)
 {
 }
Signature · 1 line
2.50.1 (Apple Git-155)
#3 lore raw

PATCH v4 3/3 target/ppc/kvm: Use host compatibility mode for nested guests

In reply to Amit Machhiwal

On POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). When running nested KVM guests, QEMU currently derives the host CPU type using mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model used by QEMU and the compatibility mode enforced by the host, leading to guest boot failures such as "KVM-NESTEDv2: couldn't set guest wide elements".

Update kvm_ppc_get_host_cpu_class() to check if the host is running in a compatibility mode using kvm_ppc_host_compat_pvr(). When available, use the compatibility PVR instead of the raw hardware PVR when selecting the CPU model. This ensures that QEMU selects a CPU model consistent with the host compatibility mode, allowing nested guests to boot correctly.

The guard uses #if defined(TARGET_PPC64) to prevent build breakage on ppc32 targets where the POWER9/10/11 PVR constants are not defined.

---
Changes in v4:
- Updated commit log
- Added a #ifndef CONFIG_KVM compile-time assertion inside the TARGET_PPC64 guard as a sanity check, since kvm_ppc_get_host_cpu_class() is only ever built with CONFIG_KVM=y.

 target/ppc/kvm.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
target/ppc/kvm.c +12 -0
index b1b596f004ef..105b838cdef9 100644
@@ -2682,6 +2682,18 @@ PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void)
     uint32_t host_pvr = mfpvr();
     PowerPCCPUClass *pvr_pcc;
 
+#if defined(TARGET_PPC64)
+#ifndef CONFIG_KVM
+#error "CONFIG_KVM is not enabled"
+#endif
+    uint32_t compat_host_pvr;
+
+    compat_host_pvr = kvm_ppc_host_compat_pvr();
+    if (compat_host_pvr) {
+        host_pvr = compat_host_pvr;
+    }
+#endif /* TARGET_PPC64 */
+
     pvr_pcc = ppc_cpu_class_by_pvr(host_pvr);
     if (pvr_pcc == NULL) {
         pvr_pcc = ppc_cpu_class_by_pvr_mask(host_pvr);
Signature · 1 line
2.50.1 (Apple Git-155)