ppc/kvm: Handle CPU compatibility mode correctly for nested guests
cover ppc/kvm: Handle CPU compatibility mode correctly for nested guests
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
This creates issues for nested virtualization. When booting a nested KVM guest, QEMU may derive the CPU model from the raw hardware PVR and attempt to configure the guest accordingly. However, the host is constrained by the compatibility level negotiated with the hypervisor, and requests exceeding that level are rejected by KVM, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series addresses the issue in two ways:
1. Do not silently fall back to raw mode when KVM rejects a requested compatibility level during CAS. Instead, propagate the error so invalid configurations are visible and fail early.
2. Query the effective CPU compatibility modes supported by the host via KVM and use this information to select an appropriate CPU model for nested guests.
With these changes, QEMU avoids masking KVM errors and ensures that nested guests are configured with CPU models consistent with the host compatibility mode, allowing them to boot correctly.
Patch summary:
[1/2] hw/ppc/spapr: Do not fallback to raw mode when KVM rejects compat [2/2] target/ppc/kvm: Use host compatibility mode for nested guests
Tested on:
- Power11 pSeries LPAR in Power10 compatibility mode
- Power10 PowerNV and QEMU PowerNV 11 TCG L0 host
Note: The corresponding Linux patches have been posted [1]
[1] https://lore.kernel.org/all/[email protected]/
Amit Machhiwal (2):
hw/ppc/spapr: Do not fallback to raw mode when KVM rejects compat target/ppc/kvm: Use host compatibility mode for nested guests
hw/ppc/spapr_hcall.c | 9 ++++++++ target/ppc/kvm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+)
base-commit: 759c456b1d22fe4083c8b384da27d3f56fd53f82 --
2.50.1
PATCH 1/2 hw/ppc/spapr: Do not fallback to raw mode when KVM rejects compat
During H_CLIENT_ARCHITECTURE_SUPPORT, QEMU attempts to set the requested CPU compatibility mode via ppc_set_compat_all(). If this fails, the existing code may fall back to raw mode when supported by the guest.
The current logic treats all failures uniformly and tried to fall back to raw mode whenever possible. As a result, errors from KVM (e.g. kvmppc_set_compat() returning -EINVAL) are silently masked, making debugging difficult and allowing the guest to proceed with an unsupported or incompatible configuration.
When running with KVM, such failures indicate that the requested compatibility level is not supported by the host. Do not fallback in this case; instead, report the error and fail the CAS negotiation.
This makes the failure visible to the user and ensures that invalid compatibility requests are rejected early rather than being hidden by fallback behavior. For example, the following error is now reported:
qemu-system-ppc64: Unable to set CPU compatibility mode in KVM: Invalid argument
--- hw/ppc/spapr_hcall.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/ppc/spapr_hcall.c +9 -0
@@ -1204,6 +1204,15 @@ target_ulong do_client_architecture_support(PowerPCCPU *cpu,Error *local_err =NULL ;if (ppc_set_compat_all (cas_pvr, &local_err) <0 ) { +/* +* If KVM rejected the compat mode, do not fallback. This indicates +* the host cannot support the requested level. +*/ +if (kvm_enabled ()) { +error_report_err (local_err); +return H_HARDWARE ; + } +/* We fail to set compat mode (likely because running with KVM PR), * but maybe we can fallback to raw mode if the guest supports it. */
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH 2/2 target/ppc/kvm: Use host compatibility mode for nested guests
In reply to Amit MachhiwalOn POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
When running nested KVM guests, QEMU currently derives the host CPU type using mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model used by QEMU and the compatibility mode enforced by the host, leading to guest boot failures.
In particular, booting a nested guest on a Power11 LPAR configured in Power10 compatibility mode fails with errors such as:
KVM-NESTEDv2: couldn't set guest wide elements
This occurs because QEMU selects a CPU model based on the physical processor version, while the host operates in a lower compatibility mode. As a result, KVM rejects the requested compatibility level during guest initialization.
Add support for querying host compatibility capabilities via the KVM_PPC_GET_COMPAT_CAPS ioctl and derive the effective PVR based on the compatibility mode reported by KVM. When available, use this compatibility PVR instead of the raw hardware PVR when selecting the CPU model.
If the capability is not supported or the query fails, fall back to the existing behavior.
This ensures that QEMU selects a CPU model consistent with the host compatibility mode, allowing nested guests to boot correctly.
--- target/ppc/kvm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)
target/ppc/kvm.c +52 -0
@@ -2602,11 +2602,63 @@ bool kvmppc_supports_ail_3(void)return cap_ail_mode_3; } +static target_ulong kvmppc_get_compat_caps (void ) +{ +struct kvm_ppc_compat_caps host_compat; +target_ulong host_caps; +int ret; + +if (!kvm_check_extension (kvm_state,KVM_CAP_PPC_COMPAT_CAPS )) { +return 0 ; + } + + ret =kvm_vm_ioctl (kvm_state,KVM_PPC_GET_COMPAT_CAPS , &host_compat); +if (ret <0 ) { +fprintf (stderr,"KVM: failed to get host capabilities\n" ); +return 0 ; + } + + host_caps = host_compat.compat_capabilities; +return host_caps; +} + +static uint32_t kvm_ppc_host_compat_pvr (void ) +{ +uint32_t compat_host_pvr =0 ; +int cap_idx =0 ; +target_ulong host_caps =kvmppc_get_compat_caps (); + +if (host_caps) { + cap_idx =63 -__builtin_ctzll (host_caps); +switch (cap_idx) { +case H_GUEST_CAP_P9_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER9_DD22 ; +break ; +case H_GUEST_CAP_P10_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER10_DD20 ; +break ; +case H_GUEST_CAP_P11_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER11_DD20 ; +break ; +default : +break ; + } + } + +return compat_host_pvr; +} +PowerPCCPUClass *kvm_ppc_get_host_cpu_class (void ) {uint32_t host_pvr =mfpvr (); +uint32_t compat_host_pvr;PowerPCCPUClass *pvr_pcc; + compat_host_pvr =kvm_ppc_host_compat_pvr (); +if (compat_host_pvr) { + host_pvr = compat_host_pvr; + } + pvr_pcc =ppc_cpu_class_by_pvr (host_pvr);if (pvr_pcc ==NULL ) { pvr_pcc =ppc_cpu_class_by_pvr_mask (host_pvr);
Signature · 1 line
2.50.1 (Apple Git-155)
Re: [PATCH 2/2] target/ppc/kvm: Use host compatibility mode for nested guests
I found a compilation issue with this patch when building for ppc32. The host PVR constants (CPU_POWERPC_POWER9_DD22, etc.) are only defined under TARGET_PPC64, and since POWER9/10/11 compatibility modes are 64-bit only, the code should be guarded with #if defined(TARGET_PPC64). I'll send v2 shortly with this fix.
~Amit
Quoted 108 lines · Amit Machhiwal
On POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
When running nested KVM guests, QEMU currently derives the host CPU type using
mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model used by QEMU and the compatibility mode enforced by the host, leading to guest boot failures.In particular, booting a nested guest on a Power11 LPAR configured in Power10 compatibility mode fails with errors such as:
KVM-NESTEDv2: couldn't set guest wide elements
This occurs because QEMU selects a CPU model based on the physical processor version, while the host operates in a lower compatibility mode. As a result, KVM rejects the requested compatibility level during guest initialization.
Add support for querying host compatibility capabilities via the
KVM_PPC_GET_COMPAT_CAPSioctl and derive the effective PVR based on the compatibility mode reported by KVM. When available, use this compatibility PVR instead of the raw hardware PVR when selecting the CPU model.If the capability is not supported or the query fails, fall back to the existing behavior.
This ensures that QEMU selects a CPU model consistent with the host compatibility mode, allowing nested guests to boot correctly.
Signed-off-by Amit Machhiwal <[email protected]>--- target/ppc/kvm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)target/ppc/kvm.c +52
-0@@ -2602,11 +2602,63 @@ bool kvmppc_supports_ail_3(void)return cap_ail_mode_3; } +static target_ulong kvmppc_get_compat_caps (void ) +{ +struct kvm_ppc_compat_caps host_compat; +target_ulong host_caps; +int ret; + +if (!kvm_check_extension (kvm_state,KVM_CAP_PPC_COMPAT_CAPS )) { +return 0 ; + } + + ret =kvm_vm_ioctl (kvm_state,KVM_PPC_GET_COMPAT_CAPS , &host_compat); +if (ret <0 ) { +fprintf (stderr,"KVM: failed to get host capabilities\n" ); +return 0 ; + } + + host_caps = host_compat.compat_capabilities; +return host_caps; +} + +static uint32_t kvm_ppc_host_compat_pvr (void ) +{ +uint32_t compat_host_pvr =0 ; +int cap_idx =0 ; +target_ulong host_caps =kvmppc_get_compat_caps (); + +if (host_caps) { + cap_idx =63 -__builtin_ctzll (host_caps); +switch (cap_idx) { +case H_GUEST_CAP_P9_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER9_DD22 ; +break ; +case H_GUEST_CAP_P10_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER10_DD20 ; +break ; +case H_GUEST_CAP_P11_MODE_BMAP : + compat_host_pvr =CPU_POWERPC_POWER11_DD20 ; +break ; +default : +break ; + } + } + +return compat_host_pvr; +} +PowerPCCPUClass *kvm_ppc_get_host_cpu_class (void ) {uint32_t host_pvr =mfpvr (); +uint32_t compat_host_pvr;PowerPCCPUClass *pvr_pcc; + compat_host_pvr =kvm_ppc_host_compat_pvr (); +if (compat_host_pvr) { + host_pvr = compat_host_pvr; + } + pvr_pcc =ppc_cpu_class_by_pvr (host_pvr);if (pvr_pcc ==NULL ) { pvr_pcc =ppc_cpu_class_by_pvr_mask (host_pvr);Signature · 1 line
2.50.1 (Apple Git-155)