KVM: PPC: Handle CPU compatibility mode for nested guests
Subsystems (4)
v2 cover KVM: PPC: Handle CPU compatibility mode for nested guests
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
This creates a problem for nested virtualization. When booting a nested KVM guest (L2) inside a host KVM guest (L1) running in a compatibility mode, userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR and attempt to configure the nested guest accordingly. However, the L1 partition is constrained by the compatibility level negotiated with the hypervisor (L0), and requests exceeding that level are rejected, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series addresses the issue in two steps:
1. Detect and reject invalid compatibility requests early in KVM to avoid late failures.
2. Provide a mechanism for userspace to query the effective CPU compatibility modes supported by the host, so it can select an appropriate CPU model for nested guests.
To achieve this, the series introduces a new KVM capability and ioctl (KVM_CAP_PPC_COMPAT_CAPS / KVM_PPC_GET_COMPAT_CAPS) that expose the compatibility modes supported by the host.
The implementation supports both:
- PowerVM (nested API v2), where compatibility information is obtained via the H_GUEST_GET_CAPABILITIES hypercall.
- PowerNV (nested API v1), where compatibility is derived from the device tree ("cpu-version") representing the effective processor compatibility level.
This allows userspace (e.g., QEMU) to select a CPU model consistent with the host compatibility mode, avoiding mismatches and enabling successful nested guest boot.
Changes in v2:
- Squashed patches 2 and 3 from v1 (capability introduction and ioctl wiring) into a single patch for better logical grouping
- Changed kvm_ppc_compat_caps.flags from __u32 to __u64 for consistency and future extensibility
- Addressed other review comments
- Improved commit messages with clearer explanations of the changes
Patch summary:
[1/5] Validate arch_compat against host compatibility mode [2/5] Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl [3/5] Implement capability retrieval for PowerVM (API v2) [4/5] Add PowerNV support (API v1)
[5/5] Document the new ioctl
Tested on:
- Power11 pSeries LPAR in Power10 compatibility mode (nested API v2)
- Power10 PowerNV system (and QEMU TCG PowerNV 11) with nested virtualization (API v1) with various combinations of KVM L1/L2 guests in various supported compatibility modes.
With this series, nested guests boot successfully in configurations where they previously failed due to compatibility mismatches.
Related QEMU series:
A corresponding QEMU series adds support for querying and using these compatibility capabilities when configuring nested KVM guests:
https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/linuxppc-dev/[email protected]/
Amit Machhiwal (5):
KVM: PPC: Book3S HV: Validate arch_compat against host compatibility
mode
KVM: PPC: Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
KVM: PPC: Book3S HV: Implement compat CPU capability retrieval for KVM
on PowerVM
KVM: PPC: Book3S HV: Add support for compat CPU capabilities for KVM
on PowerNV
KVM: PPC: Document KVM_PPC_GET_COMPAT_CAPS ioctl
Documentation/virt/kvm/api.rst | 35 ++++++++++++++++ arch/powerpc/include/asm/kvm_ppc.h | 1 + arch/powerpc/include/uapi/asm/kvm.h | 6 +++ arch/powerpc/kvm/book3s_hv.c | 63 +++++++++++++++++++++++++++++ arch/powerpc/kvm/powerpc.c | 21 ++++++++++ include/uapi/linux/kvm.h | 4 ++ 6 files changed, 130 insertions(+)
base-commit: 1d5dcaa3bd65f2e8c9baa14a393d3a2dc5db7524
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 1/5 KVM: PPC: Book3S HV: Validate arch_compat against host compatibility mode
On IBM POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations. This becomes relevant for nested virtualization, where nested KVM guests may need to run with a specific processor compatibility level.
Currently, when running a nested KVM guest (L2) inside a Power11 pSeries logical partition (L1) booted in Power10 compatibility mode, the guest fails to boot while setting 'arch_compat'. This happens because the CPU class is derived from the hardware PVR (via mfspr()), which reflects the physical processor generation (Power11), rather than the effective compatibility mode (Power10).
As a result, userspace may request a Power11 arch_compat for the L2 guest. However, the L1 partition, running in Power10 compatibility, has only negotiated support up to Power10 with the Power Hypervisor (L0). When H_SET_STATE is invoked with a Power11 Logical PVR, the hypervisor rejects the request, leading to a late guest boot failure:
KVM-NESTEDv2: couldn't set guest wide elements
[..KVM reg dump..]
This situation should be detected earlier. Rejecting unsupported 'arch_compat' values in 'kvmppc_set_arch_compat()' avoids issuing an invalid H_SET_STATE hcall and provides a clearer failure mode.
Add a check to reject Power11 'arch_compat' requests when the host is running in Power10 compatibility mode, returning -EINVAL early instead of deferring the failure to the hypervisor.
--- arch/powerpc/kvm/book3s_hv.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/powerpc/kvm/book3s_hv.c +12 -0
@@ -446,7 +446,19 @@ static int kvmppc_set_arch_compat(struct kvm_vcpu *vcpu, u32 arch_compat) guest_pcr_bit =PCR_ARCH_300 ;break ;case PVR_ARCH_31 : + guest_pcr_bit =PCR_ARCH_31 ; +break ;case PVR_ARCH_31_P11 : +/* +* Need to check this for ISA 3.1, as Power10 and +* Power11 share the same PCR. For any subsequent ISA +* versions, this will be taken care of by the guest vs +* host PCR comparison below. +*/ +if ((PVR_ARCH_31 & cur_cpu_spec->pvr_mask) == + cur_cpu_spec->pvr_value) { +return -EINVAL ; + } guest_pcr_bit =PCR_ARCH_31 ;break ;default :
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 2/5 KVM: PPC: Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
In reply to Amit MachhiwalIntroduce a new capability and ioctl to expose CPU compatibility modes supported by the host processor for nested guests.
On IBM POWER systems, newer processor generations (N) can operate in compatibility modes corresponding to earlier generations, like (N-1) and (N-2). This is particularly relevant for nested virtualization, where nested KVM guests may need to run with a specific processor compatibility level.
Introduce KVM_CAP_PPC_COMPAT_CAPS capability and the corresponding KVM_PPC_GET_COMPAT_CAPS vm ioctl. The ioctl returns a bitmap describing the compatibility modes supported by the host in respective bit numbers, allowing userspace (e.g., QEMU) to select an appropriate compatibility level when configuring nested KVM guests.
The ioctl handling is added in kvm_arch_vm_ioctl() and retrieves host CPU compatibility capabilities via a PowerPC-specific backend implementation when available. If the capability is not supported, the ioctl returns success with no capabilities set, allowing userspace to fall back gracefully.
--- arch/powerpc/include/asm/kvm_ppc.h | 1 + arch/powerpc/include/uapi/asm/kvm.h | 6 ++++++ arch/powerpc/kvm/powerpc.c | 21 +++++++++++++++++++++ include/uapi/linux/kvm.h | 4 ++++ 4 files changed, 32 insertions(+)
arch/powerpc/include/asm/kvm_ppc.h +1 -0
@@ -319,6 +319,7 @@ struct kvmppc_ops {bool (*hash_v3_possible)(void );int (*create_vm_debugfs)(struct kvm *kvm);int (*create_vcpu_debugfs)(struct kvm_vcpu *vcpu,struct dentry *debugfs_dentry); +int (*get_compat_cpu_ver)(struct kvm_ppc_compat_caps *host_caps); };extern struct kvmppc_ops *kvmppc_hv_ops;
arch/powerpc/include/uapi/asm/kvm.h +6 -0
@@ -437,6 +437,12 @@ struct kvm_ppc_cpu_char {__u64 behaviour_mask;/* valid bits in behaviour */ }; +/* For KVM_PPC_GET_COMPAT_CAPS */ +struct kvm_ppc_compat_caps { +__u64 flags;/* Reserved for future use */ +__u64 compat_capabilities;/* Capabilities supported by the host */ +}; +/* * Values for character and character_mask. * These are identical to the values used by H_GET_CPU_CHARACTERISTICS.
arch/powerpc/kvm/powerpc.c +21 -0
@@ -697,6 +697,13 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) } }break ; +#if defined(CONFIG_KVM_BOOK3S_HV_POSSIBLE ) +case KVM_CAP_PPC_COMPAT_CAPS : + r =0 ; +if (kvmhv_on_pseries ()) + r =1 ; +break ; +#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ default : r =0 ;break ;
@@ -2463,6 +2470,20 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = kvm->arch.kvm_ops->svm_off (kvm);break ; } +case KVM_PPC_GET_COMPAT_CAPS : { +struct kvm_ppc_compat_caps host_caps; + + r =0 ; +memset (&host_caps,0 ,sizeof (host_caps)); +if (!kvm->arch.kvm_ops->get_compat_cpu_ver) + goto out; + + r = kvm->arch.kvm_ops->get_compat_cpu_ver (&host_caps); +if (!r &©_to_user (argp, &host_caps, +sizeof (host_caps))) + r = -EFAULT ; +break ; + }default : {struct kvm *kvm = filp->private_data; r = kvm->arch.kvm_ops->arch_vm_ioctl (filp, ioctl, arg);
include/uapi/linux/kvm.h +4 -0
@@ -996,6 +996,7 @@ struct kvm_enable_cap {#define KVM_CAP_S390_USER_OPEREXEC 246#define KVM_CAP_S390_KEYOP 247#define KVM_CAP_S390_VSIE_ESAMODE 248 +#define KVM_CAP_PPC_COMPAT_CAPS 249struct kvm_irq_routing_irqchip {__u32 irqchip;
@@ -1349,6 +1350,9 @@ struct kvm_s390_keyop {#define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)#define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr) +/* Available with KVM_CAP_PPC_COMPAT_CAPS */ +#define KVM_PPC_GET_COMPAT_CAPS _IOR(KVMIO, 0xe4, struct kvm_ppc_compat_caps) +/* * ioctls for vcpu fds */
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 3/5 KVM: PPC: Book3S HV: Implement compat CPU capability retrieval for KVM on PowerVM
In reply to Amit MachhiwalOn POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
When running nested KVM guests, QEMU derives the host CPU type using mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model selected by QEMU and the compatibility mode enforced by the host, leading to guest boot failures.
For example, booting a nested guest on a Power11 LPAR configured in Power10 compatibility mode fails with:
KVM-NESTEDv2: couldn't set guest wide elements
[..KVM reg dump..]
This occurs because QEMU selects a CPU model corresponding to the physical processor (via mfpvr()), while the host operates in a lower compatibility mode. As a result, KVM rejects the requested compatibility level during guest initialization.
Add support for retrieving host CPU compatibility capabilities for nested guests on PowerVM (PAPR nested API v2). The hypervisor provides the effective compatibility levels via the H_GUEST_GET_CAPABILITIES hcall, which reflects the processor modes negotiated between the Power hypervisor (L0) and the host partition (L1).
On pseries systems, obtain the capability bitmap using plpar_guest_get_capabilities() and return it via struct kvm_ppc_compat_caps. This information is then exposed to userspace through the KVM_PPC_GET_COMPAT_CAPS ioctl.
Hook the implementation into the Book3S HV kvmppc_ops so that it can be invoked by the generic KVM ioctl handling code.
--- arch/powerpc/kvm/book3s_hv.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/powerpc/kvm/book3s_hv.c +16 -0
@@ -6522,6 +6522,21 @@ static bool kvmppc_hash_v3_possible(void)return true; } + +static int kvmppc_get_compat_cpu_caps (struct kvm_ppc_compat_caps *host_caps) +{ +unsigned long capabilities =0 ; +long rc = -EINVAL ; + +if (kvmhv_on_pseries ()) { +if (kvmhv_is_nestedv2 ()) + rc =plpar_guest_get_capabilities (0 , &capabilities); + host_caps->compat_capabilities = capabilities; + } + +return rc; +} +static struct kvmppc_ops kvm_ops_hv = { .get_sregs = kvm_arch_vcpu_ioctl_get_sregs_hv, .set_sregs = kvm_arch_vcpu_ioctl_set_sregs_hv,
@@ -6564,6 +6579,7 @@ static struct kvmppc_ops kvm_ops_hv = { .hash_v3_possible = kvmppc_hash_v3_possible, .create_vcpu_debugfs = kvmppc_arch_create_vcpu_debugfs_hv, .create_vm_debugfs = kvmppc_arch_create_vm_debugfs_hv, + .get_compat_cpu_ver = kvmppc_get_compat_cpu_caps, };static int kvm_init_subcore_bitmap (void )
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 4/5 KVM: PPC: Book3S HV: Add support for compat CPU capabilities for KVM on PowerNV
In reply to Amit MachhiwalCurrently, when booting a compatibility-mode KVM guest (L1) on a PowerNV hypervisor (L0), the guest runs with the expected processor compatibility level. However, when booting a nested KVM guest (L2) inside the L1, QEMU derives the CPU model from the raw host PVR and attempts to run the nested guest at that level, instead of honoring the compatibility mode of the L1.
Extend host CPU compatibility capability reporting to support nested virtualization on PowerNV systems (PAPR nested API v1).
For nested API v2 (PowerVM), compatibility capabilities are obtained from the hypervisor via the H_GUEST_GET_CAPABILITIES hcall. This information is not available on PowerNV systems.
For nested API v1, derive the compatibility capabilities from the L1 guest by reading the "cpu-version" property from the device tree, which reflects the effective (logical) processor compatibility level. Map this value to the corresponding compatibility capability bitmap.
Introduce a helper to translate CPU version values into compatibility capability bits and integrate it into kvmppc_get_compat_cpu_caps().
This allows userspace to query host CPU compatibility modes on both PowerVM and PowerNV platforms via the KVM_PPC_GET_COMPAT_CAPS ioctl.
--- arch/powerpc/kvm/book3s_hv.c | 37 +++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
arch/powerpc/kvm/book3s_hv.c +36 -1
@@ -6522,15 +6522,50 @@ static bool kvmppc_hash_v3_possible(void)return true; } +static int kvmppc_map_compat_capabilities (const __be32 cpu_version, +unsigned long *capabilities) +{ +switch (cpu_version) { +case PVR_ARCH_31_P11 : + *capabilities |=H_GUEST_CAP_POWER11 ; +break ; +case PVR_ARCH_31 : + *capabilities |=H_GUEST_CAP_POWER10 ; +break ; +case PVR_ARCH_300 : + *capabilities |=H_GUEST_CAP_POWER9 ; +break ; +default : +return -EINVAL ; + } + +return 0 ; +}static int kvmppc_get_compat_cpu_caps (struct kvm_ppc_compat_caps *host_caps) { +struct device_node *np;unsigned long capabilities =0 ; +const __be32 *prop =NULL ;long rc = -EINVAL ; +u32 cpu_version;if (kvmhv_on_pseries ()) { -if (kvmhv_is_nestedv2 ()) +if (kvmhv_is_nestedv2 ()) { rc =plpar_guest_get_capabilities (0 , &capabilities); + }else { +for_each_node_by_type (np,"cpu" ) { + prop =of_get_property (np,"cpu-version" ,NULL ); +if (prop) { + cpu_version =be32_to_cpup (prop); +break ; + } + } +if (!prop) +return -EINVAL ; + rc =kvmppc_map_compat_capabilities (cpu_version, + &capabilities); + } host_caps->compat_capabilities = capabilities; }
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 5/5 KVM: PPC: Document KVM_PPC_GET_COMPAT_CAPS ioctl
In reply to Amit MachhiwalAdd documentation for the KVM_PPC_GET_COMPAT_CAPS ioctl to the KVM API documentation.
The ioctl exposes host processor compatibility modes supported for nested KVM guests on PowerPC systems.
--- Documentation/virt/kvm/api.rst | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
Documentation/virt/kvm/api.rst +35 -0
@@ -6555,6 +6555,41 @@ KVM_S390_KEYOP_SSKE .. _kvm_run: +4.145 KVM_PPC_GET_COMPAT_CAPS +----------------------------- +:Capability: KVM_CAP_PPC_COMPAT_CAPS +:Architectures: powerpc +:Type: vm ioctl +:Parameters: struct kvm_ppc_compat_caps (out) +:Returns: + 0 on successful completion, + -EFAULT if ``struct kvm_ppc_compat_caps`` cannot be written + +IBM POWER system server-based processors provide a compatibility mode feature +where an Nth generation processor can operate in modes consistent with earlier +generations such as (N-1) and (N-2). + +This ioctl provides userspace with information about the CPU compatibility modes +supported by the current host processor for booting the nested KVM guests on +PowerNV (KVM nested APIv1) and PowerVM (KVM nested APIv2) platforms. + +:: + + struct kvm_ppc_compat_caps { + __u64 flags; /* Reserved for future use */ + __u64 compat_capabilities; /* Capabilities supported by the host */ + }; + +The ``compat_capabilities`` bit field describes the processor compatibility +modes supported by the host. For example, the following bits indicate support +for specific processor modes. + +:: + +H_GUEST_CAP_POWER9 (bit 1): KVM guests can run in Power9 processor mode +H_GUEST_CAP_POWER10 (bit 2): KVM guests can run in Power10 processor mode +H_GUEST_CAP_POWER11 (bit 3): KVM guests can run in Power11 processor mode + 5. The kvm_run structure ========================
Signature · 1 line
2.50.1 (Apple Git-155)
Ritesh Harjani (IBM)
reviewerHi Amit,
Quoted 15 lines · Amit Machhiwal
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
This creates a problem for nested virtualization. When booting a nested KVM guest (L2) inside a host KVM guest (L1) running in a compatibility mode, userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR and attempt to configure the nested guest accordingly. However, the L1 partition is constrained by the compatibility level negotiated with the hypervisor (L0), and requests exceeding that level are rejected, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
Quoted 9 lines
This series addresses the issue in two steps:
1. Detect and reject invalid compatibility requests early in KVM to avoid late failures.
2. Provide a mechanism for userspace to query the effective CPU compatibility modes supported by the host, so it can select an appropriate CPU model for nested guests.
Do we really need to add a uapi change for this? Tools like Qemu can read the device tree info of the host, isn't it?
Quoted 11 lines
To achieve this, the series introduces a new KVM capability and ioctl (
KVM_CAP_PPC_COMPAT_CAPS/KVM_PPC_GET_COMPAT_CAPS) that expose the compatibility modes supported by the host.The implementation supports both:
- PowerVM (nested API v2), where compatibility information is obtained via the
H_GUEST_GET_CAPABILITIEShypercall.
- PowerNV (nested API v1), where compatibility is derived from the device tree ("cpu-version") representing the effective processor compatibility level.
See there you go, for PowerNV if this info is provided in the device tree, then Qemu could as well just read that info, no?
... yup, kvmppc_read_int_dt() can do that I guess.
So, my request is, can we look into this to see, if there is a possible alternative to this? maybe we already have a mechanism which Qemu could use to get this info already?
btw - I haven't given a full read of the patch series, but reading the cover letter, I felt we should atleast add this info to the cover letter on, why a uapi change is really needed here, why can't the existing alternatives work for us.
-ritesh
Quoted 5 lines
This allows userspace (e.g., QEMU) to select a CPU model consistent with the host compatibility mode, avoiding mismatches and enabling successful nested guest boot.
Hi Ritesh,
Thanks for taking a look at this series. Please find my comment inline below:
Quoted 25 lines · Ritesh Harjani
Hi Amit,
Quoted 20 lines · Amit Machhiwal
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
This creates a problem for nested virtualization. When booting a nested KVM guest (L2) inside a host KVM guest (L1) running in a compatibility mode, userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR and attempt to configure the nested guest accordingly. However, the L1 partition is constrained by the compatibility level negotiated with the hypervisor (L0), and requests exceeding that level are rejected, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series addresses the issue in two steps:
1. Detect and reject invalid compatibility requests early in KVM to avoid late failures.
Quoted 7 lines
Quoted 4 lines
2. Provide a mechanism for userspace to query the effective CPU compatibility modes supported by the host, so it can select an appropriate CPU model for nested guests.
Do we really need to add a uapi change for this? Tools like Qemu can read the device tree info of the host, isn't it?
While cpu-version is available in /proc/device-tree/cpus/<cpu#>/cpu-version on both L1 booted on PowerNV and PowerVM LPARs, I believe the UAPI change is still preferable for several reasons:
1. We would want to rely on the capabilities negotiated with pHYP (L0) in KVM on PowerVM case instead of device tree property. Also, the cpu-version property only depicts the current compat mode host (L1) is booted in but doesn't really point to what all compat modes are supported for the nested guest (L2).
2. procfs dependency: Not all systems run with procfs enabled (CONFIG_PROC_FS is optional). For example, minimal configurations (like buildroot) might disable it. The KVM ioctl works regardless of procfs availability since it accesses kernel data structures directly.
3. Kernel validation: The kernel validates and normalizes the compatibility information. For example, patch 1 adds validation logic that rejects invalid compatibility requests early. The ioctl ensures userspace gets validated, consistent data.
4. Abstraction & stability: While /proc/device-tree works today, it's an implementation detail. The UAPI provides a stable interface that won't break if the underlying mechanism changes.
5. Semantic clarity: KVM_PPC_GET_COMPAT_CAPS clearly expresses what compatibility modes can I use for KVM guests vs. parsing device tree which requires understanding the semantic meaning of cpu-version.
Quoted 13 lines
Quoted 5 lines
To achieve this, the series introduces a new KVM capability and ioctl (
KVM_CAP_PPC_COMPAT_CAPS/KVM_PPC_GET_COMPAT_CAPS) that expose the compatibility modes supported by the host.Quoted 7 lines
The implementation supports both:
- PowerVM (nested API v2), where compatibility information is obtained via the
H_GUEST_GET_CAPABILITIEShypercall.
- PowerNV (nested API v1), where compatibility is derived from the device tree ("cpu-version") representing the effective processor compatibility level.
Quoted 8 lines
See there you go, for PowerNV if this info is provided in the device tree, then Qemu could as well just read that info, no?
... yup,
kvmppc_read_int_dt()can do that I guess.So, my request is, can we look into this to see, if there is a possible alternative to this? maybe we already have a mechanism which Qemu could use to get this info already?
You're right that QEMU could read the device tree from procfs. We had discussed this approach internally as well. However, we believe the UAPI approach offers additional benefits and looks more robust and future proof as outlined above.
Quoted 5 lines
btw - I haven't given a full read of the patch series, but reading the cover letter, I felt we should atleast add this info to the cover letter on, why a uapi change is really needed here, why can't the existing alternatives work for us.
I have described above why we did the UAPI change for the approach followed in this series. Could you please suggest what else can be added?
Thanks,
Amit
Quoted 7 lines
-ritesh
Quoted 5 lines
This allows userspace (e.g., QEMU) to select a CPU model consistent with the host compatibility mode, avoiding mismatches and enabling successful nested guest boot.
Quoted 90 lines · Amit Machhiwal
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation.
This creates a problem for nested virtualization. When booting a nested KVM guest (L2) inside a host KVM guest (L1) running in a compatibility mode, userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR and attempt to configure the nested guest accordingly. However, the L1 partition is constrained by the compatibility level negotiated with the hypervisor (L0), and requests exceeding that level are rejected, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series addresses the issue in two steps:
1. Detect and reject invalid compatibility requests early in KVM to avoid late failures.
2. Provide a mechanism for userspace to query the effective CPU compatibility modes supported by the host, so it can select an appropriate CPU model for nested guests.
To achieve this, the series introduces a new KVM capability and ioctl (
KVM_CAP_PPC_COMPAT_CAPS/KVM_PPC_GET_COMPAT_CAPS) that expose the compatibility modes supported by the host.The implementation supports both:
- PowerVM (nested API v2), where compatibility information is obtained via the
H_GUEST_GET_CAPABILITIEShypercall.
- PowerNV (nested API v1), where compatibility is derived from the device tree ("cpu-version") representing the effective processor compatibility level.This allows userspace (e.g., QEMU) to select a CPU model consistent with the host compatibility mode, avoiding mismatches and enabling successful nested guest boot.
Changes in v2:
- Squashed patches 2 and 3 from v1 (capability introduction and ioctl wiring) into a single patch for better logical grouping
- Changed kvm_ppc_compat_caps.flags from__u32to__u64for consistency and future extensibility
- Addressed other review comments
- Improved commit messages with clearer explanations of the changesPatch summary:
[1/5] Validatearch_compatagainst host compatibility mode [2/5] IntroduceKVM_CAP_PPC_COMPAT_CAPSand wire up ioctl [3/5] Implement capability retrieval for PowerVM (API v2) [4/5] Add PowerNV support (API v1)
[5/5] Document the new ioctlTested on:
- Power11 pSeries LPAR in Power10 compatibility mode (nested API v2)
- Power10 PowerNV system (and QEMU TCG PowerNV 11) with nested virtualization (API v1) with various combinations of KVM L1/L2 guests in various supported compatibility modes.With this series, nested guests boot successfully in configurations where they previously failed due to compatibility mismatches.
Related QEMU series:
A corresponding QEMU series adds support for querying and using these compatibility capabilities when configuring nested KVM guests:https://lore.kernel.org/all/[email protected]/v1: https://lore.kernel.org/linuxppc-dev/[email protected]/
Amit Machhiwal (5):
KVM: PPC: Book3S HV: Validatearch_compatagainst host compatibility
mode
KVM: PPC: IntroduceKVM_CAP_PPC_COMPAT_CAPSand wire up ioctl
KVM: PPC: Book3S HV: Implement compat CPU capability retrieval for KVM
on PowerVM
KVM: PPC: Book3S HV: Add support for compat CPU capabilities for KVM
on PowerNV
KVM: PPC: DocumentKVM_PPC_GET_COMPAT_CAPSioctlDocumentation/virt/kvm/api.rst | 35 ++++++++++++++++ arch/powerpc/include/asm/kvm_ppc.h | 1 + arch/powerpc/include/uapi/asm/kvm.h | 6 +++ arch/powerpc/kvm/book3s_hv.c | 63 +++++++++++++++++++++++++++++ arch/powerpc/kvm/powerpc.c | 21 ++++++++++ include/uapi/linux/kvm.h | 4 ++ 6 files changed, 130 insertions(+)
Quoted 1 line
base-commit:
1d5dcaa3bd65f2e8c9baa14a393d3a2dc5db7524
Hi Amit,
I tried booting up a guest on P11 lpar booted with P10 compat mode applying your patch along with the qemu patch series and it has been working perfectly fine.
Host lscpu:
lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Model name: POWER10 (architected), altivec supported
Guest lscpu:
lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 10
On-line CPU(s) list: 0-9
Model name: POWER10 (architected), altivec supported
Feel free to add :
Thank you!
Anushree Mathur