Skip to contentSkip to conversation

KVM: x86: Convert nested ops to static calls

PATCH Applied Sean Christopherson 3 participants 10 messages Jun 30 – Jul 14

Subsystems (4)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v2 cover KVM: x86: Convert nested ops to static calls

Convert nested_ops to use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.

v2: In patch 1, keep the get_evmcs_version != NULL check in kvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko]

v1: https://lore.kernel.org/all/[email protected]

Sean Christopherson (3):
KVM: x86: Reject nested CAP enablement if nested virtualization is
disabled
KVM: x86: Add static calls for nested virtualization ops
KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops

 arch/x86/include/asm/kvm-x86-nested-ops.h | 36 +++++++++++++
 arch/x86/include/asm/kvm_host.h           | 14 ++++-
 arch/x86/kvm/hyperv.c                     |  6 +--
 arch/x86/kvm/mmu.h                        |  5 +-
 arch/x86/kvm/mmu/paging_tmpl.h            |  2 +-
 arch/x86/kvm/svm/nested.c                 |  2 +-
 arch/x86/kvm/svm/svm.c                    |  4 +-
 arch/x86/kvm/vmx/main.c                   |  3 +-
 arch/x86/kvm/vmx/nested.c                 |  2 +-
 arch/x86/kvm/vmx/vmx.c                    |  1 +
 arch/x86/kvm/x86.c                        | 66 +++++++++++++++--------
 arch/x86/kvm/x86.h                        |  2 +-
 12 files changed, 104 insertions(+), 39 deletions(-)
 create mode 100644 arch/x86/include/asm/kvm-x86-nested-ops.h

base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370

Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
author maintainer
#1 lore raw

PATCH v2 1/3 KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

Add a flag to explicitly track if nested virtualization is enabled, and use it enumerate that various nested CAPs are unsupported, and to reject enablement of said CAPs. When the nested ops hooks were moved to their own structure, KVM's NULL-by-default behavior was deliberately dropped, with the changelog asserting that all was well. That wasn't quite true; there is no danger to KVM, but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.

Fixes 33b22172452f ("KVM: x86: move nested-related kvm_x86_ops to a separate struct")
---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/hyperv.c           |  3 ++-
 arch/x86/kvm/svm/svm.c          |  1 +
 arch/x86/kvm/vmx/vmx.c          |  1 +
 arch/x86/kvm/x86.c              | 12 +++++++-----
 5 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index b517257a6315..36de689a579c 100644
@@ -1906,6 +1906,8 @@ struct kvm_x86_ops {
 };
 
 struct kvm_x86_nested_ops {
+	bool enabled;
+
 	void (*leave_nested)(struct kvm_vcpu *vcpu);
 	bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu, u8 vector,
 				    u32 error_code);
arch/x86/kvm/hyperv.c +2 -1
index 1ee0d23f8949..7184622992d0 100644
@@ -2787,7 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 	};
 	int i, nent = ARRAY_SIZE(cpuid_entries);
 
-	if (kvm_x86_ops.nested_ops->get_evmcs_version)
+	if (kvm_x86_ops.nested_ops->enabled &&
+	    kvm_x86_ops.nested_ops->get_evmcs_version)
 		evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version(vcpu);
 
 	if (cpuid->nent < nent)
arch/x86/kvm/svm/svm.c +1 -0
index ef69a51ab27f..163440fca59f 100644
@@ -5646,6 +5646,7 @@ static __init int svm_hardware_setup(void)
 		if (r)
 			return r;
 	}
+	svm_nested_ops.enabled = nested;
 
 	/*
 	 * KVM's MMU doesn't support using 2-level paging for itself, and thus
arch/x86/kvm/vmx/vmx.c +1 -0
index 3681d565f177..ef9e1bc39edb 100644
@@ -8786,6 +8786,7 @@ __init int vmx_hardware_setup(void)
 		if (r)
 			return r;
 	}
+	vmx_nested_ops.enabled = nested;
 
 	kvm_set_posted_intr_wakeup_handler(pi_wakeup_handler);
 
arch/x86/kvm/x86.c +7 -5
index 0626e835e9eb..cd9a5ff47f6f 100644
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->get_state ?
+		r = kvm_x86_ops.nested_ops->enabled ?
 			kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0;
 		break;
 #ifdef CONFIG_KVM_HYPERV
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_x86_ops.nested_ops->enabled &&
+		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
 		break;
 #endif
 	case KVM_CAP_SMALLER_MAXPHYADDR:
@@ -3366,7 +3367,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 			uint16_t vmcs_version;
 			void __user *user_ptr;
 
-			if (!kvm_x86_ops.nested_ops->enable_evmcs)
+			if (!kvm_x86_ops.nested_ops->enabled ||
+			    !kvm_x86_ops.nested_ops->enable_evmcs)
 				return -ENOTTY;
 			r = kvm_x86_ops.nested_ops->enable_evmcs(vcpu, &vmcs_version);
 			if (!r) {
@@ -3732,7 +3734,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		u32 user_data_size;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->get_state)
+		if (!kvm_x86_ops.nested_ops->enabled)
 			break;
 
 		BUILD_BUG_ON(sizeof(user_data_size) != sizeof(user_kvm_nested_state->size));
@@ -3762,7 +3764,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		int idx;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->set_state)
+		if (!kvm_x86_ops.nested_ops->enabled)
 			break;
 
 		r = -EFAULT;
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
maintainer Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

Quoted 112 lines · Sean Christopherson

Add a flag to explicitly track if nested virtualization is enabled, and use it enumerate that various nested CAPs are unsupported, and to reject enablement of said CAPs. When the nested ops hooks were moved to their own structure, KVM's NULL-by-default behavior was deliberately dropped, with the changelog asserting that all was well. That wasn't quite true; there is no danger to KVM, but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.

Fixes 33b22172452f ("KVM: x86: move nested-related kvm_x86_ops to a separate struct")
---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/hyperv.c           |  3 ++-
 arch/x86/kvm/svm/svm.c          |  1 +
 arch/x86/kvm/vmx/vmx.c          |  1 +
 arch/x86/kvm/x86.c              | 12 +++++++-----
 5 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index b517257a6315..36de689a579c 100644
@@ -1906,6 +1906,8 @@ struct kvm_x86_ops {
 };
 
 struct kvm_x86_nested_ops {
+	bool enabled;
+
 	void (*leave_nested)(struct kvm_vcpu *vcpu);
 	bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu, u8 vector,
 				    u32 error_code);
arch/x86/kvm/hyperv.c +2 -1
index 1ee0d23f8949..7184622992d0 100644
@@ -2787,7 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 	};
 	int i, nent = ARRAY_SIZE(cpuid_entries);
 
-	if (kvm_x86_ops.nested_ops->get_evmcs_version)
+	if (kvm_x86_ops.nested_ops->enabled &&
+	    kvm_x86_ops.nested_ops->get_evmcs_version)
 		evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version(vcpu);
 
 	if (cpuid->nent < nent)
arch/x86/kvm/svm/svm.c +1 -0
index ef69a51ab27f..163440fca59f 100644
@@ -5646,6 +5646,7 @@ static __init int svm_hardware_setup(void)
 		if (r)
 			return r;
 	}
+	svm_nested_ops.enabled = nested;
 
 	/*
 	 * KVM's MMU doesn't support using 2-level paging for itself, and thus
arch/x86/kvm/vmx/vmx.c +1 -0
index 3681d565f177..ef9e1bc39edb 100644
@@ -8786,6 +8786,7 @@ __init int vmx_hardware_setup(void)
 		if (r)
 			return r;
 	}
+	vmx_nested_ops.enabled = nested;
 
 	kvm_set_posted_intr_wakeup_handler(pi_wakeup_handler);
 
arch/x86/kvm/x86.c +6 -4
index 0626e835e9eb..cd9a5ff47f6f 100644
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->get_state ?
+		r = kvm_x86_ops.nested_ops->enabled ?
 			kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0;
 		break;
 #ifdef CONFIG_KVM_HYPERV
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_x86_ops.nested_ops->enabled &&
+		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
 		break;
 #endif
 	case KVM_CAP_SMALLER_MAXPHYADDR:
@@ -3366,7 +3367,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 			uint16_t vmcs_version;
 			void __user *user_ptr;
 
-			if (!kvm_x86_ops.nested_ops->enable_evmcs)
+			if (!kvm_x86_ops.nested_ops->enabled ||
+			    !kvm_x86_ops.nested_ops->enable_evmcs)
 				return -ENOTTY;
 			r = kvm_x86_ops.nested_ops->enable_evmcs(vcpu, &vmcs_version);
 			if (!r) {
@@ -3732,7 +3734,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		u32 user_data_size;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->get_state)
+		if (!kvm_x86_ops.nested_ops->enabled)
 			break;
 
 		BUILD_BUG_ON(sizeof(user_data_size) != sizeof(user_kvm_nested_state->size));

arch/x86/kvm/x86.c long kvm_arch_vcpu_ioctl(struct file *filp, · line 3770

Quoted 8 lines
arch/x86/kvm/x86.c +1 -1
@@ -3762,7 +3764,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		int idx;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->set_state)
+		if (!kvm_x86_ops.nested_ops->enabled)
 			break;
 
 		r = -EFAULT;
Signature · 1 line
Vitaly
#3 lore raw

Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

In reply to Sean Christopherson
Quoted 5 lines

but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.

[...]

Quoted 8 lines
arch/x86/kvm/x86.c +1 -1
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->get_state ?
+		r = kvm_x86_ops.nested_ops->enabled ?
 			kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0;
 		break;

[...]

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359

Quoted 9 lines

#ifdef CONFIG_KVM_HYPERV

arch/x86/kvm/x86.c +2 -1
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_x86_ops.nested_ops->enabled &&
+		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
 

IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.

How about letting VMX/SVM code to explicitly clear the function pointer in xx_hardware_setup()?

Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of kvm_x86_nested_ops and the kvm_nested_ops_update() should treat all as either OPTIONAL or OPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return -EXXX but I guess we can live with that (kvm_x86_ops is similar too anyway) ?

author maintainer
#4 lore raw

Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

Quoted 18 lines · Kai Huang
Quoted 5 lines

but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.

[...]

Quoted 8 lines
arch/x86/kvm/x86.c +1 -1
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->get_state ?
+		r = kvm_x86_ops.nested_ops->enabled ?
 			kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0;
 		break;

[...]

Quoted 13 lines

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359

Quoted 9 lines

#ifdef CONFIG_KVM_HYPERV

arch/x86/kvm/x86.c +2 -1
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_x86_ops.nested_ops->enabled &&
+		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
 

IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.

Because in most cases, the hooks are reachable if and only if nested virtualization is actively being used. Generally speaking, the only flows that need to check "enabled" are those that enumerate support to userspace.

Quoted 2 lines

How about letting VMX/SVM code to explicitly clear the function pointer in xx_hardware_setup()?

The downside to that is approach is that all nested ops would become optional from the perspective of arch/x86/include/asm/kvm-x86-nested-ops.h, i.e. we'd lose the sanity checks provided by:

#define KVM_X86_NESTED_OP(func) \
	WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)

We could probably come up with a way to make things work, but I think we'd end up with something a lot like the "enabled" flag.

Quoted 5 lines

Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of kvm_x86_nested_ops and the kvm_nested_ops_update() should treat all as either OPTIONAL or OPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return -EXXX but I guess we can live with that (kvm_x86_ops is similar too anyway) ?

I'm concerned this would actually be less robust, due to static calls being nops by default. If KVM screws up and invokes a "mandatory" nested op that has been nullified, exactly what will happen is harder to predict, e.g. kvm_translate_gpa() would probably return stack garbage?

Any such bug is highly unlikely to occur, but I don't want to sacrifice defense-in-depth without a clear benefit, and IMO avoiding the "enabled" flag wouldn't provide much benefit in practice.

#5 lore raw

Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

Quoted 33 lines · Sean Christopherson
Quoted 18 lines · Kai Huang
Quoted 5 lines

but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.

[...]

Quoted 8 lines
arch/x86/kvm/x86.c +1 -1
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->get_state ?
+		r = kvm_x86_ops.nested_ops->enabled ?
 			kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0;
 		break;

[...]

Quoted 13 lines

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359

Quoted 9 lines

#ifdef CONFIG_KVM_HYPERV

arch/x86/kvm/x86.c +2 -1
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_x86_ops.nested_ops->enabled &&
+		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
 

IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.

Quoted 3 lines

Because in most cases, the hooks are reachable if and only if nested virtualization is actively being used.

Yeah.

Quoted 2 lines

Generally speaking, the only flows that need to check "enabled" are those that enumerate support to userspace.

I see.

Quoted 10 lines
Quoted 3 lines

How about letting VMX/SVM code to explicitly clear the function pointer in xx_hardware_setup()?

The downside to that is approach is that all nested ops would become optional from the perspective of arch/x86/include/asm/kvm-x86-nested-ops.h, i.e. we'd lose the sanity checks provided by:

#define KVM_X86_NESTED_OP(func) \
	WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)

Right.

Quoted 10 lines

We could probably come up with a way to make things work, but I think we'd end up with something a lot like the "enabled" flag.

Quoted 5 lines

Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of kvm_x86_nested_ops and the kvm_nested_ops_update() should treat all as either OPTIONAL or OPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return -EXXX but I guess we can live with that (kvm_x86_ops is similar too anyway) ?

Quoted 4 lines

I'm concerned this would actually be less robust, due to static calls being nops by default. If KVM screws up and invokes a "mandatory" nested op that has been nullified, exactly what will happen is harder to predict, e.g. kvm_translate_gpa() would probably return stack garbage?

Oh ok. So I guess when nested is truly enabled, you only want to use OPTIONAL when it is truly optional, so that the KVM_X86_NESTED_OP() can catch if a NULL pointer is provided by vendor code.

Quoted 4 lines

Any such bug is highly unlikely to occur, but I don't want to sacrifice defense-in-depth without a clear benefit, and IMO avoiding the "enabled" flag wouldn't provide much benefit in practice.

OK agreed. Yeah being able to catch bug is more important :-)

I thought about in case of nested=false, and for the callbacks that can only be called when nested=true, we can replace the NULL pointer with some stub function which does WARN_ON_ONCE() first but maybe that is not worth doing.

Thanks for explanation.

author maintainer
#6 lore raw

PATCH v2 2/3 KVM: x86: Add static calls for nested virtualization ops

In reply to Sean Christopherson

Use static calls to invoke nested virtualization ops, as many of the calls are in relatively hot paths when L2 is active, e.g. checking for events, and because there's no reason not use static calls these days.

Opportunistically use a RET0 static call for get_evmcs_version() instead of manually checking for a non-NULL vendor hook.

---
 arch/x86/include/asm/kvm-x86-nested-ops.h | 36 +++++++++++++++
 arch/x86/include/asm/kvm_host.h           |  8 ++++
 arch/x86/kvm/hyperv.c                     |  7 ++-
 arch/x86/kvm/mmu.h                        |  5 +--
 arch/x86/kvm/mmu/paging_tmpl.h            |  2 +-
 arch/x86/kvm/x86.c                        | 53 +++++++++++++++--------
 arch/x86/kvm/x86.h                        |  2 +-
 7 files changed, 86 insertions(+), 27 deletions(-)
 create mode 100644 arch/x86/include/asm/kvm-x86-nested-ops.h
arch/x86/include/asm/kvm-x86-nested-ops.h new file +36 -0
new file mode 100644
index 000000000000..4b1be5bcecaa
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#if !defined(KVM_X86_NESTED_OP) || \
+    !defined(KVM_X86_NESTED_OP_OPTIONAL) || \
+    !defined(KVM_X86_NESTED_OP_OPTIONAL_RET0)
+#error Missing one or more KVM_X86_NESTED_OP #defines
+#else
+/*
+ * KVM_X86_NESTED_OP() and KVM_X86_NESTED_OP_OPTIONAL() are used to help
+ * generate both DECLARE/DEFINE_STATIC_CALL() invocations and
+ * "static_call_update()" calls.
+ *
+ * KVM_X86_NESTED_OP_OPTIONAL() can be used for those functions that can have
+ * a NULL definition.  KVM_X86_NESTED_OP_OPTIONAL_RET0() can be used likewise
+ * to make a definition optional, but in this case the default will
+ * be __static_call_return0.
+ */
+KVM_X86_NESTED_OP(leave_nested)
+KVM_X86_NESTED_OP(is_exception_vmexit)
+KVM_X86_NESTED_OP(check_events)
+KVM_X86_NESTED_OP_OPTIONAL_RET0(has_events)
+KVM_X86_NESTED_OP(triple_fault)
+KVM_X86_NESTED_OP(get_state)
+KVM_X86_NESTED_OP(set_state)
+KVM_X86_NESTED_OP(get_nested_state_pages)
+KVM_X86_NESTED_OP_OPTIONAL_RET0(write_log_dirty)
+KVM_X86_NESTED_OP(translate_nested_gpa)
+#ifdef CONFIG_KVM_HYPERV
+KVM_X86_NESTED_OP_OPTIONAL(enable_evmcs)
+KVM_X86_NESTED_OP_OPTIONAL_RET0(get_evmcs_version)
+KVM_X86_NESTED_OP(hv_inject_synthetic_vmexit_post_tlb_flush)
+#endif
+#endif
+
+#undef KVM_X86_NESTED_OP
+#undef KVM_X86_NESTED_OP_OPTIONAL
+#undef KVM_X86_NESTED_OP_OPTIONAL_RET0
arch/x86/include/asm/kvm_host.h +8 -0
index 36de689a579c..25b1e3c1b344 100644
@@ -1963,6 +1963,14 @@ extern struct kvm_x86_ops kvm_x86_ops;
 #define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP
 #include <asm/kvm-x86-ops.h>
 
+#define kvm_nested_call(func) static_call(kvm_x86_nested_##func)
+
+#define KVM_X86_NESTED_OP(func) \
+	DECLARE_STATIC_CALL(kvm_x86_nested_##func, *(((struct kvm_x86_nested_ops *)0)->func));
+#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP
+#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP
+#include <asm/kvm-x86-nested-ops.h>
+
 #define __KVM_HAVE_ARCH_VM_ALLOC
 static inline struct kvm *kvm_arch_alloc_vm(void)
 {
arch/x86/kvm/hyperv.c +3 -4
index 7184622992d0..8b396ccd847e 100644
@@ -2406,7 +2406,7 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result)
 	ret = kvm_skip_emulated_instruction(vcpu);
 
 	if (tlb_lock_count)
-		kvm_x86_ops.nested_ops->hv_inject_synthetic_vmexit_post_tlb_flush(vcpu);
+		kvm_nested_call(hv_inject_synthetic_vmexit_post_tlb_flush)(vcpu);
 
 	return ret;
 }
@@ -2787,9 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 	};
 	int i, nent = ARRAY_SIZE(cpuid_entries);
 
-	if (kvm_x86_ops.nested_ops->enabled &&
-	    kvm_x86_ops.nested_ops->get_evmcs_version)
-		evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version(vcpu);
+	if (kvm_x86_ops.nested_ops->enabled)
+		evmcs_ver = kvm_nested_call(get_evmcs_version)(vcpu);
 
 	if (cpuid->nent < nent)
 		return -E2BIG;
arch/x86/kvm/mmu.h +2 -3
index c9f628b97dae..fa05ca2c8549 100644
@@ -385,9 +385,8 @@ static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu,
 {
 	if (!mmu_is_nested(vcpu) || w == &vcpu->arch.ngpa_walk)
 		return gpa;
-	return kvm_x86_ops.nested_ops->translate_nested_gpa(vcpu, gpa, access,
-							    exception,
-							    pte_access);
+	return kvm_nested_call(translate_nested_gpa)(vcpu, gpa, access,
+						     exception, pte_access);
 }
 
 static inline bool kvm_has_mirrored_tdp(const struct kvm *kvm)
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
index e73fc09ec4db..4ee7b03e762d 100644
@@ -235,7 +235,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm_vcpu *vcpu,
 				!(pte & PT_GUEST_DIRTY_MASK)) {
 			trace_kvm_mmu_set_dirty_bit(table_gfn, index, sizeof(pte));
 #if PTTYPE == PTTYPE_EPT
-			if (kvm_x86_ops.nested_ops->write_log_dirty(vcpu, addr))
+			if (kvm_nested_call(write_log_dirty)(vcpu, addr))
 				return -EINVAL;
 #endif
 			pte |= PT_GUEST_DIRTY_MASK;
arch/x86/kvm/x86.c +35 -18

140 lines not on this page · raw message

arch/x86/kvm/x86.h +1 -1
index 8ece468087a8..b510b631f0c5 100644
@@ -93,7 +93,7 @@ int kvm_check_nested_events(struct kvm_vcpu *vcpu);
 /* Forcibly leave the nested mode in cases like a vCPU reset */
 static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
 {
-	kvm_x86_ops.nested_ops->leave_nested(vcpu);
+	kvm_nested_call(leave_nested)(vcpu);
 }
 
 /*
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
author maintainer
#7 lore raw

PATCH v2 3/3 KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops

In reply to Sean Christopherson

Rework KVM's handling of per-vendor nested ops to copy the vendor's ops into a global structure owned by common x86, i.e. treat nested ops just like x86 and PMU ops. In addition to providing consistency across all ops implementations, making a copy of the ops prevents changes to the vendor's ops after KVM is initialized, i.e. guards against goofs where KVM *thinks* it is updating nested ops, but which won't take effect now that KVM uses static calls to invoke vendor hooks.

Ignoring the side effects of tagging {svm,vmx}_nested_ops as __initdata, no functional change intended.

---
 arch/x86/include/asm/kvm_host.h |  4 ++--
 arch/x86/kvm/hyperv.c           |  2 +-
 arch/x86/kvm/svm/nested.c       |  2 +-
 arch/x86/kvm/svm/svm.c          |  3 +--
 arch/x86/kvm/vmx/main.c         |  3 +--
 arch/x86/kvm/vmx/nested.c       |  2 +-
 arch/x86/kvm/x86.c              | 25 +++++++++++++------------
 7 files changed, 20 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
index 25b1e3c1b344..7e07a9c259d0 100644
@@ -1842,8 +1842,6 @@ struct kvm_x86_ops {
 
 	void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu);
 
-	const struct kvm_x86_nested_ops *nested_ops;
-
 	void (*vcpu_blocking)(struct kvm_vcpu *vcpu);
 	void (*vcpu_unblocking)(struct kvm_vcpu *vcpu);
 
@@ -1939,6 +1937,7 @@ struct kvm_x86_init_ops {
 
 	struct kvm_x86_ops *runtime_ops;
 	struct kvm_pmu_ops *pmu_ops;
+	struct kvm_x86_nested_ops *nested_ops;
 };
 
 struct kvm_arch_async_pf {
@@ -1954,6 +1953,7 @@ extern bool __read_mostly enable_apicv;
 extern bool __read_mostly enable_ipiv;
 extern bool __read_mostly enable_device_posted_irqs;
 extern struct kvm_x86_ops kvm_x86_ops;
+extern struct kvm_x86_nested_ops kvm_nested_ops __read_mostly;
 
 #define kvm_x86_call(func) static_call(kvm_x86_##func)
 
arch/x86/kvm/hyperv.c +1 -1
index 8b396ccd847e..b010dca37b9d 100644
@@ -2787,7 +2787,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 	};
 	int i, nent = ARRAY_SIZE(cpuid_entries);
 
-	if (kvm_x86_ops.nested_ops->enabled)
+	if (kvm_nested_ops.enabled)
 		evmcs_ver = kvm_nested_call(get_evmcs_version)(vcpu);
 
 	if (cpuid->nent < nent)
arch/x86/kvm/svm/nested.c +1 -1
index ba985a02208a..5f6d9971a3f2 100644
@@ -2162,7 +2162,7 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 	return w->gva_to_gpa(vcpu, w, gpa, access, exception);
 }
 
-struct kvm_x86_nested_ops svm_nested_ops = {
+struct kvm_x86_nested_ops svm_nested_ops __initdata = {
 	.leave_nested = svm_leave_nested,
 	.translate_nested_gpa = svm_translate_nested_gpa,
 	.is_exception_vmexit = nested_svm_is_exception_vmexit,
arch/x86/kvm/svm/svm.c +1 -2
index 163440fca59f..d3807f4abb49 100644
@@ -5426,8 +5426,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.check_intercept = svm_check_intercept,
 	.handle_exit_irqoff = svm_handle_exit_irqoff,
 
-	.nested_ops = &svm_nested_ops,
-
 	.deliver_interrupt = svm_deliver_interrupt,
 	.pi_update_irte = avic_pi_update_irte,
 	.setup_mce = svm_setup_mce,
@@ -5773,6 +5771,7 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {
 
 	.runtime_ops = &svm_x86_ops,
 	.pmu_ops = &amd_pmu_ops,
+	.nested_ops = &svm_nested_ops,
 };
 
 static void __svm_exit(void)
arch/x86/kvm/vmx/main.c +1 -2
index 83d9921277ea..04f986e3d439 100644
@@ -995,8 +995,6 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 
 	.update_cpu_dirty_logging = vt_op(update_cpu_dirty_logging),
 
-	.nested_ops = &vmx_nested_ops,
-
 	.pi_update_irte = vmx_pi_update_irte,
 	.pi_start_bypass = vmx_pi_start_bypass,
 
@@ -1038,6 +1036,7 @@ struct kvm_x86_init_ops vt_init_ops __initdata = {
 
 	.runtime_ops = &vt_x86_ops,
 	.pmu_ops = &intel_pmu_ops,
+	.nested_ops = &vmx_nested_ops,
 };
 
 static void __exit vt_exit(void)
arch/x86/kvm/vmx/nested.c +1 -1
index 0635e92471c8..cbb209d8a8f0 100644
@@ -7484,7 +7484,7 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 	return w->gva_to_gpa(vcpu, w, gpa, access, exception);
 }
 
-struct kvm_x86_nested_ops vmx_nested_ops = {
+struct kvm_x86_nested_ops vmx_nested_ops __initdata = {
 	.leave_nested = vmx_leave_nested,
 	.translate_nested_gpa = vmx_translate_nested_gpa,
 	.is_exception_vmexit = nested_vmx_is_exception_vmexit,
arch/x86/kvm/x86.c +13 -12
index 792afd15582b..5ba4aac37640 100644
@@ -154,6 +154,7 @@ static int sync_regs(struct kvm_vcpu *vcpu);
 static DEFINE_MUTEX(vendor_module_lock);
 
 struct kvm_x86_ops kvm_x86_ops __read_mostly;
+struct kvm_x86_nested_ops kvm_nested_ops __read_mostly;
 
 #define KVM_X86_OP(func)					     \
 	DEFINE_STATIC_CALL_NULL(kvm_x86_##func,			     \
@@ -2353,16 +2354,14 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 			r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST;
 		break;
 	case KVM_CAP_NESTED_STATE:
-		r = kvm_x86_ops.nested_ops->enabled ?
-			kvm_nested_call(get_state)(NULL, NULL, 0) : 0;
+		r = kvm_nested_ops.enabled ? kvm_nested_call(get_state)(NULL, NULL, 0) : 0;
 		break;
 #ifdef CONFIG_KVM_HYPERV
 	case KVM_CAP_HYPERV_DIRECT_TLBFLUSH:
 		r = kvm_x86_ops.enable_l2_tlb_flush != NULL;
 		break;
 	case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
-		r = kvm_x86_ops.nested_ops->enabled &&
-		    kvm_x86_ops.nested_ops->enable_evmcs != NULL;
+		r = kvm_nested_ops.enabled && kvm_nested_ops.enable_evmcs != NULL;
 		break;
 #endif
 	case KVM_CAP_SMALLER_MAXPHYADDR:
@@ -3374,8 +3373,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 			uint16_t vmcs_version;
 			void __user *user_ptr;
 
-			if (!kvm_x86_ops.nested_ops->enabled ||
-			    !kvm_x86_ops.nested_ops->enable_evmcs)
+			if (!kvm_nested_ops.enabled ||
+			    !kvm_nested_ops.enable_evmcs)
 				return -ENOTTY;
 			r = kvm_nested_call(enable_evmcs)(vcpu, &vmcs_version);
 			if (!r) {
@@ -3741,7 +3740,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		u32 user_data_size;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->enabled)
+		if (!kvm_nested_ops.enabled)
 			break;
 
 		BUILD_BUG_ON(sizeof(user_data_size) != sizeof(user_kvm_nested_state->size));
@@ -3770,7 +3769,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		int idx;
 
 		r = -EINVAL;
-		if (!kvm_x86_ops.nested_ops->enabled)
+		if (!kvm_nested_ops.enabled)
 			break;
 
 		r = -EFAULT;
@@ -6913,13 +6912,15 @@ static void kvm_setup_efer_caps(void)
 
 static void kvm_nested_ops_update(const struct kvm_x86_nested_ops *nested_ops)
 {
+	memcpy(&kvm_nested_ops, nested_ops, sizeof(kvm_nested_ops));
+
 #define __KVM_X86_NESTED_OP(func) \
-	static_call_update(kvm_x86_nested_##func, nested_ops->func);
+	static_call_update(kvm_x86_nested_##func, kvm_nested_ops.func);
 #define KVM_X86_NESTED_OP(func) \
-	WARN_ON(!nested_ops->func); __KVM_X86_NESTED_OP(func)
+	WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)
 #define KVM_X86_NESTED_OP_OPTIONAL __KVM_X86_NESTED_OP
 #define KVM_X86_NESTED_OP_OPTIONAL_RET0(func) \
-	static_call_update(kvm_x86_nested_##func, (void *)nested_ops->func ? : \
+	static_call_update(kvm_x86_nested_##func, (void *)kvm_nested_ops.func ? : \
 						  (void *)__static_call_return0);
 #include <asm/kvm-x86-nested-ops.h>
 #undef __KVM_X86_NESTED_OP
@@ -6940,7 +6941,7 @@ static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)
 #include <asm/kvm-x86-ops.h>
 #undef __KVM_X86_OP
 
-	kvm_nested_ops_update(kvm_x86_ops.nested_ops);
+	kvm_nested_ops_update(ops->nested_ops);
 
 	kvm_pmu_ops_update(ops->pmu_ops);
 }
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
author maintainer Applied
#8 lore raw
In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson

Convert nested_ops to use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.

v2: In patch 1, keep the get_evmcs_version != NULL check in kvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko]

[...]

Applied to kvm-x86 misc, thanks!

[1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

      https://github.com/kvm-x86/linux/commit/b65699be2c60
[2/3] KVM: x86: Add static calls for nested virtualization ops
      https://github.com/kvm-x86/linux/commit/4b9819a50674
[3/3] KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops
      https://github.com/kvm-x86/linux/commit/2c56780d8ec7

--
https://github.com/kvm-x86/linux/tree/next

Reviewed-by
#9 lore raw
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Sean Christopherson

Convert nested_ops to use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.

v2: In patch 1, keep the get_evmcs_version != NULL check in kvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko]

[...]

Quoted 9 lines

Applied to kvm-x86 misc, thanks!

[1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled

      https://github.com/kvm-x86/linux/commit/b65699be2c60
[2/3] KVM: x86: Add static calls for nested virtualization ops
      https://github.com/kvm-x86/linux/commit/4b9819a50674
[3/3] KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops
      https://github.com/kvm-x86/linux/commit/2c56780d8ec7

I saw patch 2/3 has no tags from others.

FWIW, this series LGTM: