KVM: x86: Convert nested ops to static calls
Subsystems (4)
Sean Christopherson
maintainerv2 cover KVM: x86: Convert nested ops to static calls
Convert nested_ops to use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.
v2: In patch 1, keep the get_evmcs_version != NULL check in kvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko]
v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (3):
KVM: x86: Reject nested CAP enablement if nested virtualization is
disabled
KVM: x86: Add static calls for nested virtualization ops
KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops
arch/x86/include/asm/kvm-x86-nested-ops.h | 36 +++++++++++++ arch/x86/include/asm/kvm_host.h | 14 ++++- arch/x86/kvm/hyperv.c | 6 +-- arch/x86/kvm/mmu.h | 5 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/svm.c | 4 +- arch/x86/kvm/vmx/main.c | 3 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/x86.c | 66 +++++++++++++++-------- arch/x86/kvm/x86.h | 2 +- 12 files changed, 104 insertions(+), 39 deletions(-) create mode 100644 arch/x86/include/asm/kvm-x86-nested-ops.h
base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 1/3 KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
Add a flag to explicitly track if nested virtualization is enabled, and use it enumerate that various nested CAPs are unsupported, and to reject enablement of said CAPs. When the nested ops hooks were moved to their own structure, KVM's NULL-by-default behavior was deliberately dropped, with the changelog asserting that all was well. That wasn't quite true; there is no danger to KVM, but now KVM is over-reporting support for KVM_CAP_NESTED_STATE and KVM_CAP_HYPERV_ENLIGHTENED_VMCS.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/hyperv.c | 3 ++- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/x86.c | 12 +++++++----- 5 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1906,6 +1906,8 @@ struct kvm_x86_ops { };struct kvm_x86_nested_ops { +bool enabled; +void (*leave_nested)(struct kvm_vcpu *vcpu);bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu,u8 vector,u32 error_code);
arch/x86/kvm/hyperv.c +2 -1
@@ -2787,7 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, };int i, nent =ARRAY_SIZE (cpuid_entries); -if (kvm_x86_ops.nested_ops->get_evmcs_version) +if (kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->get_evmcs_version) evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version (vcpu);if (cpuid->nent < nent)
arch/x86/kvm/svm/svm.c +1 -0
@@ -5646,6 +5646,7 @@ static __init int svm_hardware_setup(void)if (r)return r; } + svm_nested_ops.enabled = nested;/* * KVM's MMU doesn't support using 2-level paging for itself, and thus
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -8786,6 +8786,7 @@ __init int vmx_hardware_setup(void)if (r)return r; } + vmx_nested_ops.enabled = nested;kvm_set_posted_intr_wakeup_handler (pi_wakeup_handler);
arch/x86/kvm/x86.c +7 -5
@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->get_state ? + r = kvm_x86_ops.nested_ops->enabled ? kvm_x86_ops.nested_ops->get_state (NULL ,NULL ,0 ) :0 ;break ;#ifdef CONFIG_KVM_HYPERV
@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->enable_evmcs !=NULL ;break ;#endif case KVM_CAP_SMALLER_MAXPHYADDR :
@@ -3366,7 +3367,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,uint16_t vmcs_version;void __user *user_ptr; -if (!kvm_x86_ops.nested_ops->enable_evmcs) +if (!kvm_x86_ops.nested_ops->enabled || + !kvm_x86_ops.nested_ops->enable_evmcs)return -ENOTTY ; r = kvm_x86_ops.nested_ops->enable_evmcs (vcpu, &vmcs_version);if (!r) {
@@ -3732,7 +3734,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,u32 user_data_size; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->get_state) +if (!kvm_x86_ops.nested_ops->enabled)break ;BUILD_BUG_ON (sizeof (user_data_size) !=sizeof (user_kvm_nested_state->size));
@@ -3762,7 +3764,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,int idx; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->set_state) +if (!kvm_x86_ops.nested_ops->enabled)break ; r = -EFAULT ;
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
Quoted 112 lines · Sean Christopherson
Add a flag to explicitly track if nested virtualization is enabled, and use it enumerate that various nested CAPs are unsupported, and to reject enablement of said CAPs. When the nested ops hooks were moved to their own structure, KVM's NULL-by-default behavior was deliberately dropped, with the changelog asserting that all was well. That wasn't quite true; there is no danger to KVM, but now KVM is over-reporting support for
KVM_CAP_NESTED_STATEandKVM_CAP_HYPERV_ENLIGHTENED_VMCS.Fixes 33b22172452f ("KVM: x86: move nested-related kvm_x86_ops to a separate struct")Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/hyperv.c | 3 ++- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/x86.c | 12 +++++++----- 5 files changed, 13 insertions(+), 6 deletions(-)arch/x86/include/asm/kvm_host.h +2
-0@@ -1906,6 +1906,8 @@ struct kvm_x86_ops { };struct kvm_x86_nested_ops { +bool enabled; +void (*leave_nested)(struct kvm_vcpu *vcpu);bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu,u8 vector,u32 error_code);arch/x86/kvm/hyperv.c +2
-1@@ -2787,7 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, };int i, nent =ARRAY_SIZE (cpuid_entries); -if (kvm_x86_ops.nested_ops->get_evmcs_version) +if (kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->get_evmcs_version) evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version (vcpu);if (cpuid->nent < nent)arch/x86/kvm/svm/svm.c +1
-0@@ -5646,6 +5646,7 @@ static __init int svm_hardware_setup(void)if (r)return r; } + svm_nested_ops.enabled = nested;/* * KVM's MMU doesn't support using 2-level paging for itself, and thus arch/x86/kvm/vmx/vmx.c +1
-0@@ -8786,6 +8786,7 @@ __init int vmx_hardware_setup(void)if (r)return r; } + vmx_nested_ops.enabled = nested;kvm_set_posted_intr_wakeup_handler (pi_wakeup_handler);arch/x86/kvm/x86.c +6
-4@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->get_state ? + r = kvm_x86_ops.nested_ops->enabled ? kvm_x86_ops.nested_ops->get_state (NULL ,NULL ,0 ) :0 ;break ;#ifdef CONFIG_KVM_HYPERV @@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->enable_evmcs !=NULL ;break ;#endif case KVM_CAP_SMALLER_MAXPHYADDR :@@ -3366,7 +3367,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,uint16_t vmcs_version;void __user *user_ptr; -if (!kvm_x86_ops.nested_ops->enable_evmcs) +if (!kvm_x86_ops.nested_ops->enabled || + !kvm_x86_ops.nested_ops->enable_evmcs)return -ENOTTY ; r = kvm_x86_ops.nested_ops->enable_evmcs (vcpu, &vmcs_version);if (!r) {@@ -3732,7 +3734,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,u32 user_data_size; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->get_state) +if (!kvm_x86_ops.nested_ops->enabled)break ;BUILD_BUG_ON (sizeof (user_data_size) !=sizeof (user_kvm_nested_state->size));
arch/x86/kvm/x86.c long kvm_arch_vcpu_ioctl(struct file *filp, · line 3770
Quoted 8 lines
arch/x86/kvm/x86.c +1
-1@@ -3762,7 +3764,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,int idx; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->set_state) +if (!kvm_x86_ops.nested_ops->enabled)break ; r = -EFAULT ;
Signature · 1 line
Vitaly
Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
In reply to Sean ChristophersonQuoted 5 lines
but now KVM is over-reporting support for
KVM_CAP_NESTED_STATEandKVM_CAP_HYPERV_ENLIGHTENED_VMCS.
[...]
Quoted 8 lines
arch/x86/kvm/x86.c +1
-1@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->get_state ? + r = kvm_x86_ops.nested_ops->enabled ? kvm_x86_ops.nested_ops->get_state (NULL ,NULL ,0 ) :0 ;break ;
[...]
arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359
Quoted 9 lines
#ifdef
CONFIG_KVM_HYPERVarch/x86/kvm/x86.c +2
-1@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->enable_evmcs !=NULL ;
IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.
How about letting VMX/SVM code to explicitly clear the function pointer in xx_hardware_setup()?
Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of kvm_x86_nested_ops and the kvm_nested_ops_update() should treat all as either OPTIONAL or OPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return -EXXX but I guess we can live with that (kvm_x86_ops is similar too anyway) ?
Sean Christopherson
maintainerRe: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
Quoted 18 lines · Kai Huang
Quoted 5 lines
but now KVM is over-reporting support for
KVM_CAP_NESTED_STATEandKVM_CAP_HYPERV_ENLIGHTENED_VMCS.[...]
Quoted 8 lines
arch/x86/kvm/x86.c +1
-1@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->get_state ? + r = kvm_x86_ops.nested_ops->enabled ? kvm_x86_ops.nested_ops->get_state (NULL ,NULL ,0 ) :0 ;break ;[...]
Quoted 13 lines
arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359
Quoted 9 lines
#ifdef
CONFIG_KVM_HYPERVarch/x86/kvm/x86.c +2
-1@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->enable_evmcs !=NULL ;IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.
Because in most cases, the hooks are reachable if and only if nested virtualization is actively being used. Generally speaking, the only flows that need to check "enabled" are those that enumerate support to userspace.
Quoted 2 lines
How about letting VMX/SVM code to explicitly clear the function pointer in
xx_hardware_setup()?
The downside to that is approach is that all nested ops would become optional from the perspective of arch/x86/include/asm/kvm-x86-nested-ops.h, i.e. we'd lose the sanity checks provided by:
#define KVM_X86_NESTED_OP(func) \ WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)
We could probably come up with a way to make things work, but I think we'd end up with something a lot like the "enabled" flag.
Quoted 5 lines
Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of
kvm_x86_nested_opsand thekvm_nested_ops_update()should treat all as either OPTIONAL orOPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return-EXXXbut I guess we can live with that (kvm_x86_opsis similar too anyway) ?
I'm concerned this would actually be less robust, due to static calls being nops by default. If KVM screws up and invokes a "mandatory" nested op that has been nullified, exactly what will happen is harder to predict, e.g. kvm_translate_gpa() would probably return stack garbage?
Any such bug is highly unlikely to occur, but I don't want to sacrifice defense-in-depth without a clear benefit, and IMO avoiding the "enabled" flag wouldn't provide much benefit in practice.
Re: [PATCH v2 1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
Quoted 33 lines · Sean Christopherson
Quoted 18 lines · Kai Huang
Quoted 5 lines
but now KVM is over-reporting support for
KVM_CAP_NESTED_STATEandKVM_CAP_HYPERV_ENLIGHTENED_VMCS.[...]
Quoted 8 lines
arch/x86/kvm/x86.c +1
-1@@ -2346,7 +2346,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->get_state ? + r = kvm_x86_ops.nested_ops->enabled ? kvm_x86_ops.nested_ops->get_state (NULL ,NULL ,0 ) :0 ;break ;[...]
Quoted 13 lines
arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2359
Quoted 9 lines
#ifdef
CONFIG_KVM_HYPERVarch/x86/kvm/x86.c +2
-1@@ -2354,7 +2354,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_x86_ops.nested_ops->enabled && + kvm_x86_ops.nested_ops->enable_evmcs !=NULL ;IMHO an additional 'enabled' seems a bit redundant? And it seems there's no rule whether the common x86 code should check it before (checking and) making the call or not.
Quoted 3 lines
Because in most cases, the hooks are reachable if and only if nested virtualization is actively being used.
Yeah.
Quoted 2 lines
Generally speaking, the only flows that need to check "enabled" are those that enumerate support to userspace.
I see.
Quoted 10 lines
Quoted 3 lines
How about letting VMX/SVM code to explicitly clear the function pointer in
xx_hardware_setup()?The downside to that is approach is that all nested ops would become optional from the perspective of
arch/x86/include/asm/kvm-x86-nested-ops.h, i.e. we'd lose the sanity checks provided by:#define KVM_X86_NESTED_OP(func) \ WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)
Right.
Quoted 10 lines
We could probably come up with a way to make things work, but I think we'd end up with something a lot like the "enabled" flag.
Quoted 5 lines
Perhaps as an ultimate goal, I am not sure whether VMX/SVM should provide a all- NULL version of
kvm_x86_nested_opsand thekvm_nested_ops_update()should treat all as either OPTIONAL orOPTIONAL_RET0. There will be still call sites which needs to check the pointer is NULL or not and return-EXXXbut I guess we can live with that (kvm_x86_opsis similar too anyway) ?
Quoted 4 lines
I'm concerned this would actually be less robust, due to static calls being nops by default. If KVM screws up and invokes a "mandatory" nested op that has been nullified, exactly what will happen is harder to predict, e.g.
kvm_translate_gpa()would probably return stack garbage?
Oh ok. So I guess when nested is truly enabled, you only want to use OPTIONAL when it is truly optional, so that the KVM_X86_NESTED_OP() can catch if a NULL pointer is provided by vendor code.
Quoted 4 lines
Any such bug is highly unlikely to occur, but I don't want to sacrifice defense-in-depth without a clear benefit, and IMO avoiding the "enabled" flag wouldn't provide much benefit in practice.
OK agreed. Yeah being able to catch bug is more important :-)
I thought about in case of nested=false, and for the callbacks that can only be called when nested=true, we can replace the NULL pointer with some stub function which does WARN_ON_ONCE() first but maybe that is not worth doing.
Thanks for explanation.
Sean Christopherson
maintainerPATCH v2 2/3 KVM: x86: Add static calls for nested virtualization ops
In reply to Sean ChristophersonUse static calls to invoke nested virtualization ops, as many of the calls are in relatively hot paths when L2 is active, e.g. checking for events, and because there's no reason not use static calls these days.
Opportunistically use a RET0 static call for get_evmcs_version() instead of manually checking for a non-NULL vendor hook.
--- arch/x86/include/asm/kvm-x86-nested-ops.h | 36 +++++++++++++++ arch/x86/include/asm/kvm_host.h | 8 ++++ arch/x86/kvm/hyperv.c | 7 ++- arch/x86/kvm/mmu.h | 5 +-- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/x86.c | 53 +++++++++++++++-------- arch/x86/kvm/x86.h | 2 +- 7 files changed, 86 insertions(+), 27 deletions(-) create mode 100644 arch/x86/include/asm/kvm-x86-nested-ops.h
arch/x86/include/asm/kvm-x86-nested-ops.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(KVM_X86_NESTED_OP ) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL ) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL_RET0 ) +#error Missing one or more KVM_X86_NESTED_OP #defines +#else +/* +* KVM_X86_NESTED_OP() and KVM_X86_NESTED_OP_OPTIONAL() are used to help +* generate both DECLARE/DEFINE_STATIC_CALL() invocations and +* "static_call_update()" calls. +* +* KVM_X86_NESTED_OP_OPTIONAL() can be used for those functions that can have +* a NULL definition. KVM_X86_NESTED_OP_OPTIONAL_RET0() can be used likewise +* to make a definition optional, but in this case the default will +* be __static_call_return0. +*/ +KVM_X86_NESTED_OP(leave_nested) +KVM_X86_NESTED_OP(is_exception_vmexit) +KVM_X86_NESTED_OP(check_events) +KVM_X86_NESTED_OP_OPTIONAL_RET0(has_events) +KVM_X86_NESTED_OP(triple_fault) +KVM_X86_NESTED_OP(get_state) +KVM_X86_NESTED_OP(set_state) +KVM_X86_NESTED_OP(get_nested_state_pages) +KVM_X86_NESTED_OP_OPTIONAL_RET0(write_log_dirty) +KVM_X86_NESTED_OP(translate_nested_gpa) +#ifdef CONFIG_KVM_HYPERV +KVM_X86_NESTED_OP_OPTIONAL(enable_evmcs) +KVM_X86_NESTED_OP_OPTIONAL_RET0(get_evmcs_version) +KVM_X86_NESTED_OP(hv_inject_synthetic_vmexit_post_tlb_flush) +#endif +#endif + +#undef KVM_X86_NESTED_OP +#undef KVM_X86_NESTED_OP_OPTIONAL +#undef KVM_X86_NESTED_OP_OPTIONAL_RET0
arch/x86/include/asm/kvm_host.h +8 -0
@@ -1963,6 +1963,14 @@ extern struct kvm_x86_ops kvm_x86_ops;#define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP#include <asm/kvm-x86-ops.h> +#define kvm_nested_call (func) static_call(kvm_x86_nested_##func) + +#define KVM_X86_NESTED_OP (func) \ + DECLARE_STATIC_CALL(kvm_x86_nested_##func, *(((struct kvm_x86_nested_ops *)0)->func)); +#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP +#include <asm/kvm-x86-nested-ops.h> +#define __KVM_HAVE_ARCH_VM_ALLOCstatic inline struct kvm *kvm_arch_alloc_vm (void ) {
arch/x86/kvm/hyperv.c +3 -4
@@ -2406,7 +2406,7 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result) ret =kvm_skip_emulated_instruction (vcpu);if (tlb_lock_count) - kvm_x86_ops.nested_ops->hv_inject_synthetic_vmexit_post_tlb_flush (vcpu); +kvm_nested_call (hv_inject_synthetic_vmexit_post_tlb_flush)(vcpu);return ret; }
@@ -2787,9 +2787,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, };int i, nent =ARRAY_SIZE (cpuid_entries); -if (kvm_x86_ops.nested_ops->enabled && - kvm_x86_ops.nested_ops->get_evmcs_version) - evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version (vcpu); +if (kvm_x86_ops.nested_ops->enabled) + evmcs_ver =kvm_nested_call (get_evmcs_version)(vcpu);if (cpuid->nent < nent)return -E2BIG ;
arch/x86/kvm/mmu.h +2 -3
@@ -385,9 +385,8 @@ static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu, {if (!mmu_is_nested (vcpu) || w == &vcpu->arch.ngpa_walk)return gpa; -return kvm_x86_ops.nested_ops->translate_nested_gpa (vcpu, gpa, access, - exception, - pte_access); +return kvm_nested_call (translate_nested_gpa)(vcpu, gpa, access, + exception, pte_access); }static inline bool kvm_has_mirrored_tdp (const struct kvm *kvm)
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -235,7 +235,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm_vcpu *vcpu, !(pte & PT_GUEST_DIRTY_MASK)) {trace_kvm_mmu_set_dirty_bit (table_gfn, index,sizeof (pte));#if PTTYPE ==PTTYPE_EPT -if (kvm_x86_ops.nested_ops->write_log_dirty (vcpu, addr)) +if (kvm_nested_call (write_log_dirty)(vcpu, addr))return -EINVAL ;#endif pte |=PT_GUEST_DIRTY_MASK ;
arch/x86/kvm/x86.c +35 -18
140 lines not on this page · raw message
arch/x86/kvm/x86.h +1 -1
@@ -93,7 +93,7 @@ int kvm_check_nested_events(struct kvm_vcpu *vcpu);/* Forcibly leave the nested mode in cases like a vCPU reset */ static inline void kvm_leave_nested (struct kvm_vcpu *vcpu) { - kvm_x86_ops.nested_ops->leave_nested (vcpu); +kvm_nested_call (leave_nested)(vcpu); }/*
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Sean Christopherson
maintainerPATCH v2 3/3 KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops
In reply to Sean ChristophersonRework KVM's handling of per-vendor nested ops to copy the vendor's ops into a global structure owned by common x86, i.e. treat nested ops just like x86 and PMU ops. In addition to providing consistency across all ops implementations, making a copy of the ops prevents changes to the vendor's ops after KVM is initialized, i.e. guards against goofs where KVM *thinks* it is updating nested ops, but which won't take effect now that KVM uses static calls to invoke vendor hooks.
Ignoring the side effects of tagging {svm,vmx}_nested_ops as __initdata, no functional change intended.
--- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/hyperv.c | 2 +- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/svm.c | 3 +-- arch/x86/kvm/vmx/main.c | 3 +-- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 25 +++++++++++++------------ 7 files changed, 20 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
@@ -1842,8 +1842,6 @@ struct kvm_x86_ops {void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu); -const struct kvm_x86_nested_ops *nested_ops; -void (*vcpu_blocking)(struct kvm_vcpu *vcpu);void (*vcpu_unblocking)(struct kvm_vcpu *vcpu);
@@ -1939,6 +1937,7 @@ struct kvm_x86_init_ops {struct kvm_x86_ops *runtime_ops;struct kvm_pmu_ops *pmu_ops; +struct kvm_x86_nested_ops *nested_ops; };struct kvm_arch_async_pf {
@@ -1954,6 +1953,7 @@ extern bool __read_mostly enable_apicv;extern bool __read_mostly enable_ipiv;extern bool __read_mostly enable_device_posted_irqs;extern struct kvm_x86_ops kvm_x86_ops; +extern struct kvm_x86_nested_ops kvm_nested_ops__read_mostly ;#define kvm_x86_call (func) static_call(kvm_x86_##func)
arch/x86/kvm/hyperv.c +1 -1
@@ -2787,7 +2787,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, };int i, nent =ARRAY_SIZE (cpuid_entries); -if (kvm_x86_ops.nested_ops->enabled) +if (kvm_nested_ops.enabled) evmcs_ver =kvm_nested_call (get_evmcs_version)(vcpu);if (cpuid->nent < nent)
arch/x86/kvm/svm/nested.c +1 -1
@@ -2162,7 +2162,7 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,return w->gva_to_gpa (vcpu, w, gpa, access, exception); } -struct kvm_x86_nested_ops svm_nested_ops = { +struct kvm_x86_nested_ops svm_nested_ops__initdata = { .leave_nested = svm_leave_nested, .translate_nested_gpa = svm_translate_nested_gpa, .is_exception_vmexit = nested_svm_is_exception_vmexit,
arch/x86/kvm/svm/svm.c +1 -2
@@ -5426,8 +5426,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .check_intercept = svm_check_intercept, .handle_exit_irqoff = svm_handle_exit_irqoff, - .nested_ops = &svm_nested_ops, - .deliver_interrupt = svm_deliver_interrupt, .pi_update_irte = avic_pi_update_irte, .setup_mce = svm_setup_mce,
@@ -5773,6 +5771,7 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { .runtime_ops = &svm_x86_ops, .pmu_ops = &amd_pmu_ops, + .nested_ops = &svm_nested_ops, };static void __svm_exit (void )
arch/x86/kvm/vmx/main.c +1 -2
@@ -995,8 +995,6 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_cpu_dirty_logging = vt_op(update_cpu_dirty_logging), - .nested_ops = &vmx_nested_ops, - .pi_update_irte = vmx_pi_update_irte, .pi_start_bypass = vmx_pi_start_bypass,
@@ -1038,6 +1036,7 @@ struct kvm_x86_init_ops vt_init_ops __initdata = { .runtime_ops = &vt_x86_ops, .pmu_ops = &intel_pmu_ops, + .nested_ops = &vmx_nested_ops, }; static void __exit vt_exit(void)
arch/x86/kvm/vmx/nested.c +1 -1
@@ -7484,7 +7484,7 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,return w->gva_to_gpa (vcpu, w, gpa, access, exception); } -struct kvm_x86_nested_ops vmx_nested_ops = { +struct kvm_x86_nested_ops vmx_nested_ops__initdata = { .leave_nested = vmx_leave_nested, .translate_nested_gpa = vmx_translate_nested_gpa, .is_exception_vmexit = nested_vmx_is_exception_vmexit,
arch/x86/kvm/x86.c +13 -12
@@ -154,6 +154,7 @@ static int sync_regs(struct kvm_vcpu *vcpu);static DEFINE_MUTEX (vendor_module_lock);struct kvm_x86_ops kvm_x86_ops__read_mostly ; +struct kvm_x86_nested_ops kvm_nested_ops__read_mostly ;#define KVM_X86_OP (func) \ DEFINE_STATIC_CALL_NULL(kvm_x86_##func, \
@@ -2353,16 +2354,14 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST ;break ;case KVM_CAP_NESTED_STATE : - r = kvm_x86_ops.nested_ops->enabled ? -kvm_nested_call (get_state)(NULL ,NULL ,0 ) :0 ; + r = kvm_nested_ops.enabled ?kvm_nested_call (get_state)(NULL ,NULL ,0 ) :0 ;break ;#ifdef CONFIG_KVM_HYPERV case KVM_CAP_HYPERV_DIRECT_TLBFLUSH : r = kvm_x86_ops.enable_l2_tlb_flush !=NULL ;break ;case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : - r = kvm_x86_ops.nested_ops->enabled && - kvm_x86_ops.nested_ops->enable_evmcs !=NULL ; + r = kvm_nested_ops.enabled && kvm_nested_ops.enable_evmcs !=NULL ;break ;#endif case KVM_CAP_SMALLER_MAXPHYADDR :
@@ -3374,8 +3373,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,uint16_t vmcs_version;void __user *user_ptr; -if (!kvm_x86_ops.nested_ops->enabled || - !kvm_x86_ops.nested_ops->enable_evmcs) +if (!kvm_nested_ops.enabled || + !kvm_nested_ops.enable_evmcs)return -ENOTTY ; r =kvm_nested_call (enable_evmcs)(vcpu, &vmcs_version);if (!r) {
@@ -3741,7 +3740,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,u32 user_data_size; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->enabled) +if (!kvm_nested_ops.enabled)break ;BUILD_BUG_ON (sizeof (user_data_size) !=sizeof (user_kvm_nested_state->size));
@@ -3770,7 +3769,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,int idx; r = -EINVAL ; -if (!kvm_x86_ops.nested_ops->enabled) +if (!kvm_nested_ops.enabled)break ; r = -EFAULT ;
@@ -6913,13 +6912,15 @@ static void kvm_setup_efer_caps(void)static void kvm_nested_ops_update (const struct kvm_x86_nested_ops *nested_ops) { +memcpy (&kvm_nested_ops, nested_ops,sizeof (kvm_nested_ops)); +#define __KVM_X86_NESTED_OP (func) \ - static_call_update(kvm_x86_nested_##func, nested_ops->func); + static_call_update(kvm_x86_nested_##func, kvm_nested_ops.func);#define KVM_X86_NESTED_OP (func) \ - WARN_ON(!nested_ops->func); __KVM_X86_NESTED_OP(func) + WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func)#define KVM_X86_NESTED_OP_OPTIONAL __KVM_X86_NESTED_OP#define KVM_X86_NESTED_OP_OPTIONAL_RET0 (func) \ - static_call_update(kvm_x86_nested_##func, (void *)nested_ops->func ? : \ + static_call_update(kvm_x86_nested_##func, (void *)kvm_nested_ops.func ? : \ (void *)__static_call_return0);#include <asm/kvm-x86-nested-ops.h> #undef __KVM_X86_NESTED_OP
@@ -6940,7 +6941,7 @@ static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)#include <asm/kvm-x86-ops.h> #undef __KVM_X86_OP -kvm_nested_ops_update (kvm_x86_ops.nested_ops); +kvm_nested_ops_update (ops->nested_ops);kvm_pmu_ops_update (ops->pmu_ops); }
Signature · 1 line
2.55.0.rc0.799.gd6f94ed593-goog
Quoted 8 lines · Sean Christopherson
Convert
nested_opsto use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.v2: In patch 1, keep the
get_evmcs_version!= NULL check inkvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko][...]
Applied to kvm-x86 misc, thanks!
[1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
https://github.com/kvm-x86/linux/commit/b65699be2c60 [2/3] KVM: x86: Add static calls for nested virtualization ops https://github.com/kvm-x86/linux/commit/4b9819a50674 [3/3] KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops https://github.com/kvm-x86/linux/commit/2c56780d8ec7
Huang, Kai
Reviewed-byQuoted 10 lines · Sean Christopherson
Quoted 8 lines · Sean Christopherson
Convert
nested_opsto use static calls, mainly because the nested ops are bizarrely different than the x86 and PMU ops, and the kvm_x86_ops.nested_ops-> code is an eyesore (IMO). I haven't bothered to do any performance testing.v2: In patch 1, keep the
get_evmcs_version!= NULL check inkvm_get_hv_cpuid(),
then convert it to a RET0 call in patch 2. [Sashiko][...]
Quoted 9 lines
Applied to kvm-x86 misc, thanks!
[1/3] KVM: x86: Reject nested CAP enablement if nested virtualization is disabled
https://github.com/kvm-x86/linux/commit/b65699be2c60 [2/3] KVM: x86: Add static calls for nested virtualization ops https://github.com/kvm-x86/linux/commit/4b9819a50674 [3/3] KVM: x86: Move nested_ops out of kvm_x86_ops, to global kvm_nested_ops https://github.com/kvm-x86/linux/commit/2c56780d8ec7
I saw patch 2/3 has no tags from others.
FWIW, this series LGTM: