KVM: x86/mmu: Avoid retpoline on ->page_fault() with TDP
PATCH 1/1 KVM: x86/mmu: Avoid retpoline on ->page_fault() with TDP
Wrap calls to ->page_fault() with a small shim to directly invoke the TDP fault handler when the kernel is using retpolines and TDP is being used. Denote the TDP fault handler by nullifying mmu->page_fault, and annotate the TDP path as likely to coerce the compiler into preferring the TDP path.
Rename tdp_page_fault() to kvm_tdp_page_fault() as it's exposed outside of mmu.c to allow inlining the shim.
---
Haven't done any performance testing, this popped into my head when mucking with the 5-level page table crud as an easy way to shave cycles in the happy path.
arch/x86/kvm/mmu.h | 13 +++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++------ arch/x86/kvm/x86.c | 2 +- 3 files changed, 24 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu.h +13 -0
@@ -102,6 +102,19 @@ static inline void kvm_mmu_load_cr3(struct kvm_vcpu *vcpu) kvm_get_active_pcid(vcpu)); } +int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +bool prefault); + +static inline int kvm_mmu_do_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa, +u32 err,bool prefault) +{ +#ifdef CONFIG_RETPOLINE +if (likely (!vcpu->arch.mmu->page_fault)) +return kvm_tdp_page_fault (vcpu, cr2_or_gpa, err, prefault); +#endif +return vcpu->arch.mmu->page_fault (vcpu, cr2_or_gpa, err, prefault); +} +/* * Currently, we have two sorts of write-protection, a) the first one * write-protects guest page to sync the guest modification, b) another one is
arch/x86/kvm/mmu/mmu.c +10 -6
@@ -4219,8 +4219,8 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, }EXPORT_SYMBOL_GPL (kvm_handle_page_fault); -static int tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, -bool prefault) +int kvm_tdp_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code, +bool prefault) {int max_level;
@@ -4925,7 +4925,12 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)return ; context->mmu_role.as_u64 = new_role.as_u64; - context->page_fault = tdp_page_fault; +#ifdef CONFIG_RETPOLINE +/* Nullify ->page_fault() to use direct kvm_tdp_page_fault() call. */ + context->page_fault =NULL ; +#else + context->page_fault = kvm_tdp_page_fault; +#endif context->sync_page = nonpaging_sync_page; context->invlpg = nonpaging_invlpg; context->update_pte = nonpaging_update_pte;
@@ -5436,9 +5441,8 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code, }if (r ==RET_PF_INVALID ) { - r = vcpu->arch.mmu->page_fault (vcpu, cr2_or_gpa, -lower_32_bits (error_code), - false); + r =kvm_mmu_do_page_fault (vcpu, cr2_or_gpa, +lower_32_bits (error_code), false);WARN_ON (r ==RET_PF_INVALID ); }
arch/x86/kvm/x86.c +1 -1
@@ -10182,7 +10182,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) work->arch.cr3 != vcpu->arch.mmu->get_cr3(vcpu)) return; - vcpu->arch.mmu->page_fault (vcpu, work->cr2_or_gpa,0 , true); +kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); }static inline u32 kvm_async_pf_hash_fn (gfn_t gfn)
Signature · 1 line
2.24.1
Quoted 11 lines · Sean Christopherson
Wrap calls to ->page_fault() with a small shim to directly invoke the TDP fault handler when the kernel is using retpolines and TDP is being used. Denote the TDP fault handler by nullifying
mmu->page_fault, and annotate the TDP path as likely to coerce the compiler into preferring the TDP path.Rename
tdp_page_fault()tokvm_tdp_page_fault()as it's exposed outside ofmmu.cto allow inlining the shim.Signed-off-by Sean Christopherson <[email protected]>---
Out of pure curiosity, if we do something like
if (vcpu->arch.mmu->page_fault == tdp_page_fault)
tdp_page_fault(...)
else if (vcpu->arch.mmu->page_fault == nonpaging_page_fault)
nonpaging_page_fault(...)
...
we also defeat the retpoline, right? Should we use this technique ... everywhere? :-)
Signature · 1 line
Vitaly
Quoted 16 lines · Vitaly Kuznetsov
Quoted 11 lines · Sean Christopherson
Wrap calls to ->page_fault() with a small shim to directly invoke the TDP fault handler when the kernel is using retpolines and TDP is being used. Denote the TDP fault handler by nullifying
mmu->page_fault, and annotate the TDP path as likely to coerce the compiler into preferring the TDP path.Rename
tdp_page_fault()tokvm_tdp_page_fault()as it's exposed outside ofmmu.cto allow inlining the shim.Signed-off-by Sean Christopherson <[email protected]>---
Out of pure curiosity, if we do something like
Quoted 7 lines
if (
vcpu->arch.mmu->page_fault==tdp_page_fault)tdp_page_fault(...)else if (
vcpu->arch.mmu->page_fault==nonpaging_page_fault)
nonpaging_page_fault(...)
...we also defeat the retpoline, right?
Yep.
Quoted 1 line
Should we use this technique ... everywhere? :-)
It becomes a matter of weighing the maintenance cost and robustness against the performance benefits. For the TDP case, amost no one (that cares about performance) uses shadow paging, the change is very explicit, tiny and isolated, and TDP page fault are a hot path, e.g. when booting the VM. I.e. low maintenance overhead, still robust, and IMO worth the shenanigans.
The changes to VMX's VM-Exit handlers follow similar thinking: snipe off the exit handlers that are performance critical, but use a low maintenance implementation for the majority of handlers.
There have been multiple attempts to add infrastructure to solve the maintenance and robustness problems[*], but AFAIK none of them have made their way upstream.
Vitaly Kuznetsov
Reviewed-byQuoted 39 lines · Sean Christopherson
Quoted 16 lines · Vitaly Kuznetsov
Quoted 11 lines · Sean Christopherson
Wrap calls to ->page_fault() with a small shim to directly invoke the TDP fault handler when the kernel is using retpolines and TDP is being used. Denote the TDP fault handler by nullifying
mmu->page_fault, and annotate the TDP path as likely to coerce the compiler into preferring the TDP path.Rename
tdp_page_fault()tokvm_tdp_page_fault()as it's exposed outside ofmmu.cto allow inlining the shim.Signed-off-by Sean Christopherson <[email protected]>---
Out of pure curiosity, if we do something like
Quoted 7 lines
if (
vcpu->arch.mmu->page_fault==tdp_page_fault)tdp_page_fault(...)else if (
vcpu->arch.mmu->page_fault==nonpaging_page_fault)
nonpaging_page_fault(...)
...we also defeat the retpoline, right?
Yep.
Quoted 1 line
Should we use this technique ... everywhere? :-)
It becomes a matter of weighing the maintenance cost and robustness against the performance benefits. For the TDP case, amost no one (that cares about performance) uses shadow paging, the change is very explicit, tiny and isolated, and TDP page fault are a hot path, e.g. when booting the VM. I.e. low maintenance overhead, still robust, and IMO worth the shenanigans.
The changes to VMX's VM-Exit handlers follow similar thinking: snipe off the exit handlers that are performance critical, but use a low maintenance implementation for the majority of handlers.
Quoted 6 lines
There have been multiple attempts to add infrastructure to solve the maintenance and robustness problems[*], but AFAIK none of them have made their way upstream.
Oh I see, missed some of these discussion.
And I actualy forgot to say:
as the patch itself looks good to me, I was just wondering about the approach in general.
Signature · 1 line
Vitaly
Paolo Bonzini
maintainerQuoted 5 lines · Sean Christopherson
It becomes a matter of weighing the maintenance cost and robustness against the performance benefits. For the TDP case, amost no one (that cares about performance) uses shadow paging, the change is very explicit, tiny and isolated, and TDP page fault are a hot path, e.g. when booting the VM. I.e. low maintenance overhead, still robust, and IMO worth the shenanigans.
The "NULL" trick does not seem needed though. Any objections to this?
arch/x86/kvm/mmu.h +1 -1
@@ -109,7 +109,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u32 err, bool prefault) {#ifdef CONFIG_RETPOLINE -if (likely (!vcpu->arch.mmu->page_fault)) +if (likely (vcpu->arch.mmu->page_fault == kvm_tdp_page_fault))return kvm_tdp_page_fault (vcpu, cr2_or_gpa, err, prefault);#endif return vcpu->arch.mmu->page_fault (vcpu, cr2_or_gpa, err, prefault);
arch/x86/kvm/mmu/mmu.c +0 -5
@@ -4925,12 +4925,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)return ; context->mmu_role.as_u64 = new_role.as_u64; -#ifdef CONFIG_RETPOLINE -/* Nullify ->page_fault() to use direct kvm_tdp_page_fault() call. */ - context->page_fault =NULL ; -#else context->page_fault = kvm_tdp_page_fault; -#endif context->sync_page = nonpaging_sync_page; context->invlpg = nonpaging_invlpg; context->update_pte = nonpaging_update_pte;
Paolo
Quoted 8 lines · Paolo Bonzini
Quoted 5 lines · Sean Christopherson
It becomes a matter of weighing the maintenance cost and robustness against the performance benefits. For the TDP case, amost no one (that cares about performance) uses shadow paging, the change is very explicit, tiny and isolated, and TDP page fault are a hot path, e.g. when booting the VM. I.e. low maintenance overhead, still robust, and IMO worth the shenanigans.
The "NULL" trick does not seem needed though. Any objections to this?
Nope, no objections.
Quoted 33 lines
arch/x86/kvm/mmu.h +1
-1@@ -109,7 +109,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u32 err, bool prefault) {#ifdef CONFIG_RETPOLINE -if (likely (!vcpu->arch.mmu->page_fault)) +if (likely (vcpu->arch.mmu->page_fault == kvm_tdp_page_fault))return kvm_tdp_page_fault (vcpu, cr2_or_gpa, err, prefault);#endif return vcpu->arch.mmu->page_fault (vcpu, cr2_or_gpa, err, prefault);arch/x86/kvm/mmu/mmu.c +0
-5@@ -4925,12 +4925,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)return ; context->mmu_role.as_u64 = new_role.as_u64; -#ifdef CONFIG_RETPOLINE -/* Nullify ->page_fault() to use direct kvm_tdp_page_fault() call. */ - context->page_fault =NULL ; -#else context->page_fault = kvm_tdp_page_fault; -#endif context->sync_page = nonpaging_sync_page; context->invlpg = nonpaging_invlpg; context->update_pte = nonpaging_update_pte;Paolo