KVM: VMX: Optimize VMX instrs error/fault handling
v2 cover KVM: VMX: Optimize VMX instrs error/fault handling
A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL to kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.
Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".
The net impact relative to the current code base is more or less a nop when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.
Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive as CC_SET() without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.
A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.
v2:
- Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to __kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the __kvm_handle_fault_on_reboot() changes.
Sean Christopherson (5):
objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop ____kvm_handle_fault_on_reboot()
KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()
arch/x86/include/asm/kvm_host.h | 16 ++---- arch/x86/kvm/vmx/ops.h | 93 ++++++++++++++++++++------------- arch/x86/kvm/vmx/vmx.c | 42 +++++++++++++++ arch/x86/kvm/x86.c | 3 +- tools/objtool/check.c | 1 - 5 files changed, 104 insertions(+), 51 deletions(-)
Signature · 1 line
2.22.0
PATCH v2 1/5 objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
Explicitly check kvm_reboot in kvm_spurious_fault() prior to invoking BUG(), as opposed to assuming the caller has already done so. Letting kvm_spurious_fault() be called "directly" will allow VMX to better optimize its low level assembly flows.
As a happy side effect, kvm_spurious_fault() no longer needs to be marked as a dead end since it doesn't unconditionally BUG().
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/x86.c | 3 ++- tools/objtool/check.c | 1 - 3 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1496,7 +1496,7 @@ enum {#define kvm_arch_vcpu_memslots_id (vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)#define kvm_memslots_for_spte_role (kvm, role) __kvm_memslots(kvm, (role).smm) -asmlinkage void __noreturn kvm_spurious_fault(void); +asmlinkage void kvm_spurious_fault(void);/* * Hardware virtualization extension instructions may fault if a
arch/x86/kvm/x86.c +2 -1
@@ -356,7 +356,8 @@ EXPORT_SYMBOL_GPL(kvm_set_apic_base); asmlinkage __visible void kvm_spurious_fault(void) {/* Fault while not rebooting. We want the trace. */ -BUG (); +if (!kvm_rebooting) +BUG (); }EXPORT_SYMBOL_GPL (kvm_spurious_fault);
tools/objtool/check.c +0 -1
@@ -138,7 +138,6 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func,"do_task_dead" ,"__module_put_and_exit" ,"complete_and_exit" , -"kvm_spurious_fault" ,"__reiserfs_panic" ,"lbug_with_loc" , "fortify_panic",
Signature · 1 line
2.22.0
PATCH v2 2/5 KVM: VMX: Optimize VMX instruction error and fault handling
In reply to Sean ChristophersonRework the VMX instruction helpers using asm-goto to branch directly to error/fault "handlers" in lieu of using __ex(), i.e. the generic ____kvm_handle_fault_on_reboot(). Branching directly to fault handling code during fixup avoids the extra JMP that is inserted after every VMX instruction when using the generic "fault on reboot" (see commit 3901336ed9887, "x86/kvm: Don't call kvm_spurious_fault() from .fixup").
Opportunistically clean up the helpers so that they all have consistent error handling and messages.
Leave the usage of ____kvm_handle_fault_on_reboot() (via __ex()) in kvm_cpu_vmxoff() and nested_vmx_check_vmentry_hw() as is. The VMXOFF case is not a fast path, i.e. the cleanliness of __ex() is worth the JMP, and the extra JMP in nested_vmx_check_vmentry_hw() is unavoidable.
Note, VMREAD cannot get the asm-goto treatment as output operands aren't compatible with GCC's asm-goto due to internal compiler restrictions.
--- arch/x86/kvm/vmx/ops.h | 72 +++++++++++++++++++++++------------------- arch/x86/kvm/vmx/vmx.c | 34 ++++++++++++++++++++ 2 files changed, 74 insertions(+), 32 deletions(-)
arch/x86/kvm/vmx/ops.h +40 -32
@@ -14,6 +14,12 @@#define __ex_clear (x, reg) \ ____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg) +void vmwrite_error (unsigned long field,unsigned long value); +void vmclear_error (struct vmcs *vmcs,u64 phys_addr); +void vmptrld_error (struct vmcs *vmcs,u64 phys_addr); +void invvpid_error (unsigned long ext,u16 vpid,gva_t gva); +void invept_error (unsigned long ext,u64 eptp,gpa_t gpa); + static __always_inline void vmcs_check16(unsigned long field) { BUILD_BUG_ON_MSG(__builtin_constant_p(field) && ((field) & 0x6001) == 0x2000,
@@ -103,21 +109,39 @@ static __always_inline unsigned long vmcs_readl(unsigned long field)return __vmcs_readl (field); } -static noinline void vmwrite_error(unsigned long field, unsigned long value) -{ -printk (KERN_ERR "vmwrite error: reg %lx value %lx (err %d)\n" , - field, value,vmcs_read32 (VM_INSTRUCTION_ERROR )); -dump_stack (); -} +#define vmx_asm1(insn, op1, error_args...) \ +do { \ + asm_volatile_goto("1: " __stringify(insn) " %0\n\t" \ + ".byte 0x2e\n\t"/* branch not taken hint */ \ + "jna %l[error]\n\t" \ + _ASM_EXTABLE(1b, %l[fault]) \ + : : op1 : "cc" : error, fault); \ +return; \ +error: \ +insn##_error(error_args); \ +return; \ +fault: \ +kvm_spurious_fault(); \ +} while (0) + +#define vmx_asm2(insn, op1, op2, error_args...) \ +do { \ + asm_volatile_goto("1: " __stringify(insn) " %1, %0\n\t" \ +".byte 0x2e\n\t" /* branch not taken hint */ \ +"jna %l[error]\n\t" \ + _ASM_EXTABLE(1b, %l[fault]) \ + : : op1, op2 : "cc" : error, fault); \ +return ; \ +error: \ + insn##_error(error_args); \ +return ; \ +fault: \ +kvm_spurious_fault (); \ +}while (0 ) static __always_inline void __vmcs_writel(unsigned long field, unsigned long value) { -bool error; - - asm volatile (__ex("vmwrite %2, %1") CC_SET(na) - : CC_OUT(na) (error) : "r"(field), "rm"(value)); -if (unlikely (error)) -vmwrite_error (field, value); +vmx_asm2 (vmwrite,"r" (field),"rm" (value), field, value); } static __always_inline void vmcs_write16(unsigned long field, u16 value)
@@ -182,28 +206,18 @@ static __always_inline void vmcs_set_bits(unsigned long field, u32 mask)static inline void vmcs_clear (struct vmcs *vmcs) {u64 phys_addr =__pa (vmcs); -bool error; - asm volatile (__ex("vmclear %1") CC_SET(na) - : CC_OUT(na) (error) : "m"(phys_addr)); -if (unlikely (error)) -printk (KERN_ERR "kvm: vmclear fail: %p/%llx\n" , - vmcs, phys_addr); +vmx_asm1 (vmclear,"m" (phys_addr), vmcs, phys_addr); }static inline void vmcs_load (struct vmcs *vmcs) {u64 phys_addr =__pa (vmcs); -bool error;if (static_branch_unlikely (&enable_evmcs))return evmcs_load (phys_addr); - asm volatile (__ex("vmptrld %1") CC_SET(na) - : CC_OUT(na) (error) : "m"(phys_addr)); -if (unlikely (error)) -printk (KERN_ERR "kvm: vmptrld %p/%llx failed\n" , - vmcs, phys_addr); +vmx_asm1 (vmptrld,"m" (phys_addr), vmcs, phys_addr); }static inline void __invvpid (unsigned long ext,u16 vpid,gva_t gva)
@@ -213,11 +227,8 @@ static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva) u64 rsvd : 48;u64 gva; } operand = { vpid, 0, gva }; -bool error; - asm volatile (__ex("invvpid %2, %1") CC_SET(na) - : CC_OUT(na) (error) : "r"(ext), "m"(operand)); -BUG_ON (error); +vmx_asm2 (invvpid,"r" (ext),"m" (operand), ext, vpid, gva); }static inline void __invept (unsigned long ext,u64 eptp,gpa_t gpa)
@@ -225,11 +236,8 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)struct {u64 eptp, gpa; } operand = {eptp, gpa}; -bool error; - asm volatile (__ex("invept %2, %1") CC_SET(na) - : CC_OUT(na) (error) : "r"(ext), "m"(operand)); -BUG_ON (error); +vmx_asm2 (invept,"r" (ext),"m" (operand), ext, eptp, gpa); }static inline bool vpid_sync_vcpu_addr (int vpid,gva_t addr)
arch/x86/kvm/vmx/vmx.c +34 -0
@@ -343,6 +343,40 @@ static __always_inline void vmx_disable_intercept_for_msr(unsigned long *msr_bitvoid vmx_vmexit (void ); +#define vmx_insn_failed(fmt...) \ +do { \ + WARN_ONCE(1, fmt); \ + pr_warn_ratelimited(fmt); \ +} while (0) + +noinline void vmwrite_error(unsigned long field, unsigned long value) +{ +vmx_insn_failed ("kvm: vmwrite failed: field=%lx val=%lx err=%d\n" , + field, value,vmcs_read32 (VM_INSTRUCTION_ERROR )); +} + +noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr) +{ +vmx_insn_failed ("kvm: vmclear failed: %p/%llx\n" , vmcs, phys_addr); +} + +noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr) +{ +vmx_insn_failed ("kvm: vmptrld failed: %p/%llx\n" , vmcs, phys_addr); +} + +noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva) +{ +vmx_insn_failed ("kvm: invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n" , + ext, vpid, gva); +} + +noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa) +{ +vmx_insn_failed ("kvm: invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n" , + ext, eptp, gpa); +} + static DEFINE_PER_CPU(struct vmcs *, vmxarea);DEFINE_PER_CPU (struct vmcs *, current_vmcs);/*
Signature · 1 line
2.22.0
PATCH v2 3/5 KVM: VMX: Add error handling to VMREAD helper
In reply to Sean ChristophersonNow that VMREAD flows require a taken branch, courtesy of commit
3901336ed9887 ("x86/kvm: Don't call kvm_spurious_fault() from .fixup")
bite the bullet and add full error handling to VMREAD, i.e. replace the JMP added by __ex()/____kvm_handle_fault_on_reboot() with a hinted Jcc.
To minimize the code footprint, add a helper function, vmread_error(), to handle both faults and failures so that the inline flow has a single CALL.
--- arch/x86/kvm/vmx/ops.h | 21 +++++++++++++++++---- arch/x86/kvm/vmx/vmx.c | 8 ++++++++ 2 files changed, 25 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/ops.h +17 -4
@@ -11,9 +11,8 @@#include "vmcs.h" #define __ex (x) __kvm_handle_fault_on_reboot(x) -#define __ex_clear (x, reg) \ - ____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg) +asmlinkage void vmread_error(unsigned long field, bool fault);void vmwrite_error (unsigned long field,unsigned long value);void vmclear_error (struct vmcs *vmcs,u64 phys_addr);void vmptrld_error (struct vmcs *vmcs,u64 phys_addr);
@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) {unsigned long value; - asm volatile (__ex_clear("vmread %1, %0", "%k0") - : "=r"(value) : "r"(field)); + asmvolatile ("1: vmread %2, %1\n\t" +".byte 0x3e\n\t" /* branch taken hint */ +"ja 3f\n\t" +"mov %2, %%" _ASM_ARG1"\n\t" +"xor %%" _ASM_ARG2", %%" _ASM_ARG2"\n\t" +"2: call vmread_error\n\t" +"xor %k1, %k1\n\t" +"3:\n\t" + +".pushsection .fixup, \"ax\"\n\t" +"4: mov %2, %%" _ASM_ARG1"\n\t" +"mov $1, %%" _ASM_ARG2"\n\t" +"jmp 2b\n\t" +".popsection\n\t" + _ASM_EXTABLE(1b, 4b) + : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");return value; }
arch/x86/kvm/vmx/vmx.c +8 -0
@@ -349,6 +349,14 @@ do { \pr_warn_ratelimited (fmt); \ }while (0 ) +asmlinkage void vmread_error(unsigned long field, bool fault) +{ +if (fault) +kvm_spurious_fault (); +else +vmx_insn_failed ("kvm: vmread failed: field=%lx\n" , field); +} + noinline void vmwrite_error(unsigned long field, unsigned long value) { vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%d\n",
Signature · 1 line
2.22.0
Re: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper
Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/ops.h +6
-2@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) {unsigned long value; - asm volatile (__ex_clear("vmread %1, %0", "%k0") - : "=r"(value) : "r"(field)); + asmvolatile ("1: vmread %2, %1\n\t" +".byte 0x3e\n\t" /* branch taken hint */ +"ja 3f\n\t" +"mov %2, %%" _ASM_ARG1"\n\t" +"xor %%" _ASM_ARG2", %%" _ASM_ARG2"\n\t" + "2: call vmread_error\n\t"
arch/x86/kvm/vmx/ops.h static __always_inline unsigned long __vmcs_readl(unsigned long field) · line 85
Quoted 10 lines
arch/x86/kvm/vmx/ops.h +10
-0+"xor %k1, %k1\n\t" +"3:\n\t" + +".pushsection .fixup, \"ax\"\n\t" +"4: mov %2, %%" _ASM_ARG1"\n\t" +"mov $1, %%" _ASM_ARG2"\n\t" +"jmp 2b\n\t" +".popsection\n\t" + _ASM_EXTABLE(1b ,4b ) + : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");
Was there a reason you didn't do the asm goto thing here like you did for the previous patch? That seemed cleaner, and needs less asm.
I think the branch hints aren't needed -- they're ignored on modern processors. Ditto for the previous patch.
Also please use named asm operands whereever you can, like "%[field]" instead of "%2". It helps a lot with readability.
Signature · 1 line
Josh
Paolo Bonzini
maintainerRe: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper
Quoted 18 lines · Josh Poimboeuf
Quoted 17 lines · Sean Christopherson
arch/x86/kvm/vmx/ops.h +11
-2@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) {unsigned long value; - asm volatile (__ex_clear("vmread %1, %0", "%k0") - : "=r"(value) : "r"(field)); + asmvolatile ("1: vmread %2, %1\n\t" +".byte 0x3e\n\t" /* branch taken hint */ +"ja 3f\n\t" +"mov %2, %%" _ASM_ARG1"\n\t" +"xor %%" _ASM_ARG2", %%" _ASM_ARG2"\n\t" +"2: call vmread_error\n\t" +"xor %k1, %k1\n\t" +"3:\n\t" + +".pushsection .fixup, \"ax\"\n\t" + "4: mov %2, %%" _ASM_ARG1 "\n\t"
Quoted 8 lines
arch/x86/kvm/vmx/ops.h static __always_inline unsigned long __vmcs_readl(unsigned long field) · line 85
Quoted 5 lines
arch/x86/kvm/vmx/ops.h +5
-0+"mov $1, %%" _ASM_ARG2"\n\t" +"jmp 2b\n\t" +".popsection\n\t" + _ASM_EXTABLE(1b ,4b ) + : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");Was there a reason you didn't do the asm goto thing here like you did for the previous patch? That seemed cleaner, and needs less asm.
It's because asm goto doesn't support outputs.
Paolo
Quoted 6 lines
I think the branch hints aren't needed -- they're ignored on modern processors. Ditto for the previous patch.
Also please use named asm operands whereever you can, like "%[field]" instead of "%2". It helps a lot with readability.
PATCH v2 4/5 KVM: x86: Drop ____kvm_handle_fault_on_reboot()
In reply to Sean ChristophersonRemove the variation of __kvm_handle_fault_on_reboot() that accepts a post-fault cleanup instruction now that its sole user (VMREAD) uses a different method for handling faults.
--- arch/x86/include/asm/kvm_host.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -5
@@ -1504,7 +1504,7 @@ asmlinkage void kvm_spurious_fault(void);* Usually after catching the fault we just panic; during reboot * instead the instruction is ignored. */ -#define ____kvm_handle_fault_on_reboot (insn, cleanup_insn) \ +#define __kvm_handle_fault_on_reboot (insn) \ "666: \n\t" \ insn "\n\t" \ "jmp 668f \n\t" \
@@ -1513,16 +1513,12 @@ asmlinkage void kvm_spurious_fault(void);"668: \n\t" \".pushsection .fixup, \"ax\" \n\t" \"700: \n\t" \ - cleanup_insn"\n\t" \"cmpb $0, kvm_rebooting\n\t" \"je 667b \n\t" \"jmp 668b \n\t" \".popsection \n\t" \ _ASM_EXTABLE(666b, 700b) -#define __kvm_handle_fault_on_reboot (insn) \ - ____kvm_handle_fault_on_reboot(insn, "") -#define KVM_ARCH_WANT_MMU_NOTIFIER int kvm_unmap_hva_range (struct kvm *kvm,unsigned long start,unsigned long end);int kvm_age_hva (struct kvm *kvm,unsigned long start,unsigned long end);
Signature · 1 line
2.22.0
PATCH v2 5/5 KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()
In reply to Sean ChristophersonRemove the kvm_rebooting check from VMX/SVM instruction exception fixup now that kvm_spurious_fault() conditions its BUG() on !kvm_rebooting. Because the 'cleanup_insn' functionally is also gone, deferring to kvm_spurious_fault() means __kvm_handle_fault_on_reboot() can eliminate its .fixup code entirely and have its exception table entry branch directly to the call to kvm_spurious_fault().
--- arch/x86/include/asm/kvm_host.h | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -7
@@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);"667: \n\t" \"call kvm_spurious_fault \n\t" \"668: \n\t" \ -".pushsection .fixup, \"ax\" \n\t" \ -"700: \n\t" \ -"cmpb $0, kvm_rebooting\n\t" \ -"je 667b \n\t" \ -"jmp 668b \n\t" \ -".popsection \n\t" \ - _ASM_EXTABLE(666b, 700b) + _ASM_EXTABLE(666b, 667b)#define KVM_ARCH_WANT_MMU_NOTIFIER int kvm_unmap_hva_range (struct kvm *kvm,unsigned long start,unsigned long end);
Signature · 1 line
2.22.0
Re: [PATCH v2 5/5] KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()
Quoted 24 lines · Sean Christopherson
Remove the
kvm_rebootingcheck from VMX/SVM instruction exception fixup now thatkvm_spurious_fault()conditions itsBUG()on !kvm_rebooting. Because the 'cleanup_insn' functionally is also gone, deferring tokvm_spurious_fault()means__kvm_handle_fault_on_reboot()can eliminate its .fixup code entirely and have its exception table entry branch directly to the call tokvm_spurious_fault().Cc Paolo Bonzini <[email protected]>Cc Josh Poimboeuf <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm_host.h | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)arch/x86/include/asm/kvm_host.h +0
-1@@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);"667: \n\t" \"call kvm_spurious_fault \n\t" \"668: \n\t" \ -".pushsection .fixup, \"ax\" \n\t" \
arch/x86/include/asm/kvm_host.h asmlinkage void kvm_spurious_fault(void); · line 1517
Quoted 10 lines
arch/x86/include/asm/kvm_host.h +1
-6-"700: \n\t" \ -"cmpb $0, kvm_rebooting\n\t" \ -"je 667b \n\t" \ -"jmp 668b \n\t" \ -".popsection \n\t" \ - _ASM_EXTABLE(666b, 700b) + _ASM_EXTABLE(666b, 667b)#define KVM_ARCH_WANT_MMU_NOTIFIER int kvm_unmap_hva_range (struct kvm *kvm,unsigned long start,unsigned long end);
Paolo,
Any chance this can be picked up for 5.4? Josh's kvm_spurious_fault() changes went into 5.3, so this can be taken through the KVM tree, which probably makes the most sense since the non-KVM change is a one line deletion.
Quoted 49 lines · Sean Christopherson
A recent commit reworked
__kvm_handle_fault_on_reboot()to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL tokvm_spurious_fault()instead of a funky PUSH+JMP facsimile in .fixup.Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to
kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".The net impact relative to the current code base is more or less a nop when building with a compiler that supports
__GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.Without
__GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive asCC_SET()without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.
v2:
- Rebased to x86/master, commiteceffd88ca20("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to__kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the__kvm_handle_fault_on_reboot()changes.Sean Christopherson (5):
objtool: KVM: x86: Checkkvm_rebootinginkvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop____kvm_handle_fault_on_reboot()
KVM: x86: Don't checkkvm_rebootingin__kvm_handle_fault_on_reboot()arch/x86/include/asm/kvm_host.h | 16 ++---- arch/x86/kvm/vmx/ops.h | 93 ++++++++++++++++++++------------- arch/x86/kvm/vmx/vmx.c | 42 +++++++++++++++ arch/x86/kvm/x86.c | 3 +- tools/objtool/check.c | 1 - 5 files changed, 104 insertions(+), 51 deletions(-)Signature · 1 line
2.22.0
Quoted 39 lines · Sean Christopherson
A recent commit reworked
__kvm_handle_fault_on_reboot()to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL tokvm_spurious_fault()instead of a funky PUSH+JMP facsimile in .fixup.Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to
kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".The net impact relative to the current code base is more or less a nop when building with a compiler that supports
__GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.Without
__GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive asCC_SET()without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.
v2:
- Rebased to x86/master, commiteceffd88ca20("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to__kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the__kvm_handle_fault_on_reboot()changes.Sean Christopherson (5):
objtool: KVM: x86: Checkkvm_rebootinginkvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop____kvm_handle_fault_on_reboot()
KVM: x86: Don't checkkvm_rebootingin__kvm_handle_fault_on_reboot()
Quoted 7 lines
arch/x86/include/asm/kvm_host.h | 16 ++---- arch/x86/kvm/vmx/ops.h | 93 ++++++++++++++++++++------------- arch/x86/kvm/vmx/vmx.c | 42 +++++++++++++++ arch/x86/kvm/x86.c | 3 +- tools/objtool/check.c | 1 - 5 files changed, 104 insertions(+), 51 deletions(-)
Queued, thanks.
Paolo