Skip to contentSkip to conversation

KVM: VMX: Optimize VMX instrs error/fault handling

PATCH Applied Sean Christopherson 3 participants 11 messages Jul 19, 2019 – Sep 25, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: VMX: Optimize VMX instrs error/fault handling

A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL to kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.

Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".

The net impact relative to the current code base is more or less a nop when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.

Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive as CC_SET() without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.

A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.

v2:
- Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to __kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the __kvm_handle_fault_on_reboot() changes.

Sean Christopherson (5):
objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop ____kvm_handle_fault_on_reboot()
KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

 arch/x86/include/asm/kvm_host.h | 16 ++----
 arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
 arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
 arch/x86/kvm/x86.c              |  3 +-
 tools/objtool/check.c           |  1 -
 5 files changed, 104 insertions(+), 51 deletions(-)
Signature · 1 line
2.22.0
#1 lore raw

PATCH v2 1/5 objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()

Explicitly check kvm_reboot in kvm_spurious_fault() prior to invoking BUG(), as opposed to assuming the caller has already done so. Letting kvm_spurious_fault() be called "directly" will allow VMX to better optimize its low level assembly flows.

As a happy side effect, kvm_spurious_fault() no longer needs to be marked as a dead end since it doesn't unconditionally BUG().

---
 arch/x86/include/asm/kvm_host.h | 2 +-
 arch/x86/kvm/x86.c              | 3 ++-
 tools/objtool/check.c           | 1 -
 3 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index 8282b8d41209..9739ed615faf 100644
@@ -1496,7 +1496,7 @@ enum {
 #define kvm_arch_vcpu_memslots_id(vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)
 #define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, (role).smm)
 
-asmlinkage void __noreturn kvm_spurious_fault(void);
+asmlinkage void kvm_spurious_fault(void);
 
 /*
  * Hardware virtualization extension instructions may fault if a
arch/x86/kvm/x86.c +2 -1
index 4a0b74ecd1de..6bc012afb86a 100644
@@ -356,7 +356,8 @@ EXPORT_SYMBOL_GPL(kvm_set_apic_base);
 asmlinkage __visible void kvm_spurious_fault(void)
 {
 	/* Fault while not rebooting.  We want the trace. */
-	BUG();
+	if (!kvm_rebooting)
+		BUG();
 }
 EXPORT_SYMBOL_GPL(kvm_spurious_fault);
 
tools/objtool/check.c +0 -1
index 5f26620f13f5..688a9af8124d 100644
@@ -138,7 +138,6 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func,
 		"do_task_dead",
 		"__module_put_and_exit",
 		"complete_and_exit",
-		"kvm_spurious_fault",
 		"__reiserfs_panic",
 		"lbug_with_loc",
 		"fortify_panic",
Signature · 1 line
2.22.0
#2 lore raw

PATCH v2 2/5 KVM: VMX: Optimize VMX instruction error and fault handling

In reply to Sean Christopherson

Rework the VMX instruction helpers using asm-goto to branch directly to error/fault "handlers" in lieu of using __ex(), i.e. the generic ____kvm_handle_fault_on_reboot(). Branching directly to fault handling code during fixup avoids the extra JMP that is inserted after every VMX instruction when using the generic "fault on reboot" (see commit 3901336ed9887, "x86/kvm: Don't call kvm_spurious_fault() from .fixup").

Opportunistically clean up the helpers so that they all have consistent error handling and messages.

Leave the usage of ____kvm_handle_fault_on_reboot() (via __ex()) in kvm_cpu_vmxoff() and nested_vmx_check_vmentry_hw() as is. The VMXOFF case is not a fast path, i.e. the cleanliness of __ex() is worth the JMP, and the extra JMP in nested_vmx_check_vmentry_hw() is unavoidable.

Note, VMREAD cannot get the asm-goto treatment as output operands aren't compatible with GCC's asm-goto due to internal compiler restrictions.

---
 arch/x86/kvm/vmx/ops.h | 72 +++++++++++++++++++++++-------------------
 arch/x86/kvm/vmx/vmx.c | 34 ++++++++++++++++++++
 2 files changed, 74 insertions(+), 32 deletions(-)
arch/x86/kvm/vmx/ops.h +40 -32
index 2200fb698dd0..79e25d49d4d9 100644
@@ -14,6 +14,12 @@
 #define __ex_clear(x, reg) \
 	____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg)
 
+void vmwrite_error(unsigned long field, unsigned long value);
+void vmclear_error(struct vmcs *vmcs, u64 phys_addr);
+void vmptrld_error(struct vmcs *vmcs, u64 phys_addr);
+void invvpid_error(unsigned long ext, u16 vpid, gva_t gva);
+void invept_error(unsigned long ext, u64 eptp, gpa_t gpa);
+
 static __always_inline void vmcs_check16(unsigned long field)
 {
 	BUILD_BUG_ON_MSG(__builtin_constant_p(field) && ((field) & 0x6001) == 0x2000,
@@ -103,21 +109,39 @@ static __always_inline unsigned long vmcs_readl(unsigned long field)
 	return __vmcs_readl(field);
 }
 
-static noinline void vmwrite_error(unsigned long field, unsigned long value)
-{
-	printk(KERN_ERR "vmwrite error: reg %lx value %lx (err %d)\n",
-	       field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
-	dump_stack();
-}
+#define vmx_asm1(insn, op1, error_args...)				\
+do {									\
+	asm_volatile_goto("1: " __stringify(insn) " %0\n\t"		\
+			  ".byte 0x2e\n\t" /* branch not taken hint */	\
+			  "jna %l[error]\n\t"				\
+			  _ASM_EXTABLE(1b, %l[fault])			\
+			  : : op1 : "cc" : error, fault);		\
+	return;								\
+error:									\
+	insn##_error(error_args);					\
+	return;								\
+fault:									\
+	kvm_spurious_fault();						\
+} while (0)
+
+#define vmx_asm2(insn, op1, op2, error_args...)				\
+do {									\
+	asm_volatile_goto("1: "  __stringify(insn) " %1, %0\n\t"	\
+			  ".byte 0x2e\n\t" /* branch not taken hint */	\
+			  "jna %l[error]\n\t"				\
+			  _ASM_EXTABLE(1b, %l[fault])			\
+			  : : op1, op2 : "cc" : error, fault);		\
+	return;								\
+error:									\
+	insn##_error(error_args);					\
+	return;								\
+fault:									\
+	kvm_spurious_fault();						\
+} while (0)
 
 static __always_inline void __vmcs_writel(unsigned long field, unsigned long value)
 {
-	bool error;
-
-	asm volatile (__ex("vmwrite %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(field), "rm"(value));
-	if (unlikely(error))
-		vmwrite_error(field, value);
+	vmx_asm2(vmwrite, "r"(field), "rm"(value), field, value);
 }
 
 static __always_inline void vmcs_write16(unsigned long field, u16 value)
@@ -182,28 +206,18 @@ static __always_inline void vmcs_set_bits(unsigned long field, u32 mask)
 static inline void vmcs_clear(struct vmcs *vmcs)
 {
 	u64 phys_addr = __pa(vmcs);
-	bool error;
 
-	asm volatile (__ex("vmclear %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "m"(phys_addr));
-	if (unlikely(error))
-		printk(KERN_ERR "kvm: vmclear fail: %p/%llx\n",
-		       vmcs, phys_addr);
+	vmx_asm1(vmclear, "m"(phys_addr), vmcs, phys_addr);
 }
 
 static inline void vmcs_load(struct vmcs *vmcs)
 {
 	u64 phys_addr = __pa(vmcs);
-	bool error;
 
 	if (static_branch_unlikely(&enable_evmcs))
 		return evmcs_load(phys_addr);
 
-	asm volatile (__ex("vmptrld %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "m"(phys_addr));
-	if (unlikely(error))
-		printk(KERN_ERR "kvm: vmptrld %p/%llx failed\n",
-		       vmcs, phys_addr);
+	vmx_asm1(vmptrld, "m"(phys_addr), vmcs, phys_addr);
 }
 
 static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva)
@@ -213,11 +227,8 @@ static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva)
 		u64 rsvd : 48;
 		u64 gva;
 	} operand = { vpid, 0, gva };
-	bool error;
 
-	asm volatile (__ex("invvpid %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(ext), "m"(operand));
-	BUG_ON(error);
+	vmx_asm2(invvpid, "r"(ext), "m"(operand), ext, vpid, gva);
 }
 
 static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)
@@ -225,11 +236,8 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)
 	struct {
 		u64 eptp, gpa;
 	} operand = {eptp, gpa};
-	bool error;
 
-	asm volatile (__ex("invept %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(ext), "m"(operand));
-	BUG_ON(error);
+	vmx_asm2(invept, "r"(ext), "m"(operand), ext, eptp, gpa);
 }
 
 static inline bool vpid_sync_vcpu_addr(int vpid, gva_t addr)
arch/x86/kvm/vmx/vmx.c +34 -0
index 69536553446d..46689019ebf7 100644
@@ -343,6 +343,40 @@ static __always_inline void vmx_disable_intercept_for_msr(unsigned long *msr_bit
 
 void vmx_vmexit(void);
 
+#define vmx_insn_failed(fmt...)		\
+do {					\
+	WARN_ONCE(1, fmt);		\
+	pr_warn_ratelimited(fmt);	\
+} while (0)
+
+noinline void vmwrite_error(unsigned long field, unsigned long value)
+{
+	vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%d\n",
+			field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
+}
+
+noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr)
+{
+	vmx_insn_failed("kvm: vmclear failed: %p/%llx\n", vmcs, phys_addr);
+}
+
+noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr)
+{
+	vmx_insn_failed("kvm: vmptrld failed: %p/%llx\n", vmcs, phys_addr);
+}
+
+noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva)
+{
+	vmx_insn_failed("kvm: invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",
+			ext, vpid, gva);
+}
+
+noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa)
+{
+	vmx_insn_failed("kvm: invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",
+			ext, eptp, gpa);
+}
+
 static DEFINE_PER_CPU(struct vmcs *, vmxarea);
 DEFINE_PER_CPU(struct vmcs *, current_vmcs);
 /*
Signature · 1 line
2.22.0
#3 lore raw

PATCH v2 3/5 KVM: VMX: Add error handling to VMREAD helper

In reply to Sean Christopherson

Now that VMREAD flows require a taken branch, courtesy of commit

3901336ed9887 ("x86/kvm: Don't call kvm_spurious_fault() from .fixup")

bite the bullet and add full error handling to VMREAD, i.e. replace the JMP added by __ex()/____kvm_handle_fault_on_reboot() with a hinted Jcc.

To minimize the code footprint, add a helper function, vmread_error(), to handle both faults and failures so that the inline flow has a single CALL.

---
 arch/x86/kvm/vmx/ops.h | 21 +++++++++++++++++----
 arch/x86/kvm/vmx/vmx.c |  8 ++++++++
 2 files changed, 25 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/ops.h +17 -4
index 79e25d49d4d9..45eaedee2ac0 100644
@@ -11,9 +11,8 @@
 #include "vmcs.h"
 
 #define __ex(x) __kvm_handle_fault_on_reboot(x)
-#define __ex_clear(x, reg) \
-	____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg)
 
+asmlinkage void vmread_error(unsigned long field, bool fault);
 void vmwrite_error(unsigned long field, unsigned long value);
 void vmclear_error(struct vmcs *vmcs, u64 phys_addr);
 void vmptrld_error(struct vmcs *vmcs, u64 phys_addr);
@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
 {
 	unsigned long value;
 
-	asm volatile (__ex_clear("vmread %1, %0", "%k0")
-		      : "=r"(value) : "r"(field));
+	asm volatile("1: vmread %2, %1\n\t"
+		     ".byte 0x3e\n\t" /* branch taken hint */
+		     "ja 3f\n\t"
+		     "mov %2, %%" _ASM_ARG1 "\n\t"
+		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
+		     "2: call vmread_error\n\t"
+		     "xor %k1, %k1\n\t"
+		     "3:\n\t"
+
+		     ".pushsection .fixup, \"ax\"\n\t"
+		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
+		     "mov $1, %%" _ASM_ARG2 "\n\t"
+		     "jmp 2b\n\t"
+		     ".popsection\n\t"
+		     _ASM_EXTABLE(1b, 4b)
+		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");
 	return value;
 }
 
arch/x86/kvm/vmx/vmx.c +8 -0
index 46689019ebf7..9acf3d6395d3 100644
@@ -349,6 +349,14 @@ do {					\
 	pr_warn_ratelimited(fmt);	\
 } while (0)
 
+asmlinkage void vmread_error(unsigned long field, bool fault)
+{
+	if (fault)
+		kvm_spurious_fault();
+	else
+		vmx_insn_failed("kvm: vmread failed: field=%lx\n", field);
+}
+
 noinline void vmwrite_error(unsigned long field, unsigned long value)
 {
 	vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%d\n",
Signature · 1 line
2.22.0
#4 lore raw

Re: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper

Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/ops.h +6 -2
@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
 {
 	unsigned long value;
 
-	asm volatile (__ex_clear("vmread %1, %0", "%k0")
-		      : "=r"(value) : "r"(field));
+	asm volatile("1: vmread %2, %1\n\t"
+		     ".byte 0x3e\n\t" /* branch taken hint */
+		     "ja 3f\n\t"
+		     "mov %2, %%" _ASM_ARG1 "\n\t"
+		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
+		     "2: call vmread_error\n\t"

arch/x86/kvm/vmx/ops.h static __always_inline unsigned long __vmcs_readl(unsigned long field) · line 85

Quoted 10 lines
arch/x86/kvm/vmx/ops.h +10 -0
+		     "xor %k1, %k1\n\t"
+		     "3:\n\t"
+
+		     ".pushsection .fixup, \"ax\"\n\t"
+		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
+		     "mov $1, %%" _ASM_ARG2 "\n\t"
+		     "jmp 2b\n\t"
+		     ".popsection\n\t"
+		     _ASM_EXTABLE(1b, 4b)
+		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");

Was there a reason you didn't do the asm goto thing here like you did for the previous patch? That seemed cleaner, and needs less asm.

I think the branch hints aren't needed -- they're ignored on modern processors. Ditto for the previous patch.

Also please use named asm operands whereever you can, like "%[field]" instead of "%2". It helps a lot with readability.

Signature · 1 line
Josh
maintainer
#5 lore raw

Re: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper

Quoted 18 lines · Josh Poimboeuf
Quoted 17 lines · Sean Christopherson
arch/x86/kvm/vmx/ops.h +11 -2
@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
 {
 	unsigned long value;
 
-	asm volatile (__ex_clear("vmread %1, %0", "%k0")
-		      : "=r"(value) : "r"(field));
+	asm volatile("1: vmread %2, %1\n\t"
+		     ".byte 0x3e\n\t" /* branch taken hint */
+		     "ja 3f\n\t"
+		     "mov %2, %%" _ASM_ARG1 "\n\t"
+		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
+		     "2: call vmread_error\n\t"
+		     "xor %k1, %k1\n\t"
+		     "3:\n\t"
+
+		     ".pushsection .fixup, \"ax\"\n\t"
+		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
Quoted 8 lines

arch/x86/kvm/vmx/ops.h static __always_inline unsigned long __vmcs_readl(unsigned long field) · line 85

Quoted 5 lines
arch/x86/kvm/vmx/ops.h +5 -0
+		     "mov $1, %%" _ASM_ARG2 "\n\t"
+		     "jmp 2b\n\t"
+		     ".popsection\n\t"
+		     _ASM_EXTABLE(1b, 4b)
+		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");

Was there a reason you didn't do the asm goto thing here like you did for the previous patch? That seemed cleaner, and needs less asm.

It's because asm goto doesn't support outputs.

Paolo

Quoted 6 lines

I think the branch hints aren't needed -- they're ignored on modern processors. Ditto for the previous patch.

Also please use named asm operands whereever you can, like "%[field]" instead of "%2". It helps a lot with readability.

#6 lore raw

PATCH v2 4/5 KVM: x86: Drop ____kvm_handle_fault_on_reboot()

In reply to Sean Christopherson

Remove the variation of __kvm_handle_fault_on_reboot() that accepts a post-fault cleanup instruction now that its sole user (VMREAD) uses a different method for handling faults.

---
 arch/x86/include/asm/kvm_host.h | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -5
index 9739ed615faf..92c59cd923b6 100644
@@ -1504,7 +1504,7 @@ asmlinkage void kvm_spurious_fault(void);
  * Usually after catching the fault we just panic; during reboot
  * instead the instruction is ignored.
  */
-#define ____kvm_handle_fault_on_reboot(insn, cleanup_insn)		\
+#define __kvm_handle_fault_on_reboot(insn)				\
 	"666: \n\t"							\
 	insn "\n\t"							\
 	"jmp	668f \n\t"						\
@@ -1513,16 +1513,12 @@ asmlinkage void kvm_spurious_fault(void);
 	"668: \n\t"							\
 	".pushsection .fixup, \"ax\" \n\t"				\
 	"700: \n\t"							\
-	cleanup_insn "\n\t"						\
 	"cmpb	$0, kvm_rebooting\n\t"					\
 	"je	667b \n\t"						\
 	"jmp	668b \n\t"						\
 	".popsection \n\t"						\
 	_ASM_EXTABLE(666b, 700b)
 
-#define __kvm_handle_fault_on_reboot(insn)		\
-	____kvm_handle_fault_on_reboot(insn, "")
-
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
 int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);
Signature · 1 line
2.22.0
#7 lore raw

PATCH v2 5/5 KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

In reply to Sean Christopherson

Remove the kvm_rebooting check from VMX/SVM instruction exception fixup now that kvm_spurious_fault() conditions its BUG() on !kvm_rebooting. Because the 'cleanup_insn' functionally is also gone, deferring to kvm_spurious_fault() means __kvm_handle_fault_on_reboot() can eliminate its .fixup code entirely and have its exception table entry branch directly to the call to kvm_spurious_fault().

Cc Paolo Bonzini <[email protected]>
Cc Josh Poimboeuf <[email protected]>
---
 arch/x86/include/asm/kvm_host.h | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -7
index 92c59cd923b6..a5ae5562ce0a 100644
@@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);
 	"667: \n\t"							\
 	"call	kvm_spurious_fault \n\t"				\
 	"668: \n\t"							\
-	".pushsection .fixup, \"ax\" \n\t"				\
-	"700: \n\t"							\
-	"cmpb	$0, kvm_rebooting\n\t"					\
-	"je	667b \n\t"						\
-	"jmp	668b \n\t"						\
-	".popsection \n\t"						\
-	_ASM_EXTABLE(666b, 700b)
+	_ASM_EXTABLE(666b, 667b)
 
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
Signature · 1 line
2.22.0
maintainer Acked-by on v2 5/5
#8 lore raw

Re: [PATCH v2 5/5] KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

Quoted 24 lines · Sean Christopherson

Remove the kvm_rebooting check from VMX/SVM instruction exception fixup now that kvm_spurious_fault() conditions its BUG() on !kvm_rebooting. Because the 'cleanup_insn' functionally is also gone, deferring to kvm_spurious_fault() means __kvm_handle_fault_on_reboot() can eliminate its .fixup code entirely and have its exception table entry branch directly to the call to kvm_spurious_fault().

Cc Paolo Bonzini <[email protected]>
Cc Josh Poimboeuf <[email protected]>
---
 arch/x86/include/asm/kvm_host.h | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 92c59cd923b6..a5ae5562ce0a 100644
@@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);
 	"667: \n\t"							\
 	"call	kvm_spurious_fault \n\t"				\
 	"668: \n\t"							\
-	".pushsection .fixup, \"ax\" \n\t"				\

arch/x86/include/asm/kvm_host.h asmlinkage void kvm_spurious_fault(void); · line 1517

Quoted 10 lines
arch/x86/include/asm/kvm_host.h +1 -6
-	"700: \n\t"							\
-	"cmpb	$0, kvm_rebooting\n\t"					\
-	"je	667b \n\t"						\
-	"jmp	668b \n\t"						\
-	".popsection \n\t"						\
-	_ASM_EXTABLE(666b, 700b)
+	_ASM_EXTABLE(666b, 667b)
 
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
#9 lore raw
In reply to Sean Christopherson

Paolo,

Any chance this can be picked up for 5.4? Josh's kvm_spurious_fault() changes went into 5.3, so this can be taken through the KVM tree, which probably makes the most sense since the non-KVM change is a one line deletion.

Quoted 49 lines · Sean Christopherson

A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL to kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.

Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".

The net impact relative to the current code base is more or less a nop when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.

Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive as CC_SET() without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.

A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.

v2:
- Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to __kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the __kvm_handle_fault_on_reboot() changes.

Sean Christopherson (5):
objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop ____kvm_handle_fault_on_reboot()
KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

 arch/x86/include/asm/kvm_host.h | 16 ++----
 arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
 arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
 arch/x86/kvm/x86.c              |  3 +-
 tools/objtool/check.c           |  1 -
 5 files changed, 104 insertions(+), 51 deletions(-)
Signature · 1 line
2.22.0
maintainer Applied
#10 lore raw
In reply to Sean Christopherson
Quoted 39 lines · Sean Christopherson

A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with objtool. An unfortunate side effect is that JMP is now inserted after most VMX instructions so that the reboot macro can use an actual CALL to kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.

Rework the low level VMX instruction helpers to handle unexpected faults manually instead of relying on the "fault on reboot" macro. By using asm-goto, most helpers can branch directly to an in-function call to kvm_spurious_fault(), which can then be optimized by compilers to reside out-of-line at the end of the function instead of inline as done by "fault on reboot".

The net impact relative to the current code base is more or less a nop when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__. A bunch of code that was previously in .fixup gets moved into the slow paths of functions, but the fast paths are more basically unchanged.

Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net positive as CC_SET() without compiler support almost always generates a SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.

A small bonus is that the Jcc instrs are hinted to predict that the VMX instr will be successful.

v2:
- Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
- Reworded changelogs to reference the commit instead lkml link for the recent changes to __kvm_handle_fault_on_reboot().
- Added Paolo's acks for patch 1-4
- Added patch 5 to do more cleanup, which was made possible by rebasing on top of the __kvm_handle_fault_on_reboot() changes.

Sean Christopherson (5):
objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
KVM: VMX: Optimize VMX instruction error and fault handling
KVM: VMX: Add error handling to VMREAD helper
KVM: x86: Drop ____kvm_handle_fault_on_reboot()
KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

Quoted 7 lines
 arch/x86/include/asm/kvm_host.h | 16 ++----
 arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
 arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
 arch/x86/kvm/x86.c              |  3 +-
 tools/objtool/check.c           |  1 -
 5 files changed, 104 insertions(+), 51 deletions(-)

Queued, thanks.

Paolo