Skip to contentSkip to conversation

KVM: SVM: Add exception to disable objtool warning for kvm-amd.o

PATCH In review Nikunj A Dadhania 6 participants 17 messages Aug 2, 2023 – Aug 12, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: SVM: Add exception to disable objtool warning for kvm-amd.o

commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.

objtool gives the following warnings in the newer kernel builds:

arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register

As kvm-amd.o is a link time object, skipping the kvm-amd.o is not possible as per the objtool documentation, better to skip the offending functions.

Functions __svm_vcpu_run() and __svm_sev_es_vcpu_run() saves and restores RBP. Below is the snippet:

    SYM_FUNC_START(__svm_vcpu_run)
        push %_ASM_BP
    <…>
        pop %_ASM_BP
        RET

Add exceptions to skip both these functions. Remove the OBJECT_FILES_NON_STANDARD for vmenter.o

Cc Josh Poimboeuf <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Sean Christopherson <[email protected]>
---
 arch/x86/kvm/Makefile      | 4 ----
 arch/x86/kvm/svm/vmenter.S | 2 ++
 2 files changed, 2 insertions(+), 4 deletions(-)
arch/x86/kvm/Makefile +0 -4
index 80e3fe184d17..0c5c2f090e93 100644
@@ -3,10 +3,6 @@
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
-ifeq ($(CONFIG_FRAME_POINTER),y)
-OBJECT_FILES_NON_STANDARD_vmenter.o := y
-endif
-
 include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +2 -0
index 8e8295e774f0..8fd37d661c33 100644
@@ -289,6 +289,7 @@ SYM_FUNC_START(__svm_vcpu_run)
 	_ASM_EXTABLE(7b, 70b)
 
 SYM_FUNC_END(__svm_vcpu_run)
+STACK_FRAME_NON_STANDARD(__svm_vcpu_run)
 
 /**
  * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode
@@ -388,3 +389,4 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	_ASM_EXTABLE(1b, 3b)
 
 SYM_FUNC_END(__svm_sev_es_vcpu_run)
+STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)
Signature · 1 line
2.34.1
#1 lore raw
Quoted 1 line · Nikunj A Dadhania

objtool gives the following warnings in the newer kernel builds:

Define "newer". As in, exactly what commit is causing problems?

#2 lore raw
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Nikunj A Dadhania

objtool gives the following warnings in the newer kernel builds:

Define "newer". As in, exactly what commit is causing problems?

Not that new it seems, I have tried till v5.19 and the warning is there but with a different signature. Do you want me to further bisect it ?

arch/x86/kvm/kvm-amd.o: warning: objtool: .altinstr_replacement+0x4d: call without frame pointer save/setup arch/x86/kvm/kvm-amd.o: warning: objtool: .altinstr_replacement+0x57: call without frame pointer save/setup

kvm-amd.o warnings were also reported in the below thread:
https://lore.kernel.org/lkml/[email protected]/

Regards,
Nikunj

#3 lore raw
In reply to Nikunj A Dadhania
Quoted 64 lines · Nikunj A Dadhania

commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.

objtool gives the following warnings in the newer kernel builds:

arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register

As kvm-amd.o is a link time object, skipping the kvm-amd.o is not possible as per the objtool documentation, better to skip the offending functions.

Functions __svm_vcpu_run() and __svm_sev_es_vcpu_run() saves and restores RBP. Below is the snippet:

    SYM_FUNC_START(__svm_vcpu_run)
        push %_ASM_BP
    <…>
        pop %_ASM_BP
        RET

Add exceptions to skip both these functions. Remove the OBJECT_FILES_NON_STANDARD for vmenter.o

Cc Josh Poimboeuf <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Sean Christopherson <[email protected]>
---
 arch/x86/kvm/Makefile      | 4 ----
 arch/x86/kvm/svm/vmenter.S | 2 ++
 2 files changed, 2 insertions(+), 4 deletions(-)
arch/x86/kvm/Makefile +0 -4
index 80e3fe184d17..0c5c2f090e93 100644
@@ -3,10 +3,6 @@
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
-ifeq ($(CONFIG_FRAME_POINTER),y)
-OBJECT_FILES_NON_STANDARD_vmenter.o := y
-endif
-
 include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +1 -0
index 8e8295e774f0..8fd37d661c33 100644
@@ -289,6 +289,7 @@ SYM_FUNC_START(__svm_vcpu_run)
 	_ASM_EXTABLE(7b, 70b)
 
 SYM_FUNC_END(__svm_vcpu_run)
+STACK_FRAME_NON_STANDARD(__svm_vcpu_run)
 
 /**
  * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode

arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_sev_es_vcpu_run) · line 392

Quoted 4 lines
arch/x86/kvm/svm/vmenter.S +1 -0
@@ -388,3 +389,4 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	_ASM_EXTABLE(1b, 3b)
 
 SYM_FUNC_END(__svm_sev_es_vcpu_run)
+STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)

Urgh... no, no, this is all broken.

By marking them with STACK_FRAME_NON_STANDARD you will get no ORC data at all, and then you also violate the normal framepointer calling convention.

This means that if you need to unwind here you're up a creek without no paddles on.

Objtool complains for a reason, your changelog does not provide a counter argument for that reason.

Hardware/firmware interfaces that require one to violate basic calling conventions are horrible crap.

maintainer
#4 lore raw
Quoted 7 lines · Peter Zijlstra

By marking them with STACK_FRAME_NON_STANDARD you will get no ORC data at all, and then you also violate the normal framepointer calling convention.

This means that if you need to unwind here you're up a creek without no paddles on.

The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.

If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.

So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.

Paolo

#5 lore raw
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Peter Zijlstra

By marking them with STACK_FRAME_NON_STANDARD you will get no ORC data at all, and then you also violate the normal framepointer calling convention.

Quoted 6 lines
Quoted 2 lines

This means that if you need to unwind here you're up a creek without no paddles on.

The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.

This code is ran with GIF disabled, so NMIs are not in the books, right? Does GIF block #MC ?

Quoted 10 lines

If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.

So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.

But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.

Also, if you look at the makefile hunk that's being replaced, that was conditional on CONFIG_FRAMEPOINTS, while the annotation that's being added is not. I think objtool won't complain for ORC builds, only for framepoints builds.

#6 lore raw
Quoted 28 lines · Peter Zijlstra
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Peter Zijlstra

By marking them with STACK_FRAME_NON_STANDARD you will get no ORC data at all, and then you also violate the normal framepointer calling convention.

Quoted 6 lines
Quoted 2 lines

This means that if you need to unwind here you're up a creek without no paddles on.

The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.

This code is ran with GIF disabled, so NMIs are not in the books, right? Does GIF block #MC ?

Quoted 10 lines

If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.

So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.

Quoted 6 lines

But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.

Also, if you look at the makefile hunk that's being replaced, that was conditional on CONFIG_FRAMEPOINTS, while the annotation that's being added is not.

Even with the hunk present with CONFIG_FRAME_POINTER=y, I was getting the warning. That is why I had added the STACK_FRAME_NON_STANDARD and removed the hunk in makefile.

Do you recommend using the below instead ?

#ifdef CONFIG_FRAME_POINTER
STACK_FRAME_NON_STANDARD()
#endif

Regards
Nikunj

maintainer
#7 lore raw
In reply to Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines

The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.

This code is ran with GIF disabled, so NMIs are not in the books, right?

Yep.

Quoted 1 line

Does GIF block #MC ?

No, #MC is an exception not an interrupt.

Quoted 5 lines
Quoted 2 lines

So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.

But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.

It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).

I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.

Paolo

arch/x86/kvm/svm/vmenter.S +19 -6
index 8e8295e774f0..58fab5e0f7ae 100644
@@ -99,6 +99,8 @@
  */
 SYM_FUNC_START(__svm_vcpu_run)
 	push %_ASM_BP
+	mov %_ASM_SP, %_ASM_BP
+
 #ifdef CONFIG_X86_64
 	push %r15
 	push %r14
@@ -121,7 +123,8 @@ SYM_FUNC_START(__svm_vcpu_run)
 	/* Needed to restore access to percpu variables.  */
 	__ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa)
 
-	/* Finally save @svm. */
+	/* Finally save frame pointer and @svm. */
+	push %_ASM_BP
 	push %_ASM_ARG1
 
 .ifnc _ASM_ARG1, _ASM_DI
@@ -153,7 +156,6 @@ SYM_FUNC_START(__svm_vcpu_run)
 	mov VCPU_RCX(%_ASM_DI), %_ASM_CX
 	mov VCPU_RDX(%_ASM_DI), %_ASM_DX
 	mov VCPU_RBX(%_ASM_DI), %_ASM_BX
-	mov VCPU_RBP(%_ASM_DI), %_ASM_BP
 	mov VCPU_RSI(%_ASM_DI), %_ASM_SI
 #ifdef CONFIG_X86_64
 	mov VCPU_R8 (%_ASM_DI),  %r8
@@ -165,6 +167,7 @@ SYM_FUNC_START(__svm_vcpu_run)
 	mov VCPU_R14(%_ASM_DI), %r14
 	mov VCPU_R15(%_ASM_DI), %r15
 #endif
+	mov VCPU_RBP(%_ASM_DI), %_ASM_BP
 	mov VCPU_RDI(%_ASM_DI), %_ASM_DI
 
 	/* Enter guest mode */
@@ -177,11 +180,15 @@ SYM_FUNC_START(__svm_vcpu_run)
 	/* Pop @svm to RAX while it's the only available register. */
 	pop %_ASM_AX
 
-	/* Save all guest registers.  */
+	/*
+	 * Save all guest registers. Pop the frame pointer as soon as possible
+	 * to enable unwinding.
+	 */
+	mov %_ASM_BP,   VCPU_RBP(%_ASM_AX)
+	pop %_ASM_BP
 	mov %_ASM_CX,   VCPU_RCX(%_ASM_AX)
 	mov %_ASM_DX,   VCPU_RDX(%_ASM_AX)
 	mov %_ASM_BX,   VCPU_RBX(%_ASM_AX)
-	mov %_ASM_BP,   VCPU_RBP(%_ASM_AX)
 	mov %_ASM_SI,   VCPU_RSI(%_ASM_AX)
 	mov %_ASM_DI,   VCPU_RDI(%_ASM_AX)
 #ifdef CONFIG_X86_64
@@ -297,6 +304,7 @@ SYM_FUNC_END(__svm_vcpu_run)
  */
 SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	push %_ASM_BP
+	mov %_ASM_SP, %_ASM_BP
 #ifdef CONFIG_X86_64
 	push %r15
 	push %r14
@@ -316,7 +324,8 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	/* Accessed directly from the stack in RESTORE_HOST_SPEC_CTRL.  */
 	push %_ASM_ARG2
 
-	/* Save @svm. */
+	/* Save frame pointer and @svm. */
+	push %_ASM_BP
 	push %_ASM_ARG1
 
 .ifnc _ASM_ARG1, _ASM_DI
@@ -341,8 +350,12 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 
 2:	cli
 
-	/* Pop @svm to RDI, guest registers have been saved already. */
+	/*
+	 * Guest registers have been saved already.
+	 * Pop @svm to RDI and restore the frame pointer to allow unwinding.
+	 */
 	pop %_ASM_DI
+	pop %_ASM_BP
 
 #ifdef CONFIG_RETPOLINE
 	/* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
#8 lore raw
Quoted 6 lines · Paolo Bonzini

It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).

I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.

Yes, that's very helpful, the below is your patch with a few extra hints and a comment. This seems to cure things.

Specifically, your change is needed to put UNWIND_HINT_RESTORE before we go CALL things (like entry_ibpb), otherwise objtool gets upset we CALL without having a valid framepointer.

Josh, this look ok to you?

---
 arch/x86/kvm/Makefile      |  4 ----
 arch/x86/kvm/svm/vmenter.S | 38 ++++++++++++++++++++++++++++++++------
 2 files changed, 32 insertions(+), 10 deletions(-)
arch/x86/kvm/Makefile +0 -4
index 80e3fe184d17..0c5c2f090e93 100644
@@ -3,10 +3,6 @@
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
-ifeq ($(CONFIG_FRAME_POINTER),y)
-OBJECT_FILES_NON_STANDARD_vmenter.o := y
-endif
-
 include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +32 -6
index 8e8295e774f0..99b9be9a56c3 100644
@@ -99,6 +99,8 @@
  */
 SYM_FUNC_START(__svm_vcpu_run)
 	push %_ASM_BP
+	mov %_ASM_SP, %_ASM_BP
+
 #ifdef CONFIG_X86_64
 	push %r15
 	push %r14
@@ -121,7 +123,18 @@ SYM_FUNC_START(__svm_vcpu_run)
 	/* Needed to restore access to percpu variables.  */
 	__ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa)
 
-	/* Finally save @svm. */
+	/*
+	 * Finally save frame pointer and @svm.
+	 *
+	 * Clobbering BP here is mostly ok since GIF will block NMIs and with
+	 * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below
+	 * nothing untoward will happen until BP is restored.
+	 *
+	 * The kvm_rebooting exceptions should not want to unwind stack, and
+	 * while #MV might want to unwind stack, it is ultimately fatal.
+	 */
+	UNWIND_HINT_SAVE
+	push %_ASM_BP
 	push %_ASM_ARG1
 
 .ifnc _ASM_ARG1, _ASM_DI
@@ -153,7 +166,6 @@ SYM_FUNC_START(__svm_vcpu_run)
 	mov VCPU_RCX(%_ASM_DI), %_ASM_CX
 	mov VCPU_RDX(%_ASM_DI), %_ASM_DX
 	mov VCPU_RBX(%_ASM_DI), %_ASM_BX
-	mov VCPU_RBP(%_ASM_DI), %_ASM_BP
 	mov VCPU_RSI(%_ASM_DI), %_ASM_SI
 #ifdef CONFIG_X86_64
 	mov VCPU_R8 (%_ASM_DI),  %r8
@@ -165,6 +177,7 @@ SYM_FUNC_START(__svm_vcpu_run)
 	mov VCPU_R14(%_ASM_DI), %r14
 	mov VCPU_R15(%_ASM_DI), %r15
 #endif
+	mov VCPU_RBP(%_ASM_DI), %_ASM_BP
 	mov VCPU_RDI(%_ASM_DI), %_ASM_DI
 
 	/* Enter guest mode */
@@ -177,11 +190,16 @@ SYM_FUNC_START(__svm_vcpu_run)
 	/* Pop @svm to RAX while it's the only available register. */
 	pop %_ASM_AX
 
-	/* Save all guest registers.  */
+	/*
+	 * Save all guest registers. Pop the frame pointer as soon as possible
+	 * to enable unwinding.
+	 */
+	mov %_ASM_BP,   VCPU_RBP(%_ASM_AX)
+	pop %_ASM_BP
+	UNWIND_HINT_RESTORE
 	mov %_ASM_CX,   VCPU_RCX(%_ASM_AX)
 	mov %_ASM_DX,   VCPU_RDX(%_ASM_AX)
 	mov %_ASM_BX,   VCPU_RBX(%_ASM_AX)
-	mov %_ASM_BP,   VCPU_RBP(%_ASM_AX)
 	mov %_ASM_SI,   VCPU_RSI(%_ASM_AX)
 	mov %_ASM_DI,   VCPU_RDI(%_ASM_AX)
 #ifdef CONFIG_X86_64
@@ -297,6 +315,7 @@ SYM_FUNC_END(__svm_vcpu_run)
  */
 SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	push %_ASM_BP
+	mov %_ASM_SP, %_ASM_BP
 #ifdef CONFIG_X86_64
 	push %r15
 	push %r14
@@ -316,7 +335,9 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 	/* Accessed directly from the stack in RESTORE_HOST_SPEC_CTRL.  */
 	push %_ASM_ARG2
 
-	/* Save @svm. */
+	/* Save frame pointer and @svm. */
+	UNWIND_HINT_SAVE
+	push %_ASM_BP
 	push %_ASM_ARG1
 
 .ifnc _ASM_ARG1, _ASM_DI
@@ -341,8 +362,13 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run)
 
 2:	cli
 
-	/* Pop @svm to RDI, guest registers have been saved already. */
+	/*
+	 * Guest registers have been saved already.
+	 * Pop @svm to RDI and restore the frame pointer to allow unwinding.
+	 */
 	pop %_ASM_DI
+	pop %_ASM_BP
+	UNWIND_HINT_RESTORE
 
 #ifdef CONFIG_RETPOLINE
 	/* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
#9 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Paolo Bonzini

It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).

I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.

Quoted 8 lines

Yes, that's very helpful, the below is your patch with a few extra hints and a comment. This seems to cure things.

Specifically, your change is needed to put UNWIND_HINT_RESTORE before we go CALL things (like entry_ibpb), otherwise objtool gets upset we CALL without having a valid framepointer.

Josh, this look ok to you?

Looks mostly right, except this now creates an unnecessary gap in unwinding coverage for the ORC unwinder. So it's better to put the FP-specific changes behind CONFIG_FRAME_POINTER:

arch/x86/kvm/Makefile +0 -4
index 80e3fe184d17..0c5c2f090e93 100644
@@ -3,10 +3,6 @@
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
-ifeq ($(CONFIG_FRAME_POINTER),y)
-OBJECT_FILES_NON_STANDARD_vmenter.o := y
-endif
-
 include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +43 -6

101 lines not on this page · raw message

#10 lore raw
Quoted 3 lines · Josh Poimboeuf

Looks mostly right, except this now creates an unnecessary gap in unwinding coverage for the ORC unwinder. So it's better to put the FP-specific changes behind CONFIG_FRAME_POINTER:

Fair enough.

Quoted 34 lines
arch/x86/kvm/Makefile +0 -4
index 80e3fe184d17..0c5c2f090e93 100644
@@ -3,10 +3,6 @@
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
-ifeq ($(CONFIG_FRAME_POINTER),y)
-OBJECT_FILES_NON_STANDARD_vmenter.o := y
-endif
-
 include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +3 -1
index 8e8295e774f0..51f6851b1ae5 100644
@@ -99,6 +99,9 @@
  */
 SYM_FUNC_START(__svm_vcpu_run)
 	push %_ASM_BP
+#ifdef CONFIG_FRAME_POINTER
+	mov %_ASM_SP, %_ASM_BP
+#endif
 #ifdef CONFIG_X86_64
 	push %r15
 	push %r14
@@ -121,7 +124,20 @@ SYM_FUNC_START(__svm_vcpu_run)
 	/* Needed to restore access to percpu variables.  */
 	__ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa)
 
-	/* Finally save @svm. */

arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_vcpu_run) · line 136

Quoted 10 lines
arch/x86/kvm/svm/vmenter.S +10 -0
+	/*
+	 * Finally save frame pointer and @svm.
+	 *
+	 * Clobbering BP here is mostly ok since GIF will block NMIs and with
+	 * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below
+	 * nothing untoward will happen until BP is restored.
+	 *
+	 * The kvm_rebooting exceptions should not want to unwind stack, and
+	 * while #MV might want to unwind stack, it is ultimately fatal.
+	 */

Aside from me not being able to type #MC, I did realize that the kvm_reboot exception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.

maintainer
#11 lore raw
Quoted 10 lines · Peter Zijlstra

Review

Quoted 7 lines
(quoted diff) +7 -0
+	 * Clobbering BP here is mostly ok since GIF will block NMIs and with
+	 * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below
+	 * nothing untoward will happen until BP is restored.
+	 *
+	 * The kvm_rebooting exceptions should not want to unwind stack, and
+	 * while #MV might want to unwind stack, it is ultimately fatal.
+	 */

Aside from me not being able to type #MC, I did realize that the kvm_reboot exception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.

Asynchronously disabling SVM requires an IPI, so kvm_rebooting cannot change within CLGI/STGI. We can check it after CLGI instead of waiting for a #GP:

arch/x86/kvm/svm/svm.c +4 -1
index 956726d867aa..e3755f5eaf81 100644
@@ -4074,7 +4074,10 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 	if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
 		x86_spec_ctrl_set_guest(svm->virt_spec_ctrl);
 
-	svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted);
+	if (unlikely(kvm_rebooting))
+		svm->vmcb->control.exit_code = SVM_EXIT_PAUSE;
+	else
+		svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted);
 
 	if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
 		x86_spec_ctrl_restore_host(svm->virt_spec_ctrl);
arch/x86/kvm/svm/vmenter.S +4 -15
index 8e8295e774f0..34641b3a6823 100644
@@ -270,23 +270,12 @@ SYM_FUNC_START(__svm_vcpu_run)
 	RESTORE_GUEST_SPEC_CTRL_BODY
 	RESTORE_HOST_SPEC_CTRL_BODY
 
-10:	cmpb $0, kvm_rebooting
-	jne 2b
-	ud2
-30:	cmpb $0, kvm_rebooting
-	jne 4b
-	ud2
-50:	cmpb $0, kvm_rebooting
-	jne 6b
-	ud2
-70:	cmpb $0, kvm_rebooting
-	jne 8b
-	ud2
+10:	ud2
 
 	_ASM_EXTABLE(1b, 10b)
-	_ASM_EXTABLE(3b, 30b)
-	_ASM_EXTABLE(5b, 50b)
-	_ASM_EXTABLE(7b, 70b)
+	_ASM_EXTABLE(3b, 10b)
+	_ASM_EXTABLE(5b, 10b)
+	_ASM_EXTABLE(7b, 10b)
 
 SYM_FUNC_END(__svm_vcpu_run)
 

Paolo

#12 lore raw
Quoted 12 lines · Paolo Bonzini
Quoted 10 lines · Peter Zijlstra

Review

Quoted 7 lines
(quoted diff) +7 -0
+	 * Clobbering BP here is mostly ok since GIF will block NMIs and with
+	 * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below
+	 * nothing untoward will happen until BP is restored.
+	 *
+	 * The kvm_rebooting exceptions should not want to unwind stack, and
+	 * while #MV might want to unwind stack, it is ultimately fatal.
+	 */

Aside from me not being able to type #MC, I did realize that the kvm_reboot exception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.

Quoted 2 lines

Asynchronously disabling SVM requires an IPI, so kvm_rebooting cannot change
within CLGI/STGI. We can check it after CLGI instead of waiting for a #GP:

Seems fair; thanks!

Quoted 52 lines
arch/x86/kvm/svm/svm.c +4 -1
index 956726d867aa..e3755f5eaf81 100644
@@ -4074,7 +4074,10 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct
 kvm_vcpu *vcpu)
 	if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
 		x86_spec_ctrl_set_guest(svm->virt_spec_ctrl);
 
-	svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted);
+	if (unlikely(kvm_rebooting))
+		svm->vmcb->control.exit_code = SVM_EXIT_PAUSE;
+	else
+		svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted);
 
 	if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
arch/x86/kvm/svm/svm.c +0 -0
 		x86_spec_ctrl_restore_host(svm->virt_spec_ctrl);
arch/x86/kvm/svm/vmenter.S +4 -15
index 8e8295e774f0..34641b3a6823 100644
@@ -270,23 +270,12 @@ SYM_FUNC_START(__svm_vcpu_run)
 	RESTORE_GUEST_SPEC_CTRL_BODY
 	RESTORE_HOST_SPEC_CTRL_BODY
 
-10:	cmpb $0, kvm_rebooting
-	jne 2b
-	ud2
-30:	cmpb $0, kvm_rebooting
-	jne 4b
-	ud2
-50:	cmpb $0, kvm_rebooting
-	jne 6b
-	ud2
-70:	cmpb $0, kvm_rebooting
-	jne 8b
-	ud2
+10:	ud2
 
 	_ASM_EXTABLE(1b, 10b)
-	_ASM_EXTABLE(3b, 30b)
-	_ASM_EXTABLE(5b, 50b)
-	_ASM_EXTABLE(7b, 70b)
+	_ASM_EXTABLE(3b, 10b)
+	_ASM_EXTABLE(5b, 10b)
+	_ASM_EXTABLE(7b, 10b)
 
 SYM_FUNC_END(__svm_vcpu_run)
 

Paolo

#13 lore raw
In reply to Nikunj A Dadhania
Quoted 8 lines · Nikunj A Dadhania

commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.

objtool gives the following warnings in the newer kernel builds:

arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register

I wanted to poke around a little, but can't reproduce this.

I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?

#14 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Nikunj A Dadhania

commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.

objtool gives the following warnings in the newer kernel builds:

Quoted 8 lines
Quoted 3 lines

arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register

I wanted to poke around a little, but can't reproduce this.

I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?

I have attached the config.

Regards,
Nikunj

#15 lore raw
Quoted 11 lines · Nikunj A. Dadhania
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Nikunj A Dadhania

commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.

objtool gives the following warnings in the newer kernel builds:

arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register

Quoted 6 lines
Quoted 4 lines

I wanted to poke around a little, but can't reproduce this.

I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?

I have attached the config.

Thanks, that does reproduce, although now I'm left wondering what exactly makes it go beep. Also, URGH, I thought the build was broken until I noticed it had BTF on :/

#16 lore raw
In reply to Nikunj A Dadhania
kernel test robot noticed the following build errors:

Hi Nikunj,

kernel test robot noticed the following build errors:

[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Nikunj-A-Dadhania/KVM-SVM-Add-exception-to-disable-objtool-warning-for-kvm-amd-o/20230802-171219
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20230802091107.1160320-1-nikunj%40amd.com
patch subject: [PATCH] KVM: SVM: Add exception to disable objtool warning for kvm-amd.o
config: i386-randconfig-i013-20230812 (https://download.01.org/0day-ci/archive/20230812/[email protected]/config)
compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07)
reproduce: (https://download.01.org/0day-ci/archive/20230812/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> arch/x86/kvm/svm/vmenter.S:392:1: error: invalid instruction mnemonic 'stack_frame_non_standard_fp'

STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run) ^~~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/stack_frame_non_standard_fp +392 arch/x86/kvm/svm/vmenter.S

352
353 /* Clobbers RAX, RCX, RDX. */
354 RESTORE_HOST_SPEC_CTRL
355
356 /*
357 * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be
358 * untrained as soon as we exit the VM and are back to the
359 * kernel. This should be done before re-enabling interrupts
360 * because interrupt handlers won't sanitize RET if the return is
361 * from the kernel.
362 */
363 UNTRAIN_RET
364
365 /* "Pop" @spec_ctrl_intercepted. */
366 pop %_ASM_BX
367
368 pop %_ASM_BX
369
370 #ifdef CONFIG_X86_64
371 pop %r12
372 pop %r13
373 pop %r14
374 pop %r15
375 #else
376 pop %esi
377 pop %edi
378 #endif
379 pop %_ASM_BP
380 RET
381
382 RESTORE_GUEST_SPEC_CTRL_BODY
383 RESTORE_HOST_SPEC_CTRL_BODY
384
385 3: cmpb $0, kvm_rebooting
386 jne 2b
387 ud2
388
389 _ASM_EXTABLE(1b, 3b)
390
391 SYM_FUNC_END(__svm_sev_es_vcpu_run)

 > 392	STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki