KVM: SVM: Add exception to disable objtool warning for kvm-amd.o
PATCH 1/1 KVM: SVM: Add exception to disable objtool warning for kvm-amd.o
commit 7f4b5cde2409 ("kvm: Disable objtool frame pointer checking for vmenter.S") had added the vmenter.o file to the exception list.
objtool gives the following warnings in the newer kernel builds:
arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_vcpu_run+0x17d: BP used as a scratch register arch/x86/kvm/kvm-amd.o: warning: objtool: __svm_sev_es_vcpu_run+0x72: BP used as a scratch register
As kvm-amd.o is a link time object, skipping the kvm-amd.o is not possible as per the objtool documentation, better to skip the offending functions.
Functions __svm_vcpu_run() and __svm_sev_es_vcpu_run() saves and restores RBP. Below is the snippet:
SYM_FUNC_START(__svm_vcpu_run)
push %_ASM_BP
<…>
pop %_ASM_BP
RETAdd exceptions to skip both these functions. Remove the OBJECT_FILES_NON_STANDARD for vmenter.o
--- arch/x86/kvm/Makefile | 4 ---- arch/x86/kvm/svm/vmenter.S | 2 ++ 2 files changed, 2 insertions(+), 4 deletions(-)
arch/x86/kvm/Makefile +0 -4
@@ -3,10 +3,6 @@ ccflags-y += -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) += -Werror -ifeq ($(CONFIG_FRAME_POINTER),y) -OBJECT_FILES_NON_STANDARD_vmenter.o := y -endif - include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +2 -0
@@ -289,6 +289,7 @@ SYM_FUNC_START(__svm_vcpu_run) _ASM_EXTABLE(7b, 70b) SYM_FUNC_END(__svm_vcpu_run) +STACK_FRAME_NON_STANDARD(__svm_vcpu_run) /** * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode
@@ -388,3 +389,4 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) _ASM_EXTABLE(1b, 3b) SYM_FUNC_END(__svm_sev_es_vcpu_run) +STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)
Signature · 1 line
2.34.1
Sean Christopherson
maintainerQuoted 1 line · Nikunj A Dadhania
objtool gives the following warnings in the newer kernel builds:
Define "newer". As in, exactly what commit is causing problems?
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Nikunj A Dadhania
objtool gives the following warnings in the newer kernel builds:
Define "newer". As in, exactly what commit is causing problems?
Not that new it seems, I have tried till v5.19 and the warning is there but with a different signature. Do you want me to further bisect it ?
arch/x86/kvm/kvm-amd.o: warning: objtool: .altinstr_replacement+0x4d: call without frame pointer save/setup arch/x86/kvm/kvm-amd.o: warning: objtool: .altinstr_replacement+0x57: call without frame pointer save/setup
kvm-amd.o warnings were also reported in the below thread:
https://lore.kernel.org/lkml/[email protected]/
Regards,
Nikunj
Quoted 64 lines · Nikunj A Dadhania
commit
7f4b5cde2409("kvm: Disable objtool frame pointer checking forvmenter.S") had added the vmenter.o file to the exception list.objtool gives the following warnings in the newer kernel builds:
arch/x86/kvm/kvm-amd.o: warning: objtool:__svm_vcpu_run+0x17d: BP used as a scratch registerarch/x86/kvm/kvm-amd.o: warning: objtool:__svm_sev_es_vcpu_run+0x72: BP used as a scratch registerAs kvm-amd.o is a link time object, skipping the kvm-amd.o is not possible as per the objtool documentation, better to skip the offending functions.
Functions
__svm_vcpu_run()and__svm_sev_es_vcpu_run()saves and restores RBP. Below is the snippet:SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP <…> pop %_ASM_BP RETAdd exceptions to skip both these functions. Remove the
OBJECT_FILES_NON_STANDARDfor vmenter.oCc Josh Poimboeuf <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Randy Dunlap <[email protected]>Cc Tom Lendacky <[email protected]>Cc Sean Christopherson <[email protected]>Reported-by Ravi Bangoria <[email protected]>Signed-off-by Nikunj A Dadhania <[email protected]>--- arch/x86/kvm/Makefile | 4 ---- arch/x86/kvm/svm/vmenter.S | 2 ++ 2 files changed, 2 insertions(+), 4 deletions(-)arch/x86/kvm/Makefile +0
-4@@ -3,10 +3,6 @@ ccflags-y += -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) += -Werror -ifeq ($(CONFIG_FRAME_POINTER),y) -OBJECT_FILES_NON_STANDARD_vmenter.o := y -endif - include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \arch/x86/kvm/svm/vmenter.S +1
-0@@ -289,6 +289,7 @@ SYM_FUNC_START(__svm_vcpu_run) _ASM_EXTABLE(7b, 70b) SYM_FUNC_END(__svm_vcpu_run) +STACK_FRAME_NON_STANDARD(__svm_vcpu_run) /** * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode
arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_sev_es_vcpu_run) · line 392
Quoted 4 lines
arch/x86/kvm/svm/vmenter.S +1
-0@@ -388,3 +389,4 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) _ASM_EXTABLE(1b, 3b) SYM_FUNC_END(__svm_sev_es_vcpu_run) +STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)
Urgh... no, no, this is all broken.
By marking them with STACK_FRAME_NON_STANDARD you will get no ORC data at all, and then you also violate the normal framepointer calling convention.
This means that if you need to unwind here you're up a creek without no paddles on.
Objtool complains for a reason, your changelog does not provide a counter argument for that reason.
Hardware/firmware interfaces that require one to violate basic calling conventions are horrible crap.
Paolo Bonzini
maintainerQuoted 7 lines · Peter Zijlstra
By marking them with
STACK_FRAME_NON_STANDARDyou will get no ORC data at all, and then you also violate the normal framepointer calling convention.This means that if you need to unwind here you're up a creek without no paddles on.
The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.
If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.
So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.
Paolo
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Peter Zijlstra
By marking them with
STACK_FRAME_NON_STANDARDyou will get no ORC data at all, and then you also violate the normal framepointer calling convention.
Quoted 6 lines
Quoted 2 lines
This means that if you need to unwind here you're up a creek without no paddles on.
The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.
This code is ran with GIF disabled, so NMIs are not in the books, right? Does GIF block #MC ?
Quoted 10 lines
If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.
So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.
But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.
Also, if you look at the makefile hunk that's being replaced, that was conditional on CONFIG_FRAMEPOINTS, while the annotation that's being added is not. I think objtool won't complain for ORC builds, only for framepoints builds.
Quoted 28 lines · Peter Zijlstra
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Peter Zijlstra
By marking them with
STACK_FRAME_NON_STANDARDyou will get no ORC data at all, and then you also violate the normal framepointer calling convention.Quoted 6 lines
Quoted 2 lines
This means that if you need to unwind here you're up a creek without no paddles on.
The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.
This code is ran with GIF disabled, so NMIs are not in the books, right? Does GIF block #MC ?
Quoted 10 lines
If I understand correctly those ud2 would use ORC information to show the backtrace, but even then the frame pointer should be correct. Of these instructions, vmrun is the only one that runs with wrong %rbp; and it is unlikely or even impossible that a #GP happens at vmrun, because the same operand has been used for a vmload ten instructions before. The only time I saw that #GP it was due to a processor errata, but it happened consistently on the vmload.
So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.
Quoted 6 lines
But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.
Also, if you look at the makefile hunk that's being replaced, that was conditional on
CONFIG_FRAMEPOINTS, while the annotation that's being added is not.
Even with the hunk present with CONFIG_FRAME_POINTER=y, I was getting the warning. That is why I had added the STACK_FRAME_NON_STANDARD and removed the hunk in makefile.
Do you recommend using the below instead ?
#ifdef CONFIG_FRAME_POINTERSTACK_FRAME_NON_STANDARD()
#endif
Regards
Nikunj
Paolo Bonzini
maintainerQuoted 5 lines · Peter Zijlstra
Quoted 3 lines
The only weird thing that can happen is ud2 instructions that are executed in case the vmload/vmrun/vmsave instructions causes a #GP, from the exception handler.
This code is ran with GIF disabled, so NMIs are not in the books, right?
Yep.
Quoted 1 line
Does GIF block #MC ?
No, #MC is an exception not an interrupt.
Quoted 5 lines
Quoted 2 lines
So if frame pointer unwinding can be used in the absence of ORC, Nikunj patch should not break anything.
But framepointer unwinds rely on BP, and that is clobbered per the objtool complaint.
It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).
I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.
Paolo
arch/x86/kvm/svm/vmenter.S +19 -6
@@ -99,6 +99,8 @@ */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP + mov %_ASM_SP, %_ASM_BP + #ifdef CONFIG_X86_64 push %r15 push %r14
@@ -121,7 +123,8 @@ SYM_FUNC_START(__svm_vcpu_run) /* Needed to restore access to percpu variables. */ __ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa) - /* Finally save @svm. */ + /* Finally save frame pointer and @svm. */ + push %_ASM_BP push %_ASM_ARG1 .ifnc _ASM_ARG1, _ASM_DI
@@ -153,7 +156,6 @@ SYM_FUNC_START(__svm_vcpu_run) mov VCPU_RCX(%_ASM_DI), %_ASM_CX mov VCPU_RDX(%_ASM_DI), %_ASM_DX mov VCPU_RBX(%_ASM_DI), %_ASM_BX - mov VCPU_RBP(%_ASM_DI), %_ASM_BP mov VCPU_RSI(%_ASM_DI), %_ASM_SI #ifdef CONFIG_X86_64 mov VCPU_R8 (%_ASM_DI), %r8
@@ -165,6 +167,7 @@ SYM_FUNC_START(__svm_vcpu_run) mov VCPU_R14(%_ASM_DI), %r14 mov VCPU_R15(%_ASM_DI), %r15 #endif + mov VCPU_RBP(%_ASM_DI), %_ASM_BP mov VCPU_RDI(%_ASM_DI), %_ASM_DI /* Enter guest mode */
@@ -177,11 +180,15 @@ SYM_FUNC_START(__svm_vcpu_run) /* Pop @svm to RAX while it's the only available register. */ pop %_ASM_AX - /* Save all guest registers. */ + /* + * Save all guest registers. Pop the frame pointer as soon as possible + * to enable unwinding. + */ + mov %_ASM_BP, VCPU_RBP(%_ASM_AX) + pop %_ASM_BP mov %_ASM_CX, VCPU_RCX(%_ASM_AX) mov %_ASM_DX, VCPU_RDX(%_ASM_AX) mov %_ASM_BX, VCPU_RBX(%_ASM_AX) - mov %_ASM_BP, VCPU_RBP(%_ASM_AX) mov %_ASM_SI, VCPU_RSI(%_ASM_AX) mov %_ASM_DI, VCPU_RDI(%_ASM_AX) #ifdef CONFIG_X86_64
@@ -297,6 +304,7 @@ SYM_FUNC_END(__svm_vcpu_run) */ SYM_FUNC_START(__svm_sev_es_vcpu_run) push %_ASM_BP + mov %_ASM_SP, %_ASM_BP #ifdef CONFIG_X86_64 push %r15 push %r14
@@ -316,7 +324,8 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* Accessed directly from the stack in RESTORE_HOST_SPEC_CTRL. */ push %_ASM_ARG2 - /* Save @svm. */ + /* Save frame pointer and @svm. */ + push %_ASM_BP push %_ASM_ARG1 .ifnc _ASM_ARG1, _ASM_DI
@@ -341,8 +350,12 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) 2: cli - /* Pop @svm to RDI, guest registers have been saved already. */ + /* + * Guest registers have been saved already. + * Pop @svm to RDI and restore the frame pointer to allow unwinding. + */ pop %_ASM_DI + pop %_ASM_BP #ifdef CONFIG_RETPOLINE /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
Quoted 6 lines · Paolo Bonzini
It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).
I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.
Yes, that's very helpful, the below is your patch with a few extra hints and a comment. This seems to cure things.
Specifically, your change is needed to put UNWIND_HINT_RESTORE before we go CALL things (like entry_ibpb), otherwise objtool gets upset we CALL without having a valid framepointer.
Josh, this look ok to you?
--- arch/x86/kvm/Makefile | 4 ---- arch/x86/kvm/svm/vmenter.S | 38 ++++++++++++++++++++++++++++++++------ 2 files changed, 32 insertions(+), 10 deletions(-)
arch/x86/kvm/Makefile +0 -4
@@ -3,10 +3,6 @@ ccflags-y += -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) += -Werror -ifeq ($(CONFIG_FRAME_POINTER),y) -OBJECT_FILES_NON_STANDARD_vmenter.o := y -endif - include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +32 -6
@@ -99,6 +99,8 @@ */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP + mov %_ASM_SP, %_ASM_BP + #ifdef CONFIG_X86_64 push %r15 push %r14
@@ -121,7 +123,18 @@ SYM_FUNC_START(__svm_vcpu_run) /* Needed to restore access to percpu variables. */ __ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa) - /* Finally save @svm. */ + /* + * Finally save frame pointer and @svm. + * + * Clobbering BP here is mostly ok since GIF will block NMIs and with + * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below + * nothing untoward will happen until BP is restored. + * + * The kvm_rebooting exceptions should not want to unwind stack, and + * while #MV might want to unwind stack, it is ultimately fatal. + */ + UNWIND_HINT_SAVE + push %_ASM_BP push %_ASM_ARG1 .ifnc _ASM_ARG1, _ASM_DI
@@ -153,7 +166,6 @@ SYM_FUNC_START(__svm_vcpu_run) mov VCPU_RCX(%_ASM_DI), %_ASM_CX mov VCPU_RDX(%_ASM_DI), %_ASM_DX mov VCPU_RBX(%_ASM_DI), %_ASM_BX - mov VCPU_RBP(%_ASM_DI), %_ASM_BP mov VCPU_RSI(%_ASM_DI), %_ASM_SI #ifdef CONFIG_X86_64 mov VCPU_R8 (%_ASM_DI), %r8
@@ -165,6 +177,7 @@ SYM_FUNC_START(__svm_vcpu_run) mov VCPU_R14(%_ASM_DI), %r14 mov VCPU_R15(%_ASM_DI), %r15 #endif + mov VCPU_RBP(%_ASM_DI), %_ASM_BP mov VCPU_RDI(%_ASM_DI), %_ASM_DI /* Enter guest mode */
@@ -177,11 +190,16 @@ SYM_FUNC_START(__svm_vcpu_run) /* Pop @svm to RAX while it's the only available register. */ pop %_ASM_AX - /* Save all guest registers. */ + /* + * Save all guest registers. Pop the frame pointer as soon as possible + * to enable unwinding. + */ + mov %_ASM_BP, VCPU_RBP(%_ASM_AX) + pop %_ASM_BP + UNWIND_HINT_RESTORE mov %_ASM_CX, VCPU_RCX(%_ASM_AX) mov %_ASM_DX, VCPU_RDX(%_ASM_AX) mov %_ASM_BX, VCPU_RBX(%_ASM_AX) - mov %_ASM_BP, VCPU_RBP(%_ASM_AX) mov %_ASM_SI, VCPU_RSI(%_ASM_AX) mov %_ASM_DI, VCPU_RDI(%_ASM_AX) #ifdef CONFIG_X86_64
@@ -297,6 +315,7 @@ SYM_FUNC_END(__svm_vcpu_run) */ SYM_FUNC_START(__svm_sev_es_vcpu_run) push %_ASM_BP + mov %_ASM_SP, %_ASM_BP #ifdef CONFIG_X86_64 push %r15 push %r14
@@ -316,7 +335,9 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* Accessed directly from the stack in RESTORE_HOST_SPEC_CTRL. */ push %_ASM_ARG2 - /* Save @svm. */ + /* Save frame pointer and @svm. */ + UNWIND_HINT_SAVE + push %_ASM_BP push %_ASM_ARG1 .ifnc _ASM_ARG1, _ASM_DI
@@ -341,8 +362,13 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) 2: cli - /* Pop @svm to RDI, guest registers have been saved already. */ + /* + * Guest registers have been saved already. + * Pop @svm to RDI and restore the frame pointer to allow unwinding. + */ pop %_ASM_DI + pop %_ASM_BP + UNWIND_HINT_RESTORE #ifdef CONFIG_RETPOLINE /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Paolo Bonzini
It's not clobbered in a part that will cause unwinding; we can further restrict the part to a handful of instructions (and add a mov %rsp, %rbp at the top, see untested patch after signature).
I think the chance of this failure is similar or lower to the chance of a memory failure that hits the exception handler code itself.
Quoted 8 lines
Yes, that's very helpful, the below is your patch with a few extra hints and a comment. This seems to cure things.
Specifically, your change is needed to put
UNWIND_HINT_RESTOREbefore we go CALL things (likeentry_ibpb), otherwise objtool gets upset we CALL without having a valid framepointer.Josh, this look ok to you?
Looks mostly right, except this now creates an unnecessary gap in unwinding coverage for the ORC unwinder. So it's better to put the FP-specific changes behind CONFIG_FRAME_POINTER:
arch/x86/kvm/Makefile +0 -4
@@ -3,10 +3,6 @@ ccflags-y += -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) += -Werror -ifeq ($(CONFIG_FRAME_POINTER),y) -OBJECT_FILES_NON_STANDARD_vmenter.o := y -endif - include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \
arch/x86/kvm/svm/vmenter.S +43 -6
101 lines not on this page · raw message
Quoted 3 lines · Josh Poimboeuf
Looks mostly right, except this now creates an unnecessary gap in unwinding coverage for the ORC unwinder. So it's better to put the FP-specific changes behind
CONFIG_FRAME_POINTER:
Fair enough.
Quoted 34 lines
arch/x86/kvm/Makefile +0
-4@@ -3,10 +3,6 @@ ccflags-y += -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) += -Werror -ifeq ($(CONFIG_FRAME_POINTER),y) -OBJECT_FILES_NON_STANDARD_vmenter.o := y -endif - include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \arch/x86/kvm/svm/vmenter.S +3
-1@@ -99,6 +99,9 @@ */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP +#ifdef CONFIG_FRAME_POINTER + mov %_ASM_SP, %_ASM_BP +#endif #ifdef CONFIG_X86_64 push %r15 push %r14@@ -121,7 +124,20 @@ SYM_FUNC_START(__svm_vcpu_run) /* Needed to restore access to percpu variables. */ __ASM_SIZE(push) PER_CPU_VAR(svm_data + SD_save_area_pa) - /* Finally save @svm. */
arch/x86/kvm/svm/vmenter.S SYM_FUNC_START(__svm_vcpu_run) · line 136
Quoted 10 lines
arch/x86/kvm/svm/vmenter.S +10
-0+ /* + * Finally save frame pointer and @svm. + * + * Clobbering BP here is mostly ok since GIF will block NMIs and with + * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below + * nothing untoward will happen until BP is restored. + * + * The kvm_rebooting exceptions should not want to unwind stack, and + * while #MV might want to unwind stack, it is ultimately fatal. + */
Aside from me not being able to type #MC, I did realize that the kvm_reboot exception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.
Paolo Bonzini
maintainerQuoted 10 lines · Peter Zijlstra
Review
Quoted 7 lines
(quoted diff) +7
-0+ * Clobbering BP here is mostly ok since GIF will block NMIs and with + * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below + * nothing untoward will happen until BP is restored. + * + * The kvm_rebooting exceptions should not want to unwind stack, and + * while #MV might want to unwind stack, it is ultimately fatal. + */Aside from me not being able to type #MC, I did realize that the
kvm_rebootexception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.
Asynchronously disabling SVM requires an IPI, so kvm_rebooting cannot change within CLGI/STGI. We can check it after CLGI instead of waiting for a #GP:
arch/x86/kvm/svm/svm.c +4 -1
@@ -4074,7 +4074,10 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)if (!static_cpu_has (X86_FEATURE_V_SPEC_CTRL ))x86_spec_ctrl_set_guest (svm->virt_spec_ctrl); -svm_vcpu_enter_exit (vcpu, spec_ctrl_intercepted); +if (unlikely (kvm_rebooting)) + svm->vmcb->control.exit_code =SVM_EXIT_PAUSE ; +else +svm_vcpu_enter_exit (vcpu, spec_ctrl_intercepted);if (!static_cpu_has (X86_FEATURE_V_SPEC_CTRL ))x86_spec_ctrl_restore_host (svm->virt_spec_ctrl);
arch/x86/kvm/svm/vmenter.S +4 -15
@@ -270,23 +270,12 @@ SYM_FUNC_START(__svm_vcpu_run) RESTORE_GUEST_SPEC_CTRL_BODY RESTORE_HOST_SPEC_CTRL_BODY -10: cmpb $0, kvm_rebooting - jne 2b - ud2 -30: cmpb $0, kvm_rebooting - jne 4b - ud2 -50: cmpb $0, kvm_rebooting - jne 6b - ud2 -70: cmpb $0, kvm_rebooting - jne 8b - ud2 +10: ud2 _ASM_EXTABLE(1b, 10b) - _ASM_EXTABLE(3b, 30b) - _ASM_EXTABLE(5b, 50b) - _ASM_EXTABLE(7b, 70b) + _ASM_EXTABLE(3b, 10b) + _ASM_EXTABLE(5b, 10b) + _ASM_EXTABLE(7b, 10b) SYM_FUNC_END(__svm_vcpu_run)
Paolo
Quoted 12 lines · Paolo Bonzini
Quoted 10 lines · Peter Zijlstra
Review
Quoted 7 lines
(quoted diff) +7
-0+ * Clobbering BP here is mostly ok since GIF will block NMIs and with + * the exception of #MC and the kvm_rebooting _ASM_EXTABLE()s below + * nothing untoward will happen until BP is restored. + * + * The kvm_rebooting exceptions should not want to unwind stack, and + * while #MV might want to unwind stack, it is ultimately fatal. + */Aside from me not being able to type #MC, I did realize that the
kvm_rebootexception will go outside noinstr code and can hit tracing/instrumentation and do unwinds from there.
Quoted 2 lines
Asynchronously disabling SVM requires an IPI, so
kvm_rebootingcannot change
within CLGI/STGI. We can check it after CLGI instead of waiting for a #GP:
Seems fair; thanks!
Quoted 52 lines
arch/x86/kvm/svm/svm.c +4
-1@@ -4074,7 +4074,10 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)if (!static_cpu_has (X86_FEATURE_V_SPEC_CTRL ))x86_spec_ctrl_set_guest (svm->virt_spec_ctrl); -svm_vcpu_enter_exit (vcpu, spec_ctrl_intercepted); +if (unlikely (kvm_rebooting)) + svm->vmcb->control.exit_code =SVM_EXIT_PAUSE ; +else +svm_vcpu_enter_exit (vcpu, spec_ctrl_intercepted);if (!static_cpu_has (X86_FEATURE_V_SPEC_CTRL ))arch/x86/kvm/svm/svm.c +0
-0x86_spec_ctrl_restore_host (svm->virt_spec_ctrl);arch/x86/kvm/svm/vmenter.S +4
-15@@ -270,23 +270,12 @@ SYM_FUNC_START(__svm_vcpu_run) RESTORE_GUEST_SPEC_CTRL_BODY RESTORE_HOST_SPEC_CTRL_BODY -10: cmpb $0, kvm_rebooting - jne 2b - ud2 -30: cmpb $0, kvm_rebooting - jne 4b - ud2 -50: cmpb $0, kvm_rebooting - jne 6b - ud2 -70: cmpb $0, kvm_rebooting - jne 8b - ud2 +10: ud2 _ASM_EXTABLE(1b, 10b) - _ASM_EXTABLE(3b, 30b) - _ASM_EXTABLE(5b, 50b) - _ASM_EXTABLE(7b, 70b) + _ASM_EXTABLE(3b, 10b) + _ASM_EXTABLE(5b, 10b) + _ASM_EXTABLE(7b, 10b) SYM_FUNC_END(__svm_vcpu_run)Paolo
Quoted 8 lines · Nikunj A Dadhania
commit
7f4b5cde2409("kvm: Disable objtool frame pointer checking forvmenter.S") had added the vmenter.o file to the exception list.objtool gives the following warnings in the newer kernel builds:
arch/x86/kvm/kvm-amd.o: warning: objtool:__svm_vcpu_run+0x17d: BP used as a scratch registerarch/x86/kvm/kvm-amd.o: warning: objtool:__svm_sev_es_vcpu_run+0x72: BP used as a scratch register
I wanted to poke around a little, but can't reproduce this.
I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Nikunj A Dadhania
commit
7f4b5cde2409("kvm: Disable objtool frame pointer checking forvmenter.S") had added the vmenter.o file to the exception list.objtool gives the following warnings in the newer kernel builds:
Quoted 8 lines
Quoted 3 lines
arch/x86/kvm/kvm-amd.o: warning: objtool:__svm_vcpu_run+0x17d: BP used as a scratch registerarch/x86/kvm/kvm-amd.o: warning: objtool:__svm_sev_es_vcpu_run+0x72: BP used as a scratch registerI wanted to poke around a little, but can't reproduce this.
I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?
I have attached the config.
Regards,
Nikunj
Quoted 11 lines · Nikunj A. Dadhania
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Nikunj A Dadhania
commit
7f4b5cde2409("kvm: Disable objtool frame pointer checking forvmenter.S") had added the vmenter.o file to the exception list.objtool gives the following warnings in the newer kernel builds:
arch/x86/kvm/kvm-amd.o: warning: objtool:__svm_vcpu_run+0x17d: BP used as a scratch registerarch/x86/kvm/kvm-amd.o: warning: objtool:__svm_sev_es_vcpu_run+0x72: BP used as a scratch register
Quoted 6 lines
Quoted 4 lines
I wanted to poke around a little, but can't reproduce this.
I took x86_64-defconfig + KVM=m + KVM_AMD=m + UNWIND_FRAME_POINTER=y but objtool won't complain :/ What actual .config trips this?
I have attached the config.
Thanks, that does reproduce, although now I'm left wondering what exactly makes it go beep. Also, URGH, I thought the build was broken until I noticed it had BTF on :/
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Nikunj,
kernel test robot noticed the following build errors:
[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nikunj-A-Dadhania/KVM-SVM-Add-exception-to-disable-objtool-warning-for-kvm-amd-o/20230802-171219 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue patch link: https://lore.kernel.org/r/20230802091107.1160320-1-nikunj%40amd.com patch subject: [PATCH] KVM: SVM: Add exception to disable objtool warning for kvm-amd.o config: i386-randconfig-i013-20230812 (https://download.01.org/0day-ci/archive/20230812/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce: (https://download.01.org/0day-ci/archive/20230812/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> arch/x86/kvm/svm/vmenter.S:392:1: error: invalid instruction mnemonic 'stack_frame_non_standard_fp'
STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run) ^~~~~~~~~~~~~~~~~~~~~~~~~~~
vim +/stack_frame_non_standard_fp +392 arch/x86/kvm/svm/vmenter.S
352
353 /* Clobbers RAX, RCX, RDX. */
354 RESTORE_HOST_SPEC_CTRL
355
356 /*
357 * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be
358 * untrained as soon as we exit the VM and are back to the
359 * kernel. This should be done before re-enabling interrupts
360 * because interrupt handlers won't sanitize RET if the return is
361 * from the kernel.
362 */
363 UNTRAIN_RET
364
365 /* "Pop" @spec_ctrl_intercepted. */
366 pop %_ASM_BX
367
368 pop %_ASM_BX
369
370 #ifdef CONFIG_X86_64
371 pop %r12
372 pop %r13
373 pop %r14
374 pop %r15
375 #else
376 pop %esi
377 pop %edi
378 #endif
379 pop %_ASM_BP
380 RET
381
382 RESTORE_GUEST_SPEC_CTRL_BODY
383 RESTORE_HOST_SPEC_CTRL_BODY
384
385 3: cmpb $0, kvm_rebooting
386 jne 2b
387 ud2
388
389 _ASM_EXTABLE(1b, 3b)
390
391 SYM_FUNC_END(__svm_sev_es_vcpu_run)
> 392 STACK_FRAME_NON_STANDARD_FP(__svm_sev_es_vcpu_run)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki