x86/speculation: Use AMD specific retpoline for inline asm on AMD
Subsystems (2)
PATCH 1/1 x86/speculation: Use AMD specific retpoline for inline asm on AMD
Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.
For x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used by x86_64.
Add ANNOTATE_NOSPEC_ALTERNATIVE for i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE."
--- arch/x86/include/asm/nospec-branch.h | 23 ++++++++++++++++------- 1 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/include/asm/nospec-branch.h +16 -7
@@ -170,21 +170,26 @@*/ # define CALL_NOSPEC \ ANNOTATE_NOSPEC_ALTERNATIVE \ - ALTERNATIVE( \ + ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ "call __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE) + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "call *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_AMD)# define THUNK_TARGET (addr) [thunk_target] "r" (addr) -#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )/* * For i386 we use the original ret-equivalent retpoline, because * otherwise we'll run out of registers. We don't care about CET * here, anyway. */ # define CALL_NOSPEC \ - ALTERNATIVE( \ + ANNOTATE_NOSPEC_ALTERNATIVE \ + ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ " jmp 904f;\n" \
@@ -194,12 +199,16 @@ " lfence;\n" \ " jmp 902b;\n" \ " .align 16\n" \ - "903: addl $4, %%esp;\n" \ - " pushl %[thunk_target];\n" \ + "903: add $4, %%" _ASM_SP ";\n" \ + " push %[thunk_target];\n" \ " ret;\n" \ " .align 16\n" \ "904: call 901b;\n", \ - X86_FEATURE_RETPOLINE) + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "call *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_AMD)# define THUNK_TARGET (addr) [thunk_target] "rm" (addr)#else /* No retpoline for C / inline asm */
Signature · 1 line
1.7.3
Peter Zijlstra
maintainerQuoted 11 lines · Zhenzhong Duan
Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.
For
x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used byx86_64.
AddANNOTATE_NOSPEC_ALTERNATIVEfor i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it withANNOTATE_NOSPEC_ALTERNATIVE."
This Changelog is almost unreadable, please rewrite.
Reverse engineering the patch you add RETPOLINE_AMD support to the inline-asm CALL_NOSPEC so that they match the asm CALL_NOSPEC.
Quoted 15 lines
Signed-off-by Zhenzhong Duan <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 23 ++++++++++++++++------- 1 files changed, 16 insertions(+), 7 deletions(-)arch/x86/include/asm/nospec-branch.h +1
-1@@ -170,21 +170,26 @@*/ # define CALL_NOSPEC \ ANNOTATE_NOSPEC_ALTERNATIVE \ - ALTERNATIVE( \ + ALTERNATIVE_2( \
arch/x86/include/asm/nospec-branch.h line 182
Quoted 10 lines
arch/x86/include/asm/nospec-branch.h +5
-1ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ "call __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE) + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "call *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_AMD)# define THUNK_TARGET (addr) [thunk_target] "r" (addr)
That's OK.
arch/x86/include/asm/nospec-branch.h line 184
Quoted 3 lines
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )
This doesn't make any sense..
Quoted 13 lines
arch/x86/include/asm/nospec-branch.h +2
-1/* * For i386 we use the original ret-equivalent retpoline, because * otherwise we'll run out of registers. We don't care about CET * here, anyway. */ # define CALL_NOSPEC \ - ALTERNATIVE( \ + ANNOTATE_NOSPEC_ALTERNATIVE \ + ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ " jmp 904f;\n" \
arch/x86/include/asm/nospec-branch.h line 203
Quoted 7 lines
arch/x86/include/asm/nospec-branch.h +2
-2@@ -194,12 +199,16 @@" lfence;\n" \" jmp 902b;\n" \" .align 16\n" \ -"903: addl $4, %%esp;\n" \ -" pushl %[thunk_target];\n" \ +"903: add $4, %%" _ASM_SP";\n" \ +" push %[thunk_target];\n" \
Yeah, don't do that.
arch/x86/include/asm/nospec-branch.h line 211
Quoted 9 lines
arch/x86/include/asm/nospec-branch.h +5
-1" ret;\n" \" .align 16\n" \"904: call 901b;\n" , \ - X86_FEATURE_RETPOLINE) +X86_FEATURE_RETPOLINE , \ +"lfence;\n" \ +ANNOTATE_RETPOLINE_SAFE \ +"call *%[thunk_target]\n" , \ + X86_FEATURE_RETPOLINE_AMD)
And that's OK again.
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Zhenzhong Duan
Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.
For
x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used byx86_64.
AddANNOTATE_NOSPEC_ALTERNATIVEfor i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it withANNOTATE_NOSPEC_ALTERNATIVE."
Quoted 1 line
This Changelog is almost unreadable, please rewrite.
Sorry, I'll rewrite it.
Quoted 33 lines
Reverse engineering the patch you add
RETPOLINE_AMDsupport to the inline-asmCALL_NOSPECso that they match the asmCALL_NOSPEC.Quoted 15 lines
Signed-off-by Zhenzhong Duan <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 23 ++++++++++++++++------- 1 files changed, 16 insertions(+), 7 deletions(-)arch/x86/include/asm/nospec-branch.h +1
-1@@ -170,21 +170,26 @@*/ # define CALL_NOSPEC \ ANNOTATE_NOSPEC_ALTERNATIVE \ - ALTERNATIVE( \ + ALTERNATIVE_2( \arch/x86/include/asm/nospec-branch.h line 182
Quoted 10 lines
arch/x86/include/asm/nospec-branch.h +5
-1ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ "call __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE) + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "call *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_AMD)# define THUNK_TARGET (addr) [thunk_target] "r" (addr)That's OK.
Quoted 1 line
Quoted 4 lines
arch/x86/include/asm/nospec-branch.h line 184
Quoted 2 lines
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )This doesn't make any sense..
This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?
Quoted 15 lines
Quoted 15 lines
/*
* For i386 we use the original ret-equivalent retpoline, because
* otherwise we'll run out of registers. We don't care about CET
* here, anyway.
*/# define CALL_NOSPEC \arch/x86/include/asm/nospec-branch.h +2
-1-ALTERNATIVE ( \ + ANNOTATE_NOSPEC_ALTERNATIVE \ + ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \"call *%[thunk_target]\n" , \" jmp 904f;\n" \@@ -194,12 +199,16 @@" lfence;\n" \
Quoted 8 lines
arch/x86/include/asm/nospec-branch.h line 203
Quoted 6 lines
arch/x86/include/asm/nospec-branch.h +2
-2" jmp 902b;\n" \" .align 16\n" \ -"903: addl $4, %%esp;\n" \ -" pushl %[thunk_target];\n" \ +"903: add $4, %%" _ASM_SP";\n" \ +" push %[thunk_target];\n" \Yeah, don't do that.
This is the change for above reason.
Thanks
Zhenzhong
Peter Zijlstra
maintainerQuoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra
arch/x86/include/asm/nospec-branch.h line 184
Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )This doesn't make any sense..
This change is used for
x86_64to have minimal Retpoline support whenCONFIG_RETPOLINEis defined but RETPOLINE isn't defined, or I missed something?
No it doesn't.
#if defined(X86_64) && defined(RETPOLINE)
/* x86_64 retpoline goes here */
#elif defined(RETPOLINE)
/* !x86_64 retpoline goes here */
#else
/* !retpoline goes here
#endif
Quoted 19 lines · Peter Zijlstra
Quoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra
arch/x86/include/asm/nospec-branch.h line 184
Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )This doesn't make any sense..
This change is used for
x86_64to have minimal Retpoline support whenCONFIG_RETPOLINEis defined but RETPOLINE isn't defined, or I missed something?No it doesn't.
#if
defined(X86_64)&& defined(RETPOLINE)/*
x86_64retpoline goes here */#elif defined(RETPOLINE)
Quoted 7 lines
/* !x86_64 retpoline goes here */
#else
/* !retpoline goes here
#endif
Sorry, but I am confused.
So where is 'if defined(x86_64) && !defined(RETPOLINE) && defined(CONFIG_RETPOLINE)' go? In original code, it will go to "call *%[thunk_target]\n" while
we have set SPECTRE_V2_RETPOLINE_MINIMAL or SPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?
Thanks
Zhenzhong
Peter Zijlstra
maintainerQuoted 24 lines · Zhenzhong Duan
Quoted 23 lines · Peter Zijlstra
Quoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra
arch/x86/include/asm/nospec-branch.h line 184
Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )This doesn't make any sense..
This change is used for
x86_64to have minimal Retpoline support whenCONFIG_RETPOLINEis defined but RETPOLINE isn't defined, or I missed something?No it doesn't.
#if
defined(X86_64)&& defined(RETPOLINE)/*
x86_64retpoline goes here */#elif defined(RETPOLINE)
/* !x86_64 retpoline goes here */
#else
Quoted 7 lines
Quoted 3 lines
/* !retpoline goes here
#endif
Sorry, but I am confused.
So where is 'ifdefined(x86_64)&& !defined(RETPOLINE) &&defined(CONFIG_RETPOLINE)' go?
Argh, CONFIG_RETPOLINE vs RETPOLINE :/
The thing is, the one you modify has a comment on that explains why it is i386 only. CET and retpolines don't like one another much.
And the x86_64 version uses %V which requires new GCC.
So I'm all for fixing the RETPOLINE_AMD thing, but at this point nobody should use the minimal stuff, that's just delusional.
Quoted 3 lines
In original code, it will go to "call *%[thunk_target]\n" while we have set
SPECTRE_V2_RETPOLINE_MINIMALorSPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?
Yes, that is exactly right -- it does that with or without your change though.
Peter Zijlstra
maintainerQuoted 6 lines
Quoted 3 lines
In original code, it will go to "call *%[thunk_target]\n" while we have set
SPECTRE_V2_RETPOLINE_MINIMALorSPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?Yes, that is exactly right -- it does that with or without your change though.
Soryr, I wrote that before I noticed the CONFIG_RETPOLINE vs RETPOLINE thing.
In any case, since retpoline capable compilers are widely available, I'm all for making CONFIG_RETPOLINE hard depend on it. That minimal stuff really is quite silly.
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines
Quoted 3 lines
In original code, it will go to "call *%[thunk_target]\n" while we have set
SPECTRE_V2_RETPOLINE_MINIMALorSPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?Yes, that is exactly right -- it does that with or without your change though.
Quoted 6 lines
Soryr, I wrote that before I noticed the
CONFIG_RETPOLINEvs RETPOLINE thing.In any case, since retpoline capable compilers are widely available, I'm all for making
CONFIG_RETPOLINEhard depend on it. That minimal stuff really is quite silly.
Yep, now the the compiler support is widespread that makes a lot of sense.
Quoted 31 lines · Peter Zijlstra
Quoted 20 lines · Zhenzhong Duan
Quoted 19 lines · Peter Zijlstra
Quoted 8 lines · Zhenzhong Duan
Quoted 4 lines · Peter Zijlstra
arch/x86/include/asm/nospec-branch.h line 184
Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1
-1-#elif defined(CONFIG_X86_32 ) && defined(CONFIG_RETPOLINE ) +#elif defined(CONFIG_RETPOLINE )This doesn't make any sense..
This change is used for
x86_64to have minimal Retpoline support whenCONFIG_RETPOLINEis defined but RETPOLINE isn't defined, or I missed something?No it doesn't.
#if
defined(X86_64)&& defined(RETPOLINE)/* x86_64 retpoline goes here */#elif defined(RETPOLINE)
/* !x86_64 retpoline goes here */Quoted 8 lines
Quoted 5 lines
#else
/* !retpoline goes here#endif
Sorry, but I am confused.
So where is 'ifdefined(x86_64)&& !defined(RETPOLINE) &&defined(CONFIG_RETPOLINE)' go?Argh,
CONFIG_RETPOLINEvs RETPOLINE :/
Quoted 8 lines
The thing is, the one you modify has a comment on that explains why it is i386 only. CET and retpolines don't like one another much.
And the
x86_64version uses %V which requires new GCC.So I'm all for fixing the
RETPOLINE_AMDthing, but at this point nobody should use the minimal stuff, that's just delusional.
Clear, thanks for your explanation.
Zhenzhong