Skip to contentSkip to conversation

x86/speculation: Use AMD specific retpoline for inline asm on AMD

PATCH In review Zhenzhong Duan 3 participants 9 messages Sep 18, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/speculation: Use AMD specific retpoline for inline asm on AMD

Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.

For x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used by x86_64.

Add ANNOTATE_NOSPEC_ALTERNATIVE for i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE."

---
 arch/x86/include/asm/nospec-branch.h |   23 ++++++++++++++++-------
 1 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/include/asm/nospec-branch.h +16 -7
index fd2a8c1..2d49eab 100644
@@ -170,21 +170,26 @@
  */
 # define CALL_NOSPEC						\
 	ANNOTATE_NOSPEC_ALTERNATIVE				\
-	ALTERNATIVE(						\
+	ALTERNATIVE_2(						\
 	ANNOTATE_RETPOLINE_SAFE					\
 	"call *%[thunk_target]\n",				\
 	"call __x86_indirect_thunk_%V[thunk_target]\n",		\
-	X86_FEATURE_RETPOLINE)
+	X86_FEATURE_RETPOLINE,					\
+	"lfence;\n"						\
+	ANNOTATE_RETPOLINE_SAFE					\
+	"call *%[thunk_target]\n",				\
+	X86_FEATURE_RETPOLINE_AMD)
 # define THUNK_TARGET(addr) [thunk_target] "r" (addr)
 
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)
 /*
  * For i386 we use the original ret-equivalent retpoline, because
  * otherwise we'll run out of registers. We don't care about CET
  * here, anyway.
  */
 # define CALL_NOSPEC						\
-	ALTERNATIVE(						\
+	ANNOTATE_NOSPEC_ALTERNATIVE				\
+	ALTERNATIVE_2(						\
 	ANNOTATE_RETPOLINE_SAFE					\
 	"call *%[thunk_target]\n",				\
 	"       jmp    904f;\n"					\
@@ -194,12 +199,16 @@
 	"    	lfence;\n"					\
 	"       jmp    902b;\n"					\
 	"       .align 16\n"					\
-	"903:	addl   $4, %%esp;\n"				\
-	"       pushl  %[thunk_target];\n"			\
+	"903:	add   $4, %%" _ASM_SP ";\n"			\
+	"       push  %[thunk_target];\n"			\
 	"       ret;\n"						\
 	"       .align 16\n"					\
 	"904:	call   901b;\n",				\
-	X86_FEATURE_RETPOLINE)
+	X86_FEATURE_RETPOLINE,					\
+	"lfence;\n"						\
+	ANNOTATE_RETPOLINE_SAFE					\
+	"call *%[thunk_target]\n",				\
+	X86_FEATURE_RETPOLINE_AMD)
 
 # define THUNK_TARGET(addr) [thunk_target] "rm" (addr)
 #else /* No retpoline for C / inline asm */
Signature · 1 line
1.7.3
maintainer
#1 lore raw
Quoted 11 lines · Zhenzhong Duan

Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.

For x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used by x86_64.
Add ANNOTATE_NOSPEC_ALTERNATIVE for i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE."

This Changelog is almost unreadable, please rewrite.

Reverse engineering the patch you add RETPOLINE_AMD support to the inline-asm CALL_NOSPEC so that they match the asm CALL_NOSPEC.

Quoted 15 lines
---
 arch/x86/include/asm/nospec-branch.h |   23 ++++++++++++++++-------
 1 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/include/asm/nospec-branch.h +1 -1
index fd2a8c1..2d49eab 100644
@@ -170,21 +170,26 @@
  */
 # define CALL_NOSPEC						\
 	ANNOTATE_NOSPEC_ALTERNATIVE				\
-	ALTERNATIVE(						\
+	ALTERNATIVE_2(						\

arch/x86/include/asm/nospec-branch.h line 182

Quoted 10 lines
arch/x86/include/asm/nospec-branch.h +5 -1
 	ANNOTATE_RETPOLINE_SAFE					\
 	"call *%[thunk_target]\n",				\
 	"call __x86_indirect_thunk_%V[thunk_target]\n",		\
-	X86_FEATURE_RETPOLINE)
+	X86_FEATURE_RETPOLINE,					\
+	"lfence;\n"						\
+	ANNOTATE_RETPOLINE_SAFE					\
+	"call *%[thunk_target]\n",				\
+	X86_FEATURE_RETPOLINE_AMD)
 # define THUNK_TARGET(addr) [thunk_target] "r" (addr)

That's OK.

arch/x86/include/asm/nospec-branch.h line 184

Quoted 3 lines
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

Quoted 13 lines
arch/x86/include/asm/nospec-branch.h +2 -1
 /*
  * For i386 we use the original ret-equivalent retpoline, because
  * otherwise we'll run out of registers. We don't care about CET
  * here, anyway.
  */
 # define CALL_NOSPEC						\
-	ALTERNATIVE(						\
+	ANNOTATE_NOSPEC_ALTERNATIVE				\
+	ALTERNATIVE_2(						\
 	ANNOTATE_RETPOLINE_SAFE					\
 	"call *%[thunk_target]\n",				\
 	"       jmp    904f;\n"					\

arch/x86/include/asm/nospec-branch.h line 203

Quoted 7 lines
arch/x86/include/asm/nospec-branch.h +2 -2
@@ -194,12 +199,16 @@
 	"    	lfence;\n"					\
 	"       jmp    902b;\n"					\
 	"       .align 16\n"					\
-	"903:	addl   $4, %%esp;\n"				\
-	"       pushl  %[thunk_target];\n"			\
+	"903:	add   $4, %%" _ASM_SP ";\n"			\
+	"       push  %[thunk_target];\n"			\

Yeah, don't do that.

arch/x86/include/asm/nospec-branch.h line 211

Quoted 9 lines
arch/x86/include/asm/nospec-branch.h +5 -1
 	"       ret;\n"						\
 	"       .align 16\n"					\
 	"904:	call   901b;\n",				\
-	X86_FEATURE_RETPOLINE)
+	X86_FEATURE_RETPOLINE,					\
+	"lfence;\n"						\
+	ANNOTATE_RETPOLINE_SAFE					\
+	"call *%[thunk_target]\n",				\
+	X86_FEATURE_RETPOLINE_AMD)

And that's OK again.

#2 lore raw
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Zhenzhong Duan

Lfence is preferred than general retpoline on AMD, add this option in C / inline asm just as the ASM code does.

For x86_64, it still help to have minimal retpoline for kernel even if gcc doesn't support it, change the inline asm for x86 so that it could also be used by x86_64.
Add ANNOTATE_NOSPEC_ALTERNATIVE for i386 to avoid below warning:
"warning: objtool: .altinstr_replacement+0x10: unsupported intra-function call"
"warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE."

Quoted 1 line

This Changelog is almost unreadable, please rewrite.

Sorry, I'll rewrite it.

Quoted 33 lines

Reverse engineering the patch you add RETPOLINE_AMD support to the inline-asm CALL_NOSPEC so that they match the asm CALL_NOSPEC.

Quoted 15 lines
---
  arch/x86/include/asm/nospec-branch.h |   23 ++++++++++++++++-------
  1 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/include/asm/nospec-branch.h +1 -1
index fd2a8c1..2d49eab 100644
@@ -170,21 +170,26 @@
   */
  # define CALL_NOSPEC						\
  	ANNOTATE_NOSPEC_ALTERNATIVE				\
-	ALTERNATIVE(						\
+	ALTERNATIVE_2(						\

arch/x86/include/asm/nospec-branch.h line 182

Quoted 10 lines
arch/x86/include/asm/nospec-branch.h +5 -1
  	ANNOTATE_RETPOLINE_SAFE					\
  	"call *%[thunk_target]\n",				\
  	"call __x86_indirect_thunk_%V[thunk_target]\n",		\
-	X86_FEATURE_RETPOLINE)
+	X86_FEATURE_RETPOLINE,					\
+	"lfence;\n"						\
+	ANNOTATE_RETPOLINE_SAFE					\
+	"call *%[thunk_target]\n",				\
+	X86_FEATURE_RETPOLINE_AMD)
  # define THUNK_TARGET(addr) [thunk_target] "r" (addr)

That's OK.

Quoted 1 line
Quoted 4 lines

arch/x86/include/asm/nospec-branch.h line 184

Quoted 2 lines
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?

Quoted 15 lines
Quoted 15 lines

/*
* For i386 we use the original ret-equivalent retpoline, because
* otherwise we'll run out of registers. We don't care about CET
* here, anyway.
*/

  # define CALL_NOSPEC						\
arch/x86/include/asm/nospec-branch.h +2 -1
-	ALTERNATIVE(						\
+	ANNOTATE_NOSPEC_ALTERNATIVE				\
+	ALTERNATIVE_2(						\
  	ANNOTATE_RETPOLINE_SAFE					\
  	"call *%[thunk_target]\n",				\
  	"       jmp    904f;\n"					\
@@ -194,12 +199,16 @@
  	"    	lfence;\n"					\
Quoted 8 lines

arch/x86/include/asm/nospec-branch.h line 203

Quoted 6 lines
arch/x86/include/asm/nospec-branch.h +2 -2
  	"       jmp    902b;\n"					\
  	"       .align 16\n"					\
-	"903:	addl   $4, %%esp;\n"				\
-	"       pushl  %[thunk_target];\n"			\
+	"903:	add   $4, %%" _ASM_SP ";\n"			\
+	"       push  %[thunk_target];\n"			\

Yeah, don't do that.

This is the change for above reason.

Thanks
Zhenzhong

maintainer
#3 lore raw
Quoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra

arch/x86/include/asm/nospec-branch.h line 184

Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?

No it doesn't.

#if defined(X86_64) && defined(RETPOLINE)

/* x86_64 retpoline goes here */

#elif defined(RETPOLINE)

/* !x86_64 retpoline goes here */

#else

/* !retpoline goes here

#endif

#4 lore raw
Quoted 19 lines · Peter Zijlstra
Quoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra

arch/x86/include/asm/nospec-branch.h line 184

Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?

No it doesn't.

#if defined(X86_64) && defined(RETPOLINE)

/* x86_64 retpoline goes here */

#elif defined(RETPOLINE)

Quoted 7 lines

/* !x86_64 retpoline goes here */

#else

/* !retpoline goes here

#endif

Sorry, but I am confused.
So where is 'if defined(x86_64) && !defined(RETPOLINE) && defined(CONFIG_RETPOLINE)' go? In original code, it will go to "call *%[thunk_target]\n" while
we have set SPECTRE_V2_RETPOLINE_MINIMAL or SPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?

Thanks
Zhenzhong

maintainer
#5 lore raw
Quoted 24 lines · Zhenzhong Duan
Quoted 23 lines · Peter Zijlstra
Quoted 9 lines · Zhenzhong Duan
Quoted 5 lines · Peter Zijlstra

arch/x86/include/asm/nospec-branch.h line 184

Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?

No it doesn't.

#if defined(X86_64) && defined(RETPOLINE)

/* x86_64 retpoline goes here */

#elif defined(RETPOLINE)

/* !x86_64 retpoline goes here */

#else

Quoted 7 lines
Quoted 3 lines

/* !retpoline goes here

#endif

Sorry, but I am confused.
So where is 'if defined(x86_64) && !defined(RETPOLINE) && defined(CONFIG_RETPOLINE)' go?

Argh, CONFIG_RETPOLINE vs RETPOLINE :/

The thing is, the one you modify has a comment on that explains why it is i386 only. CET and retpolines don't like one another much.

And the x86_64 version uses %V which requires new GCC.

So I'm all for fixing the RETPOLINE_AMD thing, but at this point nobody should use the minimal stuff, that's just delusional.

Quoted 3 lines

In original code, it will go to "call *%[thunk_target]\n" while we have set SPECTRE_V2_RETPOLINE_MINIMAL or SPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?

Yes, that is exactly right -- it does that with or without your change though.

maintainer
#6 lore raw
Quoted 6 lines
Quoted 3 lines

In original code, it will go to "call *%[thunk_target]\n" while we have set SPECTRE_V2_RETPOLINE_MINIMAL or SPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?

Yes, that is exactly right -- it does that with or without your change though.

Soryr, I wrote that before I noticed the CONFIG_RETPOLINE vs RETPOLINE thing.

In any case, since retpoline capable compilers are widely available, I'm all for making CONFIG_RETPOLINE hard depend on it. That minimal stuff really is quite silly.

#7 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines
Quoted 3 lines

In original code, it will go to "call *%[thunk_target]\n" while we have set SPECTRE_V2_RETPOLINE_MINIMAL or SPECTRE_V2_RETPOLINE_MINIMAL_AMD. Is this expected?

Yes, that is exactly right -- it does that with or without your change though.

Quoted 6 lines

Soryr, I wrote that before I noticed the CONFIG_RETPOLINE vs RETPOLINE thing.

In any case, since retpoline capable compilers are widely available, I'm all for making CONFIG_RETPOLINE hard depend on it. That minimal stuff really is quite silly.

Yep, now the the compiler support is widespread that makes a lot of sense.

#8 lore raw
In reply to Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 20 lines · Zhenzhong Duan
Quoted 19 lines · Peter Zijlstra
Quoted 8 lines · Zhenzhong Duan
Quoted 4 lines · Peter Zijlstra

arch/x86/include/asm/nospec-branch.h line 184

Quoted 2 lines · Zhenzhong Duan
arch/x86/include/asm/nospec-branch.h +1 -1
-#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE)
+#elif defined(CONFIG_RETPOLINE)

This doesn't make any sense..

This change is used for x86_64 to have minimal Retpoline support when CONFIG_RETPOLINE is defined but RETPOLINE isn't defined, or I missed something?

No it doesn't.

#if defined(X86_64) && defined(RETPOLINE)

   /* x86_64 retpoline goes here */

#elif defined(RETPOLINE)

   /* !x86_64 retpoline goes here */
Quoted 8 lines
Quoted 5 lines

#else

   /* !retpoline goes here

#endif

Sorry, but I am confused.
So where is 'if defined(x86_64) && !defined(RETPOLINE) && defined(CONFIG_RETPOLINE)' go?

Argh, CONFIG_RETPOLINE vs RETPOLINE :/

Quoted 8 lines

The thing is, the one you modify has a comment on that explains why it is i386 only. CET and retpolines don't like one another much.

And the x86_64 version uses %V which requires new GCC.

So I'm all for fixing the RETPOLINE_AMD thing, but at this point nobody should use the minimal stuff, that's just delusional.

Clear, thanks for your explanation.

Zhenzhong