x86/kvm/emulate: Avoid RET for FASTOPs
Peter Zijlstra
maintainerv2 cover x86/kvm/emulate: Avoid RET for FASTOPs
Hi!
At long last, a respin of these patches.
The FASTOPs are special because they rely on RET to preserve CFLAGS, which is a problem with all the mitigation stuff. Also see things like: ba5ca5e5e6a1 ("x86/retpoline: Don't clobber RFLAGS during srso_safe_ret()").
Rework FASTOPs to no longer use RET and side-step the problem of trying to make the various return thunks preserve CFLAGS for just this one case.
There are two separate instances, test_cc() and fastop(). The first is basically a SETCC wrapper, which seems like a very complicated (and somewhat expensive) way to read FLAGS. Instead use the code we already have to emulate JCC to fully emulate the instruction.
That then leaves fastop(), which when marked noinline is guaranteed to exist only once. As such, CALL+RET isn't needed, because we'll always be RETurning to the same location, as such replace with JMP+JMP.
My plan is to take the objtool patches through tip/objtool/core, the nospec patches through tip/x86/core and either stick the fastop patches in that latter tree if the KVM folks agree, or they can merge the aforementioned two branches and then stick the patches on top, whatever works for people.
Peter Zijlstra
maintainerPATCH v2 1/12 objtool: Generic annotation infrastructure
Avoid endless .discard.foo sections for each annotation, create a single .discard.annotate section that takes an annotation type along with the instruction.
--- include/linux/objtool.h | 18 ++++++++++++++++++ tools/objtool/check.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
include/linux/objtool.h +18 -0
@@ -57,6 +57,13 @@".long 998b\n\t" \ ".popsection\n\t" +#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t" +#else /* __ASSEMBLY__ */ /*
@@ -146,6 +153,14 @@ .popsection .endm +.macro ANNOTATE type:req +.Lhere_\@: + .pushsection .discard.annotate,"M",@progbits,8 + .long .Lhere_\@ - . + .long \type + .popsection +.endm +#endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */
@@ -155,6 +170,7 @@#define UNWIND_HINT (type, sp_reg, sp_offset, signal) "\n\t"#define STACK_FRAME_NON_STANDARD (func)#define STACK_FRAME_NON_STANDARD_FP (func) +#define ASM_ANNOTATE (x)#define ANNOTATE_NOENDBR #define ASM_REACHABLE #else
@@ -167,6 +183,8 @@ .endm .macro REACHABLE .endm +.macro ANNOTATE type:req +.endm#endif #endif /* CONFIG_OBJTOOL */
tools/objtool/check.c +46 -0
58 lines not on this page · raw message
Re: [PATCH v2 01/12] objtool: Generic annotation infrastructure
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"
Why mergeable and progbits?
Quoted 4 lines
tools/objtool/check.c +4
-0+static int read_annotate (struct objtool_file *file,void (*func)(int type,struct instruction *insn)) +{ +struct section *rsec, *sec; +struct instruction *insn;
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2389
Quoted 10 lines
tools/objtool/check.c +10
-0+struct reloc *reloc; +int type; + + rsec =find_section_by_name (file->elf,".rela.discard.annotate" ); +if (!rsec) +return 0 ; + + sec =find_section_by_name (file->elf,".discard.annotate" ); +if (!sec) +return 0 ;
Instead of looking for .rela.discard.annotate you can just get it from sec->rsec.
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2392
Quoted 3 lines
tools/objtool/check.c +3
-0+ +if (sec->sh.sh_entsize !=8 ) { +static bool warn = false;
"warned" ?
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2396
Quoted 4 lines
tools/objtool/check.c +4
-0+if (!warn) { +WARN ("%s: dodgy linker, sh_entsize != 8" , sec->name); + warn = true; + }
Any reason not to make this a fatal error?
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2406
Quoted 10 lines
tools/objtool/check.c +10
-0+ sec->sh.sh_entsize =8 ; + } + +for_each_reloc (rsec, reloc) { + insn =find_insn (file, reloc->sym->sec, + reloc->sym->offset +reloc_addend (reloc)); +if (!insn) { +WARN ("bad .discard.annotate entry: %d" ,reloc_idx (reloc)); +return -1 ; + }
Would be nice to print the type here as well.
tools/objtool/check.c static int decode_sections(struct objtoo · line 2718
Quoted 6 lines
tools/objtool/check.c +2
-0@@ -2670,6 +2714,8 @@ static int decode_sections(struct objtooif (ret)return ret; + ret =read_annotate (file, __annotate_nop); +
'ret' is ignored here (not that it matters much as this goes away in the next patch)
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
In order to get sh_entsize ?
Quoted 9 lines
Quoted 9 lines
tools/objtool/check.c +9
-0+static int read_annotate (struct objtool_file *file,void (*func)(int type,struct instruction *insn)) +{ +struct section *rsec, *sec; +struct instruction *insn; +struct reloc *reloc; +int type; + + rsec =find_section_by_name (file->elf,".rela.discard.annotate" ); +if (!rsec)
Quoted 8 lines
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2389
Quoted 5 lines
tools/objtool/check.c +5
-0+return 0 ; + + sec =find_section_by_name (file->elf,".discard.annotate" ); +if (!sec) +return 0 ;Instead of looking for .rela.discard.annotate you can just get it from
sec->rsec.
Oh, indeed.
Quoted 5 lines
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2392
Quoted 3 lines
tools/objtool/check.c +3
-0+ +if (sec->sh.sh_entsize !=8 ) { +static bool warn = false;"warned" ?
Sure.
Quoted 6 lines
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2396
Quoted 4 lines
tools/objtool/check.c +4
-0+if (!warn) { +WARN ("%s: dodgy linker, sh_entsize != 8" , sec->name); + warn = true; + }Any reason not to make this a fatal error?
lld is currently suffering from this, it would get us build failures on llvm builds. Once that's fixed, then yes, this should become fatal.
https://github.com/ClangBuiltLinux/linux/issues/2057
Quoted 4 lines
Quoted 4 lines
tools/objtool/check.c +4
-0+ sec->sh.sh_entsize =8 ; + } + +for_each_reloc (rsec, reloc) {
Quoted 8 lines
tools/objtool/check.c static int read_unwind_hints(struct objt · line 2406
Quoted 6 lines
tools/objtool/check.c +6
-0+ insn =find_insn (file, reloc->sym->sec, + reloc->sym->offset +reloc_addend (reloc)); +if (!insn) { +WARN ("bad .discard.annotate entry: %d" ,reloc_idx (reloc)); +return -1 ; + }Would be nice to print the type here as well.
Sure.
Quoted 9 lines
tools/objtool/check.c static int decode_sections(struct objtoo · line 2718
Quoted 6 lines
tools/objtool/check.c +2
-0@@ -2670,6 +2714,8 @@ static int decode_sections(struct objtooif (ret)return ret; + ret =read_annotate (file, __annotate_nop); +'ret' is ignored here (not that it matters much as this goes away in the next patch)
Right..
Re: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
Quoted 1 line
In order to get
sh_entsize?
Is that a guess? If so, it's not very convincing as I don't see what entsize would have to do with it.
Signature · 1 line
Josh
Re: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 12 lines · Josh Poimboeuf
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
Quoted 4 lines
Quoted 1 line
In order to get
sh_entsize?Is that a guess? If so, it's not very convincing as I don't see what entsize would have to do with it.
Oh, nevermind... I see it's a gas syntax issue.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 13 lines · Josh Poimboeuf
Quoted 12 lines · Josh Poimboeuf
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
In order to get
sh_entsize?Is that a guess? If so, it's not very convincing as I don't see what entsize would have to do with it.
Oh, nevermind... I see it's a gas syntax issue.
Not a guess, only mergable gets entsize, and progbits is a required argument per the syntax in order to specify entsize.
Re: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Josh Poimboeuf
Quoted 14 lines · Josh Poimboeuf
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool.h line 65
Quoted 6 lines · Peter Zijlstra
include/linux/objtool.h +6
-0+#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
In order to get
sh_entsize?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Is that a guess? If so, it's not very convincing as I don't see what entsize would have to do with it.
Oh, nevermind... I see it's a gas syntax issue.
Not a guess, only mergable gets entsize, and progbits is a required argument per the syntax in order to specify entsize.
If you look at "readelf -WS vmlinux" there are plenty of non-mergeable sections with entsize.
Signature · 1 line
Josh
Re: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 20 lines · Josh Poimboeuf
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Josh Poimboeuf
Quoted 16 lines · Josh Poimboeuf
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
Review
Quoted 6 lines · Peter Zijlstra
(quoted diff) +6
-0+#define ASM_ANNOTATE(x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t"Why mergeable and progbits?
Quoted 1 line
In order to get
sh_entsize?Is that a guess? If so, it's not very convincing as I don't see what entsize would have to do with it.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Oh, nevermind... I see it's a gas syntax issue.
Not a guess, only mergable gets entsize, and progbits is a required argument per the syntax in order to specify entsize.
If you look at "readelf -WS vmlinux" there are plenty of non-mergeable sections with entsize.
Er, vmlinux.o
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 01/12] objtool: Generic annotation infrastructure
Quoted 4 lines · Josh Poimboeuf
Quoted 2 lines
If you look at "readelf -WS vmlinux" there are plenty of non-mergeable sections with entsize.
Er, vmlinux.o
Well yes, but how do you set entsize from as? The manual only mentions entsize in relation to M(ergable) with the .section command.
Peter Zijlstra
maintainerPATCH v2 2/12 objtool: Convert ANNOTATE_NOENDBR to ANNOTATE
In reply to Peter Zijlstra--- include/linux/objtool.h | 17 ++++------------- include/linux/objtool_types.h | 5 +++++ tools/include/linux/objtool_types.h | 5 +++++ tools/objtool/check.c | 32 +++++--------------------------- 4 files changed, 19 insertions(+), 40 deletions(-)
include/linux/objtool.h +4 -13
@@ -45,12 +45,6 @@#define STACK_FRAME_NON_STANDARD_FP (func)#endif -#define ANNOTATE_NOENDBR \ - "986: \n\t" \ - ".pushsection .discard.noendbr\n\t" \ - ".long 986b\n\t" \ - ".popsection\n\t" -#define ASM_REACHABLE \ "998:\n\t" \ ".pushsection .discard.reachable\n\t" \
@@ -64,6 +58,8 @@ ".long " __stringify(x) "\n\t" \ ".popsection\n\t" +#define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR) +#else /* __ASSEMBLY__ */ /*
@@ -122,13 +118,6 @@#endif .endm -.macro ANNOTATE_NOENDBR -.Lhere_\@: - .pushsection .discard.noendbr - .long .Lhere_\@ - .popsection -.endm -/* * Use objtool to validate the entry requirement that all code paths do * VALIDATE_UNRET_END before RET.
@@ -161,6 +150,8 @@ .popsection .endm +#define ANNOTATE_NOENDBR ANNOTATE type=ANNOTYPE_NOENDBR +#endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */
include/linux/objtool_types.h +5 -0
@@ -54,4 +54,9 @@ struct unwind_hint {#define UNWIND_HINT_TYPE_SAVE 6#define UNWIND_HINT_TYPE_RESTORE 7 +/* +* Annotate types +*/ +#define ANNOTYPE_NOENDBR 1 +#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +5 -0
@@ -54,4 +54,9 @@ struct unwind_hint {#define UNWIND_HINT_TYPE_SAVE 6#define UNWIND_HINT_TYPE_RESTORE 7 +/* +* Annotate types +*/ +#define ANNOTYPE_NOENDBR 1 +#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +5 -27
@@ -2339,32 +2339,12 @@ static int read_annotate(struct objtool_return 0 ; } -static void __annotate_nop (int type,struct instruction *insn) +static void __annotate_noendbr (int type,struct instruction *insn) { -} - -static int read_noendbr_hints (struct objtool_file *file) -{ -struct instruction *insn; -struct section *rsec; -struct reloc *reloc; - - rsec =find_section_by_name (file->elf,".rela.discard.noendbr" ); -if (!rsec) -return 0 ; +if (type !=ANNOTYPE_NOENDBR ) +return ; -for_each_reloc (rsec, reloc) { - insn =find_insn (file, reloc->sym->sec, - reloc->sym->offset +reloc_addend (reloc)); -if (!insn) { -WARN ("bad .discard.noendbr entry" ); -return -1 ; - } - - insn->noendbr =1 ; - } - -return 0 ; + insn->noendbr =1 ; }static int read_retpoline_hints (struct objtool_file *file)
@@ -2637,12 +2617,10 @@ static int decode_sections(struct objtooif (ret)return ret; - ret =read_annotate (file, __annotate_nop); -/* * Must be before read_unwind_hints() since that needs insn->noendbr. */ - ret =read_noendbr_hints (file); + ret =read_annotate (file, __annotate_noendbr);if (ret)return ret;
Peter Zijlstra
maintainerPATCH v2 3/12 objtool: Convert ANNOTATE_RETPOLINE_SAFE to ANNOTATE
In reply to Peter Zijlstra--- arch/x86/include/asm/nospec-branch.h | 13 +------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 52 ++++++++++++----------------------- 4 files changed, 22 insertions(+), 45 deletions(-)
arch/x86/include/asm/nospec-branch.h +2 -11
@@ -193,12 +193,7 @@* objtool the subsequent indirect jump/call is vouched safe for retpoline * builds. */ -.macro ANNOTATE_RETPOLINE_SAFE -.Lhere_\@: - .pushsection .discard.retpoline_safe - .long .Lhere_\@ - .popsection -.endm +#define ANNOTATE_RETPOLINE_SAFE ANNOTATE type=ANNOTYPE_RETPOLINE_SAFE/* * (ab)use RETPOLINE_SAFE on RET to annotate away 'bare' RET instructions
@@ -317,11 +312,7 @@#else /* __ASSEMBLY__ */ -#define ANNOTATE_RETPOLINE_SAFE \ - "999:\n\t" \ - ".pushsection .discard.retpoline_safe\n\t" \ - ".long 999b\n\t" \ - ".popsection\n\t" +#define ANNOTATE_RETPOLINE_SAFE ASM_ANNOTATE(ANNOTYPE_RETPOLINE_SAFE)typedef u8 retpoline_thunk_t [RETPOLINE_THUNK_SIZE ];extern retpoline_thunk_t __x86_indirect_thunk_array[];
include/linux/objtool_types.h +1 -0
@@ -58,5 +58,6 @@ struct unwind_hint {* Annotate types */ #define ANNOTYPE_NOENDBR 1 +#define ANNOTYPE_RETPOLINE_SAFE 2#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +1 -0
@@ -58,5 +58,6 @@ struct unwind_hint {* Annotate types */ #define ANNOTYPE_NOENDBR 1 +#define ANNOTYPE_RETPOLINE_SAFE 2#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +18 -34
90 lines not on this page · raw message
Re: [PATCH v2 03/12] objtool: Convert ANNOTATE_RETPOLINE_SAFE to ANNOTATE
Quoted 12 lines · Peter Zijlstra
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 13 +------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 52 ++++++++++++----------------------- 4 files changed, 22 insertions(+), 45 deletions(-)
arch/x86/include/asm/nospec-branch.h line 196
Quoted 10 lines
arch/x86/include/asm/nospec-branch.h +1
-6@@ -193,12 +193,7 @@* objtool the subsequent indirect jump/call is vouched safe for retpoline * builds. */ -.macro ANNOTATE_RETPOLINE_SAFE -.Lhere_\@: - .pushsection .discard.retpoline_safe - .long .Lhere_\@ - .popsection -.endm +#define ANNOTATE_RETPOLINE_SAFE ANNOTATE type=ANNOTYPE_RETPOLINE_SAFE
I'm thinking it would be nice to put all the ANNOTATE_* definitions in objtool.h so we can have all the annotations and their descriptions in one place.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 03/12] objtool: Convert ANNOTATE_RETPOLINE_SAFE to ANNOTATE
Quoted 19 lines · Josh Poimboeuf
Quoted 18 lines · Peter Zijlstra
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 13 +------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 52 ++++++++++++----------------------- 4 files changed, 22 insertions(+), 45 deletions(-)arch/x86/include/asm/nospec-branch.h +0
-3@@ -193,12 +193,7 @@* objtool the subsequent indirect jump/call is vouched safe for retpoline * builds. */ -.macro ANNOTATE_RETPOLINE_SAFE -.Lhere_\@: - .pushsection .discard.retpoline_safe
Quoted 8 lines
arch/x86/include/asm/nospec-branch.h line 196
Quoted 4 lines
arch/x86/include/asm/nospec-branch.h +1
-3- .long .Lhere_\@ - .popsection -.endm +#define ANNOTATE_RETPOLINE_SAFE ANNOTATE type=ANNOTYPE_RETPOLINE_SAFEI'm thinking it would be nice to put all the ANNOTATE_* definitions in
objtool.hso we can have all the annotations and their descriptions in one place.
Probably, but that's going to be somewhat of a pain. Let me do that at the end and throw it at an allyesconfig or something.
Peter Zijlstra
maintainerPATCH v2 4/12 objtool: Convert instrumentation_{begin,end}() to ANNOTATE
In reply to Peter Zijlstra--- include/linux/instrumentation.h | 11 +++----- include/linux/objtool.h | 9 ++++-- include/linux/objtool_types.h | 2 + tools/include/linux/objtool_types.h | 2 + tools/objtool/check.c | 49 +++++++----------------------------- 5 files changed, 25 insertions(+), 48 deletions(-)
include/linux/instrumentation.h +5 -6
@@ -4,14 +4,14 @@#ifdef CONFIG_NOINSTR_VALIDATION +#include <linux/objtool.h> #include <linux/stringify.h> /* Begin/end of an instrumentation safe region */ #define __instrumentation_begin (c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - ".pushsection .discard.instr_begin\n\t" \ - ".long " __stringify(c) "b - .\n\t" \ - ".popsection\n\t" : : "i" (c)); \ + __ASM_ANNOTATE(c, ANNOTYPE_INSTR_BEGIN) \ + : : "i" (c)); \ })#define instrumentation_begin () __instrumentation_begin(__COUNTER__)
@@ -48,9 +48,8 @@*/ #define __instrumentation_end (c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - ".pushsection .discard.instr_end\n\t" \ - ".long " __stringify(c) "b - .\n\t" \ - ".popsection\n\t" : : "i" (c)); \ + __ASM_ANNOTATE(c, ANNOTYPE_INSTR_END) \ + : : "i" (c)); \ })#define instrumentation_end () __instrumentation_end(__COUNTER__)#else /* !CONFIG_NOINSTR_VALIDATION */
include/linux/objtool.h +6 -3
@@ -51,13 +51,16 @@".long 998b\n\t" \ ".popsection\n\t" -#define ASM_ANNOTATE (x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE (s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \ ".long " __stringify(x) "\n\t" \ ".popsection\n\t" +#define ASM_ANNOTATE (x) \ + "911:\n\t" \ + __ASM_ANNOTATE(911, x) +#define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR)#else /* __ASSEMBLY__ */
include/linux/objtool_types.h +2 -0
@@ -59,5 +59,7 @@ struct unwind_hint {*/ #define ANNOTYPE_NOENDBR 1#define ANNOTYPE_RETPOLINE_SAFE 2 +#define ANNOTYPE_INSTR_BEGIN 3 +#define ANNOTYPE_INSTR_END 4#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +2 -0
@@ -59,5 +59,7 @@ struct unwind_hint {*/ #define ANNOTYPE_NOENDBR 1#define ANNOTYPE_RETPOLINE_SAFE 2 +#define ANNOTYPE_INSTR_BEGIN 3 +#define ANNOTYPE_INSTR_END 4#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +10 -39
65 lines not on this page · raw message
Re: [PATCH v2 04/12] objtool: Convert instrumentation_{begin,end}() to ANNOTATE
include/linux/objtool.h line 56
Quoted 11 lines · Peter Zijlstra
+++
b/include/linux/objtool.hinclude/linux/objtool.h +2
-3@@ -51,13 +51,16 @@".long 998b\n\t" \ ".popsection\n\t" -#define ASM_ANNOTATE (x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE (s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \
It would probably be better for __ASM_ANNOTATE's callers to pass in the full label name (e.g. '911b') since they know where the label is? It could even be a named label.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 04/12] objtool: Convert instrumentation_{begin,end}() to ANNOTATE
Quoted 6 lines · Josh Poimboeuf
Quoted 5 lines · Peter Zijlstra
+++
b/include/linux/objtool.hinclude/linux/objtool.h +0
-0@@ -51,13 +51,16 @@".long 998b\n\t" \".popsection\n\t"
Quoted 10 lines
include/linux/objtool.h line 56
Quoted 6 lines
include/linux/objtool.h +2
-3-#define ASM_ANNOTATE (x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE (s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \It would probably be better for __ASM_ANNOTATE's callers to pass in the full label name (e.g. '911b') since they know where the label is? It could even be a named label.
I have this somewhere later, changing it here would be a pain because the existing annotations dont do it like that.
Peter Zijlstra
maintainerRe: [PATCH v2 04/12] objtool: Convert instrumentation_{begin,end}() to ANNOTATE
Quoted 18 lines · Peter Zijlstra
Quoted 6 lines · Josh Poimboeuf
Quoted 5 lines · Peter Zijlstra
+++
b/include/linux/objtool.hinclude/linux/objtool.h +0
-0@@ -51,13 +51,16 @@".long 998b\n\t" \".popsection\n\t" Quoted 10 lines
include/linux/objtool.h line 56
Quoted 6 lines
include/linux/objtool.h +2
-3-#define ASM_ANNOTATE (x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE (s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \It would probably be better for __ASM_ANNOTATE's callers to pass in the full label name (e.g. '911b') since they know where the label is? It could even be a named label.
Quoted 2 lines
I have this somewhere later, changing it here would be a pain because the existing annotations dont do it like that.
Reading is hard, yes let me do what you said.
Peter Zijlstra
maintainerRe: [PATCH v2 04/12] objtool: Convert instrumentation_{begin,end}() to ANNOTATE
In reply to Josh PoimboeufQuoted 6 lines · Josh Poimboeuf
Quoted 5 lines · Peter Zijlstra
+++
b/include/linux/objtool.h(quoted diff) +0
-0@@ -51,13 +51,16 @@ ".long 998b\n\t" \ ".popsection\n\t"
Quoted 10 lines
Review line 56
Quoted 6 lines
(quoted diff) +2
-3-#define ASM_ANNOTATE(x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE(s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \It would probably be better for __ASM_ANNOTATE's callers to pass in the full label name (e.g. '911b') since they know where the label is? It could even be a named label.
This seems to work.
include/linux/instrumentation.h +3 -2
@@ -6,11 +6,12 @@#include <linux/objtool.h> #include <linux/stringify.h> +#include <linux/args.h> /* Begin/end of an instrumentation safe region */ #define __instrumentation_begin (c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - __ASM_ANNOTATE(c, ANNOTYPE_INSTR_BEGIN) \ + __ASM_ANNOTATE(CONCATENATE(c, b), ANNOTYPE_INSTR_BEGIN) \ : : "i" (c)); \ })#define instrumentation_begin () __instrumentation_begin(__COUNTER__)
@@ -48,7 +49,7 @@*/ #define __instrumentation_end (c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - __ASM_ANNOTATE(c, ANNOTYPE_INSTR_END) \ + __ASM_ANNOTATE(CONCATENATE(c, b), ANNOTYPE_INSTR_END) \ : : "i" (c)); \ })#define instrumentation_end () __instrumentation_end(__COUNTER__)
include/linux/objtool.h +2 -2
@@ -53,13 +53,13 @@#define __ASM_ANNOTATE (s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long " __stringify(s) "b - .\n\t" \ + ".long " __stringify(s) " - .\n\t" \ ".long " __stringify(x) "\n\t" \ ".popsection\n\t"#define ASM_ANNOTATE (x) \ "911:\n\t" \ - __ASM_ANNOTATE(911, x) + __ASM_ANNOTATE(911b, x)#define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR)
Peter Zijlstra
maintainerPATCH v2 5/12 objtool: Convert VALIDATE_UNRET_BEGIN to ANNOTATE
In reply to Peter Zijlstra--- include/linux/objtool.h | 9 +++------ include/linux/objtool_types.h | 1 + tools/include/linux/objtool_types.h | 1 + tools/objtool/check.c | 28 +++++----------------------- 4 files changed, 10 insertions(+), 29 deletions(-)
include/linux/objtool.h +3 -6
@@ -128,15 +128,12 @@* NOTE: The macro must be used at the beginning of a global symbol, otherwise * it will be ignored. */ -.macro VALIDATE_UNRET_BEGIN#if defined(CONFIG_NOINSTR_VALIDATION ) && \ (defined(CONFIG_MITIGATION_UNRET_ENTRY ) || defined(CONFIG_MITIGATION_SRSO )) -.Lhere_\@: - .pushsection .discard.validate_unret - .long .Lhere_\@ - . - .popsection +#define VALIDATE_UNRET_BEGIN ANNOTATE type=ANNOTYPE_UNRET_BEGIN +#else +#define VALIDATE_UNRET_BEGIN #endif -.endm .macro REACHABLE .Lhere_\@:
include/linux/objtool_types.h +1 -0
@@ -61,5 +61,6 @@ struct unwind_hint {#define ANNOTYPE_RETPOLINE_SAFE 2#define ANNOTYPE_INSTR_BEGIN 3#define ANNOTYPE_INSTR_END 4 +#define ANNOTYPE_UNRET_BEGIN 5#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +1 -0
@@ -61,5 +61,6 @@ struct unwind_hint {#define ANNOTYPE_RETPOLINE_SAFE 2#define ANNOTYPE_INSTR_BEGIN 3#define ANNOTYPE_INSTR_END 4 +#define ANNOTYPE_UNRET_BEGIN 5#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +5 -23
@@ -2450,33 +2450,15 @@ static int __annotate_instr(int type, streturn 0 ; } -static int read_validate_unret_hints (struct objtool_file *file) +static int __annotate_unret (int type,struct instruction *insn) { -struct section *rsec; -struct instruction *insn; -struct reloc *reloc; - - rsec =find_section_by_name (file->elf,".rela.discard.validate_unret" ); -if (!rsec) +if (type !=ANNOTYPE_UNRET_BEGIN )return 0 ; -for_each_reloc (rsec, reloc) { -if (reloc->sym->type !=STT_SECTION ) { -WARN ("unexpected relocation symbol type in %s" , rsec->name); -return -1 ; - } - - insn =find_insn (file, reloc->sym->sec,reloc_addend (reloc)); -if (!insn) { -WARN ("bad .discard.instr_end entry" ); -return -1 ; - } - insn->unret =1 ; - } - + insn->unret =1 ;return 0 ; -} +}static int read_intra_function_calls (struct objtool_file *file) {
@@ -2697,7 +2679,7 @@ static int decode_sections(struct objtooif (ret)return ret; - ret =read_validate_unret_hints (file); + ret =read_annotate (file, __annotate_unret);if (ret)return ret;
Peter Zijlstra
maintainerPATCH v2 6/12 objtool: Convert ANNOTATE_IGNORE_ALTERNATIVE to ANNOTATE
In reply to Peter Zijlstra--- arch/x86/include/asm/alternative.h | 14 ++--------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 45 ++++++++---------------------------- 4 files changed, 15 insertions(+), 46 deletions(-)
arch/x86/include/asm/alternative.h +3 -11
@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/stringify.h> +#include <linux/objtool.h> #include <asm/asm.h> #define ALT_FLAGS_SHIFT 16
@@ -55,11 +56,7 @@* objtool annotation to ignore the alternatives and only consider the original * instruction(s). */ -#define ANNOTATE_IGNORE_ALTERNATIVE \ - "999:\n\t" \ - ".pushsection .discard.ignore_alts\n\t" \ - ".long 999b\n\t" \ - ".popsection\n\t" +#define ANNOTATE_IGNORE_ALTERNATIVE ASM_ANNOTATE(ANNOTYPE_IGNORE_ALTS)/* * The patching flags are part of the upper bits of the @ft_flags parameter when
@@ -349,12 +346,7 @@ static inline int alternatives_text_rese* objtool annotation to ignore the alternatives and only consider the original * instruction(s). */ -.macro ANNOTATE_IGNORE_ALTERNATIVE - .Lannotate_\@: - .pushsection .discard.ignore_alts - .long .Lannotate_\@ - .popsection -.endm +#define ANNOTATE_IGNORE_ALTERNATIVE ANNOTATE type=ANNOTYPE_IGNORE_ALTS/* * Issue one struct alt_instr descriptor entry (need to put it into
include/linux/objtool_types.h +1 -0
@@ -62,5 +62,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_BEGIN 3#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5 +#define ANNOTYPE_IGNORE_ALTS 6#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +1 -0
@@ -62,5 +62,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_BEGIN 3#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5 +#define ANNOTYPE_IGNORE_ALTS 6#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +10 -35
63 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 7/12 objtool: Convert ANNOTATE_INTRA_FUNCTION_CALLS to ANNOTATE
In reply to Peter Zijlstra--- include/linux/objtool.h | 16 ++---- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 96 ++++++++++++++---------------------- 4 files changed, 47 insertions(+), 67 deletions(-)
include/linux/objtool.h +6 -10
@@ -66,16 +66,6 @@#else /* __ASSEMBLY__ */ /* -* This macro indicates that the following intra-function call is valid. -* Any non-annotated intra-function call will cause objtool to issue a warning. -*/ -#define ANNOTATE_INTRA_FUNCTION_CALL \ - 999: \ - .pushsection .discard.intra_function_calls; \ - .long 999b; \ - .popsection; - -/* * In asm, there are two kinds of code: normal C-type callable functions and * the rest. The normal callable functions can be called by other code, and * don't do anything unusual with the stack. Such normal callable functions
@@ -152,6 +142,12 @@#define ANNOTATE_NOENDBR ANNOTATE type=ANNOTYPE_NOENDBR +/* +* This macro indicates that the following intra-function call is valid. +* Any non-annotated intra-function call will cause objtool to issue a warning. +*/ +#define ANNOTATE_INTRA_FUNCTION_CALL ANNOTATE type=ANNOTYPE_INTRA_FUNCTION_CALLS +#endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */
include/linux/objtool_types.h +1 -0
@@ -63,5 +63,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +1 -0
@@ -63,5 +63,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +39 -57
150 lines not on this page · raw message
Re: [PATCH v2 07/12] objtool: Convert ANNOTATE_INTRA_FUNCTION_CALLS to ANNOTATE
include/linux/objtool_types.h struct unwind_hint { · line 66
Quoted 6 lines · Peter Zijlstra
+++
b/include/linux/objtool_types.hinclude/linux/objtool_types.h +1
-0@@ -63,5 +63,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7
Should be a singular "CALL"?
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 07/12] objtool: Convert ANNOTATE_INTRA_FUNCTION_CALLS to ANNOTATE
Quoted 9 lines · Josh Poimboeuf
include/linux/objtool_types.h struct unwind_hint { · line 66
Quoted 6 lines · Peter Zijlstra
+++
b/include/linux/objtool_types.hinclude/linux/objtool_types.h +1
-0@@ -63,5 +63,6 @@ struct unwind_hint {#define ANNOTYPE_INSTR_END 4#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7Should be a singular "CALL"?
Sure, one regex on the patch sorted that :-)
Peter Zijlstra
maintainerPATCH v2 8/12 objtool: Collapse annotate sequences
In reply to Peter ZijlstraReduce read_annotate() runs by collapsing subsequent runs into a single call.
--- tools/objtool/check.c | 87 ++++++++++++++++++-------------------------------- 1 file changed, 32 insertions(+), 55 deletions(-)
tools/objtool/check.c +32 -55
@@ -2308,21 +2308,24 @@ static int read_annotate(struct objtool_return 0 ; } -static int __annotate_ignore_alts (struct objtool_file *file,int type,struct instruction *insn) +static int __annotate_early (struct objtool_file *file,int type,struct instruction *insn) { -if (type !=ANNOTYPE_IGNORE_ALTS ) -return 0 ; +switch (type) { +case ANNOTYPE_IGNORE_ALTS : + insn->ignore_alts = true; +break ; - insn->ignore_alts = true; -return 0 ; -} +/* +* Must be before read_unwind_hints() since that needs insn->noendbr. +*/ +case ANNOTYPE_NOENDBR : + insn->noendbr =1 ; +break ; -static int __annotate_noendbr (struct objtool_file *file,int type,struct instruction *insn) -{ -if (type !=ANNOTYPE_NOENDBR ) -return 0 ; +default : +break ; + } - insn->noendbr =1 ;return 0 ; }
@@ -2356,26 +2359,21 @@ static int __annotate_ifc(struct objtoolreturn 0 ; } -static int __annotate_retpoline_safe (struct objtool_file *file,int type,struct instruction *insn) +static int __annotate_late (struct objtool_file *file,int type,struct instruction *insn) { -if (type !=ANNOTYPE_RETPOLINE_SAFE ) -return 0 ; - -if (insn->type !=INSN_JUMP_DYNAMIC && - insn->type !=INSN_CALL_DYNAMIC && - insn->type !=INSN_RETURN && - insn->type !=INSN_NOP ) { -WARN_INSN (insn,"retpoline_safe hint not an indirect jump/call/ret/nop" ); -return -1 ; - } +switch (type) { +case ANNOTYPE_RETPOLINE_SAFE : +if (insn->type !=INSN_JUMP_DYNAMIC && + insn->type !=INSN_CALL_DYNAMIC && + insn->type !=INSN_RETURN && + insn->type !=INSN_NOP ) { +WARN_INSN (insn,"retpoline_safe hint not an indirect jump/call/ret/nop" ); +return -1 ; + } - insn->retpoline_safe = true; -return 0 ; -} + insn->retpoline_safe = true; +break ; -static int __annotate_instr (struct objtool_file *file,int type,struct instruction *insn) -{ -switch (type) {case ANNOTYPE_INSTR_BEGIN : insn->instr++;break ;
@@ -2384,6 +2382,10 @@ static int __annotate_instr(struct objto insn->instr--;break ; +case ANNOTYPE_UNRET_BEGIN : + insn->unret =1 ; +break ; +default :break ; }
@@ -2391,16 +2393,6 @@ static int __annotate_instr(struct objtoreturn 0 ; } -static int __annotate_unret (struct objtool_file *file,int type,struct instruction *insn) -{ -if (type !=ANNOTYPE_UNRET_BEGIN ) -return 0 ; - - insn->unret =1 ; -return 0 ; - -} -/* * Return true if name matches an instrumentation function, where calls to that * function from noinstr code can safely be removed, but compilers won't do so.
@@ -2507,14 +2499,7 @@ static int decode_sections(struct objtooadd_ignores (file);add_uaccess_safe (file); - ret =read_annotate (file, __annotate_ignore_alts); -if (ret) -return ret; - -/* -* Must be before read_unwind_hints() since that needs insn->noendbr. -*/ - ret =read_annotate (file, __annotate_noendbr); + ret =read_annotate (file, __annotate_early);if (ret)return ret;
@@ -2560,15 +2545,7 @@ static int decode_sections(struct objtooif (ret)return ret; - ret =read_annotate (file, __annotate_retpoline_safe); -if (ret) -return ret; - - ret =read_annotate (file, __annotate_instr); -if (ret) -return ret; - - ret =read_annotate (file, __annotate_unret); + ret =read_annotate (file, __annotate_late);if (ret)return ret;
Peter Zijlstra
maintainerPATCH v2 9/12 x86/nospec: JMP_NOSPEC
In reply to Peter Zijlstra--- arch/x86/include/asm/nospec-branch.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
arch/x86/include/asm/nospec-branch.h +32 -0
@@ -403,6 +403,17 @@ static inline void call_depth_return_thu "call *%[thunk_target]\n", \ X86_FEATURE_RETPOLINE_LFENCE) +# define JMP_NOSPEC \ + ALTERNATIVE_2( \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + "jmp __x86_indirect_thunk_%V[thunk_target]\n", \ + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_LFENCE) +# define THUNK_TARGET (addr) [thunk_target] "r" (addr)#else /* CONFIG_X86_32 */
@@ -433,10 +444,31 @@ static inline void call_depth_return_thu "call *%[thunk_target]\n", \ X86_FEATURE_RETPOLINE_LFENCE) +# define JMP_NOSPEC \ + ALTERNATIVE_2( \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + " jmp 901f;\n" \ + " .align 16\n" \ + "901: call 903f;\n" \ + "902: pause;\n" \ + " lfence;\n" \ + " jmp 902b;\n" \ + " .align 16\n" \ + "903: lea 4(%%esp), %%esp;\n" \ + " pushl %[thunk_target];\n" \ + " ret;\n", \ + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_LFENCE) +# define THUNK_TARGET (addr) [thunk_target] "rm" (addr)#endif #else /* No retpoline for C / inline asm */ # define CALL_NOSPEC "call *%[thunk_target]\n" +# define JMP_NOSPEC "jmp *%[thunk_target]\n"# define THUNK_TARGET (addr) [thunk_target] "rm" (addr)#endif
Peter Zijlstra
maintainerPATCH v2 10/12 x86,nospec: Simplify {JMP,CALL}_NOSPEC (part 2)
In reply to Peter ZijlstraCounterpart to 09d09531a51a ("x86,nospec: Simplify {JMP,CALL}_NOSPEC"), x86_64 will rewrite all this anyway, see apply_retpoline.
--- arch/x86/include/asm/nospec-branch.h | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-)
arch/x86/include/asm/nospec-branch.h +11 -18
@@ -429,31 +429,24 @@ static inline void call_depth_return_thu#ifdef CONFIG_X86_64 +#define __CS_PREFIX \ + ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ + ".ifc %V[thunk_target],\\rs\n" \ + ".byte 0x2e\n" \ + ".endif\n" \ + ".endr\n" +/* * Inline asm uses the %V modifier which is only in newer GCC * which is ensured when CONFIG_MITIGATION_RETPOLINE is defined. */ # define CALL_NOSPEC \ - ALTERNATIVE_2( \ - ANNOTATE_RETPOLINE_SAFE \ - "call *%[thunk_target]\n", \ - "call __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE, \ - "lfence;\n" \ - ANNOTATE_RETPOLINE_SAFE \ - "call *%[thunk_target]\n", \ - X86_FEATURE_RETPOLINE_LFENCE) + __CS_PREFIX \ + "call __x86_indirect_thunk_%V[thunk_target]\n"# define JMP_NOSPEC \ - ALTERNATIVE_2( \ - ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ - "jmp __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE, \ - "lfence;\n" \ - ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ - X86_FEATURE_RETPOLINE_LFENCE) + __CS_PREFIX \ + "jmp __x86_indirect_thunk_%V[thunk_target]\n"# define THUNK_TARGET (addr) [thunk_target] "r" (addr)
Re: [PATCH v2 10/12] x86,nospec: Simplify {JMP,CALL}_NOSPEC (part 2)
Quoted 12 lines · Peter Zijlstra
Counterpart to
09d09531a51a("x86,nospec: Simplify {JMP,CALL}_NOSPEC"),x86_64will rewrite all this anyway, seeapply_retpoline.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-)
arch/x86/include/asm/nospec-branch.h static inline void call_depth_return_thu · line 437
Quoted 9 lines
arch/x86/include/asm/nospec-branch.h +6
-0@@ -429,31 +429,24 @@ static inline void call_depth_return_thu#ifdef CONFIG_X86_64 +#define __CS_PREFIX \ + ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ + ".ifc %V[thunk_target],\\rs\n" \ + ".byte 0x2e\n" \ + ".endif\n" \ + ".endr\n"
After staring at this for some minutes I'm thinking it would be helpful to add a comment saying this is equivalent to -mindirect-branch-cs-prefix.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 10/12] x86,nospec: Simplify {JMP,CALL}_NOSPEC (part 2)
Quoted 16 lines · Josh Poimboeuf
Quoted 15 lines · Peter Zijlstra
Counterpart to
09d09531a51a("x86,nospec: Simplify {JMP,CALL}_NOSPEC"),x86_64will rewrite all this anyway, seeapply_retpoline.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-)arch/x86/include/asm/nospec-branch.h +0
-0@@ -429,31 +429,24 @@ static inline void call_depth_return_thu#ifdef CONFIG_X86_64
Quoted 10 lines
arch/x86/include/asm/nospec-branch.h static inline void call_depth_return_thu · line 437
Quoted 6 lines
arch/x86/include/asm/nospec-branch.h +6
-0+#define __CS_PREFIX \ + ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ + ".ifc %V[thunk_target],\\rs\n" \ + ".byte 0x2e\n" \ + ".endif\n" \ + ".endr\n"After staring at this for some minutes I'm thinking it would be helpful to add a comment saying this is equivalent to -mindirect-branch-cs-prefix.
I'll just copy-paste the comment from the other __CS_PREFIX elsewhere in this file :-)
Peter Zijlstra
maintainerPATCH v2 11/12 x86/kvm/emulate: Implement test_cc() in C
In reply to Peter ZijlstraCurrent test_cc() uses the fastop infrastructure to test flags using SETcc instructions. However, int3_emulate_jcc() already fully implements the flags->CC mapping, use that.
Removes a pile of gnarly asm.
--- arch/x86/include/asm/text-patching.h | 20 +++++++++++++------- arch/x86/kvm/emulate.c | 34 ++-------------------------------- 2 files changed, 15 insertions(+), 39 deletions(-)
arch/x86/include/asm/text-patching.h +13 -7
@@ -176,9 +176,9 @@ void int3_emulate_ret(struct pt_regs *re }static __always_inline -void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +bool __emulate_cc(unsigned long flags, u8 cc) { -static const unsigned long jcc_mask[6 ] = { +static const unsigned long cc_mask[6 ] = { [0 ] =X86_EFLAGS_OF , [1 ] =X86_EFLAGS_CF , [2 ] =X86_EFLAGS_ZF ,
@@ -191,15 +191,21 @@ void int3_emulate_jcc(struct pt_regs *rebool match;if (cc <0xc ) { - match = regs->flags & jcc_mask[cc >>1 ]; + match = flags & cc_mask[cc >>1 ]; }else { - match = ((regs->flags &X86_EFLAGS_SF ) >>X86_EFLAGS_SF_BIT ) ^ - ((regs->flags &X86_EFLAGS_OF ) >>X86_EFLAGS_OF_BIT ); + match = ((flags &X86_EFLAGS_SF ) >>X86_EFLAGS_SF_BIT ) ^ + ((flags &X86_EFLAGS_OF ) >>X86_EFLAGS_OF_BIT );if (cc >=0xe ) - match = match || (regs->flags &X86_EFLAGS_ZF ); + match = match || (flags &X86_EFLAGS_ZF ); } -if ((match && !invert) || (!match && invert)) +return (match && !invert) || (!match && invert); +} + +static __always_inline +void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +{ +if (__emulate_cc (regs->flags, cc)) ip += disp;int3_emulate_jmp (regs, ip);
arch/x86/kvm/emulate.c +2 -32
@@ -26,6 +26,7 @@#include <asm/debugreg.h> #include <asm/nospec-branch.h> #include <asm/ibt.h> +#include <asm/text-patching.h> #include "x86.h" #include "tss.h"
@@ -416,31 +417,6 @@ static int fastop(struct x86_emulate_ctx ON64(FOP3E(op##q, rax, rdx, cl)) \ FOP_END -/* Special case for SETcc - 1 instruction per cc */ -#define FOP_SETCC (op) \ - FOP_FUNC(op) \ - #op " %al \n\t" \ - FOP_RET(op) - -FOP_START(setcc) -FOP_SETCC(seto) -FOP_SETCC(setno) -FOP_SETCC(setc) -FOP_SETCC(setnc) -FOP_SETCC(setz) -FOP_SETCC(setnz) -FOP_SETCC(setbe) -FOP_SETCC(setnbe) -FOP_SETCC(sets) -FOP_SETCC(setns) -FOP_SETCC(setp) -FOP_SETCC(setnp) -FOP_SETCC(setl) -FOP_SETCC(setnl) -FOP_SETCC(setle) -FOP_SETCC(setnle) -FOP_END; - FOP_START(salc) FOP_FUNC(salc)"pushf; sbb %al, %al; popf \n\t"
@@ -1064,13 +1040,7 @@ static int em_bsr_c(struct x86_emulate_c static __always_inline u8 test_cc(unsigned int condition, unsigned long flags) { -u8 rc; -void (*fop)(void ) = (void *)em_setcc +FASTOP_SIZE * (condition &0xf ); - - flags = (flags &EFLAGS_MASK ) |X86_EFLAGS_IF ; - asm("push %[flags]; popf; " CALL_NOSPEC - : "=a"(rc), ASM_CALL_CONSTRAINT : [thunk_target]"r"(fop), [flags]"r"(flags)); -return rc; +return __emulate_cc (flags, condition &0xf ); }static void fetch_register_operand (struct operand *op)
Re: [PATCH v2 11/12] x86/kvm/emulate: Implement test_cc() in C
Can you use "KVM: x86" for the scope? "x86/kvm" is used for guest changes, i.e. for paravirt code when running as a KVM guest.
Quoted 3 lines · Peter Zijlstra
Current
test_cc()uses the fastop infrastructure to test flags using SETcc instructions. However,int3_emulate_jcc()already fully implements theflags->CCmapping, use that.
I think it's worth presenting this as a revert of sorts, even though it's not a strict revert. KVM also emulated jcc-like operations in software prior to commit 9ae9febae950 ("KVM: x86 emulator: covert SETCC to fastop"), i.e. the fastop madness was introduced for performance reasons, not because writing the code was hard.
Looking at the output of the fastop versus __emulate_cc(), the worst case cost is two extra conditional branches. Assuming that eliminating the test_cc() fastop code avoids having to add another special case in objtool, I am a-ok with the tradeoff. Especially since emulating instructions that use test_cc() is largely limited to older hardware running guest firmware that uses (emulated) SMM, and maybe a few bespoke use cases. E.g. I get literally zero hits on test_cc() when booting a 24 vCPU VM (64-bit or 32-bit kernel) with EPT and unrestricted guest disabled, as the OVMF build I use doesn't rely on SMM.
And FWIW, a straight revert appears to generate worse code. I see no reason to bring it back.
With a massaged shortlog+changelog,
fastop:
0x0000000000042c41 <+1537>: movzbl 0x61(%rbp),%eax 0x0000000000042c45 <+1541>: mov 0x10(%rbp),%rdx 0x0000000000042c49 <+1545>: shl $0x4,%rax 0x0000000000042c4d <+1549>: and $0xf0,%eax 0x0000000000042c52 <+1554>: and $0x8d5,%edx 0x0000000000042c58 <+1560>: or $0x2,%dh 0x0000000000042c5b <+1563>: add $0x0,%rax 0x0000000000042c61 <+1569>: push %rdx 0x0000000000042c62 <+1570>: popf 0x0000000000042c63 <+1571>: call 0x42c68 <x86_emulate_insn+1576> 0x0000000000042c68 <+1576>: mov %eax,%edx 0x0000000000042c6a <+1578>: xor %eax,%eax 0x0000000000042c6c <+1580>: test %dl,%dl 0x0000000000042c6e <+1582>: jne 0x42f24 <x86_emulate_insn+2276>
__emulate_cc:
0x0000000000042b95 <+1541>: movzbl 0x61(%rbp),%eax
0x0000000000042b99 <+1545>: mov 0x10(%rbp),%rcx
0x0000000000042b9d <+1549>: and $0xf,%eax
0x0000000000042ba0 <+1552>: cmp $0xb,%al
0x0000000000042ba2 <+1554>: ja 0x42e90 <x86_emulate_insn+2304>
0x0000000000042e90 <+2304>: mov %rcx,%rdx
0x0000000000042e93 <+2307>: mov %rcx,%rsi
0x0000000000042e96 <+2310>: shr $0x7,%rdx
0x0000000000042e9a <+2314>: shr $0xb,%rsi
0x0000000000042e9e <+2318>: xor %rsi,%rdx
0x0000000000042ea1 <+2321>: cmp $0xd,%al
0x0000000000042ea3 <+2323>: ja 0x4339a <x86_emulate_insn+3594>
0x000000000004339a <+3594>: and $0x1,%edx
0x000000000004339d <+3597>: and $0x40,%ecx
0x00000000000433a0 <+3600>: or %rcx,%rdx
0x00000000000433a3 <+3603>: setne %dl
0x00000000000433a6 <+3606>: jmp 0x42bba <x86_emulate_insn+1578>
0x0000000000042ea9 <+2329>: and $0x1,%edx
0x0000000000042eac <+2332>: jmp 0x42bba <x86_emulate_insn+1578>
0x0000000000042ba8 <+1560>: mov %eax,%edx
0x0000000000042baa <+1562>: shr %dl
0x0000000000042bac <+1564>: and $0x7,%edx
0x0000000000042baf <+1567>: and 0x0(,%rdx,8),%rcx
0x0000000000042bb7 <+1575>: setne %dl
0x0000000000042bba <+1578>: test $0x1,%al
0x0000000000042bbc <+1580>: jne 0x43323 <x86_emulate_insn+3475>
0x0000000000043323 <+3475>: test %dl,%dl
0x0000000000043325 <+3477>: jne 0x4332f <x86_emulate_insn+3487>
0x0000000000043327 <+3479>: test $0x1,%al
0x0000000000043329 <+3481>: jne 0x42bca <x86_emulate_insn+1594>
0x000000000004332f <+3487>: xor %eax,%eax
0x0000000000043331 <+3489>: jmp 0x42be0 <x86_emulate_insn+1616>
0x0000000000042bc2 <+1586>: test %dl,%dl
0x0000000000042bc4 <+1588>: je 0x43327 <x86_emulate_insn+3479>
0x0000000000043327 <+3479>: test $0x1,%al
0x0000000000043329 <+3481>: jne 0x42bca <x86_emulate_insn+1594>
0x000000000004332f <+3487>: xor %eax,%eax
0x0000000000043331 <+3489>: jmp 0x42be0 <x86_emulate_insn+1616>
0x0000000000042bca <+1594>: movslq 0xd0(%rbp),%rsi
0x0000000000042bd1 <+1601>: mov %rbp,%rdi
0x0000000000042bd4 <+1604>: add 0x90(%rbp),%rsi
0x0000000000042bdb <+1611>: call 0x3a7c0 <assign_eip>revert:
0x0000000000042bc9 <+1545>: movzbl 0x61(%rbp),%esi
0x0000000000042bcd <+1549>: mov 0x10(%rbp),%rdx
0x0000000000042bd1 <+1553>: mov %esi,%eax
0x0000000000042bd3 <+1555>: shr %eax
0x0000000000042bd5 <+1557>: and $0x7,%eax
0x0000000000042bd8 <+1560>: cmp $0x4,%eax
0x0000000000042bdb <+1563>: je 0x42ed4 <x86_emulate_insn+2324>
0x0000000000042ed4 <+2324>: mov %edx,%ecx
0x0000000000042ed6 <+2326>: and $0x80,%ecx
0x0000000000042edc <+2332>: jmp 0x42bff <x86_emulate_insn+1599>
0x0000000000042be1 <+1569>: ja 0x43225 <x86_emulate_insn+3173>
0x0000000000043225 <+3173>: cmp $0x6,%eax
0x0000000000043228 <+3176>: je 0x434ec <x86_emulate_insn+3884>
0x00000000000434ec <+3884>: xor %edi,%edi
0x00000000000434ee <+3886>: jmp 0x43238 <x86_emulate_insn+3192>
0x000000000004322e <+3182>: mov %edx,%edi
0x0000000000043230 <+3184>: and $0x40,%edi
0x0000000000043233 <+3187>: cmp $0x7,%eax
0x0000000000043236 <+3190>: jne 0x4325c <x86_emulate_insn+3228>
0x000000000004325c <+3228>: mov %edx,%ecx
0x000000000004325e <+3230>: and $0x4,%ecx
0x0000000000043261 <+3233>: cmp $0x5,%eax
0x0000000000043264 <+3236>: je 0x42bff <x86_emulate_insn+1599>
0x000000000004326a <+3242>: jmp 0x43218 <x86_emulate_insn+3160>
0x0000000000043238 <+3192>: mov %rdx,%rcx
0x000000000004323b <+3195>: shr $0xb,%rdx
0x000000000004323f <+3199>: shr $0x7,%rcx
0x0000000000043243 <+3203>: xor %edx,%ecx
0x0000000000043245 <+3205>: and $0x1,%ecx
0x0000000000043248 <+3208>: or %edi,%ecx
0x000000000004324a <+3210>: jmp 0x42bff <x86_emulate_insn+1599>
0x0000000000042be7 <+1575>: mov %edx,%ecx
0x0000000000042be9 <+1577>: and $0x40,%ecx
0x0000000000042bec <+1580>: cmp $0x2,%eax
0x0000000000042bef <+1583>: je 0x42bff <x86_emulate_insn+1599>
0x0000000000042bf1 <+1585>: mov %edx,%ecx
0x0000000000042bf3 <+1587>: and $0x41,%ecx
0x0000000000042bf6 <+1590>: cmp $0x3,%eax
0x0000000000042bf9 <+1593>: jne 0x4320a <x86_emulate_insn+3146>
0x000000000004320a <+3146>: mov %edx,%ecx
0x000000000004320c <+3148>: and $0x1,%ecx
0x000000000004320f <+3151>: cmp $0x1,%eax
0x0000000000043212 <+3154>: je 0x42bff <x86_emulate_insn+1599>
0x0000000000043218 <+3160>: mov %edx,%ecx
0x000000000004321a <+3162>: and $0x800,%ecx
0x0000000000043220 <+3168>: jmp 0x42bff <x86_emulate_insn+1599>
0x0000000000042bff <+1599>: test %ecx,%ecx
0x0000000000042c01 <+1601>: setne %dl
0x0000000000042c04 <+1604>: and $0x1,%esi
0x0000000000042c07 <+1607>: xor %eax,%eax
0x0000000000042c09 <+1609>: cmp %sil,%dl
0x0000000000042c0c <+1612>: je 0x42c24 <x86_emulate_insn+1636>
0x0000000000042c0e <+1614>: movslq 0xd0(%rbp),%rsi
0x0000000000042c15 <+1621>: mov %rbp,%rdi
0x0000000000042c18 <+1624>: add 0x90(%rbp),%rsi
0x0000000000042c1f <+1631>: call 0x3a7c0 <assign_eip>Peter Zijlstra
maintainerPATCH v2 12/12 x86/kvm/emulate: Avoid RET for fastops
In reply to Peter ZijlstraSince there is only a single fastop() function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz.
Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here).
Objtool strenuously complains about this:
- indirect call without a .rodata, fails to determine JUMP_TABLE, annotate
- fastop functions fall through, exception
- unreachable instruction after fastop_return, save/restore
--- arch/x86/kvm/emulate.c | 20 +++++++++++++++----- include/linux/objtool_types.h | 1 + tools/include/linux/objtool_types.h | 1 + tools/objtool/check.c | 11 ++++++++++- 4 files changed, 27 insertions(+), 6 deletions(-)
arch/x86/kvm/emulate.c +15 -5
@@ -285,8 +285,8 @@ static void invalidate_registers(struct* different operand sizes can be reached by calculation, rather than a jump * table (which would be bigger than the code). * -* The 16 byte alignment, considering 5 bytes for the RET thunk, 3 for ENDBR -* and 1 for the straight line speculation INT3, leaves 7 bytes for the +* The 16 byte alignment, considering 5 bytes for the JMP, 4 for ENDBR +* and 1 for the straight line speculation INT3, leaves 6 bytes for the * body of the function. Currently none is larger than 4. */ static int fastop (struct x86_emulate_ctxt *ctxt,fastop_t fop);
@@ -304,7 +304,7 @@ static int fastop(struct x86_emulate_ctx __FOP_FUNC(#name)#define __FOP_RET (name) \ - "11: " ASM_RET \ + "11: jmp fastop_return; int3 \n\t" \ ".size " name ", .-" name "\n\t"#define FOP_RET (name) \
@@ -5071,14 +5071,24 @@ static void fetch_possible_mmx_operand(skvm_read_mmx_reg (op->addr.mm, &op->mm_val); } -static int fastop (struct x86_emulate_ctxt *ctxt,fastop_t fop) +/* +* All the FASTOP magic above relies on there being *one* instance of this +* so it can JMP back, avoiding RET and it's various thunks. +*/ +static noinline int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) {ulong flags = (ctxt->eflags &EFLAGS_MASK ) |X86_EFLAGS_IF ;if (!(ctxt->d & ByteOp)) fop +=__ffs (ctxt->dst.bytes) *FASTOP_SIZE ; - asm("push %[flags]; popf; " CALL_NOSPEC " ; pushf; pop %[flags]\n" + asm("push %[flags]; popf \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_SAVE, 0, 0, 0) +ASM_ANNOTATE (ANNOTYPE_JUMP_TABLE) + JMP_NOSPEC +"fastop_return: \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_RESTORE, 0, 0, 0) +"pushf; pop %[flags]\n" : "+a"(ctxt->dst.val), "+d"(ctxt->src.val), [flags]"+D"(flags), [thunk_target]"+S"(fop), ASM_CALL_CONSTRAINT : "c"(ctxt->src2.val));
include/linux/objtool_types.h +1 -0
@@ -64,5 +64,6 @@ struct unwind_hint {#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6#define ANNOTYPE_INTRA_FUNCTION_CALLS 7 +#define ANNOTYPE_JUMP_TABLE 8#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/include/linux/objtool_types.h +1 -0
@@ -64,5 +64,6 @@ struct unwind_hint {#define ANNOTYPE_UNRET_BEGIN 5#define ANNOTYPE_IGNORE_ALTS 6#define ANNOTYPE_INTRA_FUNCTION_CALLS 7 +#define ANNOTYPE_JUMP_TABLE 8#endif /* _LINUX_OBJTOOL_TYPES_H */
tools/objtool/check.c +10 -1
@@ -2386,6 +2386,14 @@ static int __annotate_late(struct objtoo insn->unret =1 ;break ; +/* +* Must be after add_jump_table(); for it doesn't set a sane +* _jump_table value. +*/ +case ANNOTYPE_JUMP_TABLE : + insn->_jump_table = (void *)1 ; +break ; +default :break ; }
@@ -3459,7 +3467,8 @@ static int validate_branch(struct objtooif (func &&insn_func (insn) && func !=insn_func (insn)->pfunc) {/* Ignore KCFI type preambles, which always fall through */ if (!strncmp (func->name,"__cfi_" ,6 ) || - !strncmp (func->name,"__pfx_" ,6 )) + !strncmp (func->name,"__pfx_" ,6 ) || + !strcmp (insn_func (insn)->name,"fastop" ))return 0 ; WARN("%s() falls through to next function %s()",
Peter Zijlstra
maintainerRe: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops
Quoted 12 lines · Peter Zijlstra
(quoted diff) +5
-1+/* + * All the FASTOP magic above relies on there being *one* instance of this + * so it can JMP back, avoiding RET and it's various thunks. + */ +static noinline int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) { ulong flags = (ctxt->eflags & EFLAGS_MASK) | X86_EFLAGS_IF; if (!(ctxt->d & ByteOp)) fop += __ffs(ctxt->dst.bytes) * FASTOP_SIZE; - asm("push %[flags]; popf; " CALL_NOSPEC " ; pushf; pop %[flags]\n"
Review
Quoted 10 lines
(quoted diff) +7
-0+ asm("push %[flags]; popf \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_SAVE, 0, 0, 0) + ASM_ANNOTATE(ANNOTYPE_JUMP_TABLE) + JMP_NOSPEC + "fastop_return: \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_RESTORE, 0, 0, 0) + "pushf; pop %[flags]\n" : "+a"(ctxt->dst.val), "+d"(ctxt->src.val), [flags]"+D"(flags), [thunk_target]"+S"(fop), ASM_CALL_CONSTRAINT : "c"(ctxt->src2.val));
Do Andrew is telling me the compiler is free to mess this up... Notably:
https://github.com/llvm/llvm-project/issues/92161
In lieu of that, I wrote the below hack. It makes objtool sad (it don't like STT_FUNC calling STT_NOTYPE), but it should work if we ever run into the compiler being daft like that (it should fail to compile because of the duplicate fastop_return label, so it's not silent failure).
Wear protective eye gear before continuing...
---
arch/x86/include/asm/nospec-branch.h +14 -11
@@ -429,9 +429,9 @@ static inline void call_depth_return_thu#ifdef CONFIG_X86_64 -#define __CS_PREFIX \ +#define __CS_PREFIX (reg) \ ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ - ".ifc %V[thunk_target],\\rs\n" \ + ".ifc " reg ",\\rs\n" \ ".byte 0x2e\n" \ ".endif\n" \ ".endr\n"
@@ -441,12 +441,12 @@ static inline void call_depth_return_thu* which is ensured when CONFIG_MITIGATION_RETPOLINE is defined. */ # define CALL_NOSPEC \ - __CS_PREFIX \ + __CS_PREFIX("%V[thunk_target]") \ "call __x86_indirect_thunk_%V[thunk_target]\n" -# define JMP_NOSPEC \ - __CS_PREFIX \ - "jmp __x86_indirect_thunk_%V[thunk_target]\n" +# define __JMP_NOSPEC (reg) \ + __CS_PREFIX(reg) \ + "jmp __x86_indirect_thunk_" reg "\n"# define THUNK_TARGET (addr) [thunk_target] "r" (addr)
@@ -478,10 +478,10 @@ static inline void call_depth_return_thu "call *%[thunk_target]\n", \ X86_FEATURE_RETPOLINE_LFENCE) -# define JMP_NOSPEC \ +# define __JMP_NOSPEC (reg) \ ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ + "jmp *%%" reg "\n", \ " jmp 901f;\n" \ " .align 16\n" \ "901: call 903f;\n" \
@@ -490,22 +490,25 @@ static inline void call_depth_return_thu " jmp 902b;\n" \ " .align 16\n" \ "903: lea 4(%%esp), %%esp;\n" \ - " pushl %[thunk_target];\n" \ + " pushl %%" reg "\n" \ " ret;\n", \ X86_FEATURE_RETPOLINE, \ "lfence;\n" \ ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ + "jmp *%%" reg "\n", \ X86_FEATURE_RETPOLINE_LFENCE)# define THUNK_TARGET (addr) [thunk_target] "rm" (addr)#endif +#else /* No retpoline for C / inline asm */ # define CALL_NOSPEC "call *%[thunk_target]\n" -# define JMP_NOSPEC "jmp *%[thunk_target]\n" +# define __JMP_NOSPEC (reg) "jmp *%%" reg "\n"# define THUNK_TARGET (addr) [thunk_target] "rm" (addr)#endif +# define JMP_NOSPEC __JMP_NOSPEC("%V[thunk_target]") +/* The Spectre V2 mitigation variants */ enum spectre_v2_mitigation {SPECTRE_V2_NONE ,
arch/x86/kvm/emulate.c +30 -8
@@ -5039,23 +5039,45 @@ static void fetch_possible_mmx_operand(s }/* +* Stub written in asm in order to ensure GCC doesn't duplicate the +* fastop_return: label. +* +* Custom calling convention. +* +* __fastop: +* ax = ctxt->dst.val +* dx = ctxt->src.val +* cx = ctxt->src.val2 +* di = flags +* si = fop +*/ +asm (ASM_FUNC_ALIGN +"__fastop: \n\t" +"push %" _ASM_DI"\n\t" +"popf \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_SAVE, 0, 0, 0) + ASM_ANNOTATE(ANNOTYPE_JUMP_TABLE) + __JMP_NOSPEC(_ASM_SI) +"fastop_return: \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_RESTORE, 0, 0, 0) +"pushf \n\t" +"pop %" _ASM_DI"\n\t" +ASM_RET +".type __fastop, @notype \n\t" + ".size __fastop, . - __fastop \n\t"); + +/* * All the FASTOP magic above relies on there being *one* instance of this * so it can JMP back, avoiding RET and it's various thunks. */ -static noinline int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) +static int fastop (struct x86_emulate_ctxt *ctxt,fastop_t fop) {ulong flags = (ctxt->eflags &EFLAGS_MASK ) |X86_EFLAGS_IF ;if (!(ctxt->d & ByteOp)) fop +=__ffs (ctxt->dst.bytes) *FASTOP_SIZE ; - asm("push %[flags]; popf \n\t" - UNWIND_HINT(UNWIND_HINT_TYPE_SAVE, 0, 0, 0) -ASM_ANNOTATE (ANNOTYPE_JUMP_TABLE) - JMP_NOSPEC -"fastop_return: \n\t" - UNWIND_HINT(UNWIND_HINT_TYPE_RESTORE, 0, 0, 0) -"pushf; pop %[flags]\n" + asm("call __fastop" :"+a" (ctxt->dst.val),"+d" (ctxt->src.val), [flags]"+D" (flags), [thunk_target]"+S"(fop), ASM_CALL_CONSTRAINT :"c" (ctxt->src2.val));
Sean Christopherson
maintainerRe: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops
In reply to Peter ZijlstraKVM: x86:
Quoted 7 lines · Peter Zijlstra
Since there is only a single
fastop()function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz.Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here).
Maybe add an example? Mostly as a reminder of how to reproduce the call depth issues.
E.g. booting with "retbleed=force,stuff spectre_v2=retpoline,generic" causes KVM-Unit-Test's "emulator" test to fail due to flags being clobbered.
Quoted 8 lines
Objtool strenuously complains about this:
- indirect call without a .rodata, fails to determine
JUMP_TABLE, annotate
- fastop functions fall through, exception
- unreachable instruction afterfastop_return, save/restoreSigned-off-by Peter Zijlstra (Intel) <[email protected]>
The original patch works, but with the fixup KVM fails emulation of an ADC and generates:
------------[ cut here ]------------ Unpatched return thunk in use. This should not happen! WARNING: CPU: 4 PID: 1452 at arch/x86/kernel/cpu/bugs.c:3063 __warn_thunk+0x26/0x30 Modules linked in: vhost_net vhost vhost_iotlb tap kvm_intel kvm CPU: 4 UID: 1000 PID: 1452 Comm: qemu Not tainted 6.12.0-rc5-22582d7d68a6-rev/fastops-miti #11 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:__warn_thunk+0x26/0x30 Code: 5e ff 7e 05 0f 1f 44 00 00 80 3d 5d 06 5c 01 00 74 05 e9 bd 7d a0 00 48 c7 c7 10 4a 13 82 c6 05 48 06 5c 01 01 e8 31 48 04 00 <0f> 0b e9 a3 7d a0 00 cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 RSP: 0018:ffffc90001743c78 EFLAGS: 00010287 RAX: 0000000000000000 RBX: ffff88811877a040 RCX: 0000000000000027 RDX: ffff88846f91b208 RSI: 0000000000000001 RDI: ffff88846f91b200 RBP: ffffc90001743cc8 R08: ffffffff825195c8 R09: 0000000000000003 R10: ffffffff824395e0 R11: ffffffff824e95e0 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 FS: 00007f3027400700(0000) GS:ffff88846f900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3029ba3001 CR3: 000000010cdc0000 CR4: 0000000000352eb0 Call Trace: <TASK> ? __warn+0x85/0x120 ? __warn_thunk+0x26/0x30 ? report_bug+0x17d/0x190 ? handle_bug+0x53/0x90 ? exc_invalid_op+0x14/0x70 ? asm_exc_invalid_op+0x16/0x20 ? __warn_thunk+0x26/0x30 ? __warn_thunk+0x26/0x30 warn_thunk_thunk+0x16/0x30 ? __kvm_mmu_topup_memory_cache+0x57/0x150 [kvm] init_emulate_ctxt+0xae/0x110 [kvm] x86_decode_emulated_instruction+0x25/0x80 [kvm] x86_emulate_instruction+0x2cb/0x6f0 [kvm] vmx_handle_exit+0x394/0x6e0 [kvm_intel] kvm_arch_vcpu_ioctl_run+0xf40/0x1db0 [kvm] kvm_vcpu_ioctl+0x2e9/0x870 [kvm] ? futex_wake+0x81/0x180 ? call_depth_return_thunk+0x6c/0x90 ? call_depth_return_thunk+0x66/0x90 ? call_depth_return_thunk+0x60/0x90 ? call_depth_return_thunk+0x5a/0x90 __x64_sys_ioctl+0x8a/0xc0 do_syscall_64+0x5b/0x170 entry_SYSCALL_64_after_hwframe+0x71/0x79 RIP: 0033:0x7f30290cedeb Code: 0f 92 c0 84 c0 75 b0 49 8d 3c 1c e8 ff 47 04 00 85 c0 78 b1 48 83 c4 08 4c 89 e0 5b 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f30273ff748 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f30290cedeb RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000000e RBP: 0000555587e2f1e0 R08: 00007f302923fc10 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f3029b90780 R14: 00007ffea5ab9640 R15: 00007f30273ffa00 </TASK> ---[ end trace 0000000000000000 ]---
Peter Zijlstra
maintainerRe: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops
Quoted 24 lines · Sean Christopherson
KVM: x86:
Quoted 7 lines · Peter Zijlstra
Since there is only a single
fastop()function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz.Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here).
Maybe add an example? Mostly as a reminder of how to reproduce the call depth issues.
E.g. booting with "retbleed=force,stuff spectre_v2=retpoline,generic" causes KVM-Unit-Test's "emulator" test to fail due to flags being clobbered.
Quoted 7 lines
Objtool strenuously complains about this:
- indirect call without a .rodata, fails to determine
JUMP_TABLE, annotate
- fastop functions fall through, exception
- unreachable instruction afterfastop_return, save/restore
Quoted 4 lines
Quoted 1 line
Signed-off-by Peter Zijlstra (Intel) <[email protected]>The original patch works, but with the fixup KVM fails emulation of an ADC and generates:
Bah, I'll go chase it down. Thanks!
Re: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Since there is only a single
fastop()function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz.Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here).
Quoted 6 lines
Objtool strenuously complains about this:
- indirect call without a .rodata, fails to determine
JUMP_TABLE, annotate
- fastop functions fall through, exception
- unreachable instruction afterfastop_return, save/restore
This wording makes it sound like this patch triggers objtool warnings, which I'm guessing is not true?
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Peter Zijlstra
Since there is only a single
fastop()function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz.Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here).
Objtool strenuously complains about this:
Quoted 7 lines
Quoted 4 lines
- indirect call without a .rodata, fails to determine
JUMP_TABLE, annotate
- fastop functions fall through, exception
- unreachable instruction afterfastop_return, save/restoreThis wording makes it sound like this patch triggers objtool warnings, which I'm guessing is not true?
Right, no, it did without the fixups. This was a (poorly worder) attempt at explaining the reasons for the various annotations in the patch.
Sean Christopherson
maintainerQuoted 16 lines · Peter Zijlstra
Hi!
At long last, a respin of these patches.
The FASTOPs are special because they rely on RET to preserve CFLAGS, which is a problem with all the mitigation stuff. Also see things like:
ba5ca5e5e6a1("x86/retpoline: Don't clobber RFLAGS duringsrso_safe_ret()").Rework FASTOPs to no longer use RET and side-step the problem of trying to make the various return thunks preserve CFLAGS for just this one case.
There are two separate instances,
test_cc()andfastop(). The first is basically a SETCC wrapper, which seems like a very complicated (and somewhat expensive) way to read FLAGS. Instead use the code we already have to emulate JCC to fully emulate the instruction.
Quoted 8 lines
That then leaves
fastop(), which when marked noinline is guaranteed to exist only once. As such, CALL+RET isn't needed, because we'll always be RETurning to the same location, as such replace with JMP+JMP.My plan is to take the objtool patches through tip/objtool/core, the nospec patches through tip/x86/core and either stick the fastop patches in that latter tree if the KVM folks agree, or they can merge the aforementioned two branches and then stick the patches on top, whatever works for people.
Unless Paolo objects, I think it makes sense to take the fastop patches through tip/x86/core.