x86,objtool: Optimize !RETPOLINE
Subsystems (4)
Peter Zijlstra
maintainerv2 cover x86,objtool: Optimize !RETPOLINE
Hi,
Respin of the !RETPOLINE optimization patches.
Boris, the first 3 should probably go into tip/x86/core, it's an ungodly tangle since it relies on the insn decoder patches in tip/x86/core, the NOP patches in tip/x86/cpu and the alternative patches in tip/x86/alternatives.
Just to make life easy I'd suggest merging everything in x86/core and forgetting about the other topic branches (that's what I ended up doing locally).
The remaining 11 patches depend on the first 3 as well as on the work in tip/objtool/core, just to make life more interesting still ;-)
All except the last 4 patches should be fairly uncontroversial (I hope...)
Peter Zijlstra
maintainerPATCH v2 1/14 x86: Add insn_decode_kernel()
Add a helper to decode kernel instructions; there's no point in endlessly repeating those last two arguments.
--- arch/x86/include/asm/insn.h | 2 ++ arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/cpu/mce/severity.c | 2 +- arch/x86/kernel/kprobes/core.c | 4 ++-- arch/x86/kernel/kprobes/opt.c | 2 +- arch/x86/kernel/traps.c | 2 +- tools/arch/x86/include/asm/insn.h | 4 +++- 7 files changed, 11 insertions(+), 7 deletions(-)
arch/x86/include/asm/insn.h +2 -0
@@ -150,6 +150,8 @@ enum insn_mode {extern int insn_decode (struct insn *insn,const void *kaddr,int buf_len,enum insn_mode m); +#define insn_decode_kernel (_insn, _ptr) insn_decode((_insn), (_ptr), MAX_INSN_SIZE, INSN_MODE_KERN) +/* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute (struct insn *insn) {
arch/x86/kernel/alternative.c +1 -1
@@ -1160,7 +1160,7 @@ static void text_poke_loc_init(struct teif (!emulate) emulate = opcode; - ret =insn_decode (&insn, emulate,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (&insn, emulate);BUG_ON (ret <0 );BUG_ON (len != insn.length);
arch/x86/kernel/cpu/mce/severity.c +1 -1
@@ -225,7 +225,7 @@ static bool is_copy_from_user(struct pt_if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE ))return false; - ret =insn_decode (&insn, insn_buf,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (&insn, insn_buf);if (ret <0 )return false;
arch/x86/kernel/kprobes/core.c +2 -2
@@ -279,7 +279,7 @@ static int can_probe(unsigned long paddrif (!__addr)return 0 ; - ret =insn_decode (&insn, (void *)__addr,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (&insn, (void *)__addr);if (ret <0 )return 0 ;
@@ -316,7 +316,7 @@ int __copy_instruction(u8 *dest, u8 *src MAX_INSN_SIZE))return 0 ; - ret =insn_decode (insn, dest,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (insn, dest);if (ret <0 )return 0 ;
arch/x86/kernel/kprobes/opt.c +1 -1
@@ -324,7 +324,7 @@ static int can_optimize(unsigned long paif (!recovered_insn)return 0 ; - ret =insn_decode (&insn, (void *)recovered_insn,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (&insn, (void *)recovered_insn);if (ret <0 )return 0 ;
arch/x86/kernel/traps.c +1 -1
@@ -504,7 +504,7 @@ static enum kernel_gp_hint get_kernel_gp MAX_INSN_SIZE))return GP_NO_HINT ; - ret =insn_decode (&insn, insn_buf,MAX_INSN_SIZE ,INSN_MODE_KERN ); + ret =insn_decode_kernel (&insn, insn_buf);if (ret <0 )return GP_NO_HINT ;
tools/arch/x86/include/asm/insn.h +2 -0
@@ -150,6 +150,8 @@ enum insn_mode {extern int insn_decode (struct insn *insn,const void *kaddr,int buf_len,enum insn_mode m); +#define insn_decode_kernel (_insn, _ptr) insn_decode((_insn), (_ptr), MAX_INSN_SIZE, INSN_MODE_KERN) +/* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute (struct insn *insn) {
Re: [PATCH v2 01/14] x86: Add insn_decode_kernel()
Quoted 4 lines · Peter Zijlstra
Add a helper to decode kernel instructions; there's no point in endlessly repeating those last two arguments.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 9 lines
--- arch/x86/include/asm/insn.h | 2 ++ arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/cpu/mce/severity.c | 2 +- arch/x86/kernel/kprobes/core.c | 4 ++-- arch/x86/kernel/kprobes/opt.c | 2 +- arch/x86/kernel/traps.c | 2 +- tools/arch/x86/include/asm/insn.h | 4 +++- 7 files changed, 11 insertions(+), 7 deletions(-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePeter Zijlstra
maintainerPATCH v2 2/14 x86/alternatives: Optimize optimize_nops()
In reply to Peter ZijlstraCurrently optimize_nops() scans to see if the alternative starts with NOPs. However, the emit pattern is:
141: \oldinstr
142: .skip (len-(142b-141b)), 0x90
That is, when oldinstr is short, we pad the tail with NOPs. This case never gets optimized.
Rewrite optimize_nops() to replace any string of NOPs inside the alternative to larger NOPs. Also run it irrespective of patching, replacing NOPs in both the original and replaced code.
A direct consequence is that padlen becomes superfluous, so remove it.
--- arch/x86/include/asm/alternative.h | 14 ++--- arch/x86/kernel/alternative.c | 61 ++++++++++++++++---------- tools/objtool/arch/x86/include/arch/special.h | 2 3 files changed, 45 insertions(+), 32 deletions(-)
arch/x86/include/asm/alternative.h +5 -9
@@ -65,7 +65,6 @@ struct alt_instr {u16 cpuid;/* cpuid bit set for replacement */ u8 instrlen;/* length of original instruction */ u8 replacementlen;/* length of new instruction */ -u8 padlen;/* length of build-time padding */ } __packed;/*
@@ -104,7 +103,6 @@ static inline int alternatives_text_rese#define alt_end_marker "663"#define alt_slen "662b-661b" -#define alt_pad_len alt_end_marker"b-662b"#define alt_total_slen alt_end_marker"b-661b"#define alt_rlen (num) e_replacement(num)"f-"b_replacement(num)"f"
@@ -151,8 +149,7 @@ static inline int alternatives_text_rese " .long " b_replacement(num)"f - .\n" /* new instruction */ \ " .word " __stringify(feature) "\n" /* feature bit */ \" .byte " alt_total_slen"\n" /* source len */ \ - " .byte " alt_rlen(num) "\n" /* replacement len */ \ - " .byte " alt_pad_len "\n" /* pad len */ + " .byte " alt_rlen(num) "\n" /* replacement len */#define ALTINSTR_REPLACEMENT (newinstr, num)/* replacement */ \ "# ALT: replacement " #num "\n" \
@@ -315,13 +312,12 @@ static inline int alternatives_text_rese* enough information for the alternatives patching code to patch an * instruction. See apply_alternatives(). */ -.macro altinstruction_entry orig alt feature orig_len alt_len pad_len +.macro altinstruction_entry orig alt feature orig_len alt_len .long \orig - . .long \alt - . .word \feature .byte \orig_len .byte \alt_len - .byte \pad_len .endm/*
@@ -338,7 +334,7 @@ static inline int alternatives_text_rese 142: .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f,142b-141b + altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f .popsection .pushsection .altinstr_replacement,"ax"
@@ -375,8 +371,8 @@ static inline int alternatives_text_rese 142: .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f,142b-141b - altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f,142b-141b + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f .popsection .pushsection .altinstr_replacement,"ax"
arch/x86/kernel/alternative.c +39 -22
91 lines not on this page · raw message
tools/objtool/arch/x86/include/arch/special.h +1 -1
@@ -10,7 +10,7 @@#define JUMP_ORIG_OFFSET 0#define JUMP_NEW_OFFSET 4 -#define ALT_ENTRY_SIZE 13 +#define ALT_ENTRY_SIZE 12#define ALT_ORIG_OFFSET 0#define ALT_NEW_OFFSET 4#define ALT_FEATURE_OFFSET 8
Borislav Petkov
maintainerRe: [PATCH v2 02/14] x86/alternatives: Optimize optimize_nops()
Quoted 39 lines · Peter Zijlstra
arch/x86/kernel/alternative.c +23
-7@@ -345,19 +345,39 @@ recompute_jump(struct alt_instr *a, u8 * static void __init_or_module noinline optimize_nops(struct alt_instr *a, u8 *instr) {unsigned long flags; -int i; +int nops =0 , i =0 ; +struct insn insn; +u8 *nop =NULL ; -for (i =0 ; i < a->padlen; i++) { -if (instr[i] !=0x90 ) + do { + if (insn_decode_kernel(&insn, &instr[i])) return; - } -local_irq_save (flags); -add_nops (instr + (a->instrlen - a->padlen), a->padlen); -local_irq_restore (flags); + if (insn.length == 1 && insn.opcode.bytes[0] == 0x90) { + if (!nop) { + nop = &instr[i]; + nops = 1; + } else { + nops++; + } + } + i += insn.length; + + if ((insn.length != 1 || i == a->instrlen) && nop) { + + local_irq_save(flags); + add_nops(nop, nops); + local_irq_restore(flags); + + DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ", + instr, (int)(unsigned long)(nop-instr), nops);
arch/x86/kernel/alternative.c recompute_jump(struct alt_instr *a, u8 * · line 381
Quoted 10 lines
arch/x86/kernel/alternative.c +6
-2+ + nop =NULL ; + } -DUMP_BYTES (instr, a->instrlen,"%px: [%d:%d) optimized NOPs: " , - instr, a->instrlen - a->padlen, a->padlen); + }while (i < a->instrlen); + +WARN_ON_ONCE (nop); }
I think I've made this simpler; pasting the whole function and not the diff because former is easier to read:
/*
* "noinline" to cause control flow change and thus invalidate I$ and
* cause refetch after modification.
*
* Jump over the non-NOP insns, the remaining bytes must be single-byte NOPs,
* optimize them.
*/
static void __init_or_module noinline optimize_nops(struct alt_instr *a, u8 *instr)
{
unsigned long flags;
struct insn insn;
int i = 0, j;
/* Skip preceding non-NOP instructions. */
do {
if (insn_decode_kernel(&insn, &instr[i]))
return;
if (insn.length == 1 && insn.opcode.bytes[0] == 0x90)
break;
i += insn.length;
} while (i < a->instrlen);
if (i >= a->instrlen - 1)
return;
/* Verify rest is NOPs - should not fire(tm) */
for (j = i; j < a->instrlen - 1; j++) {
if (WARN(instr[j] != 0x90, "Wrong insn byte 0x%hx at 0x%px\n",
instr[j], &instr[j]))
return;
}
local_irq_save(flags);
add_nops(&instr[i], a->instrlen - i);
local_irq_restore(flags);
DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ",
instr, i, a->instrlen);
}Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePeter Zijlstra
maintainerRe: [PATCH v2 02/14] x86/alternatives: Optimize optimize_nops()
Quoted 2 lines · Borislav Petkov
I think I've made this simpler; pasting the whole function and not the diff because former is easier to read:
You've make it only replace a single stream of NOPs. Which is probably fine, but... :-)
So mine, while a little more complicated, will replace any number of NOP streams in any location. Saves having to worry about if we missed some stupid corner case somewhere.
Anyway, I think I got it slightly wrong.. How's this one?
---
static void __init_or_module noinline optimize_nops(struct alt_instr *a, u8 *instr)
{
int nops = 0, i = 0;
bool done = false;
struct insn insn;
u8 *nop = NULL;
do {
if (insn_decode_kernel(&insn, &instr[i]))
return;
if (insn.length == 1 && insn.opcode.bytes[0] == 0x90)
nops++;
else if (nops)
nop = &instr[i - nops];
if (i + insn.length >= a->instrlen) {
nop = &instr[i - nops];
done = true;
}
if (nop) {
unsigned long flags;
local_irq_save(flags);
add_nops(nop, nops);
local_irq_restore(flags);
DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ",
instr, (int)(unsigned long)(nop-instr), nops);
nops = 0;
nop = NULL;
}
i += insn.length;
} while (!done);
WARN_ON_ONCE(nop);
}Borislav Petkov
maintainerRe: [PATCH v2 02/14] x86/alternatives: Optimize optimize_nops()
Quoted 2 lines · Peter Zijlstra
You've make it only replace a single stream of NOPs. Which is probably fine, but... :-)
Yap, since I added the padding thing there should be no need to put NOPs in the middle.
Quoted 1 line
So mine, while a little more complicated, will replace any number of NOP
A little? I had to sprinkle printks to figure out what it does :)
And what it does is, it calls the instruction decoder on every byte and advances by the length of each insn it decoded. But that is unnecessary because the possible opcode bytes you should get to see in any possible location are:
[(insn).* NOP*]
i.e., 0 or more instructions which are non-NOPs followed by 0 or more NOPs. Thus my simpler solution to scan past the non-NOPs and patch the rest.
So I don't see the need for the complexity, frankly.
Btw, yours needs some adjusting to the DUMP_BYTES indices call:
[ 0.145789] SMP alternatives: SKIP feat: 8*32+16, old: (entry_SYSCALL_64_after_hwframe+0x49/0xc5 (ffffffff81a0006d) len: 23), repl: (ffffffff89764c3f, len: 23) [ 0.146699] SMP alternatives: ffffffff81a0006d: old_insn: 48 c7 c0 90 90 00 00 90 90 90 cd 30 90 90 90 90 90 90 90 90 90 90 90 [ 0.148121] SMP alternatives: ffffffff89764c3f: rpl_insn: 48 bb ef be ad de 00 00 00 00 48 31 c0 48 b9 be ba ee ff c0 00 00 00 [ 0.149405] SMP alternatives: ffffffff81a0006d: [7:3) optimized NOPs: 48 c7 c0 90 90 00 00 0f 1f 00 cd 30 90 90 90 90 90 90 90 90 90 90 90 [ 0.150700] SMP alternatives: ffffffff81a0006d: [11:11) optimized NOPs: 48 c7 c0 90 90 00 00 0f 1f 00 cd 0f 1f 84 00 00 00 00 00 0f 1f 00 90
[7:3) and [11:11) look weird.
The thing I'm testing with is:
ALTERNATIVE "mov $0x9090, %rax; .byte 0x90,0x90,0x90; int $0x30", "mov $0xdeadbeef, %rbx; xor %rax, %rax; mov $0xc0ffeebabe, %rcx", \
X86_FEATURE_XENPVwhich is arbitrary, ofc.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePeter Zijlstra
maintainerPATCH v2 3/14 x86/retpoline: Simplify retpolines
In reply to Peter ZijlstraDue to commit c9c324dc22aa ("objtool: Support stack layout changes in alternatives"), it is possible to simplify the retpolines.
Currently our retpolines consist of 2 symbols, __x86_indirect_thunk_\reg, which is the compiler target, and __x86_retpoline_\reg, which is the actual retpoline. Both are consecutive in code and aligned such that for any one register they both live in the same cacheline:
0000000000000000 <__x86_indirect_thunk_rax>:
0: ff e0 jmpq *%rax 2: 90 nop 3: 90 nop 4: 90 nop
0000000000000005 <__x86_retpoline_rax>:
5: e8 07 00 00 00 callq 11 <__x86_retpoline_rax+0xc> a: f3 90 pause c: 0f ae e8 lfence f: eb f9 jmp a <__x86_retpoline_rax+0x5> 11: 48 89 04 24 mov %rax,(%rsp) 15: c3 retq 16: 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:0x0(%rax,%rax,1)
The thunk is an alternative_2, where one option is a jmp to the retpoline. This was done so that objtool didn't need to deal with alternatives with stack ops. But that problem has been solved, so now it is possible to fold the entire retpoline into the alternative to simplify and consolidate unused bytes:
0000000000000000 <__x86_indirect_thunk_rax>:
0: ff e0 jmpq *%rax 2: 90 nop 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 66 66 2e 0f 1f 84 00 00 00 00 00 data16 nopw %cs:0x0(%rax,%rax,1) 1c: 0f 1f 40 00 nopl 0x0(%rax)
Notice that since the longest alternative sequence is now:
0: e8 07 00 00 00 callq c <.altinstr_replacement+0xc> 5: f3 90 pause 7: 0f ae e8 lfence a: eb f9 jmp 5 <.altinstr_replacement+0x5> c: 48 89 04 24 mov %rax,(%rsp) 10: c3 retq
17 bytes, we have 15 bytes NOP at the end of our 32 byte slot. (IOW, if we can shrink the retpoline by 1 byte we can pack it more dense)
--- arch/x86/include/asm/asm-prototypes.h | 7 ------- arch/x86/include/asm/nospec-branch.h | 6 +++--- arch/x86/lib/retpoline.S | 34 +++++++++++++++++----------------- tools/objtool/check.c | 3 +-- 4 files changed, 21 insertions(+), 29 deletions(-)
arch/x86/include/asm/asm-prototypes.h +0 -7
@@ -22,15 +22,8 @@ extern void cmpxchg8b_emu(void);#define DECL_INDIRECT_THUNK (reg) \ extern asmlinkage void __x86_indirect_thunk_ ## reg (void); -#define DECL_RETPOLINE (reg) \ - extern asmlinkage void __x86_retpoline_ ## reg (void); -#undef GEN#define GEN (reg) DECL_INDIRECT_THUNK(reg)#include <asm/GEN-for-each-reg.h> -#undef GEN -#define GEN (reg) DECL_RETPOLINE(reg) -#include <asm/GEN-for-each-reg.h> -#endif /* CONFIG_RETPOLINE */
arch/x86/include/asm/nospec-branch.h +3 -3
@@ -81,7 +81,7 @@ .macro JMP_NOSPEC reg:req#ifdef CONFIG_RETPOLINE ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ - __stringify(jmp __x86_retpoline_\reg), X86_FEATURE_RETPOLINE, \ + __stringify(jmp __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE, \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEATURE_RETPOLINE_AMD#else jmp *%\reg
@@ -91,7 +91,7 @@ .macro CALL_NOSPEC reg:req#ifdef CONFIG_RETPOLINE ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; call *%\reg), \ - __stringify(call __x86_retpoline_\reg), X86_FEATURE_RETPOLINE, \ + __stringify(call __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE, \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; call *%\reg), X86_FEATURE_RETPOLINE_AMD#else call *%\reg
@@ -129,7 +129,7 @@ALTERNATIVE_2 ( \ANNOTATE_RETPOLINE_SAFE \"call *%[thunk_target]\n" , \ -"call __x86_retpoline_%V[thunk_target]\n" , \ +"call __x86_indirect_thunk_%V[thunk_target]\n" , \X86_FEATURE_RETPOLINE , \"lfence;\n" \ ANNOTATE_RETPOLINE_SAFE \
arch/x86/lib/retpoline.S +17 -17
@@ -10,27 +10,31 @@ #include <asm/unwind_hints.h> #include <asm/frame.h> -.macro THUNK reg - .section .text.__x86.indirect_thunk - - .align 32 -SYM_FUNC_START(__x86_indirect_thunk_\reg) - JMP_NOSPEC \reg -SYM_FUNC_END(__x86_indirect_thunk_\reg) - -SYM_FUNC_START_NOALIGN(__x86_retpoline_\reg) +.macro RETPOLINE reg ANNOTATE_INTRA_FUNCTION_CALL - call .Ldo_rop_\@ + call .Ldo_rop_\@ .Lspec_trap_\@: UNWIND_HINT_EMPTY pause lfence - jmp .Lspec_trap_\@ + jmp .Lspec_trap_\@ .Ldo_rop_\@: - mov %\reg, (%_ASM_SP) + mov %\reg, (%_ASM_SP) UNWIND_HINT_FUNC ret -SYM_FUNC_END(__x86_retpoline_\reg) +.endm + +.macro THUNK reg + .section .text.__x86.indirect_thunk + + .align 32 +SYM_FUNC_START(__x86_indirect_thunk_\reg) + + ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ + __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \ + __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEATURE_RETPOLINE_AMD + +SYM_FUNC_END(__x86_indirect_thunk_\reg) .endm
@@ -48,7 +52,6 @@ SYM_FUNC_END(__x86_retpoline_\reg) #define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym) #define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) -#define EXPORT_RETPOLINE(reg) __EXPORT_THUNK(__x86_retpoline_ ## reg) #undef GEN #define GEN(reg) THUNK reg
@@ -58,6 +61,3 @@ SYM_FUNC_END(__x86_retpoline_\reg) #define GEN(reg) EXPORT_THUNK(reg) #include <asm/GEN-for-each-reg.h> -#undef GEN -#define GEN(reg) EXPORT_RETPOLINE(reg) -#include <asm/GEN-for-each-reg.h>
tools/objtool/check.c +1 -2
@@ -800,8 +800,7 @@ static int add_jump_destinations(struct }else if (reloc->sym->type ==STT_SECTION ) { dest_sec = reloc->sym->sec; dest_off =arch_dest_reloc_offset (reloc->addend); - }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 ) || - !strncmp (reloc->sym->name,"__x86_retpoline_" ,16 )) { + }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 )) {/* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly.
RE: [PATCH v2 03/14] x86/retpoline: Simplify retpolines
From: Peter Zijlstra
Quoted 5 lines
Sent: 18 March 2021 17:11
Due to commit
c9c324dc22aa("objtool: Support stack layout changes in alternatives"), it is possible to simplify the retpolines.
...
Quoted 11 lines
Notice that since the longest alternative sequence is now:
0: e8 07 00 00 00 callq c <.altinstr_replacement+0xc> 5: f3 90 pause 7: 0f ae e8 lfence a: eb f9 jmp 5 <.altinstr_replacement+0x5> c: 48 89 04 24 mov %rax,(%rsp) 10: c3 retq17 bytes, we have 15 bytes NOP at the end of our 32 byte slot. (IOW, if we can shrink the retpoline by 1 byte we can pack it more dense)
I'm intrigued about the lfence after the pause. Clearly this is for very warped cpu behaviour. To get to the pause you have to be speculating past an unconditional call.
To get to the lfence you have to (mostly) have ignored the pause. Which is commented:
_mm_pause(); /* Abort speculation */
in a couple of examples in 248966-033.
I wonder what effect replacing the lfence with hlt would have? It would certainly save 2 bytes and allow the entire retpoline be put into a single 16byte code fetch block.
248966-033 also contains a note that the instruction(s) after an indirect jump may get executed.
It suggests adding a pause (or illegal instruction) to stop anything odd happening (they knew it could be horrid in June 2016. But then go on to say the adding pause may be a performance issue. (Presumably because if it is speculatively executed it takes ages.)
I do remember something from even longer ago about trying to never speculate any of the trig opcodes - because at last some cpu couldn't abort the instruction.
This may also mean that a big pile of 0x90 nops after the jmp (%eax) is actually better than 2 or 3 'big' nops.
Of course, if you execute the nops you always want the big ones.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Peter Zijlstra
maintainerRe: [PATCH v2 03/14] x86/retpoline: Simplify retpolines
Quoted 16 lines · David Laight
From: Peter Zijlstra
Quoted 5 lines
Sent: 18 March 2021 17:11
Due to commit
c9c324dc22aa("objtool: Support stack layout changes in alternatives"), it is possible to simplify the retpolines....
Quoted 9 lines
Notice that since the longest alternative sequence is now:
0: e8 07 00 00 00 callq c <.altinstr_replacement+0xc> 5: f3 90 pause 7: 0f ae e8 lfence a: eb f9 jmp 5 <.altinstr_replacement+0x5> c: 48 89 04 24 mov %rax,(%rsp) 10: c3 retq
Quoted 7 lines
Quoted 2 lines
17 bytes, we have 15 bytes NOP at the end of our 32 byte slot. (IOW, if we can shrink the retpoline by 1 byte we can pack it more dense)
I'm intrigued about the lfence after the pause. Clearly this is for very warped cpu behaviour. To get to the pause you have to be speculating past an unconditional call.
Please read up on retpoline... That's the speculation trap. The warped CPU behaviour is called Spectre-v2.
For others, the obvious alternative is the below; and possibly we could then also remove the loop.
The original retpoline, as per Paul's article has: 1: pause; jmp 1b;. That is, it lacks the LFENCE we have and would also fit 16 bytes.
---
arch/x86/lib/retpoline.S +2 -3
@@ -15,8 +15,7 @@ call .Ldo_rop_\@ .Lspec_trap_\@: UNWIND_HINT_EMPTY - pause - lfence + int3 jmp .Lspec_trap_\@ .Ldo_rop_\@: mov %\reg, (%_ASM_SP)
@@ -27,7 +26,7 @@ .macro THUNK reg .section .text.__x86.indirect_thunk - .align 32 + .align 16 SYM_FUNC_START(__x86_indirect_thunk_\reg) ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \
RE: [PATCH v2 03/14] x86/retpoline: Simplify retpolines
From: Peter Zijlstra
Quoted 22 lines
Sent: 22 March 2021 09:33
Quoted 19 lines · David Laight
From: Peter Zijlstra
Quoted 5 lines
Sent: 18 March 2021 17:11
Due to commit
c9c324dc22aa("objtool: Support stack layout changes in alternatives"), it is possible to simplify the retpolines....
Quoted 11 lines
Notice that since the longest alternative sequence is now:
0: e8 07 00 00 00 callq c <.altinstr_replacement+0xc> 5: f3 90 pause 7: 0f ae e8 lfence a: eb f9 jmp 5 <.altinstr_replacement+0x5> c: 48 89 04 24 mov %rax,(%rsp) 10: c3 retq17 bytes, we have 15 bytes NOP at the end of our 32 byte slot. (IOW, if we can shrink the retpoline by 1 byte we can pack it more dense)
Quoted 7 lines
Quoted 4 lines
I'm intrigued about the lfence after the pause. Clearly this is for very warped cpu behaviour. To get to the pause you have to be speculating past an unconditional call.
Please read up on retpoline... That's the speculation trap. The warped CPU behaviour is called Spectre-v2.
There is 'warped' and 'very warped' :-)
Most of Spectre-v2 (don't search for Spectra v2 by mistake) is avoiding the indirect branch prediction - which I knew.
I think the 'pause' is only executed is the cpu speculates through the mov and retq; rather the speculating past the 'call' - which some ARM cpu seem to do.
Quoted 2 lines
For others, the obvious alternative is the below; and possibly we could then also remove the loop.
Another alternative is 'hlt' with the loop (rather than int3).
Quoted 2 lines
The original retpoline, as per Paul's article has: 1: pause; jmp 1b;. That is, it lacks the LFENCE we have and would also fit 16 bytes.
Yes. Just 'jmps .' ought to be enough to block any side effects of speculative execution.
Adding 'pause' is 'a good idea' for any spin loop.
There might be another lurking performance issue.
Skylake increased the execution time of pause from ~10 to ~140 clocks. Reading between the lines I suspect this applies to speculatively executed pause.
If that happens on a regular basis it might be noticeable. So it may even be best to remove the pause!
As you say, the original retpoline lacked the lfence. The only 'load' instruction in that sequence is the 'retq'. It has to be said that given all (normal) loads are executed in program order and all (normal) stores are also executed in program order I've never actually seen what either lfence or sfence actually do for you.
(mfence synchronises reads and writes - so may be useful.) The (pre spectre) copies of the intel pdf's I have don't say anything about lfence being any kind of a barrier against speculative memory reads.
If the retpoline doesn't fit in 16 bytes it is almost certainly (probably) worth putting the called label at offset 16. This would mean that there is only one 16-byte code block read from the call target.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Peter Zijlstra
maintainerPATCH v2 4/14 objtool: Correctly handle retpoline thunk calls
In reply to Peter ZijlstraJust like JMP handling, convert a direct CALL to a retpoline thunk into a retpoline safe indirect CALL.
--- tools/objtool/check.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
tools/objtool/check.c +12 -0
@@ -953,6 +953,18 @@ static int add_call_destinations(struct dest_off);return -1 ; } + + }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 )) { +/* +* Retpoline calls are really dynamic calls in +* disguise, so convert them accordingly. +*/ + insn->type =INSN_CALL_DYNAMIC ; + insn->retpoline_safe = true; + +remove_insn_ops (insn); +continue ; + }else insn->call_dest = reloc->sym;
Peter Zijlstra
maintainerPATCH v2 5/14 objtool: Per arch retpoline naming
In reply to Peter ZijlstraThe __x86_indirect_ naming is obviously not generic. Shorten to allow matching some additional magic names later.
--- tools/objtool/arch/x86/decode.c | 5 +++++ tools/objtool/check.c | 7 ++++++- tools/objtool/include/objtool/arch.h | 2 ++ 3 files changed, 13 insertions(+), 1 deletion(-)
tools/objtool/arch/x86/decode.c +5 -0
@@ -692,3 +692,8 @@ int arch_decode_hint_reg(struct instructreturn 0 ; } + +bool arch_is_retpoline (struct symbol *sym) +{ +return !strncmp (sym->name,"__x86_indirect_" ,15 ); +}
tools/objtool/check.c +6 -1
@@ -850,6 +850,11 @@ static int add_ignore_alternatives(strucreturn 0 ; } +__weak bool arch_is_retpoline (struct symbol *sym) +{ +return false; +} +/* * Find the destination instructions for all jumps. */
@@ -872,7 +877,7 @@ static int add_jump_destinations(struct }else if (reloc->sym->type ==STT_SECTION ) { dest_sec = reloc->sym->sec; dest_off =arch_dest_reloc_offset (reloc->addend); - }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 )) { + }else if (arch_is_retpoline (reloc->sym)) {/* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly.
tools/objtool/include/objtool/arch.h +2 -0
@@ -85,4 +85,6 @@ const char *arch_nop_insn(int len);int arch_decode_hint_reg (struct instruction *insn,u8 sp_reg); +bool arch_is_retpoline (struct symbol *sym); +#endif /* _ARCH_H */
Re: [PATCH v2 05/14] objtool: Per arch retpoline naming
tools/objtool/check.c static int add_jump_destinations(struct · line 883
Quoted 9 lines · Peter Zijlstra
tools/objtool/check.c +1
-1@@ -872,7 +877,7 @@ static int add_jump_destinations(struct }else if (reloc->sym->type ==STT_SECTION ) { dest_sec = reloc->sym->sec; dest_off =arch_dest_reloc_offset (reloc->addend); - }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 )) { + }else if (arch_is_retpoline (reloc->sym)) {/* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly.
There's another one in add_call_destinations().
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 05/14] objtool: Per arch retpoline naming
Quoted 12 lines · Josh Poimboeuf
tools/objtool/check.c static int add_jump_destinations(struct · line 883
Quoted 9 lines · Peter Zijlstra
tools/objtool/check.c +1
-1@@ -872,7 +877,7 @@ static int add_jump_destinations(struct }else if (reloc->sym->type ==STT_SECTION ) { dest_sec = reloc->sym->sec; dest_off =arch_dest_reloc_offset (reloc->addend); - }else if (!strncmp (reloc->sym->name,"__x86_indirect_thunk_" ,21 )) { + }else if (arch_is_retpoline (reloc->sym)) {/* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly. There's another one in
add_call_destinations().
Bah, I'm sure I converted all of them at some point, must've lost a hunk somewhere along the way. Fixed!
Peter Zijlstra
maintainerPATCH v2 6/14 objtool: Fix static_call list generation
In reply to Peter ZijlstraCurrently objtool generates tail call entries in add_jump_destination() but waits until validate_branch() to generate the regular call entries, move these to add_call_destination() for consistency.
--- tools/objtool/check.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
tools/objtool/check.c +13 -5
@@ -1045,6 +1045,12 @@ static int add_call_destinations(struct }else insn->call_dest = reloc->sym; +if (insn->call_dest && insn->call_dest->static_call_tramp) { +list_add_tail (&insn->static_call_node, + &file->static_call_list); + } + +/* * Many compilers cannot disable KCOV with a function attribute * so they need a little help, NOP out any KCOV calls from noinstr
@@ -1788,6 +1794,9 @@ static int decode_sections(struct objtooif (ret)return ret; +/* +* Must be before add_{jump_call}_destination. +*/ ret =read_static_call_tramps (file);if (ret)return ret;
@@ -1800,6 +1809,10 @@ static int decode_sections(struct objtooif (ret)return ret; +/* +* Must be before add_call_destination(); it changes INSN_CALL to +* INSN_JUMP. +*/ ret =read_intra_function_calls (file);if (ret)return ret;
@@ -2745,11 +2758,6 @@ static int validate_branch(struct objtooif (dead_end_function (file, insn->call_dest))return 0 ; -if (insn->type ==INSN_CALL && insn->call_dest->static_call_tramp) { -list_add_tail (&insn->static_call_node, - &file->static_call_list); - } -break ;case INSN_JUMP_CONDITIONAL :
Re: [PATCH v2 06/14] objtool: Fix static_call list generation
tools/objtool/check.c static int add_call_destinations(struct · line 1053
Quoted 12 lines
tools/objtool/check.c +6
-0@@ -1045,6 +1045,12 @@ static int add_call_destinations(struct }else insn->call_dest = reloc->sym; +if (insn->call_dest && insn->call_dest->static_call_tramp) { +list_add_tail (&insn->static_call_node, + &file->static_call_list); + } + +
A nit, but too many new lines here.
Miroslav
Peter Zijlstra
maintainerPATCH v2 7/14 objtool: Rework rebuild_reloc logic
In reply to Peter ZijlstraInstead of manually calling elf_rebuild_reloc_section() on sections we've called elf_add_reloc() on, have elf_write() DTRT.
This makes it easier to add random relocations in places without carefully tracking when we're done and need to flush what section.
--- tools/objtool/check.c | 6 ------ tools/objtool/elf.c | 20 ++++++++++++++------ tools/objtool/include/objtool/elf.h | 1 - tools/objtool/orc_gen.c | 3 --- 4 files changed, 14 insertions(+), 16 deletions(-)
tools/objtool/check.c +0 -6
@@ -542,9 +542,6 @@ static int create_static_call_sections(s idx++; } -if (elf_rebuild_reloc_section (file->elf, reloc_sec)) -return -1 ; -return 0 ; }
@@ -614,9 +611,6 @@ static int create_mcount_loc_sections(st idx++; } -if (elf_rebuild_reloc_section (file->elf, reloc_sec)) -return -1 ; -return 0 ; }
tools/objtool/elf.c +14 -6
@@ -479,6 +479,8 @@ void elf_add_reloc(struct elf *elf, strulist_add_tail (&reloc->list, &sec->reloc_list);elf_hash_add (elf->reloc_hash, &reloc->hash,reloc_hash (reloc)); + + sec->changed = true; }static int read_rel_reloc (struct section *sec,int i,struct reloc *reloc,unsigned int *symndx)
@@ -558,7 +560,9 @@ static int read_relocs(struct elf *elf)return -1 ; } -elf_add_reloc (elf, reloc); +list_add_tail (&reloc->list, &sec->reloc_list); +elf_hash_add (elf->reloc_hash, &reloc->hash,reloc_hash (reloc)); + nr_reloc++; } max_reloc =max (max_reloc, nr_reloc);
@@ -873,14 +877,11 @@ static int elf_rebuild_rela_reloc_sectioreturn 0 ; } -int elf_rebuild_reloc_section (struct elf *elf,struct section *sec) +static int elf_rebuild_reloc_section (struct elf *elf,struct section *sec) {struct reloc *reloc;int nr; - sec->changed = true; - elf->changed = true; - nr =0 ;list_for_each_entry (reloc, &sec->reloc_list, list) nr++;
@@ -944,9 +945,15 @@ int elf_write(struct elf *elf)struct section *sec;Elf_Scn *s; -/* Update section headers for changed sections: */ +/* Update changed relocation sections and section headers: */ list_for_each_entry (sec, &elf->sections, list) {if (sec->changed) { +if (sec->reloc && +elf_rebuild_reloc_section (elf, sec->reloc)) { +WARN_ELF ("elf_rebuild_reloc_section" ); +return -1 ; + } + s =elf_getscn (elf->elf, sec->idx);if (!s) {WARN_ELF ("elf_getscn" );
@@ -958,6 +965,7 @@ int elf_write(struct elf *elf) } sec->changed = false; + elf->changed = true; } }
tools/objtool/include/objtool/elf.h +0 -1
@@ -142,7 +142,6 @@ struct reloc *find_reloc_by_dest_range(cstruct symbol *find_func_containing (struct section *sec,unsigned long offset);void insn_to_reloc_sym_addend (struct section *sec,unsigned long offset,struct reloc *reloc); -int elf_rebuild_reloc_section (struct elf *elf,struct section *sec);#define for_each_sec (file, sec) \ list_for_each_entry(sec, &file->elf->sections, list)
tools/objtool/orc_gen.c +0 -3
@@ -254,8 +254,5 @@ int orc_create(struct objtool_file *filereturn -1 ; } -if (elf_rebuild_reloc_section (file->elf, ip_rsec)) -return -1 ; -return 0 ; }
Peter Zijlstra
maintainerPATCH v2 8/14 objtool: Add elf_create_reloc() helper
In reply to Peter ZijlstraWe have 4 instances of adding a relocation. Create a common helper to avoid growing even more.
--- tools/objtool/check.c | 76 ++++++---------------------- tools/objtool/elf.c | 90 +++++++++++++++++++++++----------- tools/objtool/include/objtool/check.h | 7 ++ tools/objtool/include/objtool/elf.h | 10 +++ tools/objtool/orc_gen.c | 57 ++++++++------------- 5 files changed, 119 insertions(+), 121 deletions(-)
tools/objtool/check.c +17 -59
@@ -433,8 +433,7 @@ static int add_dead_ends(struct objtool_static int create_static_call_sections (struct objtool_file *file) { -struct section *sec, *reloc_sec; -struct reloc *reloc; +struct section *sec;struct static_call_site *site;struct instruction *insn;struct symbol *key_sym;
@@ -460,8 +459,7 @@ static int create_static_call_sections(sif (!sec)return -1 ; - reloc_sec =elf_create_reloc_section (file->elf, sec,SHT_RELA ); -if (!reloc_sec) +if (!elf_create_reloc_section (file->elf, sec,SHT_RELA ))return -1 ; idx =0 ;
@@ -471,26 +469,13 @@ static int create_static_call_sections(smemset (site,0 ,sizeof (struct static_call_site ));/* populate reloc for 'addr' */ - reloc =malloc (sizeof (*reloc)); - -if (!reloc) { -perror ("malloc" ); -return -1 ; - } -memset (reloc,0 ,sizeof (*reloc)); - -insn_to_reloc_sym_addend (insn->sec, insn->offset, reloc); -if (!reloc->sym) { -WARN_FUNC ("static call tramp: missing containing symbol" , - insn->sec, insn->offset); +if (!elf_create_reloc (file->elf, sec, idx *sizeof (struct static_call_site ), +R_X86_64_PC32 , + reloc_to_insn, insn,0 )) { +WARN_ELF ("elf_create_reloc: static_call_site::addr" );return -1 ; } - reloc->type =R_X86_64_PC32 ; - reloc->offset = idx *sizeof (struct static_call_site ); - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc); -/* find key symbol */ key_name =strdup (insn->call_dest->name);if (!key_name) {
@@ -526,18 +511,13 @@ static int create_static_call_sections(sfree (key_name);/* populate reloc for 'key' */ - reloc =malloc (sizeof (*reloc)); -if (!reloc) { -perror ("malloc" ); +if (!elf_create_reloc (file->elf, sec, idx *sizeof (struct static_call_site ) +4 , +R_X86_64_PC32 , + reloc_to_sym, key_sym, +is_sibling_call (insn) *STATIC_CALL_SITE_TAIL )) { +WARN_ELF ("elf_create_reloc: static_call_site::key" );return -1 ; } -memset (reloc,0 ,sizeof (*reloc)); - reloc->sym = key_sym; - reloc->addend =is_sibling_call (insn) ?STATIC_CALL_SITE_TAIL :0 ; - reloc->type =R_X86_64_PC32 ; - reloc->offset = idx *sizeof (struct static_call_site ) +4 ; - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc); idx++; }
@@ -547,8 +527,7 @@ static int create_static_call_sections(sstatic int create_mcount_loc_sections (struct objtool_file *file) { -struct section *sec, *reloc_sec; -struct reloc *reloc; +struct section *sec;unsigned long *loc;struct instruction *insn;int idx;
@@ -571,8 +550,7 @@ static int create_mcount_loc_sections(stif (!sec)return -1 ; - reloc_sec =elf_create_reloc_section (file->elf, sec,SHT_RELA ); -if (!reloc_sec) +if (!elf_create_reloc_section (file->elf, sec,SHT_RELA ))return -1 ; idx =0 ;
@@ -581,32 +559,12 @@ static int create_mcount_loc_sections(st loc = (unsigned long *)sec->data->d_buf + idx;memset (loc,0 ,sizeof (unsigned long )); - reloc =malloc (sizeof (*reloc)); -if (!reloc) { -perror ("malloc" ); +if (!elf_create_reloc (file->elf, sec, idx *sizeof (unsigned long ), +R_X86_64_64 , + reloc_to_insn, insn,0 )) { +WARN_ELF ("elf_create_reloc: __mcount_loc" );return -1 ; } -memset (reloc,0 ,sizeof (*reloc)); - -if (insn->sec->sym) { - reloc->sym = insn->sec->sym; - reloc->addend = insn->offset; - }else { - reloc->sym =find_symbol_containing (insn->sec, insn->offset); - -if (!reloc->sym) { -WARN ("missing symbol for insn at offset 0x%lx\n" , - insn->offset); -return -1 ; - } - - reloc->addend = insn->offset - reloc->sym->offset; - } - - reloc->type =R_X86_64_64 ; - reloc->offset = idx *sizeof (unsigned long ); - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc); idx++; }
tools/objtool/elf.c +63 -27
108 lines not on this page · raw message
tools/objtool/include/objtool/check.h +7 -0
@@ -9,6 +9,7 @@#include <stdbool.h> #include <objtool/cfi.h> #include <objtool/arch.h> +#include <objtool/elf.h> struct insn_state {struct cfi_state cfi;
@@ -91,4 +92,10 @@ struct instruction *find_insn(struct obj insn->sec == sec; \ insn = list_next_entry(insn, list)) +static inline void reloc_to_insn (struct reloc *reloc,void *dst) +{ +struct instruction *insn = dst; +__reloc_to_insn (reloc, insn->sec, insn->offset); +} +#endif /* _CHECK_H */
tools/objtool/include/objtool/elf.h +9 -1
@@ -123,7 +123,15 @@ static inline u32 reloc_hash(struct relostruct elf *elf_open_read (const char *name,int flags);struct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr);struct section *elf_create_reloc_section (struct elf *elf,struct section *base,int reltype); -void elf_add_reloc (struct elf *elf,struct reloc *reloc); + +void reloc_to_sym (struct reloc *reloc,void *sym); +void __reloc_to_insn (struct reloc *reloc,struct section *sec,unsigned long offset); + +struct reloc *elf_create_reloc (struct elf *elf, +struct section *sec,unsigned long offset, +unsigned int type, +void (*set_dest)(struct reloc *reloc,void *),void *dst,int addend); +int elf_write_insn (struct elf *elf,struct section *sec,unsigned long offset,unsigned int len,const char *insn);
tools/objtool/orc_gen.c +23 -34
94 lines not on this page · raw message
Re: [PATCH v2 08/14] objtool: Add elf_create_reloc() helper
Quoted 4 lines · Peter Zijlstra
We have 4 instances of adding a relocation. Create a common helper to avoid growing even more.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
I'm not a fan of the API -- how about squashing this in? Untested, of course.
tools/objtool/check.c +12 -16
@@ -469,12 +469,11 @@ static int create_static_call_sections(struct objtool_file *file)memset (site,0 ,sizeof (struct static_call_site ));/* populate reloc for 'addr' */ -if (!elf_create_reloc (file->elf, sec, idx *sizeof (struct static_call_site ), -R_X86_64_PC32 , - reloc_to_insn, insn,0 )) { -WARN_ELF ("elf_create_reloc: static_call_site::addr" ); +if (elf_add_reloc_to_insn (file->elf, sec, + idx *sizeof (struct static_call_site ), +R_X86_64_PC32 , + insn->sec, insn->offset))return -1 ; - }/* find key symbol */ key_name =strdup (insn->call_dest->name);
@@ -511,13 +510,11 @@ static int create_static_call_sections(struct objtool_file *file)free (key_name);/* populate reloc for 'key' */ -if (!elf_create_reloc (file->elf, sec, idx *sizeof (struct static_call_site ) +4 , -R_X86_64_PC32 , - reloc_to_sym, key_sym, -is_sibling_call (insn) *STATIC_CALL_SITE_TAIL )) { -WARN_ELF ("elf_create_reloc: static_call_site::key" ); +if (elf_add_reloc (file->elf, sec, + idx *sizeof (struct static_call_site ) +4 , +R_X86_64_PC32 , key_sym, +is_sibling_call (insn) *STATIC_CALL_SITE_TAIL ))return -1 ; - } idx++; }
@@ -559,12 +556,11 @@ static int create_mcount_loc_sections(struct objtool_file *file) loc = (unsigned long *)sec->data->d_buf + idx;memset (loc,0 ,sizeof (unsigned long )); -if (!elf_create_reloc (file->elf, sec, idx *sizeof (unsigned long ), -R_X86_64_64 , - reloc_to_insn, insn,0 )) { -WARN_ELF ("elf_create_reloc: __mcount_loc" ); +if (elf_add_reloc_to_insn (file->elf, sec, + idx *sizeof (unsigned long ), +R_X86_64_64 , + insn->sec, insn->offset))return -1 ; - } idx++; }
tools/objtool/elf.c +40 -43
116 lines not on this page · raw message
tools/objtool/include/objtool/check.h +0 -7
@@ -9,7 +9,6 @@#include <stdbool.h> #include <objtool/cfi.h> #include <objtool/arch.h> -#include <objtool/elf.h> struct insn_state {struct cfi_state cfi;
@@ -92,10 +91,4 @@ struct instruction *find_insn(struct objtool_file *file, insn->sec == sec; \ insn = list_next_entry(insn, list)) -static inline void reloc_to_insn (struct reloc *reloc,void *dst) -{ -struct instruction *insn = dst; -__reloc_to_insn (reloc, insn->sec, insn->offset); -} -#endif /* _CHECK_H */
tools/objtool/include/objtool/elf.h +5 -9
@@ -124,13 +124,11 @@ struct elf *elf_open_read(const char *name, int flags);struct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr);struct section *elf_create_reloc_section (struct elf *elf,struct section *base,int reltype); -void reloc_to_sym (struct reloc *reloc,void *sym); -void __reloc_to_insn (struct reloc *reloc,struct section *sec,unsigned long offset); - -struct reloc *elf_create_reloc (struct elf *elf, -struct section *sec,unsigned long offset, -unsigned int type, -void (*set_dest)(struct reloc *reloc,void *),void *dst,int addend); +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend); +int elf_add_reloc_to_insn (struct elf *elf,struct section *sec, +unsigned long offset,unsigned int type, +struct section *insn_sec,unsigned long insn_off); int elf_write_insn(struct elf *elf, struct section *sec, unsigned long offset, unsigned int len,
@@ -148,8 +146,6 @@ struct reloc *find_reloc_by_dest(const struct elf *elf, struct section *sec, unsstruct reloc *find_reloc_by_dest_range (const struct elf *elf,struct section *sec,unsigned long offset,unsigned int len);struct symbol *find_func_containing (struct section *sec,unsigned long offset); -void insn_to_reloc_sym_addend (struct section *sec,unsigned long offset, -struct reloc *reloc);#define for_each_sec (file, sec) \ list_for_each_entry(sec, &file->elf->sections, list)
tools/objtool/orc_gen.c +14 -21
@@ -81,24 +81,11 @@ static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi)return 0 ; } -struct orc_list_entry { -struct list_head list; -struct orc_entry orc; -struct section *insn_sec; -unsigned long insn_off; -}; - -static void reloc_to_orc (struct reloc *reloc,void *dst) -{ -struct orc_list_entry *entry = dst; -__reloc_to_insn (reloc, entry->insn_sec, entry->insn_off); -} -static int write_orc_entry (struct elf *elf,struct section *orc_sec,struct section *ip_sec,unsigned int idx, -struct orc_list_entry *entry) +struct section *insn_sec,unsigned long insn_off, +struct orc_entry *o) { -struct orc_entry *o = &entry->orc;struct orc_entry *orc;/* populate ORC data */
@@ -108,16 +95,20 @@ static int write_orc_entry(struct elf *elf, struct section *orc_sec, orc->bp_offset =bswap_if_needed (orc->bp_offset);/* populate reloc for ip */ -if (!elf_create_reloc (elf, ip_sec, idx *sizeof (int ), -R_X86_64_PC32 , - reloc_to_orc, entry,0 )) { -WARN_ELF ("elf_create_reloc: ORC ip" ); +if (elf_add_reloc_to_insn (elf, ip_sec, idx *sizeof (int ),R_X86_64_PC32 , + insn_sec, insn_off))return -1 ; - }return 0 ; } +struct orc_list_entry { +struct list_head list; +struct orc_entry orc; +struct section *insn_sec; +unsigned long insn_off; +}; +static int orc_list_add (struct list_head *orc_list,struct orc_entry *orc,struct section *sec,unsigned long offset) {
@@ -239,7 +230,9 @@ int orc_create(struct objtool_file *file)/* Write ORC entries to sections: */ list_for_each_entry (entry, &orc_list, list) { -if (write_orc_entry (file->elf, orc_sec, sec, idx++, entry)) +if (write_orc_entry (file->elf, orc_sec, sec, idx++, + entry->insn_sec, entry->insn_off, + &entry->orc))return -1 ; }
Peter Zijlstra
maintainerRe: [PATCH v2 08/14] objtool: Add elf_create_reloc() helper
Quoted 8 lines · Josh Poimboeuf
Quoted 4 lines · Peter Zijlstra
We have 4 instances of adding a relocation. Create a common helper to avoid growing even more.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>I'm not a fan of the API -- how about squashing this in? Untested, of course.
Can do.. I do seem to have over-cooked it a bit.
Quoted 23 lines
tools/objtool/elf.c +12
-2@@ -447,76 +447,73 @@ static int read_symbols(struct elf *elf)return -1 ; } -static void elf_add_reloc (struct elf *elf,struct reloc *reloc) +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend) { -struct section *sec = reloc->sec; +struct reloc *reloc; + + reloc =malloc (sizeof (*reloc)); +if (!reloc) { +perror ("malloc" ); +return -1 ; + } +memset (reloc,0 ,sizeof (*reloc)); + + reloc->sec = sec->reloc;
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 458
Quoted 10 lines
tools/objtool/elf.c +4
-2+ reloc->offset = offset; + reloc->type = type; + reloc->sym = sym; + reloc->addend = addend;list_add_tail (&reloc->list, &sec->reloc_list);elf_hash_add (elf->reloc_hash, &reloc->hash,reloc_hash (reloc)); - sec->changed = true; -}
I'm thinking we have to mark something changed though; I've added sec->reloc->changed = true; there.
Quoted 30 lines
tools/objtool/elf.c +14
-0+static int insn_to_sym_addend (struct section *sec,unsigned long offset, +struct symbol **sym,int *addend) {if (sec->sym) { + *sym = sec->sym; + *addend = offset; +return 0 ; }/* * The Clang assembler strips section symbols, so we have to reference * the function symbol instead: */ + *sym =find_symbol_containing (sec, offset); +if (!*sym) {/* * Hack alert. This happens when we need to reference the NOP * pad insn immediately after the function. */ + *sym =find_symbol_containing (sec, offset -1 ); } +if (!*sym) { +WARN ("can't find symbol containing %s+0x%lx" , sec->name, offset); +return -1 ; } + *addend = offset - (*sym)->offset; +return 0 ; +}
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 510
Quoted 10 lines
tools/objtool/elf.c +8
-0+int elf_add_reloc_to_insn (struct elf *elf,struct section *sec, +unsigned long offset,unsigned int type, +struct section *insn_sec,unsigned long insn_off) +{ +struct symbol *sym; +int addend; +if (insn_to_sym_addend (insn_sec, insn_off, &sym, &addend)) +return -1 ;
Might be simpler to not have that function and just do it here instead.
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 512
Quoted 3 lines
tools/objtool/elf.c +1
-0+return elf_add_reloc (elf, sec, offset, type, sym, addend); }
Full patch, because diff on a diff on a diff is getting ludicrous hard to read :-)
--- check.c | 78 +++++++++------------------------------------- elf.c | 84 ++++++++++++++++++++++++++++++++------------------ include/objtool/elf.h | 10 ++++- orc_gen.c | 30 +++-------------- 4 files changed, 84 insertions(+), 118 deletions(-)
tools/objtool/check.c +16 -62
@@ -433,8 +433,7 @@ static int add_dead_ends(struct objtool_static int create_static_call_sections (struct objtool_file *file) { -struct section *sec, *reloc_sec; -struct reloc *reloc; +struct section *sec;struct static_call_site *site;struct instruction *insn;struct symbol *key_sym;
@@ -460,8 +459,7 @@ static int create_static_call_sections(sif (!sec)return -1 ; - reloc_sec =elf_create_reloc_section (file->elf, sec,SHT_RELA ); -if (!reloc_sec) +if (!elf_create_reloc_section (file->elf, sec,SHT_RELA ))return -1 ; idx =0 ;
@@ -471,25 +469,11 @@ static int create_static_call_sections(smemset (site,0 ,sizeof (struct static_call_site ));/* populate reloc for 'addr' */ - reloc =malloc (sizeof (*reloc)); - -if (!reloc) { -perror ("malloc" ); +if (elf_add_reloc_to_insn (file->elf, sec, + idx *sizeof (struct static_call_site ), +R_X86_64_PC32 , + insn->sec, insn->offset))return -1 ; - } -memset (reloc,0 ,sizeof (*reloc)); - -insn_to_reloc_sym_addend (insn->sec, insn->offset, reloc); -if (!reloc->sym) { -WARN_FUNC ("static call tramp: missing containing symbol" , - insn->sec, insn->offset); -return -1 ; - } - - reloc->type =R_X86_64_PC32 ; - reloc->offset = idx *sizeof (struct static_call_site ); - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc);/* find key symbol */ key_name =strdup (insn->call_dest->name);
@@ -526,18 +510,11 @@ static int create_static_call_sections(sfree (key_name);/* populate reloc for 'key' */ - reloc =malloc (sizeof (*reloc)); -if (!reloc) { -perror ("malloc" ); +if (elf_add_reloc (file->elf, sec, + idx *sizeof (struct static_call_site ) +4 , +R_X86_64_PC32 , key_sym, +is_sibling_call (insn) *STATIC_CALL_SITE_TAIL ))return -1 ; - } -memset (reloc,0 ,sizeof (*reloc)); - reloc->sym = key_sym; - reloc->addend =is_sibling_call (insn) ?STATIC_CALL_SITE_TAIL :0 ; - reloc->type =R_X86_64_PC32 ; - reloc->offset = idx *sizeof (struct static_call_site ) +4 ; - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc); idx++; }
@@ -547,8 +524,7 @@ static int create_static_call_sections(sstatic int create_mcount_loc_sections (struct objtool_file *file) { -struct section *sec, *reloc_sec; -struct reloc *reloc; +struct section *sec;unsigned long *loc;struct instruction *insn;int idx;
@@ -571,8 +547,7 @@ static int create_mcount_loc_sections(stif (!sec)return -1 ; - reloc_sec =elf_create_reloc_section (file->elf, sec,SHT_RELA ); -if (!reloc_sec) +if (!elf_create_reloc_section (file->elf, sec,SHT_RELA ))return -1 ; idx =0 ;
@@ -581,32 +556,11 @@ static int create_mcount_loc_sections(st loc = (unsigned long *)sec->data->d_buf + idx;memset (loc,0 ,sizeof (unsigned long )); - reloc =malloc (sizeof (*reloc)); -if (!reloc) { -perror ("malloc" ); +if (elf_add_reloc_to_insn (file->elf, sec, + idx *sizeof (unsigned long ), +R_X86_64_64 , + insn->sec, insn->offset))return -1 ; - } -memset (reloc,0 ,sizeof (*reloc)); - -if (insn->sec->sym) { - reloc->sym = insn->sec->sym; - reloc->addend = insn->offset; - }else { - reloc->sym =find_symbol_containing (insn->sec, insn->offset); - -if (!reloc->sym) { -WARN ("missing symbol for insn at offset 0x%lx\n" , - insn->offset); -return -1 ; - } - - reloc->addend = insn->offset - reloc->sym->offset; - } - - reloc->type =R_X86_64_64 ; - reloc->offset = idx *sizeof (unsigned long ); - reloc->sec = reloc_sec; -elf_add_reloc (file->elf, reloc); idx++; }
tools/objtool/elf.c +55 -29
101 lines not on this page · raw message
tools/objtool/include/objtool/elf.h +7 -3
@@ -123,7 +123,13 @@ static inline u32 reloc_hash(struct relostruct elf *elf_open_read (const char *name,int flags);struct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr);struct section *elf_create_reloc_section (struct elf *elf,struct section *base,int reltype); -void elf_add_reloc (struct elf *elf,struct reloc *reloc); + +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend); +int elf_add_reloc_to_insn (struct elf *elf,struct section *sec, +unsigned long offset,unsigned int type, +struct section *insn_sec,unsigned long insn_off); +int elf_write_insn (struct elf *elf,struct section *sec,unsigned long offset,unsigned int len,const char *insn);
@@ -140,8 +146,6 @@ struct reloc *find_reloc_by_dest(const sstruct reloc *find_reloc_by_dest_range (const struct elf *elf,struct section *sec,unsigned long offset,unsigned int len);struct symbol *find_func_containing (struct section *sec,unsigned long offset); -void insn_to_reloc_sym_addend (struct section *sec,unsigned long offset, -struct reloc *reloc);#define for_each_sec (file, sec) \ list_for_each_entry(sec, &file->elf->sections, list)
tools/objtool/orc_gen.c +6 -24
@@ -82,12 +82,11 @@ static int init_orc_entry(struct orc_ent }static int write_orc_entry (struct elf *elf,struct section *orc_sec, -struct section *ip_rsec,unsigned int idx, +struct section *ip_sec,unsigned int idx,struct section *insn_sec,unsigned long insn_off,struct orc_entry *o) {struct orc_entry *orc; -struct reloc *reloc;/* populate ORC data */ orc = (struct orc_entry *)orc_sec->data->d_buf + idx;
@@ -96,25 +95,9 @@ static int write_orc_entry(struct elf *e orc->bp_offset =bswap_if_needed (orc->bp_offset);/* populate reloc for ip */ - reloc =malloc (sizeof (*reloc)); -if (!reloc) { -perror ("malloc" ); +if (elf_add_reloc_to_insn (elf, ip_sec, idx *sizeof (int ),R_X86_64_PC32 , + insn_sec, insn_off))return -1 ; - } -memset (reloc,0 ,sizeof (*reloc)); - -insn_to_reloc_sym_addend (insn_sec, insn_off, reloc); -if (!reloc->sym) { -WARN ("missing symbol for insn at offset 0x%lx" , - insn_off); -return -1 ; - } - - reloc->type =R_X86_64_PC32 ; - reloc->offset = idx *sizeof (int ); - reloc->sec = ip_rsec; - -elf_add_reloc (elf, reloc);return 0 ; }
@@ -153,7 +136,7 @@ static unsigned long alt_group_len(strucint orc_create (struct objtool_file *file) { -struct section *sec, *ip_rsec, *orc_sec; +struct section *sec, *orc_sec;unsigned int nr =0 , idx =0 ;struct orc_list_entry *entry;struct list_head orc_list;
@@ -242,13 +225,12 @@ int orc_create(struct objtool_file *file sec =elf_create_section (file->elf,".orc_unwind_ip" ,0 ,sizeof (int ), nr);if (!sec)return -1 ; - ip_rsec =elf_create_reloc_section (file->elf, sec,SHT_RELA ); -if (!ip_rsec) +if (!elf_create_reloc_section (file->elf, sec,SHT_RELA ))return -1 ;/* Write ORC entries to sections: */ list_for_each_entry (entry, &orc_list, list) { -if (write_orc_entry (file->elf, orc_sec, ip_rsec, idx++, +if (write_orc_entry (file->elf, orc_sec, sec, idx++, entry->insn_sec, entry->insn_off, &entry->orc))return -1 ;
Re: [PATCH v2 08/14] objtool: Add elf_create_reloc() helper
Quoted 2 lines · Peter Zijlstra
Full patch, because diff on a diff on a diff is getting ludicrous hard to read :-)
Appreciated :-)
Quoted 5 lines
tools/objtool/elf.c +2
-2-void elf_add_reloc (struct elf *elf,struct reloc *reloc) +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend) { -struct section *sec = reloc->sec;
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 462
Quoted 10 lines
tools/objtool/elf.c +10
-0+struct reloc *reloc; + + reloc =malloc (sizeof (*reloc)); +if (!reloc) { +perror ("malloc" ); +return -1 ; + } +memset (reloc,0 ,sizeof (*reloc)); + + reloc->sec = sec->reloc;
Maybe elf_add_reloc() could create the reloc section if it doesn't already exist, that would help remove the multiple calls to elf_create_reloc_section().
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 468
Quoted 6 lines
tools/objtool/elf.c +4
-0+ reloc->offset = offset; + reloc->type = type; + reloc->sym = sym; + reloc->addend = addend;list_add_tail (&reloc->list, &sec->reloc_list);
This should be sec->reloc->reloc_list?
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 08/14] objtool: Add elf_create_reloc() helper
Quoted 11 lines · Josh Poimboeuf
Quoted 11 lines
tools/objtool/elf.c +8
-2-void elf_add_reloc (struct elf *elf,struct reloc *reloc) +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend) { -struct section *sec = reloc->sec; +struct reloc *reloc; + + reloc =malloc (sizeof (*reloc)); +if (!reloc) { +perror ("malloc" ); +return -1 ;
Quoted 8 lines
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 462
Quoted 4 lines
tools/objtool/elf.c +4
-0+ } +memset (reloc,0 ,sizeof (*reloc)); + + reloc->sec = sec->reloc;Maybe
elf_add_reloc()could create the reloc section if it doesn't already exist, that would help remove the multiple calls toelf_create_reloc_section().
I'll do that as yet another patch ;-)
Quoted 8 lines
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 468
Quoted 6 lines
tools/objtool/elf.c +4
-0+ reloc->offset = offset; + reloc->type = type; + reloc->sym = sym; + reloc->addend = addend;list_add_tail (&reloc->list, &sec->reloc_list);This should be
sec->reloc->reloc_list?
Absolutely, there were a few other 'funnies' as well that I just fixed whilst trying to make the whole new stack work again :-)
Re: [PATCH v2 08/14] objtool: Add elf_create_reloc() helper
Quoted 22 lines · Peter Zijlstra
Quoted 11 lines · Josh Poimboeuf
Quoted 11 lines
tools/objtool/elf.c +8
-2-void elf_add_reloc (struct elf *elf,struct reloc *reloc) +int elf_add_reloc (struct elf *elf,struct section *sec,unsigned long offset, +unsigned int type,struct symbol *sym,int addend) { -struct section *sec = reloc->sec; +struct reloc *reloc; + + reloc =malloc (sizeof (*reloc)); +if (!reloc) { +perror ("malloc" ); +return -1 ;Quoted 8 lines
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 462
Quoted 4 lines
tools/objtool/elf.c +4
-0+ } +memset (reloc,0 ,sizeof (*reloc)); + + reloc->sec = sec->reloc;Maybe
elf_add_reloc()could create the reloc section if it doesn't already exist, that would help remove the multiple calls toelf_create_reloc_section().
Quoted 1 line
I'll do that as yet another patch ;-)
Ok!
Quoted 11 lines
Quoted 8 lines
tools/objtool/elf.c static int read_symbols(struct elf *elf) · line 468
Quoted 6 lines
tools/objtool/elf.c +4
-0+ reloc->offset = offset; + reloc->type = type; + reloc->sym = sym; + reloc->addend = addend;list_add_tail (&reloc->list, &sec->reloc_list);This should be
sec->reloc->reloc_list?Absolutely, there were a few other 'funnies' as well that I just fixed whilst trying to make the whole new stack work again :-)
I'm sure some of those were my fault, thanks for testing my crap :-)
Signature · 1 line
Josh
Peter Zijlstra
maintainerPATCH v2 9/14 objtool: Extract elf_strtab_concat()
In reply to Peter ZijlstraCreate a common helper to append strings to a strtab.
--- tools/objtool/elf.c | 73 +++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 31 deletions(-)
tools/objtool/elf.c +42 -31
@@ -676,13 +676,51 @@ struct elf *elf_open_read(const char *nareturn NULL ; } +static int elf_strtab_concat (struct elf *elf,char *name,const char *strtab_name) +{ +struct section *strtab =NULL ; +Elf_Data *data; +Elf_Scn *s; +int len; + +if (strtab_name) + strtab =find_section_by_name (elf, strtab_name); +if (!strtab) + strtab =find_section_by_name (elf,".strtab" ); +if (!strtab) { +WARN ("can't find %s or .strtab section" , strtab_name); +return -1 ; + } + + s =elf_getscn (elf->elf, strtab->idx); +if (!s) { +WARN_ELF ("elf_getscn" ); +return -1 ; + } + + data =elf_newdata (s); +if (!data) { +WARN_ELF ("elf_newdata" ); +return -1 ; + } + + data->d_buf = name; + data->d_size =strlen (name) +1 ;; + data->d_align =1 ; + + len = strtab->len; + strtab->len += data->d_size; + strtab->changed = true; + +return len; +} +struct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr) { -struct section *sec, *shstrtab; +struct section *sec;size_t size = entsize * nr;Elf_Scn *s; -Elf_Data *data; sec =malloc (sizeof (*sec));if (!sec) {
@@ -739,36 +777,9 @@ struct section *elf_create_section(struc sec->sh.sh_addralign =1 ; sec->sh.sh_flags =SHF_ALLOC | sh_flags; - -/* Add section name to .shstrtab (or .strtab for Clang) */ - shstrtab =find_section_by_name (elf,".shstrtab" ); -if (!shstrtab) - shstrtab =find_section_by_name (elf,".strtab" ); -if (!shstrtab) { -WARN ("can't find .shstrtab or .strtab section" ); -return NULL ; - } - - s =elf_getscn (elf->elf, shstrtab->idx); -if (!s) { -WARN_ELF ("elf_getscn" ); + sec->sh.sh_name =elf_strtab_concat (elf, sec->name,".shstrtab" ); +if (sec->sh.sh_name ==-1 )return NULL ; - } - - data =elf_newdata (s); -if (!data) { -WARN_ELF ("elf_newdata" ); -return NULL ; - } - - data->d_buf = sec->name; - data->d_size =strlen (name) +1 ; - data->d_align =1 ; - - sec->sh.sh_name = shstrtab->len; - - shstrtab->len +=strlen (name) +1 ; - shstrtab->changed = true;list_add_tail (&sec->list, &elf->sections);elf_hash_add (elf->section_hash, &sec->hash, sec->idx);
Re: [PATCH v2 09/14] objtool: Extract elf_strtab_concat()
Quoted 18 lines · Peter Zijlstra
Create a common helper to append strings to a strtab.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 73 +++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 31 deletions(-)tools/objtool/elf.c +5
-0@@ -676,13 +676,51 @@ struct elf *elf_open_read(const char *nareturn NULL ; } +static int elf_strtab_concat (struct elf *elf,char *name,const char *strtab_name) +{ +struct section *strtab =NULL ; +Elf_Data *data; +Elf_Scn *s;
tools/objtool/elf.c struct elf *elf_open_read(const char *na · line 693
Quoted 10 lines
tools/objtool/elf.c +10
-0+int len; + +if (strtab_name) + strtab =find_section_by_name (elf, strtab_name); +if (!strtab) + strtab =find_section_by_name (elf,".strtab" ); +if (!strtab) { +WARN ("can't find %s or .strtab section" , strtab_name); +return -1 ; + }
This part's a bit mysterious (and it loses the Clang comment). Maybe we can leave the section finding logic in elf_create_section()?
tools/objtool/elf.c +12 -6
@@ -676,19 +676,17 @@ struct elf *elf_open_read(const char *name, int flags)return NULL ; } -static int elf_strtab_concat (struct elf *elf,char *name,const char *strtab_name) +static int elf_add_string (struct elf *elf,struct section *strtab,char *str) {struct section *strtab =NULL ;Elf_Data *data;Elf_Scn *s;int len; -if (strtab_name) - strtab =find_section_by_name (elf, strtab_name);if (!strtab) strtab =find_section_by_name (elf,".strtab" );if (!strtab) { -WARN ("can't find %s or .strtab section" , strtab_name); +WARN ("can't find .strtab section" );return -1 ; }
@@ -718,7 +716,7 @@ static int elf_strtab_concat(struct elf *elf, char *name, const char *strtab_namstruct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr) { -struct section *sec; +struct section *sec, *shstrtab;size_t size = entsize * nr;Elf_Scn *s;
@@ -777,7 +775,15 @@ struct section *elf_create_section(struct elf *elf, const char *name, sec->sh.sh_addralign =1 ; sec->sh.sh_flags =SHF_ALLOC | sh_flags; - sec->sh.sh_name =elf_strtab_concat (elf, sec->name,".shstrtab" ); +/* Add section name to .shstrtab (or .strtab for Clang) */ + shstrtab =find_section_by_name (elf,".shstrtab" ); +if (!shstrtab) + shstrtab =find_section_by_name (elf,".strtab" ); +if (!shstrtab) { +WARN ("can't find .shstrtab or .strtab section" ); +return NULL ; + } + sec->sh.sh_name =elf_add_string (elf, sec->name, shstrtab);if (sec->sh.sh_name ==-1 )return NULL ;
Peter Zijlstra
maintainerRe: [PATCH v2 09/14] objtool: Extract elf_strtab_concat()
Quoted 24 lines · Josh Poimboeuf
Quoted 23 lines · Peter Zijlstra
Create a common helper to append strings to a strtab.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 73 +++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 31 deletions(-)tools/objtool/elf.c +10
-0@@ -676,13 +676,51 @@ struct elf *elf_open_read(const char *nareturn NULL ; } +static int elf_strtab_concat (struct elf *elf,char *name,const char *strtab_name) +{ +struct section *strtab =NULL ; +Elf_Data *data; +Elf_Scn *s; +int len; + +if (strtab_name) + strtab =find_section_by_name (elf, strtab_name); +if (!strtab)
Quoted 8 lines
tools/objtool/elf.c struct elf *elf_open_read(const char *na · line 693
Quoted 5 lines
tools/objtool/elf.c +5
-0+ strtab =find_section_by_name (elf,".strtab" ); +if (!strtab) { +WARN ("can't find %s or .strtab section" , strtab_name); +return -1 ; + }This part's a bit mysterious (and it loses the Clang comment). Maybe we can leave the section finding logic in
elf_create_section()?
Sure, made it compile too ;-)
Peter Zijlstra
maintainerPATCH v2 10/14 objtool: Extract elf_symbol_add()
In reply to Peter ZijlstraCreate a common helper to add symbols.
--- tools/objtool/elf.c | 57 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-)
tools/objtool/elf.c +33 -24
@@ -290,12 +290,41 @@ static int read_sections(struct elf *elfreturn 0 ; } +static bool elf_symbol_add (struct elf *elf,struct symbol *sym) +{ +struct list_head *entry; +struct rb_node *pnode; + + sym->type =GELF_ST_TYPE (sym->sym.st_info); + sym->bind =GELF_ST_BIND (sym->sym.st_info); + + sym->offset = sym->sym.st_value; + sym->len = sym->sym.st_size; + +rb_add (&sym->node, &sym->sec->symbol_tree, symbol_to_offset); + pnode =rb_prev (&sym->node); +if (pnode) + entry = &rb_entry(pnode, struct symbol, node)->list; +else + entry = &sym->sec->symbol_list; +list_add (&sym->list, entry); +elf_hash_add (elf->symbol_hash, &sym->hash, sym->idx); +elf_hash_add (elf->symbol_name_hash, &sym->name_hash,str_hash (sym->name)); + +/* +* Don't store empty STT_NOTYPE symbols in the rbtree. They +* can exist within a function, confusing the sorting. +*/ +if (!sym->len) +rb_erase (&sym->node, &sym->sec->symbol_tree); + +return true; +} +static int read_symbols (struct elf *elf) {struct section *symtab, *symtab_shndx, *sec;struct symbol *sym, *pfunc; -struct list_head *entry; -struct rb_node *pnode;int symbols_nr, i;char *coldstr;Elf_Data *shndx_data =NULL ;
@@ -340,9 +369,6 @@ static int read_symbols(struct elf *elf) goto err; } - sym->type =GELF_ST_TYPE (sym->sym.st_info); - sym->bind =GELF_ST_BIND (sym->sym.st_info); -if ((sym->sym.st_shndx >SHN_UNDEF && sym->sym.st_shndx <SHN_LORESERVE ) || (shndx_data && sym->sym.st_shndx ==SHN_XINDEX )) {
@@ -362,25 +388,8 @@ static int read_symbols(struct elf *elf) }else sym->sec =find_section_by_index (elf,0 ); - sym->offset = sym->sym.st_value; - sym->len = sym->sym.st_size; - -rb_add (&sym->node, &sym->sec->symbol_tree, symbol_to_offset); - pnode =rb_prev (&sym->node); -if (pnode) - entry = &rb_entry(pnode, struct symbol, node)->list; -else - entry = &sym->sec->symbol_list; -list_add (&sym->list, entry); -elf_hash_add (elf->symbol_hash, &sym->hash, sym->idx); -elf_hash_add (elf->symbol_name_hash, &sym->name_hash,str_hash (sym->name)); - -/* -* Don't store empty STT_NOTYPE symbols in the rbtree. They -* can exist within a function, confusing the sorting. -*/ -if (!sym->len) -rb_erase (&sym->node, &sym->sec->symbol_tree); +if (!elf_symbol_add (elf, sym)) + goto err; }if (stats)
Re: [PATCH v2 10/14] objtool: Extract elf_symbol_add()
Quoted 10 lines · Peter Zijlstra
Create a common helper to add symbols.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 57 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-)
tools/objtool/elf.c static int read_sections(struct elf *elf · line 293
Quoted 4 lines
tools/objtool/elf.c +1
-0@@ -290,12 +290,41 @@ static int read_sections(struct elf *elfreturn 0 ; } +static bool elf_symbol_add (struct elf *elf,struct symbol *sym)
How about "elf_add_symbol()" for consistency with my other suggestions (elf_add_reloc() and elf_add_string()). And return an int.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 10/14] objtool: Extract elf_symbol_add()
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines · Peter Zijlstra
Create a common helper to add symbols.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 57 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-)tools/objtool/elf.c +0
-0@@ -290,12 +290,41 @@ static int read_sections(struct elf *elfreturn 0 ; }
Quoted 4 lines
tools/objtool/elf.c static int read_sections(struct elf *elf · line 293
Quoted 1 line
tools/objtool/elf.c +1
-0+static bool elf_symbol_add (struct elf *elf,struct symbol *sym)How about "
elf_add_symbol()" for consistency with my other suggestions (elf_add_reloc()andelf_add_string()). And return an int.
Changed the nane, how about void? This latest incarnation doesn't actually have an error path. Still doesn't hurt to have one and not use it I suppose...
Re: [PATCH v2 10/14] objtool: Extract elf_symbol_add()
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines · Peter Zijlstra
Create a common helper to add symbols.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 57 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-)tools/objtool/elf.c +0
-0@@ -290,12 +290,41 @@ static int read_sections(struct elf *elfreturn 0 ; }
Quoted 8 lines
Quoted 4 lines
tools/objtool/elf.c static int read_sections(struct elf *elf · line 293
Quoted 1 line
tools/objtool/elf.c +1
-0+static bool elf_symbol_add (struct elf *elf,struct symbol *sym)How about "
elf_add_symbol()" for consistency with my other suggestions (elf_add_reloc()andelf_add_string()). And return an int.Changed the nane, how about void? This latest incarnation doesn't actually have an error path. Still doesn't hurt to have one and not use it I suppose...
void would be my preference, just to avoid unnecessary error handling boilerplate in the caller.
Signature · 1 line
Josh
Peter Zijlstra
maintainerPATCH v2 11/14 objtool: Add elf_create_undef_symbol()
In reply to Peter ZijlstraAllow objtool to create undefined symbols; this allows creating relocations to symbols not currently in the symbol table.
--- tools/objtool/elf.c | 63 ++++++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/elf.h | 1 2 files changed, 64 insertions(+)
tools/objtool/elf.c +63 -0
@@ -724,6 +724,69 @@ static int elf_strtab_concat(struct elfreturn len; } +struct symbol *elf_create_undef_symbol (struct elf *elf,const char *name) +{ +struct section *symtab; +struct symbol *sym; +Elf_Data *data; +Elf_Scn *s; + + sym =malloc (sizeof (*sym)); +if (!sym) { +perror ("malloc" ); +return NULL ; + } +memset (sym,0 ,sizeof (*sym)); + + sym->name =strdup (name); + + sym->sym.st_name =elf_strtab_concat (elf, sym->name,NULL ); +if (sym->sym.st_name ==-1 ) +return NULL ; + + sym->sym.st_info =0x10 ;/* STB_GLOBAL, STT_NOTYPE */ +// st_other 0 +// st_shndx 0 +// st_value 0 +// st_size 0 + + symtab =find_section_by_name (elf,".symtab" ); +if (!symtab) { +WARN ("can't find .symtab" ); +return NULL ; + } + + s =elf_getscn (elf->elf, symtab->idx); +if (!s) { +WARN_ELF ("elf_getscn" ); +return NULL ; + } + + data =elf_newdata (s); +if (!data) { +WARN_ELF ("elf_newdata" ); +return NULL ; + } + + data->d_buf = &sym->sym; + data->d_size =sizeof (sym->sym); + data->d_align =1 ; + + sym->idx = symtab->len /sizeof (sym->sym); + + symtab->len += data->d_size; + symtab->changed = true; + + sym->sec =find_section_by_index (elf,0 ); + +if (!elf_symbol_add (elf, sym)) { +WARN ("elf_symbol_add" ); +return NULL ; + } + +return sym; +} +struct section *elf_create_section (struct elf *elf,const char *name,unsigned int sh_flags,size_t entsize,int nr) {
tools/objtool/include/objtool/elf.h +1 -0
@@ -136,6 +136,7 @@ int elf_write_insn(struct elf *elf, stru unsigned long offset, unsigned int len, const char *insn);int elf_write_reloc (struct elf *elf,struct reloc *reloc); +struct symbol *elf_create_undef_symbol (struct elf *elf,const char *name);int elf_write (struct elf *elf);void elf_close (struct elf *elf);
Re: [PATCH v2 11/14] objtool: Add elf_create_undef_symbol()
Quoted 26 lines · Peter Zijlstra
Allow objtool to create undefined symbols; this allows creating relocations to symbols not currently in the symbol table.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 63 ++++++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/elf.h | 1 2 files changed, 64 insertions(+)tools/objtool/elf.c +11
-0@@ -724,6 +724,69 @@ static int elf_strtab_concat(struct elfreturn len; } +struct symbol *elf_create_undef_symbol (struct elf *elf,const char *name) +{ +struct section *symtab; +struct symbol *sym; +Elf_Data *data; +Elf_Scn *s; + + sym =malloc (sizeof (*sym)); +if (!sym) { +perror ("malloc" ); +return NULL ;
tools/objtool/elf.c static int elf_strtab_concat(struct elf · line 747
Quoted 10 lines
tools/objtool/elf.c +10
-0+ } +memset (sym,0 ,sizeof (*sym)); + + sym->name =strdup (name); + + sym->sym.st_name =elf_strtab_concat (elf, sym->name,NULL ); +if (sym->sym.st_name ==-1 ) +return NULL ; + + sym->sym.st_info =0x10 ;/* STB_GLOBAL, STT_NOTYPE */
There's a generic macro for this:
sym->sym.st_info = GELF_ST_INFO(STB_GLOBAL, STT_NOTYPE);
And sym->bind and sym->type should probably get set.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 11/14] objtool: Add elf_create_undef_symbol()
Quoted 40 lines · Josh Poimboeuf
Quoted 26 lines · Peter Zijlstra
Allow objtool to create undefined symbols; this allows creating relocations to symbols not currently in the symbol table.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- tools/objtool/elf.c | 63 ++++++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/elf.h | 1 2 files changed, 64 insertions(+)tools/objtool/elf.c +11
-0@@ -724,6 +724,69 @@ static int elf_strtab_concat(struct elfreturn len; } +struct symbol *elf_create_undef_symbol (struct elf *elf,const char *name) +{ +struct section *symtab; +struct symbol *sym; +Elf_Data *data; +Elf_Scn *s; + + sym =malloc (sizeof (*sym)); +if (!sym) { +perror ("malloc" ); +return NULL ;tools/objtool/elf.c static int elf_strtab_concat(struct elf · line 747
Quoted 10 lines
tools/objtool/elf.c +10
-0+ } +memset (sym,0 ,sizeof (*sym)); + + sym->name =strdup (name); + + sym->sym.st_name =elf_strtab_concat (elf, sym->name,NULL ); +if (sym->sym.st_name ==-1 ) +return NULL ; + + sym->sym.st_info =0x10 ;/* STB_GLOBAL, STT_NOTYPE */ There's a generic macro for this:
Quoted 1 line
sym->sym.st_info = GELF_ST_INFO(STB_GLOBAL, STT_NOTYPE);
Ah, I remember not finding that many moons ago when I wrote that ..
Quoted 1 line
And
sym->bindandsym->typeshould probably get set.
They are, it's in that elf_add_symbol() thing.
Peter Zijlstra
maintainerPATCH v2 12/14 objtool: Allow archs to rewrite retpolines
In reply to Peter ZijlstraWhen retpolines are employed, compilers typically emit calls to retpoline thunks. Objtool recognises these calls and marks them as dynamic calls.
Provide infrastructure for architectures to rewrite/augment what the compiler wrote for us.
--- tools/objtool/check.c | 23 +++++++++++++++++++++-- tools/objtool/include/objtool/arch.h | 3 +++ 2 files changed, 24 insertions(+), 2 deletions(-)
tools/objtool/check.c +21 -2
@@ -852,6 +852,14 @@ __weak bool arch_is_retpoline(struct symreturn false; } + +__weak int arch_rewrite_retpoline (struct objtool_file *file, +struct instruction *insn, +struct reloc *reloc) +{ +return 0 ; +} +/* * Find the destination instructions for all jumps. */
@@ -885,6 +893,9 @@ static int add_jump_destinations(struct insn->type =INSN_JUMP_DYNAMIC_CONDITIONAL ; insn->retpoline_safe = true; + +arch_rewrite_retpoline (file, insn, reloc); +continue ; }else if (insn->func) {/* internal or external sibling call (with reloc) */
@@ -1036,6 +1047,8 @@ static int add_call_destinations(struct insn->type =INSN_CALL_DYNAMIC ; insn->retpoline_safe = true; +arch_rewrite_retpoline (file, insn, reloc); +remove_insn_ops (insn);continue ;
@@ -1212,6 +1225,8 @@ static int handle_group_alt(struct objto dest_off =arch_jump_destination (insn);if (dest_off == special_alt->new_off + special_alt->new_len) insn->jump_dest =next_insn_same_sec (file, last_orig_insn); +else + insn->jump_dest =find_insn (file, insn->sec, dest_off);if (!insn->jump_dest) { WARN_FUNC("can't find alternative jump destination",
@@ -1797,11 +1812,15 @@ static int decode_sections(struct objtooif (ret)return ret; - ret =add_jump_destinations (file); +/* +* Must be before add_{jump,call}_destination; for they can add +* magic alternatives. +*/ + ret =add_special_section_alts (file);if (ret)return ret; - ret =add_special_section_alts (file); + ret =add_jump_destinations (file);if (ret)return ret;
tools/objtool/include/objtool/arch.h +3 -0
@@ -82,6 +82,9 @@ unsigned long arch_jump_destination(struunsigned long arch_dest_reloc_offset (int addend);const char *arch_nop_insn (int len); +int arch_rewrite_retpoline (struct objtool_file *file, +struct instruction *insn, +struct reloc *reloc);int arch_decode_hint_reg (struct instruction *insn,u8 sp_reg);
Re: [PATCH v2 12/14] objtool: Allow archs to rewrite retpolines
tools/objtool/check.c static int handle_group_alt(struct objto · line 1232
Quoted 9 lines · Peter Zijlstra
tools/objtool/check.c +2
-0@@ -1212,6 +1225,8 @@ static int handle_group_alt(struct objto dest_off =arch_jump_destination (insn);if (dest_off == special_alt->new_off + special_alt->new_len) insn->jump_dest =next_insn_same_sec (file, last_orig_insn); +else + insn->jump_dest =find_insn (file, insn->sec, dest_off);if (!insn->jump_dest) { WARN_FUNC("can't find alternative jump destination",
So I assume this change is because of the ordering change: now this is done before add_jump_destinations().
But doesn't that mean the alternative jump modification (changing the dest to the end of the original insns) will get overwritten later?
Also the new hunk to be an oversimplified version of add_jump_destinations(). I'm not quite convinced that it will always do the right thing for this case.
tools/objtool/check.c static int decode_sections(struct objtoo · line 1819
Quoted 10 lines
tools/objtool/check.c +5
-1@@ -1797,11 +1812,15 @@ static int decode_sections(struct objtooif (ret)return ret; - ret =add_jump_destinations (file); +/* +* Must be before add_{jump,call}_destination; for they can add +* magic alternatives. +*/ + ret =add_special_section_alts (file);
This reordering is unfortunate. Maybe there's a better way, though I don't have any ideas, at least until I get to the most controversial patch.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 12/14] objtool: Allow archs to rewrite retpolines
Quoted 8 lines · Josh Poimboeuf
Quoted 7 lines · Peter Zijlstra
tools/objtool/check.c +2
-0@@ -1212,6 +1225,8 @@ static int handle_group_alt(struct objto dest_off =arch_jump_destination (insn);if (dest_off == special_alt->new_off + special_alt->new_len) insn->jump_dest =next_insn_same_sec (file, last_orig_insn); +else + insn->jump_dest =find_insn (file, insn->sec, dest_off);
Quoted 5 lines
tools/objtool/check.c static int handle_group_alt(struct objto · line 1232
Quoted 2 lines
tools/objtool/check.c +0
-0if (!insn->jump_dest) { WARN_FUNC("can't find alternative jump destination",So I assume this change is because of the ordering change: now this is done before
add_jump_destinations().
Correct.
Quoted 3 lines
Also the new hunk to be an oversimplified version of
add_jump_destinations(). I'm not quite convinced that it will always do the right thing for this case.
You're right; this is because of the reorder. At the time I thought this was right, but looking at it now, I'm not sure. Esp. so when ARM64 comes along and allows more relocations in alternatives.
Let me see if I can come up with something better.
Quoted 2 lines
But doesn't that mean the alternative jump modification (changing the dest to the end of the original insns) will get overwritten later?
Good point, should be simple enough to fix by having add_jump_destination() skip all that already have insn->jump_dest set I suppose.
Quoted 4 lines
Quoted 4 lines
tools/objtool/check.c +0
-0@@ -1797,11 +1812,15 @@ static int decode_sections(struct objtooif (ret)return ret;
Quoted 10 lines
tools/objtool/check.c static int decode_sections(struct objtoo · line 1819
Quoted 6 lines
tools/objtool/check.c +5
-1- ret =add_jump_destinations (file); +/* +* Must be before add_{jump,call}_destination; for they can add +* magic alternatives. +*/ + ret =add_special_section_alts (file);This reordering is unfortunate. Maybe there's a better way, though I don't have any ideas, at least until I get to the most controversial patch.
So the thing no longer crashes on the alternatives it writes, so we *could* read back our own alternatives, but it does seem somewhat unfortunate to do that. Too easy to get into cycles that way.
Peter Zijlstra
maintainerRe: [PATCH v2 12/14] objtool: Allow archs to rewrite retpolines
Quoted 1 line · Peter Zijlstra
Let me see if I can come up with something better.
Duh, we can put the instructions on a list and process them later, just like static_call, can even use the same insn list_head since a retpoline cannot be a static call.
Lemme go make that happen.
Peter Zijlstra
maintainerPATCH v2 13/14 objtool: Skip magical retpoline .altinstr_replacement
In reply to Peter ZijlstraWhen the .altinstr_replacement is a retpoline, skip the alternative. We already special case retpolines anyway.
--- tools/objtool/special.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
tools/objtool/special.c +11 -1
@@ -106,6 +106,14 @@ static int get_alt_entry(struct elf *elfreturn -1 ; } +/* +* Skip retpoline .altinstr_replacement... we already rewrite the +* instructions for retpolines anyway, see arch_is_retpoline() +* usage in add_{call,jump}_destinations(). +*/ +if (arch_is_retpoline (new_reloc->sym)) +return 1 ; + alt->new_sec = new_reloc->sym->sec; alt->new_off = (unsigned int )new_reloc->addend;
@@ -154,7 +162,9 @@ int special_get_alts(struct elf *elf, stmemset (alt,0 ,sizeof (*alt)); ret =get_alt_entry (elf, entry, sec, idx, alt); -if (ret) +if (ret >0 ) +continue ; +if (ret <0 )return ret;list_add_tail (&alt->list, alts);
Peter Zijlstra
maintainerPATCH v2 14/14 objtool,x86: Rewrite retpoline thunk calls
In reply to Peter ZijlstraWhen the compiler emits: "CALL __x86_indirect_thunk_\reg" for an indirect call, have objtool rewrite it to:
ALTERNATIVE "call __x86_indirect_thunk_\reg", "call *%reg", ALT_NOT(X86_FEATURE_RETPOLINE)
Additionally, in order to not emit endless identical .altinst_replacement chunks, use a global symbol for them, see __x86_indirect_alt_*.
This also avoids objtool from having to do code generation.
--- arch/x86/include/asm/asm-prototypes.h | 12 ++- arch/x86/lib/retpoline.S | 33 +++++++++ tools/objtool/arch/x86/decode.c | 116 ++++++++++++++++++++++++++++++++++ 3 files changed, 158 insertions(+), 3 deletions(-)
arch/x86/include/asm/asm-prototypes.h +10 -2
@@ -19,11 +19,19 @@ extern void cmpxchg8b_emu(void);#ifdef CONFIG_RETPOLINE -#define DECL_INDIRECT_THUNK (reg) \ +#undef GEN +#define GEN (reg) \ extern asmlinkage void __x86_indirect_thunk_ ## reg (void); +#include <asm/GEN-for-each-reg.h> + +#undef GEN +#define GEN (reg) \ + extern asmlinkage void __x86_indirect_alt_call_ ## reg (void); +#include <asm/GEN-for-each-reg.h> #undef GEN -#define GEN (reg) DECL_INDIRECT_THUNK(reg) +#define GEN (reg) \ + extern asmlinkage void __x86_indirect_alt_jmp_ ## reg (void);#include <asm/GEN-for-each-reg.h> #endif /* CONFIG_RETPOLINE */
arch/x86/lib/retpoline.S +32 -1
@@ -10,6 +10,8 @@ #include <asm/unwind_hints.h> #include <asm/frame.h> + .section .text.__x86.indirect_thunk + .macro RETPOLINE reg ANNOTATE_INTRA_FUNCTION_CALL call .Ldo_rop_\@
@@ -25,7 +27,6 @@ .endm .macro THUNK reg - .section .text.__x86.indirect_thunk .align 32 SYM_FUNC_START(__x86_indirect_thunk_\reg)
@@ -38,6 +39,24 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) .endm +.macro ALT_THUNK reg + + .align 1 + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg) + ANNOTATE_RETPOLINE_SAFE +1: call *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_call_\reg) + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) + ANNOTATE_RETPOLINE_SAFE +1: jmp *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_jmp_\reg) + +.endm + /* * Despite being an assembler file we can't just use .irp here * because __KSYM_DEPS__ only uses the C preprocessor and would
@@ -61,3 +80,15 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) #define GEN(reg) EXPORT_THUNK(reg) #include <asm/GEN-for-each-reg.h> +#undef GEN +#define GEN(reg) ALT_THUNK reg +#include <asm/GEN-for-each-reg.h> + +#undef GEN +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_call_ ## reg) +#include <asm/GEN-for-each-reg.h> + +#undef GEN +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_jmp_ ## reg) +#include <asm/GEN-for-each-reg.h> +
tools/objtool/arch/x86/decode.c +116 -0
128 lines not on this page · raw message
Re: [PATCH v2 14/14] objtool,x86: Rewrite retpoline thunk calls
Quoted 6 lines · Peter Zijlstra
When the compiler emits: "CALL __x86_indirect_thunk_\reg" for an indirect call, have objtool rewrite it to:
ALTERNATIVE "call __x86_indirect_thunk_\reg", "call *%reg", ALT_NOT(X86_FEATURE_RETPOLINE)
Quoted 7 lines
Additionally, in order to not emit endless identical .altinst_replacement chunks, use a global symbol for them, see __x86_indirect_alt_*.
This also avoids objtool from having to do code generation.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
This is better than I expected. Nice workaround for not generating code.
Quoted 5 lines
+.macro
ALT_THUNKregarch/x86/lib/retpoline.S +4
-0+ + .align 1 + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg)
arch/x86/lib/retpoline.S SYM_FUNC_END(__x86_indirect_thunk_\reg) · line 56
Quoted 10 lines
arch/x86/lib/retpoline.S +10
-0+ ANNOTATE_RETPOLINE_SAFE +1: call *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_call_\reg) + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) + ANNOTATE_RETPOLINE_SAFE +1: jmp *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_jmp_\reg)
This mysterious code needs a comment. Shouldn't it be in .altinstr_replacement or something?
Also doesn't the alternative code already insert nops?
Quoted 18 lines
tools/objtool/arch/x86/decode.c +18
-0+int arch_rewrite_retpoline (struct objtool_file *file, +struct instruction *insn, +struct reloc *reloc) +{ +struct symbol *sym; +char name[32 ] ="" ; + +if (!strcmp (insn->sec->name,".text.__x86.indirect_thunk" )) +return 0 ; + +sprintf (name,"__x86_indirect_alt_%s_%s" , + insn->type ==INSN_JUMP_DYNAMIC ?"jmp" :"call" , + reloc->sym->name +21 ); + + sym =find_symbol_by_name (file->elf, name); +if (!sym) { + sym =elf_create_undef_symbol (file->elf, name); +if (!sym) {
tools/objtool/arch/x86/decode.c const char *arch_nop_insn(int len) · line 774
Quoted 10 lines
tools/objtool/arch/x86/decode.c +10
-0+WARN ("elf_create_undef_symbol" ); +return -1 ; + } + } + +elf_add_alternative (file->elf, insn, sym, +ALT_NOT (X86_FEATURE_RETPOLINE ),5 ,5 ); + +return 0 ; +}
Need to propagate the error.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 14/14] objtool,x86: Rewrite retpoline thunk calls
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Peter Zijlstra
When the compiler emits: "CALL __x86_indirect_thunk_\reg" for an indirect call, have objtool rewrite it to:
ALTERNATIVE "call __x86_indirect_thunk_\reg", "call *%reg", ALT_NOT(X86_FEATURE_RETPOLINE)Additionally, in order to not emit endless identical .altinst_replacement chunks, use a global symbol for them, see __x86_indirect_alt_*.
Quoted 6 lines
Quoted 3 lines
This also avoids objtool from having to do code generation.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>This is better than I expected. Nice workaround for not generating code.
Thanks :-)
Quoted 10 lines
Quoted 10 lines
+.macro
ALT_THUNKregarch/x86/lib/retpoline.S +9
-0+ + .align 1 + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg) + ANNOTATE_RETPOLINE_SAFE +1: call *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_call_\reg) +
Quoted 8 lines
arch/x86/lib/retpoline.S SYM_FUNC_END(__x86_indirect_thunk_\reg) · line 56
Quoted 5 lines
arch/x86/lib/retpoline.S +5
-0+SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) + ANNOTATE_RETPOLINE_SAFE +1: jmp *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_jmp_\reg)This mysterious code needs a comment. Shouldn't it be in .altinstr_replacement or something?
Comment, yes, I suppose so. And no, if we stick it in .altinstr_replacement we'll throw them away with initmem and module alternative patching (which will also refer to these symbols) will go side-ways.
Quoted 1 line
Also doesn't the alternative code already insert nops?
Problem is that the {call,jmp} *%\reg thing is not fixed length. They're 2 or 3 bytes depending on which register is picked.
We could make them all 3 long and insert 0,1 nop I suppose.
Initially alternatives wouldn't re-optimize nops on patched things, it would simply add nops on. And I had the above be:
1: INSN *%\reg
2: .nops 5-(2b-1b)
and we'd get a single right sized nop. But the .nops directive it too new, we support binutils that don't have it :/
Hence, it now reads:
2: .skip 5-(2b-1b), 0x90
End result is that alternative NOP optimizer patch at the start of the series that now also optimizes a bunch of cases that are unrelated and were previously missed -- but crucially, it covers this case too :-)
Anyway, yes I could make it 3 long.
Quoted 22 lines
Quoted 22 lines
tools/objtool/arch/x86/decode.c +22
-0+int arch_rewrite_retpoline (struct objtool_file *file, +struct instruction *insn, +struct reloc *reloc) +{ +struct symbol *sym; +char name[32 ] ="" ; + +if (!strcmp (insn->sec->name,".text.__x86.indirect_thunk" )) +return 0 ; + +sprintf (name,"__x86_indirect_alt_%s_%s" , + insn->type ==INSN_JUMP_DYNAMIC ?"jmp" :"call" , + reloc->sym->name +21 ); + + sym =find_symbol_by_name (file->elf, name); +if (!sym) { + sym =elf_create_undef_symbol (file->elf, name); +if (!sym) { +WARN ("elf_create_undef_symbol" ); +return -1 ; + } + }
Quoted 8 lines
tools/objtool/arch/x86/decode.c const char *arch_nop_insn(int len) · line 774
Quoted 6 lines
tools/objtool/arch/x86/decode.c +6
-0+ +elf_add_alternative (file->elf, insn, sym, +ALT_NOT (X86_FEATURE_RETPOLINE ),5 ,5 ); + +return 0 ; +}Need to propagate the error.
Oh, indeed so.
Re: [PATCH v2 14/14] objtool,x86: Rewrite retpoline thunk calls
Quoted 4 lines · Peter Zijlstra
Quoted 1 line
Also doesn't the alternative code already insert nops?
Problem is that the {call,jmp} *%\reg thing is not fixed length. They're 2 or 3 bytes depending on which register is picked.
Why do they need to be fixed length? Objtool can use sym->len as the alternative replacement length. Then alternatives can add nops as needed.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH v2 14/14] objtool,x86: Rewrite retpoline thunk calls
Quoted 9 lines · Josh Poimboeuf
Quoted 4 lines · Peter Zijlstra
Quoted 1 line
Also doesn't the alternative code already insert nops?
Problem is that the {call,jmp} *%\reg thing is not fixed length. They're 2 or 3 bytes depending on which register is picked.
Why do they need to be fixed length? Objtool can use
sym->lenas the alternative replacement length. Then alternatives can add nops as needed.
UNDEF has size 0. That is, unless these symbols exist in the translation unit (they do not) we have no clue.
Arguably I could parse the symbol name again and then we know the register number and thus if we need REX etc.. but I figured we wanted to avoid all that.
Re: [PATCH v2 14/14] objtool,x86: Rewrite retpoline thunk calls
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Josh Poimboeuf
Quoted 4 lines · Peter Zijlstra
Quoted 1 line
Also doesn't the alternative code already insert nops?
Problem is that the {call,jmp} *%\reg thing is not fixed length. They're 2 or 3 bytes depending on which register is picked.
Why do they need to be fixed length? Objtool can use
sym->lenas the alternative replacement length. Then alternatives can add nops as needed.
Quoted 6 lines
UNDEF has size 0. That is, unless these symbols exist in the translation unit (they do not) we have no clue.
Arguably I could parse the symbol name again and then we know the register number and thus if we need REX etc.. but I figured we wanted to avoid all that.
Ah, makes sense now.
Signature · 1 line
Josh