x86, alternatives: Instruction padding and more robust JMPs
Subsystems (3)
Borislav Petkov
maintainerv2 cover x86, alternatives: Instruction padding and more robust JMPs
From: Borislav Petkov <[email protected]>
[ Changelog is in version-increasing number so that one can follow the
evolution of the patch set in a more natural way (i.e., latest version comes at the end. ]
v0:
this is something which hpa and I talked about recently: the ability for the alternatives code to add padding to the original instruction in case the replacement is longer and also to be able to simply write "jmp" and not care about which JMP exactly the compiler generates and whether the relative offsets are correct.
So this is a stab at it, it seems to boot in kvm here but it needs more staring to make sure we're actually generating the proper code at all times.
Thus the RFC tag, comments/suggestions are welcome.
v1:
This is the first version which passes testing on AMD/Intel, 32/64-bit boxes I have here. For more info what it does, you can boot with "debug-alternative" to see some verbose information about what gets changed into what.
Patches 1 and 2 are cleanups.
Patch 3 is adding the padding at build time and patch 4 simplifies using JMPs in alternatives without having to do crazy math with labels, as a user of the alternatives facilities.
Patch 5 optimizes the single-byte NOPs we're adding at build time to longer NOPs which should go easier through the frontend.
Patches 6-12 then convert most of the alternative callsites to the generic macros and kill the homegrown fun.
v2:
This version reworks the NOP padding by adding a field to struct alt_instr which holds the padding length and thus makes the padding more robust than what we did before, instead of us trying to figure out which byte is a NOP and which byte is something else (part of a relative offset or immediate...).
Thanks to Andy Lutomirsky for pointing that out.
As always, constructive comments/suggestions are welcome.
Borislav Petkov (15):
x86/lib/copy_user_64.S: Remove FIX_ALIGNMENT define x86/alternatives: Cleanup DPRINTK macro x86/alternatives: Add instruction padding x86/alternatives: Make JMPs more robust x86/alternatives: Use optimized NOPs for padding x86/lib/copy_page_64.S: Use generic ALTERNATIVE macro x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2 x86/smap: Use ALTERNATIVE macro
x86/entry_32: Convert X86_INVD_BUG to ALTERNATIVE macro x86/lib/clear_page_64.S: Convert to ALTERNATIVE_2 macro x86/asm: Use alternative_2() in rdtsc_barrier() x86/asm: Cleanup prefetch primitives x86/lib/memset_64.S: Convert to ALTERNATIVE_2 macro x86/lib/memmove_64.S: Convert memmove() to ALTERNATIVE macro x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
arch/x86/include/asm/alternative-asm.h | 43 ++++++++- arch/x86/include/asm/alternative.h | 65 ++++++++------ arch/x86/include/asm/apic.h | 2 +- arch/x86/include/asm/barrier.h | 6 +- arch/x86/include/asm/cpufeature.h | 30 ++++--- arch/x86/include/asm/processor.h | 16 ++-- arch/x86/include/asm/smap.h | 30 ++----- arch/x86/kernel/alternative.c | 158 ++++++++++++++++++++++++++++----- arch/x86/kernel/cpu/amd.c | 5 ++ arch/x86/kernel/entry_32.S | 12 +-- arch/x86/lib/clear_page_64.S | 66 ++++++-------- arch/x86/lib/copy_page_64.S | 37 +++----- arch/x86/lib/copy_user_64.S | 46 +++------- arch/x86/lib/memcpy_64.S | 68 +++++--------- arch/x86/lib/memmove_64.S | 19 +--- arch/x86/lib/memset_64.S | 61 +++++-------- arch/x86/um/asm/barrier.h | 4 +- 17 files changed, 360 insertions(+), 308 deletions(-)
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 1/15 x86/lib/copy_user_64.S: Remove FIX_ALIGNMENT define
From: Borislav Petkov <[email protected]>
It is unconditionally enabled so remove it. No object file change.
--- arch/x86/lib/copy_user_64.S | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/lib/copy_user_64.S +0 -5
@@ -8,9 +8,6 @@ #include <linux/linkage.h> #include <asm/dwarf2.h> - -#define FIX_ALIGNMENT 1 - #include <asm/current.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h>
@@ -45,7 +42,6 @@ .endm .macro ALIGN_DESTINATION -#ifdef FIX_ALIGNMENT /* check for bad alignment of destination */ movl %edi,%ecx andl $7,%ecx
@@ -67,7 +63,6 @@ _ASM_EXTABLE(100b,103b) _ASM_EXTABLE(101b,103b) -#endif .endm /* Standard copy_to_user with segment limit checking */
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 2/15 x86/alternatives: Cleanup DPRINTK macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Make it pass __func__ implicitly. Also, dump info about each replacing we're doing. Fixup comments and style while at it.
--- arch/x86/kernel/alternative.c | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-)
arch/x86/kernel/alternative.c +25 -16
@@ -52,10 +52,10 @@ static int __init setup_noreplace_paravirt(char *str)__setup ("noreplace-paravirt" , setup_noreplace_paravirt);#endif -#define DPRINTK (fmt, ...) \ -do { \ - if (debug_alternative) \ - printk(KERN_DEBUG fmt, ##__VA_ARGS__); \ +#define DPRINTK(fmt, args...) \ +do { \ + if (debug_alternative) \ + printk(KERN_DEBUG "%s: " fmt "\n", __func__, ##args); \ } while (0)/*
@@ -243,12 +243,13 @@ extern struct alt_instr __alt_instructions[], __alt_instructions_end[];extern s32 __smp_locks[], __smp_locks_end[];void *text_poke_early (void *addr,const void *opcode,size_t len); -/* Replace instructions with better alternatives for this CPU type. -This runs before SMP is initialized to avoid SMP problems with -self modifying code. This implies that asymmetric systems where -APs have less capabilities than the boot processor are not handled. -Tough. Make sure you disable such features by hand. */ - +/* +* Replace instructions with better alternatives for this CPU type. This runs +* before SMP is initialized to avoid SMP problems with self modifying code. +* This implies that asymmetric systems where APs have less capabilities than +* the boot processor are not handled. Tough. Make sure you disable such +* features by hand. +*/ void __init_or_module apply_alternatives(struct alt_instr *start,struct alt_instr *end) {
@@ -256,10 +257,10 @@ void __init_or_module apply_alternatives(struct alt_instr *start,u8 *instr, *replacement;u8 insnbuf[MAX_PATCH_LEN ]; -DPRINTK ("%s: alt table %p -> %p\n" , __func__, start, end); +DPRINTK ("alt table %p -> %p" , start, end);/* * The scan order should be from start to end. A later scanned -* alternative code can overwrite a previous scanned alternative code. +* alternative code can overwrite previously scanned alternative code. * Some kernel functions (e.g. memcpy, memset, etc) use this order to * patch code. *
@@ -275,11 +276,19 @@ void __init_or_module apply_alternatives(struct alt_instr *start,if (!boot_cpu_has (a->cpuid))continue ; +DPRINTK ("feat: %d*32+%d, old: (%p, len: %d), repl: (%p, len: %d)" , + a->cpuid >>5 , + a->cpuid &0x1f , + instr, a->instrlen, + replacement, a->replacementlen); +memcpy (insnbuf, replacement, a->replacementlen);/* 0xe8 is a relative jump; fix the offset. */ -if (*insnbuf ==0xe8 && a->replacementlen ==5 ) - *(s32 *)(insnbuf +1 ) += replacement - instr; +if (*insnbuf ==0xe8 && a->replacementlen ==5 ) { + *(s32 *)(insnbuf +1 ) += replacement - instr; +DPRINTK ("Fix CALL offset: 0x%x" , *(s32 *)(insnbuf +1 )); + }add_nops (insnbuf + a->replacementlen, a->instrlen - a->replacementlen);
@@ -371,8 +380,8 @@ void __init_or_module alternatives_smp_module_add(struct module *mod, smp->locks_end = locks_end; smp->text = text; smp->text_end = text_end; -DPRINTK ("%s: locks %p -> %p, text %p -> %p, name %s\n" , - __func__, smp->locks, smp->locks_end, +DPRINTK ("locks %p -> %p, text %p -> %p, name %s\n" , + smp->locks, smp->locks_end, smp->text, smp->text_end, smp->name);list_add_tail (&smp->next, &smp_alt_modules);
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 3/15 x86/alternatives: Add instruction padding
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Up until now we have always paid attention to make sure the length of the new instruction replacing the old one is at least less or equal to the length of the old instruction. If the new instruction is longer, at the time it replaces the old instruction it will overwrite the beginning of the next instruction in the kernel image and cause your pants to catch fire.
So instead of having to pay attention, teach the alternatives framework to pad shorter old instructions with NOPs at buildtime - but only in the case when
len(old instruction(s)) < len(new instruction(s))
and add nothing in the >= case. (In that case we do add_nops() when patching).
This way the alternatives user shouldn't have to care about instruction sizes and simply use the macros.
Add asm ALTERNATIVE* flavor macros too, while at it.
Also, we need to save the pad length in a separate struct alt_instr member for NOP optimization and the way to do that reliably is to carry the pad length instead of trying to detect whether we're looking at single-byte NOPs or at pathological instruction offsets like e9 90 90 90 90, for example, which is a valid instruction.
Thanks to Michael Matz for the great help with toolchain questions.
--- arch/x86/include/asm/alternative-asm.h | 43 +++++++++++++++++++++- arch/x86/include/asm/alternative.h | 65 +++++++++++++++++++++------------- arch/x86/include/asm/cpufeature.h | 22 ++++++++---- arch/x86/include/asm/smap.h | 4 +-- arch/x86/kernel/alternative.c | 6 ++-- arch/x86/kernel/entry_32.S | 2 +- arch/x86/lib/clear_page_64.S | 4 +-- arch/x86/lib/copy_page_64.S | 2 +- arch/x86/lib/copy_user_64.S | 4 +-- arch/x86/lib/memcpy_64.S | 4 +-- arch/x86/lib/memmove_64.S | 2 +- arch/x86/lib/memset_64.S | 4 +-- 12 files changed, 114 insertions(+), 48 deletions(-)
arch/x86/include/asm/alternative-asm.h +42 -1
@@ -18,12 +18,53 @@ .endm #endif -.macro altinstruction_entry orig alt feature orig_len alt_len +.macro altinstruction_entry orig alt feature orig_len alt_len pad_len .long \orig - . .long \alt - . .word \feature .byte \orig_len .byte \alt_len + .byte \pad_len +.endm + +.macro ALTERNATIVE oldinstr, newinstr, feature +140: + \oldinstr +141: + .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f,142b-141b + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr +144: + .popsection +.endm + +.macro ALTERNATIVE_2 oldinstr, newinstr1, feature1, newinstr2, feature2 +140: + \oldinstr +141: + .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 + .skip -(((145f-144f)-(144f-143f)-(141b-140b)) > 0) * ((145f-144f)-(144f-143f)-(141b-140b)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f,142b-141b + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f,142b-141b + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr1 +144: + \newinstr2 +145: + .popsection .endm #endif /* __ASSEMBLY__ */
arch/x86/include/asm/alternative.h +40 -25
104 lines not on this page · raw message
arch/x86/include/asm/cpufeature.h +16 -6
@@ -418,6 +418,7 @@ static __always_inline __pure bool __static_cpu_has(u16 bit) " .word %P0\n" /* 1: do replace */ " .byte 2b - 1b\n" /* source len */ " .byte 0\n" /* replacement len */ + " .byte 0\n" /* pad len */ ".previous\n" /* skipping size check since replacement size = 0 */ : : "i" (X86_FEATURE_ALWAYS) : : t_warn);
@@ -432,6 +433,7 @@ static __always_inline __pure bool __static_cpu_has(u16 bit) " .word %P0\n" /* feature bit */ " .byte 2b - 1b\n" /* source len */ " .byte 0\n" /* replacement len */ + " .byte 0\n" /* pad len */ ".previous\n" /* skipping size check since replacement size = 0 */ : : "i" (bit) : : t_no);
@@ -457,6 +459,7 @@ static __always_inline __pure bool __static_cpu_has(u16 bit)" .word %P1\n" /* feature bit */ " .byte 2b - 1b\n" /* source len */ " .byte 4f - 3f\n" /* replacement len */ +" .byte 0\n" /* pad len */ ".previous\n" ".section .discard,\"aw\",@progbits\n" " .byte 0xff + (4f-3f) - (2b-1b)\n" /* size check */
@@ -491,23 +494,28 @@ static __always_inline __pure bool _static_cpu_has_safe(u16 bit)*/ asm_volatile_goto ("1: .byte 0xe9\n .long %l[t_dynamic] - 2f\n" "2:\n" +".skip -(((5f-4f) - (2b-1b)) > 0) * " +"((5f-4f) - (2b-1b)),0x90\n" +"3:\n" ".section .altinstructions,\"a\"\n" " .long 1b - .\n" /* src offset */ -" .long 3f - .\n" /* repl offset */ +" .long 4f - .\n" /* repl offset */ " .word %P1\n" /* always replace */ -" .byte 2b - 1b\n" /* src len */ -" .byte 4f - 3f\n" /* repl len */ +" .byte 3b - 1b\n" /* src len */ +" .byte 5f - 4f\n" /* repl len */ +" .byte 3b - 2b\n" /* pad len */ ".previous\n" ".section .altinstr_replacement,\"ax\"\n" -"3: .byte 0xe9\n .long %l[t_no] - 2b\n" -"4:\n" +"4: .byte 0xe9\n .long %l[t_no] - 2b\n" +"5:\n" ".previous\n" ".section .altinstructions,\"a\"\n" " .long 1b - .\n" /* src offset */ " .long 0\n" /* no replacement */ " .word %P0\n" /* feature bit */ -" .byte 2b - 1b\n" /* src len */ +" .byte 3b - 1b\n" /* src len */ " .byte 0\n" /* repl len */ +" .byte 0\n" /* pad len */ ".previous\n" : : "i" (bit), "i" (X86_FEATURE_ALWAYS) : : t_dynamic, t_no);
@@ -527,6 +535,7 @@ static __always_inline __pure bool _static_cpu_has_safe(u16 bit)" .word %P2\n" /* always replace */ " .byte 2b - 1b\n" /* source len */ " .byte 4f - 3f\n" /* replacement len */ +" .byte 0\n" /* pad len */ ".previous\n" ".section .discard,\"aw\",@progbits\n" " .byte 0xff + (4f-3f) - (2b-1b)\n" /* size check */
@@ -541,6 +550,7 @@ static __always_inline __pure bool _static_cpu_has_safe(u16 bit)" .word %P1\n" /* feature bit */ " .byte 4b - 3b\n" /* src len */ " .byte 6f - 5f\n" /* repl len */ +" .byte 0\n" /* pad len */ ".previous\n" ".section .discard,\"aw\",@progbits\n" " .byte 0xff + (6f-5f) - (4b-3b)\n" /* size check */
arch/x86/include/asm/smap.h +2 -2
@@ -33,7 +33,7 @@ 662: __ASM_CLAC ; \ .popsection ; \ .pushsection .altinstructions, "a" ; \ - altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3 ; \ + altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3, 0 ; \ .popsection #define ASM_STAC \
@@ -42,7 +42,7 @@ 662: __ASM_STAC ; \ .popsection ; \ .pushsection .altinstructions, "a" ; \ - altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3 ; \ + altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3, 0 ; \ .popsection #else /* CONFIG_X86_SMAP */
arch/x86/kernel/alternative.c +3 -3
@@ -270,7 +270,6 @@ void __init_or_module apply_alternatives(struct alt_instr *start,for (a = start; a < end; a++) { instr = (u8 *)&a->instr_offset + a->instr_offset; replacement = (u8 *)&a->repl_offset + a->repl_offset; -BUG_ON (a->replacementlen > a->instrlen);BUG_ON (a->instrlen >sizeof (insnbuf));BUG_ON (a->cpuid >= (NCAPINTS +NBUGINTS ) *32 );if (!boot_cpu_has (a->cpuid))
@@ -290,8 +289,9 @@ void __init_or_module apply_alternatives(struct alt_instr *start,DPRINTK ("Fix CALL offset: 0x%x" , *(s32 *)(insnbuf +1 )); } -add_nops (insnbuf + a->replacementlen, - a->instrlen - a->replacementlen); +if (a->instrlen > a->replacementlen) +add_nops (insnbuf + a->replacementlen, + a->instrlen - a->replacementlen);text_poke_early (instr, insnbuf, a->instrlen); }
arch/x86/kernel/entry_32.S +1 -1
@@ -819,7 +819,7 @@ ENTRY(simd_coprocessor_error) 661: pushl_cfi $do_general_protection 662: .section .altinstructions,"a" - altinstruction_entry 661b, 663f, X86_FEATURE_XMM, 662b-661b, 664f-663f + altinstruction_entry 661b, 663f, X86_FEATURE_XMM, 662b-661b, 664f-663f, 0 .previous .section .altinstr_replacement,"ax" 663: pushl $do_simd_coprocessor_error
arch/x86/lib/clear_page_64.S +2 -2
@@ -67,7 +67,7 @@ ENDPROC(clear_page) .previous .section .altinstructions,"a" altinstruction_entry clear_page,1b,X86_FEATURE_REP_GOOD,\ - .Lclear_page_end-clear_page, 2b-1b + .Lclear_page_end-clear_page, 2b-1b, 0 altinstruction_entry clear_page,2b,X86_FEATURE_ERMS, \ - .Lclear_page_end-clear_page,3b-2b + .Lclear_page_end-clear_page,3b-2b, 0 .previous
arch/x86/lib/copy_page_64.S +1 -1
@@ -106,5 +106,5 @@ ENDPROC(copy_page) .previous .section .altinstructions,"a" altinstruction_entry copy_page, 1b, X86_FEATURE_REP_GOOD, \ - .Lcopy_page_end-copy_page, 2b-1b + .Lcopy_page_end-copy_page, 2b-1b, 0 .previous
arch/x86/lib/copy_user_64.S +2 -2
@@ -36,8 +36,8 @@ .previous .section .altinstructions,"a" - altinstruction_entry 0b,2b,\feature1,5,5 - altinstruction_entry 0b,3b,\feature2,5,5 + altinstruction_entry 0b,2b,\feature1,5,5,0 + altinstruction_entry 0b,3b,\feature2,5,5,0 .previous .endm
arch/x86/lib/memcpy_64.S +2 -2
@@ -202,7 +202,7 @@ ENDPROC(__memcpy) */ .section .altinstructions, "a" altinstruction_entry __memcpy,.Lmemcpy_c,X86_FEATURE_REP_GOOD,\ - .Lmemcpy_e-.Lmemcpy_c,.Lmemcpy_e-.Lmemcpy_c + .Lmemcpy_e-.Lmemcpy_c,.Lmemcpy_e-.Lmemcpy_c,0 altinstruction_entry __memcpy,.Lmemcpy_c_e,X86_FEATURE_ERMS, \ - .Lmemcpy_e_e-.Lmemcpy_c_e,.Lmemcpy_e_e-.Lmemcpy_c_e + .Lmemcpy_e_e-.Lmemcpy_c_e,.Lmemcpy_e_e-.Lmemcpy_c_e,0 .previous
arch/x86/lib/memmove_64.S +1 -1
@@ -221,7 +221,7 @@ ENTRY(__memmove) altinstruction_entry .Lmemmove_begin_forward, \ .Lmemmove_begin_forward_efs,X86_FEATURE_ERMS, \ .Lmemmove_end_forward-.Lmemmove_begin_forward, \ - .Lmemmove_end_forward_efs-.Lmemmove_begin_forward_efs + .Lmemmove_end_forward_efs-.Lmemmove_begin_forward_efs,0 .previous ENDPROC(__memmove) ENDPROC(memmove)
arch/x86/lib/memset_64.S +2 -2
@@ -150,7 +150,7 @@ ENDPROC(__memset) */ .section .altinstructions,"a" altinstruction_entry __memset,.Lmemset_c,X86_FEATURE_REP_GOOD,\ - .Lfinal-__memset,.Lmemset_e-.Lmemset_c + .Lfinal-__memset,.Lmemset_e-.Lmemset_c,0 altinstruction_entry __memset,.Lmemset_c_e,X86_FEATURE_ERMS, \ - .Lfinal-__memset,.Lmemset_e_e-.Lmemset_c_e + .Lfinal-__memset,.Lmemset_e_e-.Lmemset_c_e,0 .previous
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 4/15 x86/alternatives: Make JMPs more robust
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Up until now we had to pay attention to relative JMPs in alternatives about how their relative offset gets computed so that the jump target is still correct. Or, as it is the case for near CALLs (opcode e8), we still have to go and readjust the offset at patching time.
What is more, the static_cpu_has_safe() facility had to forcefully generate 5-byte JMPs since we couldn't rely on the compiler to generate properly sized ones so we had to force the longest ones. Worse than that, sometimes it would generate a replacement JMP which is longer than the original one, thus overwriting the beginning of the next instruction at patching time.
So, in order to alleviate all that and make using JMPs more straight-forward we go and pad the original instruction in an alternative block with NOPs at build time, should the replacement(s) be longer. This way, alternatives users shouldn't pay special attention so that original and replacement instruction sizes are fine but the assembler would simply add padding where needed and not do anything otherwise.
As a second aspect, we go and recompute JMPs at patching time so that we can try to make 5-byte JMPs into two-byte ones if possible. If not, we still have to recompute the offsets as the replacement JMP gets put far away in the .altinstr_replacement section leading to a wrong offset if copied verbatim.
For example, on a locally generated kernel image
old insn VA: 0xffffffff810014bd, CPU feat: X86_FEATURE_ALWAYS, size: 2 __switch_to:
ffffffff810014bd: eb 21 jmp ffffffff810014e0
repl insn: size: 5
ffffffff81d0b23c: e9 b1 62 2f ff jmpq ffffffff810014f2
gets corrected to a 2-byte JMP:
apply_alternatives: feat: 3*32+21, old: (ffffffff810014bd, len: 2), repl: (ffffffff81d0b23c, len: 5)
alt_insn: e9 b1 62 2f ff
recompute_jumps: next_rip: ffffffff81d0b241, tgt_rip: ffffffff810014f2, new_displ: 0x00000033, ret len: 2 converted to: eb 33 90 90 90
and a 5-byte JMP:
old insn VA: 0xffffffff81001516, CPU feat: X86_FEATURE_ALWAYS, size: 2 __switch_to:
ffffffff81001516: eb 30 jmp ffffffff81001548
repl insn: size: 5
ffffffff81d0b241: e9 10 63 2f ff jmpq ffffffff81001556
gets shortened into a two-byte one:
apply_alternatives: feat: 3*32+21, old: (ffffffff81001516, len: 2), repl: (ffffffff81d0b241, len: 5)
alt_insn: e9 10 63 2f ff
recompute_jumps: next_rip: ffffffff81d0b246, tgt_rip: ffffffff81001556, new_displ: 0x0000003e, ret len: 2 converted to: eb 3e 90 90 90
... and so on.
This leads to a net win of around
40ish replacements * 3 bytes savings =~ 120 bytes of I$
on an AMD guest which means some savings of precious instruction cache bandwidth. The padding to the shorter 2-byte JMPs are single-byte NOPs which on smart microarchitectures means discarding NOPs at decode time and thus freeing up execution bandwidth.
--- arch/x86/include/asm/cpufeature.h | 10 +--- arch/x86/kernel/alternative.c | 103 ++++++++++++++++++++++++++++++++++++-- arch/x86/lib/copy_user_64.S | 11 ++-- 3 files changed, 105 insertions(+), 19 deletions(-)
arch/x86/include/asm/cpufeature.h +2 -8
@@ -486,13 +486,7 @@ static __always_inline __pure bool __static_cpu_has(u16 bit) static __always_inline __pure bool _static_cpu_has_safe(u16 bit) {#ifdef CC_HAVE_ASM_GOTO -/* -* We need to spell the jumps to the compiler because, depending on the offset, -* the replacement jump can be bigger than the original jump, and this we cannot -* have. Thus, we force the jump to the widest, 4-byte, signed relative -* offset even though the last would often fit in less bytes. -*/ -asm_volatile_goto ("1: .byte 0xe9\n .long %l[t_dynamic] - 2f\n" +asm_volatile_goto ("1: jmp %l[t_dynamic]\n" "2:\n" ".skip -(((5f-4f) - (2b-1b)) > 0) * " "((5f-4f) - (2b-1b)),0x90\n"
@@ -506,7 +500,7 @@ static __always_inline __pure bool _static_cpu_has_safe(u16 bit)" .byte 3b - 2b\n" /* pad len */ ".previous\n" ".section .altinstr_replacement,\"ax\"\n" -"4: .byte 0xe9\n .long %l[t_no] - 2b\n" +"4: jmp %l[t_no]\n" "5:\n" ".previous\n" ".section .altinstructions,\"a\"\n"
arch/x86/kernel/alternative.c +98 -5
147 lines not on this page · raw message
arch/x86/lib/copy_user_64.S +5 -6
@@ -25,14 +25,13 @@ */ .macro ALTERNATIVE_JUMP feature1,feature2,orig,alt1,alt2 0: - .byte 0xe9 /* 32bit jump */ - .long \orig-1f /* by default jump to orig */ + jmp \orig 1: .section .altinstr_replacement,"ax" -2: .byte 0xe9 /* near jump with 32bit immediate */ - .long \alt1-1b /* offset */ /* or alternatively to alt1 */ -3: .byte 0xe9 /* near jump with 32bit immediate */ - .long \alt2-1b /* offset */ /* or alternatively to alt2 */ +2: + jmp \alt1 +3: + jmp \alt2 .previous .section .altinstructions,"a"
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 5/15 x86/alternatives: Use optimized NOPs for padding
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Alternatives allow now for an empty old instruction. In this case we go and pad the space with NOPs at assembly time. However, there are the optimal, longer NOPs which should be used. Do that at patching time by adding alt_instr.padlen-sized NOPs at the old instruction address.
--- arch/x86/kernel/alternative.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/x86/kernel/alternative.c +13 -1
@@ -323,6 +323,14 @@ done: n_dspl, (unsigned long)orig_insn + n_dspl + repl_len); } +static void __init_or_module optimize_nops(struct alt_instr *a, u8 *instr) +{ +add_nops (instr + (a->instrlen - a->padlen), a->padlen); + +DUMP_BYTES (instr, a->instrlen,"%p: [%d:%d) optimized NOPs: " , + instr, a->instrlen - a->padlen, a->padlen); +} +/* * Replace instructions with better alternatives for this CPU type. This runs * before SMP is initialized to avoid SMP problems with self modifying code.
@@ -354,8 +362,12 @@ void __init_or_module apply_alternatives(struct alt_instr *start, replacement = (u8 *)&a->repl_offset + a->repl_offset;BUG_ON (a->instrlen >sizeof (insnbuf));BUG_ON (a->cpuid >= (NCAPINTS +NBUGINTS ) *32 ); -if (!boot_cpu_has (a->cpuid)) +if (!boot_cpu_has (a->cpuid)) { +if (a->padlen >1 ) +optimize_nops (a, instr); +continue ; + } DPRINTK("feat: %d*32+%d, old: (%p, len: %d), repl: (%p, len: %d)", a->cpuid >> 5,
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH v2 05/15] x86/alternatives: Use optimized NOPs for padding
Quoted 18 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Alternatives allow now for an empty old instruction. In this case we go and pad the space with NOPs at assembly time. However, there are the optimal, longer NOPs which should be used. Do that at patching time by adding alt_instr.padlen-sized NOPs at the old instruction address.
Cc Andy Lutomirski <[email protected]>Signed-off-by Borislav Petkov <[email protected]>--- arch/x86/kernel/alternative.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)arch/x86/kernel/alternative.c +0
-0
arch/x86/kernel/alternative.c done: · line 328
Quoted 6 lines
arch/x86/kernel/alternative.c +3
-0@@ -323,6 +323,14 @@ done: n_dspl, (unsigned long)orig_insn + n_dspl + repl_len); } +static void __init_or_module optimize_nops(struct alt_instr *a, u8 *instr) +{ +add_nops (instr + (a->instrlen - a->padlen), a->padlen);
So while looking at this patch I was wondering about the following
question: right now add_nops() does the obvious 'fill with large NOPs first, then fill the remaining bytes with a smaller NOP' logic:
/* Use this to add nops to a buffer, then text_poke the whole buffer. */ static void __init_or_module add_nops(void *insns, unsigned int len)
{
while (len > 0) {
unsigned int noplen = len;
if (noplen > ASM_NOP_MAX)
noplen = ASM_NOP_MAX;
memcpy(insns, ideal_nops[noplen], noplen);
insns += noplen;
len -= noplen;
}
}this works perfectly fine, but I'm wondering how current decoders work when a large NOP crosses a cache line boundary or a page boundary. Is there any inefficiency in that case, and if yes, could we avoid that by not spilling NOPs across cachelines or page boundaries?
With potentially thousands of patched instructions both situations are bound to occur dozens of times in the cacheline case, and a few times in the page boundary case.
There's also the following special case, of a large NOP followed by a small NOP, when the number of NOPs would not change if we padded differently:
[ large NOP ][smaller NOP]
[ cacheline 1 ][ cacheline 2 ]
which might be more optimally filled with two mid-size NOPs:
[ midsize NOP ][ midsize NOP ]
[ cacheline 1 ][ cacheline 2 ]
So that any special boundary is not partially covered by a NOP instruction.
But the main question is, do such alignment details ever matter to decoder performance?
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 05/15] x86/alternatives: Use optimized NOPs for padding
Quoted 2 lines · Ingo Molnar
But the main question is, do such alignment details ever matter to decoder performance?
Right, so I have some ideas about it but this all is probably very uarch-specific so below is probably only speculation.
So both optimization manuals suggest using longer NOPs is better probably because the less micro-ops you decode from the NOPs, the less instructions you have in the pipe, less resources, etc etc. Thus, making them longer with prefixes is better than having multiple unprefixed and shorter nops.
Now, there's the dependency on operands and NOPs generally reference rAX. On some uarches this dependency is broken much earlier so the micro-op gets scheduled earlier. Thus freeing resources earlier, etc, etc.
If the NOP is crossing cacheline, you obviously can't know it is a NOP yet so you have to fetch the next cacheline to finish decoding it. So in the best case you'll go down to L1 and in the worst case go to memory.
And on modern machines this is probably so very unnoticeable. I hardly doubt we'll see that in a hotpath even. But it could be a good measuring exercise :)
Btw, when we pad JMPs with NOPs, we shouldn't be affected because the unconditional JMP will not have us decode the NOPs behind it. And modern, hungry prefetching beasts would've probably fetched and decoded a bunch of instructions along with the NOPs. But they should see the JMP and stop prefetching after it though. Who knows, uarch stuff.
See, all speculations :)
Btw #2 and more importantly: this patchset of mine doesn't necessarily enlarge the patch sites - before it, you'll have to add proper-sized NOPs explicitly and now we let the toolchain do it for us, which sometimes even leads to smaller NOPs depending on the instruction the toolchain generates.
With the JMP optimizations, the instructions become smaller too.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Borislav Petkov
maintainerPATCH v2 6/15 x86/lib/copy_page_64.S: Use generic ALTERNATIVE macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
... instead of the semi-version with the spelled out sections.
What is more, make the REP_GOOD version be the default copy_page() version as the majority of the relevant x86 CPUs do set X86_FEATURE_REP_GOOD. Thus, copy_page gets compiled to:
ffffffff8130af80 <copy_page>:
ffffffff8130af80: e9 0b 00 00 00 jmpq ffffffff8130af90 <copy_page_regs> ffffffff8130af85: b9 00 02 00 00 mov $0x200,%ecx ffffffff8130af8a: f3 48 a5 rep movsq %ds:(%rsi),%es:(%rdi) ffffffff8130af8d: c3 retq ffffffff8130af8e: 66 90 xchg %ax,%ax
ffffffff8130af90 <copy_page_regs>:
...
and after the alternatives have run, the JMP to the old, unrolled version gets NOPed out:
ffffffff8130af80 <copy_page>:
ffffffff8130af80: 66 66 90 xchg %ax,%ax ffffffff8130af83: 66 90 xchg %ax,%ax ffffffff8130af85: b9 00 02 00 00 mov $0x200,%ecx ffffffff8130af8a: f3 48 a5 rep movsq %ds:(%rsi),%es:(%rdi) ffffffff8130af8d: c3 retq
On modern uarches, those NOPs are cheaper than the unconditional JMP previously.
--- arch/x86/lib/copy_page_64.S | 37 ++++++++++++------------------------- 1 file changed, 12 insertions(+), 25 deletions(-)
arch/x86/lib/copy_page_64.S +12 -25
@@ -2,23 +2,26 @@ #include <linux/linkage.h> #include <asm/dwarf2.h> +#include <asm/cpufeature.h> #include <asm/alternative-asm.h> +/* + * Some CPUs run faster using the string copy instructions (sane microcode). + * It is also a lot simpler. Use this when possible. But, don't use streaming + * copy unless the CPU indicates X86_FEATURE_REP_GOOD. Could vary the + * prefetch distance based on SMP/UP. + */ ALIGN -copy_page_rep: +ENTRY(copy_page) CFI_STARTPROC + ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD movl $4096/8, %ecx rep movsq ret CFI_ENDPROC -ENDPROC(copy_page_rep) - -/* - * Don't use streaming copy unless the CPU indicates X86_FEATURE_REP_GOOD. - * Could vary the prefetch distance based on SMP/UP. -*/ +ENDPROC(copy_page) -ENTRY(copy_page) +ENTRY(copy_page_regs) CFI_STARTPROC subq $2*8, %rsp CFI_ADJUST_CFA_OFFSET 2*8
@@ -90,21 +93,5 @@ ENTRY(copy_page) addq $2*8, %rsp CFI_ADJUST_CFA_OFFSET -2*8 ret -.Lcopy_page_end: CFI_ENDPROC -ENDPROC(copy_page) - - /* Some CPUs run faster using the string copy instructions. - It is also a lot simpler. Use this when possible */ - -#include <asm/cpufeature.h> - - .section .altinstr_replacement,"ax" -1: .byte 0xeb /* jmp <disp8> */ - .byte (copy_page_rep - copy_page) - (2f - 1b) /* offset */ -2: - .previous - .section .altinstructions,"a" - altinstruction_entry copy_page, 1b, X86_FEATURE_REP_GOOD, \ - .Lcopy_page_end-copy_page, 2b-1b, 0 - .previous +ENDPROC(copy_page_regs)
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 7/15 x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Use the asm macro and drop the locally grown version.
--- arch/x86/lib/copy_user_64.S | 40 ++++++++++------------------------------ 1 file changed, 10 insertions(+), 30 deletions(-)
arch/x86/lib/copy_user_64.S +10 -30
@@ -16,30 +16,6 @@ #include <asm/asm.h> #include <asm/smap.h> -/* - * By placing feature2 after feature1 in altinstructions section, we logically - * implement: - * If CPU has feature2, jmp to alt2 is used - * else if CPU has feature1, jmp to alt1 is used - * else jmp to orig is used. - */ - .macro ALTERNATIVE_JUMP feature1,feature2,orig,alt1,alt2 -0: - jmp \orig -1: - .section .altinstr_replacement,"ax" -2: - jmp \alt1 -3: - jmp \alt2 - .previous - - .section .altinstructions,"a" - altinstruction_entry 0b,2b,\feature1,5,5,0 - altinstruction_entry 0b,3b,\feature2,5,5,0 - .previous - .endm - .macro ALIGN_DESTINATION /* check for bad alignment of destination */ movl %edi,%ecx
@@ -73,9 +49,11 @@ ENTRY(_copy_to_user) jc bad_to_user cmpq TI_addr_limit(%rax),%rcx ja bad_to_user - ALTERNATIVE_JUMP X86_FEATURE_REP_GOOD,X86_FEATURE_ERMS, \ - copy_user_generic_unrolled,copy_user_generic_string, \ - copy_user_enhanced_fast_string + ALTERNATIVE_2 "jmp copy_user_generic_unrolled", \ + "jmp copy_user_generic_string", \ + X86_FEATURE_REP_GOOD, \ + "jmp copy_user_enhanced_fast_string", \ + X86_FEATURE_ERMS CFI_ENDPROC ENDPROC(_copy_to_user)
@@ -88,9 +66,11 @@ ENTRY(_copy_from_user) jc bad_from_user cmpq TI_addr_limit(%rax),%rcx ja bad_from_user - ALTERNATIVE_JUMP X86_FEATURE_REP_GOOD,X86_FEATURE_ERMS, \ - copy_user_generic_unrolled,copy_user_generic_string, \ - copy_user_enhanced_fast_string + ALTERNATIVE_2 "jmp copy_user_generic_unrolled", \ + "jmp copy_user_generic_string", \ + X86_FEATURE_REP_GOOD, \ + "jmp copy_user_enhanced_fast_string", \ + X86_FEATURE_ERMS CFI_ENDPROC ENDPROC(_copy_from_user)
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
arch/x86/lib/copy_user_64.S ENTRY(_copy_to_user) · line 56
Quoted 13 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Use the asm macro and drop the locally grown version.
arch/x86/lib/copy_user_64.S +5
-0@@ -73,9 +49,11 @@ ENTRY(_copy_to_user) jc bad_to_user cmpq TI_addr_limit(%rax),%rcx ja bad_to_user + ALTERNATIVE_2 "jmp copy_user_generic_unrolled", \ + "jmp copy_user_generic_string", \ + X86_FEATURE_REP_GOOD, \ + "jmp copy_user_enhanced_fast_string", \ + X86_FEATURE_ERMS
Btw., as a future optimization, wouldn't it be useful to patch this function at its first instruction, i.e. to have three fully functional copy_user_generic_ variants and choose to jmp to one of them in the first instruction of the original function?
The advantage would be two-fold:
1) right now: smart microarchitectures that are able to optimize jump-after-jump (and jump-after-call) targets in their branch target cache can do so in this case, reducing the overhead of the patching, possibly close to zero in the cached case.
2) in the future: we could actually do a (limited) re-link of the kernel during bootup, and patch up the original copy_to_user call sites directly to one of the three variants. Alternatives patching done at the symbol level. Does current tooling allow something like this already?
Thanks,
Ingo
Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 22 lines · Ingo Molnar
arch/x86/lib/copy_user_64.S ENTRY(_copy_to_user) · line 56
Quoted 13 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Use the asm macro and drop the locally grown version.
arch/x86/lib/copy_user_64.S +5
-0@@ -73,9 +49,11 @@ ENTRY(_copy_to_user) jc bad_to_user cmpq TI_addr_limit(%rax),%rcx ja bad_to_user + ALTERNATIVE_2 "jmp copy_user_generic_unrolled", \ + "jmp copy_user_generic_string", \ + X86_FEATURE_REP_GOOD, \ + "jmp copy_user_enhanced_fast_string", \ + X86_FEATURE_ERMSBtw., as a future optimization, wouldn't it be useful to patch this function at its first instruction, i.e. to have three fully functional
copy_user_generic_variants and choose to jmp to one of them in the first instruction of the original function?
Quoted 6 lines
The advantage would be two-fold:
1) right now: smart microarchitectures that are able to optimize jump-after-jump (and jump-after-call) targets in their branch target cache can do so in this case, reducing the overhead of the patching, possibly close to zero in the cached case.
Btw., the x86 memset() variants are using this today, and I think this is the most optimal jump-patching variant, even if it means a small amount of code duplication between the copy_user variants.
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 3 lines · Ingo Molnar
Btw., the x86
memset()variants are using this today, and I think this is the most optimal jump-patching variant, even if it means a small amount of code duplication between thecopy_uservariants.
Yeah, the problem with that one was that we patch a huge amount of code, see the dump below.
The X86_FEATURE_REP_GOOD thing replaces a 172 bytes memset with the 42 byte REP;STOSQ version and the X86_FEATURE_ERMS does the same with REP;STOSB.
The 172-42 bytes at the end got padded with 130 bytes worth of NOPs.
Now, I've changed those to do simple JMPs for now so that we're working with much less bytes at patching time (5 and if we're lucky 2). The next step would be to do what you suggest and simply CALL the respective variants at the call sites directly.
old insn VA: 0xffffffff812c4060, CPU feat: X86_FEATURE_REP_GOOD, size: 172 memset:
ffffffff812c4060 <memset>:
ffffffff812c4060: 49 89 fa mov %rdi,%r10 ffffffff812c4063: 40 0f b6 ce movzx %sil,%ecx ffffffff812c4067: 48 b8 01 01 01 01 01 mov $0x101010101010101,%rax ffffffff812c406e: 01 01 01 ffffffff812c4071: 48 0f af c1 imul %rcx,%rax ffffffff812c4075: 41 89 f9 mov %edi,%r9d ffffffff812c4078: 41 83 e1 07 and $0x7,%r9d ffffffff812c407c: 75 70 jne ffffffff812c40ee ffffffff812c407e: 48 89 d1 mov %rdx,%rcx ffffffff812c4081: 48 c1 e9 06 shr $0x6,%rcx ffffffff812c4085: 74 39 jz ffffffff812c40c0 ffffffff812c4087: 66 0f 1f 84 00 00 00 nop 0x0(%rax,%rax,1) ffffffff812c408e: 00 00 ffffffff812c4090: 48 ff c9 dec %rcx ffffffff812c4093: 48 89 07 mov %rax,(%rdi) ffffffff812c4096: 48 89 47 08 mov %rax,0x8(%rdi) ffffffff812c409a: 48 89 47 10 mov %rax,0x10(%rdi) ffffffff812c409e: 48 89 47 18 mov %rax,0x18(%rdi) ffffffff812c40a2: 48 89 47 20 mov %rax,0x20(%rdi) ffffffff812c40a6: 48 89 47 28 mov %rax,0x28(%rdi) ffffffff812c40aa: 48 89 47 30 mov %rax,0x30(%rdi) ffffffff812c40ae: 48 89 47 38 mov %rax,0x38(%rdi) ffffffff812c40b2: 48 8d 7f 40 lea 0x40(%rdi),%rdi ffffffff812c40b6: 75 d8 jne ffffffff812c4090 ffffffff812c40b8: 0f 1f 84 00 00 00 00 nop 0x0(%rax,%rax,1) ffffffff812c40bf: 00 ffffffff812c40c0: 89 d1 mov %edx,%ecx ffffffff812c40c2: 83 e1 38 and $0x38,%ecx ffffffff812c40c5: 74 14 jz ffffffff812c40db ffffffff812c40c7: c1 e9 03 shr $0x3,%ecx ffffffff812c40ca: 66 0f 1f 44 00 00 nop 0x0(%rax,%rax,1) ffffffff812c40d0: ff c9 dec %ecx ffffffff812c40d2: 48 89 07 mov %rax,(%rdi) ffffffff812c40d5: 48 8d 7f 08 lea 0x8(%rdi),%rdi ffffffff812c40d9: 75 f5 jne ffffffff812c40d0 ffffffff812c40db: 83 e2 07 and $0x7,%edx ffffffff812c40de: 74 0a jz ffffffff812c40ea ffffffff812c40e0: ff ca dec %edx ffffffff812c40e2: 88 07 mov %al,(%rdi) ffffffff812c40e4: 48 8d 7f 01 lea 0x1(%rdi),%rdi ffffffff812c40e8: 75 f6 jne ffffffff812c40e0 ffffffff812c40ea: 4c 89 d0 mov %r10,%rax ffffffff812c40ed: c3 retq ffffffff812c40ee: 48 83 fa 07 cmp $0x7,%rdx ffffffff812c40f2: 76 e7 jbe ffffffff812c40db ffffffff812c40f4: 48 89 07 mov %rax,(%rdi) ffffffff812c40f7: 49 c7 c0 08 00 00 00 mov $0x8,%r8 ffffffff812c40fe: 4d 29 c8 sub %r9,%r8 ffffffff812c4101: 4c 01 c7 add %r8,%rdi ffffffff812c4104: 4c 29 c2 sub %r8,%rdx ffffffff812c4107: e9 72 ff ff ff jmpq ffffffff812c407e repl insn: 0xffffffff81e1d68d, size: 42 ffffffff81e1d68d: 49 89 f9 mov %rdi,%r9 ffffffff81e1d690: 48 89 d1 mov %rdx,%rcx ffffffff81e1d693: 83 e2 07 and $0x7,%edx ffffffff81e1d696: 48 c1 e9 03 shr $0x3,%rcx ffffffff81e1d69a: 40 0f b6 f6 movzx %sil,%esi ffffffff81e1d69e: 48 b8 01 01 01 01 01 mov $0x101010101010101,%rax ffffffff81e1d6a5: 01 01 01 ffffffff81e1d6a8: 48 0f af c6 imul %rsi,%rax ffffffff81e1d6ac: f3 48 ab rep stos %rax,%es:(%rdi) ffffffff81e1d6af: 89 d1 mov %edx,%ecx ffffffff81e1d6b1: f3 aa rep stos %al,%es:(%rdi) ffffffff81e1d6b3: 4c 89 c8 mov %r9,%rax ffffffff81e1d6b6: c3 retq
old insn VA: 0xffffffff812c4060, CPU feat: X86_FEATURE_ERMS, size: 172 memset:
ffffffff812c4060 <memset>:
ffffffff812c4060: 49 89 fa mov %rdi,%r10 ffffffff812c4063: 40 0f b6 ce movzx %sil,%ecx ffffffff812c4067: 48 b8 01 01 01 01 01 mov $0x101010101010101,%rax ffffffff812c406e: 01 01 01 ffffffff812c4071: 48 0f af c1 imul %rcx,%rax ffffffff812c4075: 41 89 f9 mov %edi,%r9d ffffffff812c4078: 41 83 e1 07 and $0x7,%r9d ffffffff812c407c: 75 70 jne ffffffff812c40ee ffffffff812c407e: 48 89 d1 mov %rdx,%rcx ffffffff812c4081: 48 c1 e9 06 shr $0x6,%rcx ffffffff812c4085: 74 39 jz ffffffff812c40c0 ffffffff812c4087: 66 0f 1f 84 00 00 00 nop 0x0(%rax,%rax,1) ffffffff812c408e: 00 00 ffffffff812c4090: 48 ff c9 dec %rcx ffffffff812c4093: 48 89 07 mov %rax,(%rdi) ffffffff812c4096: 48 89 47 08 mov %rax,0x8(%rdi) ffffffff812c409a: 48 89 47 10 mov %rax,0x10(%rdi) ffffffff812c409e: 48 89 47 18 mov %rax,0x18(%rdi) ffffffff812c40a2: 48 89 47 20 mov %rax,0x20(%rdi) ffffffff812c40a6: 48 89 47 28 mov %rax,0x28(%rdi) ffffffff812c40aa: 48 89 47 30 mov %rax,0x30(%rdi) ffffffff812c40ae: 48 89 47 38 mov %rax,0x38(%rdi) ffffffff812c40b2: 48 8d 7f 40 lea 0x40(%rdi),%rdi ffffffff812c40b6: 75 d8 jne ffffffff812c4090 ffffffff812c40b8: 0f 1f 84 00 00 00 00 nop 0x0(%rax,%rax,1) ffffffff812c40bf: 00 ffffffff812c40c0: 89 d1 mov %edx,%ecx ffffffff812c40c2: 83 e1 38 and $0x38,%ecx ffffffff812c40c5: 74 14 jz ffffffff812c40db ffffffff812c40c7: c1 e9 03 shr $0x3,%ecx ffffffff812c40ca: 66 0f 1f 44 00 00 nop 0x0(%rax,%rax,1) ffffffff812c40d0: ff c9 dec %ecx ffffffff812c40d2: 48 89 07 mov %rax,(%rdi) ffffffff812c40d5: 48 8d 7f 08 lea 0x8(%rdi),%rdi ffffffff812c40d9: 75 f5 jne ffffffff812c40d0 ffffffff812c40db: 83 e2 07 and $0x7,%edx ffffffff812c40de: 74 0a jz ffffffff812c40ea ffffffff812c40e0: ff ca dec %edx ffffffff812c40e2: 88 07 mov %al,(%rdi) ffffffff812c40e4: 48 8d 7f 01 lea 0x1(%rdi),%rdi ffffffff812c40e8: 75 f6 jne ffffffff812c40e0 ffffffff812c40ea: 4c 89 d0 mov %r10,%rax ffffffff812c40ed: c3 retq ffffffff812c40ee: 48 83 fa 07 cmp $0x7,%rdx ffffffff812c40f2: 76 e7 jbe ffffffff812c40db ffffffff812c40f4: 48 89 07 mov %rax,(%rdi) ffffffff812c40f7: 49 c7 c0 08 00 00 00 mov $0x8,%r8 ffffffff812c40fe: 4d 29 c8 sub %r9,%r8 ffffffff812c4101: 4c 01 c7 add %r8,%rdi ffffffff812c4104: 4c 29 c2 sub %r8,%rdx ffffffff812c4107: e9 72 ff ff ff jmpq ffffffff812c407e repl insn: 0xffffffff81e1d6b7, size: 15 ffffffff81e1d6b7: 49 89 f9 mov %rdi,%r9 ffffffff81e1d6ba: 40 88 f0 mov %sil,%al ffffffff81e1d6bd: 48 89 d1 mov %rdx,%rcx ffffffff81e1d6c0: f3 aa rep stos %al,%es:(%rdi) ffffffff81e1d6c2: 4c 89 c8 mov %r9,%rax ffffffff81e1d6c5: c3 retq
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 11 lines · Borislav Petkov
Quoted 3 lines · Ingo Molnar
Btw., the x86
memset()variants are using this today, and I think this is the most optimal jump-patching variant, even if it means a small amount of code duplication between thecopy_uservariants.Yeah, the problem with that one was that we patch a huge amount of code, see the dump below.
The
X86_FEATURE_REP_GOODthing replaces a 172 bytes memset with the 42 byte REP;STOSQ version and theX86_FEATURE_ERMSdoes the same with REP;STOSB.
So I'd not patch in the large _orig variant, it's legacy mostly - but the two fast variants?
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 2 lines · Ingo Molnar
So I'd not patch in the large
_origvariant, it's legacy mostly - but the two fast variants?
Yes, so the optimal thing would be to have the kernel generate the CALL <REP_GOOD> variant when building. Those are going to be the default opcodes in vmlinux coming out of build.
Then, when booted on ERMS machines, we'll patch in CALL <ERMS version> and as a last resort fallback, the CALL to the legacy variant on really old boxes.
/me puts it on the TODO.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Borislav Petkov
maintainerRe: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
In reply to Ingo MolnarQuoted 4 lines · Ingo Molnar
Btw., as a future optimization, wouldn't it be useful to patch this function at its first instruction, i.e. to have three fully functional
copy_user_generic_variants and choose to jmp to one of them in the first instruction of the original function?
Yeah, we already have callsites doing that:
old insn VA: 0xffffffff8100e8b3, CPU feat: X86_FEATURE_REP_GOOD, size: 5 xfpregs_get:
ffffffff8100e8b3: e8 f8 40 2b 00 callq ffffffff812c29b0
repl insn: 0xffffffff81e1c255, size: 5
ffffffff81e1c255: e8 16 68 4a ff callq ffffffff812c2a70 ffffffff8100e8b3: e8 16 68 4a ff callq ffffffff804b50ce
old insn VA: 0xffffffff8100e8b3, CPU feat: X86_FEATURE_ERMS, size: 5 xfpregs_get:
ffffffff8100e8b3: e8 f8 40 2b 00 callq ffffffff812c29b0
repl insn: 0xffffffff81e1c25a, size: 5
ffffffff81e1c25a: e8 51 68 4a ff callq ffffffff812c2ab0 ffffffff8100e8b3: e8 51 68 4a ff callq ffffffff804b5109
That's copy_user_generic() which does the alternative_call_2().
And yes, it is on the TODO list to optimize those other alternatives sites.
Quoted 6 lines
The advantage would be two-fold:
1) right now: smart microarchitectures that are able to optimize jump-after-jump (and jump-after-call) targets in their branch target cache can do so in this case, reducing the overhead of the patching, possibly close to zero in the cached case.
Yeah, it would still be better to simply do CALL and no unconditional JMPs in there later. But this is future work, one thing at a time :)
Quoted 5 lines
2) in the future: we could actually do a (limited) re-link of the kernel during bootup, and patch up the original
copy_to_usercall sites directly to one of the three variants. Alternatives patching done at the symbol level. Does current tooling allow something like this already?
Well, I have a patchset which uses relocs to patch vmlinux at build time. And that was the initial approach to this but you cannot know which features a CPU supports until boot time so you have to boot.
BUT(!), you can replace stuff like X86_FEATURE_ALWAYS at build time already (this is the static_cpu_has_safe() stuff). I'll look into that later and dust off my relocs pile.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 6 lines · Borislav Petkov
Quoted 5 lines
2) in the future: we could actually do a (limited) re-link of the kernel during bootup, and patch up the original
copy_to_usercall sites directly to one of the three variants. Alternatives patching done at the symbol level. Does current tooling allow something like this already?
Quoted 7 lines
Well, I have a patchset which uses relocs to patch vmlinux at build time. And that was the initial approach to this but you cannot know which features a CPU supports until boot time so you have to boot.
BUT(!), you can replace stuff like
X86_FEATURE_ALWAYSat build time already (this is thestatic_cpu_has_safe()stuff). I'll look into that later and dust off my relocs pile.
We could also do a (limited) relink during early bootup, as part of the alternatives patching pass in essence: for that we need to stick the relocation info into a section and put that into the vmlinux.
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 3 lines · Ingo Molnar
We could also do a (limited) relink during early bootup, as part of the alternatives patching pass in essence: for that we need to stick the relocation info into a section and put that into the vmlinux.
Oh, you want us to do our own asm inlining, so to speak, and save us even a CALL. Well, that would be more involved and saving a single CALL is making me question the effort.
Also, with shifted virtual addresses at boot time, debuggability becomes a serious pain because "objdump -D vmlinux" output is worthless all of a sudden.
For example, even during doing those patches, I had to go and dump kernel memory to see what actually gets patched in. And without kvm and the monitor console, I would've had a serious hard time. If you change larger portions of the kernel at early boot, that might make the whole endeavor of debugging a serious undertaking. I already dread the moment when I'd have to look at a kaslr'ed splat.
We'll see.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Ingo Molnar
We could also do a (limited) relink during early bootup, as part of the alternatives patching pass in essence: for that we need to stick the relocation info into a section and put that into the vmlinux.
Oh, you want us to do our own asm inlining, so to speak, and save us even a CALL. [...]
No, I didn't want to go that far - I just suggested to patch the CALL to the target function.
But it's an equivalent method to patch in the most popular full function variants via ALTERNATIVE_2().
Quoted 3 lines
Also, with shifted virtual addresses at boot time, debuggability becomes a serious pain because "objdump -D vmlinux" output is worthless all of a sudden.
Yeah, that way lies madness.
Quoted 6 lines
For example, even during doing those patches, I had to go and dump kernel memory to see what actually gets patched in. And without kvm and the monitor console, I would've had a serious hard time. If you change larger portions of the kernel at early boot, that might make the whole endeavor of debugging a serious undertaking. I already dread the moment when I'd have to look at a kaslr'ed splat.
Ha! There's a neat alternatives debugging trick with perf that you might not know about: if you run 'perf top' as root then perf will use /proc/kcore to disassemble the live kernel image and if you look at the assembly output of hot functions then you'll see the real, patched instructions on the live kernel, not the vmlinux instructions.
I have two enhancement suggestions to the perf tooling developers for this usecase:
1) We should perhaps add a 'perf top --all-kernel-symbols' kind of mode (with a better command name), which would make all kernel symbols visible in 'perf top' at startup, which could be searched and inspected freely - without having to first wait for them to show up in the profile?
2) it would be absolutely, totally nice to have a TUI mode where the raw address and raw instruction opcodes are output as well, like objdump does:
ffffffff818740e2: 90 nop ffffffff818740e3: 48 83 ec 78 sub $0x78,%rsp ffffffff818740e7: e8 64 03 00 00 callq ffffffff81874450 <save_paranoid> ffffffff818740ec: 48 89 e7 mov %rsp,%rdi ffffffff818740ef: 48 8b 74 24 78 mov 0x78(%rsp),%rsi ffffffff818740f4: 48 c7 44 24 78 ff ff movq $0xffffffffffffffff,0x78(%rsp)
The 'o' hotkey already shows the raw address - but the raw instruction opcodes are still hidden - would be nice to get them too!
This kind of disassembly/annotation mode could be called 'perf inspect' or 'perf disasm' or 'perf kernel-annotate'? It would be a perf.data-less mode of operation, i.e. like 'perf top'.
Thanks,
Ingo
Re: [PATCH v2 07/15] x86/lib/copy_user_64.S: Convert to ALTERNATIVE_2
Quoted 9 lines · Ingo Molnar
Ha! There's a neat alternatives debugging trick with perf that you might not know about: if you run 'perf top' as root then perf will use
/proc/kcoreto disassemble the live kernel image and if you look at the assembly output of hot functions then you'll see the real, patched instructions on the live kernel, not the vmlinux instructions.I have two enhancement suggestions to the perf tooling developers for this usecase:
and I've got a bugreport as well, when I use the kcore annotations on 'fput' symbol (i.e. running 'perf top' as root), go into the disassembly window and then try to exit that window, then I get this segfault:
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff3c41700 (LWP 29134)] lock__delete (ops=0x7fffec001818) at util/annotate.c:219
219 ins__delete(ops->locked.ops);
(gdb) bt
#0 lock__delete (ops=0x7fffec001818) at util/annotate.c:219
#1 0x000000000046e568 in disasm_line__free (dl=0x7fffec0017e0) at util/annotate.c:619
#2 0x00000000004f0743 in symbol__tui_annotate (sym=<optimized out>, map=<optimized out>, evsel=evsel@entry=0x83d770, hbt=hbt@entry=0x7ffff3c40e60)
at ui/browsers/annotate.c:976
#3 0x00000000004f0973 in hist_entry__tui_annotate (he=he@entry=0x151ba40, evsel=evsel@entry=0x83d770, hbt=hbt@entry=0x7ffff3c40e60) at ui/browsers/annotate.c:835
#4 0x00000000004f5685 in perf_evsel__hists_browse (evsel=0x83d770, nr_events=nr_events@entry=1,
helpline=helpline@entry=0x51d350 "For a higher level overview, try: perf top --sort comm,dso", left_exits=left_exits@entry=false, hbt=hbt@entry=0x7ffff3c40e60,
min_pcnt=<optimized out>, env=0x83de40) at ui/browsers/hists.c:1716
#5 0x00000000004f7ae4 in perf_evlist__tui_browse_hists (evlist=0x7cbb90, help=help@entry=0x51d350 "For a higher level overview, try: perf top --sort comm,dso",
hbt=hbt@entry=0x7ffff3c40e60, min_pcnt=<optimized out>, env=<optimized out>) at ui/browsers/hists.c:2006
#6 0x0000000000435850 in display_thread_tui (arg=0x7fffffffa580) at builtin-top.c:582
#7 0x00007ffff7bc40a5 in start_thread (arg=0x7ffff3c41700) at pthread_create.c:309
#8 0x00007ffff5fe4cfd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111using latestest tools/perf.
Thanks,
Ingo
Borislav Petkov
maintainerPATCH v2 8/15 x86/smap: Use ALTERNATIVE macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
... and drop unfolded version. No need for ASM_NOP3 anymore either as the alternatives do the proper padding at build time and insert proper NOPs at boot time.
There should be no apparent operational change from this patch.
--- arch/x86/include/asm/smap.h | 30 +++++++++--------------------- 1 file changed, 9 insertions(+), 21 deletions(-)
arch/x86/include/asm/smap.h +9 -21
@@ -27,23 +27,11 @@#ifdef CONFIG_X86_SMAP -#define ASM_CLAC \ - 661: ASM_NOP3 ; \ - .pushsection .altinstr_replacement, "ax" ; \ - 662: __ASM_CLAC ; \ - .popsection ; \ - .pushsection .altinstructions, "a" ; \ - altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3, 0 ; \ - .popsection - -#define ASM_STAC \ - 661: ASM_NOP3 ; \ - .pushsection .altinstr_replacement, "ax" ; \ - 662: __ASM_STAC ; \ - .popsection ; \ - .pushsection .altinstructions, "a" ; \ - altinstruction_entry 661b, 662b, X86_FEATURE_SMAP, 3, 3, 0 ; \ - .popsection +#define ASM_CLAC \ + ALTERNATIVE "", __stringify(__ASM_CLAC), X86_FEATURE_SMAP + +#define ASM_STAC \ + ALTERNATIVE "", __stringify(__ASM_STAC), X86_FEATURE_SMAP#else /* CONFIG_X86_SMAP */
@@ -61,20 +49,20 @@ static __always_inline void clac(void) {/* Note: a barrier is implicit in alternative() */ -alternative (ASM_NOP3 ,__stringify (__ASM_CLAC),X86_FEATURE_SMAP ); +alternative ("" ,__stringify (__ASM_CLAC),X86_FEATURE_SMAP ); } static __always_inline void stac(void) {/* Note: a barrier is implicit in alternative() */ -alternative (ASM_NOP3 ,__stringify (__ASM_STAC),X86_FEATURE_SMAP ); +alternative ("" ,__stringify (__ASM_STAC),X86_FEATURE_SMAP ); }/* These macros can be used in asm() statements */ #define ASM_CLAC \ - ALTERNATIVE(ASM_NOP3, __stringify(__ASM_CLAC), X86_FEATURE_SMAP) + ALTERNATIVE("", __stringify(__ASM_CLAC), X86_FEATURE_SMAP)#define ASM_STAC \ - ALTERNATIVE(ASM_NOP3, __stringify(__ASM_STAC), X86_FEATURE_SMAP) + ALTERNATIVE("", __stringify(__ASM_STAC), X86_FEATURE_SMAP)#else /* CONFIG_X86_SMAP */
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 9/15 x86/entry_32: Convert X86_INVD_BUG to ALTERNATIVE macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Booting a 486 kernel on an AMD guest with this patch applied, says:
apply_alternatives: feat: 0*32+25, old: (c160a475, len: 5), repl: (c19557d4, len: 5)
c160a475: alt_insn: 68 10 35 00 c1
c19557d4: rpl_insn: 68 80 39 00 c1
which is:
old insn VA: 0xc160a475, CPU feat: X86_FEATURE_XMM, size: 5 simd_coprocessor_error:
c160a475: 68 10 35 00 c1 push $0xc1003510 <do_general_protection>
repl insn: 0xc19557d4, size: 5
c160a475: 68 80 39 00 c1 push $0xc1003980 <do_simd_coprocessor_error>--- arch/x86/kernel/entry_32.S | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)
arch/x86/kernel/entry_32.S +3 -9
@@ -816,15 +816,9 @@ ENTRY(simd_coprocessor_error) pushl_cfi $0 #ifdef CONFIG_X86_INVD_BUG /* AMD 486 bug: invd from userspace calls exception 19 instead of #GP */ -661: pushl_cfi $do_general_protection -662: -.section .altinstructions,"a" - altinstruction_entry 661b, 663f, X86_FEATURE_XMM, 662b-661b, 664f-663f, 0 -.previous -.section .altinstr_replacement,"ax" -663: pushl $do_simd_coprocessor_error -664: -.previous + ALTERNATIVE "pushl_cfi $do_general_protection", \ + "pushl $do_simd_coprocessor_error", \ + X86_FEATURE_XMM #else pushl_cfi $do_simd_coprocessor_error #endif
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 10/15 x86/lib/clear_page_64.S: Convert to ALTERNATIVE_2 macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Move clear_page() up so that we can get 2-byte forward JMPs when patching:
apply_alternatives: feat: 3*32+16, old: (ffffffff8130adb0, len: 5), repl: (ffffffff81d0b859, len: 5)
ffffffff8130adb0: alt_insn: 90 90 90 90 90
recompute_jump: new_displ: 0x0000003e
ffffffff81d0b859: rpl_insn: eb 3e 66 66 90
even though the compiler generated 5-byte JMPs which we padded with 5 NOPs.
Also, make the REP_GOOD version be the default as the majority of machines set REP_GOOD. This way we get to save ourselves the JMP:
old insn VA: 0xffffffff813038b0, CPU feat: X86_FEATURE_REP_GOOD, size: 5, padlen: 0 clear_page:
ffffffff813038b0 <clear_page>:
ffffffff813038b0: e9 0b 00 00 00 jmpq ffffffff813038c0
repl insn: 0xffffffff81cf0e92, size: 0
old insn VA: 0xffffffff813038b0, CPU feat: X86_FEATURE_ERMS, size: 5, padlen: 0 clear_page:
ffffffff813038b0 <clear_page>:
ffffffff813038b0: e9 0b 00 00 00 jmpq ffffffff813038c0
repl insn: 0xffffffff81cf0e92, size: 5
ffffffff81cf0e92: e9 69 2a 61 ff jmpq ffffffff81303900
ffffffff813038b0 <clear_page>:
ffffffff813038b0: e9 69 2a 61 ff jmpq ffffffff8091631e
--- arch/x86/lib/clear_page_64.S | 66 ++++++++++++++++++-------------------------- 1 file changed, 27 insertions(+), 39 deletions(-)
arch/x86/lib/clear_page_64.S +27 -39
@@ -1,31 +1,35 @@ #include <linux/linkage.h> #include <asm/dwarf2.h> +#include <asm/cpufeature.h> #include <asm/alternative-asm.h> /* - * Zero a page. - * rdi page - */ -ENTRY(clear_page_c) + * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is + * recommended to use this when possible and we do use them by default. + * If enhanced REP MOVSB/STOSB is not available, try to use fast string. + * Otherwise, use original. + */ + +/* + * Zero a page. + * %rdi - page + */ +ENTRY(clear_page) CFI_STARTPROC + + ALTERNATIVE_2 "jmp clear_page_orig", "", X86_FEATURE_REP_GOOD, \ + "jmp clear_page_c_e", X86_FEATURE_ERMS + movl $4096/8,%ecx xorl %eax,%eax rep stosq ret CFI_ENDPROC -ENDPROC(clear_page_c) +ENDPROC(clear_page) -ENTRY(clear_page_c_e) +ENTRY(clear_page_orig) CFI_STARTPROC - movl $4096,%ecx - xorl %eax,%eax - rep stosb - ret - CFI_ENDPROC -ENDPROC(clear_page_c_e) -ENTRY(clear_page) - CFI_STARTPROC xorl %eax,%eax movl $4096/64,%ecx .p2align 4
@@ -45,29 +49,13 @@ ENTRY(clear_page) nop ret CFI_ENDPROC -.Lclear_page_end: -ENDPROC(clear_page) - - /* - * Some CPUs support enhanced REP MOVSB/STOSB instructions. - * It is recommended to use this when possible. - * If enhanced REP MOVSB/STOSB is not available, try to use fast string. - * Otherwise, use original function. - * - */ +ENDPROC(clear_page_orig) -#include <asm/cpufeature.h> - - .section .altinstr_replacement,"ax" -1: .byte 0xeb /* jmp <disp8> */ - .byte (clear_page_c - clear_page) - (2f - 1b) /* offset */ -2: .byte 0xeb /* jmp <disp8> */ - .byte (clear_page_c_e - clear_page) - (3f - 2b) /* offset */ -3: - .previous - .section .altinstructions,"a" - altinstruction_entry clear_page,1b,X86_FEATURE_REP_GOOD,\ - .Lclear_page_end-clear_page, 2b-1b, 0 - altinstruction_entry clear_page,2b,X86_FEATURE_ERMS, \ - .Lclear_page_end-clear_page,3b-2b, 0 - .previous +ENTRY(clear_page_c_e) + CFI_STARTPROC + movl $4096,%ecx + xorl %eax,%eax + rep stosb + ret + CFI_ENDPROC +ENDPROC(clear_page_c_e)
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 11/15 x86/asm: Use alternative_2() in rdtsc_barrier()
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
... now that we have it.
--- arch/x86/include/asm/barrier.h | 6 ++---- arch/x86/um/asm/barrier.h | 4 ++-- 2 files changed, 4 insertions(+), 6 deletions(-)
arch/x86/include/asm/barrier.h +2 -4
@@ -95,13 +95,11 @@ do { \* Stop RDTSC speculation. This is needed when you need to use RDTSC * (or get_cycles or vread that possibly accesses the TSC) in a defined * code region. -* -* (Could use an alternative three way for this if there was one.) */ static __always_inline void rdtsc_barrier(void) { -alternative (ASM_NOP3 ,"mfence" ,X86_FEATURE_MFENCE_RDTSC ); -alternative (ASM_NOP3 ,"lfence" ,X86_FEATURE_LFENCE_RDTSC ); +alternative_2 ("" ,"mfence" ,X86_FEATURE_MFENCE_RDTSC , +"lfence" ,X86_FEATURE_LFENCE_RDTSC ); }#endif /* _ASM_X86_BARRIER_H */
arch/x86/um/asm/barrier.h +2 -2
@@ -64,8 +64,8 @@*/ static inline void rdtsc_barrier (void ) { -alternative (ASM_NOP3 ,"mfence" ,X86_FEATURE_MFENCE_RDTSC ); -alternative (ASM_NOP3 ,"lfence" ,X86_FEATURE_LFENCE_RDTSC ); +alternative_2 ("" ,"mfence" ,X86_FEATURE_MFENCE_RDTSC , +"lfence" ,X86_FEATURE_LFENCE_RDTSC ); }#endif
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 12/15 x86/asm: Cleanup prefetch primitives
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1 as a mem8 operand constraint and rely on the toolchain to generate the proper instruction sizes. For example, on 32-bit, where we use an empty old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the replacement instruction the compiler generates.
--- arch/x86/include/asm/apic.h | 2 +- arch/x86/include/asm/processor.h | 16 +++++++--------- arch/x86/kernel/cpu/amd.c | 5 +++++ 3 files changed, 13 insertions(+), 10 deletions(-)
arch/x86/include/asm/apic.h +1 -1
@@ -91,7 +91,7 @@ static inline void native_apic_mem_write(u32 reg, u32 v) {volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); -alternative_io ("movl %0, %1" ,"xchgl %0, %1" ,X86_BUG_11AP , +alternative_io ("movl %0, %P1" ,"xchgl %0, %P1" ,X86_BUG_11AP ,ASM_OUTPUT2 ("=r" (v),"=m" (*addr)),ASM_OUTPUT2 ("0" (v),"m" (*addr))); }
arch/x86/include/asm/processor.h +7 -9
@@ -761,10 +761,10 @@ extern char ignore_fpu_irq;#define ARCH_HAS_SPINLOCK_PREFETCH #ifdef CONFIG_X86_32 -# define BASE_PREFETCH ASM_NOP4 +# define BASE_PREFETCH ""# define ARCH_HAS_PREFETCH #else -# define BASE_PREFETCH "prefetcht0 (%1)" +# define BASE_PREFETCH "prefetcht0 %P1"#endif /*
@@ -775,10 +775,9 @@ extern char ignore_fpu_irq;*/ static inline void prefetch (const void *x) { -alternative_input (BASE_PREFETCH , -"prefetchnta (%1)" , +alternative_input (BASE_PREFETCH ,"prefetchnta %P1" ,X86_FEATURE_XMM , -"r" (x)); +"m" (*(const char *)x)); }/*
@@ -788,10 +787,9 @@ static inline void prefetch(const void *x)*/ static inline void prefetchw (const void *x) { -alternative_input (BASE_PREFETCH , -"prefetchw (%1)" , -X86_FEATURE_3DNOW , -"r" (x)); +alternative_input (BASE_PREFETCH ,"prefetchw %P1" , +X86_FEATURE_3DNOWPREFETCH , +"m" (*(const char *)x)); }static inline void spin_lock_prefetch (const void *x)
arch/x86/kernel/cpu/amd.c +5 -0
@@ -711,6 +711,11 @@ static void init_amd(struct cpuinfo_x86 *c)set_cpu_bug (c,X86_BUG_AMD_APIC_C1E );rdmsr_safe (MSR_AMD64_PATCH_LEVEL , &c->microcode, &dummy); + +/* 3DNow or LM implies PREFETCHW */ +if (!cpu_has (c,X86_FEATURE_3DNOWPREFETCH )) +if (cpu_has (c,X86_FEATURE_3DNOW ) ||cpu_has (c,X86_FEATURE_LM )) +set_cpu_cap (c,X86_FEATURE_3DNOWPREFETCH ); }#ifdef CONFIG_X86_32
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH v2 12/15] x86/asm: Cleanup prefetch primitives
Quoted 98 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1 as a mem8 operand constraint and rely on the toolchain to generate the proper instruction sizes. For example, on 32-bit, where we use an empty old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b:alt_insn: 90 90 90 90
c195566c:rpl_insn: 0f 0d 4b 5c...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4:alt_insn: 90 90 90
c1955948:rpl_insn: 0f 0d 08...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9:alt_insn: 90 90 90 90 90 90 90
c1955a79:rpl_insn: 0f 0d 0d a0 d4 85 c1all with the proper padding done depending on the size of the replacement instruction the compiler generates.
Signed-off-by Borislav Petkov <[email protected]>Cc H. Peter Anvin <[email protected]>--- arch/x86/include/asm/apic.h | 2 +- arch/x86/include/asm/processor.h | 16 +++++++--------- arch/x86/kernel/cpu/amd.c | 5 +++++ 3 files changed, 13 insertions(+), 10 deletions(-)arch/x86/include/asm/apic.h +1
-1@@ -91,7 +91,7 @@ static inline void native_apic_mem_write(u32 reg, u32 v) {volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); -alternative_io ("movl %0, %1" ,"xchgl %0, %1" ,X86_BUG_11AP , +alternative_io ("movl %0, %P1" ,"xchgl %0, %P1" ,X86_BUG_11AP ,ASM_OUTPUT2 ("=r" (v),"=m" (*addr)),ASM_OUTPUT2 ("0" (v),"m" (*addr))); }arch/x86/include/asm/processor.h +7
-9@@ -761,10 +761,10 @@ extern char ignore_fpu_irq;#define ARCH_HAS_SPINLOCK_PREFETCH #ifdef CONFIG_X86_32 -# define BASE_PREFETCH ASM_NOP4 +# define BASE_PREFETCH ""# define ARCH_HAS_PREFETCH #else -# define BASE_PREFETCH "prefetcht0 (%1)" +# define BASE_PREFETCH "prefetcht0 %P1"#endif /* @@ -775,10 +775,9 @@ extern char ignore_fpu_irq;*/ static inline void prefetch (const void *x) { -alternative_input (BASE_PREFETCH , -"prefetchnta (%1)" , +alternative_input (BASE_PREFETCH ,"prefetchnta %P1" ,X86_FEATURE_XMM , -"r" (x)); +"m" (*(const char *)x)); }/* @@ -788,10 +787,9 @@ static inline void prefetch(const void *x)*/ static inline void prefetchw (const void *x) { -alternative_input (BASE_PREFETCH , -"prefetchw (%1)" , -X86_FEATURE_3DNOW , -"r" (x)); +alternative_input (BASE_PREFETCH ,"prefetchw %P1" , +X86_FEATURE_3DNOWPREFETCH , +"m" (*(const char *)x)); }static inline void spin_lock_prefetch (const void *x)arch/x86/kernel/cpu/amd.c +0
-0
arch/x86/kernel/cpu/amd.c static void init_amd(struct cpuinfo_x86 *c) · line 720
Quoted 10 lines
arch/x86/kernel/cpu/amd.c +5
-0@@ -711,6 +711,11 @@ static void init_amd(struct cpuinfo_x86 *c)set_cpu_bug (c,X86_BUG_AMD_APIC_C1E );rdmsr_safe (MSR_AMD64_PATCH_LEVEL , &c->microcode, &dummy); + +/* 3DNow or LM implies PREFETCHW */ +if (!cpu_has (c,X86_FEATURE_3DNOWPREFETCH )) +if (cpu_has (c,X86_FEATURE_3DNOW ) ||cpu_has (c,X86_FEATURE_LM )) +set_cpu_cap (c,X86_FEATURE_3DNOWPREFETCH ); }
For future reference: you forgot to declare the cpu/amd.c X86_FEATURE_3DNOWPREFETCH enablement change in the changelog. I guess you noticed it during testing and forgot about it?
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 12/15] x86/asm: Cleanup prefetch primitives
Quoted 3 lines · Ingo Molnar
For future reference: you forgot to declare the
cpu/amd.cX86_FEATURE_3DNOWPREFETCHenablement change in the changelog. I guess you noticed it during testing and forgot about it?
In the changelog of the patch? Oh sorry.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Borislav Petkov
maintainerPATCH v2 13/15 x86/lib/memset_64.S: Convert to ALTERNATIVE_2 macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Make alternatives replace single JMPs instead of whole memset functions, thus decreasing the amount of instructions copied during patching time at boot.
While at it, make it use the REP_GOOD version by default which means alternatives NOP out the JMP to the other versions, as REP_GOOD is set by default on the majority of relevant x86 processors.
--- arch/x86/lib/memset_64.S | 61 +++++++++++++++++++----------------------------- 1 file changed, 24 insertions(+), 37 deletions(-)
arch/x86/lib/memset_64.S +24 -37
@@ -5,19 +5,30 @@ #include <asm/cpufeature.h> #include <asm/alternative-asm.h> +.weak memset + /* * ISO C memset - set a memory block to a byte value. This function uses fast * string to get better performance than the original function. The code is * simpler and shorter than the orignal function as well. - * + * * rdi destination - * rsi value (char) - * rdx count (bytes) - * + * rsi value (char) + * rdx count (bytes) + * * rax original destination - */ - .section .altinstr_replacement, "ax", @progbits -.Lmemset_c: + */ +ENTRY(memset) +ENTRY(__memset) + /* + * Some CPUs support enhanced REP MOVSB/STOSB feature. It is recommended + * to use it when possible. If not available, use fast string instructions. + * + * Otherwise, use original memset function. + */ + ALTERNATIVE_2 "jmp memset_orig", "", X86_FEATURE_REP_GOOD, \ + "jmp memset_erms", X86_FEATURE_ERMS + movq %rdi,%r9 movq %rdx,%rcx andl $7,%edx
@@ -31,8 +42,8 @@ rep stosb movq %r9,%rax ret -.Lmemset_e: - .previous +ENDPROC(memset) +ENDPROC(__memset) /* * ISO C memset - set a memory block to a byte value. This function uses
@@ -45,21 +56,16 @@ * * rax original destination */ - .section .altinstr_replacement, "ax", @progbits -.Lmemset_c_e: +ENTRY(memset_erms) movq %rdi,%r9 movb %sil,%al movq %rdx,%rcx rep stosb movq %r9,%rax ret -.Lmemset_e_e: - .previous - -.weak memset +ENDPROC(memset_erms) -ENTRY(memset) -ENTRY(__memset) +ENTRY(memset_orig) CFI_STARTPROC movq %rdi,%r10
@@ -134,23 +140,4 @@ ENTRY(__memset) jmp .Lafter_bad_alignment .Lfinal: CFI_ENDPROC -ENDPROC(memset) -ENDPROC(__memset) - - /* Some CPUs support enhanced REP MOVSB/STOSB feature. - * It is recommended to use this when possible. - * - * If enhanced REP MOVSB/STOSB feature is not available, use fast string - * instructions. - * - * Otherwise, use original memset function. - * - * In .altinstructions section, ERMS feature is placed after REG_GOOD - * feature to implement the right patch order. - */ - .section .altinstructions,"a" - altinstruction_entry __memset,.Lmemset_c,X86_FEATURE_REP_GOOD,\ - .Lfinal-__memset,.Lmemset_e-.Lmemset_c,0 - altinstruction_entry __memset,.Lmemset_c_e,X86_FEATURE_ERMS, \ - .Lfinal-__memset,.Lmemset_e_e-.Lmemset_c_e,0 - .previous +ENDPROC(memset_orig)
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH v2 14/15 x86/lib/memmove_64.S: Convert memmove() to ALTERNATIVE macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Make it execute the ERMS version if support is present and we're in the forward memmove() part and remove the unfolded alternatives section definition.
--- arch/x86/lib/memmove_64.S | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-)
arch/x86/lib/memmove_64.S +2 -17
@@ -5,7 +5,6 @@ * This assembly file is re-written from memmove_64.c file. * - Copyright 2011 Fenghua Yu <[email protected]> */ -#define _STRING_C #include <linux/linkage.h> #include <asm/dwarf2.h> #include <asm/cpufeature.h>
@@ -44,6 +43,8 @@ ENTRY(__memmove) jg 2f .Lmemmove_begin_forward: + ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_ERMS + /* * movsq instruction have many startup latency * so we handle small size by general register.
@@ -207,21 +208,5 @@ ENTRY(__memmove) 13: retq CFI_ENDPROC - - .section .altinstr_replacement,"ax" -.Lmemmove_begin_forward_efs: - /* Forward moving data. */ - movq %rdx, %rcx - rep movsb - retq -.Lmemmove_end_forward_efs: - .previous - - .section .altinstructions,"a" - altinstruction_entry .Lmemmove_begin_forward, \ - .Lmemmove_begin_forward_efs,X86_FEATURE_ERMS, \ - .Lmemmove_end_forward-.Lmemmove_begin_forward, \ - .Lmemmove_end_forward_efs-.Lmemmove_begin_forward_efs,0 - .previous ENDPROC(__memmove) ENDPROC(memmove)
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH v2 14/15] x86/lib/memmove_64.S: Convert memmove() to ALTERNATIVE macro
Quoted 24 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Make it execute the ERMS version if support is present and we're in the forward
memmove()part and remove the unfolded alternatives section definition.Signed-off-by Borislav Petkov <[email protected]>--- arch/x86/lib/memmove_64.S | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-)arch/x86/lib/memmove_64.S +0
-1@@ -5,7 +5,6 @@ * This assembly file is re-written from memmove_64.c file. * - Copyright 2011 Fenghua Yu <[email protected]> */ -#define _STRING_C #include <linux/linkage.h> #include <asm/dwarf2.h> #include <asm/cpufeature.h>
arch/x86/lib/memmove_64.S ENTRY(__memmove) · line 50
Quoted 8 lines
arch/x86/lib/memmove_64.S +2
-0@@ -44,6 +43,8 @@ ENTRY(__memmove) jg 2f .Lmemmove_begin_forward: + ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_ERMS + /* * movsq instruction have many startup latency * so we handle small size by general register.
Btw., it might make sense to translate that comment to English, while at it? (in a separate patch)
Thanks,
Ingo
Borislav Petkov
maintainerPATCH v2 15/15 x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Make REP_GOOD variant the default after alternatives have run.
--- arch/x86/lib/memcpy_64.S | 68 +++++++++++++++--------------------------------- 1 file changed, 21 insertions(+), 47 deletions(-)
arch/x86/lib/memcpy_64.S +21 -47
@@ -1,12 +1,20 @@ /* Copyright 2002 Andi Kleen */ #include <linux/linkage.h> - #include <asm/cpufeature.h> #include <asm/dwarf2.h> #include <asm/alternative-asm.h> /* + * We build a jump to memcpy_orig by default which gets NOPped out on + * the majority of x86 CPUs which set REP_GOOD. In addition, CPUs which + * have the enhanced REP MOVSB/STOSB feature (ERMS), change those NOPs + * to a jmp to memcpy_erms which does the REP; MOVSB mem copy. + */ + +.weak memcpy + +/* * memcpy - Copy a memory block. * * Input:
@@ -17,15 +25,11 @@ * Output: * rax original destination */ +ENTRY(__memcpy) +ENTRY(memcpy) + ALTERNATIVE_2 "jmp memcpy_orig", "", X86_FEATURE_REP_GOOD, \ + "jmp memcpy_erms", X86_FEATURE_ERMS -/* - * memcpy_c() - fast string ops (REP MOVSQ) based variant. - * - * This gets patched over the unrolled variant (below) via the - * alternative instructions framework: - */ - .section .altinstr_replacement, "ax", @progbits -.Lmemcpy_c: movq %rdi, %rax movq %rdx, %rcx shrq $3, %rcx
@@ -34,29 +38,21 @@ movl %edx, %ecx rep movsb ret -.Lmemcpy_e: - .previous +ENDPROC(memcpy) +ENDPROC(__memcpy) /* - * memcpy_c_e() - enhanced fast string memcpy. This is faster and simpler than - * memcpy_c. Use memcpy_c_e when possible. - * - * This gets patched over the unrolled variant (below) via the - * alternative instructions framework: + * memcpy_erms() - enhanced fast string memcpy. This is faster and + * simpler than memcpy. Use memcpy_erms when possible. */ - .section .altinstr_replacement, "ax", @progbits -.Lmemcpy_c_e: +ENTRY(memcpy_erms) movq %rdi, %rax movq %rdx, %rcx rep movsb ret -.Lmemcpy_e_e: - .previous - -.weak memcpy +ENDPROC(memcpy_erms) -ENTRY(__memcpy) -ENTRY(memcpy) +ENTRY(memcpy_orig) CFI_STARTPROC movq %rdi, %rax
@@ -183,26 +179,4 @@ ENTRY(memcpy) .Lend: retq CFI_ENDPROC -ENDPROC(memcpy) -ENDPROC(__memcpy) - - /* - * Some CPUs are adding enhanced REP MOVSB/STOSB feature - * If the feature is supported, memcpy_c_e() is the first choice. - * If enhanced rep movsb copy is not available, use fast string copy - * memcpy_c() when possible. This is faster and code is simpler than - * original memcpy(). - * Otherwise, original memcpy() is used. - * In .altinstructions section, ERMS feature is placed after REG_GOOD - * feature to implement the right patch order. - * - * Replace only beginning, memcpy is used to apply alternatives, - * so it is silly to overwrite itself with nops - reboot is the - * only outcome... - */ - .section .altinstructions, "a" - altinstruction_entry __memcpy,.Lmemcpy_c,X86_FEATURE_REP_GOOD,\ - .Lmemcpy_e-.Lmemcpy_c,.Lmemcpy_e-.Lmemcpy_c,0 - altinstruction_entry __memcpy,.Lmemcpy_c_e,X86_FEATURE_ERMS, \ - .Lmemcpy_e_e-.Lmemcpy_c_e,.Lmemcpy_e_e-.Lmemcpy_c_e,0 - .previous +ENDPROC(memcpy_orig)
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH v2 15/15] x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
Quoted 42 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Make
REP_GOODvariant the default after alternatives have run.Signed-off-by Borislav Petkov <[email protected]>--- arch/x86/lib/memcpy_64.S | 68 +++++++++++++++--------------------------------- 1 file changed, 21 insertions(+), 47 deletions(-)arch/x86/lib/memcpy_64.S +12
-1@@ -1,12 +1,20 @@ /* Copyright 2002 Andi Kleen */ #include <linux/linkage.h> - #include <asm/cpufeature.h> #include <asm/dwarf2.h> #include <asm/alternative-asm.h> /* + * We build a jump to memcpy_orig by default which gets NOPped out on + * the majority of x86 CPUs which set REP_GOOD. In addition, CPUs which + * have the enhanced REP MOVSB/STOSB feature (ERMS), change those NOPs + * to a jmp to memcpy_erms which does the REP; MOVSB mem copy. + */ + +.weak memcpy + +/* * memcpy - Copy a memory block. * * Input:@@ -17,15 +25,11 @@ * Output: * rax original destination */ +ENTRY(__memcpy) +ENTRY(memcpy) + ALTERNATIVE_2 "jmp memcpy_orig", "", X86_FEATURE_REP_GOOD, \
arch/x86/lib/memcpy_64.S
Quoted 10 lines
arch/x86/lib/memcpy_64.S +4
-0+ "jmp memcpy_erms", X86_FEATURE_ERMS +ENDPROC(memcpy) +ENDPROC(__memcpy) +ENTRY(memcpy_erms) movq %rdi, %rax movq %rdx, %rcx rep movsb ret
Since most CPUs we care about have ERMS, wouldn't it be better to patch in the actual memcpy_erms sequence into the primary memcpy() function? It's just about 9 bytes AFAICT.
This would remove a jump instruction from the most common memcpy
variant: worth it!
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 15/15] x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
Quoted 3 lines · Ingo Molnar
Since most CPUs we care about have ERMS, wouldn't it be better to patch in the actual
memcpy_ermssequence into the primarymemcpy()function? It's just about 9 bytes AFAICT.
Actually, most set REP_GOOD - all Intel family 6 and all relevant AMDs.
And only the newer Intels have ERMS. My SNB, for example doesn't while IVB has it. So I'd guess everything >= IVB would have it.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 15/15] x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Ingo Molnar
Since most CPUs we care about have ERMS, wouldn't it be better to patch in the actual
memcpy_ermssequence into the primarymemcpy()function? It's just about 9 bytes AFAICT.Actually, most set
REP_GOOD- all Intel family 6 and all relevant AMDs.And only the newer Intels have ERMS. My SNB, for example doesn't while IVB has it. So I'd guess everything >= IVB would have it.
Well, my point equally applies to all variants: it's better to avoid the NOP or JMP overhead (however small it may be), by simply copying the ideal memcpy routine into memcpy()?
I.e. while I'd not want to patch in memcpy_orig (it's legacy really), but the other two variants, ERMS and REP MOVSQ could be patched in directly via ALTERNATIVE_2()?
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 15/15] x86/lib/memcpy_64.S: Convert memcpy to ALTERNATIVE_2 macro
Quoted 3 lines · Ingo Molnar
I.e. while I'd not want to patch in
memcpy_orig(it's legacy really), but the other two variants, ERMS and REP MOVSQ could be patched in directly viaALTERNATIVE_2()?
Certainly worth the try, it is on the TODO list.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Quoted 41 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
[ Changelog is in version-increasing number so that one can follow the
evolution of the patch set in a more natural way (i.e., latest version comes at the end. ]v0:
this is something which hpa and I talked about recently: the ability for the alternatives code to add padding to the original instruction in case the replacement is longer and also to be able to simply write "jmp" and not care about which JMP exactly the compiler generates and whether the relative offsets are correct.
So this is a stab at it, it seems to boot in kvm here but it needs more staring to make sure we're actually generating the proper code at all times.
Thus the RFC tag, comments/suggestions are welcome.
v1:
This is the first version which passes testing on AMD/Intel, 32/64-bit boxes I have here. For more info what it does, you can boot with "debug-alternative" to see some verbose information about what gets changed into what.
Patches 1 and 2 are cleanups.
Patch 3 is adding the padding at build time and patch 4 simplifies using JMPs in alternatives without having to do crazy math with labels, as a user of the alternatives facilities.
Patch 5 optimizes the single-byte NOPs we're adding at build time to longer NOPs which should go easier through the frontend.
Patches 6-12 then convert most of the alternative callsites to the generic macros and kill the homegrown fun.
v2:
Quoted 7 lines
This version reworks the NOP padding by adding a field to struct
alt_instrwhich holds the padding length and thus makes the padding more robust than what we did before, instead of us trying to figure out which byte is a NOP and which byte is something else (part of a relative offset or immediate...).Thanks to Andy Lutomirsky for pointing that out.
:)
I'll read the NOP stuff in a minute. In the mean time, the series seems to work correctly for the vdso, and the vdso has fewer pointless nops.
--Andy
Quoted 42 lines
As always, constructive comments/suggestions are welcome.
Borislav Petkov (15):
x86/lib/copy_user_64.S: RemoveFIX_ALIGNMENTdefine x86/alternatives: Cleanup DPRINTK macro x86/alternatives: Add instruction padding x86/alternatives: Make JMPs more robust x86/alternatives: Use optimized NOPs for paddingx86/lib/copy_page_64.S: Use generic ALTERNATIVE macrox86/lib/copy_user_64.S: Convert toALTERNATIVE_2x86/smap: Use ALTERNATIVE macro
x86/entry_32: ConvertX86_INVD_BUGto ALTERNATIVE macrox86/lib/clear_page_64.S: Convert toALTERNATIVE_2macro x86/asm: Usealternative_2()inrdtsc_barrier()x86/asm: Cleanup prefetch primitivesx86/lib/memset_64.S: Convert toALTERNATIVE_2macrox86/lib/memmove_64.S: Convertmemmove()to ALTERNATIVE macrox86/lib/memcpy_64.S: Convert memcpy toALTERNATIVE_2macroarch/x86/include/asm/alternative-asm.h | 43 ++++++++- arch/x86/include/asm/alternative.h | 65 ++++++++------ arch/x86/include/asm/apic.h | 2 +- arch/x86/include/asm/barrier.h | 6 +- arch/x86/include/asm/cpufeature.h | 30 ++++--- arch/x86/include/asm/processor.h | 16 ++-- arch/x86/include/asm/smap.h | 30 ++----- arch/x86/kernel/alternative.c | 158 ++++++++++++++++++++++++++++----- arch/x86/kernel/cpu/amd.c | 5 ++ arch/x86/kernel/entry_32.S | 12 +-- arch/x86/lib/clear_page_64.S | 66 ++++++-------- arch/x86/lib/copy_page_64.S | 37 +++----- arch/x86/lib/copy_user_64.S | 46 +++------- arch/x86/lib/memcpy_64.S | 68 +++++--------- arch/x86/lib/memmove_64.S | 19 +--- arch/x86/lib/memset_64.S | 61 +++++-------- arch/x86/um/asm/barrier.h | 4 +- 17 files changed, 360 insertions(+), 308 deletions(-)--
2.2.0.33.gc18b867
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC
Borislav Petkov
maintainerHi all,
So this alternatives patchset breaks perf bench mem, here are a couple of patches ontop, you guys tell me whether it makes sense. I wanted to make it run all memset/memcpy routines so here are a couple of patches which do this:
./perf bench mem memset -l 20MB -r all
# Running 'mem/memset' benchmark:
Routine default (Default memset() provided by glibc) # Copying 20MB Bytes ...
1.136000 GB/Sec
6.026304 GB/Sec (with prefault)Routine x86-64-unrolled (unrolled memset() in arch/x86/lib/memset_64.S) # Copying 20MB Bytes ...
5.333493 GB/Sec
5.633473 GB/Sec (with prefault)Routine x86-64-stosq (movsq-based memset() in arch/x86/lib/memset_64.S) # Copying 20MB Bytes ...
5.828484 GB/Sec
5.851183 GB/Sec (with prefault)Routine x86-64-stosb (movsb-based memset() in arch/x86/lib/memset_64.S) # Copying 20MB Bytes ...
5.553384 GB/Sec
5.956465 GB/Sec (with prefault)This way you can see all results by executing one command only with "-r all".
Patches coming as a reply to this message.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Borislav Petkov
maintainerPATCH 1/3 perf/bench: Fix mem* routines usage after alternatives change
From: Borislav Petkov <[email protected]>
Adjust perf bench to the new changes in the alternatives code for memcpy/memset.
--- tools/perf/bench/mem-memcpy-x86-64-asm-def.h | 6 +++--- tools/perf/bench/mem-memcpy-x86-64-asm.S | 2 -- tools/perf/bench/mem-memset-x86-64-asm-def.h | 6 +++--- tools/perf/bench/mem-memset-x86-64-asm.S | 2 -- tools/perf/util/include/asm/alternative-asm.h | 1 + 5 files changed, 7 insertions(+), 10 deletions(-)
tools/perf/bench/mem-memcpy-x86-64-asm-def.h +3 -3
@@ -1,12 +1,12 @@ -MEMCPY_FN (__memcpy, +MEMCPY_FN (memcpy_orig,"x86-64-unrolled" , "unrolled memcpy() in arch/x86/lib/memcpy_64.S") -MEMCPY_FN (memcpy_c, +MEMCPY_FN (__memcpy,"x86-64-movsq" , "movsq-based memcpy() in arch/x86/lib/memcpy_64.S") -MEMCPY_FN (memcpy_c_e, +MEMCPY_FN (memcpy_erms,"x86-64-movsb" ,"movsb-based memcpy() in arch/x86/lib/memcpy_64.S" )
tools/perf/bench/mem-memcpy-x86-64-asm.S +0 -2
@@ -1,8 +1,6 @@ #define memcpy MEMCPY /* don't hide glibc's memcpy() */ #define altinstr_replacement text #define globl p2align 4; .globl -#define Lmemcpy_c globl memcpy_c; memcpy_c -#define Lmemcpy_c_e globl memcpy_c_e; memcpy_c_e #include "../../../arch/x86/lib/memcpy_64.S" /* * We need to provide note.GNU-stack section, saying that we want
tools/perf/bench/mem-memset-x86-64-asm-def.h +3 -3
@@ -1,12 +1,12 @@ -MEMSET_FN (__memset, +MEMSET_FN (memset_orig,"x86-64-unrolled" , "unrolled memset() in arch/x86/lib/memset_64.S") -MEMSET_FN (memset_c, +MEMSET_FN (__memset,"x86-64-stosq" , "movsq-based memset() in arch/x86/lib/memset_64.S") -MEMSET_FN (memset_c_e, +MEMSET_FN (memset_erms,"x86-64-stosb" ,"movsb-based memset() in arch/x86/lib/memset_64.S" )
tools/perf/bench/mem-memset-x86-64-asm.S +0 -2
@@ -1,8 +1,6 @@ #define memset MEMSET /* don't hide glibc's memset() */ #define altinstr_replacement text #define globl p2align 4; .globl -#define Lmemset_c globl memset_c; memset_c -#define Lmemset_c_e globl memset_c_e; memset_c_e #include "../../../arch/x86/lib/memset_64.S" /*
tools/perf/util/include/asm/alternative-asm.h +1 -0
@@ -4,5 +4,6 @@/* Just disable it so we can build arch/x86/lib/memcpy_64.S for perf bench: */ #define altinstruction_entry # +#define ALTERNATIVE_2 ##endif
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH 2/3 perf/bench: Carve out mem routine benchmarking
From: Borislav Petkov <[email protected]>
... so that we can call it multiple times. See next patch.
--- tools/perf/bench/mem-memcpy.c | 120 ++++++++++++++++++++---------------------- 1 file changed, 58 insertions(+), 62 deletions(-)
tools/perf/bench/mem-memcpy.c +58 -62
@@ -135,84 +135,32 @@ struct bench_mem_info {const char *const *usage; }; -static int bench_mem_common (int argc,const char **argv, -const char *prefix__maybe_unused , -struct bench_mem_info *info) +static void __bench_mem_routine (struct bench_mem_info *info,int r_idx,size_t len,double totallen) { -int i; -size_t len; -double totallen; +const struct routine *r = &info->routines[r_idx];double result_bps[2 ];u64 result_cycle[2 ]; - argc =parse_options (argc, argv, options, - info->usage,0 ); - -if (no_prefault && only_prefault) { -fprintf (stderr,"Invalid options: -o and -n are mutually exclusive\n" ); -return 1 ; - } - -if (use_cycle) -init_cycle (); - - len = (size_t )perf_atoll ((char *)length_str); - totallen = (double )len * iterations; - result_cycle[0 ] = result_cycle[1 ] =0ULL ; result_bps[0 ] = result_bps[1 ] =0.0 ; -if ((s64 )len <=0 ) { -fprintf (stderr,"Invalid length:%s\n" , length_str); -return 1 ; - } - -/* same to without specifying either of prefault and no-prefault */ -if (only_prefault && no_prefault) - only_prefault = no_prefault = false; - -for (i =0 ; info->routines[i].name; i++) { -if (!strcmp (info->routines[i].name, routine)) -break ; - } -if (!info->routines[i].name) { -printf ("Unknown routine:%s\n" , routine); -printf ("Available routines...\n" ); -for (i =0 ; info->routines[i].name; i++) { -printf ("\t%s ... %s\n" , - info->routines[i].name, info->routines[i].desc); - } -return 1 ; - } -if (bench_format ==BENCH_FORMAT_DEFAULT )printf ("# Copying %s Bytes ...\n\n" , length_str);if (!only_prefault && !no_prefault) {/* show both of results */ if (use_cycle) { - result_cycle[0 ] = - info->do_cycle (&info->routines[i], len, false); - result_cycle[1 ] = - info->do_cycle (&info->routines[i], len, true); + result_cycle[0 ] = info->do_cycle (r, len, false); + result_cycle[1 ] = info->do_cycle (r, len, true); }else { - result_bps[0 ] = - info->do_gettimeofday (&info->routines[i], - len, false); - result_bps[1 ] = - info->do_gettimeofday (&info->routines[i], - len, true); + result_bps[0 ] = info->do_gettimeofday (r, len, false); + result_bps[1 ] = info->do_gettimeofday (r, len, true); } }else { -if (use_cycle) { - result_cycle[pf] = - info->do_cycle (&info->routines[i], - len, only_prefault); - }else { - result_bps[pf] = - info->do_gettimeofday (&info->routines[i], - len, only_prefault); - } +if (use_cycle) + result_cycle[pf] = info->do_cycle (r, len, only_prefault); +else + result_bps[pf] = info->do_gettimeofday (r, len, only_prefault); }switch (bench_format) {
@@ -265,6 +213,54 @@ static int bench_mem_common(int argc, const char **argv,die ("unknown format: %d\n" , bench_format);break ; } +} + +static int bench_mem_common (int argc,const char **argv, +const char *prefix__maybe_unused , +struct bench_mem_info *info) +{ +int i; +size_t len; +double totallen; + + argc =parse_options (argc, argv, options, + info->usage,0 ); + +if (no_prefault && only_prefault) { +fprintf (stderr,"Invalid options: -o and -n are mutually exclusive\n" ); +return 1 ; + } + +if (use_cycle) +init_cycle (); + + len = (size_t )perf_atoll ((char *)length_str); + totallen = (double )len * iterations; + +if ((s64 )len <=0 ) { +fprintf (stderr,"Invalid length:%s\n" , length_str); +return 1 ; + } + +/* same to without specifying either of prefault and no-prefault */ +if (only_prefault && no_prefault) + only_prefault = no_prefault = false; + +for (i =0 ; info->routines[i].name; i++) { +if (!strcmp (info->routines[i].name, routine)) +break ; + } +if (!info->routines[i].name) { +printf ("Unknown routine:%s\n" , routine); +printf ("Available routines...\n" ); +for (i =0 ; info->routines[i].name; i++) { +printf ("\t%s ... %s\n" , + info->routines[i].name, info->routines[i].desc); + } +return 1 ; + } + +__bench_mem_routine (info, i, len, totallen);return 0 ; }
Signature · 1 line
2.2.0.33.gc18b867
Borislav Petkov
maintainerPATCH 3/3 perf/bench: Add -r all so that you can run all mem* routines
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
perf bench mem mem{set,cpy} -r all thus runs all available mem benchmarking routines.
--- tools/perf/bench/mem-memcpy.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
tools/perf/bench/mem-memcpy.c +9 -1
@@ -36,7 +36,7 @@ static const struct option options[] = {"Specify length of memory to copy. " "Available units: B, KB, MB, GB and TB (upper and lower)"),OPT_STRING ('r' ,"routine" , &routine,"default" , -"Specify routine to copy" ), +"Specify routine to copy, \"all\" runs all available routines" ),OPT_INTEGER ('i' ,"iterations" , &iterations,"repeat memcpy() invocation this number of times" ), OPT_BOOLEAN('c', "cycle", &use_cycle,
@@ -144,6 +144,8 @@ static void __bench_mem_routine(struct bench_mem_info *info, int r_idx, size_t l result_cycle[0 ] = result_cycle[1 ] =0ULL ; result_bps[0 ] = result_bps[1 ] =0.0 ; +printf ("Routine %s (%s)\n" , r->name, r->desc); +if (bench_format ==BENCH_FORMAT_DEFAULT )printf ("# Copying %s Bytes ...\n\n" , length_str);
@@ -246,6 +248,12 @@ static int bench_mem_common(int argc, const char **argv,if (only_prefault && no_prefault) only_prefault = no_prefault = false; +if (!strncmp (routine,"all" ,3 )) { +for (i =0 ; info->routines[i].name; i++) +__bench_mem_routine (info, i, len, totallen); +return 0 ; + } +for (i =0 ; info->routines[i].name; i++) {if (!strcmp (info->routines[i].name, routine))break ;
Signature · 1 line
2.2.0.33.gc18b867
Re: [PATCH 3/3] perf/bench: Add -r all so that you can run all mem* routines
Quoted 4 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
perf bench mem mem{set,cpy} -r all thus runs all available mem benchmarking routines.
So I think 'perf bench mem all' should also imply '-r all' by default:
I completely forgot that -r even existed and not much hints at its existence ...
Thanks,
Ingo
Hitoshi Mitake
Reviewed-byHi Borislav,
At Thu, 26 Feb 2015 19:13:38 +0100,
Quoted 28 lines · Borislav Petkov
Hi all,
So this alternatives patchset breaks perf bench mem, here are a couple of patches ontop, you guys tell me whether it makes sense. I wanted to make it run all memset/memcpy routines so here are a couple of patches which do this:
./perf bench mem memset -l 20MB -r all
# Running 'mem/memset' benchmark:
Routine default (Defaultmemset()provided by glibc) # Copying 20MB Bytes ...1.136000 GB/Sec 6.026304 GB/Sec (with prefault)Routine x86-64-unrolled (unrolled
memset()inarch/x86/lib/memset_64.S) # Copying 20MB Bytes ...5.333493 GB/Sec 5.633473 GB/Sec (with prefault)Routine x86-64-stosq (movsq-based
memset()inarch/x86/lib/memset_64.S) # Copying 20MB Bytes ...5.828484 GB/Sec 5.851183 GB/Sec (with prefault)Routine x86-64-stosb (movsb-based
memset()inarch/x86/lib/memset_64.S) # Copying 20MB Bytes ...
Quoted 7 lines
5.553384 GB/Sec 5.956465 GB/Sec (with prefault)This way you can see all results by executing one command only with "-r all".
Patches coming as a reply to this message.
I'm not sure I'm a suitable person for reviewing your patch, but I tested this patchset for perf bench with your latest (v2) patchset for x86 alternatives. It looks good to me.
Thanks,
Hitoshi
Borislav Petkov
maintainerQuoted 4 lines · Hitoshi Mitake
I'm not sure I'm a suitable person for reviewing your patch, but I tested this patchset for perf bench with your latest (v2) patchset for x86 alternatives. It looks good to me.
Reviewed-by Hitoshi Mitake <[email protected]>
Thanks a lot, I'll add your tag to the series. We'll see what Arnaldo has to say :-)
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--