The killing of ideal_nops[]
Subsystems (5)
Peter Zijlstra
maintainerHi,
AFAICT everything made in the past 10 years ends up using p6_nops. Is it time to kill off ideal_nops[] and simplify life?
Steven Rostedt
maintainerQuoted 5 lines · On Tue, 9 Mar 2021 17:58:17 +0100 Peter Zijlstra <[email protected]> wrote:
Hi,
AFAICT everything made in the past 10 years ends up using
p6_nops. Is it time to kill off ideal_nops[] and simplify life?
Well, the one bug that was reported recently was due to a box that uses a different "ideal_nops" than p6_nops. Perhaps we should ask him if there's any noticeable difference between using p6_nops for every function than the ideal_nops that as found for that box.
-- Steve
Peter Zijlstra
maintainerQuoted 5 lines · Steven Rostedt
Quoted 2 lines · On Tue, 9 Mar 2021 17:58:17 +0100 Peter Zijlstra <[email protected]> wrote:
Hi,
Quoted 8 lines
Quoted 3 lines
AFAICT everything made in the past 10 years ends up using
p6_nops. Is it time to kill off ideal_nops[] and simplify life?Well, the one bug that was reported recently was due to a box that uses a different "
ideal_nops" thanp6_nops. Perhaps we should ask him if there's any noticeable difference between usingp6_nopsfor every function than theideal_nopsthat as found for that box.
If the machine is more than a decade old, I'm not really caring about optimal performance. If it is 32bit, I really couldn't be arsed as long as it boots.
[email protected]
reviewerQuoted 14 lines · Peter Zijlstra
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Tue, 9 Mar 2021 17:58:17 +0100 Peter Zijlstra <[email protected]> wrote:
Hi,
AFAICT everything made in the past 10 years ends up using
p6_nops.Is it
Quoted 4 lines
Quoted 2 lines
time to kill off ideal_nops[] and simplify life?
Well, the one bug that was reported recently was due to a box that
uses a
Quoted 1 line
different "
ideal_nops" thanp6_nops. Perhaps we should ask him if
Quoted 8 lines
there's
Quoted 1 line
any noticeable difference between using
p6_nopsfor every functionthan the
Quoted 1 line
ideal_nopsthat as found for that box.If the machine is more than a decade old, I'm not really caring about optimal performance. If it is 32bit, I really couldn't be arsed as long as it boots.
p6_nops don't boot on all 32-bit chips.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Peter Zijlstra
maintainerQuoted 17 lines · On Tue, Mar 09, 2021 at 04:33:45PM -0800, [email protected] wrote:
Quoted 16 lines · Peter Zijlstra
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Tue, 9 Mar 2021 17:58:17 +0100 Peter Zijlstra <[email protected]> wrote:
Hi,
AFAICT everything made in the past 10 years ends up using
p6_nops.Is it
Quoted 4 lines
Quoted 2 lines
time to kill off ideal_nops[] and simplify life?
Well, the one bug that was reported recently was due to a box that
uses a
Quoted 1 line
different "
ideal_nops" thanp6_nops. Perhaps we should ask him ifthere's
Quoted 1 line
any noticeable difference between using
p6_nopsfor every function
Quoted 8 lines
Quoted 6 lines
than the
Quoted 1 line
ideal_nopsthat as found for that box.If the machine is more than a decade old, I'm not really caring about optimal performance. If it is 32bit, I really couldn't be arsed as long as it boots.
p6_nopsdon't boot on all 32-bit chips.
Sure, but we can have one set on 32bit and another set on 64bit.
$ cat nops.s .section .text nop1: .nops 1 nop2: .nops 2 nop3: .nops 3 nop4: .nops 4 nop5: .nops 5 nop6: .nops 6 nop7: .nops 7 nop8: .nops 8
$ as --32 nops.s -o nops.o ; objdump -wd nops.o
nops.o: file format elf32-i386
Disassembly of section .text:
00000000 <nop1>:
0: 90 nop
00000001 <nop2>:
1: 66 90 xchg %ax,%ax
00000003 <nop3>:
3: 8d 76 00 lea 0x0(%esi),%esi
00000006 <nop4>:
6: 8d 74 26 00 lea 0x0(%esi,%eiz,1),%esi
0000000a <nop5>:
a: 8d 74 26 00 lea 0x0(%esi,%eiz,1),%esi e: 90 nop
0000000f <nop6>:
f: 8d b6 00 00 00 00 lea 0x0(%esi),%esi
00000015 <nop7>:
15: 8d b4 26 00 00 00 00 lea 0x0(%esi,%eiz,1),%esi
0000001c <nop8>:
1c: 8d b4 26 00 00 00 00 lea 0x0(%esi,%eiz,1),%esi 23: 90 nop
$ as --64 nops.s -o nops.o ; objdump -wd nops.o
nops.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <nop1>:
0: 90 nop
0000000000000001 <nop2>:
1: 66 90 xchg %ax,%ax
0000000000000003 <nop3>:
3: 0f 1f 00 nopl (%rax)
0000000000000006 <nop4>:
6: 0f 1f 40 00 nopl 0x0(%rax)
000000000000000a <nop5>:
a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
000000000000000f <nop6>:
f: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
0000000000000015 <nop7>:
15: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
000000000000001c <nop8>:
1c: 0f 1f 84 00 00 00 00 00 nopl 0x0(%rax,%rax,1)
---
Although I would use DS prefix nops for 32bit nop5/nop8 to keep them single instructions.
Then we can do away with runtime nop selection and special atomic nops and simplify things.
All this runtime faffing about nops is tedious and causes complications we can do without.
Peter Zijlstra
maintainerQuoted 10 lines · Peter Zijlstra
Sure, but we can have one set on 32bit and another set on 64bit.
Although I would use DS prefix nops for 32bit nop5/nop8 to keep them single instructions.
Then we can do away with runtime nop selection and special atomic nops and simplify things.
All this runtime faffing about nops is tedious and causes complications we can do without.
Something like so, except I think there's more cleanups possible.
(*completely* untested)
--- arch/x86/include/asm/nops.h | 176 ++++++++++++------------------------ arch/x86/kernel/alternative.c | 199 +++++------------------------------------ arch/x86/kernel/ftrace.c | 4 +- arch/x86/kernel/jump_label.c | 21 +---- arch/x86/kernel/kprobes/core.c | 2 +- arch/x86/kernel/setup.c | 1 - arch/x86/kernel/static_call.c | 4 +- arch/x86/net/bpf_jit_comp.c | 8 +- 8 files changed, 88 insertions(+), 327 deletions(-)
arch/x86/include/asm/nops.h +55 -121
@@ -4,89 +4,58 @@/* * Define nops for use with alternative() and for tracing. -* -* *_NOP5_ATOMIC must be a single instruction. */ -#define NOP_DS_PREFIX 0x3e +#ifndef CONFIG_64BIT -/* generic versions from gas -1: nop -the following instructions are NOT nops in 64-bit mode, -for 64-bit mode use K8 or P6 nops instead -2: movl %esi,%esi -3: leal 0x00(%esi),%esi -4: leal 0x00(,%esi,1),%esi -6: leal 0x00000000(%esi),%esi -7: leal 0x00000000(,%esi,1),%esi -*/ -#define GENERIC_NOP1 0x90 -#define GENERIC_NOP2 0x89,0xf6 -#define GENERIC_NOP3 0x8d,0x76,0x00 -#define GENERIC_NOP4 0x8d,0x74,0x26,0x00 -#define GENERIC_NOP5 GENERIC_NOP1,GENERIC_NOP4 -#define GENERIC_NOP6 0x8d,0xb6,0x00,0x00,0x00,0x00 -#define GENERIC_NOP7 0x8d,0xb4,0x26,0x00,0x00,0x00,0x00 -#define GENERIC_NOP8 GENERIC_NOP1,GENERIC_NOP7 -#define GENERIC_NOP5_ATOMIC NOP_DS_PREFIX,GENERIC_NOP4 +/* +* Generic 32bit nops from GAS: +* +* 1: nop +* 2: movl %esi,%esi +* 3: leal 0x00(%esi),%esi +* 4: leal 0x00(,%esi,1),%esi +* 5: leal %ds:0x00(,%esi,1),%esi +* 6: leal 0x00000000(%esi),%esi +* 7: leal 0x00000000(,%esi,1),%esi +* 8: leal %ds:0x00000000(,%esi,1),%esi +* +* Except 5 and 8, which are DS prefixed 4 and 7 resp, where GAS would emit 2 +* nop instructions. +*/ +#define BYTES_NOP1 0x90 +#define BYTES_NOP2 0x89,0xf6 +#define BYTES_NOP3 0x8d,0x76,0x00 +#define BYTES_NOP4 0x8d,0x74,0x26,0x00 +#define BYTES_NOP5 0x3e,BYTES_NOP4 +#define BYTES_NOP6 0x8d,0xb6,0x00,0x00,0x00,0x00 +#define BYTES_NOP7 0x8d,0xb4,0x26,0x00,0x00,0x00,0x00 +#define BYTES_NOP8 0x3e,BYTES_NOP7 -/* Opteron 64bit nops -1: nop -2: osp nop -3: osp osp nop -4: osp osp osp nop -*/ -#define K8_NOP1 GENERIC_NOP1 -#define K8_NOP2 0x66,K8_NOP1 -#define K8_NOP3 0x66,K8_NOP2 -#define K8_NOP4 0x66,K8_NOP3 -#define K8_NOP5 K8_NOP3,K8_NOP2 -#define K8_NOP6 K8_NOP3,K8_NOP3 -#define K8_NOP7 K8_NOP4,K8_NOP3 -#define K8_NOP8 K8_NOP4,K8_NOP4 -#define K8_NOP5_ATOMIC 0x66,K8_NOP4 +#else -/* K7 nops -uses eax dependencies (arbitrary choice) -1: nop -2: movl %eax,%eax -3: leal (,%eax,1),%eax -4: leal 0x00(,%eax,1),%eax -6: leal 0x00000000(%eax),%eax -7: leal 0x00000000(,%eax,1),%eax -*/ -#define K7_NOP1 GENERIC_NOP1 -#define K7_NOP2 0x8b,0xc0 -#define K7_NOP3 0x8d,0x04,0x20 -#define K7_NOP4 0x8d,0x44,0x20,0x00 -#define K7_NOP5 K7_NOP4,K7_NOP1 -#define K7_NOP6 0x8d,0x80,0,0,0,0 -#define K7_NOP7 0x8D,0x04,0x05,0,0,0,0 -#define K7_NOP8 K7_NOP7,K7_NOP1 -#define K7_NOP5_ATOMIC NOP_DS_PREFIX,K7_NOP4 +/* +* Generic 64bit nops from GAS: +* +* 1: nop +* 2: osp nop +* 3: nopl (%eax) +* 4: nopl 0x00(%eax) +* 5: nopl 0x00(%eax,%eax,1) +* 6: osp nopl 0x00(%eax,%eax,1) +* 7: nopl 0x00000000(%eax) +* 8: nopl 0x00000000(%eax,%eax,1) +*/ +#define BYTES_NOP1 0x90 +#define BYTES_NOP2 0x66,0x90 +#define BYTES_NOP3 0x0f,0x1f,0x00 +#define BYTES_NOP4 0x0f,0x1f,0x40,0 +#define BYTES_NOP5 0x0f,0x1f,0x44,0x00,0 +#define BYTES_NOP6 0x66,0x0f,0x1f,0x44,0x00,0 +#define BYTES_NOP7 0x0f,0x1f,0x80,0,0,0,0 +#define BYTES_NOP8 0x0f,0x1f,0x84,0x00,0,0,0,0 -/* P6 nops -uses eax dependencies (Intel-recommended choice) -1: nop -2: osp nop -3: nopl (%eax) -4: nopl 0x00(%eax) -5: nopl 0x00(%eax,%eax,1) -6: osp nopl 0x00(%eax,%eax,1) -7: nopl 0x00000000(%eax) -8: nopl 0x00000000(%eax,%eax,1) -Note: All the above are assumed to be a single instruction. -There is kernel code that depends on this. -*/ -#define P6_NOP1 GENERIC_NOP1 -#define P6_NOP2 0x66,0x90 -#define P6_NOP3 0x0f,0x1f,0x00 -#define P6_NOP4 0x0f,0x1f,0x40,0 -#define P6_NOP5 0x0f,0x1f,0x44,0x00,0 -#define P6_NOP6 0x66,0x0f,0x1f,0x44,0x00,0 -#define P6_NOP7 0x0f,0x1f,0x80,0,0,0,0 -#define P6_NOP8 0x0f,0x1f,0x84,0x00,0,0,0,0 -#define P6_NOP5_ATOMIC P6_NOP5 +#endif /* CONFIG_64BIT */ #ifdef __ASSEMBLY__#define _ASM_MK_NOP (x) .byte x
@@ -94,54 +63,19 @@#define _ASM_MK_NOP (x) ".byte " __stringify(x) "\n"#endif -#if defined(CONFIG_MK7 ) -#define ASM_NOP1 _ASM_MK_NOP(K7_NOP1) -#define ASM_NOP2 _ASM_MK_NOP(K7_NOP2) -#define ASM_NOP3 _ASM_MK_NOP(K7_NOP3) -#define ASM_NOP4 _ASM_MK_NOP(K7_NOP4) -#define ASM_NOP5 _ASM_MK_NOP(K7_NOP5) -#define ASM_NOP6 _ASM_MK_NOP(K7_NOP6) -#define ASM_NOP7 _ASM_MK_NOP(K7_NOP7) -#define ASM_NOP8 _ASM_MK_NOP(K7_NOP8) -#define ASM_NOP5_ATOMIC _ASM_MK_NOP(K7_NOP5_ATOMIC) -#elif defined(CONFIG_X86_P6_NOP ) -#define ASM_NOP1 _ASM_MK_NOP(P6_NOP1) -#define ASM_NOP2 _ASM_MK_NOP(P6_NOP2) -#define ASM_NOP3 _ASM_MK_NOP(P6_NOP3) -#define ASM_NOP4 _ASM_MK_NOP(P6_NOP4) -#define ASM_NOP5 _ASM_MK_NOP(P6_NOP5) -#define ASM_NOP6 _ASM_MK_NOP(P6_NOP6) -#define ASM_NOP7 _ASM_MK_NOP(P6_NOP7) -#define ASM_NOP8 _ASM_MK_NOP(P6_NOP8) -#define ASM_NOP5_ATOMIC _ASM_MK_NOP(P6_NOP5_ATOMIC) -#elif defined(CONFIG_X86_64 ) -#define ASM_NOP1 _ASM_MK_NOP(K8_NOP1) -#define ASM_NOP2 _ASM_MK_NOP(K8_NOP2) -#define ASM_NOP3 _ASM_MK_NOP(K8_NOP3) -#define ASM_NOP4 _ASM_MK_NOP(K8_NOP4) -#define ASM_NOP5 _ASM_MK_NOP(K8_NOP5) -#define ASM_NOP6 _ASM_MK_NOP(K8_NOP6) -#define ASM_NOP7 _ASM_MK_NOP(K8_NOP7) -#define ASM_NOP8 _ASM_MK_NOP(K8_NOP8) -#define ASM_NOP5_ATOMIC _ASM_MK_NOP(K8_NOP5_ATOMIC) -#else -#define ASM_NOP1 _ASM_MK_NOP(GENERIC_NOP1) -#define ASM_NOP2 _ASM_MK_NOP(GENERIC_NOP2) -#define ASM_NOP3 _ASM_MK_NOP(GENERIC_NOP3) -#define ASM_NOP4 _ASM_MK_NOP(GENERIC_NOP4) -#define ASM_NOP5 _ASM_MK_NOP(GENERIC_NOP5) -#define ASM_NOP6 _ASM_MK_NOP(GENERIC_NOP6) -#define ASM_NOP7 _ASM_MK_NOP(GENERIC_NOP7) -#define ASM_NOP8 _ASM_MK_NOP(GENERIC_NOP8) -#define ASM_NOP5_ATOMIC _ASM_MK_NOP(GENERIC_NOP5_ATOMIC) -#endif +#define ASM_NOP1 _ASM_MK_NOP(BYTES_NOP1) +#define ASM_NOP2 _ASM_MK_NOP(BYTES_NOP2) +#define ASM_NOP3 _ASM_MK_NOP(BYTES_NOP3) +#define ASM_NOP4 _ASM_MK_NOP(BYTES_NOP4) +#define ASM_NOP5 _ASM_MK_NOP(BYTES_NOP5) +#define ASM_NOP6 _ASM_MK_NOP(BYTES_NOP6) +#define ASM_NOP7 _ASM_MK_NOP(BYTES_NOP7) +#define ASM_NOP8 _ASM_MK_NOP(BYTES_NOP8)#define ASM_NOP_MAX 8 -#define NOP_ATOMIC5 (ASM_NOP_MAX+1)/* Entry for the 5-byte atomic NOP */ #ifndef __ASSEMBLY__ -extern const unsigned char *const *ideal_nops; -extern void arch_init_ideal_nops (void ); +extern const unsigned char *const *x86_nops;#endif #endif /* _ASM_X86_NOPS_H */
arch/x86/kernel/alternative.c +22 -177
228 lines not on this page · raw message
arch/x86/kernel/ftrace.c +2 -2
@@ -66,7 +66,7 @@ int ftrace_arch_code_modify_post_process(void)static const char *ftrace_nop_replace (void ) { -return ideal_nops[NOP_ATOMIC5 ]; +return x86_nops[5 ]; }static const char *ftrace_call_replace (unsigned long ip,unsigned long addr)
@@ -377,7 +377,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) ip = trampoline + (jmp_offset - start_offset);if (WARN_ON (*(char *)ip !=0x75 )) goto fail; - ret =copy_from_kernel_nofault (ip, ideal_nops[2 ],2 ); + ret =copy_from_kernel_nofault (ip, x86_nops[2 ],2 );if (ret <0 ) goto fail; }
arch/x86/kernel/jump_label.c +2 -19
@@ -31,7 +31,6 @@ static const void * __jump_label_set_jump_code(struct jump_entry *entry, enum jump_label_type type, int init) { const unsigned char default_nop[] = { STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ];const void *expect, *code;const void *addr, *dest;int line;
@@ -44,7 +43,7 @@ __jump_label_set_jump_code(struct jump_entry *entry, enum jump_label_type type,if (init) { expect = default_nop; line = __LINE__; }else if (type ==JUMP_LABEL_JMP ) { - expect = ideal_nop; line = __LINE__; + expect = x86_nops[5 ]; line = __LINE__; }else { expect = code; line = __LINE__; }
@@ -53,7 +52,7 @@ __jump_label_set_jump_code(struct jump_entry *entry, enum jump_label_type type,bug_at (addr, line);if (type ==JUMP_LABEL_NOP ) - code = ideal_nop; + code = x86_nops[5 ];return code; }
@@ -136,22 +135,6 @@ static enum { __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry,enum jump_label_type type) { -/* -* This function is called at boot up and when modules are -* first loaded. Check if the default nop, the one that is -* inserted at compile time, is the ideal nop. If it is, then -* we do not need to update the nop, and we can leave it as is. -* If it is not, then we need to update the nop to the ideal nop. -*/ -if (jlstate ==JL_STATE_START ) { -const unsigned char default_nop[] = {STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ]; - -if (memcmp (ideal_nop, default_nop,5 ) !=0 ) - jlstate =JL_STATE_UPDATE ; -else - jlstate =JL_STATE_NO_UPDATE ; - }if (jlstate ==JL_STATE_UPDATE )jump_label_transform (entry, type,1 ); }
arch/x86/kernel/kprobes/core.c +1 -1
@@ -229,7 +229,7 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)return 0UL ;if (faddr) -memcpy (buf, ideal_nops[NOP_ATOMIC5 ],5 ); +memcpy (buf, x86_nops[5 ],5 );else buf[0 ] = kp->opcode;return (unsigned long )buf;
arch/x86/kernel/setup.c +0 -1
@@ -822,7 +822,6 @@ void __init setup_arch(char **cmdline_p)idt_setup_early_traps ();early_cpu_init (); -arch_init_ideal_nops ();jump_label_init ();static_call_init ();early_ioremap_init ();
arch/x86/kernel/static_call.c +2 -2
@@ -34,7 +34,7 @@ static void __ref __static_call_transform(void *insn, enum insn_type type, voidbreak ;case NOP : - code = ideal_nops[NOP_ATOMIC5 ]; + code = x86_nops[5 ];break ;case JMP :
@@ -66,7 +66,7 @@ static void __static_call_validate(void *insn, bool tail)return ; }else {if (opcode ==CALL_INSN_OPCODE || - !memcmp (insn, ideal_nops[NOP_ATOMIC5 ],5 ) || + !memcmp (insn, x86_nops[5 ],5 ) || !memcmp (insn, xor5rax,5 ))return ; }
arch/x86/net/bpf_jit_comp.c +4 -4
@@ -282,7 +282,7 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf,/* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ -memcpy (prog, ideal_nops[NOP_ATOMIC5 ], cnt); +memcpy (prog, x86_nops[5 ], cnt); prog += cnt;if (!ebpf_from_cbpf) {if (tail_call_reachable && !is_subprog)
@@ -330,7 +330,7 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *old_addr, void *new_addr, const bool text_live) { - const u8 *nop_insn = ideal_nops[NOP_ATOMIC5]; + const u8 *nop_insn = x86_nops[5];u8 old_insn[X86_PATCH_SIZE ];u8 new_insn[X86_PATCH_SIZE ];u8 *prog;
@@ -560,7 +560,7 @@ static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke,if (stack_depth)EMIT3_off32 (0x48 ,0x81 ,0xC4 ,round_up (stack_depth,8 )); -memcpy (prog, ideal_nops[NOP_ATOMIC5 ],X86_PATCH_SIZE ); +memcpy (prog, x86_nops[5 ],X86_PATCH_SIZE ); prog +=X86_PATCH_SIZE ;/* out: */
@@ -881,7 +881,7 @@ static int emit_nops(u8 **pprog, int len) noplen =ASM_NOP_MAX ;for (i =0 ; i < noplen; i++) -EMIT1 (ideal_nops[noplen][i]); +EMIT1 (x86_nops[noplen][i]); len -= noplen; }
Peter Zijlstra
maintainerQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Sure, but we can have one set on 32bit and another set on 64bit.
Although I would use DS prefix nops for 32bit nop5/nop8 to keep them single instructions.
Quoted 7 lines
Quoted 5 lines
Then we can do away with runtime nop selection and special atomic nops and simplify things.
All this runtime faffing about nops is tedious and causes complications we can do without.
Something like so, except I think there's more cleanups possible.
This one builds and boots x86_64-defconfig. Still, more cleanups are possible.
--- arch/x86/include/asm/jump_label.h | 12 +-- arch/x86/include/asm/nops.h | 176 ++++++++++--------------------- arch/x86/include/asm/special_insns.h | 4 +- arch/x86/kernel/alternative.c | 198 ++++------------------------------- arch/x86/kernel/ftrace.c | 4 +- arch/x86/kernel/jump_label.c | 23 +--- arch/x86/kernel/kprobes/core.c | 2 +- arch/x86/kernel/setup.c | 1 - arch/x86/kernel/static_call.c | 4 +- arch/x86/net/bpf_jit_comp.c | 8 +- 10 files changed, 93 insertions(+), 339 deletions(-)
arch/x86/include/asm/jump_label.h +3 -9
@@ -6,12 +6,6 @@#define JUMP_LABEL_NOP_SIZE 5 -#ifdef CONFIG_X86_64 -# define STATIC_KEY_INIT_NOP P6_NOP5_ATOMIC -#else -# define STATIC_KEY_INIT_NOP GENERIC_NOP5_ATOMIC -#endif -#include <asm/asm.h> #include <asm/nops.h>
@@ -23,7 +17,7 @@ static __always_inline bool arch_static_branch(struct static_key * const key, const bool branch) { asm_volatile_goto("1:" - ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" + ".byte " __stringify(BYTES_NOP5) "\n\t"".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN"\n\t" ".long 1b - ., %l[l_yes] - . \n\t"
@@ -63,7 +57,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key * const ke .long \target - .Lstatic_jump_after_\@ .Lstatic_jump_after_\@: .else - .byte STATIC_KEY_INIT_NOP + .byte BYTES_NOP5 .endif .pushsection __jump_table, "aw" _ASM_ALIGN
@@ -75,7 +69,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key * const ke .macro STATIC_JUMP_IF_FALSE target, key, def .Lstatic_jump_\@: .if \def - .byte STATIC_KEY_INIT_NOP + .byte BYTES_NOP5 .else/* Equivalent to "jmp.d32 \target" */ .byte 0xe9
arch/x86/include/asm/nops.h +55 -121
198 lines not on this page · raw message
arch/x86/include/asm/special_insns.h +2 -2
@@ -214,7 +214,7 @@ static inline void clflush(volatile void *__p)static inline void clflushopt (volatile void *__p) { - alternative_io(".byte " __stringify(NOP_DS_PREFIX) "; clflush %P0", +alternative_io (".byte 0x3e; clflush %P0" ,".byte 0x66; clflush %P0" ,X86_FEATURE_CLFLUSHOPT ,"+m" (*(volatile char __force *)__p));
@@ -225,7 +225,7 @@ static inline void clwb(volatile void *__p)volatile struct {char x[64 ]; } *p = __p; asm volatile(ALTERNATIVE_2( - ".byte " __stringify(NOP_DS_PREFIX) "; clflush (%[pax])", + ".byte 0x3e; clflush (%[pax])",".byte 0x66; clflush (%[pax])" ,/* clflushopt (%%rax) */ X86_FEATURE_CLFLUSHOPT , ".byte 0x66, 0x0f, 0xae, 0x30", /* clwb (%%rax) */
arch/x86/kernel/alternative.c +21 -177
227 lines not on this page · raw message
arch/x86/kernel/ftrace.c +2 -2
@@ -66,7 +66,7 @@ int ftrace_arch_code_modify_post_process(void)static const char *ftrace_nop_replace (void ) { -return ideal_nops[NOP_ATOMIC5 ]; +return x86_nops[5 ]; }static const char *ftrace_call_replace (unsigned long ip,unsigned long addr)
@@ -377,7 +377,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) ip = trampoline + (jmp_offset - start_offset);if (WARN_ON (*(char *)ip !=0x75 )) goto fail; - ret =copy_from_kernel_nofault (ip, ideal_nops[2 ],2 ); + ret =copy_from_kernel_nofault (ip, x86_nops[2 ],2 );if (ret <0 ) goto fail; }
arch/x86/kernel/jump_label.c +3 -20
@@ -30,8 +30,7 @@ static void bug_at(const void *ip, int line)static const void *__jump_label_set_jump_code (struct jump_entry *entry,enum jump_label_type type,int init) { -const unsigned char default_nop[] = {STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ]; +const unsigned char default_nop[] = {BYTES_NOP5 };const void *expect, *code;const void *addr, *dest;int line;
@@ -44,7 +43,7 @@ __jump_label_set_jump_code(struct jump_entry *entry, enum jump_label_type type,if (init) { expect = default_nop; line = __LINE__; }else if (type ==JUMP_LABEL_JMP ) { - expect = ideal_nop; line = __LINE__; + expect = x86_nops[5 ]; line = __LINE__; }else { expect = code; line = __LINE__; }
@@ -53,7 +52,7 @@ __jump_label_set_jump_code(struct jump_entry *entry, enum jump_label_type type,bug_at (addr, line);if (type ==JUMP_LABEL_NOP ) - code = ideal_nop; + code = x86_nops[5 ];return code; }
@@ -136,22 +135,6 @@ static enum { __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry,enum jump_label_type type) { -/* -* This function is called at boot up and when modules are -* first loaded. Check if the default nop, the one that is -* inserted at compile time, is the ideal nop. If it is, then -* we do not need to update the nop, and we can leave it as is. -* If it is not, then we need to update the nop to the ideal nop. -*/ -if (jlstate ==JL_STATE_START ) { -const unsigned char default_nop[] = {STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ]; - -if (memcmp (ideal_nop, default_nop,5 ) !=0 ) - jlstate =JL_STATE_UPDATE ; -else - jlstate =JL_STATE_NO_UPDATE ; - }if (jlstate ==JL_STATE_UPDATE )jump_label_transform (entry, type,1 ); }
arch/x86/kernel/kprobes/core.c +1 -1
@@ -229,7 +229,7 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)return 0UL ;if (faddr) -memcpy (buf, ideal_nops[NOP_ATOMIC5 ],5 ); +memcpy (buf, x86_nops[5 ],5 );else buf[0 ] = kp->opcode;return (unsigned long )buf;
arch/x86/kernel/setup.c +0 -1
@@ -822,7 +822,6 @@ void __init setup_arch(char **cmdline_p)idt_setup_early_traps ();early_cpu_init (); -arch_init_ideal_nops ();jump_label_init ();static_call_init ();early_ioremap_init ();
arch/x86/kernel/static_call.c +2 -2
@@ -34,7 +34,7 @@ static void __ref __static_call_transform(void *insn, enum insn_type type, voidbreak ;case NOP : - code = ideal_nops[NOP_ATOMIC5 ]; + code = x86_nops[5 ];break ;case JMP :
@@ -66,7 +66,7 @@ static void __static_call_validate(void *insn, bool tail)return ; }else {if (opcode ==CALL_INSN_OPCODE || - !memcmp (insn, ideal_nops[NOP_ATOMIC5 ],5 ) || + !memcmp (insn, x86_nops[5 ],5 ) || !memcmp (insn, xor5rax,5 ))return ; }
arch/x86/net/bpf_jit_comp.c +4 -4
@@ -282,7 +282,7 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf,/* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ -memcpy (prog, ideal_nops[NOP_ATOMIC5 ], cnt); +memcpy (prog, x86_nops[5 ], cnt); prog += cnt;if (!ebpf_from_cbpf) {if (tail_call_reachable && !is_subprog)
@@ -330,7 +330,7 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *old_addr, void *new_addr, const bool text_live) { - const u8 *nop_insn = ideal_nops[NOP_ATOMIC5]; + const u8 *nop_insn = x86_nops[5];u8 old_insn[X86_PATCH_SIZE ];u8 new_insn[X86_PATCH_SIZE ];u8 *prog;
@@ -560,7 +560,7 @@ static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke,if (stack_depth)EMIT3_off32 (0x48 ,0x81 ,0xC4 ,round_up (stack_depth,8 )); -memcpy (prog, ideal_nops[NOP_ATOMIC5 ],X86_PATCH_SIZE ); +memcpy (prog, x86_nops[5 ],X86_PATCH_SIZE ); prog +=X86_PATCH_SIZE ;/* out: */
@@ -881,7 +881,7 @@ static int emit_nops(u8 **pprog, int len) noplen =ASM_NOP_MAX ;for (i =0 ; i < noplen; i++) -EMIT1 (ideal_nops[noplen][i]); +EMIT1 (x86_nops[noplen][i]); len -= noplen; }
Peter Zijlstra
maintainerQuoted 42 lines · Peter Zijlstra
arch/x86/kernel/alternative.c +0
-42-void __init arch_init_ideal_nops (void ) -{ -switch (boot_cpu_data.x86_vendor) { -case X86_VENDOR_INTEL : -/* -* Due to a decoder implementation quirk, some -* specific Intel CPUs actually perform better with -* the "k8_nops" than with the SDM-recommended NOPs. -*/ -if (boot_cpu_data.x86 ==6 && - boot_cpu_data.x86_model >=0x0f && - boot_cpu_data.x86_model !=0x1c && - boot_cpu_data.x86_model !=0x26 && - boot_cpu_data.x86_model !=0x27 && - boot_cpu_data.x86_model <0x30 ) { - ideal_nops = k8_nops; - }else if (boot_cpu_has (X86_FEATURE_NOPL )) { - ideal_nops = p6_nops; - }else { -#ifdef CONFIG_X86_64 - ideal_nops = k8_nops; -#else - ideal_nops = intel_nops; -#endif - } -break ; - -case X86_VENDOR_HYGON : - ideal_nops = p6_nops; -return ; - -case X86_VENDOR_AMD : -if (boot_cpu_data.x86 >0xf ) { - ideal_nops = p6_nops; -return ; - } - - fallthrough; - -default : -#ifdef CONFIG_X86_64 - ideal_nops = k8_nops;
arch/x86/kernel/alternative.c do { \ · line 256
Quoted 10 lines
arch/x86/kernel/alternative.c +0
-10-#else -if (boot_cpu_has (X86_FEATURE_K8 )) - ideal_nops = k8_nops; -else if (boot_cpu_has (X86_FEATURE_K7 )) - ideal_nops = k7_nops; -else - ideal_nops = intel_nops; -#endif - } -}
After this FEATURE_NOPL is unused except for required-features for x86_64. FEATURE_K8 is only used for PTI and FEATURE_K7 is unused.
AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD (K8) which is from 2003 and almost 2 decades old by now (SHRUG).
Everything x86_64 since AMD K10 (2007) was using p6_nops.
And per FEATURE_NOPL being required for x86_64, all those CPUs can use p6_nops. So stop caring about NOPs, simplify things and get on with life :-)
Steven Rostedt
maintainerQuoted 7 lines · On Wed, 10 Mar 2021 11:22:48 +0100 Peter Zijlstra <[email protected]> wrote:
After this
FEATURE_NOPLis unused except for required-features forx86_64.FEATURE_K8is only used for PTI andFEATURE_K7is unused.AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD (K8) which is from 2003 and almost 2 decades old by now (SHRUG).
Quoted 5 lines
Everything
x86_64since AMD K10 (2007) was usingp6_nops.And per
FEATURE_NOPLbeing required forx86_64, all those CPUs can usep6_nops. So stop caring about NOPs, simplify things and get on with life :-)
Before ripping out all the ideal_nop logic, I wonder if we should just force the nops you want now (that is, don't change the selected ideal_nops, just "pretend" that the CPU wants p6_nops), and see if anyone complains. After a few releases, if there's no complaints, then we can rip out the ideal_nop logic.
-- Steve
Peter Zijlstra
maintainerQuoted 16 lines · Steven Rostedt
Quoted 7 lines · On Wed, 10 Mar 2021 11:22:48 +0100 Peter Zijlstra <[email protected]> wrote:
After this
FEATURE_NOPLis unused except for required-features forx86_64.FEATURE_K8is only used for PTI andFEATURE_K7is unused.AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD (K8) which is from 2003 and almost 2 decades old by now (SHRUG).
Quoted 5 lines
Everything
x86_64since AMD K10 (2007) was usingp6_nops.And per
FEATURE_NOPLbeing required forx86_64, all those CPUs can usep6_nops. So stop caring about NOPs, simplify things and get on with life :-)
Quoted 5 lines
Before ripping out all the
ideal_noplogic, I wonder if we should just force the nops you want now (that is, don't change the selectedideal_nops, just "pretend" that the CPU wantsp6_nops), and see if anyone complains. After a few releases, if there's no complaints, then we can rip out theideal_noplogic.
Nah, just rip the entire thing out. You should be happy about deterministic NOPs :-)
NOP encoding is not something CPUs should differentiate on, that's just bollocks.
Peter Zijlstra
maintainerQuoted 23 lines · Peter Zijlstra
Quoted 16 lines · Steven Rostedt
Quoted 7 lines · On Wed, 10 Mar 2021 11:22:48 +0100 Peter Zijlstra <[email protected]> wrote:
After this
FEATURE_NOPLis unused except for required-features forx86_64.FEATURE_K8is only used for PTI andFEATURE_K7is unused.AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD (K8) which is from 2003 and almost 2 decades old by now (SHRUG).
Quoted 5 lines
Everything
x86_64since AMD K10 (2007) was usingp6_nops.And per
FEATURE_NOPLbeing required forx86_64, all those CPUs can usep6_nops. So stop caring about NOPs, simplify things and get on with life :-)Quoted 5 lines
Before ripping out all the
ideal_noplogic, I wonder if we should just force the nops you want now (that is, don't change the selectedideal_nops, just "pretend" that the CPU wantsp6_nops), and see if anyone complains. After a few releases, if there's no complaints, then we can rip out theideal_noplogic.
Quoted 5 lines
Nah, just rip the entire thing out. You should be happy about deterministic NOPs :-)
NOP encoding is not something CPUs should differentiate on, that's just bollocks.
Also, you seem to have fallen off of IRC. Anyway, weren't you the one that was complaining x86 was 'special' for having different NOPs the other day?
Fixed it ;-)
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Steven Rostedt
Quoted 12 lines · On Wed, 10 Mar 2021 11:22:48 +0100 Peter Zijlstra <[email protected]> wrote:
After this
FEATURE_NOPLis unused except for required-features forx86_64.FEATURE_K8is only used for PTI andFEATURE_K7is unused.AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD (K8) which is from 2003 and almost 2 decades old by now (SHRUG).
Everything
x86_64since AMD K10 (2007) was usingp6_nops.And per
FEATURE_NOPLbeing required forx86_64, all those CPUs can usep6_nops. So stop caring about NOPs, simplify things and get on with life :-)
Quoted 8 lines
Quoted 5 lines
Before ripping out all the
ideal_noplogic, I wonder if we should just force the nops you want now (that is, don't change the selectedideal_nops, just "pretend" that the CPU wantsp6_nops), and see if anyone complains. After a few releases, if there's no complaints, then we can rip out theideal_noplogic.Nah, just rip the entire thing out. You should be happy about deterministic NOPs :-)
Ack for bpf bits.
I think the cleanup is good from the point of having one way to do things. Though I won't be surprised if somebody comes along with a patch to use different nops eventually.
When I first looked at it years ago I was wondering why segment selector prefix is not used. afaik windows was using it, because having cs: ds: nop makes intel use only one of the instruction decoders (the big and slow one) which allegedly saves power, since the pipeline has bubbles. Things could be completely different now in modern u-arches.
Peter Zijlstra
maintainerQuoted 1 line · Alexei Starovoitov
Ack for bpf bits.
Thanks!
Quoted 1 line
I think the cleanup is good from the point of having one way to do things.
Right, that was the motivation. Currently x86 is the sole architecture (and thus weird and more complicated) where NOPs vary at runtime.
Quoted 2 lines
Though I won't be surprised if somebody comes along with a patch to use different nops eventually.
I don't particularly care about the exact NOP encoding, as long as we can all agree on it at build time. Having to either do boot time NOP rewrites of everything or runtime compare against multiple NOPs is complexity we don't need.
(and ideally the kernel and gas/clang .nops directives should agree with one another, although that's not the case with the below patch for 32bit).
Quoted 5 lines
When I first looked at it years ago I was wondering why segment selector prefix is not used. afaik windows was using it, because having cs: ds: nop makes intel use only one of the instruction decoders (the big and slow one) which allegedly saves power, since the pipeline has bubbles. Things could be completely different now in modern u-arches.
So we do use DS prefix for NOP[58] on 32bit. 64bit seems to prefer OSP prefixes.
Below is the latest version which I just pushed out to my git tree so that the robots can have a go at it.
---
Subject: x86: Remove dynamic NOP selection
From: Peter Zijlstra <[email protected]>
Date: Wed Mar 10 11:43:35 CET 2021
This ensures that a NOP is a NOP and not a random other instruction that is also a NOP. It allows simplification of dynamic code patching that wants to verify existing code before writing new instructions (ftrace, jump_label, static_call, etc..).
Differentiating on NOPs is not a 'feature'.
After this FEATURE_NOPL is unused except for required-features for x86_64. FEATURE_K8 is only used for PTI and FEATURE_K7 is unused.
AFAICT this negatively affects lots of 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS) and early AMD K8 which is from 2003 and almost 2 decades old by now (SHRUG).
Everything x86_64 since AMD K10 (2007) is using p6_nops.
And per FEATURE_NOPL being required for x86_64, all those CPUs can use p6_nops. So stop caring about NOPs, simplify things and get on with life.
(more cleanups possible)
--- arch/x86/include/asm/jump_label.h | 12 -- arch/x86/include/asm/nops.h | 180 ++++++++++--------------------- arch/x86/include/asm/special_insns.h | 4 arch/x86/kernel/alternative.c | 198 +++-------------------------------- arch/x86/kernel/ftrace.c | 4 arch/x86/kernel/jump_label.c | 32 +---- arch/x86/kernel/kprobes/core.c | 2 arch/x86/kernel/setup.c | 1 arch/x86/kernel/static_call.c | 4 arch/x86/net/bpf_jit_comp.c | 8 - 10 files changed, 98 insertions(+), 347 deletions(-)
arch/x86/include/asm/jump_label.h +3 -9
@@ -6,12 +6,6 @@#define JUMP_LABEL_NOP_SIZE 5 -#ifdef CONFIG_X86_64 -# define STATIC_KEY_INIT_NOP P6_NOP5_ATOMIC -#else -# define STATIC_KEY_INIT_NOP GENERIC_NOP5_ATOMIC -#endif -#include <asm/asm.h> #include <asm/nops.h>
@@ -23,7 +17,7 @@ static __always_inline bool arch_static_branch(struct static_key * const key, const bool branch) { asm_volatile_goto("1:" - ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" + ".byte " __stringify(BYTES_NOP5) "\n\t"".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN"\n\t" ".long 1b - ., %l[l_yes] - . \n\t"
@@ -63,7 +57,7 @@ static __always_inline bool arch_static_ .long \target - .Lstatic_jump_after_\@ .Lstatic_jump_after_\@: .else - .byte STATIC_KEY_INIT_NOP + .byte BYTES_NOP5 .endif .pushsection __jump_table, "aw" _ASM_ALIGN
@@ -75,7 +69,7 @@ static __always_inline bool arch_static_ .macro STATIC_JUMP_IF_FALSE target, key, def .Lstatic_jump_\@: .if \def - .byte STATIC_KEY_INIT_NOP + .byte BYTES_NOP5 .else/* Equivalent to "jmp.d32 \target" */ .byte 0xe9
arch/x86/include/asm/nops.h +57 -123
200 lines not on this page · raw message
arch/x86/include/asm/special_insns.h +2 -2
@@ -214,7 +214,7 @@ static inline void clflush(volatile voidstatic inline void clflushopt (volatile void *__p) { - alternative_io(".byte " __stringify(NOP_DS_PREFIX) "; clflush %P0", +alternative_io (".byte 0x3e; clflush %P0" ,".byte 0x66; clflush %P0" ,X86_FEATURE_CLFLUSHOPT ,"+m" (*(volatile char __force *)__p));
@@ -225,7 +225,7 @@ static inline void clwb(volatile void *_volatile struct {char x[64 ]; } *p = __p; asm volatile(ALTERNATIVE_2( - ".byte " __stringify(NOP_DS_PREFIX) "; clflush (%[pax])", + ".byte 0x3e; clflush (%[pax])",".byte 0x66; clflush (%[pax])" ,/* clflushopt (%%rax) */ X86_FEATURE_CLFLUSHOPT , ".byte 0x66, 0x0f, 0xae, 0x30", /* clwb (%%rax) */
arch/x86/kernel/alternative.c +21 -177
227 lines not on this page · raw message
arch/x86/kernel/ftrace.c +2 -2
@@ -66,7 +66,7 @@ int ftrace_arch_code_modify_post_processstatic const char *ftrace_nop_replace (void ) { -return ideal_nops[NOP_ATOMIC5 ]; +return x86_nops[5 ]; }static const char *ftrace_call_replace (unsigned long ip,unsigned long addr)
@@ -377,7 +377,7 @@ create_trampoline(struct ftrace_ops *ops ip = trampoline + (jmp_offset - start_offset);if (WARN_ON (*(char *)ip !=0x75 )) goto fail; - ret =copy_from_kernel_nofault (ip, ideal_nops[2 ],2 ); + ret =copy_from_kernel_nofault (ip, x86_nops[2 ],2 );if (ret <0 ) goto fail; }
arch/x86/kernel/jump_label.c +6 -26
@@ -28,10 +28,8 @@ static void bug_at(const void *ip, int l }static const void * -__jump_label_set_jump_code (struct jump_entry *entry,enum jump_label_type type,int init) +__jump_label_set_jump_code (struct jump_entry *entry,enum jump_label_type type) { -const unsigned char default_nop[] = {STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ];const void *expect, *code;const void *addr, *dest;int line;
@@ -41,10 +39,8 @@ __jump_label_set_jump_code(struct jump_e code =text_gen_insn (JMP32_INSN_OPCODE , addr, dest); -if (init) { - expect = default_nop; line = __LINE__; - }else if (type ==JUMP_LABEL_JMP ) { - expect = ideal_nop; line = __LINE__; +if (type ==JUMP_LABEL_JMP ) { + expect = x86_nops[5 ]; line = __LINE__; }else { expect = code; line = __LINE__; }
@@ -53,7 +49,7 @@ __jump_label_set_jump_code(struct jump_ebug_at (addr, line);if (type ==JUMP_LABEL_NOP ) - code = ideal_nop; + code = x86_nops[5 ];return code; }
@@ -62,7 +58,7 @@ static inline void __jump_label_transfor enum jump_label_type type, int init) { -const void *opcode =__jump_label_set_jump_code (entry, type, init); +const void *opcode =__jump_label_set_jump_code (entry, type);/* * As long as only a single processor is running and the code is still
@@ -113,7 +109,7 @@ bool arch_jump_label_transform_queue(str }mutex_lock (&text_mutex); - opcode =__jump_label_set_jump_code (entry, type,0 ); + opcode =__jump_label_set_jump_code (entry, type);text_poke_queue ((void *)jump_entry_code (entry), opcode,JUMP_LABEL_NOP_SIZE ,NULL );mutex_unlock (&text_mutex);
@@ -136,22 +132,6 @@ static enum { __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry,enum jump_label_type type) { -/* -* This function is called at boot up and when modules are -* first loaded. Check if the default nop, the one that is -* inserted at compile time, is the ideal nop. If it is, then -* we do not need to update the nop, and we can leave it as is. -* If it is not, then we need to update the nop to the ideal nop. -*/ -if (jlstate ==JL_STATE_START ) { -const unsigned char default_nop[] = {STATIC_KEY_INIT_NOP }; -const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5 ]; - -if (memcmp (ideal_nop, default_nop,5 ) !=0 ) - jlstate =JL_STATE_UPDATE ; -else - jlstate =JL_STATE_NO_UPDATE ; - }if (jlstate ==JL_STATE_UPDATE )jump_label_transform (entry, type,1 ); }
arch/x86/kernel/kprobes/core.c +1 -1
@@ -229,7 +229,7 @@ __recover_probed_insn(kprobe_opcode_t *breturn 0UL ;if (faddr) -memcpy (buf, ideal_nops[NOP_ATOMIC5 ],5 ); +memcpy (buf, x86_nops[5 ],5 );else buf[0 ] = kp->opcode;return (unsigned long )buf;
arch/x86/kernel/setup.c +0 -1
@@ -822,7 +822,6 @@ void __init setup_arch(char **cmdline_p)idt_setup_early_traps ();early_cpu_init (); -arch_init_ideal_nops ();jump_label_init ();static_call_init ();early_ioremap_init ();
arch/x86/kernel/static_call.c +2 -2
@@ -34,7 +34,7 @@ static void __ref __static_call_transforbreak ;case NOP : - code = ideal_nops[NOP_ATOMIC5 ]; + code = x86_nops[5 ];break ;case JMP :
@@ -66,7 +66,7 @@ static void __static_call_validate(voidreturn ; }else {if (opcode ==CALL_INSN_OPCODE || - !memcmp (insn, ideal_nops[NOP_ATOMIC5 ],5 ) || + !memcmp (insn, x86_nops[5 ],5 ) || !memcmp (insn, xor5rax,5 ))return ; }
arch/x86/net/bpf_jit_comp.c +4 -4
@@ -282,7 +282,7 @@ static void emit_prologue(u8 **pprog, u3/* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ -memcpy (prog, ideal_nops[NOP_ATOMIC5 ], cnt); +memcpy (prog, x86_nops[5 ], cnt); prog += cnt;if (!ebpf_from_cbpf) {if (tail_call_reachable && !is_subprog)
@@ -330,7 +330,7 @@ static int __bpf_arch_text_poke(void *ip void *old_addr, void *new_addr, const bool text_live) { - const u8 *nop_insn = ideal_nops[NOP_ATOMIC5]; + const u8 *nop_insn = x86_nops[5];u8 old_insn[X86_PATCH_SIZE ];u8 new_insn[X86_PATCH_SIZE ];u8 *prog;
@@ -560,7 +560,7 @@ static void emit_bpf_tail_call_direct(stif (stack_depth)EMIT3_off32 (0x48 ,0x81 ,0xC4 ,round_up (stack_depth,8 )); -memcpy (prog, ideal_nops[NOP_ATOMIC5 ],X86_PATCH_SIZE ); +memcpy (prog, x86_nops[5 ],X86_PATCH_SIZE ); prog +=X86_PATCH_SIZE ;/* out: */
@@ -881,7 +881,7 @@ static int emit_nops(u8 **pprog, int len noplen =ASM_NOP_MAX ;for (i =0 ; i < noplen; i++) -EMIT1 (ideal_nops[noplen][i]); +EMIT1 (x86_nops[noplen][i]); len -= noplen; }
From: Peter Zijlstra <[email protected]>
...
Quoted 2 lines
Below is the latest version which I just pushed out to my git tree so that the robots can have a go at it.
Why not delete the indirection table?
So you end up with:
Quoted 16 lines
arch/x86/include/asm/nops.h +14
-1+#ifndef CONFIG_64BIT + +/* +* Generic 32bit nops from GAS: +* +* 1: nop +* 2: movl %esi,%esi +* 3: leal 0x00(%esi),%esi +* 4: leal 0x00(,%esi,1),%esi +* 5: leal %ds:0x00(,%esi,1),%esi +* 6: leal 0x00000000(%esi),%esi +* 7: leal 0x00000000(,%esi,1),%esi +* 8: leal %ds:0x00000000(,%esi,1),%esi * -* *_NOP5_ATOMIC must be a single instruction. +* Except 5 and 8, which are DS prefixed 4 and 7 resp, where GAS would emit 2
arch/x86/include/asm/nops.h line 33
Quoted 10 lines
arch/x86/include/asm/nops.h +9
-0+* nop instructions. */ +#define BYTES_NOP1 0x90 +#define BYTES_NOP2 0x89,0xf6 +#define BYTES_NOP3 0x8d,0x76,0x00 +#define BYTES_NOP4 0x8d,0x74,0x26,0x00 +#define BYTES_NOP5 0x3e,BYTES_NOP4 +#define BYTES_NOP6 0x8d,0xb6,0x00,0x00,0x00,0x00 +#define BYTES_NOP7 0x8d,0xb4,0x26,0x00,0x00,0x00,0x00 +#define BYTES_NOP8 0x3e,BYTES_NOP7
const unsigned char const x86_nops[8][8] = {
{ BYTES_NOP1 },
{ BYTES_NOP2 },
{ BYTES_NOP3 },
{ BYTES_NOP4 },
{ BYTES_NOP5 },
{ BYTES_NOP6 },
{ BYTES_NOP7 },
{ BYTES_NOP8 }
};The rest of the patch may not need changing.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 2 lines · Peter Zijlstra
Below is the latest version which I just pushed out to my git tree so that the robots can have a go at it.
Do you want me to quickly check it with a real i486, or is it already
covered by said robots (NB I wouldn't trust QEMU with such stuff)?
Maciej