Skip to contentSkip to conversation

x86: Rewrite the retpoline rewrite logic

PATCH Superseded Peter Zijlstra 8 participants 45 messages Oct 20, 2021 – Oct 26, 2021

Subsystems (7)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

v2 cover x86: Rewrite the retpoline rewrite logic

Hi,

These patches rewrite the way retpolines are rewritten. Currently objtool emits alternative entries for most retpoline calls. However trying to extend that led to trouble (ELF files are horrid).

Therefore completely overhaul this and have objtool emit a .retpoline_sites section that lists all compiler generated retpoline thunk calls. Then the kernel can do with them as it pleases.

Notably it will:

- rewrite them to indirect instructions for !RETPOLINE
- rewrite them to lfence; indirect; for RETPOLINE_AMD, where size allows (boo clang!)

Specifically, the !RETPOLINE case can now also deal with the clang-special conditional-indirect-tail-call:

Jcc __x86_indirect_thunk_\reg.

Finally, also update the x86 BPF jit to catch up to recent times and do these same things.

All this should help improve performance by removing an indirection.

Patches can also be found here:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git objtool/core

Changes since v1:
- objtool: avoid self-modifying-code in .altinstr_replacement
- objtool: shrink struct instruction
- objtool: more cleanups
- x86: more #ifdef
- x86: GEN-for-each-reg cleanups/comments
- x86: retpoline thunk array
- x86,bpf: more complete retpoline replacement
- build fixes

---
 arch/um/kernel/um_arch.c                |   4 +
 arch/x86/include/asm/GEN-for-each-reg.h |  13 ++-
 arch/x86/include/asm/alternative.h      |   1 +
 arch/x86/include/asm/asm-prototypes.h   |   6 +-
 arch/x86/include/asm/nospec-branch.h    |  59 ----------
 arch/x86/kernel/alternative.c           | 192 ++++++++++++++++++++++++++++++-
 arch/x86/kernel/cpu/bugs.c              |   7 --
 arch/x86/kernel/module.c                |   9 +-
 arch/x86/kernel/vmlinux.lds.S           |  14 +++
 arch/x86/lib/retpoline.S                |  50 ++------
 arch/x86/net/bpf_jit_comp.c             |  71 +++++++-----
 arch/x86/net/bpf_jit_comp32.c           |  22 +++-
 tools/objtool/arch/x86/decode.c         | 120 --------------------
 tools/objtool/check.c                   | 195 ++++++++++++++++++++++----------
 tools/objtool/elf.c                     |  84 --------------
 tools/objtool/include/objtool/check.h   |   1 -
 tools/objtool/include/objtool/elf.h     |   2 +-
 tools/objtool/special.c                 |   8 --
 18 files changed, 431 insertions(+), 427 deletions(-)
author maintainer
#1 lore raw

PATCH v2 1/14 objtool: Tag retpoline thunk symbols

In order to avoid calling arch_is_retpoline() (which does a strcmp) over and over on symbols, tag them once upfront.

XXX do we also want to do __fentry__ ?
XXX do we want an enum instead of a bunch of bools ?

---
 tools/objtool/check.c               |   17 +++++++++++------
 tools/objtool/include/objtool/elf.h |    1 +
 2 files changed, 12 insertions(+), 6 deletions(-)
tools/objtool/check.c +11 -6
@@ -1077,7 +1077,7 @@ static int add_jump_destinations(struct
 		} else if (reloc->sym->type == STT_SECTION) {
 			dest_sec = reloc->sym->sec;
 			dest_off = arch_dest_reloc_offset(reloc->addend);
-		} else if (arch_is_retpoline(reloc->sym)) {
+		} else if (reloc->sym->retpoline_thunk) {
 			/*
 			 * Retpoline jumps are really dynamic jumps in
 			 * disguise, so convert them accordingly.
@@ -1218,7 +1218,7 @@ static int add_call_destinations(struct
 
 			add_call_dest(file, insn, dest, false);
 
-		} else if (arch_is_retpoline(reloc->sym)) {
+		} else if (reloc->sym->retpoline_thunk) {
 			/*
 			 * Retpoline calls are really dynamic calls in
 			 * disguise, so convert them accordingly.
@@ -1907,17 +1907,22 @@ static int read_intra_function_calls(str
 	return 0;
 }
 
-static int read_static_call_tramps(struct objtool_file *file)
+static int classify_symbols(struct objtool_file *file)
 {
 	struct section *sec;
 	struct symbol *func;
 
 	for_each_sec(file, sec) {
 		list_for_each_entry(func, &sec->symbol_list, list) {
-			if (func->bind == STB_GLOBAL &&
-			    !strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR,
+			if (func->bind != STB_GLOBAL)
+				continue;
+
+			if (!strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR,
 				     strlen(STATIC_CALL_TRAMP_PREFIX_STR)))
 				func->static_call_tramp = true;
+
+			if (arch_is_retpoline(func))
+				func->retpoline_thunk = true;
 		}
 	}
 
@@ -1983,7 +1988,7 @@ static int decode_sections(struct objtoo
 	/*
 	 * Must be before add_{jump_call}_destination.
 	 */
-	ret = read_static_call_tramps(file);
+	ret = classify_symbols(file);
 	if (ret)
 		return ret;
 
tools/objtool/include/objtool/elf.h +1 -0
@@ -56,6 +56,7 @@ struct symbol {
 	struct symbol *pfunc, *cfunc, *alias;
 	bool uaccess_safe;
 	bool static_call_tramp;
+	bool retpoline_thunk;
 	struct list_head pv_target;
 };
 
#2 lore raw

Re: [PATCH v2 01/14] objtool: Tag retpoline thunk symbols

Quoted 4 lines · Peter Zijlstra

In order to avoid calling arch_is_retpoline() (which does a strcmp) over and over on symbols, tag them once upfront.

XXX do we also want to do __fentry__ ?

Might as well.

Quoted 1 line

XXX do we want an enum instead of a bunch of bools ?

Maybe, or convert the bools to bit fields.

Signature · 1 line
Josh
author maintainer
#3 lore raw

Re: [PATCH v2 01/14] objtool: Tag retpoline thunk symbols

Quoted 11 lines · Josh Poimboeuf
Quoted 4 lines · Peter Zijlstra

In order to avoid calling arch_is_retpoline() (which does a strcmp) over and over on symbols, tag them once upfront.

XXX do we also want to do __fentry__ ?

Might as well.

Quoted 1 line

XXX do we want an enum instead of a bunch of bools ?

Maybe, or convert the bools to bit fields.

Like so then...

---
Subject: objtool: Classify symbols
From: Peter Zijlstra <[email protected]>
Date: Wed Oct 20 10:05:55 CEST 2021

In order to avoid calling str*cmp() on symbol names, over and over, do them all once upfront and store the result.

---
 tools/objtool/check.c               |   34 ++++++++++++++++++++++------------
 tools/objtool/include/objtool/elf.h |    5 ++++-
 2 files changed, 26 insertions(+), 13 deletions(-)
tools/objtool/check.c +22 -12
@@ -1012,8 +1012,7 @@ static void add_call_dest(struct objtool
 	 * so they need a little help, NOP out any KCOV calls from noinstr
 	 * text.
 	 */
-	if (insn->sec->noinstr &&
-	    !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) {
+	if (insn->sec->noinstr && insn->call_dest->kcov) {
 		if (reloc) {
 			reloc->type = R_NONE;
 			elf_write_reloc(file->elf, reloc);
@@ -1027,7 +1026,7 @@ static void add_call_dest(struct objtool
 		insn->type = sibling ? INSN_RETURN : INSN_NOP;
 	}
 
-	if (mcount && !strcmp(insn->call_dest->name, "__fentry__")) {
+	if (mcount && insn->call_dest->fentry) {
 		if (sibling)
 			WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
 
@@ -1077,7 +1076,7 @@ static int add_jump_destinations(struct
 		} else if (reloc->sym->type == STT_SECTION) {
 			dest_sec = reloc->sym->sec;
 			dest_off = arch_dest_reloc_offset(reloc->addend);
-		} else if (arch_is_retpoline(reloc->sym)) {
+		} else if (reloc->sym->retpoline_thunk) {
 			/*
 			 * Retpoline jumps are really dynamic jumps in
 			 * disguise, so convert them accordingly.
@@ -1218,7 +1217,7 @@ static int add_call_destinations(struct
 
 			add_call_dest(file, insn, dest, false);
 
-		} else if (arch_is_retpoline(reloc->sym)) {
+		} else if (reloc->sym->retpoline_thunk) {
 			/*
 			 * Retpoline calls are really dynamic calls in
 			 * disguise, so convert them accordingly.
@@ -1919,17 +1918,28 @@ static int read_intra_function_calls(str
 	return 0;
 }
 
-static int read_static_call_tramps(struct objtool_file *file)
+static int classify_symbols(struct objtool_file *file)
 {
 	struct section *sec;
 	struct symbol *func;
 
 	for_each_sec(file, sec) {
 		list_for_each_entry(func, &sec->symbol_list, list) {
-			if (func->bind == STB_GLOBAL &&
-			    !strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR,
+			if (func->bind != STB_GLOBAL)
+				continue;
+
+			if (!strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR,
 				     strlen(STATIC_CALL_TRAMP_PREFIX_STR)))
 				func->static_call_tramp = true;
+
+			if (arch_is_retpoline(func))
+				func->retpoline_thunk = true;
+
+			if (!strcmp(func->name, "__fentry__"))
+				func->fentry = true;
+
+			if (!strncmp(func->name, "__sanitizer_cov_", 16))
+				func->kcov = true;
 		}
 	}
 
@@ -1995,7 +2005,7 @@ static int decode_sections(struct objtoo
 	/*
 	 * Must be before add_{jump_call}_destination.
 	 */
-	ret = read_static_call_tramps(file);
+	ret = classify_symbols(file);
 	if (ret)
 		return ret;
 
@@ -2053,9 +2063,9 @@ static int decode_sections(struct objtoo
 
 static bool is_fentry_call(struct instruction *insn)
 {
-	if (insn->type == INSN_CALL && insn->call_dest &&
-	    insn->call_dest->type == STT_NOTYPE &&
-	    !strcmp(insn->call_dest->name, "__fentry__"))
+	if (insn->type == INSN_CALL &&
+	    insn->call_dest &&
+	    insn->call_dest->fentry)
 		return true;
 
 	return false;
tools/objtool/include/objtool/elf.h +4 -1
@@ -55,7 +55,10 @@ struct symbol {
 	unsigned int len;
 	struct symbol *pfunc, *cfunc, *alias;
 	bool uaccess_safe;
-	bool static_call_tramp;
+	u8 static_call_tramp : 1;
+	u8 retpoline_thunk   : 1;
+	u8 fentry            : 1;
+	u8 kcov              : 1;
 	struct list_head pv_target;
 };
 
author maintainer
#4 lore raw

PATCH v2 2/14 objtool: Explicitly avoid self modifying code in .altinstr_replacement

In reply to Peter Zijlstra

Assume ALTERNATIVE()s know what they're doing and do not change, or cause to change, instructions in .altinstr_replacement sections.

---
 tools/objtool/check.c |   42 +++++++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 11 deletions(-)
tools/objtool/check.c +31 -11
@@ -993,18 +993,27 @@ static void remove_insn_ops(struct instr
 	}
 }
 
-static void add_call_dest(struct objtool_file *file, struct instruction *insn,
-			  struct symbol *dest, bool sibling)
+static void annotate_call_site(struct objtool_file *file,
+			       struct instruction *insn, bool sibling)
 {
 	struct reloc *reloc = insn_reloc(file, insn);
+	struct symbol *sym = insn->call_dest;
 
-	insn->call_dest = dest;
-	if (!dest)
+	if (!sym)
+		sym = reloc->sym;
+
+	/*
+	 * Alternative replacement code is just template code which is
+	 * sometimes copied to the original instruction, For now, don't
+	 * annotate it. (In the future we might consider annotating the
+	 * original instruction if/when it ever makes sense to do so.).
+	 */
+	if (!strcmp(insn->sec->name, ".altinstr_replacement"))
 		return;
 
-	if (insn->call_dest->static_call_tramp) {
-		list_add_tail(&insn->call_node,
-			      &file->static_call_list);
+	if (sym->static_call_tramp) {
+		list_add_tail(&insn->call_node, &file->static_call_list);
+		return;
 	}
 
 	/*
@@ -1013,7 +1022,7 @@ static void add_call_dest(struct objtool
 	 * text.
 	 */
 	if (insn->sec->noinstr &&
-	    !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) {
+	    !strncmp(sym->name, "__sanitizer_cov_", 16)) {
 		if (reloc) {
 			reloc->type = R_NONE;
 			elf_write_reloc(file->elf, reloc);
@@ -1025,9 +1034,10 @@ static void add_call_dest(struct objtool
 			               : arch_nop_insn(insn->len));
 
 		insn->type = sibling ? INSN_RETURN : INSN_NOP;
+		return;
 	}
 
-	if (mcount && !strcmp(insn->call_dest->name, "__fentry__")) {
+	if (mcount && !strcmp(sym->name, "__fentry__")) {
 		if (sibling)
 			WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
 
@@ -1042,9 +1052,17 @@ static void add_call_dest(struct objtool
 
 		insn->type = INSN_NOP;
 
-		list_add_tail(&insn->mcount_loc_node,
-			      &file->mcount_loc_list);
+		list_add_tail(&insn->mcount_loc_node, &file->mcount_loc_list);
+		return;
 	}
+}
+
+static void add_call_dest(struct objtool_file *file, struct instruction *insn,
+			  struct symbol *dest, bool sibling)
+{
+	insn->call_dest = dest;
+	if (!dest)
+		return;
 
 	/*
 	 * Whatever stack impact regular CALLs have, should be undone
@@ -1054,6 +1072,8 @@ static void add_call_dest(struct objtool
 	 * are converted to JUMP, see read_intra_function_calls().
 	 */
 	remove_insn_ops(insn);
+
+	annotate_call_site(file, insn, sibling);
 }
 
 /*
author maintainer
#5 lore raw

PATCH v2 3/14 objtool: Shrink struct instruction

In reply to Peter Zijlstra

Any one instruction can only ever call a single function, therefore insn->mcount_loc_node is superfluous and can use insn->call_node.

This shrinks struct instruction, which is by far the most numerous structure objtool creates.

---
 tools/objtool/check.c                 |    6 +++---
 tools/objtool/include/objtool/check.h |    1 -
 2 files changed, 3 insertions(+), 4 deletions(-)
tools/objtool/check.c +3 -3
@@ -701,7 +701,7 @@ static int create_mcount_loc_sections(st
 		return 0;
 
 	idx = 0;
-	list_for_each_entry(insn, &file->mcount_loc_list, mcount_loc_node)
+	list_for_each_entry(insn, &file->mcount_loc_list, call_node)
 		idx++;
 
 	sec = elf_create_section(file->elf, "__mcount_loc", 0, sizeof(unsigned long), idx);
@@ -709,7 +709,7 @@ static int create_mcount_loc_sections(st
 		return -1;
 
 	idx = 0;
-	list_for_each_entry(insn, &file->mcount_loc_list, mcount_loc_node) {
+	list_for_each_entry(insn, &file->mcount_loc_list, call_node) {
 
 		loc = (unsigned long *)sec->data->d_buf + idx;
 		memset(loc, 0, sizeof(unsigned long));
@@ -1049,7 +1049,7 @@ static void annotate_call_site(struct ob
 
 		insn->type = INSN_NOP;
 
-		list_add_tail(&insn->mcount_loc_node, &file->mcount_loc_list);
+		list_add_tail(&insn->call_node, &file->mcount_loc_list);
 		return;
 	}
 }
tools/objtool/include/objtool/check.h +0 -1
@@ -40,7 +40,6 @@ struct instruction {
 	struct list_head list;
 	struct hlist_node hash;
 	struct list_head call_node;
-	struct list_head mcount_loc_node;
 	struct section *sec;
 	unsigned long offset;
 	unsigned int len;
author maintainer
#6 lore raw

PATCH v2 4/14 objtool,x86: Replace alternatives with .retpoline_sites

In reply to Peter Zijlstra

Instead of writing complete alternatives, simply provide a list of all the retpoline thunk calls. Then the kernel is free to do with them as it pleases. Simpler code all-round.

---
 arch/x86/kernel/vmlinux.lds.S       |   14 +++
 tools/objtool/arch/x86/decode.c     |  120 --------------------------------
 tools/objtool/check.c               |  132 +++++++++++++++++++++++++-----------
 tools/objtool/elf.c                 |   84 ----------------------
 tools/objtool/include/objtool/elf.h |    1 
 tools/objtool/special.c             |    8 --
 6 files changed, 107 insertions(+), 252 deletions(-)
arch/x86/kernel/vmlinux.lds.S +14 -0
@@ -272,6 +272,20 @@ SECTIONS
 		__parainstructions_end = .;
 	}
 
+#ifdef CONFIG_RETPOLINE
+	/*
+	 * List of instructions that call/jmp/jcc to retpoline thunks
+	 * __x86_indirect_thunk_*(). These instructions can be patched along
+	 * with alternatives, after which the section can be freed.
+	 */
+	. = ALIGN(8);
+	.retpoline_sites : AT(ADDR(.retpoline_sites) - LOAD_OFFSET) {
+		__retpoline_sites = .;
+		*(.retpoline_sites)
+		__retpoline_sites_end = .;
+	}
+#endif
+
 	/*
 	 * struct alt_inst entries. From the header (alternative.h):
 	 * "Alternative instructions for different CPU types or capabilities"
tools/objtool/arch/x86/decode.c +0 -120

126 lines not on this page · raw message

tools/objtool/check.c +93 -39

180 lines not on this page · raw message

tools/objtool/elf.c +0 -84

90 lines not on this page · raw message

tools/objtool/include/objtool/elf.h +0 -1
@@ -142,7 +142,6 @@ int elf_write_insn(struct elf *elf, stru
 		   unsigned long offset, unsigned int len,
 		   const char *insn);
 int elf_write_reloc(struct elf *elf, struct reloc *reloc);
-struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name);
 int elf_write(struct elf *elf);
 void elf_close(struct elf *elf);
 
tools/objtool/special.c +0 -8
@@ -109,14 +109,6 @@ static int get_alt_entry(struct elf *elf
 			return -1;
 		}
 
-		/*
-		 * Skip retpoline .altinstr_replacement... we already rewrite the
-		 * instructions for retpolines anyway, see arch_is_retpoline()
-		 * usage in add_{call,jump}_destinations().
-		 */
-		if (arch_is_retpoline(new_reloc->sym))
-			return 1;
-
 		reloc_to_sec_off(new_reloc, &alt->new_sec, &alt->new_off);
 
 		/* _ASM_EXTABLE_EX hack */
author maintainer
#7 lore raw

PATCH v2 5/14 x86/retpoline: Remove unused replacement symbols

In reply to Peter Zijlstra

Now that objtool no longer creates alternatives, these replacement symbols are no longer needed, remove them.

---
 arch/x86/lib/retpoline.S |   42 ------------------------------------------
 1 file changed, 42 deletions(-)
arch/x86/lib/retpoline.S +0 -42
@@ -41,36 +41,6 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg)
 .endm
 
 /*
- * This generates .altinstr_replacement symbols for use by objtool. They,
- * however, must not actually live in .altinstr_replacement since that will be
- * discarded after init, but module alternatives will also reference these
- * symbols.
- *
- * Their names matches the "__x86_indirect_" prefix to mark them as retpolines.
- */
-.macro ALT_THUNK reg
-
-	.align 1
-
-SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg)
-	ANNOTATE_RETPOLINE_SAFE
-1:	call	*%\reg
-2:	.skip	5-(2b-1b), 0x90
-SYM_FUNC_END(__x86_indirect_alt_call_\reg)
-
-STACK_FRAME_NON_STANDARD(__x86_indirect_alt_call_\reg)
-
-SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg)
-	ANNOTATE_RETPOLINE_SAFE
-1:	jmp	*%\reg
-2:	.skip	5-(2b-1b), 0x90
-SYM_FUNC_END(__x86_indirect_alt_jmp_\reg)
-
-STACK_FRAME_NON_STANDARD(__x86_indirect_alt_jmp_\reg)
-
-.endm
-
-/*
  * Despite being an assembler file we can't just use .irp here
  * because __KSYM_DEPS__ only uses the C preprocessor and would
  * only see one instance of "__x86_indirect_thunk_\reg" rather
@@ -92,15 +62,3 @@ STACK_FRAME_NON_STANDARD(__x86_indirect_
 #undef GEN
 #define GEN(reg) EXPORT_THUNK(reg)
 #include <asm/GEN-for-each-reg.h>
-
-#undef GEN
-#define GEN(reg) ALT_THUNK reg
-#include <asm/GEN-for-each-reg.h>
-
-#undef GEN
-#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_call_ ## reg)
-#include <asm/GEN-for-each-reg.h>
-
-#undef GEN
-#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_jmp_ ## reg)
-#include <asm/GEN-for-each-reg.h>
author maintainer
#8 lore raw

PATCH v2 6/14 x86/asm: Fix register order

In reply to Peter Zijlstra

Ensure the register order is correct; this allows for easy translation between register number and trampoline and vice-versa.

---
 arch/x86/include/asm/GEN-for-each-reg.h |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
arch/x86/include/asm/GEN-for-each-reg.h +9 -4
@@ -1,11 +1,15 @@
+/*
+ * These are in machine order; things rely on that.
+ */
 #ifdef CONFIG_64BIT
 GEN(rax)
-GEN(rbx)
 GEN(rcx)
 GEN(rdx)
+GEN(rbx)
+GEN(rsp)
+GEN(rbp)
 GEN(rsi)
 GEN(rdi)
-GEN(rbp)
 GEN(r8)
 GEN(r9)
 GEN(r10)
@@ -16,10 +20,11 @@ GEN(r14)
 GEN(r15)
 #else
 GEN(eax)
-GEN(ebx)
 GEN(ecx)
 GEN(edx)
+GEN(ebx)
+GEN(esp)
+GEN(ebp)
 GEN(esi)
 GEN(edi)
-GEN(ebp)
 #endif
#9 lore raw

RE: [PATCH v2 06/14] x86/asm: Fix register order

From: Peter Zijlstra

Quoted 4 lines

Sent: 20 October 2021 11:45

Ensure the register order is correct; this allows for easy translation between register number and trampoline and vice-versa.

There is an alternate cpp pattern that can be used to have the same effect.
Possibly less error prone.
Stealing some text from 2 patches you'd have:

asm/GEN-for-each-reg.h would contain:

#define GEN_FOR_EACH_REG(GEN) \
GEN(rax) \
GEN(rcx) \
GEN(rdx) \
GEN(rbx) \
GEN(rsp) \
GEN(rbp) \
GEN(rsi) \
GEN(rdi) \
GEN(r8) \
GEN(r9) \
GEN(r10)

and the user would contain:

#include <asm/GEN-for-each-reg.h>

#define GEN(reg) \
extern asmlinkage void __x86_indirect_thunk_ ## reg (void);
GEN_FOR_EACH_REG(GEN)
#undef GEN

#define GEN(reg) \
extern asmlinkage void __x86_indirect_alt_call_ ## reg (void);
GEN_FOR_EACH_REG(GEN)
#undef GEN

It is probably possible to use ... and _VA_LIST_.
Something like:

#define GEN_FOR_EACH_REG(GEN, ...) \
GEN(rax, _VA_LIST_) \
...

#define GEN_THUNK(reg, thunk) \
	extern asmlinkage void thunk ## reg (void);

GEN_FOR_EACH_REG(GEN_THUNK, __x86_indirect_thunk_) GEN_FOR_EACH_REG(GEN_THUNK, __x86_indirect_alt_call_) #undef GEN_THUNK

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

maintainer
#10 lore raw

Re: [PATCH v2 06/14] x86/asm: Fix register order

In reply to Peter Zijlstra

arch/x86/include/asm/GEN-for-each-reg.h

Quoted 11 lines · Peter Zijlstra

Ensure the register order is correct; this allows for easy translation between register number and trampoline and vice-versa.

---
 arch/x86/include/asm/GEN-for-each-reg.h |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
arch/x86/include/asm/GEN-for-each-reg.h +0 -0
@@ -1,11 +1,15 @@

<--- while you're at it:

WARNING: Missing or malformed SPDX-License-Identifier tag in line 1 #75: FILE: arch/x86/include/asm/GEN-for-each-reg.h:1:
+/*

I.e., put

/* SPDX-License-Identifier: GPL-2.0 */

at the 1st line there.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#11 lore raw

PATCH v2 7/14 x86/asm: Fixup odd GEN-for-each-reg.h usage

In reply to Peter Zijlstra

Currently GEN-for-each-reg.h usage leaves GEN defined, relying on any subsequent usage to start with #undef, which is rude.

---
 arch/x86/include/asm/asm-prototypes.h |    6 +++---
 arch/x86/lib/retpoline.S              |    4 ++--
 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/asm-prototypes.h +3 -3
@@ -19,19 +19,19 @@ extern void cmpxchg8b_emu(void);
 
 #ifdef CONFIG_RETPOLINE
 
-#undef GEN
 #define GEN(reg) \
 	extern asmlinkage void __x86_indirect_thunk_ ## reg (void);
 #include <asm/GEN-for-each-reg.h>
-
 #undef GEN
+
 #define GEN(reg) \
 	extern asmlinkage void __x86_indirect_alt_call_ ## reg (void);
 #include <asm/GEN-for-each-reg.h>
-
 #undef GEN
+
 #define GEN(reg) \
 	extern asmlinkage void __x86_indirect_alt_jmp_ ## reg (void);
 #include <asm/GEN-for-each-reg.h>
+#undef GEN
 
 #endif /* CONFIG_RETPOLINE */
arch/x86/lib/retpoline.S +2 -2
@@ -55,10 +55,10 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg)
 #define __EXPORT_THUNK(sym)	_ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym)
 #define EXPORT_THUNK(reg)	__EXPORT_THUNK(__x86_indirect_thunk_ ## reg)
 
-#undef GEN
 #define GEN(reg) THUNK reg
 #include <asm/GEN-for-each-reg.h>
-
 #undef GEN
+
 #define GEN(reg) EXPORT_THUNK(reg)
 #include <asm/GEN-for-each-reg.h>
+#undef GEN
author maintainer
#12 lore raw

PATCH v2 8/14 x86/retpoline: Create a retpoline thunk array

In reply to Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

---
 arch/x86/lib/retpoline.S |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/lib/retpoline.S +7 -1
@@ -30,7 +30,7 @@
 
 	.align 32
 
-SYM_FUNC_START(__x86_indirect_thunk_\reg)
+SYM_INNER_LABEL(__x86_indirect_thunk_\reg, SYM_L_GLOBAL)
 
 	ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \
 		      __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \
@@ -55,10 +55,16 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg)
 #define __EXPORT_THUNK(sym)	_ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym)
 #define EXPORT_THUNK(reg)	__EXPORT_THUNK(__x86_indirect_thunk_ ## reg)
 
+	.align 32
+SYM_CODE_START(__x86_indirect_thunk_array)
+
 #define GEN(reg) THUNK reg
 #include <asm/GEN-for-each-reg.h>
 #undef GEN
 
+	.align 32
+SYM_CODE_END(__x86_indirect_thunk_array)
+
 #define GEN(reg) EXPORT_THUNK(reg)
 #include <asm/GEN-for-each-reg.h>
 #undef GEN
#13 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

Signature · 1 line
Josh
#14 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

Quoted 7 lines · Josh Poimboeuf
Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

The array property is what makes:

Review

Quoted 2 lines
(quoted diff) +2 -0
+	reg = (target - &__x86_indirect_thunk_rax) /
+	      (&__x86_indirect_thunk_rcx - &__x86_indirect_thunk_rax);

safe in the next path.

~Andrew

#15 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

Quoted 9 lines · Andrew Cooper
Quoted 7 lines · Josh Poimboeuf
Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

Quoted 6 lines

The array property is what makes:

Review

Quoted 2 lines
(quoted diff) +2 -0
+	reg = (target - &__x86_indirect_thunk_rax) /
+	      (&__x86_indirect_thunk_rcx - &__x86_indirect_thunk_rax);

safe in the next path.

The thunks were already 32-byte aligned. I don't see how slapping a few unused symbols around them does anything.

Signature · 1 line
Josh
author maintainer
#16 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

Quoted 12 lines · Josh Poimboeuf
Quoted 11 lines · Andrew Cooper
Quoted 7 lines · Josh Poimboeuf
Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

The array property is what makes:

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	reg = (target - &__x86_indirect_thunk_rax) /
+	      (&__x86_indirect_thunk_rcx - &__x86_indirect_thunk_rax);

safe in the next path.

The thunks were already 32-byte aligned. I don't see how slapping a few unused symbols around them does anything.

Previously there were 16 (or rather 15 without rsp) separate symbols and a toolchain might reasonably expect it could displace them however it liked, with disregard for the relative position.

However, now they're part of a larger symbol. Any change to their relative position would disrupt this larger _array symbol and thus not be sound.

This is I think the same reasoning used for data symbols. On their own there is no guarantee about their relative position wrt to one aonther, but we're still able to do arrays because an array as a whole is a single larger symbol.

#17 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

Quoted 25 lines · Peter Zijlstra
Quoted 12 lines · Josh Poimboeuf
Quoted 11 lines · Andrew Cooper
Quoted 7 lines · Josh Poimboeuf
Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

The array property is what makes:

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	reg = (target - &__x86_indirect_thunk_rax) /
+	      (&__x86_indirect_thunk_rcx - &__x86_indirect_thunk_rax);

safe in the next path.

The thunks were already 32-byte aligned. I don't see how slapping a few unused symbols around them does anything.

Previously there were 16 (or rather 15 without rsp) separate symbols and a toolchain might reasonably expect it could displace them however it liked, with disregard for the relative position.

Quoted 8 lines

However, now they're part of a larger symbol. Any change to their relative position would disrupt this larger _array symbol and thus not be sound.

This is I think the same reasoning used for data symbols. On their own there is no guarantee about their relative position wrt to one aonther, but we're still able to do arrays because an array as a whole is a single larger symbol.

Makes sense, I think (and good fodder for the commit log).

Signature · 1 line
Josh
author maintainer
#18 lore raw

Re: [PATCH v2 08/14] x86/retpoline: Create a retpoline thunk array

In reply to Andrew Cooper
Quoted 9 lines · Andrew Cooper
Quoted 7 lines · Josh Poimboeuf
Quoted 2 lines · Peter Zijlstra

Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout.

How so?

Just wondering what the purpose of the array is. It doesn't seem to be referenced anywhere.

Quoted 6 lines

The array property is what makes:

Review

Quoted 2 lines
(quoted diff) +2 -0
+	reg = (target - &__x86_indirect_thunk_rax) /
+	      (&__x86_indirect_thunk_rcx - &__x86_indirect_thunk_rax);

safe in the next path.

Arguably I should probably write that like:

	(target - &__x86_indirect_thunk_array) / __x86_indirect_thunk_size;

or something, and also generate and expose that latter symbol from .S.

author maintainer
#19 lore raw

PATCH v2 9/14 x86/alternative: Implement .retpoline_sites support

In reply to Peter Zijlstra

Rewrite retpoline thunk call sites to be indirect calls for spectre_v2=off. This ensures spectre_v2=off is as near to a RETPOLINE=n build as possible.

This is the replacement for objtool writing alternative entries to ensure the same and achieves feature-parity with the previous approach.

One noteworthy feature is that it relies on the thunks to be in machine order to compute the register index.

Specifically, this does not yet address the Jcc __x86_indirect_thunk_* calls generated by clang, a future patch will add this.

---
 arch/um/kernel/um_arch.c           |    4 +
 arch/x86/include/asm/alternative.h |    1 
 arch/x86/kernel/alternative.c      |  142 +++++++++++++++++++++++++++++++++++--
 arch/x86/kernel/module.c           |    9 ++
 4 files changed, 151 insertions(+), 5 deletions(-)
arch/um/kernel/um_arch.c +4 -0
@@ -421,6 +421,10 @@ void __init check_bugs(void)
 	os_check_bugs();
 }
 
+void apply_retpolines(s32 *start, s32 *end)
+{
+}
+
 void apply_alternatives(struct alt_instr *start, struct alt_instr *end)
 {
 }
arch/x86/include/asm/alternative.h +1 -0
@@ -75,6 +75,7 @@ extern int alternatives_patched;
 
 extern void alternative_instructions(void);
 extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
+extern void apply_retpolines(s32 *start, s32 *end);
 
 struct module;
 
arch/x86/kernel/alternative.c +138 -4

184 lines not on this page · raw message

arch/x86/kernel/module.c +8 -1
@@ -251,7 +251,8 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    struct module *me)
 {
 	const Elf_Shdr *s, *text = NULL, *alt = NULL, *locks = NULL,
-		*para = NULL, *orc = NULL, *orc_ip = NULL;
+		*para = NULL, *orc = NULL, *orc_ip = NULL,
+		*retpolines = NULL;
 	char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 
 	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
@@ -267,8 +268,14 @@ int module_finalize(const Elf_Ehdr *hdr,
 			orc = s;
 		if (!strcmp(".orc_unwind_ip", secstrings + s->sh_name))
 			orc_ip = s;
+		if (!strcmp(".retpoline_sites", secstrings + s->sh_name))
+			retpolines = s;
 	}
 
+	if (retpolines) {
+		void *rseg = (void *)retpolines->sh_addr;
+		apply_retpolines(rseg, rseg + retpolines->sh_size);
+	}
 	if (alt) {
 		/* patch .altinstructions */
 		void *aseg = (void *)alt->sh_addr;
author maintainer
#20 lore raw

PATCH v2 10/14 x86/alternative: Handle Jcc __x86_indirect_thunk_\reg

In reply to Peter Zijlstra

Handle the rare cases where the compiler (clang) does an indirect conditional tail-call using:

Jcc __x86_indirect_thunk_\reg

For the !RETPOLINE case this can be rewritten to fit the original (6 byte) instruction like:

Jncc.d8 1f
JMP *%\reg
NOP
1:

---
 arch/x86/kernel/alternative.c |   38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)
arch/x86/kernel/alternative.c +34 -4
@@ -393,7 +393,8 @@ static int emit_indirect(int op, int reg
 static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 {
 	void (*target)(void);
-	int reg, i = 0;
+	int reg, ret, i = 0;
+	u8 op, cc;
 
 	target = addr + insn->length + insn->immediate.value;
 	reg = (target - &__x86_indirect_thunk_rax) /
@@ -408,9 +409,34 @@ static int patch_retpoline(void *addr, s
 	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE))
 		return -1;
 
-	i = emit_indirect(insn->opcode.bytes[0], reg, bytes);
-	if (i < 0)
-		return i;
+	op = insn->opcode.bytes[0];
+
+	/*
+	 * Convert:
+	 *
+	 *   Jcc.d32 __x86_indirect_thunk_\reg
+	 *
+	 * into:
+	 *
+	 *   Jncc.d8 1f
+	 *   jmp *%\reg
+	 *   nop
+	 * 1:
+	 */
+	if (op == 0x0f && (insn->opcode.bytes[1] & 0xf0) == 0x80) {
+		cc = insn->opcode.bytes[1] & 0xf;
+		cc ^= 1; /* invert condition */
+
+		bytes[i++] = 0x70 + cc; /* Jcc.d8 */
+		bytes[i++] = insn->length - 2;
+
+		op = JMP32_INSN_OPCODE;
+	}
+
+	ret = emit_indirect(op, reg, bytes + i);
+	if (ret < 0)
+		return ret;
+	i += ret;
 
 	for (; i < insn->length;)
 		bytes[i++] = BYTES_NOP1;
@@ -444,6 +470,10 @@ void __init_or_module noinline apply_ret
 		case JMP32_INSN_OPCODE:
 			break;
 
+		case 0x0f: /* escape */
+			if (op2 >= 0x80 && op2 <= 0x8f)
+				break;
+			fallthrough;
 		default:
 			WARN_ON_ONCE(1);
 			continue;
author maintainer
#21 lore raw

PATCH v2 11/14 x86/alternative: Try inline spectre_v2=retpoline,amd

In reply to Peter Zijlstra

Try and replace retpoline thunk calls with:

lfence
call *%\reg

for spectre_v2=retpoline,amd.

Specifically, the sequence above is 5 bytes for the low 8 registers, but 6 bytes for the high 8 registers. This means that unless the compilers prefix stuff the call with higher registers this replacement will fail.

Luckily GCC strongly favours RAX for the indirect calls and most (95%+ for defconfig-x86_64) will be converted. OTOH clang strongly favours R11 and almost nothing gets converted.

---
 arch/x86/kernel/alternative.c |   20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
arch/x86/kernel/alternative.c +17 -3
@@ -389,12 +389,13 @@ static int emit_indirect(int op, int reg
  *
  *   CALL *%\reg
  *
+ * It also tries to inline spectre_v2=retpoline,amd when size permits.
  */
 static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 {
+	u8 cc, op = insn->opcode.bytes[0];
 	void (*target)(void);
 	int reg, ret, i = 0;
-	u8 op, cc;
 
 	target = addr + insn->length + insn->immediate.value;
 	reg = (target - &__x86_indirect_thunk_rax) /
@@ -406,11 +407,23 @@ static int patch_retpoline(void *addr, s
 	/* If anyone ever does: CALL/JMP *%rsp, we're in deep trouble. */
 	BUG_ON(reg == 4);
 
+	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_AMD)) {
+		/*
+		 * Can't do nothing about the Jcc case here.
+		 */
+		if (op != JMP32_INSN_OPCODE && op != CALL_INSN_OPCODE)
+			return -1;
+
+		bytes[i++] = 0x0f;
+		bytes[i++] = 0xae;
+		bytes[i++] = 0xe8; /* lfence */
+
+		goto indirect;
+	}
+
 	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE))
 		return -1;
 
-	op = insn->opcode.bytes[0];
-
 	/*
 	 * Convert:
 	 *
@@ -433,6 +446,7 @@ static int patch_retpoline(void *addr, s
 		op = JMP32_INSN_OPCODE;
 	}
 
+indirect:
 	ret = emit_indirect(op, reg, bytes + i);
 	if (ret < 0)
 		return ret;
author maintainer
#22 lore raw

PATCH v2 12/14 x86/alternative: Add debug prints to apply_retpolines()

In reply to Peter Zijlstra

Make sure we can see the text changes when booting with 'debug-alternative'.

Example output:

[ ] SMP alternatives: retpoline at: __traceiter_initcall_level+0x1f/0x30 (ffffffff8100066f) len: 5 to: __x86_indirect_thunk_rax+0x0/0x20 [ ] SMP alternatives: ffffffff82603e58: [2:5) optimized NOPs: ff d0 0f 1f 00
[ ] SMP alternatives: ffffffff8100066f: orig: e8 cc 30 00 01
[ ] SMP alternatives: ffffffff8100066f: repl: ff d0 0f 1f 00

---
 arch/x86/kernel/alternative.c |    6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kernel/alternative.c +6 -0
@@ -487,9 +487,15 @@ void __init_or_module noinline apply_ret
 			continue;
 		}
 
+		DPRINTK("retpoline at: %pS (%px) len: %d to: %pS",
+			addr, addr, insn.length,
+			addr + insn.length + insn.immediate.value);
+
 		len = patch_retpoline(addr, &insn, bytes);
 		if (len == insn.length) {
 			optimize_nops(bytes, len);
+			DUMP_BYTES(((u8*)addr),  len, "%px: orig: ", addr);
+			DUMP_BYTES(((u8*)bytes), len, "%px: repl: ", addr);
 			text_poke_early(addr, bytes, len);
 		}
 	}
author maintainer
#23 lore raw

PATCH v2 13/14 x86,bugs: Unconditionally allow spectre_v2=retpoline,amd

In reply to Peter Zijlstra

Currently Linux prevents usage of retpoline,amd on !AMD hardware, this is unfriendly and gets in the way of testing. Remove this restriction.

---
 arch/x86/kernel/cpu/bugs.c |    7 -------
 1 file changed, 7 deletions(-)
arch/x86/kernel/cpu/bugs.c +0 -7
@@ -882,13 +882,6 @@ static enum spectre_v2_mitigation_cmd __
 		return SPECTRE_V2_CMD_AUTO;
 	}
 
-	if (cmd == SPECTRE_V2_CMD_RETPOLINE_AMD &&
-	    boot_cpu_data.x86_vendor != X86_VENDOR_HYGON &&
-	    boot_cpu_data.x86_vendor != X86_VENDOR_AMD) {
-		pr_err("retpoline,amd selected but CPU is not AMD. Switching to AUTO select\n");
-		return SPECTRE_V2_CMD_AUTO;
-	}
-
 	spec_v2_print_cond(mitigation_options[i].option,
 			   mitigation_options[i].secure);
 	return cmd;
author maintainer
#24 lore raw

PATCH v2 14/14 bpf,x86: Respect X86_FEATURE_RETPOLINE*

In reply to Peter Zijlstra

Current BPF codegen doesn't respect X86_FEATURE_RETPOLINE* flags and unconditionally emits a thunk call, this is sub-optimal and doesn't match the regular, compiler generated, code.

---
 arch/x86/include/asm/nospec-branch.h |   59 -----------------------------
 arch/x86/net/bpf_jit_comp.c          |   71 ++++++++++++++++++++---------------
 arch/x86/net/bpf_jit_comp32.c        |   22 ++++++++--
 3 files changed, 59 insertions(+), 93 deletions(-)
arch/x86/include/asm/nospec-branch.h +0 -59
@@ -303,63 +303,4 @@ static inline void mds_idle_clear_cpu_bu
 
 #endif /* __ASSEMBLY__ */
 
-/*
- * Below is used in the eBPF JIT compiler and emits the byte sequence
- * for the following assembly:
- *
- * With retpolines configured:
- *
- *    callq do_rop
- *  spec_trap:
- *    pause
- *    lfence
- *    jmp spec_trap
- *  do_rop:
- *    mov %rcx,(%rsp) for x86_64
- *    mov %edx,(%esp) for x86_32
- *    retq
- *
- * Without retpolines configured:
- *
- *    jmp *%rcx for x86_64
- *    jmp *%edx for x86_32
- */
-#ifdef CONFIG_RETPOLINE
-# ifdef CONFIG_X86_64
-#  define RETPOLINE_RCX_BPF_JIT_SIZE	17
-#  define RETPOLINE_RCX_BPF_JIT()				\
-do {								\
-	EMIT1_off32(0xE8, 7);	 /* callq do_rop */		\
-	/* spec_trap: */					\
-	EMIT2(0xF3, 0x90);       /* pause */			\
-	EMIT3(0x0F, 0xAE, 0xE8); /* lfence */			\
-	EMIT2(0xEB, 0xF9);       /* jmp spec_trap */		\
-	/* do_rop: */						\
-	EMIT4(0x48, 0x89, 0x0C, 0x24); /* mov %rcx,(%rsp) */	\
-	EMIT1(0xC3);             /* retq */			\
-} while (0)
-# else /* !CONFIG_X86_64 */
-#  define RETPOLINE_EDX_BPF_JIT()				\
-do {								\
-	EMIT1_off32(0xE8, 7);	 /* call do_rop */		\
-	/* spec_trap: */					\
-	EMIT2(0xF3, 0x90);       /* pause */			\
-	EMIT3(0x0F, 0xAE, 0xE8); /* lfence */			\
-	EMIT2(0xEB, 0xF9);       /* jmp spec_trap */		\
-	/* do_rop: */						\
-	EMIT3(0x89, 0x14, 0x24); /* mov %edx,(%esp) */		\
-	EMIT1(0xC3);             /* ret */			\
-} while (0)
-# endif
-#else /* !CONFIG_RETPOLINE */
-# ifdef CONFIG_X86_64
-#  define RETPOLINE_RCX_BPF_JIT_SIZE	2
-#  define RETPOLINE_RCX_BPF_JIT()				\
-	EMIT2(0xFF, 0xE1);       /* jmp *%rcx */
-# else /* !CONFIG_X86_64 */
-#  define RETPOLINE_EDX_BPF_JIT()				\
-	EMIT2(0xFF, 0xE2)        /* jmp *%edx */
-# endif
-#endif
-
 #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */
arch/x86/net/bpf_jit_comp.c +41 -30

133 lines not on this page · raw message

arch/x86/net/bpf_jit_comp32.c +18 -4
@@ -15,6 +15,7 @@
 #include <asm/cacheflush.h>
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>
+#include <asm/asm-prototypes.h>
 #include <linux/bpf.h>
 
 /*
@@ -1267,6 +1268,19 @@ static void emit_epilogue(u8 **pprog, u3
 	*pprog = prog;
 }
 
+static void emit_jmp_edx(u8 **pprog, u8 *ip)
+{
+	u8 *prog = *pprog;
+	int cnt = 0;
+
+#ifdef CONFIG_RETPOLINE
+	EMIT1_off32(0xE9, (u8 *)__x86_indirect_thunk_edx - (ip + 5));
+#else
+	EMIT2(0xFF, 0xE2);
+#endif
+	*pprog = prog;
+}
+
 /*
  * Generate the following code:
  * ... bpf_tail_call(void *ctx, struct bpf_array *array, u64 index) ...
@@ -1280,9 +1294,9 @@ static void emit_epilogue(u8 **pprog, u3
  *   goto *(prog->bpf_func + prologue_size);
  * out:
  */
-static void emit_bpf_tail_call(u8 **pprog)
+static void emit_bpf_tail_call(u8 **pprog, u8 *ip)
 {
-	u8 *prog = *pprog;
+	u8 *prog = *pprog, *start = *pprog;
 	int cnt = 0;
 	const u8 *r1 = bpf2ia32[BPF_REG_1];
 	const u8 *r2 = bpf2ia32[BPF_REG_2];
@@ -1362,7 +1376,7 @@ static void emit_bpf_tail_call(u8 **ppro
 	 * eax == ctx (1st arg)
 	 * edx == prog->bpf_func + prologue_size
 	 */
-	RETPOLINE_EDX_BPF_JIT();
+	emit_jmp_edx(&prog, ip + (prog - start));
 
 	if (jmp_label1 == -1)
 		jmp_label1 = cnt;
@@ -2122,7 +2136,7 @@ static int do_jit(struct bpf_prog *bpf_p
 			break;
 		}
 		case BPF_JMP | BPF_TAIL_CALL:
-			emit_bpf_tail_call(&prog);
+			emit_bpf_tail_call(&prog, image + addrs[i - 1]);
 			break;
 
 		/* cond jump */
author maintainer
#25 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 50 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +10 -5
 
+#ifdef CONFIG_RETPOLINE
+#define INDIRECT_SIZE (2 + 3*cpu_feature_enabled(X86_FEATURE_RETPOLINE))
+#else
+#define INDIRECT_SIZE (2)
+#endif
@@ -411,10 +442,10 @@ static int get_pop_bytes(bool *callee_re
  * out:
  */
 static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used,
-					u32 stack_depth)
+					u32 stack_depth, u8 *ip)
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
-	u8 *prog = *pprog;
+	u8 *prog = *pprog, *start = *pprog;
 	int pop_bytes = 0;
 	int off1 = 42;
 	int off2 = 31;
@@ -448,7 +479,7 @@ static void emit_bpf_tail_call_indirect(
 	EMIT2(0x89, 0xD2);                        /* mov edx, edx */
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
-#define OFFSET1 (off1 + RETPOLINE_RCX_BPF_JIT_SIZE) /* Number of bytes to jump */
+#define OFFSET1 (off1 + INDIRECT_SIZE) /* Number of bytes to jump */
 	EMIT2(X86_JBE, OFFSET1);                  /* jbe out */
 
 	/*
@@ -457,7 +488,7 @@ static void emit_bpf_tail_call_indirect(
 	 */
 	EMIT2_off32(0x8B, 0x85, tcc_off);         /* mov eax, dword ptr [rbp - tcc_off] */
 	EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT);     /* cmp eax, MAX_TAIL_CALL_CNT */
-#define OFFSET2 (off2 + RETPOLINE_RCX_BPF_JIT_SIZE)
+#define OFFSET2 (off2 + INDIRECT_SIZE)
 	EMIT2(X86_JA, OFFSET2);                   /* ja out */
 	EMIT3(0x83, 0xC0, 0x01);                  /* add eax, 1 */
 	EMIT2_off32(0x89, 0x85, tcc_off);         /* mov dword ptr [rbp - tcc_off], eax */
@@ -471,7 +502,7 @@ static void emit_bpf_tail_call_indirect(
 	 *	goto out;
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
-#define OFFSET3 (off3 + RETPOLINE_RCX_BPF_JIT_SIZE)
+#define OFFSET3 (off3 + INDIRECT_SIZE)
 	EMIT2(X86_JE, OFFSET3);                   /* je out */
 
 	*pprog = prog;

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 530

Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +1 -1
@@ -493,7 +524,7 @@ static void emit_bpf_tail_call_indirect(
 	 * rdi == ctx (1st arg)
 	 * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET
 	 */
-	RETPOLINE_RCX_BPF_JIT();
+	emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start));
 
 	/* out: */
 	*pprog = prog;

Alexei; could the above not be further improved with something like the below?

Despite several hours trying and Song helping, I can't seem to run anything bpf, that stuff is cursed. So I've no idea if the below actually works, but it seems reasonable.

---

arch/x86/net/bpf_jit_comp.c +13 -31
@@ -398,12 +398,6 @@ static int get_pop_bytes(bool *callee_re
 
 #define EMIT_LFENCE()	EMIT3(0x0F, 0xAE, 0xE8)
 
-#ifdef CONFIG_RETPOLINE
-#define INDIRECT_SIZE (2 + 3*cpu_feature_enabled(X86_FEATURE_RETPOLINE))
-#else
-#define INDIRECT_SIZE (2)
-#endif
-
 static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)
 {
 	u8 *prog = *pprog;
@@ -446,25 +440,8 @@ static void emit_bpf_tail_call_indirect(
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
 	u8 *prog = *pprog, *start = *pprog;
-	int pop_bytes = 0;
-	int off1 = 42;
-	int off2 = 31;
-	int off3 = 9;
-
-	/* count the additional bytes used for popping callee regs from stack
-	 * that need to be taken into account for each of the offsets that
-	 * are used for bailing out of the tail call
-	 */
-	pop_bytes = get_pop_bytes(callee_regs_used);
-	off1 += pop_bytes;
-	off2 += pop_bytes;
-	off3 += pop_bytes;
-
-	if (stack_depth) {
-		off1 += 7;
-		off2 += 7;
-		off3 += 7;
-	}
+	static int out_label = -1;
+	int offset;
 
 	/*
 	 * rdi - pointer to ctx
@@ -479,8 +456,9 @@ static void emit_bpf_tail_call_indirect(
 	EMIT2(0x89, 0xD2);                        /* mov edx, edx */
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
-#define OFFSET1 (off1 + INDIRECT_SIZE) /* Number of bytes to jump */
-	EMIT2(X86_JBE, OFFSET1);                  /* jbe out */
+
+	offset = out_label - (prog - start) + 2;
+	EMIT2(X86_JBE, offset);                  /* jbe out */
 
 	/*
 	 * if (tail_call_cnt > MAX_TAIL_CALL_CNT)
@@ -488,8 +466,9 @@ static void emit_bpf_tail_call_indirect(
 	 */
 	EMIT2_off32(0x8B, 0x85, tcc_off);         /* mov eax, dword ptr [rbp - tcc_off] */
 	EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT);     /* cmp eax, MAX_TAIL_CALL_CNT */
-#define OFFSET2 (off2 + INDIRECT_SIZE)
-	EMIT2(X86_JA, OFFSET2);                   /* ja out */
+
+	offset = out_label - (prog - start) + 2;
+	EMIT2(X86_JA, offset);                   /* ja out */
 	EMIT3(0x83, 0xC0, 0x01);                  /* add eax, 1 */
 	EMIT2_off32(0x89, 0x85, tcc_off);         /* mov dword ptr [rbp - tcc_off], eax */
 
@@ -502,8 +481,9 @@ static void emit_bpf_tail_call_indirect(
 	 *	goto out;
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
-#define OFFSET3 (off3 + INDIRECT_SIZE)
-	EMIT2(X86_JE, OFFSET3);                   /* je out */
+
+	offset = out_label - (prog - start) + 2;
+	EMIT2(X86_JE, offset);                   /* je out */
 
 	*pprog = prog;
 	pop_callee_regs(pprog, callee_regs_used);
@@ -527,6 +507,8 @@ static void emit_bpf_tail_call_indirect(
 	emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start));
 
 	/* out: */
+	out_label = (prog - start);
+
 	*pprog = prog;
 }
 
#26 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 6 lines · Peter Zijlstra

Alexei; could the above not be further improved with something like the below?

Despite several hours trying and Song helping, I can't seem to run anything bpf, that stuff is cursed. So I've no idea if the below actually works, but it seems reasonable.

The below fix gets it to run with test_verififer.

I do like it, it does seem less fiddly and more robust against future changes, though it probably needs a comment for 'out_label' clarifying it only works because this function is always called at least twice for a given bpf_tail_call emission.

---

arch/x86/net/bpf_jit_comp.c +3 -3
index 08f32c9fceaa..c9230c5bbca5 100644
@@ -457,7 +457,7 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used,
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
 
-	offset = out_label - (prog - start) + 2;
+	offset = out_label - (prog - start + 2);
 	EMIT2(X86_JBE, offset);                  /* jbe out */
 
 	/*
@@ -467,7 +467,7 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used,
 	EMIT2_off32(0x8B, 0x85, tcc_off);         /* mov eax, dword ptr [rbp - tcc_off] */
 	EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT);     /* cmp eax, MAX_TAIL_CALL_CNT */
 
-	offset = out_label - (prog - start) + 2;
+	offset = out_label - (prog - start + 2);
 	EMIT2(X86_JA, offset);                   /* ja out */
 	EMIT3(0x83, 0xC0, 0x01);                  /* add eax, 1 */
 	EMIT2_off32(0x89, 0x85, tcc_off);         /* mov dword ptr [rbp - tcc_off], eax */
@@ -482,7 +482,7 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used,
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
 
-	offset = out_label - (prog - start) + 2;
+	offset = out_label - (prog - start + 2);
 	EMIT2(X86_JE, offset);                   /* je out */
 
 	*pprog = prog;
author maintainer
#27 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 9 lines · Josh Poimboeuf
Quoted 6 lines · Peter Zijlstra

Alexei; could the above not be further improved with something like the below?

Despite several hours trying and Song helping, I can't seem to run anything bpf, that stuff is cursed. So I've no idea if the below actually works, but it seems reasonable.

The below fix gets it to run with test_verififer.

Argh, I;m fast running out of brown paper bags at this rate :/

Quoted 5 lines

I do like it, it does seem less fiddly and more robust against future changes, though it probably needs a comment for 'out_label' clarifying it only works because this function is always called at least twice for a given bpf_tail_call emission.

Right.

#28 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 512

Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +1 -1
-	RETPOLINE_RCX_BPF_JIT();
+	emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start));
 
 	/* out: */
 	*pprog = prog;

Alexei; could the above not be further improved with something like the below?

sorry for delay. I was traveling last week
and Daniel is on PTO this week.

Quoted 3 lines

Despite several hours trying and Song helping, I can't seem to run anything bpf, that stuff is cursed. So I've no idea if the below actually works, but it seems reasonable.

It's certainly delicate.

Quoted 14 lines
arch/x86/net/bpf_jit_comp.c +0 -10
@@ -446,25 +440,8 @@ static void emit_bpf_tail_call_indirect(
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
 	u8 *prog = *pprog, *start = *pprog;
-	int pop_bytes = 0;
-	int off1 = 42;
-	int off2 = 31;
-	int off3 = 9;
-
-	/* count the additional bytes used for popping callee regs from stack
-	 * that need to be taken into account for each of the offsets that
-	 * are used for bailing out of the tail call
-	 */
-	pop_bytes = get_pop_bytes(callee_regs_used);

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 443

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +1 -9
-	off1 += pop_bytes;
-	off2 += pop_bytes;
-	off3 += pop_bytes;
-
-	if (stack_depth) {
-		off1 += 7;
-		off2 += 7;
-		off3 += 7;
-	}
+	static int out_label = -1;

Interesting idea!
All insn emits trying to do the right thing from the start. Here the logic assumes that there will be at least two passes over image. I think that is correct, but we never had such assumption.
A comment is certainly must have.
The race is possible too. Not sure whether READ_ONCE/WRITE_ONCE are really warranted though. Might be overkill.

Nice that Josh's test_verifier is passing, but it doesn't provide a ton of coverage. test_progs has a lot more.
Once you have a git branch with all the changes I can give it a go. Also you can rely on our BPF CI.
Just cc your patchset to bpf@vger and add [PATCH bpf-next] to a subject. In patchwork there will be "bpf/vmtest-bpf-next" link that builds kernel, selftests and runs everything.
It's pretty much the same as selftests/bpf/vmtest.sh, but with the latest clang nightly and other deps like pahole.

author maintainer
#29 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +0 -14
@@ -446,25 +440,8 @@ static void emit_bpf_tail_call_indirect(
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
 	u8 *prog = *pprog, *start = *pprog;
-	int pop_bytes = 0;
-	int off1 = 42;
-	int off2 = 31;
-	int off3 = 9;
-
-	/* count the additional bytes used for popping callee regs from stack
-	 * that need to be taken into account for each of the offsets that
-	 * are used for bailing out of the tail call
-	 */
-	pop_bytes = get_pop_bytes(callee_regs_used);
-	off1 += pop_bytes;
-	off2 += pop_bytes;
-	off3 += pop_bytes;
-
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 443

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +1 -5
-	if (stack_depth) {
-		off1 += 7;
-		off2 += 7;
-		off3 += 7;
-	}
+	static int out_label = -1;

Interesting idea!

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Quoted 3 lines

All insn emits trying to do the right thing from the start. Here the logic assumes that there will be at least two passes over image. I think that is correct, but we never had such assumption.

That's not exactly true; I think image is NULL on every first run, so all insn that depend on it will be wrong to start with. Equally there's a number of insn that seem to depend on addrs[i], that also requires at least two passes.

Quoted 1 line

A comment is certainly must have.

I can certainly add one, although I think we'll disagree on the comment style :-)

Quoted 2 lines

The race is possible too. Not sure whether READ_ONCE/WRITE_ONCE are really warranted though. Might be overkill.

Is there concurrency on the jit?

Quoted 1 line

Once you have a git branch with all the changes I can give it a go.

Ok, I'll go polish this thing and stick it in the tree mentioned in the cover letter.

Quoted 4 lines

Also you can rely on our BPF CI.
Just cc your patchset to bpf@vger and add [PATCH bpf-next] to a subject. In patchwork there will be "bpf/vmtest-bpf-next" link that builds kernel, selftests and runs everything.

What's a patchwork and where do I find it?

Quoted 2 lines

It's pretty much the same as selftests/bpf/vmtest.sh, but with the latest clang nightly and other deps like pahole.

nice.

#30 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 31 lines · Peter Zijlstra
Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +0 -14
@@ -446,25 +440,8 @@ static void emit_bpf_tail_call_indirect(
 {
    int tcc_off = -4 - round_up(stack_depth, 8);
    u8 *prog = *pprog, *start = *pprog;
-   int pop_bytes = 0;
-   int off1 = 42;
-   int off2 = 31;
-   int off3 = 9;
-
-   /* count the additional bytes used for popping callee regs from stack
-    * that need to be taken into account for each of the offsets that
-    * are used for bailing out of the tail call
-    */
-   pop_bytes = get_pop_bytes(callee_regs_used);
-   off1 += pop_bytes;
-   off2 += pop_bytes;
-   off3 += pop_bytes;
-
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 443

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +1 -5
-   if (stack_depth) {
-           off1 += 7;
-           off2 += 7;
-           off3 += 7;
-   }
+   static int out_label = -1;

Interesting idea!

Quoted 3 lines

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Interesting. Daniel will recognize that trick then :)

Quoted 8 lines
Quoted 3 lines

All insn emits trying to do the right thing from the start. Here the logic assumes that there will be at least two passes over image. I think that is correct, but we never had such assumption.

That's not exactly true; I think image is NULL on every first run, so all insn that depend on it will be wrong to start with. Equally there's a number of insn that seem to depend on addrs[i], that also requires at least two passes.

Right. The image will be allocated after size converges and addrs[] is inited with 64.
So there is certainly more than one pass.
I was saying that emit* helpers didn't have that assumption. Looks like 32-bit JIT had.

Quoted 4 lines
Quoted 1 line

A comment is certainly must have.

I can certainly add one, although I think we'll disagree on the comment style :-)

I think we're on the same page actually.

Quoted 4 lines
Quoted 2 lines

The race is possible too. Not sure whether READ_ONCE/WRITE_ONCE are really warranted though. Might be overkill.

Is there concurrency on the jit?

The JIT of different progs can happen in parallel.

Quoted 11 lines
Quoted 1 line

Once you have a git branch with all the changes I can give it a go.

Ok, I'll go polish this thing and stick it in the tree mentioned in the cover letter.

Quoted 4 lines

Also you can rely on our BPF CI.
Just cc your patchset to bpf@vger and add [PATCH bpf-next] to a subject. In patchwork there will be "bpf/vmtest-bpf-next" link that builds kernel, selftests and runs everything.

What's a patchwork and where do I find it?

https://patchwork.kernel.org/project/netdevbpf/list/?delegate=121173
Click on any patch, search for 'bpf/vmtest-bpf-next' and follow the 'VM_Test' link.
The summary of the test run is available without logging in into github. To see detailed logs you need to be logged in with your github account. It's a silly limitation they have.
They even have a button 'Sign in to view logs'. Oh well.

Quoted 4 lines
Quoted 2 lines

It's pretty much the same as selftests/bpf/vmtest.sh, but with the latest clang nightly and other deps like pahole.

nice.

One more thing. There is test_bpf.ko.
Just insmod it and it will run a ton of JIT tests that we cannot do from user space.
Please use bpf-next tree though. Few weeks ago Johan Almbladh added a lot more tests to it.

author maintainer
#31 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 9 lines · Alexei Starovoitov
Quoted 3 lines

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Interesting. Daniel will recognize that trick then :)

Quoted 1 line

Is there concurrency on the jit?

The JIT of different progs can happen in parallel.

In that case I don't think the patch is safe. I'll see if I can find a variant that doesn't use static storage.

#32 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Alexei Starovoitov
Quoted 3 lines

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Quoted 8 lines
Quoted 5 lines

Interesting. Daniel will recognize that trick then :)

Quoted 1 line

Is there concurrency on the jit?

The JIT of different progs can happen in parallel.

In that case I don't think the patch is safe. I'll see if I can find a variant that doesn't use static storage.

The variable can only change from one fixed value to another fixed value. Different threads will compute the same value. So I think it's safe as-is. READ_ONCE/WRITE_ONCE won't hurt though.

#33 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 12 lines · Alexei Starovoitov
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Alexei Starovoitov
Quoted 3 lines

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Interesting. Daniel will recognize that trick then :)

Quoted 1 line

Is there concurrency on the jit?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The JIT of different progs can happen in parallel.

In that case I don't think the patch is safe. I'll see if I can find a variant that doesn't use static storage.

The variable can only change from one fixed value to another fixed value. Different threads will compute the same value. So I think it's safe as-is. READ_ONCE/WRITE_ONCE won't hurt though.

But the size of the generated code differs based on the emit_bpf_tail_call_indirect() args: 'callee_regs_used' and 'stack_depth'. So the fixed value can change.

Signature · 1 line
Josh
#34 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 19 lines · Josh Poimboeuf
Quoted 17 lines · Alexei Starovoitov
Quoted 15 lines · Peter Zijlstra
Quoted 9 lines · Alexei Starovoitov
Quoted 3 lines

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Interesting. Daniel will recognize that trick then :)

Quoted 1 line

Is there concurrency on the jit?

The JIT of different progs can happen in parallel.

In that case I don't think the patch is safe. I'll see if I can find a variant that doesn't use static storage.

Quoted 7 lines
Quoted 3 lines

The variable can only change from one fixed value to another fixed value. Different threads will compute the same value. So I think it's safe as-is. READ_ONCE/WRITE_ONCE won't hurt though.

But the size of the generated code differs based on the emit_bpf_tail_call_indirect() args: 'callee_regs_used' and 'stack_depth'. So the fixed value can change.

Ahh. Right. It's potentially a different offset for every prog. Let's put it into struct jit_context then.

author maintainer
#35 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 2 lines · Alexei Starovoitov

Ahh. Right. It's potentially a different offset for every prog. Let's put it into struct jit_context then.

Something like this...

---

arch/x86/net/bpf_jit_comp.c +42 -81
@@ -225,6 +225,14 @@ static void jit_fill_hole(void *area, un
 
 struct jit_context {
 	int cleanup_addr; /* Epilogue code offset */
+
+	/*
+	 * Program specific offsets of labels in the code; these rely on the
+	 * JIT doing at least 2 passes, recording the position on the first
+	 * pass, only to generate the correct offset on the second pass.
+	 */
+	int tail_call_direct_label;
+	int tail_call_indirect_label;
 };
 
 /* Maximum number of bytes emitted while JITing one eBPF insn */
@@ -380,22 +388,6 @@ int bpf_arch_text_poke(void *ip, enum bp
 	return __bpf_arch_text_poke(ip, t, old_addr, new_addr, true);
 }
 
-static int get_pop_bytes(bool *callee_regs_used)
-{
-	int bytes = 0;
-
-	if (callee_regs_used[3])
-		bytes += 2;
-	if (callee_regs_used[2])
-		bytes += 2;
-	if (callee_regs_used[1])
-		bytes += 2;
-	if (callee_regs_used[0])
-		bytes += 1;
-
-	return bytes;
-}
-
 /*
  * Generate the following code:
  *
@@ -411,29 +403,12 @@ static int get_pop_bytes(bool *callee_re
  * out:
  */
 static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used,
-					u32 stack_depth)
+					u32 stack_depth, u8 *ip,
+					struct jit_context *ctx)
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
-	u8 *prog = *pprog;
-	int pop_bytes = 0;
-	int off1 = 42;
-	int off2 = 31;
-	int off3 = 9;
-
-	/* count the additional bytes used for popping callee regs from stack
-	 * that need to be taken into account for each of the offsets that
-	 * are used for bailing out of the tail call
-	 */
-	pop_bytes = get_pop_bytes(callee_regs_used);
-	off1 += pop_bytes;
-	off2 += pop_bytes;
-	off3 += pop_bytes;
-
-	if (stack_depth) {
-		off1 += 7;
-		off2 += 7;
-		off3 += 7;
-	}
+	u8 *prog = *pprog, *start = *pprog;
+	int offset;
 
 	/*
 	 * rdi - pointer to ctx
@@ -448,8 +423,9 @@ static void emit_bpf_tail_call_indirect(
 	EMIT2(0x89, 0xD2);                        /* mov edx, edx */
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
-#define OFFSET1 (off1 + RETPOLINE_RCX_BPF_JIT_SIZE) /* Number of bytes to jump */
-	EMIT2(X86_JBE, OFFSET1);                  /* jbe out */
+
+	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	EMIT2(X86_JBE, offset);                   /* jbe out */
 
 	/*
 	 * if (tail_call_cnt > MAX_TAIL_CALL_CNT)
@@ -457,8 +433,9 @@ static void emit_bpf_tail_call_indirect(
 	 */
 	EMIT2_off32(0x8B, 0x85, tcc_off);         /* mov eax, dword ptr [rbp - tcc_off] */
 	EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT);     /* cmp eax, MAX_TAIL_CALL_CNT */
-#define OFFSET2 (off2 + RETPOLINE_RCX_BPF_JIT_SIZE)
-	EMIT2(X86_JA, OFFSET2);                   /* ja out */
+
+	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	EMIT2(X86_JA, offset);                    /* ja out */
 	EMIT3(0x83, 0xC0, 0x01);                  /* add eax, 1 */
 	EMIT2_off32(0x89, 0x85, tcc_off);         /* mov dword ptr [rbp - tcc_off], eax */
 
@@ -471,12 +448,11 @@ static void emit_bpf_tail_call_indirect(
 	 *	goto out;
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
-#define OFFSET3 (off3 + RETPOLINE_RCX_BPF_JIT_SIZE)
-	EMIT2(X86_JE, OFFSET3);                   /* je out */
 
-	*pprog = prog;
-	pop_callee_regs(pprog, callee_regs_used);
-	prog = *pprog;
+	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	EMIT2(X86_JE, offset);                    /* je out */
+
+	pop_callee_regs(&prog, callee_regs_used);
 
 	EMIT1(0x58);                              /* pop rax */
 	if (stack_depth)
@@ -496,38 +472,18 @@ static void emit_bpf_tail_call_indirect(
 	RETPOLINE_RCX_BPF_JIT();
 
 	/* out: */
+	ctx->tail_call_indirect_label = prog - start;
 	*pprog = prog;
 }
 
 static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke,
-				      u8 **pprog, int addr, u8 *image,
-				      bool *callee_regs_used, u32 stack_depth)
+				      u8 **pprog, u8 *ip,
+				      bool *callee_regs_used, u32 stack_depth,
+				      struct jit_context *ctx)
 {
 	int tcc_off = -4 - round_up(stack_depth, 8);
-	u8 *prog = *pprog;
-	int pop_bytes = 0;
-	int off1 = 20;
-	int poke_off;
-
-	/* count the additional bytes used for popping callee regs to stack
-	 * that need to be taken into account for jump offset that is used for
-	 * bailing out from of the tail call when limit is reached
-	 */
-	pop_bytes = get_pop_bytes(callee_regs_used);
-	off1 += pop_bytes;
-
-	/*
-	 * total bytes for:
-	 * - nop5/ jmpq $off
-	 * - pop callee regs
-	 * - sub rsp, $val if depth > 0
-	 * - pop rax
-	 */
-	poke_off = X86_PATCH_SIZE + pop_bytes + 1;
-	if (stack_depth) {
-		poke_off += 7;
-		off1 += 7;
-	}
+	u8 *prog = *pprog, *start = *pprog;
+	int offset;
 
 	/*
 	 * if (tail_call_cnt > MAX_TAIL_CALL_CNT)
@@ -535,28 +491,30 @@ static void emit_bpf_tail_call_direct(st
 	 */
 	EMIT2_off32(0x8B, 0x85, tcc_off);             /* mov eax, dword ptr [rbp - tcc_off] */
 	EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT);         /* cmp eax, MAX_TAIL_CALL_CNT */
-	EMIT2(X86_JA, off1);                          /* ja out */
+
+	offset = ctx->tail_call_direct_label - (prog + 2 - start);
+	EMIT2(X86_JA, offset);                        /* ja out */
 	EMIT3(0x83, 0xC0, 0x01);                      /* add eax, 1 */
 	EMIT2_off32(0x89, 0x85, tcc_off);             /* mov dword ptr [rbp - tcc_off], eax */
 
-	poke->tailcall_bypass = image + (addr - poke_off - X86_PATCH_SIZE);
+	poke->tailcall_bypass = ip + (prog - start);
 	poke->adj_off = X86_TAIL_CALL_OFFSET;
-	poke->tailcall_target = image + (addr - X86_PATCH_SIZE);
+	poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;
 	poke->bypass_addr = (u8 *)poke->tailcall_target + X86_PATCH_SIZE;
 
 	emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE,
 		  poke->tailcall_bypass);
 
-	*pprog = prog;
-	pop_callee_regs(pprog, callee_regs_used);
-	prog = *pprog;
+	pop_callee_regs(&prog, callee_regs_used);
 	EMIT1(0x58);                                  /* pop rax */
 	if (stack_depth)
 		EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));
 
 	memcpy(prog, x86_nops[5], X86_PATCH_SIZE);
 	prog += X86_PATCH_SIZE;
+
 	/* out: */
+	ctx->tail_call_direct_label = prog - start;
 
 	*pprog = prog;
 }
@@ -1405,13 +1363,16 @@ st:			if (is_imm8(insn->off))
 		case BPF_JMP | BPF_TAIL_CALL:
 			if (imm32)
 				emit_bpf_tail_call_direct(&bpf_prog->aux->poke_tab[imm32 - 1],
-							  &prog, addrs[i], image,
+							  &prog, image + addrs[i - 1],
 							  callee_regs_used,
-							  bpf_prog->aux->stack_depth);
+							  bpf_prog->aux->stack_depth,
+							  ctx);
 			else
 				emit_bpf_tail_call_indirect(&prog,
 							    callee_regs_used,
-							    bpf_prog->aux->stack_depth);
+							    bpf_prog->aux->stack_depth,
+							    image + addrs[i - 1],
+							    ctx);
 			break;
 
 			/* cond jump */
#36 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov

Ahh. Right. It's potentially a different offset for every prog. Let's put it into struct jit_context then.

Something like this...

Yep. Looks nice and clean to me.

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_direct(st · line 502

Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +2 -2
-       poke->tailcall_bypass = image + (addr - poke_off - X86_PATCH_SIZE);
+       poke->tailcall_bypass = ip + (prog - start);
        poke->adj_off = X86_TAIL_CALL_OFFSET;
-       poke->tailcall_target = image + (addr - X86_PATCH_SIZE);
+       poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;

This part looks correct too, but this is Daniel's magic. He'll probably take a look next week when he comes back from PTO. I don't recall which test exercises this tailcall poking logic. It's only used with dynamic updates to prog_array. insmod test_bpf.ko and test_verifier won't go down this path.

#37 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 17 lines · Alexei Starovoitov
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov

Ahh. Right. It's potentially a different offset for every prog. Let's put it into struct jit_context then.

Something like this...

Yep. Looks nice and clean to me.

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_direct(st · line 502

Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +2 -2
-       poke->tailcall_bypass = image + (addr - poke_off - X86_PATCH_SIZE);
+       poke->tailcall_bypass = ip + (prog - start);
        poke->adj_off = X86_TAIL_CALL_OFFSET;
-       poke->tailcall_target = image + (addr - X86_PATCH_SIZE);
+       poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;
Quoted 5 lines

This part looks correct too, but this is Daniel's magic. He'll probably take a look next week when he comes back from PTO. I don't recall which test exercises this tailcall poking logic. It's only used with dynamic updates to prog_array. insmod test_bpf.ko and test_verifier won't go down this path.

Please run ./test_progs -t tailcalls from tools/testing/selftests/bpf and make sure that all of the tests are passing in there, especially the tailcall_bpf2bpf* subset.

Thanks!

author maintainer
#38 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 24 lines · Maciej Fijalkowski
Quoted 17 lines · Alexei Starovoitov
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Alexei Starovoitov

Ahh. Right. It's potentially a different offset for every prog. Let's put it into struct jit_context then.

Something like this...

Yep. Looks nice and clean to me.

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_direct(st · line 502

Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +2 -2
-       poke->tailcall_bypass = image + (addr - poke_off - X86_PATCH_SIZE);
+       poke->tailcall_bypass = ip + (prog - start);
        poke->adj_off = X86_TAIL_CALL_OFFSET;
-       poke->tailcall_target = image + (addr - X86_PATCH_SIZE);
+       poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;
Quoted 5 lines

This part looks correct too, but this is Daniel's magic. He'll probably take a look next week when he comes back from PTO. I don't recall which test exercises this tailcall poking logic. It's only used with dynamic updates to prog_array. insmod test_bpf.ko and test_verifier won't go down this path.

Quoted 3 lines

Please run ./test_progs -t tailcalls from tools/testing/selftests/bpf and make sure that all of the tests are passing in there, especially the tailcall_bpf2bpf* subset.

Yeah, so nothing from that selftests crud wants to work for me; also I *really* dislike how vmtest.sh as found there tramples all over my source dir without asking.

Note that even when eventually supplied with O=builddir (confusingly in front of it), it doesn't want to work and bails with lots of -ENOSPC warnings (I double checked, my disks are nowhere near full). (and this is after installing some horrendous python rst crap because clearly running a test needs to build documentation :/)

I've spend hours on that, I'm not sinking more time into it. If you want me to run that crap, fix it first.

#39 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

In reply to Peter Zijlstra
Quoted 59 lines · Peter Zijlstra
Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +0 -14
@@ -446,25 +440,8 @@ static void emit_bpf_tail_call_indirect(
 {
    int tcc_off = -4 - round_up(stack_depth, 8);
    u8 *prog = *pprog, *start = *pprog;
-   int pop_bytes = 0;
-   int off1 = 42;
-   int off2 = 31;
-   int off3 = 9;
-
-   /* count the additional bytes used for popping callee regs from stack
-    * that need to be taken into account for each of the offsets that
-    * are used for bailing out of the tail call
-    */
-   pop_bytes = get_pop_bytes(callee_regs_used);
-   off1 += pop_bytes;
-   off2 += pop_bytes;
-   off3 += pop_bytes;
-
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_bpf_tail_call_indirect( · line 443

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +1 -5
-   if (stack_depth) {
-           off1 += 7;
-           off2 += 7;
-           off3 += 7;
-   }
+   static int out_label = -1;

Interesting idea!

I nicked it from emit_bpf_tail_call() in the 32bit jit :-) It seemed a lot more robust than the 64bit one and I couldn't figure out why the difference.

Quoted 3 lines

All insn emits trying to do the right thing from the start. Here the logic assumes that there will be at least two passes over image. I think that is correct, but we never had such assumption.

That's not exactly true; I think image is NULL on every first run, so all insn that depend on it will be wrong to start with. Equally there's a number of insn that seem to depend on addrs[i], that also requires at least two passes.

Quoted 1 line

A comment is certainly must have.

I can certainly add one, although I think we'll disagree on the comment style :-)

Quoted 2 lines

The race is possible too. Not sure whether READ_ONCE/WRITE_ONCE are really warranted though. Might be overkill.

Is there concurrency on the jit?

Quoted 1 line

Once you have a git branch with all the changes I can give it a go.

Ok, I'll go polish this thing and stick it in the tree mentioned in the cover letter.

Quoted 7 lines
Quoted 4 lines

Also you can rely on our BPF CI.
Just cc your patchset to bpf@vger and add [PATCH bpf-next] to a subject. In patchwork there will be "bpf/vmtest-bpf-next" link that builds kernel, selftests and runs everything.

What's a patchwork and where do I find it?

Patchwork[0] tracks the status of patches from submission through to merge (and beyond?).

[0]: https://patchwork.kernel.org/

Quoted 4 lines
Quoted 2 lines

It's pretty much the same as selftests/bpf/vmtest.sh, but with the latest clang nightly and other deps like pahole.

nice.

author maintainer
#40 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 5 lines · Zvi Effron
Quoted 2 lines

What's a patchwork and where do I find it?

Patchwork[0] tracks the status of patches from submission through to merge (and beyond?).

Yeah, I sorta know that :-) Even though I loathe the things because web-browser, but the second part of the question was genuine, there's a *lot* of patchwork instances around, not everyone uses the new k.org based one.

#41 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Zvi Effron
Quoted 2 lines

What's a patchwork and where do I find it?

Quoted 8 lines
Quoted 2 lines

Patchwork[0] tracks the status of patches from submission through to merge (and beyond?).

Yeah, I sorta know that :-) Even though I loathe the things because web-browser, but the second part of the question was genuine, there's a *lot* of patchwork instances around, not everyone uses the new k.org based one.

BPF and NetDev share one: https://patchwork.kernel.org/project/netdevbpf/list/

--Zvi

#42 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

In reply to Peter Zijlstra

arch/x86/net/bpf_jit_comp.c static int get_pop_bytes(bool *callee_re · line 423

Quoted 7 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +7 -0
+
+	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_AMD)) {
+		EMIT_LFENCE();
+		EMIT2(0xFF, 0xE0 + reg);
+	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
+		emit_jump(&prog, reg_thunk[reg], ip);
+	} else

One more question.
What's a deal with AMD? I thought the retpoline is effective on it as well. lfence is an optimization or retpoline turned out to be not enough in some cases?

#43 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 13 lines · Alexei Starovoitov

arch/x86/net/bpf_jit_comp.c static int get_pop_bytes(bool *callee_re · line 423

Quoted 7 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +7 -0
+
+	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_AMD)) {
+		EMIT_LFENCE();
+		EMIT2(0xFF, 0xE0 + reg);
+	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
+		emit_jump(&prog, reg_thunk[reg], ip);
+	} else

One more question.
What's a deal with AMD? I thought the retpoline is effective on it as well. lfence is an optimization or retpoline turned out to be not enough in some cases?

Yes, it's basically an optimization. AMD recommends it presumably because it's quite a bit faster than a retpoline.

According to AMD it shrinks the speculative execution window enough so that Spectre v2 isn't a threat.

Signature · 1 line
Josh
author maintainer
#44 lore raw

Re: [PATCH v2 14/14] bpf,x86: Respect X86_FEATURE_RETPOLINE*

Quoted 15 lines · Josh Poimboeuf
Quoted 13 lines · Alexei Starovoitov

arch/x86/net/bpf_jit_comp.c static int get_pop_bytes(bool *callee_re · line 423

Quoted 7 lines · Peter Zijlstra
arch/x86/net/bpf_jit_comp.c +7 -0
+
+	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_AMD)) {
+		EMIT_LFENCE();
+		EMIT2(0xFF, 0xE0 + reg);
+	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
+		emit_jump(&prog, reg_thunk[reg], ip);
+	} else

One more question.
What's a deal with AMD? I thought the retpoline is effective on it as well. lfence is an optimization or retpoline turned out to be not enough in some cases?

Quoted 5 lines

Yes, it's basically an optimization. AMD recommends it presumably because it's quite a bit faster than a retpoline.

According to AMD it shrinks the speculative execution window enough so that Spectre v2 isn't a threat.

Right, also note that we've been using alternatives to patch the thunk to lfence;jmp for AMD pretty much forever.

Inlining it is better tho; just a shame clang seems to insist on r11, which means we cannot fit it in the thunk call site for them :/