x86/bpf: unwinder fixes
Subsystems (9)
v2 cover x86/bpf: unwinder fixes
v2:
- Simplified the frame pointer fixes - instead of using R12, just continue to use RBP for BPF_REG_FP, but use nested frames so the unwinder can understand (suggested by David Laight).
- Dropped the AT&T syntax patches for now. I'm about to disappear for a week and I don't have time to argue about whether code readability is a good thing.
- I can do the 32-bit version of the fix when I get back. It should be easy enough.
v1 is here:
https://lkml.kernel.org/r/[email protected]
Josh Poimboeuf (4):
objtool: Fix ORC unwinding in non-JIT BPF generated code x86/bpf: Move epilogue generation to a dedicated function x86/bpf: Fix 64-bit JIT frame pointer usage x86/unwind/orc: Fall back to using frame pointers for generated code
Song Liu (1):
perf/x86: Always store regs->ip in perf_callchain_kernel()
arch/x86/events/core.c | 10 +-- arch/x86/kernel/unwind_orc.c | 26 ++++++-- arch/x86/net/bpf_jit_comp.c | 115 +++++++++++++++++++---------------- kernel/bpf/core.c | 5 +- tools/objtool/check.c | 16 ++++- 5 files changed, 107 insertions(+), 65 deletions(-)
Signature · 1 line
2.20.1
PATCH v2 1/5 perf/x86: Always store regs->ip in perf_callchain_kernel()
From: Song Liu <[email protected]>
The stacktrace_map_raw_tp BPF selftest is failing because the RIP saved by perf_arch_fetch_caller_regs() isn't getting saved by perf_callchain_kernel().
This was broken by the following commit:
d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")
With that change, when starting with non-HW regs, the unwinder starts with the current stack frame and unwinds until it passes up the frame which called perf_arch_fetch_caller_regs(). So regs->ip needs to be saved deliberately.
--- arch/x86/events/core.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/events/core.c +5 -5
@@ -2328,13 +2328,13 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *rereturn ; } -if (perf_hw_regs (regs)) { -if (perf_callchain_store (entry, regs->ip)) -return ; +if (perf_callchain_store (entry, regs->ip)) +return ; + +if (perf_hw_regs (regs))unwind_start (&state, current, regs,NULL ); - }else { +else unwind_start (&state, current,NULL , (void *)regs->sp); - }for (; !unwind_done (&state);unwind_next_frame (&state)) { addr =unwind_get_return_address (&state);
Signature · 1 line
2.20.1
Re: [PATCH v2 1/5] perf/x86: Always store regs->ip in perf_callchain_kernel()
Quoted 10 lines · Josh Poimboeuf
From: Song Liu <[email protected]>
The
stacktrace_map_raw_tpBPF selftest is failing because the RIP saved byperf_arch_fetch_caller_regs()isn't getting saved byperf_callchain_kernel().This was broken by the following commit:
d15d356887e7("perf/x86: Make perf callchains work withoutCONFIG_FRAME_POINTER")
Quoted 8 lines
With that change, when starting with non-HW regs, the unwinder starts with the current stack frame and unwinds until it passes up the frame which called
perf_arch_fetch_caller_regs(). Soregs->ipneeds to be saved deliberately.Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Signed-off-by Josh Poimboeuf <[email protected]>
It's not cool to remove people's SOB.
It's Song's patch. His should be first and your second.
Re: [PATCH v2 1/5] perf/x86: Always store regs->ip in perf_callchain_kernel()
Quoted 16 lines · Alexei Starovoitov
Quoted 15 lines · Josh Poimboeuf
From: Song Liu <[email protected]>
The
stacktrace_map_raw_tpBPF selftest is failing because the RIP saved byperf_arch_fetch_caller_regs()isn't getting saved byperf_callchain_kernel().This was broken by the following commit:
d15d356887e7("perf/x86: Make perf callchains work withoutCONFIG_FRAME_POINTER")With that change, when starting with non-HW regs, the unwinder starts with the current stack frame and unwinds until it passes up the frame which called
perf_arch_fetch_caller_regs(). Soregs->ipneeds to be saved deliberately.
Quoted 6 lines
Quoted 3 lines
Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Signed-off-by Josh Poimboeuf <[email protected]>It's not cool to remove people's SOB.
It's Song's patch. His should be first and your second.
His original patch didn't have an SOB. I preserved the "From" field.
Signature · 1 line
Josh
Re: [PATCH v2 1/5] perf/x86: Always store regs->ip in perf_callchain_kernel()
Quoted 4 lines · On Fri, 14 Jun 2019 16:06:19 -0500 Josh Poimboeuf <[email protected]> wrote:
Quoted 2 lines
It's not cool to remove people's SOB.
It's Song's patch. His should be first and your second.His original patch didn't have an SOB. I preserved the "From" field.
Then it can't be accepted. It needs an SOB from the original author.
Song, Please reply with a Signed-off-by tag.
Thanks!
-- Steve
Re: [PATCH v2 1/5] perf/x86: Always store regs->ip in perf_callchain_kernel()
Quoted 12 lines · Steven Rostedt
Quoted 4 lines · On Fri, 14 Jun 2019 16:06:19 -0500 Josh Poimboeuf <[email protected]> wrote:
Quoted 2 lines
It's not cool to remove people's SOB.
It's Song's patch. His should be first and your second.His original patch didn't have an SOB. I preserved the "From" field.
Quoted 7 lines
Then it can't be accepted. It needs an SOB from the original author.
Song, Please reply with a Signed-off-by tag.
Thanks!
-- Steve
Yes, Sir!
It was my fault to forget it in the first place. Sorry for the confusion.
Song
PATCH v2 2/5 objtool: Fix ORC unwinding in non-JIT BPF generated code
In reply to Josh PoimboeufObjtool currently ignores ___bpf_prog_run() because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.
Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for ___bpf_prog_run().
--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)
kernel/bpf/core.c +2 -3
@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),
@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \
@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \
tools/objtool/check.c +14 -2
@@ -18,6 +18,8 @@#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table." +struct alternative {struct list_head list;struct instruction *insn;
@@ -997,6 +999,7 @@ static struct rela *find_switch_table(struct objtool_file *file,struct instruction *orig_insn = insn;struct section *rodata_sec;unsigned long table_offset; +struct symbol *sym;/* * Backward search using the @first_jump_src links, these help avoid
@@ -1035,9 +1038,18 @@ static struct rela *find_switch_table(struct objtool_file *file,/* * Make sure the .rodata address isn't associated with a -* symbol. gcc jump tables are anonymous data. +* symbol. GCC jump tables are anonymous data. +* +* Also support C jump tables which are in the same format as +* switch jump tables. Each jump table should be a static +* local const array named "jump_table" for objtool to +* recognize it. Note: GCC will add a numbered suffix to the +* ELF symbol name, like "jump_table.12345", which it does for +* all static local variables. */ -if (find_symbol_containing (rodata_sec, table_offset)) + sym =find_symbol_containing (rodata_sec, table_offset); +if (sym &&strncmp (sym->name,JUMP_TABLE_SYM_PREFIX , +strlen (JUMP_TABLE_SYM_PREFIX )))continue ; rodata_rela =find_rela_by_dest (rodata_sec, table_offset);
Signature · 1 line
2.20.1
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 55 lines · Josh Poimboeuf
Objtool currently ignores
___bpf_prog_run()because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "
jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for___bpf_prog_run().Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Reported-by Song Liu <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)kernel/bpf/core.c +2
-3@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \tools/objtool/check.c +0
-0
tools/objtool/check.c line 21
Quoted 4 lines
tools/objtool/check.c +1
-0@@ -18,6 +18,8 @@#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table."
since external tool will be looking at it should it be named "bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 57 lines · Alexei Starovoitov
Quoted 56 lines · Josh Poimboeuf
Objtool currently ignores
___bpf_prog_run()because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "
jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for___bpf_prog_run().Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Reported-by Song Liu <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)kernel/bpf/core.c +2
-3@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \tools/objtool/check.c +0
-0@@ -18,6 +18,8 @@
Quoted 8 lines
tools/objtool/check.c line 21
Quoted 3 lines
tools/objtool/check.c +1
-0#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
Signature · 1 line
Josh
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 68 lines · Josh Poimboeuf
Quoted 57 lines · Alexei Starovoitov
Quoted 56 lines · Josh Poimboeuf
Objtool currently ignores
___bpf_prog_run()because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "
jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for___bpf_prog_run().Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Reported-by Song Liu <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)kernel/bpf/core.c +2
-3@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \tools/objtool/check.c +0
-0@@ -18,6 +18,8 @@Quoted 8 lines
tools/objtool/check.c line 21
Quoted 3 lines
tools/objtool/check.c +1
-0#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
Quoted 2 lines
No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 69 lines · Alexei Starovoitov
Quoted 68 lines · Josh Poimboeuf
Quoted 65 lines · Alexei Starovoitov
Quoted 55 lines · Josh Poimboeuf
Objtool currently ignores
___bpf_prog_run()because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "
jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for___bpf_prog_run().Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Reported-by Song Liu <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)kernel/bpf/core.c +2
-3@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \tools/objtool/check.c +0
-0tools/objtool/check.c line 21
Quoted 4 lines
tools/objtool/check.c +1
-0@@ -18,6 +18,8 @@#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
Quoted 5 lines
Quoted 2 lines
No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
That's the idea.
Signature · 1 line
Josh
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 8 lines · Josh Poimboeuf
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
That's the idea.
Then it needs much wider discussion.
I suggest to rename it to "bpf_interpreter_jump_table."
so it can be resolved now for this specific issue.
While bigger kernel-wide naming convention get resolved. Later we can rename it to whatever the "standard name for jump table" will be.
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 12 lines · Alexei Starovoitov
Quoted 11 lines · Josh Poimboeuf
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
That's the idea.
Then it needs much wider discussion.
Why would it need wider discussion? It only has one user. If you honestly believe that it will be controversial to require future users to call a static jump table "jump_table" then we can have that discussion when it comes up.
Signature · 1 line
Josh
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 17 lines · Josh Poimboeuf
Quoted 15 lines · Alexei Starovoitov
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
That's the idea.
Then it needs much wider discussion.
Why would it need wider discussion? It only has one user. If you honestly believe that it will be controversial to require future users to call a static jump table "
jump_table" then we can have that discussion when it comes up.
It's clearly controversial.
I nacked it already on pointless name change from "jumptable" to "jump_table" and now you're saying that no one will complain about "jump_table" name for all jump tables in the kernel that will ever appear?
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Quoted 27 lines · Alexei Starovoitov
Quoted 17 lines · Josh Poimboeuf
Quoted 15 lines · Alexei Starovoitov
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
That's the idea.
Then it needs much wider discussion.
Why would it need wider discussion? It only has one user. If you honestly believe that it will be controversial to require future users to call a static jump table "
jump_table" then we can have that discussion when it comes up.
Quoted 5 lines
It's clearly controversial.
I nacked it already on pointless name change from "jumptable" to "jump_table" and now you're saying that no one will complain about "jump_table" name for all jump tables in the kernel that will ever appear?
Let me get this straight. You're saying that "jumptable" and "bpf_interpreter_jump_table" are both acceptable.
But NACK to "jump_table".
Ok...
Signature · 1 line
Josh
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
Maybe add more multidimensional arrays?
Sent from my iPhone
Quoted 45 lines
Quoted 30 lines · Josh Poimboeuf
Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Josh Poimboeuf
On Fri, Jun 14, 2019 at 02:22:59PM -0700, Alexei Starovoitov wrote:
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
That's the idea.
Then it needs much wider discussion.
Why would it need wider discussion? It only has one user. If you honestly believe that it will be controversial to require future users to call a static jump table "
jump_table" then we can have that discussion when it comes up.Quoted 5 lines
It's clearly controversial.
I nacked it already on pointless name change from "jumptable" to "jump_table" and now you're saying that no one will complain about "jump_table" name for all jump tables in the kernel that will ever appear?Let me get this straight. You're saying that "jumptable" and "
bpf_interpreter_jump_table" are both acceptable.But NACK to "
jump_table".Ok...
Signature · 1 line
Josh
Re: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
In reply to Josh PoimboeufQuoted 35 lines · Josh Poimboeuf
Quoted 27 lines · Alexei Starovoitov
Quoted 17 lines · Josh Poimboeuf
Quoted 15 lines · Alexei Starovoitov
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
+#define
JUMP_TABLE_SYM_PREFIX"jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?No, the point is that it's a generic feature which can also be used any non-BPF code which might also have a jump table.
and you're proposing to name all such jump tables in the kernel as static foo jump_table[] ?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
That's the idea.
Then it needs much wider discussion.
Why would it need wider discussion? It only has one user. If you honestly believe that it will be controversial to require future users to call a static jump table "
jump_table" then we can have that discussion when it comes up.Quoted 5 lines
It's clearly controversial.
I nacked it already on pointless name change from "jumptable" to "jump_table" and now you're saying that no one will complain about "jump_table" name for all jump tables in the kernel that will ever appear?
Quoted 6 lines
Let me get this straight. You're saying that "jumptable" and "
bpf_interpreter_jump_table" are both acceptable.But NACK to "
jump_table".Ok...
Correct. I think I explained the reasons behind, right?
RE: [PATCH v2 2/5] objtool: Fix ORC unwinding in non-JIT BPF generated code
In reply to Alexei StarovoitovFrom: Alexei Starovoitov
Quoted 59 lines
Sent: 14 June 2019 21:59
Quoted 56 lines · Josh Poimboeuf
Objtool currently ignores
___bpf_prog_run()because it doesn't understand the jump table. This results in the ORC unwinder not being able to unwind through non-JIT BPF code.Luckily, the BPF jump table resembles a GCC switch jump table, which objtool already knows how to read.
Add generic support for reading any static local jump table array named "
jump_table", and rename the BPF variable accordingly, so objtool can generate ORC data for___bpf_prog_run().Fixes d15d356887e7 ("perf/x86: Make perf callchains work without CONFIG_FRAME_POINTER")Reported-by Song Liu <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- kernel/bpf/core.c | 5 ++--- tools/objtool/check.c | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 5 deletions(-)kernel/bpf/core.c +2
-3@@ -1299,7 +1299,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) {#define BPF_INSN_2_LBL (x, y) [BPF_##x | BPF_##y] = &&x##_##y#define BPF_INSN_3_LBL (x, y, z) [BPF_##x | BPF_##y | BPF_##z] = &&x##_##y##_##z -static const void *jumptable[256 ] = { +static const void *jump_table[256 ] = { [0 ...255 ] = &&default_label,/* Now overwrite non-defaults ... */ BPF_INSN_MAP (BPF_INSN_2_LBL ,BPF_INSN_3_LBL ),@@ -1315,7 +1315,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)#define CONT_JMP ({ insn++; goto select_insn; }) select_insn: - goto *jumptable[insn->code]; + goto *jump_table[insn->code];/* ALU */ #define ALU (OPCODE ,OP ) \@@ -1558,7 +1558,6 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)BUG_ON (1 );return 0 ; } -STACK_FRAME_NON_STANDARD (___bpf_prog_run);/* jump table */ #define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \tools/objtool/check.c +0
-0@@ -18,6 +18,8 @@
Quoted 8 lines
tools/objtool/check.c line 21
Quoted 3 lines
tools/objtool/check.c +1
-0#define FAKE_JUMP_OFFSET -1 +#define JUMP_TABLE_SYM_PREFIX "jump_table."since external tool will be looking at it should it be named "
bpf_jump_table." to avoid potential name conflicts? Or even more unique name?
Like "bpf_interpreter_jump_table." ?
If external code might need to process such symbols then jump_table_bpf_interpreter would (probably) make the symbols easier to locate.
Oh, and blue is a good colour :-)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
PATCH v2 3/5 x86/bpf: Move epilogue generation to a dedicated function
In reply to Josh PoimboeufImprove code readability by moving the BPF JIT function epilogue generation code to a dedicated emit_epilogue() function, analagous to the existing emit_prologue() function.
--- arch/x86/net/bpf_jit_comp.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
arch/x86/net/bpf_jit_comp.c +24 -13
@@ -240,6 +240,28 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) *pprog = prog; } +static void emit_epilogue (u8 **pprog) +{ +u8 *prog = *pprog; +int cnt =0 ; + +/* mov rbx, qword ptr [rbp+0] */ +EMIT4 (0x48 ,0x8B ,0x5D ,0 ); +/* mov r13, qword ptr [rbp+8] */ +EMIT4 (0x4C ,0x8B ,0x6D ,8 ); +/* mov r14, qword ptr [rbp+16] */ +EMIT4 (0x4C ,0x8B ,0x75 ,16 ); +/* mov r15, qword ptr [rbp+24] */ +EMIT4 (0x4C ,0x8B ,0x7D ,24 ); + +/* add rbp, AUX_STACK_SPACE */ +EMIT4 (0x48 ,0x83 ,0xC5 ,AUX_STACK_SPACE ); +EMIT1 (0xC9 );/* leave */ +EMIT1 (0xC3 );/* ret */ + + *pprog = prog; +} +/* * Generate the following code: *
@@ -1036,19 +1058,8 @@ xadd: if (is_imm8(insn->off)) seen_exit = true;/* Update cleanup_addr */ ctx->cleanup_addr = proglen; -/* mov rbx, qword ptr [rbp+0] */ -EMIT4 (0x48 ,0x8B ,0x5D ,0 ); -/* mov r13, qword ptr [rbp+8] */ -EMIT4 (0x4C ,0x8B ,0x6D ,8 ); -/* mov r14, qword ptr [rbp+16] */ -EMIT4 (0x4C ,0x8B ,0x75 ,16 ); -/* mov r15, qword ptr [rbp+24] */ -EMIT4 (0x4C ,0x8B ,0x7D ,24 ); - -/* add rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xC5 ,AUX_STACK_SPACE ); -EMIT1 (0xC9 );/* leave */ -EMIT1 (0xC3 );/* ret */ + +emit_epilogue (&prog);break ;default :
Signature · 1 line
2.20.1
PATCH v2 4/5 x86/bpf: Fix 64-bit JIT frame pointer usage
In reply to Josh PoimboeufThe BPF JIT code clobbers RBP. This breaks frame pointer convention and thus prevents the FP unwinder from unwinding through JIT generated code.
Fix it by saving the new RBP value on the stack before updating it. This effectively creates a new stack frame which the unwinder can understand.
Also, simplify the BPF JIT prologue such that it more closely resembles a typical compiler-generated prologue. This also reduces the prologue size quite a bit overall.
--- arch/x86/net/bpf_jit_comp.c | 106 ++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 52 deletions(-)
arch/x86/net/bpf_jit_comp.c +54 -52
@@ -186,56 +186,54 @@ struct jit_context {#define BPF_MAX_INSN_SIZE 128#define BPF_INSN_SAFETY 64 -#define AUX_STACK_SPACE 40/* Space for RBX, R13, R14, R15, tailcnt */ - -#define PROLOGUE_SIZE 37 +#define PROLOGUE_SIZE 24/* * Emit x86-64 prologue code for BPF program and check its size. * bpf_tail_call helper will skip it while jumping into another program */ -static void emit_prologue (u8 **pprog,u32 stack_depth,bool ebpf_from_cbpf) +static void emit_prologue (u8 **pprog,u32 stack_depth) {u8 *prog = *pprog;int cnt =0 ;/* push rbp */ EMIT1 (0x55 ); - -/* mov rbp,rsp */ +/* mov rbp, rsp */ EMIT3 (0x48 ,0x89 ,0xE5 ); -/* sub rsp, rounded_stack_depth + AUX_STACK_SPACE */ -EMIT3_off32 (0x48 ,0x81 ,0xEC , -round_up (stack_depth,8 ) +AUX_STACK_SPACE ); +/* push r15 */ +EMIT2 (0x41 ,0x57 ); +/* push r14 */ +EMIT2 (0x41 ,0x56 ); +/* push r13 */ +EMIT2 (0x41 ,0x55 ); +/* push rbx */ +EMIT1 (0x53 ); -/* sub rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xED ,AUX_STACK_SPACE ); - -/* mov qword ptr [rbp+0],rbx */ -EMIT4 (0x48 ,0x89 ,0x5D ,0 ); -/* mov qword ptr [rbp+8],r13 */ -EMIT4 (0x4C ,0x89 ,0x6D ,8 ); -/* mov qword ptr [rbp+16],r14 */ -EMIT4 (0x4C ,0x89 ,0x75 ,16 ); -/* mov qword ptr [rbp+24],r15 */ -EMIT4 (0x4C ,0x89 ,0x7D ,24 ); +/* +* Push the tail call counter (tail_call_cnt) for eBPF tail calls. +* Initialized to zero. +* +* push $0 +*/ +EMIT2 (0x6a ,0x00 ); -if (!ebpf_from_cbpf) { -/* -* Clear the tail call counter (tail_call_cnt): for eBPF tail -* calls we need to reset the counter to 0. It's done in two -* instructions, resetting RAX register to 0, and moving it -* to the counter location. -*/ +/* +* RBP is used for the BPF program's FP register. It points to the end +* of the program's stack area. Create another stack frame so the +* unwinder can unwind through the generated code. The tail_call_cnt +* value doubles as an (invalid) RIP address. +*/ +/* push rbp */ +EMIT1 (0x55 ); +/* mov rbp, rsp */ +EMIT3 (0x48 ,0x89 ,0xE5 ); -/* xor eax, eax */ -EMIT2 (0x31 ,0xc0 ); -/* mov qword ptr [rbp+32], rax */ -EMIT4 (0x48 ,0x89 ,0x45 ,32 ); +/* sub rsp, rounded_stack_depth */ +EMIT3_off32 (0x48 ,0x81 ,0xEC ,round_up (stack_depth,8 )); -BUILD_BUG_ON (cnt !=PROLOGUE_SIZE ); - } +BUILD_BUG_ON (cnt !=PROLOGUE_SIZE ); *pprog = prog; }
@@ -245,19 +243,24 @@ static void emit_epilogue(u8 **pprog)u8 *prog = *pprog;int cnt =0 ; -/* mov rbx, qword ptr [rbp+0] */ -EMIT4 (0x48 ,0x8B ,0x5D ,0 ); -/* mov r13, qword ptr [rbp+8] */ -EMIT4 (0x4C ,0x8B ,0x6D ,8 ); -/* mov r14, qword ptr [rbp+16] */ -EMIT4 (0x4C ,0x8B ,0x75 ,16 ); -/* mov r15, qword ptr [rbp+24] */ -EMIT4 (0x4C ,0x8B ,0x7D ,24 ); - -/* add rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xC5 ,AUX_STACK_SPACE ); -EMIT1 (0xC9 );/* leave */ -EMIT1 (0xC3 );/* ret */ +/* leave (restore rsp and rbp) */ +EMIT1 (0xC9 ); +/* pop rbx (skip over tail_call_cnt) */ +EMIT1 (0x5B ); + +/* pop rbx */ +EMIT1 (0x5B ); +/* pop r13 */ +EMIT2 (0x41 ,0x5D ); +/* pop r14 */ +EMIT2 (0x41 ,0x5E ); +/* pop r15 */ +EMIT2 (0x41 ,0x5F ); +/* pop rbp */ +EMIT1 (0x5D ); + +/* ret */ +EMIT1 (0xC3 ); *pprog = prog; }
@@ -295,7 +298,7 @@ static void emit_bpf_tail_call(u8 **pprog)EMIT2 (0x89 ,0xD2 );/* mov edx, edx */ EMIT3 (0x39 ,0x56 ,/* cmp dword ptr [rsi + 16], edx */ offsetof(struct bpf_array, map.max_entries)); -#define OFFSET1 (41 + RETPOLINE_RAX_BPF_JIT_SIZE)/* Number of bytes to jump */ +#define OFFSET1 (35 + RETPOLINE_RAX_BPF_JIT_SIZE)/* Number of bytes to jump */ EMIT2 (X86_JBE ,OFFSET1 );/* jbe out */ label1 = cnt;
@@ -303,13 +306,13 @@ static void emit_bpf_tail_call(u8 **pprog)* if (tail_call_cnt > MAX_TAIL_CALL_CNT) * goto out; */ -EMIT2_off32 (0x8B ,0x85 ,36 );/* mov eax, dword ptr [rbp + 36] */ +EMIT3 (0x8B ,0x45 ,0x0C );/* mov eax, dword ptr [rbp + 12] */ EMIT3 (0x83 ,0xF8 ,MAX_TAIL_CALL_CNT );/* cmp eax, MAX_TAIL_CALL_CNT */ -#define OFFSET2 (30 + RETPOLINE_RAX_BPF_JIT_SIZE) +#define OFFSET2 (27 + RETPOLINE_RAX_BPF_JIT_SIZE)EMIT2 (X86_JA ,OFFSET2 );/* ja out */ label2 = cnt;EMIT3 (0x83 ,0xC0 ,0x01 );/* add eax, 1 */ -EMIT2_off32 (0x89 ,0x85 ,36 );/* mov dword ptr [rbp + 36], eax */ +EMIT3 (0x89 ,0x45 ,0x0C );/* mov dword ptr [rbp + 12], eax */ /* prog = array->ptrs[index]; */ EMIT4_off32(0x48, 0x8B, 0x84, 0xD6, /* mov rax, [rsi + rdx * 8 + offsetof(...)] */
@@ -437,8 +440,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,int proglen =0 ;u8 *prog = temp; -emit_prologue (&prog, bpf_prog->aux->stack_depth, -bpf_prog_was_classic (bpf_prog)); +emit_prologue (&prog, bpf_prog->aux->stack_depth);for (i =0 ; i < insn_cnt; i++, insn++) {const s32 imm32 = insn->imm;
Signature · 1 line
2.20.1
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 85 lines · Josh Poimboeuf
The BPF JIT code clobbers RBP. This breaks frame pointer convention and thus prevents the FP unwinder from unwinding through JIT generated code.
Fix it by saving the new RBP value on the stack before updating it. This effectively creates a new stack frame which the unwinder can understand.
Also, simplify the BPF JIT prologue such that it more closely resembles a typical compiler-generated prologue. This also reduces the prologue size quite a bit overall.
Suggested-by David Laight <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 106 ++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 52 deletions(-)arch/x86/net/bpf_jit_comp.c +18
-27@@ -186,56 +186,54 @@ struct jit_context {#define BPF_MAX_INSN_SIZE 128#define BPF_INSN_SAFETY 64 -#define AUX_STACK_SPACE 40/* Space for RBX, R13, R14, R15, tailcnt */ - -#define PROLOGUE_SIZE 37 +#define PROLOGUE_SIZE 24/* * Emit x86-64 prologue code for BPF program and check its size. * bpf_tail_call helper will skip it while jumping into another program */ -static void emit_prologue (u8 **pprog,u32 stack_depth,bool ebpf_from_cbpf) +static void emit_prologue (u8 **pprog,u32 stack_depth) {u8 *prog = *pprog;int cnt =0 ;/* push rbp */ EMIT1 (0x55 ); - -/* mov rbp,rsp */ +/* mov rbp, rsp */ EMIT3 (0x48 ,0x89 ,0xE5 ); -/* sub rsp, rounded_stack_depth + AUX_STACK_SPACE */ -EMIT3_off32 (0x48 ,0x81 ,0xEC , -round_up (stack_depth,8 ) +AUX_STACK_SPACE ); +/* push r15 */ +EMIT2 (0x41 ,0x57 ); +/* push r14 */ +EMIT2 (0x41 ,0x56 ); +/* push r13 */ +EMIT2 (0x41 ,0x55 ); +/* push rbx */ +EMIT1 (0x53 ); -/* sub rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xED ,AUX_STACK_SPACE ); - -/* mov qword ptr [rbp+0],rbx */ -EMIT4 (0x48 ,0x89 ,0x5D ,0 ); -/* mov qword ptr [rbp+8],r13 */ -EMIT4 (0x4C ,0x89 ,0x6D ,8 ); -/* mov qword ptr [rbp+16],r14 */ -EMIT4 (0x4C ,0x89 ,0x75 ,16 ); -/* mov qword ptr [rbp+24],r15 */ -EMIT4 (0x4C ,0x89 ,0x7D ,24 ); +/* +* Push the tail call counter (tail_call_cnt) for eBPF tail calls. +* Initialized to zero. +* +* push $0 +*/ +EMIT2 (0x6a ,0x00 ); -if (!ebpf_from_cbpf) { -/* -* Clear the tail call counter (tail_call_cnt): for eBPF tail -* calls we need to reset the counter to 0. It's done in two -* instructions, resetting RAX register to 0, and moving it -* to the counter location. -*/
arch/x86/net/bpf_jit_comp.c struct jit_context { · line 231
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+/* +* RBP is used for the BPF program's FP register. It points to the end +* of the program's stack area. Create another stack frame so the +* unwinder can unwind through the generated code. The tail_call_cnt +* value doubles as an (invalid) RIP address. +*/ +/* push rbp */ +EMIT1 (0x55 ); +/* mov rbp, rsp */ +EMIT3 (0x48 ,0x89 ,0xE5 );
Have you tested it ?
I really doubt, since in my test both CONFIG_UNWINDER_ORC and CONFIG_UNWINDER_FRAME_POINTER failed to unwind through such odd frame.
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
It makes one proper frame and both frame and orc unwinders are happy.
From 442d91571a7f7f92a5cd3bd4a1b139390befbee3 Mon Sep 17 00:00:00 2001
From: Alexei Starovoitov <[email protected]>
Date: Fri, 14 Jun 2019 12:56:43 -0500
Subject: [PATCH bpf] x86/bpf: Fix 64-bit JIT frame pointer usage
The BPF JIT code clobbers RBP. This breaks frame pointer convention and thus prevents the FP unwinder from unwinding through JIT generated code. Fix it by moving callee saved space to the bottom.
Similar to how it was before commit 177366bf7ceb.
--- arch/x86/net/bpf_jit_comp.c | 84 +++++++++++++++---------------------- 1 file changed, 34 insertions(+), 50 deletions(-)
arch/x86/net/bpf_jit_comp.c +34 -50
@@ -190,56 +190,38 @@ struct jit_context {#define BPF_MAX_INSN_SIZE 128#define BPF_INSN_SAFETY 64 -#define AUX_STACK_SPACE 40/* Space for RBX, R13, R14, R15, tailcnt */ - -#define PROLOGUE_SIZE 37 +#define PROLOGUE_SIZE 20/* * Emit x86-64 prologue code for BPF program and check its size. * bpf_tail_call helper will skip it while jumping into another program */ -static void emit_prologue (u8 **pprog,u32 stack_depth,bool ebpf_from_cbpf) +static void emit_prologue (u8 **pprog,u32 stack_depth) {u8 *prog = *pprog;int cnt =0 ;/* push rbp */ EMIT1 (0x55 ); - -/* mov rbp,rsp */ +/* mov rbp, rsp */ EMIT3 (0x48 ,0x89 ,0xE5 ); -/* sub rsp, rounded_stack_depth + AUX_STACK_SPACE */ -EMIT3_off32 (0x48 ,0x81 ,0xEC , -round_up (stack_depth,8 ) +AUX_STACK_SPACE ); - -/* sub rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xED ,AUX_STACK_SPACE ); - -/* mov qword ptr [rbp+0],rbx */ -EMIT4 (0x48 ,0x89 ,0x5D ,0 ); -/* mov qword ptr [rbp+8],r13 */ -EMIT4 (0x4C ,0x89 ,0x6D ,8 ); -/* mov qword ptr [rbp+16],r14 */ -EMIT4 (0x4C ,0x89 ,0x75 ,16 ); -/* mov qword ptr [rbp+24],r15 */ -EMIT4 (0x4C ,0x89 ,0x7D ,24 ); +/* sub rsp, rounded_stack_depth */ +EMIT3_off32 (0x48 ,0x81 ,0xEC ,round_up (stack_depth,8 )); -if (!ebpf_from_cbpf) { -/* -* Clear the tail call counter (tail_call_cnt): for eBPF tail -* calls we need to reset the counter to 0. It's done in two -* instructions, resetting RAX register to 0, and moving it -* to the counter location. -*/ +/* push r15 */ +EMIT2 (0x41 ,0x57 ); +/* push r14 */ +EMIT2 (0x41 ,0x56 ); +/* push r13 */ +EMIT2 (0x41 ,0x55 ); +/* push rbx */ +EMIT1 (0x53 ); -/* xor eax, eax */ -EMIT2 (0x31 ,0xc0 ); -/* mov qword ptr [rbp+32], rax */ -EMIT4 (0x48 ,0x89 ,0x45 ,32 ); +/* zero init tail_call_cnt */ +EMIT2 (0x6a ,0x00 ); -BUILD_BUG_ON (cnt !=PROLOGUE_SIZE ); - } +BUILD_BUG_ON (cnt !=PROLOGUE_SIZE ); *pprog = prog; }
@@ -249,19 +231,22 @@ static void emit_epilogue(u8 **pprog)u8 *prog = *pprog;int cnt =0 ; -/* mov rbx, qword ptr [rbp+0] */ -EMIT4 (0x48 ,0x8B ,0x5D ,0 ); -/* mov r13, qword ptr [rbp+8] */ -EMIT4 (0x4C ,0x8B ,0x6D ,8 ); -/* mov r14, qword ptr [rbp+16] */ -EMIT4 (0x4C ,0x8B ,0x75 ,16 ); -/* mov r15, qword ptr [rbp+24] */ -EMIT4 (0x4C ,0x8B ,0x7D ,24 ); +/* pop rbx (skip over tail_call_cnt) */ +EMIT1 (0x5B ); + +/* pop rbx */ +EMIT1 (0x5B ); +/* pop r13 */ +EMIT2 (0x41 ,0x5D ); +/* pop r14 */ +EMIT2 (0x41 ,0x5E ); +/* pop r15 */ +EMIT2 (0x41 ,0x5F ); +/* leave (restore rsp and rbp) */ +EMIT1 (0xC9 ); -/* add rbp, AUX_STACK_SPACE */ -EMIT4 (0x48 ,0x83 ,0xC5 ,AUX_STACK_SPACE ); -EMIT1 (0xC9 );/* leave */ -EMIT1 (0xC3 );/* ret */ +/* ret */ +EMIT1 (0xC3 ); *pprog = prog; }
@@ -307,13 +292,13 @@ static void emit_bpf_tail_call(u8 **pprog)* if (tail_call_cnt > MAX_TAIL_CALL_CNT) * goto out; */ -EMIT2_off32 (0x8B ,0x85 ,36 );/* mov eax, dword ptr [rbp + 36] */ +EMIT2_off32 (0x8B ,0x85 ,-36 -MAX_BPF_STACK );/* mov eax, dword ptr [rbp - 548] */ EMIT3 (0x83 ,0xF8 ,MAX_TAIL_CALL_CNT );/* cmp eax, MAX_TAIL_CALL_CNT */ #define OFFSET2 (30 + RETPOLINE_RAX_BPF_JIT_SIZE)EMIT2 (X86_JA ,OFFSET2 );/* ja out */ label2 = cnt;EMIT3 (0x83 ,0xC0 ,0x01 );/* add eax, 1 */ -EMIT2_off32 (0x89 ,0x85 ,36 );/* mov dword ptr [rbp + 36], eax */ +EMIT2_off32 (0x89 ,0x85 ,-36 -MAX_BPF_STACK );/* mov dword ptr [rbp -548], eax */ /* prog = array->ptrs[index]; */ EMIT4_off32(0x48, 0x8B, 0x84, 0xD6, /* mov rax, [rsi + rdx * 8 + offsetof(...)] */
@@ -441,8 +426,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,int proglen =0 ;u8 *prog = temp; -emit_prologue (&prog, bpf_prog->aux->stack_depth, -bpf_prog_was_classic (bpf_prog)); +emit_prologue (&prog, bpf_prog->aux->stack_depth);for (i =0 ; i < insn_cnt; i++, insn++) {const s32 imm32 = insn->imm;
Signature · 1 line
2.20.0
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.
Hm, are you seeing selftest failures? They seem to work for me.
Quoted 4 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Some of the code and patch description look familiar, please be sure to give me proper credit.
Signature · 1 line
Josh
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 11 lines · Josh Poimboeuf
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.Hm, are you seeing selftest failures? They seem to work for me.
Quoted 3 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
Quoted 6 lines
Quoted 1 line
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Some of the code and patch description look familiar, please be sure to give me proper credit.
credit means something positive.
your contribution to bpf jit fix was negative. I'm going to rewrite the fix from relying on patch 3.
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 12 lines · Alexei Starovoitov
Quoted 11 lines · Josh Poimboeuf
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.Hm, are you seeing selftest failures? They seem to work for me.
Quoted 3 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Some of the code and patch description look familiar, please be sure to give me proper credit.
credit means something positive.
So you only give credit for *good* stolen code. I must have missed that section of the kernel patch guidelines.
Ironically you posted this patch right after you accused me of removing Song's SOB (which I didn't).
Quoted 1 line
your contribution to bpf jit fix was negative.
You have a delightful personality.
Signature · 1 line
Josh
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 18 lines · Josh Poimboeuf
Quoted 16 lines · Alexei Starovoitov
Quoted 15 lines · Josh Poimboeuf
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.Hm, are you seeing selftest failures? They seem to work for me.
Quoted 4 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Some of the code and patch description look familiar, please be sure to give me proper credit.
credit means something positive.
So you only give credit for *good* stolen code. I must have missed that section of the kernel patch guidelines.
what are you talking about?
you've posted one bad patch. I pointed out multiple issues in it. Then proposed another bad idea. I pointed out another set of issues. Than David proposed yet another idea that you've implemented and claimed that it's working when it was not.
Then I got fed up with this thread and fix it for real by reverting that old commit that I mentioned way earlier.
https://patchwork.ozlabs.org/patch/1116307/
Where do you see your code or ideas being used?
I see none.
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 27 lines · Alexei Starovoitov
Quoted 18 lines · Josh Poimboeuf
Quoted 16 lines · Alexei Starovoitov
Quoted 15 lines · Josh Poimboeuf
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.Hm, are you seeing selftest failures? They seem to work for me.
Quoted 4 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Some of the code and patch description look familiar, please be sure to give me proper credit.
credit means something positive.
So you only give credit for *good* stolen code. I must have missed that section of the kernel patch guidelines.
Quoted 10 lines
what are you talking about?
you've posted one bad patch. I pointed out multiple issues in it. Then proposed another bad idea. I pointed out another set of issues. Than David proposed yet another idea that you've implemented and claimed that it's working when it was not.
Then I got fed up with this thread and fix it for real by reverting that old commit that I mentioned way earlier.
https://patchwork.ozlabs.org/patch/1116307/
Where do you see your code or ideas being used?
I see none.
Obviously I wasn't referring to this new whitewashed patch for which I wasn't even on Cc, despite being one of the people (along with Peter Z) who convinced you that there was a problem to begin with.
The previous patch you posted has my patch description, push/pop and comment changes, with no credit:
https://lkml.kernel.org/r/[email protected]
Signature · 1 line
Josh
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 40 lines · Josh Poimboeuf
Quoted 27 lines · Alexei Starovoitov
Quoted 18 lines · Josh Poimboeuf
Quoted 16 lines · Alexei Starovoitov
Quoted 15 lines · Josh Poimboeuf
Quoted 3 lines · Alexei Starovoitov
Have you tested it ?
I really doubt, since in my test bothCONFIG_UNWINDER_ORCandCONFIG_UNWINDER_FRAME_POINTERfailed to unwind through such odd frame.Hm, are you seeing selftest failures? They seem to work for me.
Quoted 4 lines
Here is much simple patch that I mentioned in the email yesterday, but you failed to listen instead of focusing on perceived 'code readability'.
It makes one proper frame and both frame and orc unwinders are happy.
I'm on my way out the door and I just skimmed it, but it looks fine.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Some of the code and patch description look familiar, please be sure to give me proper credit.
credit means something positive.
So you only give credit for *good* stolen code. I must have missed that section of the kernel patch guidelines.
Quoted 10 lines
what are you talking about?
you've posted one bad patch. I pointed out multiple issues in it. Then proposed another bad idea. I pointed out another set of issues. Than David proposed yet another idea that you've implemented and claimed that it's working when it was not.
Then I got fed up with this thread and fix it for real by reverting that old commit that I mentioned way earlier.
https://patchwork.ozlabs.org/patch/1116307/
Where do you see your code or ideas being used?
I see none.
Quoted 3 lines
Obviously I wasn't referring to this new whitewashed patch for which I wasn't even on Cc, despite being one of the people (along with Peter Z) who convinced you that there was a problem to begin with.
vger has a small limit on cc list. I always trim it to the minimum.
Quoted 2 lines
The previous patch you posted has my patch description, push/pop and comment changes, with no credit:
I'm sorry for reusing one sentence from your commit log and not realizing you want credit for that.
Will not happen again.
I also suggest you never touch anything bpf related. Just to avoid this credit claims and threads like this one.
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 9 lines · Alexei Starovoitov
Quoted 4 lines · Josh Poimboeuf
The previous patch you posted has my patch description, push/pop and comment changes, with no credit:
I'm sorry for reusing one sentence from your commit log and not realizing you want credit for that.
Will not happen again.
Um. What are you talking about? The entire patch was clearly derived from mine. Not just "one sentence from your commit log". The title, the pushes/pops in the prologue/epilogue, the removal of the "ebpf_from_cbpf" argument, the code spacing, and some of the non trivial comment changes were the same.
Quoted 2 lines
I also suggest you never touch anything bpf related. Just to avoid this credit claims and threads like this one.
Wth. I made a simple request for credit. Anybody can see the patch was derived from mine. It's not like I really care. It's just basic human decency.
Signature · 1 line
Josh
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 18 lines · Josh Poimboeuf
Quoted 9 lines · Alexei Starovoitov
Quoted 4 lines · Josh Poimboeuf
The previous patch you posted has my patch description, push/pop and comment changes, with no credit:
I'm sorry for reusing one sentence from your commit log and not realizing you want credit for that.
Will not happen again.Um. What are you talking about? The entire patch was clearly derived from mine. Not just "one sentence from your commit log". The title, the pushes/pops in the prologue/epilogue, the removal of the "
ebpf_from_cbpf" argument, the code spacing, and some of the non trivial comment changes were the same.
Quoted 6 lines
Quoted 2 lines
I also suggest you never touch anything bpf related. Just to avoid this credit claims and threads like this one.
Wth. I made a simple request for credit. Anybody can see the patch was derived from mine. It's not like I really care. It's just basic human decency.
derived? do you really think so ?
Please fix your orc stuff that is still broken.
Human decency is fixing stuff that you're responsible for. Your commit d15d356887e7 on April 23 broke stack traces. And we reported it 3 weeks ago.
Yet instead of fixing it you kept arguing about JIT frame pointers that is orthogonal issue and was in this state for the last 2 years.
Re: [PATCH v2 4/5] x86/bpf: Fix 64-bit JIT frame pointer usage
Quoted 26 lines · Alexei Starovoitov
Quoted 18 lines · Josh Poimboeuf
Quoted 9 lines · Alexei Starovoitov
Quoted 4 lines · Josh Poimboeuf
The previous patch you posted has my patch description, push/pop and comment changes, with no credit:
I'm sorry for reusing one sentence from your commit log and not realizing you want credit for that.
Will not happen again.Um. What are you talking about? The entire patch was clearly derived from mine. Not just "one sentence from your commit log". The title, the pushes/pops in the prologue/epilogue, the removal of the "
ebpf_from_cbpf" argument, the code spacing, and some of the non trivial comment changes were the same.Quoted 6 lines
Quoted 2 lines
I also suggest you never touch anything bpf related. Just to avoid this credit claims and threads like this one.
Wth. I made a simple request for credit. Anybody can see the patch was derived from mine. It's not like I really care. It's just basic human decency.
Quoted 7 lines
derived? do you really think so ?
Please fix your orc stuff that is still broken.
Human decency is fixing stuff that you're responsible for. Your commitd15d356887e7on April 23 broke stack traces. And we reported it 3 weeks ago.
Yet instead of fixing it you kept arguing about JIT frame pointers that is orthogonal issue and was in this state for the last 2 years.
Again you're not making sense. The fix has already been posted. That was the point of this patch set.
It's your call if you want to cherry pick the FP fix (which is a dependency of the ORC fix) without taking the others.
Signature · 1 line
Josh
PATCH v2 5/5 x86/unwind/orc: Fall back to using frame pointers for generated code
In reply to Josh PoimboeufThe ORC unwinder can't unwind through BPF JIT generated code because there are no ORC entries associated with the code.
If an ORC entry isn't available, try to fall back to frame pointers. If BPF and other generated code always do frame pointer setup (even with CONFIG_FRAME_POINTERS=n) then this will allow ORC to unwind through most generated code despite there being no corresponding ORC entries.
--- arch/x86/kernel/unwind_orc.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-)
arch/x86/kernel/unwind_orc.c +22 -4
@@ -82,9 +82,9 @@ static struct orc_entry *orc_find(unsigned long ip);* But they are copies of the ftrace entries that are static and * defined in ftrace_*.S, which do have orc entries. * -* If the undwinder comes across a ftrace trampoline, then find the +* If the unwinder comes across a ftrace trampoline, then find the * ftrace function that was used to create it, and use that ftrace -* function's orc entrie, as the placement of the return code in +* function's orc entry, as the placement of the return code in * the stack will be identical. */ static struct orc_entry *orc_ftrace_find (unsigned long ip)
@@ -128,6 +128,16 @@ static struct orc_entry null_orc_entry = { .type = ORC_TYPE_CALL }; +/* Fake frame pointer entry -- used as a fallback for generated code */ +static struct orc_entry orc_fp_entry = { + .type =ORC_TYPE_CALL , + .sp_reg =ORC_REG_BP , + .sp_offset =16 , + .bp_reg =ORC_REG_PREV_SP , + .bp_offset =-16 , + .end =0 , +}; +static struct orc_entry *orc_find (unsigned long ip) {static struct orc_entry *orc;
@@ -392,8 +402,16 @@ bool unwind_next_frame(struct unwind_state *state)* calls and calls to noreturn functions. */ orc =orc_find (state->signal ? state->ip : state->ip -1 ); -if (!orc) - goto err; +if (!orc) { +/* +* As a fallback, try to assume this code uses a frame pointer. +* This is useful for generated code, like BPF, which ORC +* doesn't know about. This is just a guess, so the rest of +* the unwind is no longer considered reliable. +*/ + orc = &orc_fp_entry; + state->error = true; + }/* End-of-stack check for kernel threads: */ if (orc->sp_reg ==ORC_REG_UNDEFINED ) {
Signature · 1 line
2.20.1