kprobes: rethook: x86: Replace kretprobe trampoline with rethook
Subsystems (6)
Masami Hiramatsu
maintainerv3 cover kprobes: rethook: x86: Replace kretprobe trampoline with rethook
Hi,
Here are the 3rd version for generic kretprobe and kretprobe on x86 for replacing the kretprobe trampoline with rethook. The previous version is here[1]
[1] https://lore.kernel.org/all/164821817332.2373735.12048266953420821089.stgit@devnote2/T/#u
This version fixed typo and build issues for bpf-next and CONFIG_RETHOOK=y error. I also add temporary mitigation lines for ANNOTATE_NOENDBR macro issue for bpf-next tree [2/4].
#ifndef ANNOTATE_NOENDBR
#define ANNOTATE_NOENDBR
#endif
This will be removed after merging kernel IBT series.
Background:
This rethook came from Jiri's request of multiple kprobe for bpf[2]. He tried to solve an issue that starting bpf with multiple kprobe will take a long time because bpf-kprobe will wait for RCU grace period for sync rcu events.
Jiri wanted to attach a single bpf handler to multiple kprobes and he tried to introduce multiple-probe interface to kprobe. So I asked him to use ftrace and kretprobe-like hook if it is only for the function entry and exit, instead of adding ad-hoc interface to kprobes.
For this purpose, I introduced the fprobe (kprobe like interface for ftrace) with the rethook (this is a generic return hook feature for fprobe exit handler)[3].
[2] https://lore.kernel.org/all/[email protected]/T/#u
[3] https://lore.kernel.org/all/164191321766.806991.7930388561276940676.stgit@devnote2/T/#u
The rethook is basically same as the kretprobe trampoline. I just made it decoupled from kprobes. Eventually, the all arch dependent kretprobe trampolines will be replaced with the rethook trampoline instead of cloning and set HAVE_RETHOOK=y.
When I port the rethook for all arch which supports kretprobe, the legacy kretprobe specific code (which is for CONFIG_KRETPROBE_ON_RETHOOK=n) will be removed eventually.
Thank you,
---
Masami Hiramatsu (3):
kprobes: Use rethook for kretprobe if possible x86,rethook,kprobes: Replace kretprobe with rethook on x86 x86,kprobes: Fix optprobe trampoline to generate complete pt_regs
Peter Zijlstra (1):
x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
arch/Kconfig | 8 ++ arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 106 -------------------------------- arch/x86/kernel/kprobes/opt.c | 25 +++++-- arch/x86/kernel/rethook.c | 127 ++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/unwind_orc.c | 10 +-- include/linux/kprobes.h | 51 +++++++++++++++ kernel/Makefile | 1 kernel/kprobes.c | 124 +++++++++++++++++++++++++++++++------ kernel/trace/trace_kprobe.c | 4 + 13 files changed, 325 insertions(+), 157 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerPATCH v3 1/4 kprobes: Use rethook for kretprobe if possible
Use rethook for kretprobe function return hooking if the arch sets CONFIG_HAVE_RETHOOK=y. In this case, CONFIG_KRETPROBE_ON_RETHOOK is set to 'y' automatically, and the kretprobe internal data fields switches to use rethook. If not, it continues to use kretprobe specific function return hooks.
---
Changes in v3:
- Fix to build kernel/trace dir if CONFIG_RETHOOK (which will be enabled by CONFIG_KPROBES) is enabled.
- Fix CONFIG_KRETPROBE_ON_RETHOOK depends on CONFIG_KRETPROBES.
--- arch/Kconfig | 8 ++- include/linux/kprobes.h | 51 +++++++++++++++++- kernel/Makefile | 1 kernel/kprobes.c | 124 ++++++++++++++++++++++++++++++++++++------- kernel/trace/trace_kprobe.c | 4 + 5 files changed, 163 insertions(+), 25 deletions(-)
arch/Kconfig +7 -1
@@ -164,7 +164,13 @@ config ARCH_USE_BUILTIN_BSWAP config KRETPROBES def_bool y - depends on KPROBES && HAVE_KRETPROBES + depends on KPROBES && (HAVE_KRETPROBES || HAVE_RETHOOK) + +config KRETPROBE_ON_RETHOOK + def_bool y + depends on HAVE_RETHOOK + depends on KRETPROBES + select RETHOOK config USER_RETURN_NOTIFIER bool
include/linux/kprobes.h +49 -2
109 lines not on this page · raw message
kernel/Makefile +1 -0
@@ -108,6 +108,7 @@ obj-$(CONFIG_TRACING) += trace/ obj-$(CONFIG_TRACE_CLOCK) += trace/ obj-$(CONFIG_RING_BUFFER) += trace/ obj-$(CONFIG_TRACEPOINTS) += trace/ +obj-$(CONFIG_RETHOOK) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/
kernel/kprobes.c +104 -20
180 lines not on this page · raw message
kernel/trace/trace_kprobe.c +2 -2
@@ -1433,7 +1433,7 @@ __kretprobe_trace_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, fbuffer.regs = regs; entry = fbuffer.entry =ring_buffer_event_data (fbuffer.event); entry->func = (unsigned long )tk->rp.kp.addr; - entry->ret_ip = (unsigned long )ri->ret_addr; + entry->ret_ip =get_kretprobe_retaddr (ri);store_trace_args (&entry[1 ], &tk->tp, regs,sizeof (*entry), dsize);trace_event_buffer_commit (&fbuffer);
@@ -1628,7 +1628,7 @@ kretprobe_perf_func(struct trace_kprobe *tk, struct kretprobe_instance *ri,return ; entry->func = (unsigned long )tk->rp.kp.addr; - entry->ret_ip = (unsigned long )ri->ret_addr; + entry->ret_ip =get_kretprobe_retaddr (ri);store_trace_args (&entry[1 ], &tk->tp, regs,sizeof (*entry), dsize);perf_trace_buf_submit (entry, size, rctx, call->event.type,1 , regs, head,NULL );
Masami Hiramatsu
maintainerPATCH v3 2/4 x86,rethook,kprobes: Replace kretprobe with rethook on x86
In reply to Masami HiramatsuReplaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.
---
Changes in v3:
- Add a temporary mitigation for bpf-next tree.
Changes in v2:
- Fix remaining unwind_recover_kretprobe() with unwind_recover_rethook() (Thanks Peter!)
- Replace remaining "kretprobe" with "rethook" in comment.
--- arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 106 -------------------------------- arch/x86/kernel/rethook.c | 125 ++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/unwind_orc.c | 10 ++- 7 files changed, 144 insertions(+), 123 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
arch/x86/Kconfig +1 -0
@@ -224,6 +224,7 @@ config X86 select HAVE_KPROBES_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES + select HAVE_RETHOOK select HAVE_KVM select HAVE_LIVEPATCH if X86_64 select HAVE_MIXED_BREAKPOINTS_REGS
arch/x86/include/asm/unwind.h +11 -12
@@ -4,7 +4,7 @@#include <linux/sched.h> #include <linux/ftrace.h> -#include <linux/kprobes.h> +#include <linux/rethook.h> #include <asm/ptrace.h> #include <asm/stacktrace.h>
@@ -16,7 +16,7 @@ struct unwind_state {unsigned long stack_mask;struct task_struct *task;int graph_idx; -#ifdef CONFIG_KRETPROBES +#if defined(CONFIG_RETHOOK )struct llist_node *kr_cur;#endif bool error;
@@ -104,19 +104,18 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size,#endif static inline -unsigned long unwind_recover_kretprobe (struct unwind_state *state, -unsigned long addr,unsigned long *addr_p) +unsigned long unwind_recover_rethook (struct unwind_state *state, +unsigned long addr,unsigned long *addr_p) { -#ifdef CONFIG_KRETPROBES -return is_kretprobe_trampoline (addr) ? -kretprobe_find_ret_addr (state->task, addr_p, &state->kr_cur) : - addr; -#else -return addr; +#ifdef CONFIG_RETHOOK +if (is_rethook_trampoline (addr)) +return rethook_find_ret_addr (state->task, (unsigned long )addr_p, + &state->kr_cur);#endif +return addr; } -/* Recover the return address modified by kretprobe and ftrace_graph. */ +/* Recover the return address modified by rethook and ftrace_graph. */ static inline unsigned long unwind_recover_ret_addr (struct unwind_state *state,unsigned long addr,unsigned long *addr_p)
@@ -125,7 +124,7 @@ unsigned long unwind_recover_ret_addr(struct unwind_state *state, ret =ftrace_graph_ret_addr (state->task, &state->graph_idx, addr, addr_p); -return unwind_recover_kretprobe (state, ret, addr_p); +return unwind_recover_rethook (state, ret, addr_p); }/*
arch/x86/kernel/Makefile +1 -0
@@ -103,6 +103,7 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o obj-$(CONFIG_X86_TSC) += trace_clock.o obj-$(CONFIG_TRACING) += trace.o +obj-$(CONFIG_RETHOOK) += rethook.o obj-$(CONFIG_CRASH_CORE) += crash_core_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += machine_kexec_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += relocate_kernel_$(BITS).o crash.o
arch/x86/kernel/kprobes/common.h +1 -0
@@ -6,6 +6,7 @@#include <asm/asm.h> #include <asm/frame.h> +#include <asm/insn.h> #ifdef CONFIG_X86_64
arch/x86/kernel/kprobes/core.c +0 -106
118 lines not on this page · raw message
arch/x86/kernel/rethook.c new file +125 -0
125 lines not on this page · raw message
arch/x86/kernel/unwind_orc.c +5 -5
@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state) }/* * There is a small chance to interrupt at the entry of -* __kretprobe_trampoline() where the ORC info doesn't exist. -* That point is right after the RET to __kretprobe_trampoline() +* arch_rethook_trampoline() where the ORC info doesn't exist. +* That point is right after the RET to arch_rethook_trampoline() * which was modified return address. -* At that point, the @addr_p of the unwind_recover_kretprobe() +* At that point, the @addr_p of the unwind_recover_rethook() * (this has to point the address of the stack entry storing * the modified return address) must be "SP - (a stack entry)" * because SP is incremented by the RET. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long ))); state->regs = (struct pt_regs *)sp; state->prev_regs =NULL ;
@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state) goto err; }/* See UNWIND_HINT_TYPE_REGS case comment. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long )));if (state->full_regs)
Masami Hiramatsu
maintainerPATCH v3 3/4 x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
In reply to Masami HiramatsuFrom: Peter Zijlstra <[email protected]>
Currently arch_rethook_trampoline() generates an almost complete pt_regs on-stack, everything except regs->ss that is, that currently points to the fake return address, which is not a valid segment descriptor.
Since interpretation of regs->[sb]p should be done in the context of regs->ss, and we have code actually doing that (see arch/x86/lib/insn-eval.c for instance), complete the job by also pushing ss.
This ensures that anybody who does do look at regs->ss doesn't mysteriously malfunction, avoiding much future pain.
--- arch/x86/kernel/rethook.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
arch/x86/kernel/rethook.c +13 -11
@@ -29,29 +29,31 @@ asm(/* Push a fake return address to tell the unwinder it's a rethook. */ " pushq $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 8', this will be fixed later. */ + " pushq $" __stringify(__KERNEL_DS) "\n" +/* Save the 'sp - 16', this will be fixed later. */ " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addq $8, %rsp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addq $16, %rsp\n" " popfq\n"#else /* Push a fake return address to tell the unwinder it's a rethook. */ " pushl $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 4', this will be fixed later. */ +" pushl %ss\n" +/* Save the 'sp - 8', this will be fixed later. */ " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addl $4, %esp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addl $8, %esp\n" " popfl\n"#endif ASM_RET
@@ -73,8 +75,8 @@ __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)#endif regs->ip = (unsigned long )&arch_rethook_trampoline; regs->orig_ax = ~0UL ; - regs->sp +=sizeof (long ); - frame_pointer = ®s->sp +1 ; + regs->sp +=2 *sizeof (long ); + frame_pointer = (long *)(regs +1 );/* * The return address at 'frame_pointer' is recovered by the
@@ -84,10 +86,10 @@ __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)rethook_trampoline_handler (regs, (unsigned long )frame_pointer);/* -* Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline() +* Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline() * can do RET right after POPF. */ - regs->sp = regs->flags; + *(unsigned long *)®s->ss = regs->flags; }NOKPROBE_SYMBOL (arch_rethook_trampoline_callback);
@@ -105,7 +107,7 @@ STACK_FRAME_NON_STANDARD_FP(arch_rethook_trampoline);void arch_rethook_fixup_return (struct pt_regs *regs,unsigned long correct_ret_addr) { -unsigned long *frame_pointer = ®s->sp +1 ; +unsigned long *frame_pointer = (void *)(regs +1 );/* Replace fake return address with real one. */ *frame_pointer = correct_ret_addr;
Masami Hiramatsu
maintainerPATCH v3 4/4 x86,kprobes: Fix optprobe trampoline to generate complete pt_regs
In reply to Masami HiramatsuCurrently the optprobe trampoline template code ganerate an almost complete pt_regs on-stack, everything except regs->ss. The 'regs->ss' points to the top of stack, which is not a valid segment decriptor.
As same as the rethook does, complete the job by also pushing ss.
--- arch/x86/kernel/kprobes/opt.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
arch/x86/kernel/kprobes/opt.c +16 -9
@@ -106,7 +106,8 @@ asm (".global optprobe_template_entry\n" "optprobe_template_entry:\n" #ifdef CONFIG_X86_64 -/* We don't bother saving the ss register */ + " pushq $" __stringify(__KERNEL_DS) "\n" +/* Save the 'sp - 8', this will be fixed later. */ " pushq %rsp\n" " pushfq\n" ".global optprobe_template_clac\n"
@@ -121,14 +122,17 @@ asm (".global optprobe_template_call\n" "optprobe_template_call:\n" ASM_NOP5 -/* Move flags to rsp */ +/* Copy 'regs->flags' into 'regs->ss'. */ " movq 18*8(%rsp), %rdx\n" -" movq %rdx, 19*8(%rsp)\n" +" movq %rdx, 20*8(%rsp)\n" RESTORE_REGS_STRING -/* Skip flags entry */ -" addq $8, %rsp\n" +/* Skip 'regs->flags' and 'regs->sp'. */ +" addq $16, %rsp\n" +/* And pop flags register from 'regs->ss'. */ " popfq\n"#else /* CONFIG_X86_32 */ +" pushl %ss\n" +/* Save the 'sp - 4', this will be fixed later. */ " pushl %esp\n" " pushfl\n" ".global optprobe_template_clac\n"
@@ -142,12 +146,13 @@ asm (".global optprobe_template_call\n" "optprobe_template_call:\n" ASM_NOP5 -/* Move flags into esp */ +/* Copy 'regs->flags' into 'regs->ss'. */ " movl 14*4(%esp), %edx\n" -" movl %edx, 15*4(%esp)\n" +" movl %edx, 16*4(%esp)\n" RESTORE_REGS_STRING -/* Skip flags entry */ -" addl $4, %esp\n" +/* Skip 'regs->flags' and 'regs->sp'. */ +" addl $8, %esp\n" +/* And pop flags register from 'regs->ss'. */ " popfl\n"#endif ".global optprobe_template_end\n"
@@ -179,6 +184,8 @@ optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs)kprobes_inc_nmissed_count (&op->kp); }else {struct kprobe_ctlblk *kcb =get_kprobe_ctlblk (); +/* Adjust stack pointer */ + regs->sp +=sizeof (long );/* Save skipped registers */ regs->cs = __KERNEL_CS;#ifdef CONFIG_X86_32
This series was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Sat, 26 Mar 2022 11:26:49 +0900 you wrote:
Hi,
Here are the 3rd version for generic kretprobe and kretprobe on x86 for replacing the kretprobe trampoline with rethook. The previous version is here[1]
[1] https://lore.kernel.org/all/164821817332.2373735.12048266953420821089.stgit@devnote2/T/#u
[...]
Here is the summary with links:
- [bpf-next,v3,1/4] kprobes: Use rethook for kretprobe if possible
https://git.kernel.org/bpf/bpf/c/73f9b911faa7 - [bpf-next,v3,2/4] x86,rethook,kprobes: Replace kretprobe with rethook on x86 https://git.kernel.org/bpf/bpf/c/f3a112c0c40d - [bpf-next,v3,3/4] x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs https://git.kernel.org/bpf/bpf/c/0ef6f5c09371 - [bpf-next,v3,4/4] x86,kprobes: Fix optprobe trampoline to generate complete pt_regs https://git.kernel.org/bpf/bpf/c/45c23bf4d1a4
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html