kprobes: rethook: x86: Replace kretprobe trampoline with rethook
Subsystems (6)
Masami Hiramatsu
maintainercover kprobes: rethook: x86: Replace kretprobe trampoline with rethook
Hi,
Here are the patch set for generic kretprobe and kretprobe on x86 for replacing the kretprobe trampoline with rethook. For the other archs, I will port rethook to those after this has been merged.
This is previously called as "rethook: x86: Add rethook x86 implementation" The previous thread is here[1].
[1] https://lore.kernel.org/all/164800288611.1716332.7053663723617614668.stgit@devnote2/T/#u
Background:
This rethook came from Jiri's request of multiple kprobe for bpf[1]. He tried to solve an issue that starting bpf with multiple kprobe will take a long time because bpf-kprobe will wait for RCU grace period for sync rcu events.
Jiri wanted to attach a single bpf handler to multiple kprobes and he tried to introduce multiple-probe interface to kprobe. So I asked him to use ftrace and kretprobe-like hook if it is only for the function entry and exit, instead of adding ad-hoc interface to kprobes.
For this purpose, I introduced the fprobe (kprobe like interface for ftrace) with the rethook (this is a generic return hook feature for fprobe exit handler)[2].
[1] https://lore.kernel.org/all/[email protected]/T/#u
[2] https://lore.kernel.org/all/164191321766.806991.7930388561276940676.stgit@devnote2/T/#u
The rethook is basically same as the kretprobe trampoline. I just made it decoupled from kprobes. Eventually, the all arch dependent kretprobe trampolines will be replaced with the rethook trampoline instead of cloning and set HAVE_RETHOOK=y.
When I port the rethook for all arch which supports kretprobe, the legacy kretprobe specific code (which is for CONFIG_KRETPROBE_ON_RETHOOK=n) will be removed eventually.
BTW, this patch can be applied to next-20220324, not the bpf-next tree directly, because this depends on ANNOTATE_NOENDBR macro. However, since the fprobe is merged in the bpf-next, I marked this for bpf-next. So until merging the both of fprobes and ENDBR series, to compile this you need below 2 lines in arch/x86/kernel/rethook.c.
#ifndef ANNOTATE_NOENDBR
#define ANNOTATE_NOENDBR
But after those are merged, these lines will be unneeded. How should I handle this issue? (Just remove ANNOTATE_NOENDBR line in bpf-next?)
Thank you,
---
Masami Hiramatsu (2):
kprobes: Use rethook for kretprobe if possible
rethook: kprobes: x86: Replace kretprobe with rethook on x86
arch/Kconfig | 7 ++ arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 107 --------------------------------- arch/x86/kernel/rethook.c | 121 +++++++++++++++++++++++++++++++++++++ include/linux/kprobes.h | 51 +++++++++++++++- kernel/kprobes.c | 124 ++++++++++++++++++++++++++++++++------ kernel/trace/trace_kprobe.c | 4 + 10 files changed, 296 insertions(+), 144 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerPATCH 1/2 kprobes: Use rethook for kretprobe if possible
Use rethook for kretprobe function return hooking if the arch sets CONFIG_HAVE_RETHOOK=y. In this case, CONFIG_KRETPROBE_ON_RETHOOK is set to 'y' automatically, and the kretprobe internal data fields switches to use rethook. If not, it continues to use kretprobe specific function return hooks.
--- arch/Kconfig | 7 ++ include/linux/kprobes.h | 51 +++++++++++++++++- kernel/kprobes.c | 124 ++++++++++++++++++++++++++++++++++++------- kernel/trace/trace_kprobe.c | 4 + 4 files changed, 161 insertions(+), 25 deletions(-)
arch/Kconfig +6 -1
@@ -164,7 +164,12 @@ config ARCH_USE_BUILTIN_BSWAP config KRETPROBES def_bool y - depends on KPROBES && HAVE_KRETPROBES + depends on KPROBES && (HAVE_KRETPROBES || HAVE_RETHOOK) + +config KRETPROBE_ON_RETHOOK + def_bool y + depends on HAVE_RETHOOK + select RETHOOK config USER_RETURN_NOTIFIER bool
include/linux/kprobes.h +49 -2
109 lines not on this page · raw message
kernel/kprobes.c +104 -20
180 lines not on this page · raw message
kernel/trace/trace_kprobe.c +2 -2
@@ -1433,7 +1433,7 @@ __kretprobe_trace_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, fbuffer.regs = regs; entry = fbuffer.entry =ring_buffer_event_data (fbuffer.event); entry->func = (unsigned long )tk->rp.kp.addr; - entry->ret_ip = (unsigned long )ri->ret_addr; + entry->ret_ip =get_kretprobe_retaddr (ri);store_trace_args (&entry[1 ], &tk->tp, regs,sizeof (*entry), dsize);trace_event_buffer_commit (&fbuffer);
@@ -1628,7 +1628,7 @@ kretprobe_perf_func(struct trace_kprobe *tk, struct kretprobe_instance *ri,return ; entry->func = (unsigned long )tk->rp.kp.addr; - entry->ret_ip = (unsigned long )ri->ret_addr; + entry->ret_ip =get_kretprobe_retaddr (ri);store_trace_args (&entry[1 ], &tk->tp, regs,sizeof (*entry), dsize);perf_trace_buf_submit (entry, size, rctx, call->event.type,1 , regs, head,NULL );
Masami Hiramatsu
maintainerPATCH 2/2 rethook: kprobes: x86: Replace kretprobe with rethook on x86
In reply to Masami HiramatsuReplaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.
--- arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 107 ---------------------------------- arch/x86/kernel/rethook.c | 121 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 135 insertions(+), 119 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
arch/x86/Kconfig +1 -0
@@ -224,6 +224,7 @@ config X86 select HAVE_KPROBES_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES + select HAVE_RETHOOK select HAVE_KVM select HAVE_LIVEPATCH if X86_64 select HAVE_MIXED_BREAKPOINTS_REGS
arch/x86/include/asm/unwind.h +11 -12
@@ -4,7 +4,7 @@#include <linux/sched.h> #include <linux/ftrace.h> -#include <linux/kprobes.h> +#include <linux/rethook.h> #include <asm/ptrace.h> #include <asm/stacktrace.h>
@@ -16,7 +16,7 @@ struct unwind_state {unsigned long stack_mask;struct task_struct *task;int graph_idx; -#ifdef CONFIG_KRETPROBES +#if defined(CONFIG_RETHOOK )struct llist_node *kr_cur;#endif bool error;
@@ -104,19 +104,18 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size,#endif static inline -unsigned long unwind_recover_kretprobe (struct unwind_state *state, -unsigned long addr,unsigned long *addr_p) +unsigned long unwind_recover_rethook (struct unwind_state *state, +unsigned long addr,unsigned long *addr_p) { -#ifdef CONFIG_KRETPROBES -return is_kretprobe_trampoline (addr) ? -kretprobe_find_ret_addr (state->task, addr_p, &state->kr_cur) : - addr; -#else -return addr; +#ifdef CONFIG_RETHOOK +if (is_rethook_trampoline (addr)) +return rethook_find_ret_addr (state->task, (unsigned long )addr_p, + &state->kr_cur);#endif +return addr; } -/* Recover the return address modified by kretprobe and ftrace_graph. */ +/* Recover the return address modified by rethook and ftrace_graph. */ static inline unsigned long unwind_recover_ret_addr (struct unwind_state *state,unsigned long addr,unsigned long *addr_p)
@@ -125,7 +124,7 @@ unsigned long unwind_recover_ret_addr(struct unwind_state *state, ret =ftrace_graph_ret_addr (state->task, &state->graph_idx, addr, addr_p); -return unwind_recover_kretprobe (state, ret, addr_p); +return unwind_recover_rethook (state, ret, addr_p); }/*
arch/x86/kernel/Makefile +1 -0
@@ -103,6 +103,7 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o obj-$(CONFIG_X86_TSC) += trace_clock.o obj-$(CONFIG_TRACING) += trace.o +obj-$(CONFIG_RETHOOK) += rethook.o obj-$(CONFIG_CRASH_CORE) += crash_core_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += machine_kexec_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += relocate_kernel_$(BITS).o crash.o
arch/x86/kernel/kprobes/common.h +1 -0
@@ -6,6 +6,7 @@#include <asm/asm.h> #include <asm/frame.h> +#include <asm/insn.h> #ifdef CONFIG_X86_64
arch/x86/kernel/kprobes/core.c +0 -107
119 lines not on this page · raw message
arch/x86/kernel/rethook.c new file +121 -0
121 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86
Quoted 5 lines · Masami Hiramatsu
Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.
Signed-off-by Masami Hiramatsu <[email protected]>
Quoted 9 lines
--- arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 107 ---------------------------------- arch/x86/kernel/rethook.c | 121 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 135 insertions(+), 119 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
I'm thinking you'll find it builds much better with this on...
arch/x86/kernel/unwind_orc.c +5 -5
@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state) }/* * There is a small chance to interrupt at the entry of -* __kretprobe_trampoline() where the ORC info doesn't exist. -* That point is right after the RET to __kretprobe_trampoline() +* arch_rethook_trampoline() where the ORC info doesn't exist. +* That point is right after the RET to arch_rethook_trampoline() * which was modified return address. -* At that point, the @addr_p of the unwind_recover_kretprobe() +* At that point, the @addr_p of the unwind_recover_rethook() * (this has to point the address of the stack entry storing * the modified return address) must be "SP - (a stack entry)" * because SP is incremented by the RET. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long ))); state->regs = (struct pt_regs *)sp; state->prev_regs =NULL ;
@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state) goto err; }/* See UNWIND_HINT_TYPE_REGS case comment. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long )));if (state->full_regs)
Peter Zijlstra
maintainerRe: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86
In reply to Masami Hiramatsuarch/x86/kernel/rethook.c line 39
Quoted 2 lines · Masami Hiramatsu
arch/x86/kernel/rethook.c new file +2
-0+/* Push a fake return address to tell the unwinder it's a kretprobe. */ +/* Push a fake return address to tell the unwinder it's a kretprobe. */
s/kretprobe/rethook/ went missing
Peter Zijlstra
maintainer[PATCH bpf-next 3/2] x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
In reply to Masami HiramatsuYou lost the regs->ss bit again..
Boot tested on tigerlake with IBT enabled -- passed the boot time kretprobe selftests.
---
Subject: x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
From: Peter Zijlstra <[email protected]>
Date: Fri Mar 25 10:25:56 CET 2022
Currently arch_rethook_trampoline() generates an almost complete pt_regs on-stack, everything except regs->ss that is, that currently points to the fake return address, which is not a valid segment descriptor.
Since interpretation of regs->[sb]p should be done in the context of regs->ss, and we have code actually doing that (see arch/x86/lib/insn-eval.c for instance), complete the job by also pushing ss.
This ensures that anybody who does do look at regs->ss doesn't mysteriously malfunction, avoiding much future pain.
--- arch/x86/kernel/rethook.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
arch/x86/kernel/rethook.c +13 -11
@@ -25,29 +25,31 @@ asm(/* Push a fake return address to tell the unwinder it's a kretprobe. */ " pushq $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 8', this will be fixed later. */ + " pushq $" __stringify(__KERNEL_DS) "\n" +/* Save the 'sp - 16', this will be fixed later. */ " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addq $8, %rsp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addq $16, %rsp\n" " popfq\n"#else /* Push a fake return address to tell the unwinder it's a kretprobe. */ " pushl $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 4', this will be fixed later. */ +" pushl %ss\n" +/* Save the 'sp - 8', this will be fixed later. */ " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addl $4, %esp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addl $8, %esp\n" " popfl\n"#endif ASM_RET
@@ -69,8 +71,8 @@ __used __visible void arch_rethook_tramp#endif regs->ip = (unsigned long )&arch_rethook_trampoline; regs->orig_ax = ~0UL ; - regs->sp +=sizeof (long ); - frame_pointer = ®s->sp +1 ; + regs->sp +=2 *sizeof (long ); + frame_pointer = (long *)(regs +1 );/* * The return address at 'frame_pointer' is recovered by the
@@ -80,10 +82,10 @@ __used __visible void arch_rethook_tramprethook_trampoline_handler (regs, (unsigned long )frame_pointer);/* -* Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline() +* Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline() * can do RET right after POPF. */ - regs->sp = regs->flags; + *(unsigned long *)®s->ss = regs->flags; }NOKPROBE_SYMBOL (arch_rethook_trampoline_callback);
@@ -101,7 +103,7 @@ STACK_FRAME_NON_STANDARD_FP(arch_rethookvoid arch_rethook_fixup_return (struct pt_regs *regs,unsigned long correct_ret_addr) { -unsigned long *frame_pointer = ®s->sp +1 ; +unsigned long *frame_pointer = (void *)(regs +1 );/* Replace fake return address with real one. */ *frame_pointer = correct_ret_addr;
Jiri Olsa
Tested-byRe: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86
In reply to Peter ZijlstraQuoted 16 lines · Peter Zijlstra
Quoted 5 lines · Masami Hiramatsu
Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.
Signed-off-by Masami Hiramatsu <[email protected]>Quoted 9 lines
--- arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 107 ---------------------------------- arch/x86/kernel/rethook.c | 121 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 135 insertions(+), 119 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
Quoted 1 line
I'm thinking you'll find it builds much better with this on...
I built it with Peter's fix and ran bpf selftests, looks good
thanks,
jirka
Quoted 34 lines
arch/x86/kernel/unwind_orc.c +5
-5@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state) }/* * There is a small chance to interrupt at the entry of -* __kretprobe_trampoline() where the ORC info doesn't exist. -* That point is right after the RET to __kretprobe_trampoline() +* arch_rethook_trampoline() where the ORC info doesn't exist. +* That point is right after the RET to arch_rethook_trampoline() * which was modified return address. -* At that point, the @addr_p of the unwind_recover_kretprobe() +* At that point, the @addr_p of the unwind_recover_rethook() * (this has to point the address of the stack entry storing * the modified return address) must be "SP - (a stack entry)" * because SP is incremented by the RET. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long ))); state->regs = (struct pt_regs *)sp; state->prev_regs =NULL ;@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state) goto err; }/* See UNWIND_HINT_TYPE_REGS case comment. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long )));if (state->full_regs)
Masami Hiramatsu
maintainerRe: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86
In reply to Peter ZijlstraQuoted 16 lines · On Fri, 25 Mar 2022 11:09:40 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu
Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.
Signed-off-by Masami Hiramatsu <[email protected]>Quoted 9 lines
--- arch/x86/Kconfig | 1 arch/x86/include/asm/unwind.h | 23 +++---- arch/x86/kernel/Makefile | 1 arch/x86/kernel/kprobes/common.h | 1 arch/x86/kernel/kprobes/core.c | 107 ---------------------------------- arch/x86/kernel/rethook.c | 121 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 135 insertions(+), 119 deletions(-) create mode 100644 arch/x86/kernel/rethook.c
Quoted 1 line
I'm thinking you'll find it builds much better with this on...
Oops, Thanks. I've tested it with framepointer based unwinder...
Quoted 34 lines
arch/x86/kernel/unwind_orc.c +5
-5@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state) }/* * There is a small chance to interrupt at the entry of -* __kretprobe_trampoline() where the ORC info doesn't exist. -* That point is right after the RET to __kretprobe_trampoline() +* arch_rethook_trampoline() where the ORC info doesn't exist. +* That point is right after the RET to arch_rethook_trampoline() * which was modified return address. -* At that point, the @addr_p of the unwind_recover_kretprobe() +* At that point, the @addr_p of the unwind_recover_rethook() * (this has to point the address of the stack entry storing * the modified return address) must be "SP - (a stack entry)" * because SP is incremented by the RET. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long ))); state->regs = (struct pt_regs *)sp; state->prev_regs =NULL ;@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state) goto err; }/* See UNWIND_HINT_TYPE_REGS case comment. */ - state->ip =unwind_recover_kretprobe (state, state->ip, + state->ip =unwind_recover_rethook (state, state->ip, (unsigned long *)(state->sp -sizeof (long )));if (state->full_regs)
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH bpf-next 3/2] x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
In reply to Peter ZijlstraQuoted 2 lines · On Fri, 25 Mar 2022 12:40:12 +0100 Peter Zijlstra <[email protected]> wrote:
You lost the
regs->ssbit again..
Yeah, I planed to split it in another series with optprobe update because the optprobe also skips regs->ss. Anyway...
Quoted 15 lines
Boot tested on tigerlake with IBT enabled -- passed the boot time kretprobe selftests.
---
Subject: x86,rethook: Fix
arch_rethook_trampoline()to generate a completept_regs
From: Peter Zijlstra <[email protected]>
Date: Fri Mar 25 10:25:56 CET 2022Currently
arch_rethook_trampoline()generates an almost completept_regson-stack, everything exceptregs->ssthat is, that currently points to the fake return address, which is not a valid segment descriptor.
Quoted 4 lines
Since interpretation of regs->[sb]p should be done in the context of
regs->ss, and we have code actually doing that (seearch/x86/lib/insn-eval.cfor instance), complete the job by also pushing ss.
This looks good to me.
Thank you!
Quoted 82 lines
This ensures that anybody who does do look at
regs->ssdoesn't mysteriously malfunction, avoiding much future pain.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/kernel/rethook.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)arch/x86/kernel/rethook.c +13
-11@@ -25,29 +25,31 @@ asm(/* Push a fake return address to tell the unwinder it's a kretprobe. */ " pushq $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 8', this will be fixed later. */ + " pushq $" __stringify(__KERNEL_DS) "\n" +/* Save the 'sp - 16', this will be fixed later. */ " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addq $8, %rsp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addq $16, %rsp\n" " popfq\n"#else /* Push a fake return address to tell the unwinder it's a kretprobe. */ " pushl $arch_rethook_trampoline\n" UNWIND_HINT_FUNC -/* Save the 'sp - 4', this will be fixed later. */ +" pushl %ss\n" +/* Save the 'sp - 8', this will be fixed later. */ " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call arch_rethook_trampoline_callback\n" RESTORE_REGS_STRING -/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */ -" addl $4, %esp\n" +/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */ +" addl $8, %esp\n" " popfl\n"#endif ASM_RET @@ -69,8 +71,8 @@ __used __visible void arch_rethook_tramp#endif regs->ip = (unsigned long )&arch_rethook_trampoline; regs->orig_ax = ~0UL ; - regs->sp +=sizeof (long ); - frame_pointer = ®s->sp +1 ; + regs->sp +=2 *sizeof (long ); + frame_pointer = (long *)(regs +1 );/* * The return address at 'frame_pointer' is recovered by the @@ -80,10 +82,10 @@ __used __visible void arch_rethook_tramprethook_trampoline_handler (regs, (unsigned long )frame_pointer);/* -* Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline() +* Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline() * can do RET right after POPF. */ - regs->sp = regs->flags; + *(unsigned long *)®s->ss = regs->flags; }NOKPROBE_SYMBOL (arch_rethook_trampoline_callback);@@ -101,7 +103,7 @@ STACK_FRAME_NON_STANDARD_FP(arch_rethookvoid arch_rethook_fixup_return (struct pt_regs *regs,unsigned long correct_ret_addr) { -unsigned long *frame_pointer = ®s->sp +1 ; +unsigned long *frame_pointer = (void *)(regs +1 );/* Replace fake return address with real one. */ *frame_pointer = correct_ret_addr;
Signature · 1 line
Masami Hiramatsu <[email protected]>