Skip to contentSkip to conversation

kprobes: rethook: x86: Replace kretprobe trampoline with rethook

PATCH Superseded Masami Hiramatsu 3 participants 9 messages Mar 25, 2022

Subsystems (6)
x86 Kernel janitors BPF Tracing +2 more subsystems

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

cover kprobes: rethook: x86: Replace kretprobe trampoline with rethook

Hi,

Here are the patch set for generic kretprobe and kretprobe on x86 for replacing the kretprobe trampoline with rethook. For the other archs, I will port rethook to those after this has been merged.
This is previously called as "rethook: x86: Add rethook x86 implementation" The previous thread is here[1].

[1] https://lore.kernel.org/all/164800288611.1716332.7053663723617614668.stgit@devnote2/T/#u

Background:

This rethook came from Jiri's request of multiple kprobe for bpf[1]. He tried to solve an issue that starting bpf with multiple kprobe will take a long time because bpf-kprobe will wait for RCU grace period for sync rcu events.

Jiri wanted to attach a single bpf handler to multiple kprobes and he tried to introduce multiple-probe interface to kprobe. So I asked him to use ftrace and kretprobe-like hook if it is only for the function entry and exit, instead of adding ad-hoc interface to kprobes.
For this purpose, I introduced the fprobe (kprobe like interface for ftrace) with the rethook (this is a generic return hook feature for fprobe exit handler)[2].

[1] https://lore.kernel.org/all/[email protected]/T/#u
[2] https://lore.kernel.org/all/164191321766.806991.7930388561276940676.stgit@devnote2/T/#u

The rethook is basically same as the kretprobe trampoline. I just made it decoupled from kprobes. Eventually, the all arch dependent kretprobe trampolines will be replaced with the rethook trampoline instead of cloning and set HAVE_RETHOOK=y.
When I port the rethook for all arch which supports kretprobe, the legacy kretprobe specific code (which is for CONFIG_KRETPROBE_ON_RETHOOK=n) will be removed eventually.

BTW, this patch can be applied to next-20220324, not the bpf-next tree directly, because this depends on ANNOTATE_NOENDBR macro. However, since the fprobe is merged in the bpf-next, I marked this for bpf-next. So until merging the both of fprobes and ENDBR series, to compile this you need below 2 lines in arch/x86/kernel/rethook.c.

#ifndef ANNOTATE_NOENDBR
#define ANNOTATE_NOENDBR

But after those are merged, these lines will be unneeded. How should I handle this issue? (Just remove ANNOTATE_NOENDBR line in bpf-next?)

Thank you,

---

Masami Hiramatsu (2):
kprobes: Use rethook for kretprobe if possible
rethook: kprobes: x86: Replace kretprobe with rethook on x86

 arch/Kconfig                     |    7 ++
 arch/x86/Kconfig                 |    1 
 arch/x86/include/asm/unwind.h    |   23 +++----
 arch/x86/kernel/Makefile         |    1 
 arch/x86/kernel/kprobes/common.h |    1 
 arch/x86/kernel/kprobes/core.c   |  107 ---------------------------------
 arch/x86/kernel/rethook.c        |  121 +++++++++++++++++++++++++++++++++++++
 include/linux/kprobes.h          |   51 +++++++++++++++-
 kernel/kprobes.c                 |  124 ++++++++++++++++++++++++++++++++------
 kernel/trace/trace_kprobe.c      |    4 +
 10 files changed, 296 insertions(+), 144 deletions(-)
 create mode 100644 arch/x86/kernel/rethook.c

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

PATCH 1/2 kprobes: Use rethook for kretprobe if possible

Use rethook for kretprobe function return hooking if the arch sets CONFIG_HAVE_RETHOOK=y. In this case, CONFIG_KRETPROBE_ON_RETHOOK is set to 'y' automatically, and the kretprobe internal data fields switches to use rethook. If not, it continues to use kretprobe specific function return hooks.

---
 arch/Kconfig                |    7 ++
 include/linux/kprobes.h     |   51 +++++++++++++++++-
 kernel/kprobes.c            |  124 ++++++++++++++++++++++++++++++++++++-------
 kernel/trace/trace_kprobe.c |    4 +
 4 files changed, 161 insertions(+), 25 deletions(-)
arch/Kconfig +6 -1
index e12a4268c01d..9570854c4683 100644
@@ -164,7 +164,12 @@ config ARCH_USE_BUILTIN_BSWAP
 
 config KRETPROBES
 	def_bool y
-	depends on KPROBES && HAVE_KRETPROBES
+	depends on KPROBES && (HAVE_KRETPROBES || HAVE_RETHOOK)
+
+config KRETPROBE_ON_RETHOOK
+	def_bool y
+	depends on HAVE_RETHOOK
+	select RETHOOK
 
 config USER_RETURN_NOTIFIER
 	bool
include/linux/kprobes.h +49 -2

109 lines not on this page · raw message

kernel/kprobes.c +104 -20

180 lines not on this page · raw message

kernel/trace/trace_kprobe.c +2 -2
index b62fd785b599..47cebef78532 100644
@@ -1433,7 +1433,7 @@ __kretprobe_trace_func(struct trace_kprobe *tk, struct kretprobe_instance *ri,
 	fbuffer.regs = regs;
 	entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event);
 	entry->func = (unsigned long)tk->rp.kp.addr;
-	entry->ret_ip = (unsigned long)ri->ret_addr;
+	entry->ret_ip = get_kretprobe_retaddr(ri);
 	store_trace_args(&entry[1], &tk->tp, regs, sizeof(*entry), dsize);
 
 	trace_event_buffer_commit(&fbuffer);
@@ -1628,7 +1628,7 @@ kretprobe_perf_func(struct trace_kprobe *tk, struct kretprobe_instance *ri,
 		return;
 
 	entry->func = (unsigned long)tk->rp.kp.addr;
-	entry->ret_ip = (unsigned long)ri->ret_addr;
+	entry->ret_ip = get_kretprobe_retaddr(ri);
 	store_trace_args(&entry[1], &tk->tp, regs, sizeof(*entry), dsize);
 	perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs,
 			      head, NULL);
author maintainer
#2 lore raw

PATCH 2/2 rethook: kprobes: x86: Replace kretprobe with rethook on x86

In reply to Masami Hiramatsu

Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.

---
 arch/x86/Kconfig                 |    1 
 arch/x86/include/asm/unwind.h    |   23 +++----
 arch/x86/kernel/Makefile         |    1 
 arch/x86/kernel/kprobes/common.h |    1 
 arch/x86/kernel/kprobes/core.c   |  107 ----------------------------------
 arch/x86/kernel/rethook.c        |  121 ++++++++++++++++++++++++++++++++++++++
 6 files changed, 135 insertions(+), 119 deletions(-)
 create mode 100644 arch/x86/kernel/rethook.c
arch/x86/Kconfig +1 -0
index db3bedcbf6be..e85b52a6f612 100644
@@ -224,6 +224,7 @@ config X86
 	select HAVE_KPROBES_ON_FTRACE
 	select HAVE_FUNCTION_ERROR_INJECTION
 	select HAVE_KRETPROBES
+	select HAVE_RETHOOK
 	select HAVE_KVM
 	select HAVE_LIVEPATCH			if X86_64
 	select HAVE_MIXED_BREAKPOINTS_REGS
arch/x86/include/asm/unwind.h +11 -12
index 2a1f8734416d..7cede4dc21f0 100644
@@ -4,7 +4,7 @@
 
 #include <linux/sched.h>
 #include <linux/ftrace.h>
-#include <linux/kprobes.h>
+#include <linux/rethook.h>
 #include <asm/ptrace.h>
 #include <asm/stacktrace.h>
 
@@ -16,7 +16,7 @@ struct unwind_state {
 	unsigned long stack_mask;
 	struct task_struct *task;
 	int graph_idx;
-#ifdef CONFIG_KRETPROBES
+#if defined(CONFIG_RETHOOK)
 	struct llist_node *kr_cur;
 #endif
 	bool error;
@@ -104,19 +104,18 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size,
 #endif
 
 static inline
-unsigned long unwind_recover_kretprobe(struct unwind_state *state,
-				       unsigned long addr, unsigned long *addr_p)
+unsigned long unwind_recover_rethook(struct unwind_state *state,
+				     unsigned long addr, unsigned long *addr_p)
 {
-#ifdef CONFIG_KRETPROBES
-	return is_kretprobe_trampoline(addr) ?
-		kretprobe_find_ret_addr(state->task, addr_p, &state->kr_cur) :
-		addr;
-#else
-	return addr;
+#ifdef CONFIG_RETHOOK
+	if (is_rethook_trampoline(addr))
+		return rethook_find_ret_addr(state->task, (unsigned long)addr_p,
+					     &state->kr_cur);
 #endif
+	return addr;
 }
 
-/* Recover the return address modified by kretprobe and ftrace_graph. */
+/* Recover the return address modified by rethook and ftrace_graph. */
 static inline
 unsigned long unwind_recover_ret_addr(struct unwind_state *state,
 				     unsigned long addr, unsigned long *addr_p)
@@ -125,7 +124,7 @@ unsigned long unwind_recover_ret_addr(struct unwind_state *state,
 
 	ret = ftrace_graph_ret_addr(state->task, &state->graph_idx,
 				    addr, addr_p);
-	return unwind_recover_kretprobe(state, ret, addr_p);
+	return unwind_recover_rethook(state, ret, addr_p);
 }
 
 /*
arch/x86/kernel/Makefile +1 -0
index 6462e3dd98f4..c41ef42adbe8 100644
@@ -103,6 +103,7 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o
 obj-$(CONFIG_FTRACE_SYSCALLS)	+= ftrace.o
 obj-$(CONFIG_X86_TSC)		+= trace_clock.o
 obj-$(CONFIG_TRACING)		+= trace.o
+obj-$(CONFIG_RETHOOK)		+= rethook.o
 obj-$(CONFIG_CRASH_CORE)	+= crash_core_$(BITS).o
 obj-$(CONFIG_KEXEC_CORE)	+= machine_kexec_$(BITS).o
 obj-$(CONFIG_KEXEC_CORE)	+= relocate_kernel_$(BITS).o crash.o
arch/x86/kernel/kprobes/common.h +1 -0
index 7d3a2e2daf01..c993521d4933 100644
@@ -6,6 +6,7 @@
 
 #include <asm/asm.h>
 #include <asm/frame.h>
+#include <asm/insn.h>
 
 #ifdef CONFIG_X86_64
 
arch/x86/kernel/kprobes/core.c +0 -107

119 lines not on this page · raw message

arch/x86/kernel/rethook.c new file +121 -0

121 lines not on this page · raw message

maintainer
#3 lore raw

Re: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86

Quoted 5 lines · Masami Hiramatsu

Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.

Quoted 9 lines
---
 arch/x86/Kconfig                 |    1 
 arch/x86/include/asm/unwind.h    |   23 +++----
 arch/x86/kernel/Makefile         |    1 
 arch/x86/kernel/kprobes/common.h |    1 
 arch/x86/kernel/kprobes/core.c   |  107 ----------------------------------
 arch/x86/kernel/rethook.c        |  121 ++++++++++++++++++++++++++++++++++++++
 6 files changed, 135 insertions(+), 119 deletions(-)
 create mode 100644 arch/x86/kernel/rethook.c

I'm thinking you'll find it builds much better with this on...

arch/x86/kernel/unwind_orc.c +5 -5
index 2de3c8c5eba9..794fdef2501a 100644
@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state)
 		}
 		/*
 		 * There is a small chance to interrupt at the entry of
-		 * __kretprobe_trampoline() where the ORC info doesn't exist.
-		 * That point is right after the RET to __kretprobe_trampoline()
+		 * arch_rethook_trampoline() where the ORC info doesn't exist.
+		 * That point is right after the RET to arch_rethook_trampoline()
 		 * which was modified return address.
-		 * At that point, the @addr_p of the unwind_recover_kretprobe()
+		 * At that point, the @addr_p of the unwind_recover_rethook()
 		 * (this has to point the address of the stack entry storing
 		 * the modified return address) must be "SP - (a stack entry)"
 		 * because SP is incremented by the RET.
 		 */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 		state->regs = (struct pt_regs *)sp;
 		state->prev_regs = NULL;
@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state)
 			goto err;
 		}
 		/* See UNWIND_HINT_TYPE_REGS case comment. */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 
 		if (state->full_regs)
Tested-by
#4 lore raw

Re: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86

Quoted 16 lines · Peter Zijlstra
Quoted 5 lines · Masami Hiramatsu

Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.

Quoted 9 lines
---
 arch/x86/Kconfig                 |    1 
 arch/x86/include/asm/unwind.h    |   23 +++----
 arch/x86/kernel/Makefile         |    1 
 arch/x86/kernel/kprobes/common.h |    1 
 arch/x86/kernel/kprobes/core.c   |  107 ----------------------------------
 arch/x86/kernel/rethook.c        |  121 ++++++++++++++++++++++++++++++++++++++
 6 files changed, 135 insertions(+), 119 deletions(-)
 create mode 100644 arch/x86/kernel/rethook.c
Quoted 1 line

I'm thinking you'll find it builds much better with this on...

I built it with Peter's fix and ran bpf selftests, looks good

thanks,
jirka

Quoted 34 lines
arch/x86/kernel/unwind_orc.c +5 -5
index 2de3c8c5eba9..794fdef2501a 100644
@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state)
 		}
 		/*
 		 * There is a small chance to interrupt at the entry of
-		 * __kretprobe_trampoline() where the ORC info doesn't exist.
-		 * That point is right after the RET to __kretprobe_trampoline()
+		 * arch_rethook_trampoline() where the ORC info doesn't exist.
+		 * That point is right after the RET to arch_rethook_trampoline()
 		 * which was modified return address.
-		 * At that point, the @addr_p of the unwind_recover_kretprobe()
+		 * At that point, the @addr_p of the unwind_recover_rethook()
 		 * (this has to point the address of the stack entry storing
 		 * the modified return address) must be "SP - (a stack entry)"
 		 * because SP is incremented by the RET.
 		 */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 		state->regs = (struct pt_regs *)sp;
 		state->prev_regs = NULL;
@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state)
 			goto err;
 		}
 		/* See UNWIND_HINT_TYPE_REGS case comment. */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 
 		if (state->full_regs)
author maintainer
#5 lore raw

Re: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86

In reply to Peter Zijlstra
Quoted 16 lines · On Fri, 25 Mar 2022 11:09:40 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu

Replaces the kretprobe code with rethook on x86. With this patch, kretprobe on x86 uses the rethook instead of kretprobe specific trampoline code.

Quoted 9 lines
---
 arch/x86/Kconfig                 |    1 
 arch/x86/include/asm/unwind.h    |   23 +++----
 arch/x86/kernel/Makefile         |    1 
 arch/x86/kernel/kprobes/common.h |    1 
 arch/x86/kernel/kprobes/core.c   |  107 ----------------------------------
 arch/x86/kernel/rethook.c        |  121 ++++++++++++++++++++++++++++++++++++++
 6 files changed, 135 insertions(+), 119 deletions(-)
 create mode 100644 arch/x86/kernel/rethook.c
Quoted 1 line

I'm thinking you'll find it builds much better with this on...

Oops, Thanks. I've tested it with framepointer based unwinder...

Quoted 34 lines
arch/x86/kernel/unwind_orc.c +5 -5
index 2de3c8c5eba9..794fdef2501a 100644
@@ -550,15 +550,15 @@ bool unwind_next_frame(struct unwind_state *state)
 		}
 		/*
 		 * There is a small chance to interrupt at the entry of
-		 * __kretprobe_trampoline() where the ORC info doesn't exist.
-		 * That point is right after the RET to __kretprobe_trampoline()
+		 * arch_rethook_trampoline() where the ORC info doesn't exist.
+		 * That point is right after the RET to arch_rethook_trampoline()
 		 * which was modified return address.
-		 * At that point, the @addr_p of the unwind_recover_kretprobe()
+		 * At that point, the @addr_p of the unwind_recover_rethook()
 		 * (this has to point the address of the stack entry storing
 		 * the modified return address) must be "SP - (a stack entry)"
 		 * because SP is incremented by the RET.
 		 */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 		state->regs = (struct pt_regs *)sp;
 		state->prev_regs = NULL;
@@ -573,7 +573,7 @@ bool unwind_next_frame(struct unwind_state *state)
 			goto err;
 		}
 		/* See UNWIND_HINT_TYPE_REGS case comment. */
-		state->ip = unwind_recover_kretprobe(state, state->ip,
+		state->ip = unwind_recover_rethook(state, state->ip,
 				(unsigned long *)(state->sp - sizeof(long)));
 
 		if (state->full_regs)
Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#6 lore raw

Re: [PATCH bpf-next 2/2] rethook: kprobes: x86: Replace kretprobe with rethook on x86

In reply to Masami Hiramatsu

arch/x86/kernel/rethook.c line 39

Quoted 2 lines · Masami Hiramatsu
arch/x86/kernel/rethook.c new file +2 -0
+	/* Push a fake return address to tell the unwinder it's a kretprobe. */
+	/* Push a fake return address to tell the unwinder it's a kretprobe. */

s/kretprobe/rethook/ went missing

maintainer
#7 lore raw

[PATCH bpf-next 3/2] x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs

In reply to Masami Hiramatsu

You lost the regs->ss bit again..

Boot tested on tigerlake with IBT enabled -- passed the boot time kretprobe selftests.

---

Subject: x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
From: Peter Zijlstra <[email protected]>
Date: Fri Mar 25 10:25:56 CET 2022

Currently arch_rethook_trampoline() generates an almost complete pt_regs on-stack, everything except regs->ss that is, that currently points to the fake return address, which is not a valid segment descriptor.

Since interpretation of regs->[sb]p should be done in the context of regs->ss, and we have code actually doing that (see arch/x86/lib/insn-eval.c for instance), complete the job by also pushing ss.

This ensures that anybody who does do look at regs->ss doesn't mysteriously malfunction, avoiding much future pain.

---
 arch/x86/kernel/rethook.c |   24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
arch/x86/kernel/rethook.c +13 -11
@@ -25,29 +25,31 @@ asm(
 	/* Push a fake return address to tell the unwinder it's a kretprobe. */
 	"	pushq $arch_rethook_trampoline\n"
 	UNWIND_HINT_FUNC
-	/* Save the 'sp - 8', this will be fixed later. */
+	"       pushq $" __stringify(__KERNEL_DS) "\n"
+	/* Save the 'sp - 16', this will be fixed later. */
 	"	pushq %rsp\n"
 	"	pushfq\n"
 	SAVE_REGS_STRING
 	"	movq %rsp, %rdi\n"
 	"	call arch_rethook_trampoline_callback\n"
 	RESTORE_REGS_STRING
-	/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */
-	"	addq $8, %rsp\n"
+	/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */
+	"	addq $16, %rsp\n"
 	"	popfq\n"
 #else
 	/* Push a fake return address to tell the unwinder it's a kretprobe. */
 	"	pushl $arch_rethook_trampoline\n"
 	UNWIND_HINT_FUNC
-	/* Save the 'sp - 4', this will be fixed later. */
+	"	pushl %ss\n"
+	/* Save the 'sp - 8', this will be fixed later. */
 	"	pushl %esp\n"
 	"	pushfl\n"
 	SAVE_REGS_STRING
 	"	movl %esp, %eax\n"
 	"	call arch_rethook_trampoline_callback\n"
 	RESTORE_REGS_STRING
-	/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */
-	"	addl $4, %esp\n"
+	/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */
+	"	addl $8, %esp\n"
 	"	popfl\n"
 #endif
 	ASM_RET
@@ -69,8 +71,8 @@ __used __visible void arch_rethook_tramp
 #endif
 	regs->ip = (unsigned long)&arch_rethook_trampoline;
 	regs->orig_ax = ~0UL;
-	regs->sp += sizeof(long);
-	frame_pointer = &regs->sp + 1;
+	regs->sp += 2*sizeof(long);
+	frame_pointer = (long *)(regs + 1);
 
 	/*
 	 * The return address at 'frame_pointer' is recovered by the
@@ -80,10 +82,10 @@ __used __visible void arch_rethook_tramp
 	rethook_trampoline_handler(regs, (unsigned long)frame_pointer);
 
 	/*
-	 * Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline()
+	 * Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline()
 	 * can do RET right after POPF.
 	 */
-	regs->sp = regs->flags;
+	*(unsigned long *)&regs->ss = regs->flags;
 }
 NOKPROBE_SYMBOL(arch_rethook_trampoline_callback);
 
@@ -101,7 +103,7 @@ STACK_FRAME_NON_STANDARD_FP(arch_rethook
 void arch_rethook_fixup_return(struct pt_regs *regs,
 			       unsigned long correct_ret_addr)
 {
-	unsigned long *frame_pointer = &regs->sp + 1;
+	unsigned long *frame_pointer = (void *)(regs + 1);
 
 	/* Replace fake return address with real one. */
 	*frame_pointer = correct_ret_addr;
author maintainer Acked-by
#8 lore raw

Re: [PATCH bpf-next 3/2] x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs

Quoted 2 lines · On Fri, 25 Mar 2022 12:40:12 +0100 Peter Zijlstra <[email protected]> wrote:

You lost the regs->ss bit again..

Yeah, I planed to split it in another series with optprobe update because the optprobe also skips regs->ss. Anyway...

Quoted 15 lines

Boot tested on tigerlake with IBT enabled -- passed the boot time kretprobe selftests.

---

Subject: x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs
From: Peter Zijlstra <[email protected]>
Date: Fri Mar 25 10:25:56 CET 2022

Currently arch_rethook_trampoline() generates an almost complete pt_regs on-stack, everything except regs->ss that is, that currently points to the fake return address, which is not a valid segment descriptor.

Quoted 4 lines

Since interpretation of regs->[sb]p should be done in the context of regs->ss, and we have code actually doing that (see arch/x86/lib/insn-eval.c for instance), complete the job by also pushing ss.

This looks good to me.

Thank you!

Quoted 82 lines

This ensures that anybody who does do look at regs->ss doesn't mysteriously malfunction, avoiding much future pain.

---
 arch/x86/kernel/rethook.c |   24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
arch/x86/kernel/rethook.c +13 -11
@@ -25,29 +25,31 @@ asm(
 	/* Push a fake return address to tell the unwinder it's a kretprobe. */
 	"	pushq $arch_rethook_trampoline\n"
 	UNWIND_HINT_FUNC
-	/* Save the 'sp - 8', this will be fixed later. */
+	"       pushq $" __stringify(__KERNEL_DS) "\n"
+	/* Save the 'sp - 16', this will be fixed later. */
 	"	pushq %rsp\n"
 	"	pushfq\n"
 	SAVE_REGS_STRING
 	"	movq %rsp, %rdi\n"
 	"	call arch_rethook_trampoline_callback\n"
 	RESTORE_REGS_STRING
-	/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */
-	"	addq $8, %rsp\n"
+	/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */
+	"	addq $16, %rsp\n"
 	"	popfq\n"
 #else
 	/* Push a fake return address to tell the unwinder it's a kretprobe. */
 	"	pushl $arch_rethook_trampoline\n"
 	UNWIND_HINT_FUNC
-	/* Save the 'sp - 4', this will be fixed later. */
+	"	pushl %ss\n"
+	/* Save the 'sp - 8', this will be fixed later. */
 	"	pushl %esp\n"
 	"	pushfl\n"
 	SAVE_REGS_STRING
 	"	movl %esp, %eax\n"
 	"	call arch_rethook_trampoline_callback\n"
 	RESTORE_REGS_STRING
-	/* In the callback function, 'regs->flags' is copied to 'regs->sp'. */
-	"	addl $4, %esp\n"
+	/* In the callback function, 'regs->flags' is copied to 'regs->ss'. */
+	"	addl $8, %esp\n"
 	"	popfl\n"
 #endif
 	ASM_RET
@@ -69,8 +71,8 @@ __used __visible void arch_rethook_tramp
 #endif
 	regs->ip = (unsigned long)&arch_rethook_trampoline;
 	regs->orig_ax = ~0UL;
-	regs->sp += sizeof(long);
-	frame_pointer = &regs->sp + 1;
+	regs->sp += 2*sizeof(long);
+	frame_pointer = (long *)(regs + 1);
 
 	/*
 	 * The return address at 'frame_pointer' is recovered by the
@@ -80,10 +82,10 @@ __used __visible void arch_rethook_tramp
 	rethook_trampoline_handler(regs, (unsigned long)frame_pointer);
 
 	/*
-	 * Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline()
+	 * Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline()
 	 * can do RET right after POPF.
 	 */
-	regs->sp = regs->flags;
+	*(unsigned long *)&regs->ss = regs->flags;
 }
 NOKPROBE_SYMBOL(arch_rethook_trampoline_callback);
 
@@ -101,7 +103,7 @@ STACK_FRAME_NON_STANDARD_FP(arch_rethook
 void arch_rethook_fixup_return(struct pt_regs *regs,
 			       unsigned long correct_ret_addr)
 {
-	unsigned long *frame_pointer = &regs->sp + 1;
+	unsigned long *frame_pointer = (void *)(regs + 1);
 
 	/* Replace fake return address with real one. */
 	*frame_pointer = correct_ret_addr;
Signature · 1 line
Masami Hiramatsu <[email protected]>