Skip to contentSkip to conversation

x86/ftrace: Make created trampolines not call direct code

PATCH In review Steven Rostedt 2 participants 6 messages Apr 22, 2020

Subsystems (3)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover x86/ftrace: Make created trampolines not call direct code

Peter,

These three patches are built on top of your patch series.

This is what I was thinking of having. This isn't fully tested (as your patch series is still running on my test suite).

-- Steve

Steven Rostedt (VMware) (3):
x86/ftrace: Make non direct case the default in ftrace_regs_caller x86/ftrace: Only have the builtin ftrace_regs_caller call direct hooks x86/ftrace: Do not jump to direct code in created trampolines

----

 arch/x86/kernel/ftrace.c    | 14 +++++++++++---
 arch/x86/kernel/ftrace_64.S | 29 ++++++++++++++---------------
 kernel/trace/ftrace.c       |  8 ++++++++
 3 files changed, 33 insertions(+), 18 deletions(-)
author maintainer
#1 lore raw

PATCH 1/3 x86/ftrace: Make non direct case the default in ftrace_regs_caller

From: "Steven Rostedt (VMware)" <[email protected]>

If a direct function is hooked along with one of the ftrace registered functions, then the ftrace_regs_caller is attached to the function that shares the direct hook as well as the ftrace hook. The ftrace_regs_caller will call ftrace_ops_list_func() that iterates through all the registered ftrace callbacks, and if there's a direct callback attached to that function, the direct ftrace_ops callback is called to notify that ftrace_regs_caller to return to the direct caller instead of going back to the function that called it.

But this is a very uncommon case. Currently, the code has it as the default case. Modify ftrace_regs_caller to make the default case (the non jump) to just return normally, and have the jump to the handling of the direct caller.

---
 arch/x86/kernel/ftrace_64.S | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
arch/x86/kernel/ftrace_64.S +15 -16
index 9738ed23964e..c93ee7195af0 100644
@@ -241,22 +241,9 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
 	 */
 	movq ORIG_RAX(%rsp), %rax
 	testq	%rax, %rax
-	jz	1f
+	jnz	1f
 
-	/* Swap the flags with orig_rax */
-	movq MCOUNT_REG_SIZE(%rsp), %rdi
-	movq %rdi, MCOUNT_REG_SIZE-8(%rsp)
-	movq %rax, MCOUNT_REG_SIZE(%rsp)
-
-	restore_mcount_regs 8
-	/* Restore flags */
-	popfq
-
-SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL);
-	UNWIND_HINT_RET_OFFSET
-	jmp	ftrace_epilogue
-
-1:	restore_mcount_regs
+	restore_mcount_regs
 	/* Restore flags */
 	popfq
 
@@ -266,9 +253,21 @@ SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL);
 	 * The trampoline will add the code to jump
 	 * to the return.
 	 */
-SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
+SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL)
 	jmp ftrace_epilogue
 
+	/* Swap the flags with orig_rax */
+1:	movq MCOUNT_REG_SIZE(%rsp), %rdi
+	movq %rdi, MCOUNT_REG_SIZE-8(%rsp)
+	movq %rax, MCOUNT_REG_SIZE(%rsp)
+
+	restore_mcount_regs 8
+	/* Restore flags */
+	popfq
+SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
+	UNWIND_HINT_RET_OFFSET
+	jmp	ftrace_epilogue
+
 SYM_FUNC_END(ftrace_regs_caller)
 
 
Signature · 1 line
2.26.1
author maintainer
#2 lore raw

PATCH 2/3 x86/ftrace: Only have the builtin ftrace_regs_caller call direct hooks

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If a direct hook is attached to a function that ftrace also has a function attached to it, then it is required that the ftrace_ops_list_func() is used to iterate over the registered ftrace callbacks. This will also include the direct ftrace_ops helper, that tells ftrace_regs_caller where to return to (the direct callback and not the function that called it).

As this direct helper is only to handle the case of ftrace callbacks attached to the same function as the direct callback, the ftrace callback allocated trampolines (used to only call them), should never be used to return back to a direct callback.

Only copy the portion of the ftrace_regs_caller that will return back to what called it, and not the portion that returns back to the direct caller.

The direct ftrace_ops must then pick the ftrace_regs_caller builtin function as its own trampoline to ensure that it will never have one allocated for it (which would not include the handling of direct callbacks).

---
 arch/x86/kernel/ftrace.c    | 7 -------
 arch/x86/kernel/ftrace_64.S | 3 +--
 kernel/trace/ftrace.c       | 8 ++++++++
 3 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/kernel/ftrace.c +0 -7
index 867c126ddabe..2e11250d5647 100644
@@ -367,13 +367,6 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	if (WARN_ON(ret < 0))
 		goto fail;
 
-	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
-		ip = trampoline + (ftrace_regs_caller_ret - ftrace_regs_caller);
-		ret = probe_kernel_read(ip, (void *)retq, RET_SIZE);
-		if (WARN_ON(ret < 0))
-			goto fail;
-	}
-
 	/*
 	 * The address of the ftrace_ops that is used for this trampoline
 	 * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +1 -2
index c93ee7195af0..0882758d165a 100644
@@ -253,7 +253,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
 	 * The trampoline will add the code to jump
 	 * to the return.
 	 */
-SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL)
+SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
 	jmp ftrace_epilogue
 
 	/* Swap the flags with orig_rax */
@@ -264,7 +264,6 @@ SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL)
 	restore_mcount_regs 8
 	/* Restore flags */
 	popfq
-SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
 	UNWIND_HINT_RET_OFFSET
 	jmp	ftrace_epilogue
 
kernel/trace/ftrace.c +8 -0
index 041694a1eb74..d4bf39e2e76a 100644
@@ -2399,6 +2399,14 @@ struct ftrace_ops direct_ops = {
 	.flags		= FTRACE_OPS_FL_IPMODIFY | FTRACE_OPS_FL_RECURSION_SAFE
 			  | FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_REGS
 			  | FTRACE_OPS_FL_PERMANENT,
+	/*
+	 * By declaring the main trampoline as this trampoline
+	 * it will never have one allocated for it. Allocated
+	 * trampolines should not call direct functions.
+	 * The direct_ops should only be called by the builtin
+	 * ftrace_regs_caller trampoline.
+	 */
+	.trampoline	= FTRACE_REGS_ADDR,
 };
 #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
 
Signature · 1 line
2.26.1
author maintainer
#3 lore raw

PATCH 3/3 x86/ftrace: Do not jump to direct code in created trampolines

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

When creating a trampoline based on the ftrace_regs_caller code, nop out the jnz test that would jmup to the code that would return to a direct caller (stored in the ORIG_RAX field) and not back to the function that called it.

---
 arch/x86/kernel/ftrace.c    | 15 +++++++++++++++
 arch/x86/kernel/ftrace_64.S |  1 +
 2 files changed, 16 insertions(+)
arch/x86/kernel/ftrace.c +15 -0
index 2e11250d5647..1db87ddb268b 100644
@@ -286,6 +286,7 @@ extern void ftrace_regs_caller_ret(void);
 extern void ftrace_caller_end(void);
 extern void ftrace_caller_op_ptr(void);
 extern void ftrace_regs_caller_op_ptr(void);
+extern void ftrace_regs_caller_jmp(void);
 
 /* movq function_trace_op(%rip), %rdx */
 /* 0x48 0x8b 0x15 <offset-to-ftrace_trace_op (4 bytes)> */
@@ -316,6 +317,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	unsigned long end_offset;
 	unsigned long op_offset;
 	unsigned long call_offset;
+	unsigned long jmp_offset;
 	unsigned long offset;
 	unsigned long npages;
 	unsigned long size;
@@ -333,11 +335,13 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 		end_offset = (unsigned long)ftrace_regs_caller_end;
 		op_offset = (unsigned long)ftrace_regs_caller_op_ptr;
 		call_offset = (unsigned long)ftrace_regs_call;
+		jmp_offset = (unsigned long)ftrace_regs_caller_jmp;
 	} else {
 		start_offset = (unsigned long)ftrace_caller;
 		end_offset = (unsigned long)ftrace_caller_end;
 		op_offset = (unsigned long)ftrace_caller_op_ptr;
 		call_offset = (unsigned long)ftrace_call;
+		jmp_offset = 0;
 	}
 
 	size = end_offset - start_offset;
@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	if (WARN_ON(ret < 0))
 		goto fail;
 
+	/* No need to test direct calls on created trampolines */
+	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
+		/* NOP the jnz 1f; but make sure it's a 2 byte jnz */
+		ip = trampoline + (jmp_offset - start_offset);
+		if (WARN_ON(*(char *)ip != 0x75))
+			goto fail;
+		ret = probe_kernel_read(ip, ideal_nops[2], 2);
+		if (ret < 0)
+			goto fail;
+	}
+
 	/*
 	 * The address of the ftrace_ops that is used for this trampoline
 	 * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +1 -0
index 0882758d165a..f72ef157feb3 100644
@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
 	 */
 	movq ORIG_RAX(%rsp), %rax
 	testq	%rax, %rax
+SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL)
 	jnz	1f
 
 	restore_mcount_regs
Signature · 1 line
2.26.1
#4 lore raw

Re: [PATCH 3/3] x86/ftrace: Do not jump to direct code in created trampolines

arch/x86/kernel/ftrace.c create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) · line 380

Quoted 11 lines · Steven Rostedt
arch/x86/kernel/ftrace.c +7 -0
@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	if (WARN_ON(ret < 0))
 		goto fail;
 
+	/* No need to test direct calls on created trampolines */
+	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
+		/* NOP the jnz 1f; but make sure it's a 2 byte jnz */
+		ip = trampoline + (jmp_offset - start_offset);
+		if (WARN_ON(*(char *)ip != 0x75))
+			goto fail;
+		ret = probe_kernel_read(ip, ideal_nops[2], 2);

Now you're just being silly, are you really, actually worried you can't read ideal_nops[] ?

Quoted 12 lines
arch/x86/kernel/ftrace.c +4 -0
+		if (ret < 0)
+			goto fail;
+	}
+
 	/*
 	 * The address of the ftrace_ops that is used for this trampoline
 	 * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +0 -0
index 0882758d165a..f72ef157feb3 100644

arch/x86/kernel/ftrace_64.S SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) · line 246

Quoted 6 lines
arch/x86/kernel/ftrace_64.S +1 -0
@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
 	 */
 	movq ORIG_RAX(%rsp), %rax
 	testq	%rax, %rax
+SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL)
 	jnz	1f
 

I you worry about performance, it would make more sense to do something like so:

SYM_INNER_LABEL(ftrace_regs_caller_from, SYM_L_GLOBAL)
movq ORIG_RAX(%rsp), %rax
testq %rax, %rax
jnz 1f
SYM_INNER_LABEL(ftrace_regs_caller_to, SYM_L_GLOBAL)

	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
		ip = trampoline + (ftrace_regs_caller_from - start_offset);
		((u8[])ip)[0] = JMP8_INSN_OPCODE;
		((u8[])ip)[1] = ftrace_regs_caller_to - ftrace_regs_caller_from - JMP8_INSN_SIZE;
	}

Or nop the whole range, but it's like 10 bytes so I'm not sure that's actually faster.

author maintainer
#5 lore raw

Re: [PATCH 3/3] x86/ftrace: Do not jump to direct code in created trampolines

Quoted 6 lines · On Wed, 22 Apr 2020 22:08:08 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt
arch/x86/kernel/ftrace.c +0 -0
@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	if (WARN_ON(ret < 0))
 		goto fail;
 
Quoted 10 lines

arch/x86/kernel/ftrace.c create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) · line 380

Quoted 7 lines
arch/x86/kernel/ftrace.c +7 -0
+	/* No need to test direct calls on created trampolines */
+	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
+		/* NOP the jnz 1f; but make sure it's a 2 byte jnz */
+		ip = trampoline + (jmp_offset - start_offset);
+		if (WARN_ON(*(char *)ip != 0x75))
+			goto fail;
+		ret = probe_kernel_read(ip, ideal_nops[2], 2);  

Now you're just being silly, are you really, actually worried you can't read ideal_nops[] ?

Hah, that was more cut and paste. I guess a memcpy() would be more appropriate.

Quoted 29 lines
Quoted 13 lines
arch/x86/kernel/ftrace.c +4 -0
+		if (ret < 0)
+			goto fail;
+	}
+
 	/*
 	 * The address of the ftrace_ops that is used for this trampoline
 	 * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +0 -0
index 0882758d165a..f72ef157feb3 100644

arch/x86/kernel/ftrace_64.S SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) · line 246

Quoted 6 lines
arch/x86/kernel/ftrace_64.S +1 -0
@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
 	 */
 	movq ORIG_RAX(%rsp), %rax
 	testq	%rax, %rax
+SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL)
 	jnz	1f
 

I you worry about performance, it would make more sense to do something like so:

SYM_INNER_LABEL(ftrace_regs_caller_from, SYM_L_GLOBAL)
movq ORIG_RAX(%rsp), %rax
testq %rax, %rax
jnz 1f
SYM_INNER_LABEL(ftrace_regs_caller_to, SYM_L_GLOBAL)

Quoted 8 lines
	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
		ip = trampoline + (ftrace_regs_caller_from - start_offset);
		((u8[])ip)[0] = JMP8_INSN_OPCODE;
		((u8[])ip)[1] = ftrace_regs_caller_to - ftrace_regs_caller_from - JMP8_INSN_SIZE;
	}

Or nop the whole range, but it's like 10 bytes so I'm not sure that's actually faster.

That could work too. I'll play with that and actually do some benchmarks to see how much it affects things.

Thanks!

-- Steve