x86/ftrace: Make created trampolines not call direct code
Steven Rostedt
maintainercover x86/ftrace: Make created trampolines not call direct code
Peter,
These three patches are built on top of your patch series.
This is what I was thinking of having. This isn't fully tested (as your patch series is still running on my test suite).
-- Steve
Steven Rostedt (VMware) (3):
x86/ftrace: Make non direct case the default in ftrace_regs_caller x86/ftrace: Only have the builtin ftrace_regs_caller call direct hooks x86/ftrace: Do not jump to direct code in created trampolines
----
arch/x86/kernel/ftrace.c | 14 +++++++++++--- arch/x86/kernel/ftrace_64.S | 29 ++++++++++++++--------------- kernel/trace/ftrace.c | 8 ++++++++ 3 files changed, 33 insertions(+), 18 deletions(-)
Steven Rostedt
maintainerPATCH 1/3 x86/ftrace: Make non direct case the default in ftrace_regs_caller
From: "Steven Rostedt (VMware)" <[email protected]>
If a direct function is hooked along with one of the ftrace registered functions, then the ftrace_regs_caller is attached to the function that shares the direct hook as well as the ftrace hook. The ftrace_regs_caller will call ftrace_ops_list_func() that iterates through all the registered ftrace callbacks, and if there's a direct callback attached to that function, the direct ftrace_ops callback is called to notify that ftrace_regs_caller to return to the direct caller instead of going back to the function that called it.
But this is a very uncommon case. Currently, the code has it as the default case. Modify ftrace_regs_caller to make the default case (the non jump) to just return normally, and have the jump to the handling of the direct caller.
--- arch/x86/kernel/ftrace_64.S | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
arch/x86/kernel/ftrace_64.S +15 -16
@@ -241,22 +241,9 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) */ movq ORIG_RAX(%rsp), %rax testq %rax, %rax - jz 1f + jnz 1f - /* Swap the flags with orig_rax */ - movq MCOUNT_REG_SIZE(%rsp), %rdi - movq %rdi, MCOUNT_REG_SIZE-8(%rsp) - movq %rax, MCOUNT_REG_SIZE(%rsp) - - restore_mcount_regs 8 - /* Restore flags */ - popfq - -SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL); - UNWIND_HINT_RET_OFFSET - jmp ftrace_epilogue - -1: restore_mcount_regs + restore_mcount_regs /* Restore flags */ popfq
@@ -266,9 +253,21 @@ SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL); * The trampoline will add the code to jump * to the return. */ -SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) +SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL) jmp ftrace_epilogue + /* Swap the flags with orig_rax */ +1: movq MCOUNT_REG_SIZE(%rsp), %rdi + movq %rdi, MCOUNT_REG_SIZE-8(%rsp) + movq %rax, MCOUNT_REG_SIZE(%rsp) + + restore_mcount_regs 8 + /* Restore flags */ + popfq +SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) + UNWIND_HINT_RET_OFFSET + jmp ftrace_epilogue + SYM_FUNC_END(ftrace_regs_caller)
Signature · 1 line
2.26.1
Steven Rostedt
maintainerPATCH 2/3 x86/ftrace: Only have the builtin ftrace_regs_caller call direct hooks
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
If a direct hook is attached to a function that ftrace also has a function attached to it, then it is required that the ftrace_ops_list_func() is used to iterate over the registered ftrace callbacks. This will also include the direct ftrace_ops helper, that tells ftrace_regs_caller where to return to (the direct callback and not the function that called it).
As this direct helper is only to handle the case of ftrace callbacks attached to the same function as the direct callback, the ftrace callback allocated trampolines (used to only call them), should never be used to return back to a direct callback.
Only copy the portion of the ftrace_regs_caller that will return back to what called it, and not the portion that returns back to the direct caller.
The direct ftrace_ops must then pick the ftrace_regs_caller builtin function as its own trampoline to ensure that it will never have one allocated for it (which would not include the handling of direct callbacks).
--- arch/x86/kernel/ftrace.c | 7 ------- arch/x86/kernel/ftrace_64.S | 3 +-- kernel/trace/ftrace.c | 8 ++++++++ 3 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/kernel/ftrace.c +0 -7
@@ -367,13 +367,6 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)if (WARN_ON (ret <0 )) goto fail; -if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) { - ip = trampoline + (ftrace_regs_caller_ret - ftrace_regs_caller); - ret =probe_kernel_read (ip, (void *)retq,RET_SIZE ); -if (WARN_ON (ret <0 )) - goto fail; - } -/* * The address of the ftrace_ops that is used for this trampoline * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +1 -2
@@ -253,7 +253,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) * The trampoline will add the code to jump * to the return. */ -SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL) +SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) jmp ftrace_epilogue /* Swap the flags with orig_rax */
@@ -264,7 +264,6 @@ SYM_INNER_LABEL(ftrace_regs_caller_ret, SYM_L_GLOBAL) restore_mcount_regs 8 /* Restore flags */ popfq -SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) UNWIND_HINT_RET_OFFSET jmp ftrace_epilogue
kernel/trace/ftrace.c +8 -0
@@ -2399,6 +2399,14 @@ struct ftrace_ops direct_ops = { .flags = FTRACE_OPS_FL_IPMODIFY | FTRACE_OPS_FL_RECURSION_SAFE |FTRACE_OPS_FL_DIRECT |FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_PERMANENT, +/* +* By declaring the main trampoline as this trampoline +* it will never have one allocated for it. Allocated +* trampolines should not call direct functions. +* The direct_ops should only be called by the builtin +* ftrace_regs_caller trampoline. +*/ + .trampoline = FTRACE_REGS_ADDR, };#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
Signature · 1 line
2.26.1
Steven Rostedt
maintainerPATCH 3/3 x86/ftrace: Do not jump to direct code in created trampolines
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
When creating a trampoline based on the ftrace_regs_caller code, nop out the jnz test that would jmup to the code that would return to a direct caller (stored in the ORIG_RAX field) and not back to the function that called it.
--- arch/x86/kernel/ftrace.c | 15 +++++++++++++++ arch/x86/kernel/ftrace_64.S | 1 + 2 files changed, 16 insertions(+)
arch/x86/kernel/ftrace.c +15 -0
@@ -286,6 +286,7 @@ extern void ftrace_regs_caller_ret(void);extern void ftrace_caller_end (void );extern void ftrace_caller_op_ptr (void );extern void ftrace_regs_caller_op_ptr (void ); +extern void ftrace_regs_caller_jmp (void );/* movq function_trace_op(%rip), %rdx */ /* 0x48 0x8b 0x15 <offset-to-ftrace_trace_op (4 bytes)> */
@@ -316,6 +317,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)unsigned long end_offset;unsigned long op_offset;unsigned long call_offset; +unsigned long jmp_offset;unsigned long offset;unsigned long npages;unsigned long size;
@@ -333,11 +335,13 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) end_offset = (unsigned long )ftrace_regs_caller_end; op_offset = (unsigned long )ftrace_regs_caller_op_ptr; call_offset = (unsigned long )ftrace_regs_call; + jmp_offset = (unsigned long )ftrace_regs_caller_jmp; }else { start_offset = (unsigned long )ftrace_caller; end_offset = (unsigned long )ftrace_caller_end; op_offset = (unsigned long )ftrace_caller_op_ptr; call_offset = (unsigned long )ftrace_call; + jmp_offset =0 ; } size = end_offset - start_offset;
@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)if (WARN_ON (ret <0 )) goto fail; +/* No need to test direct calls on created trampolines */ +if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) { +/* NOP the jnz 1f; but make sure it's a 2 byte jnz */ + ip = trampoline + (jmp_offset - start_offset); +if (WARN_ON (*(char *)ip !=0x75 )) + goto fail; + ret =probe_kernel_read (ip, ideal_nops[2 ],2 ); +if (ret <0 ) + goto fail; + } +/* * The address of the ftrace_ops that is used for this trampoline * is stored at the end of the trampoline. This will be used to
arch/x86/kernel/ftrace_64.S +1 -0
@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) */ movq ORIG_RAX(%rsp), %rax testq %rax, %rax +SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) jnz 1f restore_mcount_regs
Signature · 1 line
2.26.1
Re: [PATCH 3/3] x86/ftrace: Do not jump to direct code in created trampolines
arch/x86/kernel/ftrace.c create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) · line 380
Quoted 11 lines · Steven Rostedt
arch/x86/kernel/ftrace.c +7
-0@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)if (WARN_ON (ret <0 )) goto fail; +/* No need to test direct calls on created trampolines */ +if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) { +/* NOP the jnz 1f; but make sure it's a 2 byte jnz */ + ip = trampoline + (jmp_offset - start_offset); +if (WARN_ON (*(char *)ip !=0x75 )) + goto fail; + ret =probe_kernel_read (ip, ideal_nops[2 ],2 );
Now you're just being silly, are you really, actually worried you can't read ideal_nops[] ?
Quoted 12 lines
arch/x86/kernel/ftrace.c +4
-0+if (ret <0 ) + goto fail; + } +/* * The address of the ftrace_ops that is used for this trampoline * is stored at the end of the trampoline. This will be used to arch/x86/kernel/ftrace_64.S +0
-0
arch/x86/kernel/ftrace_64.S SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) · line 246
Quoted 6 lines
arch/x86/kernel/ftrace_64.S +1
-0@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) */ movq ORIG_RAX(%rsp), %rax testq %rax, %rax +SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) jnz 1f
I you worry about performance, it would make more sense to do something like so:
SYM_INNER_LABEL(ftrace_regs_caller_from, SYM_L_GLOBAL)
movq ORIG_RAX(%rsp), %rax
testq %rax, %rax
jnz 1fSYM_INNER_LABEL(ftrace_regs_caller_to, SYM_L_GLOBAL)
if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
ip = trampoline + (ftrace_regs_caller_from - start_offset);
((u8[])ip)[0] = JMP8_INSN_OPCODE;
((u8[])ip)[1] = ftrace_regs_caller_to - ftrace_regs_caller_from - JMP8_INSN_SIZE;
}Or nop the whole range, but it's like 10 bytes so I'm not sure that's actually faster.
Steven Rostedt
maintainerRe: [PATCH 3/3] x86/ftrace: Do not jump to direct code in created trampolines
Quoted 6 lines · On Wed, 22 Apr 2020 22:08:08 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt
arch/x86/kernel/ftrace.c +0
-0@@ -367,6 +371,17 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)if (WARN_ON (ret <0 )) goto fail;
Quoted 10 lines
arch/x86/kernel/ftrace.c create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) · line 380
Quoted 7 lines
arch/x86/kernel/ftrace.c +7
-0+/* No need to test direct calls on created trampolines */ +if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) { +/* NOP the jnz 1f; but make sure it's a 2 byte jnz */ + ip = trampoline + (jmp_offset - start_offset); +if (WARN_ON (*(char *)ip !=0x75 )) + goto fail; + ret =probe_kernel_read (ip, ideal_nops[2 ],2 );Now you're just being silly, are you really, actually worried you can't read ideal_nops[] ?
Hah, that was more cut and paste. I guess a memcpy() would be more appropriate.
Quoted 29 lines
Quoted 13 lines
arch/x86/kernel/ftrace.c +4
-0+if (ret <0 ) + goto fail; + } +/* * The address of the ftrace_ops that is used for this trampoline * is stored at the end of the trampoline. This will be used to arch/x86/kernel/ftrace_64.S +0
-0arch/x86/kernel/ftrace_64.S SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) · line 246
Quoted 6 lines
arch/x86/kernel/ftrace_64.S +1
-0@@ -241,6 +241,7 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) */ movq ORIG_RAX(%rsp), %rax testq %rax, %rax +SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) jnz 1fI you worry about performance, it would make more sense to do something like so:
SYM_INNER_LABEL(ftrace_regs_caller_from, SYM_L_GLOBAL)
movqORIG_RAX(%rsp), %rax
testq %rax, %rax
jnz 1fSYM_INNER_LABEL(ftrace_regs_caller_to, SYM_L_GLOBAL)
Quoted 8 lines
if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) { ip = trampoline + (ftrace_regs_caller_from - start_offset); ((u8[])ip)[0] = JMP8_INSN_OPCODE; ((u8[])ip)[1] = ftrace_regs_caller_to - ftrace_regs_caller_from - JMP8_INSN_SIZE; }Or nop the whole range, but it's like 10 bytes so I'm not sure that's actually faster.
That could work too. I'll play with that and actually do some benchmarks to see how much it affects things.
Thanks!
-- Steve