x86_64/ftrace: Emulate calls from int3 when patching functions
Subsystems (5)
Steven Rostedt
maintainercover x86_64/ftrace: Emulate calls from int3 when patching functions
[
This is the non-RFC version.
It went through and passed all my tests. If there's no objections I'm going to include this in my pull request. I still have patches in my INBOX that may still be included, so I need to run those through my tests as well, so a pull request wont be immediate.
]
Nicolai Stange discovered that Live Kernel Patching can have unforseen consequences if tracing is enabled when there are functions that are patched. The reason being, is that Live Kernel patching is built on top of ftrace, which will have the patched functions call the live kernel trampoline directly, and that trampoline will modify the regs->ip address to return to the patched function.
But in the transition between changing the call to the customized trampoline, the tracing code is needed to have its handler called an well, so the function fentry location must be changed from calling the live kernel patching trampoline, to the ftrace_reg_caller trampoline which will iterate through all the registered ftrace handlers for that function.
During this transition, a break point is added to do the live code modifications. But if that break point is hit, it just skips calling any handler, and makes the call site act as a nop. For tracing, the worse that can happen is that you miss a function being traced, but for live kernel patching the affects are more severe, as the old buggy function is now called.
To solve this, an int3_emulate_call() is created for x86_64 to allow ftrace on x86_64 to emulate the call to ftrace_regs_caller() which will make sure all the registered handlers to that function are still called. And this keeps live kernel patching happy!
To mimimize the changes, and to avoid controversial patches, this only changes x86_64. Due to the way x86_32 implements the regs->sp the complexity of emulating calls on that platform is too much for stable patches, and live kernel patching does not support x86_32 anyway.
Josh Poimboeuf (1):
x86_64: Add gap to int3 to allow for call emulation
Peter Zijlstra (2):
x86_64: Allow breakpoints to emulate call instructions ftrace/x86_64: Emulate call function while updating in breakpoint handler
----
arch/x86/entry/entry_64.S | 18 ++++++++++++++++-- arch/x86/include/asm/text-patching.h | 28 ++++++++++++++++++++++++++++ arch/x86/kernel/ftrace.c | 32 +++++++++++++++++++++++++++----- 3 files changed, 71 insertions(+), 7 deletions(-)
Steven Rostedt
maintainerPATCH 1/3 x86_64: Add gap to int3 to allow for call emulation
From: Josh Poimboeuf <[email protected]>
To allow an int3 handler to emulate a call instruction, it must be able to push a return address onto the stack. Add a gap to the stack to allow the int3 handler to push the return address and change the return from int3 to jump straight to the emulated called function target.
[
Note, this is needed to allow Live Kernel Patching to not miss calling a patched function when tracing is enabled. -- Steven Rostedt
]
--- arch/x86/entry/entry_64.S | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +16 -2
@@ -879,7 +879,7 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt * @paranoid == 2 is special: the stub will never switch stacks. This is for * #DF: if the thread stack is somehow unusable, we'll still get a useful OOPS. */ -.macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 +.macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 create_gap=0 ENTRY(\sym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8
@@ -899,6 +899,20 @@ ENTRY(\sym) jnz .Lfrom_usermode_switch_stack_\@ .endif + .if \create_gap == 1 + /* + * If coming from kernel space, create a 6-word gap to allow the + * int3 handler to emulate a call instruction. + */ + testb $3, CS-ORIG_RAX(%rsp) + jnz .Lfrom_usermode_no_gap_\@ + .rept 6 + pushq 5*8(%rsp) + .endr + UNWIND_HINT_IRET_REGS offset=8 +.Lfrom_usermode_no_gap_\@: + .endif + .if \paranoid call paranoid_entry .else
@@ -1130,7 +1144,7 @@ apicinterrupt3 HYPERV_STIMER0_VECTOR \ #endif /* CONFIG_HYPERV */ idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK -idtentry int3 do_int3 has_error_code=0 +idtentry int3 do_int3 has_error_code=0 create_gap=1 idtentry stack_segment do_stack_segment has_error_code=1 #ifdef CONFIG_XEN_PV
Signature · 1 line
2.20.1
Steven Rostedt
maintainerPATCH 2/3 x86_64: Allow breakpoints to emulate call instructions
In reply to Steven RostedtFrom: Peter Zijlstra <peterz at infradead.org>
In order to allow breakpoints to emulate call instructions, they need to push the return address onto the stack. The x86_64 int3 handler adds a small gap to allow the stack to grow some. Use this gap to add the return address to be able to emulate a call instruction at the breakpoint location.
These helper functions are added:
int3_emulate_jmp(): changes the location of the regs->ip to return there.
(The next two are only for x86_64)
int3_emulate_push(): to push the address onto the gap in the stack int3_emulate_call(): push the return address and change regs->ip
--- arch/x86/include/asm/text-patching.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/x86/include/asm/text-patching.h +28 -0
@@ -39,4 +39,32 @@ extern int poke_int3_handler(struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +static inline void int3_emulate_jmp (struct pt_regs *regs,unsigned long ip) +{ + regs->ip = ip; +} + +#define INT3_INSN_SIZE 1 +#define CALL_INSN_SIZE 5 + +#ifdef CONFIG_X86_64 +static inline void int3_emulate_push (struct pt_regs *regs,unsigned long val) +{ +/* +* The int3 handler in entry_64.S adds a gap between the +* stack where the break point happened, and the saving of +* pt_regs. We can extend the original stack because of +* this gap. See the idtentry macro's create_gap option. +*/ + regs->sp -=sizeof (unsigned long ); + *(unsigned long *)regs->sp = val; +} + +static inline void int3_emulate_call (struct pt_regs *regs,unsigned long func) +{ +int3_emulate_push (regs, regs->ip -INT3_INSN_SIZE +CALL_INSN_SIZE ); +int3_emulate_jmp (regs, func); +} +#endif +#endif /* _ASM_X86_TEXT_PATCHING_H */
Signature · 1 line
2.20.1
Steven Rostedt
maintainerPATCH 3/3 ftrace/x86_64: Emulate call function while updating in breakpoint handler
In reply to Steven RostedtFrom: Peter Zijlstra <[email protected]>
Nicolai Stange discovered[1] that if live kernel patching is enabled, and the function tracer started tracing the same function that was patched, the conversion of the fentry call site during the translation of going from calling the live kernel patch trampoline to the iterator trampoline, would have as slight window where it didn't call anything.
As live kernel patching depends on ftrace to always call its code (to prevent the function being traced from being called, as it will redirect it). This small window would allow the old buggy function to be called, and this can cause undesirable results.
Nicolai submitted new patches[2] but these were controversial. As this is similar to the static call emulation issues that came up a while ago[3]. But after some debate[4][5] adding a gap in the stack when entering the breakpoint handler allows for pushing the return address onto the stack to easily emulate a call.
[1] http://lkml.kernel.org/r/20180726104029.7736-1-nstange at suse.de
[2] http://lkml.kernel.org/r/20190427100639.15074-1-nstange at suse.de
[3] http://lkml.kernel.org/r/3cf04e113d71c9f8e4be95fb84a510f085aa4afa.1541711457.git.jpoimboe at redhat.com
[4] http://lkml.kernel.org/r/CAHk-=wh5OpheSU8Em_Q3Hg8qw_JtoijxOdPtHru6d+5K8TWM=A at mail.gmail.com
[5] http://lkml.kernel.org/r/CAHk-=wjvQxY4DvPrJ6haPgAa6b906h=MwZXO6G8OtiTGe=N7_w at mail.gmail.com
[
Live kernel patching is not implemented on x86_32, thus the emulate calls are only for x86_64.
]
--- arch/x86/kernel/ftrace.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)
arch/x86/kernel/ftrace.c +27 -5
@@ -29,6 +29,7 @@#include <asm/kprobes.h> #include <asm/ftrace.h> #include <asm/nops.h> +#include <asm/text-patching.h> #ifdef CONFIG_DYNAMIC_FTRACE
@@ -231,6 +232,7 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, }static unsigned long ftrace_update_func; +static unsigned long ftrace_update_func_call;static int update_ftrace_func (unsigned long ip,void *new) {
@@ -259,6 +261,8 @@ int ftrace_update_ftrace_func(ftrace_func_t func)unsigned char *new;int ret; + ftrace_update_func_call = (unsigned long )func; + new =ftrace_call_replace (ip, (unsigned long )func); ret =update_ftrace_func (ip, new);
@@ -294,13 +298,28 @@ int ftrace_int3_handler(struct pt_regs *regs)if (WARN_ON_ONCE (!regs))return 0 ; - ip = regs->ip -1 ; -if (!ftrace_location (ip) && !is_ftrace_caller (ip)) -return 0 ; + ip = regs->ip -INT3_INSN_SIZE ; - regs->ip +=MCOUNT_INSN_SIZE -1 ; +#ifdef CONFIG_X86_64 +if (ftrace_location (ip)) { +int3_emulate_call (regs, (unsigned long )ftrace_regs_caller); +return 1 ; + }else if (is_ftrace_caller (ip)) { +if (!ftrace_update_func_call) { +int3_emulate_jmp (regs, ip +CALL_INSN_SIZE ); +return 1 ; + } +int3_emulate_call (regs, ftrace_update_func_call); +return 1 ; + } +#else +if (ftrace_location (ip) ||is_ftrace_caller (ip)) { +int3_emulate_jmp (regs, ip +CALL_INSN_SIZE ); +return 1 ; + } +#endif -return 1 ; +return 0 ; }NOKPROBE_SYMBOL (ftrace_int3_handler);
@@ -859,6 +878,8 @@ void arch_ftrace_update_trampoline(struct ftrace_ops *ops) func =ftrace_ops_get_func (ops); + ftrace_update_func_call = (unsigned long )func; +/* Do a safe modify in case the trampoline is executing */ new =ftrace_call_replace (ip, (unsigned long )func); ret =update_ftrace_func (ip, new);
@@ -960,6 +981,7 @@ static int ftrace_mod_jmp(unsigned long ip, void *func) {unsigned char *new; + ftrace_update_func_call =0UL ; new =ftrace_jmp_replace (ip, (unsigned long )func);return update_ftrace_func (ip, new);
Signature · 1 line
2.20.1
Quoted 31 lines · On Tue, 07 May 2019 21:55:59 -0400 Steven Rostedt <[email protected]> wrote:
[
This is the non-RFC version.It went through and passed all my tests. If there's no objections I'm going to include this in my pull request. I still have patches in my INBOX that may still be included, so I need to run those through my tests as well, so a pull request wont be immediate.
]Nicolai Stange discovered that Live Kernel Patching can have unforseen consequences if tracing is enabled when there are functions that are patched. The reason being, is that Live Kernel patching is built on top of ftrace, which will have the patched functions call the live kernel trampoline directly, and that trampoline will modify the
regs->ipaddress to return to the patched function.But in the transition between changing the call to the customized trampoline, the tracing code is needed to have its handler called an well, so the function fentry location must be changed from calling the live kernel patching trampoline, to the
ftrace_reg_callertrampoline which will iterate through all the registered ftrace handlers for that function.During this transition, a break point is added to do the live code modifications. But if that break point is hit, it just skips calling any handler, and makes the call site act as a nop. For tracing, the worse that can happen is that you miss a function being traced, but for live kernel patching the affects are more severe, as the old buggy function is now called.
Quoted 4 lines
To solve this, an
int3_emulate_call()is created forx86_64to allow ftrace onx86_64to emulate the call toftrace_regs_caller()which will make sure all the registered handlers to that function are still called. And this keeps live kernel patching happy!
Out of curiosity, would you have any idea to re-use these function for other use-case? Maybe kprobes can reuse it, but very limited use-case.
Quoted 4 lines
To mimimize the changes, and to avoid controversial patches, this only changes
x86_64. Due to the wayx86_32implements theregs->spthe complexity of emulating calls on that platform is too much for stable patches, and live kernel patching does not supportx86_32anyway.
This series looks good to me.
Thanks!
Signature · 1 line
Masami Hiramatsu <mhiramat at kernel.org>
Steven Rostedt
maintainerQuoted 7 lines · On Wed, 8 May 2019 13:30:22 +0900 Masami Hiramatsu <mhiramat at kernel.org>…
Quoted 4 lines
To solve this, an
int3_emulate_call()is created forx86_64to allow ftrace onx86_64to emulate the call toftrace_regs_caller()which will make sure all the registered handlers to that function are still called. And this keeps live kernel patching happy!Out of curiosity, would you have any idea to re-use these function for other use-case? Maybe kprobes can reuse it, but very limited use-case.
Yes, but only for x86_64.
Quoted 9 lines
Quoted 5 lines
To mimimize the changes, and to avoid controversial patches, this only changes
x86_64. Due to the wayx86_32implements theregs->spthe complexity of emulating calls on that platform is too much for stable patches, and live kernel patching does not supportx86_32anyway.This series looks good to me.
Reviewed-by Masami Hiramatsu <mhiramat at kernel.org>
Thanks Masami!
-- Steve
Quoted 8 lines · Steven Rostedt
[
This is the non-RFC version.It went through and passed all my tests. If there's no objections I'm going to include this in my pull request. I still have patches in my INBOX that may still be included, so I need to run those through my tests as well, so a pull request wont be immediate.
]
<snip />
Quoted 6 lines
Josh Poimboeuf (1):
x86_64: Add gap to int3 to allow for call emulationPeter Zijlstra (2):
x86_64: Allow breakpoints to emulate call instructions ftrace/x86_64: Emulate call function while updating in breakpoint handler
Reviewed-and-tested-by: Nicolai Stange <nstange at suse.de>
for the whole series. Many, many thanks to everybody involved!
I'll resend that live patching selftest once this fix here has been merged.
Nicolai
Signature · 2 lines
SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)