arm64: implement KPROBES_ON_FTRACE
Subsystems (3)
PATCH v4 1/1 arm64: implement KPROBES_ON_FTRACE
KPROBES_ON_FTRACE avoids much of the overhead with regular kprobes as it eliminates the need for a trap, as well as the need to emulate or single-step instructions.
Tested on berlin arm64 platform.
~ # mount -t debugfs debugfs /sys/kernel/debug/
~ # cd /sys/kernel/debug//sys/kernel/debug # echo 'p _do_fork' > tracing/kprobe_events
before the patch:
/sys/kernel/debug # cat kprobes/listffffff801009fe28 k _do_fork+0x0 [DISABLED]
after the patch:
/sys/kernel/debug # cat kprobes/listffffff801009ff54 k _do_fork+0x4 [DISABLED][FTRACE]
---KPROBES_ON_FTRACE avoids much of the overhead with regular kprobes as it eliminates the need for a trap, as well as the need to emulate or single-step instructions.
Applied after arm64 FTRACE_WITH_REGS:
http://lists.infradead.org/pipermail/linux-arm-kernel/2019-August/674404.html
Changes since v3:
- move kprobe_lookup_name() and arch_kprobe_on_func_entry to ftrace.c since we only want to choose the ftrace entry for KPROBES_ON_FTRACE.
- only choose ftrace entry if (addr && !offset)
Changes since v2:
- remove patch1, make it a single cleanup patch
- remove "This patch" in the change log
- implement arm64's kprobe_lookup_name() and arch_kprobe_on_func_entry instead of patching the common kprobes code
Changes since v1:
- make the kprobes/x86: use instruction_pointer and instruction_pointer_set as patch1
- add Masami's ACK to patch1
- add some description about KPROBES_ON_FTRACE and why we need it on
arm64
- correct the log before the patch
- remove the consolidation patch, make it as TODO
- only adjust kprobe's addr when KPROBE_FLAG_FTRACE is set
- if KPROBES_ON_FTRACE, ftrace_call_adjust() the kprobe's addr before calling ftrace_location()
- update the kprobes-on-ftrace/arch-support.txt in doc
.../debug/kprobes-on-ftrace/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/kernel/probes/Makefile | 1 + arch/arm64/kernel/probes/ftrace.c | 84 +++++++++++++++++++ 4 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/probes/ftrace.c
Documentation/features/debug/kprobes-on-ftrace/arch-support.txt +1 -1
@@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | csky: | TODO | | h8300: | TODO |
arch/arm64/Kconfig +1 -0
@@ -167,6 +167,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES + select HAVE_KPROBES_ON_FTRACE select HAVE_KRETPROBES select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT
arch/arm64/kernel/probes/Makefile +1 -0
@@ -4,3 +4,4 @@ obj-$(CONFIG_KPROBES) += kprobes.o decode-insn.o \ simulate-insn.o obj-$(CONFIG_UPROBES) += uprobes.o decode-insn.o \ simulate-insn.o +obj-$(CONFIG_KPROBES_ON_FTRACE) += ftrace.o
arch/arm64/kernel/probes/ftrace.c new file +84 -0
84 lines not on this page · raw message
Signature · 7 lines
2.23.0.rc1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 15 lines · Jisheng Zhang
KPROBES_ON_FTRACEavoids much of the overhead with regular kprobes as it eliminates the need for a trap, as well as the need to emulate or single-step instructions.Tested on berlin arm64 platform.
~ # mount -t debugfs debugfs
/sys/kernel/debug/
~ # cd/sys/kernel/debug//sys/kernel/debug# echo 'p_do_fork' > tracing/kprobe_eventsbefore the patch:
/sys/kernel/debug# cat kprobes/listffffff801009fe28k_do_fork+0x0[DISABLED]
Quoted 6 lines
after the patch:
/sys/kernel/debug# cat kprobes/listffffff801009ff54k_do_fork+0x4[DISABLED][FTRACE]Signed-off-by Jisheng Zhang <[email protected]>
This looks good to me. Except for a small confirmation below:
Quoted 100 lines
---
KPROBES_ON_FTRACEavoids much of the overhead with regular kprobes as it eliminates the need for a trap, as well as the need to emulate or single-step instructions.Applied after arm64
FTRACE_WITH_REGS:
http://lists.infradead.org/pipermail/linux-arm-kernel/2019-August/674404.htmlChanges since v3:
- movekprobe_lookup_name()andarch_kprobe_on_func_entrytoftrace.csince we only want to choose the ftrace entry forKPROBES_ON_FTRACE.
- only choose ftrace entry if (addr && !offset)Changes since v2:
- remove patch1, make it a single cleanup patch
- remove "This patch" in the change log
- implement arm64'skprobe_lookup_name()andarch_kprobe_on_func_entryinstead of patching the common kprobes codeChanges since v1:
- make the kprobes/x86: useinstruction_pointerandinstruction_pointer_setas patch1
- add Masami's ACK to patch1
- add some description aboutKPROBES_ON_FTRACEand why we need it onarm64- correct the log before the patch
- remove the consolidation patch, make it as TODO
- only adjust kprobe's addr whenKPROBE_FLAG_FTRACEis set
- ifKPROBES_ON_FTRACE,ftrace_call_adjust()the kprobe's addr before callingftrace_location()
- update thekprobes-on-ftrace/arch-support.txtin doc.../debug/kprobes-on-ftrace/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/kernel/probes/Makefile | 1 + arch/arm64/kernel/probes/ftrace.c | 84 +++++++++++++++++++ 4 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/probes/ftrace.cDocumentation/features/debug/kprobes-on-ftrace/arch-support.txt +1
-1@@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | csky: | TODO | | h8300: | TODO |arch/arm64/Kconfig +1
-0@@ -167,6 +167,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES + select HAVE_KPROBES_ON_FTRACE select HAVE_KRETPROBES select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORTarch/arm64/kernel/probes/Makefile +1
-0@@ -4,3 +4,4 @@ obj-$(CONFIG_KPROBES) += kprobes.o decode-insn.o \ simulate-insn.o obj-$(CONFIG_UPROBES) += uprobes.o decode-insn.o \ simulate-insn.o +obj-$(CONFIG_KPROBES_ON_FTRACE) += ftrace.oarch/arm64/kernel/probes/ftrace.c new file +20
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Dynamic Ftrace based Kprobes Optimization +* +* Copyright (C) Hitachi Ltd., 2012 +* Copyright (C) 2019 Jisheng Zhang <[email protected]> +* Synaptics Incorporated +*/ + +#include <linux/kprobes.h> + +/* Ftrace callback handler for kprobes -- called under preepmt disabed */ +void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct pt_regs *regs) +{ +struct kprobe *p; +struct kprobe_ctlblk *kcb; + +/* Preempt is disabled by ftrace */ + p =get_kprobe ((kprobe_opcode_t *)ip);
arch/arm64/kernel/probes/ftrace.c line 30
Quoted 10 lines
arch/arm64/kernel/probes/ftrace.c new file +10
-0+if (unlikely (!p) ||kprobe_disabled (p)) +return ; + + kcb =get_kprobe_ctlblk (); +if (kprobe_running ()) { +kprobes_inc_nmissed_count (p); + }else { +unsigned long orig_ip =instruction_pointer (regs); +/* Kprobe handler expects regs->pc = pc + 4 as breakpoint hit */ +instruction_pointer_set (regs, ip +sizeof (kprobe_opcode_t));
Just want to make sure that you've confirmed that this is what happens with a regular trap/brk based kprobe on ARM64. The reason for setting the instruction pointer here is to ensure that it is set to the same value as would be set if there was a trap/brk instruction at the ftrace location. This ensures that the kprobe pre handler sees the same value regardless.
Further changes to the instruction pointer are to achieve the same effect for kprobe post handlers.
- Naveen
Hi,
Quoted 29 lines · On Thu, 22 Aug 2019 12:23:58 +0530 "Naveen N. Rao"…
Quoted 15 lines · Jisheng Zhang
KPROBES_ON_FTRACEavoids much of the overhead with regular kprobes as it eliminates the need for a trap, as well as the need to emulate or single-step instructions.Tested on berlin arm64 platform.
~ # mount -t debugfs debugfs
/sys/kernel/debug/
~ # cd/sys/kernel/debug//sys/kernel/debug# echo 'p_do_fork' > tracing/kprobe_eventsbefore the patch:
/sys/kernel/debug# cat kprobes/listffffff801009fe28k_do_fork+0x0[DISABLED]Quoted 6 lines
after the patch:
/sys/kernel/debug# cat kprobes/listffffff801009ff54k_do_fork+0x4[DISABLED][FTRACE]Signed-off-by Jisheng Zhang <[email protected]>This looks good to me. Except for a small confirmation below:
Reviewed-by Naveen N. Rao <[email protected]>
<...>
Quoted 18 lines
Quoted 18 lines
arch/arm64/kernel/probes/ftrace.c new file +18
-0+/* Ftrace callback handler for kprobes -- called under preepmt disabed */ +void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct pt_regs *regs) +{ +struct kprobe *p; +struct kprobe_ctlblk *kcb; + +/* Preempt is disabled by ftrace */ + p =get_kprobe ((kprobe_opcode_t *)ip); +if (unlikely (!p) ||kprobe_disabled (p)) +return ; + + kcb =get_kprobe_ctlblk (); +if (kprobe_running ()) { +kprobes_inc_nmissed_count (p); + }else { +unsigned long orig_ip =instruction_pointer (regs); +/* Kprobe handler expects regs->pc = pc + 4 as breakpoint hit */
Quoted 8 lines
Quoted 1 line
+
instruction_pointer_set(regs, ip + sizeof(kprobe_opcode_t));Just want to make sure that you've confirmed that this is what happens with a regular trap/brk based kprobe on ARM64. The reason for setting the instruction pointer here is to ensure that it is set to the same value as would be set if there was a trap/brk instruction at the ftrace location. This ensures that the kprobe pre handler sees the same value regardless.
Due to the arm64's DYNAMIC_FTRACE_WITH_REGS implementation, the code itself is correct. But this doesn't look like "there was a trap instruction at the ftrace location".
W/O KPROBE_ON_FTRACE:
foo:
00 insA
04 insB
08 insC
kprobe's pre_handler() will see pc points to 00.
W/ KPROBE_ON_FTRACE:
foo:
00 lr saver
04 nop // will be modified to ftrace call ins when KPROBE is armed
08 insA
0c insB
later, kprobe_ftrace_handler() will see pc points to 04, so pc + 4 will point to 08 the same as the one w/o KPROBE_ON_FTRACE.
It seems I need to fix the comment.
Quoted 5 lines
Further changes to the instruction pointer are to achieve the same effect for kprobe post handlers.
Quoted 5 lines · Jisheng Zhang
Hi,
Quoted 1 line · On Thu, 22 Aug 2019 12:23:58 +0530 "Naveen N. Rao"…
Jisheng Zhang wrote:
...
Quoted 32 lines
Quoted 19 lines
Quoted 19 lines
+/* Ftrace callback handler for kprobes -- called under preepmt disabed */
arch/arm64/kernel/probes/ftrace.c new file +17
-0+void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct pt_regs *regs) +{ +struct kprobe *p; +struct kprobe_ctlblk *kcb; + +/* Preempt is disabled by ftrace */ + p =get_kprobe ((kprobe_opcode_t *)ip); +if (unlikely (!p) ||kprobe_disabled (p)) +return ; + + kcb =get_kprobe_ctlblk (); +if (kprobe_running ()) { +kprobes_inc_nmissed_count (p); + }else { +unsigned long orig_ip =instruction_pointer (regs); +/* Kprobe handler expects regs->pc = pc + 4 as breakpoint hit */ Quoted 8 lines
Quoted 1 line
+
instruction_pointer_set(regs, ip + sizeof(kprobe_opcode_t));Just want to make sure that you've confirmed that this is what happens with a regular trap/brk based kprobe on ARM64. The reason for setting the instruction pointer here is to ensure that it is set to the same value as would be set if there was a trap/brk instruction at the ftrace location. This ensures that the kprobe pre handler sees the same value regardless.
Due to the arm64's
DYNAMIC_FTRACE_WITH_REGSimplementation, the code itself is correct. But this doesn't look like "there was a trap instruction at the ftrace location".
Quoted 8 lines
W/O
KPROBE_ON_FTRACE:foo:
00 insA
04 insB
08 insCkprobe's
pre_handler()will see pc points to 00.
In this case, the probe will be placed at foo+0x00, so pre_handler() seeing that address in pt_regs is correct behavior - as long as arm64 'brk' instruction causes an exception with the instruction pointer set *to* the 'brk' instruction. This is similar to how powerpc 'trap' works. However, x86 'int3' causes an exception *after* execution of the instruction.
Quoted 8 lines
W/
KPROBE_ON_FTRACE:foo:
00 lr saver
04 nop // will be modified to ftrace call ins when KPROBE is armed
08 insA
0c insB
In this case, if user asks for a probe to be placed at 'foo', we will choose foo+0x04 and from that point on, the behavior should reflect that a kprobe was placed at foo+0x04. In particular, the pre_handler() should see foo+0x04 in pt_regs. The post_handler() would then see foo+0x08.
Quoted 3 lines
later,
kprobe_ftrace_handler()will see pc points to 04, so pc + 4 will point to 08 the same as the one w/oKPROBE_ON_FTRACE.
I didn't mean to compare regular trap/brk based kprobes with KPROBES_ON_FTRACE. The only important aspect is that the handlers see consistent pt_regs in both cases, depending on where the kprobe was placed. Choosing a different address/offset to place a kprobe during its registration is an orthogonal aspect.
Quoted 2 lines
It seems I need to fix the comment.
Given your explanation above, I think you can simply drop the first adjustment to the instruction pointer before the pre handler invocation. The rest of the code looks fine.
- Naveen
Quoted 48 lines · On Thu, 22 Aug 2019 15:52:05 +0530 "Naveen N. Rao"…
Quoted 5 lines · Jisheng Zhang
Hi,
Quoted 1 line · On Thu, 22 Aug 2019 12:23:58 +0530 "Naveen N. Rao"…
Jisheng Zhang wrote:
...
Quoted 39 lines
Quoted 19 lines
Quoted 19 lines
+/* Ftrace callback handler for kprobes -- called under preepmt disabed */
arch/arm64/kernel/probes/ftrace.c new file +17
-0+void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct pt_regs *regs) +{ +struct kprobe *p; +struct kprobe_ctlblk *kcb; + +/* Preempt is disabled by ftrace */ + p =get_kprobe ((kprobe_opcode_t *)ip); +if (unlikely (!p) ||kprobe_disabled (p)) +return ; + + kcb =get_kprobe_ctlblk (); +if (kprobe_running ()) { +kprobes_inc_nmissed_count (p); + }else { +unsigned long orig_ip =instruction_pointer (regs); +/* Kprobe handler expects regs->pc = pc + 4 as breakpoint hit */ Quoted 8 lines
Quoted 1 line
+
instruction_pointer_set(regs, ip + sizeof(kprobe_opcode_t));Just want to make sure that you've confirmed that this is what happens with a regular trap/brk based kprobe on ARM64. The reason for setting the instruction pointer here is to ensure that it is set to the same value as would be set if there was a trap/brk instruction at the ftrace location. This ensures that the kprobe pre handler sees the same value regardless.
Due to the arm64's
DYNAMIC_FTRACE_WITH_REGSimplementation, the code itself is correct. But this doesn't look like "there was a trap instruction at the ftrace location".W/O
KPROBE_ON_FTRACE:foo:
00 insA 04 insB 08 insC
Quoted 5 lines
Quoted 1 line
kprobe's
pre_handler()will see pc points to 00.In this case, the probe will be placed at
foo+0x00, sopre_handler()seeing that address inpt_regsis correct behavior - as long as arm64 'brk' instruction causes an exception with the instruction pointer set
Yep, confirmed with regular trap/brk based kprobes, I do see PC set to the "brk" instruction.
Quoted 3 lines
*to* the 'brk' instruction. This is similar to how powerpc 'trap' works. However, x86 'int3' causes an exception *after* execution of the instruction.
Got it. I understand where's the comment "expects regs->pc = pc + 1" from.
Quoted 16 lines
Quoted 9 lines
W/
KPROBE_ON_FTRACE:foo:
00 lr saver 04 nop // will be modified to ftrace call ins when KPROBE is armed 08 insA 0c insBIn this case, if user asks for a probe to be placed at 'foo', we will choose
foo+0x04and from that point on, the behavior should reflect that a kprobe was placed atfoo+0x04. In particular, thepre_handler()should seefoo+0x04inpt_regs. Thepost_handler()would then seefoo+0x08.Quoted 1 line
Quoted 8 lines
Quoted 2 lines
later,
kprobe_ftrace_handler()will see pc points to 04, so pc + 4 will point to 08 the same as the one w/oKPROBE_ON_FTRACE.I didn't mean to compare regular trap/brk based kprobes with
KPROBES_ON_FTRACE. The only important aspect is that the handlers see consistentpt_regsin both cases, depending on where the kprobe was placed. Choosing a different address/offset to place a kprobe during its registration is an orthogonal aspect.
Indeed, previously, I want to let the PC point to the same instruction, it seems I misunderstood the "consistent" meaning.
Quoted 9 lines
Quoted 3 lines
It seems I need to fix the comment.
Given your explanation above, I think you can simply drop the first adjustment to the instruction pointer before the pre handler invocation. The rest of the code looks fine.
Yep, thanks a lot. Will send out a new version soon.
Hi,
Quoted 85 lines · On Thu, 22 Aug 2019 18:32:54 +0800 Jisheng Zhang <[email protected]>…
Quoted 48 lines · On Thu, 22 Aug 2019 15:52:05 +0530 "Naveen N. Rao"…
Quoted 5 lines · Jisheng Zhang
Hi,
Quoted 1 line · On Thu, 22 Aug 2019 12:23:58 +0530 "Naveen N. Rao"…
Jisheng Zhang wrote:
...
Quoted 39 lines
Quoted 19 lines
Quoted 19 lines
+/* Ftrace callback handler for kprobes -- called under preepmt disabed */
arch/arm64/kernel/probes/ftrace.c new file +17
-0+void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct pt_regs *regs) +{ +struct kprobe *p; +struct kprobe_ctlblk *kcb; + +/* Preempt is disabled by ftrace */ + p =get_kprobe ((kprobe_opcode_t *)ip); +if (unlikely (!p) ||kprobe_disabled (p)) +return ; + + kcb =get_kprobe_ctlblk (); +if (kprobe_running ()) { +kprobes_inc_nmissed_count (p); + }else { +unsigned long orig_ip =instruction_pointer (regs); +/* Kprobe handler expects regs->pc = pc + 4 as breakpoint hit */ Quoted 8 lines
Quoted 1 line
+
instruction_pointer_set(regs, ip + sizeof(kprobe_opcode_t));Just want to make sure that you've confirmed that this is what happens with a regular trap/brk based kprobe on ARM64. The reason for setting the instruction pointer here is to ensure that it is set to the same value as would be set if there was a trap/brk instruction at the ftrace location. This ensures that the kprobe pre handler sees the same value regardless.
Due to the arm64's
DYNAMIC_FTRACE_WITH_REGSimplementation, the code itself is correct. But this doesn't look like "there was a trap instruction at the ftrace location".W/O
KPROBE_ON_FTRACE:foo:
00 insA 04 insB 08 insCQuoted 5 lines
Quoted 1 line
kprobe's
pre_handler()will see pc points to 00.In this case, the probe will be placed at
foo+0x00, sopre_handler()seeing that address inpt_regsis correct behavior - as long as arm64 'brk' instruction causes an exception with the instruction pointer setYep, confirmed with regular trap/brk based kprobes, I do see PC set to the "brk" instruction.
Quoted 3 lines
*to* the 'brk' instruction. This is similar to how powerpc 'trap' works. However, x86 'int3' causes an exception *after* execution of the instruction.
Got it. I understand where's the comment "expects
regs->pc= pc + 1" from.Quoted 19 lines
Quoted 9 lines
W/
KPROBE_ON_FTRACE:foo:
00 lr saver 04 nop // will be modified to ftrace call ins when KPROBE is armed 08 insA 0c insBIn this case, if user asks for a probe to be placed at 'foo', we will choose
foo+0x04and from that point on, the behavior should reflect that a kprobe was placed atfoo+0x04. In particular, thepre_handler()should seefoo+0x04inpt_regs. Thepost_handler()would then seefoo+0x08.Quoted 3 lines
later,
kprobe_ftrace_handler()will see pc points to 04, so pc + 4 will point to 08 the same as the one w/oKPROBE_ON_FTRACE.
Quoted 15 lines
Quoted 5 lines
I didn't mean to compare regular trap/brk based kprobes with
KPROBES_ON_FTRACE. The only important aspect is that the handlers see consistentpt_regsin both cases, depending on where the kprobe was placed. Choosing a different address/offset to place a kprobe during its registration is an orthogonal aspect.Indeed, previously, I want to let the PC point to the same instruction, it seems I misunderstood the "consistent" meaning.
Quoted 6 lines
Quoted 3 lines
It seems I need to fix the comment.
Given your explanation above, I think you can simply drop the first adjustment to the instruction pointer before the pre handler invocation.
Just send out v5. But the first adjustment is modified as instruction_pointer_set(regs, ip);
Because in entry of kprobe_ftrace_handler() pc/ip(the first parameter) points to foo+0x4, while regs->pc points to foo+0x8. Based on your previous explanation, I think we should instruction_pointer_set(regs, ip) to let the pre_handler see foo+0x4
Thanks a lot for your help