Early kprobe: enable kprobes at very early
Subsystems (3)
cover Early kprobe: enable kprobes at very early
This patch series shows early kprobe, a mechanism allows users to track events at very early. It should be useful for optimization of system booting. This can also be used by BSP developers to hook their platform specific procedures at kernel booting stages after setup_arch().
This patch series provides X86 and ARM support for early kprobes. The ARM portion is based on my OPTPROBES for ARM 32 patches (ARM: kprobes: OPTPROBES and other improvements), which have not been accepted yet.
Kprobes is very useful for tracking events. However, it can only be used after system fully initialized. When debugging kernel booting stage, for example, checking memory consumption during booting, analyzing boot phase processes creation and optimization of booting speed, specific tools must be created. Sometimes we have to modify kernel code.
Early kprobes is my idea on it. By utilizing OPTPROBES which converts probed instructions into branches instead of breakpoints, kprobe can be used even before setup of exception handlers. By adding cmdline options, one can insert kprobes to track kernel booting stage without code modification.
BSP developers can also benefit from it. For example, when booting an SoC equipped with unstoppable watchdog like IMP706, wathdog writting code must be inserted into different places to avoid watchdog resetting system before watchdogd is pulled up (especially during memory initialization, which is the most time-consuming portion of booting). With early kprobe, BSP developers are able to put such code at their private directory without disturbing arch-independent code.
In this patch series, early kprobes simply print messagees when the probed instructions are hit. My futher plan is to connect 'ekprobe=' cmdline parameters to '/sys/kernel/debug/tracing/kprobe_events', allows installing kprobe events from kernel cmdline, and dump early kprobe messages into ring buffer without print them out.
Patch 1 - 4 are architecture dependent code, allow text modification before kprobes_initialized is setup, and alloc resources statically from vmlinux.lds. Currently only x86 and ARM are supported.
Patch 5 - 8 define required flags and macros.
Patch 9 is the core logic of early kprobes. When register_kprobe() is called before kprobes_initialized, it marks the probed kprobes as 'KPROBE_FLAG_EARLY' and allocs resources from slots which is reserved during linking. After kprobe is fully initialized, it converts early kprobes to normal kprobes.
Patch 10 enables cmdline option 'ekprobe=', allows setup probe at cmdline. However, currently the kprobe handler is only a simple printk.
Patch 11 introduces required Kconfig options to actually enable early kprobes.
Usage of early kprobe is as follow:
Booting kernel with cmdline 'ekprobe=', like:
... rdinit=/sbin/init ekprobe=0xc00f3c2c ekprobe=__free_pages ...
During boot, kernel will print trace using printk:
...
Hit early kprobe at __alloc_pages_nodemask+0x4
Hit early kprobe at __free_pages+0x0
Hit early kprobe at __alloc_pages_nodemask+0x4
Hit early kprobe at __free_pages+0x0
Hit early kprobe at __free_pages+0x0
Hit early kprobe at __alloc_pages_nodemask+0x4
...
After fully initialized, early kprobes will be converted to normal kprobes, and can be turned-off using:
echo 0 > /sys/kernel/debug/kprobes/enabled
And reenabled using:
echo 1 > /sys/kernel/debug/kprobes/enabled
Also, optimization can be turned off using:
echo 0 > /proc/sys/debug/kprobes-optimization
There's no way to remove specific early kprobe now. I'd like to convert early kprobes into kprobe events in futher patches, and then they can be totally removed through event interface.
Wang Nan (11):
ARM: kprobes: directly modify code if kprobe is not initialized.
ARM: kprobes: introduce early kprobes related code area.
x86: kprobes: directly modify code if kprobe is not initialized.
x86: kprobes: introduce early kprobes related code area.
kprobes: Add an KPROBE_FLAG_EARLY for early kprobe.
kprobes: makes kprobes_initialized globally visable.
kprobes: introduces macros for allocing early kprobe resources.
kprobes: allows __alloc_insn_slot() from early kprobes slots.
kprobes: core logic of eraly kprobes.
kprobes: enable 'ekprobe=' cmdline option for early kprobes.
kprobes: add CONFIG_EARLY_KPROBES option.
arch/Kconfig | 12 ++ arch/arm/include/asm/kprobes.h | 29 ++++- arch/arm/kernel/vmlinux.lds.S | 2 + arch/arm/probes/kprobes/opt-arm.c | 11 +- arch/x86/include/asm/insn.h | 7 +- arch/x86/include/asm/kprobes.h | 44 +++++-- arch/x86/kernel/kprobes/opt.c | 7 +- arch/x86/kernel/vmlinux.lds.S | 2 + include/linux/kprobes.h | 109 ++++++++++++++++++ kernel/kprobes.c | 237 ++++++++++++++++++++++++++++++++++++-- 10 files changed, 437 insertions(+), 23 deletions(-)
Signature · 1 line
1.8.4
RFC PATCH 1/11 ARM: kprobes: directly modify code if kprobe is not initialized.
If kprobe is optimized before kprobe is initialized, there should be only one core, the probed instruction is not armed with breakpoint, so simply patch text is okay.
--- arch/arm/probes/kprobes/opt-arm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/arm/probes/kprobes/opt-arm.c +10 -1
@@ -325,8 +325,17 @@ void __kprobes arch_optimize_kprobes(struct list_head *oplist)* Similar to __arch_disarm_kprobe, operations which * removing breakpoints must be wrapped by stop_machine * to avoid racing. +* +* If this function is called before kprobes initialized, +* the kprobe should be an early kprobe, the instruction +* is not armed with breakpoint. There should be only +* one core now, so directly __patch_text is enough. */ -kprobes_remove_breakpoint (op->kp.addr, insn); +if (unlikely (!kprobes_initialized)) { +BUG_ON (!(op->kp.flags &KPROBE_FLAG_EARLY )); +__patch_text (op->kp.addr, insn); + }else +kprobes_remove_breakpoint (op->kp.addr, insn);list_del_init (&op->list); }
Signature · 1 line
1.8.4
Re: [RFC PATCH 01/11] ARM: kprobes: directly modify code if kprobe is not initialized.
Quoted 19 lines · Wang Nan
If kprobe is optimized before kprobe is initialized, there should be only one core, the probed instruction is not armed with breakpoint, so simply patch text is okay.
Signed-off-by Wang Nan <[email protected]>--- arch/arm/probes/kprobes/opt-arm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/arm/probes/kprobes/opt-arm.c +2
-0@@ -325,8 +325,17 @@ void __kprobes arch_optimize_kprobes(struct list_head *oplist)* Similar to __arch_disarm_kprobe, operations which * removing breakpoints must be wrapped by stop_machine * to avoid racing. +* +* If this function is called before kprobes initialized,
arch/arm/probes/kprobes/opt-arm.c void __kprobes arch_optimize_kprobes(struct list_head *oplist) · line 338
Quoted 10 lines
arch/arm/probes/kprobes/opt-arm.c +8
-1+* the kprobe should be an early kprobe, the instruction +* is not armed with breakpoint. There should be only +* one core now, so directly __patch_text is enough. */ -kprobes_remove_breakpoint (op->kp.addr, insn); +if (unlikely (!kprobes_initialized)) { +BUG_ON (!(op->kp.flags &KPROBE_FLAG_EARLY )); +__patch_text (op->kp.addr, insn); + }else +kprobes_remove_breakpoint (op->kp.addr, insn);
"...if only one branch of a conditional statement is a single statement ... use braces in both branches".
arch/arm/probes/kprobes/opt-arm.c void __kprobes arch_optimize_kprobes(struct list_head *oplist)
Quoted 4 lines
arch/arm/probes/kprobes/opt-arm.c +0
-0list_del_init (&op->list); }
Regards,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Re: [RFC PATCH 01/11] ARM: kprobes: directly modify code if kprobe is not initialized.
In reply to Wang NanQuoted 3 lines · Wang Nan
If kprobe is optimized before kprobe is initialized, there should be only one core, the probed instruction is not armed with breakpoint, so simply patch text is okay.
This patch looks very hacky. If kprobes is not initialized, why anyone can optimize kprobes? I think you must introduce early kprobes init routine and set init flag at that point.
Thank you,
Quoted 30 lines
Signed-off-by Wang Nan <[email protected]>--- arch/arm/probes/kprobes/opt-arm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/arm/probes/kprobes/opt-arm.c +10
-1@@ -325,8 +325,17 @@ void __kprobes arch_optimize_kprobes(struct list_head *oplist)* Similar to __arch_disarm_kprobe, operations which * removing breakpoints must be wrapped by stop_machine * to avoid racing. +* +* If this function is called before kprobes initialized, +* the kprobe should be an early kprobe, the instruction +* is not armed with breakpoint. There should be only +* one core now, so directly __patch_text is enough. */ -kprobes_remove_breakpoint (op->kp.addr, insn); +if (unlikely (!kprobes_initialized)) { +BUG_ON (!(op->kp.flags &KPROBE_FLAG_EARLY )); +__patch_text (op->kp.addr, insn); + }else +kprobes_remove_breakpoint (op->kp.addr, insn);list_del_init (&op->list); }
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt at hitachi.com
RFC PATCH 2/11 ARM: kprobes: introduce early kprobes related code area.
In reply to Wang NanIn arm's vmlinux.lds, introduces code area inside text section. Executable area used by early kprobes will be allocated from there.
--- arch/arm/include/asm/kprobes.h | 29 +++++++++++++++++++++++++++-- arch/arm/kernel/vmlinux.lds.S | 2 ++ 2 files changed, 29 insertions(+), 2 deletions(-)
arch/arm/include/asm/kprobes.h +27 -2
@@ -17,16 +17,40 @@#define _ARM_KPROBES_H#include <linux/types.h> -#include <linux/ptrace.h> -#include <linux/notifier.h> #define __ARCH_WANT_KPROBES_INSN_SLOT#define MAX_INSN_SIZE 2 +#ifdef __ASSEMBLY__ + +#define KPROBE_OPCODE_SIZE 4 +#define MAX_OPTINSN_SIZE (optprobe_template_end - optprobe_template_entry) + +#ifdef CONFIG_EARLY_KPROBES +#define EARLY_KPROBES_CODES_AREA \ + . = ALIGN(8); \ + VMLINUX_SYMBOL(__early_kprobes_code_area_start) = .; \ + . = . + MAX_OPTINSN_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS; \ + VMLINUX_SYMBOL(__early_kprobes_code_area_end) = .; \ + . = ALIGN(8); \ + VMLINUX_SYMBOL(__early_kprobes_insn_slot_start) = .; \ + . = . + MAX_INSN_SIZE * KPROBE_OPCODE_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS;\ + VMLINUX_SYMBOL(__early_kprobes_insn_slot_end) = .; + +#else +#define EARLY_KPROBES_CODES_AREA +#endif + +#else + +#include <linux/ptrace.h> +#include <linux/notifier.h> +#define flush_insn_slot (p) do { } while (0)#define kretprobe_blacklist_size 0typedef u32 kprobe_opcode_t ; +#define KPROBE_OPCODE_SIZE sizeof(kprobe_opcode_t)struct kprobe ;#include <asm/probes.h>
@@ -83,4 +107,5 @@ struct arch_optimized_insn {*/ }; +#endif /* __ASSEMBLY__ */ #endif /* _ARM_KPROBES_H */
arch/arm/kernel/vmlinux.lds.S +2 -0
@@ -11,6 +11,7 @@ #ifdef CONFIG_ARM_KERNMEM_PERMS #include <asm/pgtable.h> #endif +#include <asm/kprobes.h> #define PROC_INFO \ . = ALIGN(4); \
@@ -108,6 +109,7 @@ SECTIONS SCHED_TEXT LOCK_TEXT KPROBES_TEXT + EARLY_KPROBES_CODES_AREA IDMAP_TEXT #ifdef CONFIG_MMU *(.fixup)
Signature · 1 line
1.8.4
RFC PATCH 3/11 x86: kprobes: directly modify code if kprobe is not initialized.
In reply to Wang NanWhen registering early kprobes, SMP should has not been enabled, so doesn't require synchronization in text_poke_bp(). Simply memcpy is enough.
--- arch/x86/kernel/kprobes/opt.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
arch/x86/kernel/kprobes/opt.c +5 -2
@@ -397,8 +397,11 @@ void arch_optimize_kprobes(struct list_head *oplist) insn_buf[0 ] =RELATIVEJUMP_OPCODE ; *(s32 *)(&insn_buf[1 ]) = rel; -text_poke_bp (op->kp.addr, insn_buf,RELATIVEJUMP_SIZE , - op->optinsn.insn); +if (unlikely (!kprobes_initialized)) +memcpy (op->kp.addr, insn_buf,RELATIVEJUMP_SIZE ); +else +text_poke_bp (op->kp.addr, insn_buf,RELATIVEJUMP_SIZE , + op->optinsn.insn);list_del_init (&op->list); }
Signature · 1 line
1.8.4
RFC PATCH 4/11 x86: kprobes: introduce early kprobes related code area.
In reply to Wang NanThis patch introduces EARLY_KPROBES_CODES_AREA in x86 vmlinux for early kprobes.
--- arch/x86/include/asm/insn.h | 7 ++++--- arch/x86/include/asm/kprobes.h | 45 ++++++++++++++++++++++++++++++++++-------- arch/x86/kernel/vmlinux.lds.S | 2 ++ 3 files changed, 43 insertions(+), 11 deletions(-)
arch/x86/include/asm/insn.h +4 -3
@@ -20,6 +20,9 @@* Copyright (C) IBM Corporation, 2009 */ +#define MAX_INSN_SIZE 16 + +#ifndef __ASSEMBLY__/* insn_attr_t is defined in inat.h */ #include <asm/inat.h>
@@ -69,8 +72,6 @@ struct insn {const insn_byte_t *next_byte; }; -#define MAX_INSN_SIZE 16 -#define X86_MODRM_MOD (modrm) (((modrm) & 0xc0) >> 6)#define X86_MODRM_REG (modrm) (((modrm) & 0x38) >> 3)#define X86_MODRM_RM (modrm) ((modrm) & 0x07)
@@ -197,5 +198,5 @@ static inline int insn_offset_immediate(struct insn *insn) {return insn_offset_displacement (insn) + insn->displacement.nbytes; } - +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_INSN_H */
arch/x86/include/asm/kprobes.h +37 -8
71 lines not on this page · raw message
arch/x86/kernel/vmlinux.lds.S +2 -0
@@ -26,6 +26,7 @@ #include <asm/page_types.h> #include <asm/cache.h> #include <asm/boot.h> +#include <asm/kprobes.h> #undef i386 /* in case the preprocessor is a 32bit one */
@@ -100,6 +101,7 @@ SECTIONS SCHED_TEXT LOCK_TEXT KPROBES_TEXT + EARLY_KPROBES_CODES_AREA ENTRY_TEXT IRQENTRY_TEXT *(.fixup)
Signature · 1 line
1.8.4
RFC PATCH 5/11 kprobes: Add an KPROBE_FLAG_EARLY for early kprobe.
In reply to Wang NanIntroduce a KPROBE_FLAG_EARLY for futher expansion. KPROBE_FLAG_EARLY indicates a kprobe is installed at very early stage, its resources should be allocated statically.
--- include/linux/kprobes.h | 1 + 1 file changed, 1 insertion(+)
include/linux/kprobes.h +1 -0
@@ -128,6 +128,7 @@ struct kprobe {* this flag is only for optimized_kprobe. */ #define KPROBE_FLAG_FTRACE 8/* probe is using ftrace */ +#define KPROBE_FLAG_EARLY 16/* early kprobe */ /* Has this kprobe gone ? */ static inline int kprobe_gone (struct kprobe *p)
Signature · 1 line
1.8.4
RFC PATCH 6/11 kprobes: makes kprobes_initialized globally visable.
In reply to Wang NanFollowing patches will enable kprobe registering very early, before kprobe system initialized. Arch code can use it to do special treatments for such kprobes.
--- include/linux/kprobes.h | 2 ++ kernel/kprobes.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/kprobes.h +2 -0
@@ -50,6 +50,8 @@#define KPROBE_REENTER 0x00000004#define KPROBE_HIT_SSDONE 0x00000008 +extern int kprobes_initialized; +#else /* CONFIG_KPROBES */ typedef int kprobe_opcode_t ; struct arch_specific_insn {
kernel/kprobes.c +1 -1
@@ -67,7 +67,7 @@ addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))#endif -static int kprobes_initialized; +int kprobes_initialized;static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE ];static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE ];
Signature · 1 line
1.8.4
RFC PATCH 7/11 kprobes: introduces macros for allocing early kprobe resources.
In reply to Wang NanIntroduces macros to genearte common early kprobe related resource allocator.
All early kprobe related resources are statically allocated during linking for each early kprobe slot. For each type of resource, a bitmap is used to track allocation. __DEFINE_EKPROBE_ALLOC_OPS defines alloc and free handler for them. The range of the resource and the bitmap should be provided for allocaing and freeing. DEFINE_EKPROBE_ALLOC_OPS defines bitmap and the array used by it.
--- include/linux/kprobes.h | 69 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)
include/linux/kprobes.h +69 -0
@@ -270,6 +270,75 @@ extern void show_registers(struct pt_regs *regs);extern void kprobes_inc_nmissed_count (struct kprobe *p);extern bool arch_within_kprobe_blacklist (unsigned long addr); +#ifdef CONFIG_EARLY_KPROBES + +#define NR_EARLY_KPROBES_SLOTS CONFIG_NR_EARLY_KPROBES_SLOTS +#define ALIGN_UP (v, a) (((v) + ((a) - 1)) & ~((a) - 1)) +#define EARLY_KPROBES_BITMAP_SZ ALIGN_UP(NR_EARLY_KPROBES_SLOTS, BITS_PER_LONG) + +#define __ek_in_range (v, s, e) (((v) >= (s)) && ((v) < (e))) +#define __ek_buf_sz (s, e) ((void *)(e) - (void *)(s)) +#define __ek_elem_sz_b (s, e) (__ek_buf_sz(s, e) / NR_EARLY_KPROBES_SLOTS) +#define __ek_elem_sz (s, e) (__ek_elem_sz_b(s, e) / sizeof(s[0])) +#define __ek_elem_idx (v, s, e) (__ek_buf_sz(s, v) / __ek_elem_sz_b(s, e)) +#define __ek_get_elem (i, s, e) (&((s)[__ek_elem_sz(s, e) * (i)])) +#define __DEFINE_EKPROBE_ALLOC_OPS (__t, __name) \ +static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\ +{ \ + int __i = find_next_zero_bit(__b, NR_EARLY_KPROBES_SLOTS, 0); \ + if (__i >= NR_EARLY_KPROBES_SLOTS) \ + return NULL; \ + set_bit(__i, __b); \ + return __ek_get_elem(__i, __s, __e); \ +} \ +static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b) \ +{ \ + if (!__ek_in_range(__v, __s, __e)) \ + return 0; \ + clear_bit(__ek_elem_idx(__v, __s, __e), __b); \ + return 1; \ +} + +#define DEFINE_EKPROBE_ALLOC_OPS (__t, __name, __static) \ +__static __t __ek_##__name##_slots[NR_EARLY_KPROBES_SLOTS]; \ +__static unsigned long __ek_##__name##_bitmap[EARLY_KPROBES_BITMAP_SZ]; \ +__DEFINE_EKPROBE_ALLOC_OPS(__t, __name) \ +static inline __t *ek_alloc_##__name(void) \ +{ \ + return __ek_alloc_##__name(&((__ek_##__name##_slots)[0]), \ + &((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\ + (__ek_##__name##_bitmap)); \ +} \ +static inline int ek_free_##__name(__t *__s) \ +{ \ + return __ek_free_##__name(__s, &((__ek_##__name##_slots)[0]), \ + &((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\ + (__ek_##__name##_bitmap)); \ +} + + +#else +#define __DEFINE_EKPROBE_ALLOC_OPS (__t, __name) \ +static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\ +{ \ + return NULL; \ +} \ +static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)\ +{ \ + return 0; \ +} + +#define DEFINE_EKPROBE_ALLOC_OPS (__t, __name, __static) \ +static inline __t *ek_alloc_##__name(void) \ +{ \ + return NULL; \ +} \ +static inline void ek_free_##__name(__t *__s) \ +{ \ +} + +#endif +struct kprobe_insn_cache {struct mutex mutex;void *(*alloc)(void );/* allocate insn page */
Signature · 1 line
1.8.4
Re: [RFC PATCH 07/11] kprobes: introduces macros for allocing early kprobe resources.
Quoted 80 lines · Wang Nan
Introduces macros to genearte common early kprobe related resource allocator.
All early kprobe related resources are statically allocated during linking for each early kprobe slot. For each type of resource, a bitmap is used to track allocation.
__DEFINE_EKPROBE_ALLOC_OPSdefines alloc and free handler for them. The range of the resource and the bitmap should be provided for allocaing and freeing.DEFINE_EKPROBE_ALLOC_OPSdefines bitmap and the array used by it.Signed-off-by Wang Nan <[email protected]>--- include/linux/kprobes.h | 69 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)include/linux/kprobes.h +57
-0@@ -270,6 +270,75 @@ extern void show_registers(struct pt_regs *regs);extern void kprobes_inc_nmissed_count (struct kprobe *p);extern bool arch_within_kprobe_blacklist (unsigned long addr); +#ifdef CONFIG_EARLY_KPROBES + +#define NR_EARLY_KPROBES_SLOTS CONFIG_NR_EARLY_KPROBES_SLOTS +#define ALIGN_UP (v, a) (((v) + ((a) - 1)) & ~((a) - 1)) +#define EARLY_KPROBES_BITMAP_SZ ALIGN_UP(NR_EARLY_KPROBES_SLOTS, BITS_PER_LONG) + +#define __ek_in_range (v, s, e) (((v) >= (s)) && ((v) < (e))) +#define __ek_buf_sz (s, e) ((void *)(e) - (void *)(s)) +#define __ek_elem_sz_b (s, e) (__ek_buf_sz(s, e) / NR_EARLY_KPROBES_SLOTS) +#define __ek_elem_sz (s, e) (__ek_elem_sz_b(s, e) / sizeof(s[0])) +#define __ek_elem_idx (v, s, e) (__ek_buf_sz(s, v) / __ek_elem_sz_b(s, e)) +#define __ek_get_elem (i, s, e) (&((s)[__ek_elem_sz(s, e) * (i)])) +#define __DEFINE_EKPROBE_ALLOC_OPS (__t, __name) \ +static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\ +{ \ + int __i = find_next_zero_bit(__b, NR_EARLY_KPROBES_SLOTS, 0); \ + if (__i >= NR_EARLY_KPROBES_SLOTS) \ + return NULL; \ + set_bit(__i, __b); \ + return __ek_get_elem(__i, __s, __e); \ +} \ +static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b) \ +{ \ + if (!__ek_in_range(__v, __s, __e)) \ + return 0; \ + clear_bit(__ek_elem_idx(__v, __s, __e), __b); \ + return 1; \ +} + +#define DEFINE_EKPROBE_ALLOC_OPS (__t, __name, __static) \ +__static __t __ek_##__name##_slots[NR_EARLY_KPROBES_SLOTS]; \ +__static unsigned long __ek_##__name##_bitmap[EARLY_KPROBES_BITMAP_SZ]; \ +__DEFINE_EKPROBE_ALLOC_OPS(__t, __name) \ +static inline __t *ek_alloc_##__name(void) \ +{ \ + return __ek_alloc_##__name(&((__ek_##__name##_slots)[0]), \ + &((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\ + (__ek_##__name##_bitmap)); \ +} \ +static inline int ek_free_##__name(__t *__s) \ +{ \ + return __ek_free_##__name(__s, &((__ek_##__name##_slots)[0]), \ + &((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\ + (__ek_##__name##_bitmap)); \ +} + + +#else +#define __DEFINE_EKPROBE_ALLOC_OPS (__t, __name) \ +static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\ +{ \ + return NULL; \ +} \ +static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)\ +{ \ + return 0; \ +}
include/linux/kprobes.h extern void show_registers(struct pt_regs *regs); · line 339
Quoted 10 lines
include/linux/kprobes.h +10
-0+ +#define DEFINE_EKPROBE_ALLOC_OPS (__t, __name, __static) \ +static inline __t *ek_alloc_##__name(void) \ +{ \ + return NULL; \ +} \ +static inline void ek_free_##__name(__t *__s) \ +{ \ +} +
Sorry, here is a small problem. Should be:
static inline int ek_free_##__name(__t *__s) \
{ \
return 0; \
}Quoted 6 lines
include/linux/kprobes.h +2
-0+#endif +struct kprobe_insn_cache {struct mutex mutex;void *(*alloc)(void );/* allocate insn page */
RFC PATCH 8/11 kprobes: allows __alloc_insn_slot() from early kprobes slots.
In reply to Wang NanIntroduces early_slots_start/end and bitmap for struct kprobe_insn_cache then uses previous introduced macro to generate allocator. This patch makes get/free_insn_slot() and get/free_optinsn_slot() transparent to early kprobes.
--- include/linux/kprobes.h | 33 +++++++++++++++++++++++++++++++++ kernel/kprobes.c | 14 ++++++++++++++ 2 files changed, 47 insertions(+)
include/linux/kprobes.h +33 -0
@@ -316,6 +316,10 @@ static inline int ek_free_##__name(__t *__s) \ (__ek_##__name##_bitmap)); \ } +extern kprobe_opcode_t __early_kprobes_code_area_start[]; +extern kprobe_opcode_t __early_kprobes_code_area_end[]; +extern kprobe_opcode_t __early_kprobes_insn_slot_start[]; +extern kprobe_opcode_t __early_kprobes_insn_slot_end[];#else #define __DEFINE_EKPROBE_ALLOC_OPS (__t, __name) \
@@ -339,6 +343,8 @@ static inline void ek_free_##__name(__t *__s) \#endif +__DEFINE_EKPROBE_ALLOC_OPS(kprobe_opcode_t, opcode) +struct kprobe_insn_cache {struct mutex mutex;void *(*alloc)(void );/* allocate insn page */
@@ -346,8 +352,35 @@ struct kprobe_insn_cache {struct list_head pages;/* list of kprobe_insn_page */ size_t insn_size;/* size of instruction slot */ int nr_garbage; +#ifdef CONFIG_EARLY_KPROBES +# define slots_start (c) ((c)->early_slots_start) +# define slots_end (c) ((c)->early_slots_end) +# define slots_bitmap (c) ((c)->early_slots_bitmap) +kprobe_opcode_t *early_slots_start; +kprobe_opcode_t *early_slots_end; +unsigned long early_slots_bitmap[EARLY_KPROBES_BITMAP_SZ ]; +#else +# define slots_start (c) NULL +# define slots_end (c) NULL +# define slots_bitmap (c) NULL +#endif }; +static inline kprobe_opcode_t * +__get_insn_slot_early (struct kprobe_insn_cache *c) +{ +return __ek_alloc_opcode (slots_start (c), +slots_end (c),slots_bitmap (c)); +} + +static inline int +__free_insn_slot_early (struct kprobe_insn_cache *c, +kprobe_opcode_t *slot) +{ +return __ek_free_opcode (slot,slots_start (c), +slots_end (c),slots_bitmap (c)); +} +extern kprobe_opcode_t *__get_insn_slot (struct kprobe_insn_cache *c);extern void __free_insn_slot (struct kprobe_insn_cache *c,kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +14 -0
@@ -137,6 +137,10 @@ struct kprobe_insn_cache kprobe_insn_slots = { .pages = LIST_HEAD_INIT(kprobe_insn_slots.pages), .insn_size = MAX_INSN_SIZE, .nr_garbage =0 , +#ifdef CONFIG_EARLY_KPROBES + .early_slots_start = __early_kprobes_insn_slot_start, + .early_slots_end = __early_kprobes_insn_slot_end, +#endif };static int collect_garbage_slots (struct kprobe_insn_cache *c);
@@ -149,6 +153,9 @@ kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c)struct kprobe_insn_page *kip;kprobe_opcode_t *slot =NULL ; +if (unlikely (!kprobes_initialized)) +return __get_insn_slot_early (c); +mutex_lock (&c->mutex); retry:list_for_each_entry (kip, &c->pages, list) {
@@ -249,6 +256,9 @@ void __free_insn_slot(struct kprobe_insn_cache *c, {struct kprobe_insn_page *kip; +if (unlikely (__free_insn_slot_early (c, slot))) +return ; +mutex_lock (&c->mutex);list_for_each_entry (kip, &c->pages, list) { long idx = ((long)slot - (long)kip->insns) /
@@ -280,6 +290,10 @@ struct kprobe_insn_cache kprobe_optinsn_slots = { .pages = LIST_HEAD_INIT(kprobe_optinsn_slots.pages),/* .insn_size is initialized later */ .nr_garbage =0 , +#ifdef CONFIG_EARLY_KPROBES + .early_slots_start = __early_kprobes_code_area_start, + .early_slots_end = __early_kprobes_code_area_end, +#endif };#endif #endif
Signature · 1 line
1.8.4
RFC PATCH 9/11 kprobes: core logic of eraly kprobes.
In reply to Wang NanThis patch is the main logic of early kprobe.
If register_kprobe() is called before kprobes_initialized, an early kprobe is allocated. Try to utilize existing OPTPROBE mechanism to replace the target instruction by a branch instead of breakpoint, because interrupt handlers may not been initialized yet.
All resources required by early kprobes are allocated statically. CONFIG_NR_EARLY_KPROBES_SLOTS is used to control number of possible early kprobes.
--- include/linux/kprobes.h | 4 ++ kernel/kprobes.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 149 insertions(+), 6 deletions(-)
include/linux/kprobes.h +4 -0
@@ -434,6 +434,10 @@ extern int proc_kprobes_optimization_handler(struct ctl_table *table, size_t *length, loff_t *ppos);#endif +struct early_kprobe_slot { +struct optimized_kprobe op; +}; +#endif /* CONFIG_OPTPROBES */ #ifdef CONFIG_KPROBES_ON_FTRACE extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
kernel/kprobes.c +145 -6
215 lines not on this page · raw message
Signature · 1 line
1.8.4
RFC PATCH 10/11 kprobes: enable 'ekprobe=' cmdline option for early kprobes.
In reply to Wang NanThis patch shows the basic idea of usage of early kprobes. By adding kernel cmdline options such as 'ekprobe=__alloc_pages_nodemask' or 'ekprobe=0xc00f3c2c', early kprobes are installed. When the probed instructions get hit, a message is printed.
--- kernel/kprobes.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+)
kernel/kprobes.c +71 -0
@@ -2611,8 +2611,79 @@ convert_early_kprobes(void)hlist_for_each_entry_safe (p, tmp, &early_kprobe_hlist, hlist)convert_early_kprobe (p); }; + +static int early_kprobe_pre_handler (struct kprobe *p,struct pt_regs *regs) +{ +const char *sym =NULL ; +char *modname, namebuf[KSYM_NAME_LEN ]; +unsigned long offset =0 ; + + sym =kallsyms_lookup ((unsigned long )p->addr,NULL , + &offset, &modname, namebuf); +if (sym) +pr_info ("Hit early kprobe at %s+0x%lx%s%s\n" , + sym, offset, + (modname ?" " :"" ), + (modname ? modname :"" )); +else +pr_info ("Hit early kprobe at %p\n" , p->addr); +return 0 ; +} + +DEFINE_EKPROBE_ALLOC_OPS(struct kprobe, early_kprobe_setup, static); +static int __init early_kprobe_setup (char *p) +{ +unsigned long long addr; +struct kprobe *kp; +int len =strlen (p); +int err; + +if (len <=0 ) { +pr_err ("early kprobe: wrong param: %s\n" , p); +return 0 ; + } + +if ((p[0 ] =='0' ) && (p[1 ] =='x' )) { + err =kstrtoull (p,16 , &addr); +if (err) { +pr_err ("early kprobe: wrong address: %p\n" , p); +return 0 ; + } + }else { + addr =kallsyms_lookup_name (p); +if (!addr) { +pr_err ("early kprobe: wrong symbol: %s\n" , p); +return 0 ; + } + } + +if ((addr < (unsigned long )_text) || + (addr >= (unsigned long )_etext)) +pr_err ("early kprobe: address of %p out of range\n" , p); + + kp =ek_alloc_early_kprobe_setup (); +if (kp ==NULL ) { +pr_err ("early kprobe: no enough early kprobe slot\n" ); +return 0 ; + } + kp->addr = (void *)(unsigned long )(addr); + kp->pre_handler = early_kprobe_pre_handler; + err =register_kprobe (kp); +if (err) { +pr_err ("early kprobe: register early kprobe %s failed\n" , p); +ek_free_early_kprobe_setup (kp); + } +return 0 ; +}#else static int register_early_kprobe (struct kprobe *p) {return -ENOSYS ; }static int ek_free_early_kprobe (struct early_kprobe_slot *slot) {return 0 ; }static void convert_early_kprobes (void ) {}; + +static int __init early_kprobe_setup (char *p) +{ +return 0 ; +}#endif + +early_param ("ekprobe" , early_kprobe_setup);
Signature · 1 line
1.8.4
RFC PATCH 11/11 kprobes: add CONFIG_EARLY_KPROBES option.
In reply to Wang NanEnable early kprobes in Kconfig.
--- arch/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/Kconfig +12 -0
@@ -46,6 +46,18 @@ config KPROBES for kernel debugging, non-intrusive instrumentation and testing. If in doubt, say "N". +config EARLY_KPROBES + depends on KPROBES && OPTPROBES + def_bool y + +config NR_EARLY_KPROBES_SLOTS + int "Number of possible early kprobes" + range 1 64 + default 16 + depends on EARLY_KPROBES + help + Number of early kprobes slots. + config JUMP_LABEL bool "Optimize very unlikely/likely branches" depends on HAVE_ARCH_JUMP_LABEL
Signature · 1 line
1.8.4
RFC PATCH 11/11 kprobes: add CONFIG_EARLY_KPROBES option.
In reply to Wang NanEnable early kprobes in Kconfig.
--- arch/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/Kconfig +12 -0
@@ -46,6 +46,18 @@ config KPROBES for kernel debugging, non-intrusive instrumentation and testing. If in doubt, say "N". +config EARLY_KPROBES + depends on KPROBES && OPTPROBES + def_bool y + +config NR_EARLY_KPROBES_SLOTS + int "Number of possible early kprobes" + range 1 64 + default 16 + depends on EARLY_KPROBES + help + Number of early kprobes slots. + config JUMP_LABEL bool "Optimize very unlikely/likely branches" depends on HAVE_ARCH_JUMP_LABEL
Signature · 1 line
1.8.4
Quoted 4 lines · Wang Nan
This patch series shows early kprobe, a mechanism allows users to track events at very early. It should be useful for optimization of system booting. This can also be used by BSP developers to hook their platform specific procedures at kernel booting stages after
setup_arch().
Good work!! :)
Quoted 10 lines
This patch series provides X86 and ARM support for early kprobes. The ARM portion is based on my OPTPROBES for ARM 32 patches (ARM: kprobes: OPTPROBES and other improvements), which have not been accepted yet.
Kprobes is very useful for tracking events. However, it can only be used after system fully initialized. When debugging kernel booting stage, for example, checking memory consumption during booting, analyzing boot phase processes creation and optimization of booting speed, specific tools must be created. Sometimes we have to modify kernel code.
Quoted 4 lines
Early kprobes is my idea on it. By utilizing OPTPROBES which converts probed instructions into branches instead of breakpoints, kprobe can be used even before setup of exception handlers. By adding cmdline options, one can insert kprobes to track kernel booting stage without code modification.
Hmm, for arm32, this strategy is good. but on x86, not so many instructions can be optimized. I doubt that we really need to use it before initializing exception handlers. Since any exception can be happen on early point, we need to initialize it on very early stage.
Quoted 8 lines
BSP developers can also benefit from it. For example, when booting an SoC equipped with unstoppable watchdog like IMP706, wathdog writting code must be inserted into different places to avoid watchdog resetting system before watchdogd is pulled up (especially during memory initialization, which is the most time-consuming portion of booting). With early kprobe, BSP developers are able to put such code at their private directory without disturbing arch-independent code.
Quoted 5 lines
In this patch series, early kprobes simply print messagees when the probed instructions are hit. My futher plan is to connect 'ekprobe=' cmdline parameters to '
/sys/kernel/debug/tracing/kprobe_events', allows installing kprobe events from kernel cmdline, and dump early kprobe messages into ring buffer without print them out.
Yeah, I really need this early-ftrace (event-trace) feature to trace booting kernel, even without kprobe events.
Quoted 12 lines
Patch 1 - 4 are architecture dependent code, allow text modification before
kprobes_initializedis setup, and alloc resources statically fromvmlinux.lds. Currently only x86 and ARM are supported.Patch 5 - 8 define required flags and macros.
Patch 9 is the core logic of early kprobes. When
register_kprobe()is called beforekprobes_initialized, it marks the probed kprobes as 'KPROBE_FLAG_EARLY' and allocs resources from slots which is reserved during linking. After kprobe is fully initialized, it converts early kprobes to normal kprobes.
Quoted 5 lines
Patch 10 enables cmdline option 'ekprobe=', allows setup probe at cmdline. However, currently the kprobe handler is only a simple printk.
Patch 11 introduces required
Kconfigoptions to actually enable early kprobes.
BTW, did you ensure all patches in the series are "bisect-clean" ? It seems some early patches in the series depend on later patches.
Quoted 18 lines
Usage of early kprobe is as follow:
Booting kernel with cmdline 'ekprobe=', like:
... rdinit=/sbin/init ekprobe=0xc00f3c2c ekprobe=__free_pages ...
During boot, kernel will print trace using printk:
...
Hit early kprobe at__alloc_pages_nodemask+0x4
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__alloc_pages_nodemask+0x4
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__alloc_pages_nodemask+0x4
...
Quoted 2 lines
After fully initialized, early kprobes will be converted to normal kprobes, and can be turned-off using:
I think it should be just removed automatically instead of converting.
Thank you!
Quoted 40 lines
echo 0 >
/sys/kernel/debug/kprobes/enabledAnd reenabled using:
echo 1 >
/sys/kernel/debug/kprobes/enabledAlso, optimization can be turned off using:
echo 0 >
/proc/sys/debug/kprobes-optimizationThere's no way to remove specific early kprobe now. I'd like to convert early kprobes into kprobe events in futher patches, and then they can be totally removed through event interface.
Wang Nan (11):
ARM: kprobes: directly modify code if kprobe is not initialized.
ARM: kprobes: introduce early kprobes related code area.
x86: kprobes: directly modify code if kprobe is not initialized.
x86: kprobes: introduce early kprobes related code area.
kprobes: Add anKPROBE_FLAG_EARLYfor early kprobe.
kprobes: makeskprobes_initializedglobally visable.
kprobes: introduces macros for allocing early kprobe resources.
kprobes: allows__alloc_insn_slot()from early kprobes slots.
kprobes: core logic of eraly kprobes.
kprobes: enable 'ekprobe=' cmdline option for early kprobes.
kprobes: addCONFIG_EARLY_KPROBESoption.arch/Kconfig | 12 ++ arch/arm/include/asm/kprobes.h | 29 ++++- arch/arm/kernel/vmlinux.lds.S | 2 + arch/arm/probes/kprobes/opt-arm.c | 11 +- arch/x86/include/asm/insn.h | 7 +- arch/x86/include/asm/kprobes.h | 44 +++++-- arch/x86/kernel/kprobes/opt.c | 7 +- arch/x86/kernel/vmlinux.lds.S | 2 + include/linux/kprobes.h | 109 ++++++++++++++++++ kernel/kprobes.c | 237 ++++++++++++++++++++++++++++++++++++-- 10 files changed, 437 insertions(+), 23 deletions(-)
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt at hitachi.com
RFC PATCH 1/1 x86: kprobes: enable optmize relative call insn
In reply to Masami Hiramatsu's question on my previous early kprobe patch series at:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-January/315771.html
that on x86, early kprobe's applications range is limited by the type of optimizable instructions, I made this patch, which enables optimizing relative call instructions by introducing specific template for them. Such instructions make up about 7% of the kernel. In addition, when ftrace is enabled, funtion entry will be it, so early kprobe will be much useful than before.
The relationship between ftrace and kprobe is interesting. Under normal circumstances, kprobe utilizes ftrace. However, under early case, there's no way to tell whether the probing instruction is an ftrace entry. Another possible method on that is to move part of ftrace init ahead. However, to allow optimize more instructions should also be good for performance.
Masami, I'd like to hear your reply on it. Do you think this patch is also useful for the normal cases?
--- arch/x86/include/asm/kprobes.h | 17 +++++++-- arch/x86/kernel/kprobes/opt.c | 82 ++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 5 deletions(-)
arch/x86/include/asm/kprobes.h +15 -2
@@ -31,6 +31,7 @@#define RELATIVEJUMP_OPCODE 0xe9#define RELATIVEJUMP_SIZE 5#define RELATIVECALL_OPCODE 0xe8 +#define RELATIVECALL_SIZE 5#define RELATIVE_ADDR_SIZE 4#define MAX_STACK_SIZE 64#define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE)
@@ -38,8 +39,10 @@#ifdef __ASSEMBLY__#define KPROBE_OPCODE_SIZE 1 +#define OPT_CALL_TEMPLATE_SIZE (optprobe_call_template_end - \ + optprobe_call_template_entry)#define MAX_OPTINSN_SIZE ((optprobe_template_end - optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + OPT_CALL_TEMPLATE_SIZE)#ifdef CONFIG_EARLY_KPROBES # define EARLY_KPROBES_CODES_AREA \
@@ -81,10 +84,20 @@ extern __visible kprobe_opcode_t optprobe_template_entry; extern __visible kprobe_opcode_t optprobe_template_val; extern __visible kprobe_opcode_t optprobe_template_call; extern __visible kprobe_opcode_t optprobe_template_end; + +extern __visible kprobe_opcode_t optprobe_call_template_entry; +extern __visible kprobe_opcode_t optprobe_call_template_val_destaddr; +extern __visible kprobe_opcode_t optprobe_call_template_val_retaddr; +extern __visible kprobe_opcode_t optprobe_call_template_end; + +#define OPT_CALL_TEMPLATE_SIZE \ + ((unsigned long)&optprobe_call_template_end - \ + (unsigned long)&optprobe_call_template_entry)#define MAX_OPTINSN_SIZE \ (((unsigned long)&optprobe_template_end - \ (unsigned long)&optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + \ + OPT_CALL_TEMPLATE_SIZE)extern const int kretprobe_blacklist_size;
arch/x86/kernel/kprobes/opt.c +79 -3
112 lines not on this page · raw message
Signature · 1 line
1.8.4
Re: [RFC PATCH] x86: kprobes: enable optmize relative call insn
Quoted 11 lines · Wang Nan
In reply to Masami Hiramatsu's question on my previous early kprobe patch series at:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-January/315771.html
that on x86, early kprobe's applications range is limited by the type of optimizable instructions, I made this patch, which enables optimizing relative call instructions by introducing specific template for them. Such instructions make up about 7% of the kernel. In addition, when ftrace is enabled, funtion entry will be it, so early kprobe will be much useful than before.
Sorry, I couldn't understand this part. If you put kprobe on ftrace site after ftrace enabled, it uses ftrace directly, instead of int3 or jump. Anyway, ftrace-site instruction should be controlled by ftrace, not kprobes.
Quoted 10 lines
The relationship between ftrace and kprobe is interesting. Under normal circumstances, kprobe utilizes ftrace. However, under early case, there's no way to tell whether the probing instruction is an ftrace entry. Another possible method on that is to move part of ftrace init ahead. However, to allow optimize more instructions should also be good for performance.
Masami, I'd like to hear your reply on it. Do you think this patch is also useful for the normal cases?
Expanding "optimizability" is good, but I don't like add new asm-templates which reduces maintainability less. Perhaps, we'd better start with reviewing can_boost table again...
Thank you,
Quoted 175 lines
Signed-off-by Wang Nan <[email protected]>--- arch/x86/include/asm/kprobes.h | 17 +++++++-- arch/x86/kernel/kprobes/opt.c | 82 ++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 5 deletions(-)arch/x86/include/asm/kprobes.h +15
-2@@ -31,6 +31,7 @@#define RELATIVEJUMP_OPCODE 0xe9#define RELATIVEJUMP_SIZE 5#define RELATIVECALL_OPCODE 0xe8 +#define RELATIVECALL_SIZE 5#define RELATIVE_ADDR_SIZE 4#define MAX_STACK_SIZE 64#define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE)@@ -38,8 +39,10 @@#ifdef __ASSEMBLY__#define KPROBE_OPCODE_SIZE 1 +#define OPT_CALL_TEMPLATE_SIZE (optprobe_call_template_end - \ + optprobe_call_template_entry)#define MAX_OPTINSN_SIZE ((optprobe_template_end - optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + OPT_CALL_TEMPLATE_SIZE)#ifdef CONFIG_EARLY_KPROBES # define EARLY_KPROBES_CODES_AREA \@@ -81,10 +84,20 @@ extern __visible kprobe_opcode_t optprobe_template_entry; extern __visible kprobe_opcode_t optprobe_template_val; extern __visible kprobe_opcode_t optprobe_template_call; extern __visible kprobe_opcode_t optprobe_template_end; + +extern __visible kprobe_opcode_t optprobe_call_template_entry; +extern __visible kprobe_opcode_t optprobe_call_template_val_destaddr; +extern __visible kprobe_opcode_t optprobe_call_template_val_retaddr; +extern __visible kprobe_opcode_t optprobe_call_template_end; + +#define OPT_CALL_TEMPLATE_SIZE \ + ((unsigned long)&optprobe_call_template_end - \ + (unsigned long)&optprobe_call_template_entry)#define MAX_OPTINSN_SIZE \ (((unsigned long)&optprobe_template_end - \ (unsigned long)&optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + \ + OPT_CALL_TEMPLATE_SIZE)extern const int kretprobe_blacklist_size;arch/x86/kernel/kprobes/opt.c +79
-3@@ -39,6 +39,23 @@#include "common.h" +static inline bool +is_relcall (u8 *addr) +{ +return (*(u8 *)(addr) ==RELATIVECALL_OPCODE ); +} + +static inline void * +get_relcall_target (u8 *addr) +{ +struct __arch_relative_insn { +u8 op; +s32 raddr; + } __packed *insn; + insn = (struct __arch_relative_insn *)addr; +return (void *)((unsigned long )addr +RELATIVECALL_SIZE + insn->raddr); +} +unsigned long __recover_optprobed_insn (kprobe_opcode_t *buf,unsigned long addr) {struct optimized_kprobe *op;@@ -89,6 +106,48 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val) } asm ( +#ifdef CONFIG_X86_64 +".global optprobe_call_template_entry\n" +"optprobe_call_template_entry:" +"pushq %rdi\n" +".global optprobe_call_template_val_destaddr\n" +"optprobe_call_template_val_destaddr:" +ASM_NOP5 +ASM_NOP5 +"pushq %rdi\n" +".global optprobe_call_template_val_retaddr\n" +"optprobe_call_template_val_retaddr:" +ASM_NOP5 +ASM_NOP5 +"xchgq %rdi, 8(%rsp)\n" + "retq\n" +#else /* CONFIG_X86_32 */ +".global optprobe_call_template_entry\n" +"optprobe_call_template_entry:" +"push %edi\n" +".global optprobe_call_template_val_destaddr\n" +"optprobe_call_template_val_destaddr:" +ASM_NOP5 +"push %edi\n" +".global optprobe_call_template_val_retaddr\n" +"optprobe_call_template_val_retaddr:" +ASM_NOP5 +"xchg %edi, 4(%esp)\n" + "ret\n" +#endif +".global optprobe_call_template_end\n" +"optprobe_call_template_end:\n" +); + +#define __OPTCALL_TMPL_MOVE_DESTADDR_IDX \ + ((long)&optprobe_call_template_val_destaddr - (long)&optprobe_call_template_entry) +#define __OPTCALL_TMPL_MOVE_RETADDR_IDX \ + ((long)&optprobe_call_template_val_retaddr - (long)&optprobe_call_template_entry) +#define __OPTCALL_TMPL_END_IDX \ + ((long)&optprobe_call_template_end - (long)&optprobe_call_template_entry) +#define OPTCALL_TMPL_SIZE __OPTCALL_TMPL_END_IDX + +asm ( ".global optprobe_template_entry\n" "optprobe_template_entry:\n"#ifdef CONFIG_X86_64 @@ -135,6 +194,10 @@ asm (#define TMPL_END_IDX \ ((long)&optprobe_template_end - (long)&optprobe_template_entry) +#define TMPL_OPTCALL_MOVE_DESTADDR_IDX (TMPL_END_IDX + __OPTCALL_TMPL_MOVE_DESTADDR_IDX) +#define TMPL_OPTCALL_MOVE_RETADDR_IDX (TMPL_END_IDX + __OPTCALL_TMPL_MOVE_RETADDR_IDX) +#define TMPL_OPTCALL_END_IDX (TMPL_END_IDX + __OPTCALL_TMPL_END_IDX) +#define INT3_SIZE sizeof(kprobe_opcode_t)/* Optimized kprobe call back function: called from optinsn */ @@ -175,6 +238,12 @@ static int copy_optimized_instructions(u8 *dest, u8 *src) {int len =0 , ret; +if (is_relcall (src)) { +memcpy (dest, &optprobe_call_template_entry, +OPTCALL_TMPL_SIZE ); +return OPTCALL_TMPL_SIZE ; + } +while (len <RELATIVEJUMP_SIZE ) { ret =__copy_instruction (dest + len, src + len);if (!ret || !can_boost (dest + len))@@ -365,9 +434,16 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,/* Set probe function call */ synthesize_relcall (buf +TMPL_CALL_IDX , optimized_callback); -/* Set returning jmp instruction at the tail of out-of-line buffer */ -synthesize_reljump (buf +TMPL_END_IDX + op->optinsn.size, - (u8 *)op->kp.addr + op->optinsn.size); +if (!is_relcall (op->kp.addr)) { +/* Set returning jmp instruction at the tail of out-of-line buffer */ +synthesize_reljump (buf +TMPL_END_IDX + op->optinsn.size, + (u8 *)op->kp.addr + op->optinsn.size); + }else { +synthesize_set_arg1 (buf +TMPL_OPTCALL_MOVE_DESTADDR_IDX , + (unsigned long )(get_relcall_target (op->kp.addr))); +synthesize_set_arg1 (buf +TMPL_OPTCALL_MOVE_RETADDR_IDX , + (unsigned long )(op->kp.addr +RELATIVECALL_SIZE )); + } flush_icache_range((unsigned long) buf, (unsigned long) buf + TMPL_END_IDX +
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt at hitachi.com
Re: [RFC PATCH] x86: kprobes: enable optmize relative call insn
Quoted 13 lines · Masami Hiramatsu
Quoted 11 lines · Wang Nan
In reply to Masami Hiramatsu's question on my previous early kprobe patch series at:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-January/315771.html
that on x86, early kprobe's applications range is limited by the type of optimizable instructions, I made this patch, which enables optimizing relative call instructions by introducing specific template for them. Such instructions make up about 7% of the kernel. In addition, when ftrace is enabled, funtion entry will be it, so early kprobe will be much useful than before.
Quoted 4 lines
Sorry, I couldn't understand this part. If you put kprobe on ftrace site after ftrace enabled, it uses ftrace directly, instead of int3 or jump. Anyway, ftrace-site instruction should be controlled by ftrace, not kprobes.
I agree with you. Let ftrace control ftrace things.
The goal of this patch is to expand early kprobe's use. I posted early kprobe patch series serval weeks before.
Early kprobe relies on kprobeopt: it directly optimizes probed instruction to avoid exception, so we can use it before exception handler installed, and definitely before ftrace_init() (that function converts function entries to NOP from CALL when ftrace is on).
As you pointed out, on x86, this strategy is not good because not so many instructions can be optimized. In my opinion, the most intolerable thing is that we are unable to optimize 'call' instruction, so we are unable to probe function entries before ftrace_init().
After this patch my early kprobe is able to probe function entries. I thought you may allow 'CALL' to be a special case which deserve a new template for early kprobe. Anyway, thank you for your quick response so I won't spent my time on it.
My next plan is to split ftrace_init() into two parts: converting 'CALL' to 'NOP' before early kprobe and doing other thing at its current place.
Quoted 194 lines
Quoted 10 lines
The relationship between ftrace and kprobe is interesting. Under normal circumstances, kprobe utilizes ftrace. However, under early case, there's no way to tell whether the probing instruction is an ftrace entry. Another possible method on that is to move part of ftrace init ahead. However, to allow optimize more instructions should also be good for performance.
Masami, I'd like to hear your reply on it. Do you think this patch is also useful for the normal cases?
Expanding "optimizability" is good, but I don't like add new asm-templates which reduces maintainability less. Perhaps, we'd better start with reviewing
can_boosttable again...Thank you,
Quoted 175 lines
Signed-off-by Wang Nan <[email protected]>--- arch/x86/include/asm/kprobes.h | 17 +++++++-- arch/x86/kernel/kprobes/opt.c | 82 ++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 5 deletions(-)arch/x86/include/asm/kprobes.h +15
-2@@ -31,6 +31,7 @@#define RELATIVEJUMP_OPCODE 0xe9#define RELATIVEJUMP_SIZE 5#define RELATIVECALL_OPCODE 0xe8 +#define RELATIVECALL_SIZE 5#define RELATIVE_ADDR_SIZE 4#define MAX_STACK_SIZE 64#define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE)@@ -38,8 +39,10 @@#ifdef __ASSEMBLY__#define KPROBE_OPCODE_SIZE 1 +#define OPT_CALL_TEMPLATE_SIZE (optprobe_call_template_end - \ + optprobe_call_template_entry)#define MAX_OPTINSN_SIZE ((optprobe_template_end - optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + OPT_CALL_TEMPLATE_SIZE)#ifdef CONFIG_EARLY_KPROBES # define EARLY_KPROBES_CODES_AREA \@@ -81,10 +84,20 @@ extern __visible kprobe_opcode_t optprobe_template_entry; extern __visible kprobe_opcode_t optprobe_template_val; extern __visible kprobe_opcode_t optprobe_template_call; extern __visible kprobe_opcode_t optprobe_template_end; + +extern __visible kprobe_opcode_t optprobe_call_template_entry; +extern __visible kprobe_opcode_t optprobe_call_template_val_destaddr; +extern __visible kprobe_opcode_t optprobe_call_template_val_retaddr; +extern __visible kprobe_opcode_t optprobe_call_template_end; + +#define OPT_CALL_TEMPLATE_SIZE \ + ((unsigned long)&optprobe_call_template_end - \ + (unsigned long)&optprobe_call_template_entry)#define MAX_OPTINSN_SIZE \ (((unsigned long)&optprobe_template_end - \ (unsigned long)&optprobe_template_entry) + \ - MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE) + MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE + \ + OPT_CALL_TEMPLATE_SIZE)extern const int kretprobe_blacklist_size;arch/x86/kernel/kprobes/opt.c +79
-3@@ -39,6 +39,23 @@#include "common.h" +static inline bool +is_relcall (u8 *addr) +{ +return (*(u8 *)(addr) ==RELATIVECALL_OPCODE ); +} + +static inline void * +get_relcall_target (u8 *addr) +{ +struct __arch_relative_insn { +u8 op; +s32 raddr; + } __packed *insn; + insn = (struct __arch_relative_insn *)addr; +return (void *)((unsigned long )addr +RELATIVECALL_SIZE + insn->raddr); +} +unsigned long __recover_optprobed_insn (kprobe_opcode_t *buf,unsigned long addr) {struct optimized_kprobe *op;@@ -89,6 +106,48 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val) } asm ( +#ifdef CONFIG_X86_64 +".global optprobe_call_template_entry\n" +"optprobe_call_template_entry:" +"pushq %rdi\n" +".global optprobe_call_template_val_destaddr\n" +"optprobe_call_template_val_destaddr:" +ASM_NOP5 +ASM_NOP5 +"pushq %rdi\n" +".global optprobe_call_template_val_retaddr\n" +"optprobe_call_template_val_retaddr:" +ASM_NOP5 +ASM_NOP5 +"xchgq %rdi, 8(%rsp)\n" + "retq\n" +#else /* CONFIG_X86_32 */ +".global optprobe_call_template_entry\n" +"optprobe_call_template_entry:" +"push %edi\n" +".global optprobe_call_template_val_destaddr\n" +"optprobe_call_template_val_destaddr:" +ASM_NOP5 +"push %edi\n" +".global optprobe_call_template_val_retaddr\n" +"optprobe_call_template_val_retaddr:" +ASM_NOP5 +"xchg %edi, 4(%esp)\n" + "ret\n" +#endif +".global optprobe_call_template_end\n" +"optprobe_call_template_end:\n" +); + +#define __OPTCALL_TMPL_MOVE_DESTADDR_IDX \ + ((long)&optprobe_call_template_val_destaddr - (long)&optprobe_call_template_entry) +#define __OPTCALL_TMPL_MOVE_RETADDR_IDX \ + ((long)&optprobe_call_template_val_retaddr - (long)&optprobe_call_template_entry) +#define __OPTCALL_TMPL_END_IDX \ + ((long)&optprobe_call_template_end - (long)&optprobe_call_template_entry) +#define OPTCALL_TMPL_SIZE __OPTCALL_TMPL_END_IDX + +asm ( ".global optprobe_template_entry\n" "optprobe_template_entry:\n"#ifdef CONFIG_X86_64 @@ -135,6 +194,10 @@ asm (#define TMPL_END_IDX \ ((long)&optprobe_template_end - (long)&optprobe_template_entry) +#define TMPL_OPTCALL_MOVE_DESTADDR_IDX (TMPL_END_IDX + __OPTCALL_TMPL_MOVE_DESTADDR_IDX) +#define TMPL_OPTCALL_MOVE_RETADDR_IDX (TMPL_END_IDX + __OPTCALL_TMPL_MOVE_RETADDR_IDX) +#define TMPL_OPTCALL_END_IDX (TMPL_END_IDX + __OPTCALL_TMPL_END_IDX) +#define INT3_SIZE sizeof(kprobe_opcode_t)/* Optimized kprobe call back function: called from optinsn */ @@ -175,6 +238,12 @@ static int copy_optimized_instructions(u8 *dest, u8 *src) {int len =0 , ret; +if (is_relcall (src)) { +memcpy (dest, &optprobe_call_template_entry, +OPTCALL_TMPL_SIZE ); +return OPTCALL_TMPL_SIZE ; + } +while (len <RELATIVEJUMP_SIZE ) { ret =__copy_instruction (dest + len, src + len);if (!ret || !can_boost (dest + len))@@ -365,9 +434,16 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,/* Set probe function call */ synthesize_relcall (buf +TMPL_CALL_IDX , optimized_callback); -/* Set returning jmp instruction at the tail of out-of-line buffer */ -synthesize_reljump (buf +TMPL_END_IDX + op->optinsn.size, - (u8 *)op->kp.addr + op->optinsn.size); +if (!is_relcall (op->kp.addr)) { +/* Set returning jmp instruction at the tail of out-of-line buffer */ +synthesize_reljump (buf +TMPL_END_IDX + op->optinsn.size, + (u8 *)op->kp.addr + op->optinsn.size); + }else { +synthesize_set_arg1 (buf +TMPL_OPTCALL_MOVE_DESTADDR_IDX , + (unsigned long )(get_relcall_target (op->kp.addr))); +synthesize_set_arg1 (buf +TMPL_OPTCALL_MOVE_RETADDR_IDX , + (unsigned long )(op->kp.addr +RELATIVECALL_SIZE )); + } flush_icache_range((unsigned long) buf, (unsigned long) buf + TMPL_END_IDX +
Quoted 4 lines · Wang Nan
Patch 10 enables cmdline option 'ekprobe=', allows setup probe at cmdline. However, currently the kprobe handler is only a simple printk.
Quoted 8 lines
Patch 11 introduces required
Kconfigoptions to actually enable early kprobes.Usage of early kprobe is as follow:
Booting kernel with cmdline 'ekprobe=', like:
... rdinit=/sbin/init ekprobe=0xc00f3c2c ekprobe=__free_pages ...
Perhaps you should specify what the probe will do. For now it is only printk. For example:
ekprobe=printk,__free_pages
That is, here you are attaching a printk to __free_pages.
Later, when you implement tracing, you could have:
ekprobe=trace,__free_pages
where it will be sent to the ring buffer.
This will maintain backward compatibility when you add new features. Instead of getting something like printk working now, and people use it for such, and then when you switch it over to tracing, it breaks the printk version.
Also, if you plan on converting early kprobes to normal kprobes during boot, then it should just be kprobes=..., why add the 'e'. It being early is just an implementation detail, not something that should be expressed by the users of the facility.
-- Steve
Quoted 11 lines
During boot, kernel will print trace using printk:
...
Hit early kprobe at__alloc_pages_nodemask+0x4
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__alloc_pages_nodemask+0x4
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__free_pages+0x0
Hit early kprobe at__alloc_pages_nodemask+0x4
...