kprobes: batch (un)optimization support
v2 cover kprobes: batch (un)optimization support
Hi,
Here is the 2nd version of batch optimizing patch series.
Since current kprobes jump optimization calls stop_machine() for each probe, it can make a lot latency noise when (un)registering a lot of probes (~1000) at once. For example, on 4 core Xeon, 256 probes optimization takes 770us in average (max 3.3ms).
This patch series introduces batch (un)optimization which modifies code with just one stop_machine(), and it improves optimization time to 90us in average (max 330us).
- Introduce text_poke_smp_batch() which modifies multiple codes with one stop_machine().
- Limit how many probes can be (un)optimized at once.
- Introduce delayed unoptimization for batch processing.
text_poke_smp_batch() also helps Jason's Jump label to reduce its overhead coming from text_poke_smp().
Changes in v2:
- Add kprobes selftest bugfix patch.
- Add some comments about locks according to Mathieu's comment.
- Allocate working buffers when initializing kprobes, instead of using static arraies.
- Merge max optimization limit patch into batch optimizing patch.
Thank you,
---
Masami Hiramatsu (5):
kprobes: Support delayed unoptimization
kprobes/x86: Use text_poke_smp_batch
x86: Introduce text_poke_smp_batch() for batch-code modifying [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize [BUGFIX] kprobes: Fix selftest to clear flags field for reusing probes
arch/x86/include/asm/alternative.h | 7 + arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/alternative.c | 49 +++- arch/x86/kernel/kprobes.c | 114 +++++++++ include/linux/kprobes.h | 4 kernel/kprobes.c | 442 ++++++++++++++++++++++++------------ kernel/test_kprobes.c | 12 + 7 files changed, 464 insertions(+), 168 deletions(-)
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v2 1/5 kprobes: Fix selftest to clear flags field for reusing probes
Fix selftest to clear flags field for reusing probes because the flags field can be modified by Kprobes.
---
kernel/test_kprobes.c | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-)
kernel/test_kprobes.c +9 -3
@@ -115,7 +115,9 @@ static int test_kprobes(void)int ret;struct kprobe *kps[2 ] = {&kp, &kp2}; - kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + kp.addr =0 ; + kp.flags =0 ; ret =register_kprobes (kps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
@@ -210,7 +212,9 @@ static int test_jprobes(void)int ret;struct jprobe *jps[2 ] = {&jp, &jp2}; - jp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + jp.kp.addr =0 ; + jp.kp.flags =0 ; ret =register_jprobes (jps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
@@ -323,7 +327,9 @@ static int test_kretprobes(void)int ret;struct kretprobe *rps[2 ] = {&rp, &rp2}; - rp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + rp.kp.addr =0 ; + rp.kp.flags =0 ; ret =register_kretprobes (rps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip v2 1/5] [BUGFIX] kprobes: Fix selftest to clear flags field for reusing probes
Quoted 8 lines · Masami Hiramatsu
kernel/test_kprobes.c | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-)kernel/test_kprobes.c +0
-0
kernel/test_kprobes.c static int test_kprobes(void) · line 119
Quoted 6 lines
kernel/test_kprobes.c +2
-1@@ -115,7 +115,9 @@ static int test_kprobes(void)int ret;struct kprobe *kps[2 ] = {&kp, &kp2}; - kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + kp.addr =0 ;
kp.addr = NULL;
kernel/test_kprobes.c static int test_jprobes(void) · line 216
Quoted 11 lines
kernel/test_kprobes.c +3
-1+ kp.flags =0 ; ret =register_kprobes (kps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "@@ -210,7 +212,9 @@ static int test_jprobes(void)int ret;struct jprobe *jps[2 ] = {&jp, &jp2}; - jp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + jp.kp.addr =0 ;
Same here...
kernel/test_kprobes.c static int test_kretprobes(void) · line 331
Quoted 11 lines
kernel/test_kprobes.c +3
-1+ jp.kp.flags =0 ; ret =register_jprobes (jps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "@@ -323,7 +327,9 @@ static int test_kretprobes(void)int ret;struct kretprobe *rps[2 ] = {&rp, &rp2}; - rp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + rp.kp.addr =0 ;
Here too.
Ananth
Re: [PATCH -tip v2 1/5] [BUGFIX] kprobes: Fix selftest to clear flags field for reusing probes
Quoted 13 lines · Ananth N Mavinakayanahalli
Quoted 11 lines · Masami Hiramatsu
kernel/test_kprobes.c | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-)kernel/test_kprobes.c +0
-0@@ -115,7 +115,9 @@ static int test_kprobes(void)int ret;struct kprobe *kps[2 ] = {&kp, &kp2};
Quoted 5 lines
kernel/test_kprobes.c static int test_kprobes(void) · line 119
Quoted 3 lines
kernel/test_kprobes.c +2
-1- kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + kp.addr =0 ;kp.addr = NULL;
OK, I'll fix that (and below too).
Thank you!
Quoted 35 lines
kernel/test_kprobes.c static int test_jprobes(void) · line 216
Quoted 12 lines
kernel/test_kprobes.c +3
-1+ kp.flags =0 ; ret =register_kprobes (kps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "@@ -210,7 +212,9 @@ static int test_jprobes(void)int ret;struct jprobe *jps[2 ] = {&jp, &jp2}; - jp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + jp.kp.addr =0 ;Same here...
kernel/test_kprobes.c static int test_kretprobes(void) · line 331
Quoted 11 lines
kernel/test_kprobes.c +3
-1+ jp.kp.flags =0 ; ret =register_jprobes (jps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "@@ -323,7 +327,9 @@ static int test_kretprobes(void)int ret;struct kretprobe *rps[2 ] = {&rp, &rp2}; - rp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + rp.kp.addr =0 ;Here too.
Ananth
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v2 2/5 kprobes: Remove redundant text_mutex lock in optimize
In reply to Masami HiramatsuRemove text_mutex locking in optimize_all_kprobes, because this function doesn't modify text. It simply queues probes on optimization list for kprobe_optimizer worker thread.
Changes in v2:
- Update changelog.
- Add comments for (un)optimize_all_kprobes.
- Update kprobe_mutex comment.
---
kernel/kprobes.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-)
kernel/kprobes.c +4 -3
@@ -73,7 +73,8 @@ static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];/* NOTE: change this value only with kprobe_mutex held */ static bool kprobes_all_disarmed; -static DEFINE_MUTEX (kprobe_mutex);/* Protects kprobe_table */ +/* This protects kprobe_table and optimizing_list */ +static DEFINE_MUTEX (kprobe_mutex); static DEFINE_PER_CPU(struct kprobe *, kprobe_instance) = NULL; static struct { spinlock_t lock ____cacheline_aligned_in_smp;
@@ -594,6 +595,7 @@ static __kprobes void try_to_optimize_kprobe(struct kprobe *p) }#ifdef CONFIG_SYSCTL +/* This should be called with kprobe_mutex locked */ static void __kprobes optimize_all_kprobes (void ) {struct hlist_head *head;
@@ -606,17 +608,16 @@ static void __kprobes optimize_all_kprobes(void)return ; kprobes_allow_optimization = true; -mutex_lock (&text_mutex);for (i =0 ; i <KPROBE_TABLE_SIZE ; i++) { head = &kprobe_table[i];hlist_for_each_entry_rcu (p, node, head, hlist)if (!kprobe_disabled (p))optimize_kprobe (p); } -mutex_unlock (&text_mutex);printk (KERN_INFO "Kprobes globally optimized\n" ); } +/* This should be called with kprobe_mutex locked */ static void __kprobes unoptimize_all_kprobes (void ) {struct hlist_head *head;
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v2 3/5 x86: Introduce text_poke_smp_batch() for batch-code modifying
In reply to Masami HiramatsuIntroduce text_poke_smp_batch(). This function modifies several text areas with one stop_machine() on SMPr. Because calling stop_machine() is heavy task, it is better to aggregate text_poke requests.
---
arch/x86/include/asm/alternative.h | 7 +++++ arch/x86/kernel/alternative.c | 49 +++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 9 deletions(-)
arch/x86/include/asm/alternative.h +7 -0
@@ -176,7 +176,14 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,* On the local CPU you need to be protected again NMI or MCE handlers seeing an * inconsistent instruction while you patch. */ +struct text_poke_param { +void *addr; +const void *opcode; +size_t len; +}; +extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +extern void text_poke_smp_batch (struct text_poke_param *params,int n);#endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +40 -9
78 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v2 4/5 kprobes/x86: Use text_poke_smp_batch
In reply to Masami HiramatsuUse text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues. If the number of optimizing probes is more than MAX_OPTIMIZE_PROBES(=256), kprobes optimizes first MAX_OPTIMIZE_PROBES probes and kicks optimizer for remaining probes.
Changes in v2:
- Allocate code buffer and parameters in arch_init_kprobes() instead of using static arraies.
- Merge previous max optimization limit patch into this patch. So, this patch introduces upper limit of optimization at once.
---
arch/x86/kernel/kprobes.c | 69 ++++++++++++++++++++++++++++++++++++++++----- include/linux/kprobes.h | 2 + kernel/kprobes.c | 11 ++----- 3 files changed, 65 insertions(+), 17 deletions(-)
arch/x86/kernel/kprobes.c +61 -8
@@ -1385,10 +1385,16 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256 +static struct text_poke_param *jump_poke_params; +static struct jump_poke_buffer { +u8 buf[RELATIVEJUMP_SIZE ]; +} *jump_poke_bufs; + +static void __kprobes setup_optimize_kprobe (struct text_poke_param *tprm, +u8 *insn_buf, +struct optimized_kprobe *op) { -unsigned char jmp_code[RELATIVEJUMP_SIZE ];s32 rel = (s32)((long )op->optinsn.insn - ((long )op->kp.addr +RELATIVEJUMP_SIZE ));
@@ -1396,16 +1402,39 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)memcpy (op->optinsn.copied_insn, op->kp.addr +INT3_SIZE ,RELATIVE_ADDR_SIZE ); - jmp_code[0 ] =RELATIVEJUMP_OPCODE ; - *(s32 *)(&jmp_code[1 ]) = rel; + insn_buf[0 ] =RELATIVEJUMP_OPCODE ; + *(s32 *)(&insn_buf[1 ]) = rel; + + tprm->addr = op->kp.addr; + tprm->opcode = insn_buf; + tprm->len =RELATIVEJUMP_SIZE ; +} + +/* +* Replace breakpoints (int3) with relative jumps. +* Caller must call with locking kprobe_mutex. +*/ +void __kprobes arch_optimize_kprobes (struct list_head *oplist) +{ +struct optimized_kprobe *op, *tmp; +int c =0 ; + +list_for_each_entry_safe (op, tmp, oplist, list) { +WARN_ON (kprobe_disabled (&op->kp)); +/* Setup param */ +setup_optimize_kprobe (&jump_poke_params[c], + jump_poke_bufs[c].buf, op); +list_del_init (&op->list); +if (++c >=MAX_OPTIMIZE_PROBES ) +break ; + }/* * text_poke_smp doesn't support NMI/MCE code modifying. * However, since kprobes itself also doesn't support NMI/MCE * code probing, it's not a problem. */ -text_poke_smp (op->kp.addr, jmp_code,RELATIVEJUMP_SIZE ); -return 0 ; +text_poke_smp_batch (jump_poke_params, c); }/* Replace a relative jump with a breakpoint (int3). */
@@ -1437,11 +1466,35 @@ static int __kprobes setup_detour_execution(struct kprobe *p, }return 0 ; } + +static int __kprobes init_poke_params (void ) +{ +/* Allocate code buffer and parameter array */ + jump_poke_bufs =kmalloc (sizeof (struct jump_poke_buffer ) * +MAX_OPTIMIZE_PROBES ,GFP_KERNEL ); +if (!jump_poke_bufs) +return -ENOMEM ; + + jump_poke_params =kmalloc (sizeof (struct text_poke_param ) * +MAX_OPTIMIZE_PROBES ,GFP_KERNEL ); +if (!jump_poke_params) { +kfree (jump_poke_bufs); + jump_poke_bufs =NULL ; +return -ENOMEM ; + } + +return 0 ; +} +#else /* !CONFIG_OPTPROBES */ +static int __kprobes init_poke_params (void ) +{ +return 0 ; +}#endif int __init arch_init_kprobes (void ) { -return 0 ; +return init_poke_params (); }int __kprobes arch_trampoline_kprobe (struct kprobe *p)
include/linux/kprobes.h +1 -1
@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);extern int arch_check_optimized_kprobe (struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op); -extern int arch_optimize_kprobe (struct optimized_kprobe *op); +extern void arch_optimize_kprobes (struct list_head *oplist);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +3 -8
@@ -429,8 +429,6 @@ static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);/* Kprobe jump optimizer */ static __kprobes void kprobe_optimizer (struct work_struct *work) { -struct optimized_kprobe *op, *tmp; -/* Lock modules while optimizing kprobes */ mutex_lock (&module_mutex);mutex_lock (&kprobe_mutex);
@@ -458,14 +456,11 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)*/ get_online_cpus ();mutex_lock (&text_mutex); -list_for_each_entry_safe (op, tmp, &optimizing_list, list) { -WARN_ON (kprobe_disabled (&op->kp)); -if (arch_optimize_kprobe (op) <0 ) - op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; -list_del_init (&op->list); - } +arch_optimize_kprobes (&optimizing_list);mutex_unlock (&text_mutex);put_online_cpus (); +if (!list_empty (&optimizing_list)) +schedule_delayed_work (&optimizing_work,OPTIMIZE_DELAY ); end:mutex_unlock (&kprobe_mutex);mutex_unlock (&module_mutex);
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v2 5/5 kprobes: Support delayed unoptimization
In reply to Masami HiramatsuUnoptimization occurs when a probe is unregistered or disabled, and is heavy because it recovers instructions by using stop_machine(). This patch delays unoptimization operations and unoptimize several probes at once by using text_poke_smp_batch().
This can avoid unexpected system slowdown coming from stop_machine().
Changes in v2:
- Use dynamic allocated buffers and params.
---
arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/kprobes.c | 45 ++++ include/linux/kprobes.h | 2 kernel/kprobes.c | 434 +++++++++++++++++++++++++++------------- 4 files changed, 344 insertions(+), 141 deletions(-)
arch/x86/include/asm/kprobes.h +2 -2
@@ -79,12 +79,12 @@ struct arch_specific_insn { };struct arch_optimized_insn { -/* copy of the original instructions */ -kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ];/* detour code buffer */ kprobe_opcode_t *insn;/* the size of instructions copied to detour code buffer */ size_t size; +/* copy of the original instructions */ +kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ]; };/* Return true (!0) if optinsn is prepared for optimization. */
arch/x86/kernel/kprobes.c +45 -0
57 lines not on this page · raw message
include/linux/kprobes.h +2 -0
@@ -276,6 +276,8 @@ extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op);extern void arch_optimize_kprobes (struct list_head *oplist); +extern void arch_unoptimize_kprobes (struct list_head *oplist, +struct list_head *done_list);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +295 -139
633 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]