kprobes: batch (un)optimization support
v3 cover kprobes: batch (un)optimization support
Hi,
Here is batch optimizing patch series version 3.
Since current kprobes jump optimization calls stop_machine() for each probe, it can make a lot latency noise when (un)registering a lot of probes (~1000) at once. For example, on 4 core Xeon, 256 probes optimization takes 770us in average (max 3.3ms).
This patch series introduces batch (un)optimization which modifies code with just one stop_machine(), and it improves optimization time to 90us in average (max 330us).
- Introduce text_poke_smp_batch() which modifies multiple codes with one stop_machine().
- Limit how many probes can be (un)optimized at once.
- Introduce delayed unoptimization for batch processing.
text_poke_smp_batch() also helps Jason's Jump label to reduce its overhead coming from text_poke_smp().
Changes in v3:
- Set kp.addr = NULL according to Ananth's comment.
Changes in v2:
- Add kprobes selftest bugfix patch.
- Add some comments about locks according to Mathieu's comment.
- Allocate working buffers when initializing kprobes, instead of using static arraies.
- Merge max optimization limit patch into batch optimizing patch.
Thank you,
---
Masami Hiramatsu (5):
kprobes: Support delayed unoptimization
kprobes/x86: Use text_poke_smp_batch
x86: Introduce text_poke_smp_batch() for batch-code modifying [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize [BUGFIX] kprobes: Fix selftest to clear flags field for reusing probes
arch/x86/include/asm/alternative.h | 7 + arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/alternative.c | 49 +++- arch/x86/kernel/kprobes.c | 114 +++++++++ include/linux/kprobes.h | 4 kernel/kprobes.c | 442 ++++++++++++++++++++++++------------ kernel/test_kprobes.c | 12 + 7 files changed, 464 insertions(+), 168 deletions(-)
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v3 1/5 kprobes: Fix selftest to clear flags field for reusing probes
Fix selftest to clear flags field for reusing probes because the flags field can be modified by Kprobes.
Changes in v3:
- Set kp.addr = NULL according to Ananth's comment.
---
kernel/test_kprobes.c | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-)
kernel/test_kprobes.c +9 -3
@@ -115,7 +115,9 @@ static int test_kprobes(void)int ret;struct kprobe *kps[2 ] = {&kp, &kp2}; - kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + kp.addr =NULL ; + kp.flags =0 ; ret =register_kprobes (kps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
@@ -210,7 +212,9 @@ static int test_jprobes(void)int ret;struct jprobe *jps[2 ] = {&jp, &jp2}; - jp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + jp.kp.addr =NULL ; + jp.kp.flags =0 ; ret =register_jprobes (jps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
@@ -323,7 +327,9 @@ static int test_kretprobes(void)int ret;struct kretprobe *rps[2 ] = {&rp, &rp2}; - rp.kp.addr =0 ;/* addr should be cleard for reusing kprobe. */ +/* addr and flags should be cleard for reusing kprobe. */ + rp.kp.addr =NULL ; + rp.kp.flags =0 ; ret =register_kretprobes (rps,2 );if (ret <0 ) { printk(KERN_ERR "Kprobe smoke test failed: "
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v3 2/5 kprobes: Remove redundant text_mutex lock in optimize
In reply to Masami HiramatsuRemove text_mutex locking in optimize_all_kprobes, because this function doesn't modify text. It simply queues probes on optimization list for kprobe_optimizer worker thread.
Changes in v2:
- Update changelog.
- Add comments for (un)optimize_all_kprobes.
- Update kprobe_mutex comment.
---
kernel/kprobes.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-)
kernel/kprobes.c +4 -3
@@ -73,7 +73,8 @@ static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];/* NOTE: change this value only with kprobe_mutex held */ static bool kprobes_all_disarmed; -static DEFINE_MUTEX (kprobe_mutex);/* Protects kprobe_table */ +/* This protects kprobe_table and optimizing_list */ +static DEFINE_MUTEX (kprobe_mutex); static DEFINE_PER_CPU(struct kprobe *, kprobe_instance) = NULL; static struct { spinlock_t lock ____cacheline_aligned_in_smp;
@@ -594,6 +595,7 @@ static __kprobes void try_to_optimize_kprobe(struct kprobe *p) }#ifdef CONFIG_SYSCTL +/* This should be called with kprobe_mutex locked */ static void __kprobes optimize_all_kprobes (void ) {struct hlist_head *head;
@@ -606,17 +608,16 @@ static void __kprobes optimize_all_kprobes(void)return ; kprobes_allow_optimization = true; -mutex_lock (&text_mutex);for (i =0 ; i <KPROBE_TABLE_SIZE ; i++) { head = &kprobe_table[i];hlist_for_each_entry_rcu (p, node, head, hlist)if (!kprobe_disabled (p))optimize_kprobe (p); } -mutex_unlock (&text_mutex);printk (KERN_INFO "Kprobes globally optimized\n" ); } +/* This should be called with kprobe_mutex locked */ static void __kprobes unoptimize_all_kprobes (void ) {struct hlist_head *head;
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v3 3/5 x86: Introduce text_poke_smp_batch() for batch-code modifying
In reply to Masami HiramatsuIntroduce text_poke_smp_batch(). This function modifies several text areas with one stop_machine() on SMPr. Because calling stop_machine() is heavy task, it is better to aggregate text_poke requests.
---
arch/x86/include/asm/alternative.h | 7 +++++ arch/x86/kernel/alternative.c | 49 +++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 9 deletions(-)
arch/x86/include/asm/alternative.h +7 -0
@@ -176,7 +176,14 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,* On the local CPU you need to be protected again NMI or MCE handlers seeing an * inconsistent instruction while you patch. */ +struct text_poke_param { +void *addr; +const void *opcode; +size_t len; +}; +extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +extern void text_poke_smp_batch (struct text_poke_param *params,int n);#endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +40 -9
78 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip v3 3/5] x86: Introduce text_poke_smp_batch() for batch-code modifying
Quoted 11 lines · Masami Hiramatsu
Introduce
text_poke_smp_batch(). This function modifies several text areas with onestop_machine()on SMPr. Because callingstop_machine()is heavy task, it is better to aggregatetext_pokerequests.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>
<snip>
Quoted 9 lines
arch/x86/kernel/alternative.c +9
-0+/** +* text_poke_smp_batch - Update instructions on a live kernel on SMP +* @params: an array of text_poke parameters +* @n: the number of elements in params. +* +* Modify multi-byte instruction by using stop_machine() on SMP. Since the +* stop_machine() is heavy task, it is better to aggregate text_poke requests +* and do it once if possible. +*
arch/x86/kernel/alternative.c static int __kprobes stop_machine_text_poke(void *data) · line 672
Quoted 10 lines
arch/x86/kernel/alternative.c +10
-0+* Note: Must be called under get_online_cpus() and text_mutex. +*/ +void __kprobes text_poke_smp_batch (struct text_poke_param *params,int n) +{ +struct text_poke_params tpp = {.params = params, .nparams = n}; + +atomic_set (&stop_machine_first,1 ); + wrote_text =0 ; +stop_machine (stop_machine_text_poke, (void *)&tpp,NULL ); +}
Looks good. But wouldn't it be even better to get stop_machine() be able to support batches itself?
We could have stop_machine_queue() and stop_machine_flush(), stop_machine() would be a shortcut for both, to execute single jobs, may be that could simplify some code here and there.
Re: [Dle-develop] [PATCH -tip v3 3/5] x86: Introduce text_poke_smp_batch() for batch-code modifying
Quoted 37 lines · Frederic Weisbecker
Quoted 11 lines · Masami Hiramatsu
Introduce
text_poke_smp_batch(). This function modifies several text areas with onestop_machine()on SMPr. Because callingstop_machine()is heavy task, it is better to aggregatetext_pokerequests.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]><snip>
Quoted 9 lines
arch/x86/kernel/alternative.c +9
-0+/** +* text_poke_smp_batch - Update instructions on a live kernel on SMP +* @params: an array of text_poke parameters +* @n: the number of elements in params. +* +* Modify multi-byte instruction by using stop_machine() on SMP. Since the +* stop_machine() is heavy task, it is better to aggregate text_poke requests +* and do it once if possible. +* arch/x86/kernel/alternative.c static int __kprobes stop_machine_text_poke(void *data) · line 672
Quoted 10 lines
arch/x86/kernel/alternative.c +10
-0+* Note: Must be called under get_online_cpus() and text_mutex. +*/ +void __kprobes text_poke_smp_batch (struct text_poke_param *params,int n) +{ +struct text_poke_params tpp = {.params = params, .nparams = n}; + +atomic_set (&stop_machine_first,1 ); + wrote_text =0 ; +stop_machine (stop_machine_text_poke, (void *)&tpp,NULL ); +}Looks good. But wouldn't it be even better to get
stop_machine()be able to support batches itself?
Quoted 4 lines
We could have
stop_machine_queue()andstop_machine_flush(),stop_machine()would be a shortcut for both, to execute single jobs, may be that could simplify some code here and there.
Ah, that's nice too :) But I don't know other people who need that interfaces. Would you know there are any other potential users?
Thank you,
PATCH v3 4/5 kprobes/x86: Use text_poke_smp_batch
In reply to Masami HiramatsuUse text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues. If the number of optimizing probes is more than MAX_OPTIMIZE_PROBES(=256), kprobes optimizes first MAX_OPTIMIZE_PROBES probes and kicks optimizer for remaining probes.
Changes in v2:
- Allocate code buffer and parameters in arch_init_kprobes() instead of using static arraies.
- Merge previous max optimization limit patch into this patch. So, this patch introduces upper limit of optimization at once.
---
arch/x86/kernel/kprobes.c | 69 ++++++++++++++++++++++++++++++++++++++++----- include/linux/kprobes.h | 2 + kernel/kprobes.c | 11 ++----- 3 files changed, 65 insertions(+), 17 deletions(-)
arch/x86/kernel/kprobes.c +61 -8
@@ -1385,10 +1385,16 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256 +static struct text_poke_param *jump_poke_params; +static struct jump_poke_buffer { +u8 buf[RELATIVEJUMP_SIZE ]; +} *jump_poke_bufs; + +static void __kprobes setup_optimize_kprobe (struct text_poke_param *tprm, +u8 *insn_buf, +struct optimized_kprobe *op) { -unsigned char jmp_code[RELATIVEJUMP_SIZE ];s32 rel = (s32)((long )op->optinsn.insn - ((long )op->kp.addr +RELATIVEJUMP_SIZE ));
@@ -1396,16 +1402,39 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)memcpy (op->optinsn.copied_insn, op->kp.addr +INT3_SIZE ,RELATIVE_ADDR_SIZE ); - jmp_code[0 ] =RELATIVEJUMP_OPCODE ; - *(s32 *)(&jmp_code[1 ]) = rel; + insn_buf[0 ] =RELATIVEJUMP_OPCODE ; + *(s32 *)(&insn_buf[1 ]) = rel; + + tprm->addr = op->kp.addr; + tprm->opcode = insn_buf; + tprm->len =RELATIVEJUMP_SIZE ; +} + +/* +* Replace breakpoints (int3) with relative jumps. +* Caller must call with locking kprobe_mutex. +*/ +void __kprobes arch_optimize_kprobes (struct list_head *oplist) +{ +struct optimized_kprobe *op, *tmp; +int c =0 ; + +list_for_each_entry_safe (op, tmp, oplist, list) { +WARN_ON (kprobe_disabled (&op->kp)); +/* Setup param */ +setup_optimize_kprobe (&jump_poke_params[c], + jump_poke_bufs[c].buf, op); +list_del_init (&op->list); +if (++c >=MAX_OPTIMIZE_PROBES ) +break ; + }/* * text_poke_smp doesn't support NMI/MCE code modifying. * However, since kprobes itself also doesn't support NMI/MCE * code probing, it's not a problem. */ -text_poke_smp (op->kp.addr, jmp_code,RELATIVEJUMP_SIZE ); -return 0 ; +text_poke_smp_batch (jump_poke_params, c); }/* Replace a relative jump with a breakpoint (int3). */
@@ -1437,11 +1466,35 @@ static int __kprobes setup_detour_execution(struct kprobe *p, }return 0 ; } + +static int __kprobes init_poke_params (void ) +{ +/* Allocate code buffer and parameter array */ + jump_poke_bufs =kmalloc (sizeof (struct jump_poke_buffer ) * +MAX_OPTIMIZE_PROBES ,GFP_KERNEL ); +if (!jump_poke_bufs) +return -ENOMEM ; + + jump_poke_params =kmalloc (sizeof (struct text_poke_param ) * +MAX_OPTIMIZE_PROBES ,GFP_KERNEL ); +if (!jump_poke_params) { +kfree (jump_poke_bufs); + jump_poke_bufs =NULL ; +return -ENOMEM ; + } + +return 0 ; +} +#else /* !CONFIG_OPTPROBES */ +static int __kprobes init_poke_params (void ) +{ +return 0 ; +}#endif int __init arch_init_kprobes (void ) { -return 0 ; +return init_poke_params (); }int __kprobes arch_trampoline_kprobe (struct kprobe *p)
include/linux/kprobes.h +1 -1
@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);extern int arch_check_optimized_kprobe (struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op); -extern int arch_optimize_kprobe (struct optimized_kprobe *op); +extern void arch_optimize_kprobes (struct list_head *oplist);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +3 -8
@@ -429,8 +429,6 @@ static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);/* Kprobe jump optimizer */ static __kprobes void kprobe_optimizer (struct work_struct *work) { -struct optimized_kprobe *op, *tmp; -/* Lock modules while optimizing kprobes */ mutex_lock (&module_mutex);mutex_lock (&kprobe_mutex);
@@ -458,14 +456,11 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)*/ get_online_cpus ();mutex_lock (&text_mutex); -list_for_each_entry_safe (op, tmp, &optimizing_list, list) { -WARN_ON (kprobe_disabled (&op->kp)); -if (arch_optimize_kprobe (op) <0 ) - op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; -list_del_init (&op->list); - } +arch_optimize_kprobes (&optimizing_list);mutex_unlock (&text_mutex);put_online_cpus (); +if (!list_empty (&optimizing_list)) +schedule_delayed_work (&optimizing_work,OPTIMIZE_DELAY ); end:mutex_unlock (&kprobe_mutex);mutex_unlock (&module_mutex);
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH v3 5/5 kprobes: Support delayed unoptimization
In reply to Masami HiramatsuUnoptimization occurs when a probe is unregistered or disabled, and is heavy because it recovers instructions by using stop_machine(). This patch delays unoptimization operations and unoptimize several probes at once by using text_poke_smp_batch().
This can avoid unexpected system slowdown coming from stop_machine().
Changes in v2:
- Use dynamic allocated buffers and params.
---
arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/kprobes.c | 45 ++++ include/linux/kprobes.h | 2 kernel/kprobes.c | 434 +++++++++++++++++++++++++++------------- 4 files changed, 344 insertions(+), 141 deletions(-)
arch/x86/include/asm/kprobes.h +2 -2
@@ -79,12 +79,12 @@ struct arch_specific_insn { };struct arch_optimized_insn { -/* copy of the original instructions */ -kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ];/* detour code buffer */ kprobe_opcode_t *insn;/* the size of instructions copied to detour code buffer */ size_t size; +/* copy of the original instructions */ +kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ]; };/* Return true (!0) if optinsn is prepared for optimization. */
arch/x86/kernel/kprobes.c +45 -0
57 lines not on this page · raw message
include/linux/kprobes.h +2 -0
@@ -276,6 +276,8 @@ extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op);extern void arch_optimize_kprobes (struct list_head *oplist); +extern void arch_unoptimize_kprobes (struct list_head *oplist, +struct list_head *done_list);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +295 -139
633 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]