kprobes: batch (un)optimization support
cover kprobes: batch (un)optimization support
Hi,
Since current kprobes jump optimization calls stop_machine() for each probe, it will almost freeze machine when registering a lot of probes (~1000) at once.
For avoiding this issue, this patch series enhances kprobes and text_poke.
- Introduces text_poke_smp_batch() which modifies multiple codes with one stop_machine().
- Limits how many probes can be optimized at once.
- Introduce delayed unoptimization for batch processing.
text_poke_smp_batch() also helps Jason's Jump label to reduce its overhead coming from text_poke_smp().
Currently, this patch allocates working memory static, however, it is possible to allocate dynamically, because its not in a critical section. it will be done by the next step, or next version.
Thank you,
---
Masami Hiramatsu (5):
kprobes: Support delayed unoptimization
kprobes/x86: Use text_poke_smp_batch
x86: Introduce text_poke_smp_batch() for batch-code modifying
kprobes: Limit maximum number of optimization at once [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize
arch/x86/include/asm/alternative.h | 7 + arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/alternative.c | 49 +++- arch/x86/kernel/kprobes.c | 78 ++++++ include/linux/kprobes.h | 4 kernel/kprobes.c | 437 ++++++++++++++++++++++++------------ 6 files changed, 416 insertions(+), 163 deletions(-)
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH 1/5 kprobes: Remove redundant text_mutex lock in optimize
Remove text_mutex locking in optimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.
---
kernel/kprobes.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
kernel/kprobes.c +0 -2
@@ -606,14 +606,12 @@ static void __kprobes optimize_all_kprobes(void)return ; kprobes_allow_optimization = true; -mutex_lock (&text_mutex);for (i =0 ; i <KPROBE_TABLE_SIZE ; i++) { head = &kprobe_table[i];hlist_for_each_entry_rcu (p, node, head, hlist)if (!kprobe_disabled (p))optimize_kprobe (p); } -mutex_unlock (&text_mutex);printk (KERN_INFO "Kprobes globally optimized\n" ); }
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 1/5] [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize
Quoted 3 lines · * Masami Hiramatsu ([email protected]) wrote:
Remove
text_mutexlocking inoptimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.
Hi Masami,
A few comments:
1) optimize_all_kprobes/unoptimize_all_kprobes should have comments saying that they are always called with kprobe_mutex held.
2) The sentence above in the changelog could be changed into:
..."because this function doesn't modify text. It simply queues optimizations
for the kprobe_optimizer worker thread."
3)
static DEFINE_MUTEX(kprobe_mutex); /* Protects kprobe_table */
.. should also state that it protects optimizing_list.
Thanks,
Mathieu
Quoted 36 lines
Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
kernel/kprobes.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)kernel/kprobes.c +0
-2@@ -606,14 +606,12 @@ static void __kprobes optimize_all_kprobes(void)return ; kprobes_allow_optimization = true; -mutex_lock (&text_mutex);for (i =0 ; i <KPROBE_TABLE_SIZE ; i++) { head = &kprobe_table[i];hlist_for_each_entry_rcu (p, node, head, hlist)if (!kprobe_disabled (p))optimize_kprobe (p); } -mutex_unlock (&text_mutex);printk (KERN_INFO "Kprobes globally optimized\n" ); }Signature · 3 lines
-- Masami Hiramatsu e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Re: [PATCH -tip 1/5] [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize
Hi Mathieu,
Quoted 14 lines · Mathieu Desnoyers
Quoted 3 lines · * Masami Hiramatsu ([email protected]) wrote:
Remove
text_mutexlocking inoptimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.Hi Masami,
A few comments:
1) optimize_all_kprobes/unoptimize_all_kprobes should have comments saying that they are always called with
kprobe_mutexheld.2) The sentence above in the changelog could be changed into:
Quoted 9 lines
..."because this function doesn't modify text. It simply queues optimizations
for thekprobe_optimizerworker thread."3)
static
DEFINE_MUTEX(kprobe_mutex); /* Protectskprobe_table*/.. should also state that it protects
optimizing_list.
Thanks! all comments are good to me!
Thank you again,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH 2/5 kprobes: Limit maximum number of optimization at once
In reply to Masami HiramatsuSince text_poke_smp() uses stop_machine(), the machine almost stop a while if we optimize a lot of probes at once. This patch limits the maximum number of probes to optimize (means calling text_poke_smp()) at once, and try it again after a while if there are more probes waiting for optimization than the limitation.
---
kernel/kprobes.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/kprobes.c +6 -0
@@ -424,11 +424,13 @@ static LIST_HEAD(optimizing_list);static void kprobe_optimizer (struct work_struct *work); static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);#define OPTIMIZE_DELAY 5 +#define MAX_OPTIMIZE_PROBES 64/* Kprobe jump optimizer */ static __kprobes void kprobe_optimizer (struct work_struct *work) {struct optimized_kprobe *op, *tmp; +int c =0 ;/* Lock modules while optimizing kprobes */ mutex_lock (&module_mutex);
@@ -462,9 +464,13 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)if (arch_optimize_kprobe (op) <0 ) op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ;list_del_init (&op->list); +if (++c >=MAX_OPTIMIZE_PROBES ) +break ; }mutex_unlock (&text_mutex);put_online_cpus (); +if (!list_empty (&optimizing_list)) +schedule_delayed_work (&optimizing_work,OPTIMIZE_DELAY ); end:mutex_unlock (&kprobe_mutex);mutex_unlock (&module_mutex);
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH 3/5 x86: Introduce text_poke_smp_batch() for batch-code modifying
In reply to Masami HiramatsuIntroduce text_poke_smp_batch(). This function modifies several text areas with one stop_machine() on SMPr. Because calling stop_machine() is heavy task, it is better to aggregate text_poke requests.
---
arch/x86/include/asm/alternative.h | 7 +++++ arch/x86/kernel/alternative.c | 49 +++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 9 deletions(-)
arch/x86/include/asm/alternative.h +7 -0
@@ -176,7 +176,14 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,* On the local CPU you need to be protected again NMI or MCE handlers seeing an * inconsistent instruction while you patch. */ +struct text_poke_param { +void *addr; +const void *opcode; +size_t len; +}; +extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +extern void text_poke_smp_batch (struct text_poke_param *params,int n);#endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +40 -9
78 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
PATCH 4/5 kprobes/x86: Use text_poke_smp_batch
In reply to Masami HiramatsuUse text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.
---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +30 -7
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256 +static struct text_poke_param jump_params[MAX_OPTIMIZE_PROBES ]; +static char jump_code_buf[MAX_OPTIMIZE_PROBES ][RELATIVEJUMP_SIZE ]; + +static void __kprobes setup_optimize_kprobe (struct text_poke_param *tprm, +char *insn_buf, +struct optimized_kprobe *op) { -unsigned char jmp_code[RELATIVEJUMP_SIZE ];s32 rel = (s32)((long )op->optinsn.insn - ((long )op->kp.addr +RELATIVEJUMP_SIZE ));
@@ -1396,16 +1400,35 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)memcpy (op->optinsn.copied_insn, op->kp.addr +INT3_SIZE ,RELATIVE_ADDR_SIZE ); - jmp_code[0 ] =RELATIVEJUMP_OPCODE ; - *(s32 *)(&jmp_code[1 ]) = rel; + insn_buf[0 ] =RELATIVEJUMP_OPCODE ; + *(s32 *)(&insn_buf[1 ]) = rel; + + tprm->addr = op->kp.addr; + tprm->opcode = insn_buf; + tprm->len =RELATIVEJUMP_SIZE ; +} + +/* Replace a breakpoint (int3) with a relative jump. */ +void __kprobes arch_optimize_kprobes (struct list_head *oplist) +{ +struct optimized_kprobe *op, *tmp; +int c =0 ; + +list_for_each_entry_safe (op, tmp, oplist, list) { +WARN_ON (kprobe_disabled (&op->kp)); +/* Setup param */ +setup_optimize_kprobe (&jump_params[c], jump_code_buf[c], op); +list_del_init (&op->list); +if (++c >=MAX_OPTIMIZE_PROBES ) +break ; + }/* * text_poke_smp doesn't support NMI/MCE code modifying. * However, since kprobes itself also doesn't support NMI/MCE * code probing, it's not a problem. */ -text_poke_smp (op->kp.addr, jmp_code,RELATIVEJUMP_SIZE ); -return 0 ; +text_poke_smp_batch (jump_params, c); }/* Replace a relative jump with a breakpoint (int3). */
include/linux/kprobes.h +1 -1
@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);extern int arch_check_optimized_kprobe (struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op); -extern int arch_optimize_kprobe (struct optimized_kprobe *op); +extern void arch_optimize_kprobes (struct list_head *oplist);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +1 -12
@@ -424,14 +424,10 @@ static LIST_HEAD(optimizing_list);static void kprobe_optimizer (struct work_struct *work); static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);#define OPTIMIZE_DELAY 5 -#define MAX_OPTIMIZE_PROBES 64/* Kprobe jump optimizer */ static __kprobes void kprobe_optimizer (struct work_struct *work) { -struct optimized_kprobe *op, *tmp; -int c =0 ; -/* Lock modules while optimizing kprobes */ mutex_lock (&module_mutex);mutex_lock (&kprobe_mutex);
@@ -459,14 +455,7 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)*/ get_online_cpus ();mutex_lock (&text_mutex); -list_for_each_entry_safe (op, tmp, &optimizing_list, list) { -WARN_ON (kprobe_disabled (&op->kp)); -if (arch_optimize_kprobe (op) <0 ) - op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; -list_del_init (&op->list); -if (++c >=MAX_OPTIMIZE_PROBES ) -break ; - } +arch_optimize_kprobes (&optimizing_list);mutex_unlock (&text_mutex);put_online_cpus ();if (!list_empty (&optimizing_list))
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0
arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256
So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Thanks,
Mathieu
Quoted 104 lines
arch/x86/kernel/kprobes.c +29
-5+static struct text_poke_param jump_params[MAX_OPTIMIZE_PROBES ]; +static char jump_code_buf[MAX_OPTIMIZE_PROBES ][RELATIVEJUMP_SIZE ]; + +static void __kprobes setup_optimize_kprobe (struct text_poke_param *tprm, +char *insn_buf, +struct optimized_kprobe *op) { -unsigned char jmp_code[RELATIVEJUMP_SIZE ];s32 rel = (s32)((long )op->optinsn.insn - ((long )op->kp.addr +RELATIVEJUMP_SIZE ));@@ -1396,16 +1400,35 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)memcpy (op->optinsn.copied_insn, op->kp.addr +INT3_SIZE ,RELATIVE_ADDR_SIZE ); - jmp_code[0 ] =RELATIVEJUMP_OPCODE ; - *(s32 *)(&jmp_code[1 ]) = rel; + insn_buf[0 ] =RELATIVEJUMP_OPCODE ; + *(s32 *)(&insn_buf[1 ]) = rel; + + tprm->addr = op->kp.addr; + tprm->opcode = insn_buf; + tprm->len =RELATIVEJUMP_SIZE ; +} + +/* Replace a breakpoint (int3) with a relative jump. */ +void __kprobes arch_optimize_kprobes (struct list_head *oplist) +{ +struct optimized_kprobe *op, *tmp; +int c =0 ; + +list_for_each_entry_safe (op, tmp, oplist, list) { +WARN_ON (kprobe_disabled (&op->kp)); +/* Setup param */ +setup_optimize_kprobe (&jump_params[c], jump_code_buf[c], op); +list_del_init (&op->list); +if (++c >=MAX_OPTIMIZE_PROBES ) +break ; + }/* * text_poke_smp doesn't support NMI/MCE code modifying. * However, since kprobes itself also doesn't support NMI/MCE * code probing, it's not a problem. */ -text_poke_smp (op->kp.addr, jmp_code,RELATIVEJUMP_SIZE ); -return 0 ; +text_poke_smp_batch (jump_params, c); }/* Replace a relative jump with a breakpoint (int3). */ include/linux/kprobes.h +1
-1@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);extern int arch_check_optimized_kprobe (struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op); -extern int arch_optimize_kprobe (struct optimized_kprobe *op); +extern void arch_optimize_kprobes (struct list_head *oplist);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);kernel/kprobes.c +1
-12@@ -424,14 +424,10 @@ static LIST_HEAD(optimizing_list);static void kprobe_optimizer (struct work_struct *work); static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);#define OPTIMIZE_DELAY 5 -#define MAX_OPTIMIZE_PROBES 64/* Kprobe jump optimizer */ static __kprobes void kprobe_optimizer (struct work_struct *work) { -struct optimized_kprobe *op, *tmp; -int c =0 ; -/* Lock modules while optimizing kprobes */ mutex_lock (&module_mutex);mutex_lock (&kprobe_mutex);@@ -459,14 +455,7 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)*/ get_online_cpus ();mutex_lock (&text_mutex); -list_for_each_entry_safe (op, tmp, &optimizing_list, list) { -WARN_ON (kprobe_disabled (&op->kp)); -if (arch_optimize_kprobe (op) <0 ) - op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED ; -list_del_init (&op->list); -if (++c >=MAX_OPTIMIZE_PROBES ) -break ; - } +arch_optimize_kprobes (&optimizing_list);mutex_unlock (&text_mutex);put_online_cpus ();if (!list_empty (&optimizing_list))Signature · 3 lines
-- Masami Hiramatsu e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; }
Quoted 8 lines
arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 3 lines
arch/x86/kernel/kprobes.c +1
-2-/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
From my experiment on kvm/4cpu, it took about 3 seconds in average.
With this patch, it went down to 30ms. (x100 faster :))
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 26 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; }
Quoted 8 lines
Quoted 6 lines
arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 3 lines
arch/x86/kernel/kprobes.c +1
-2-/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
Thanks,
Mathieu
Quoted 6 lines
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Quoted 4 lines
Quoted 1 line
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to
stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?
Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.
Quoted 4 lines
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 47 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Quoted 4 lines
Quoted 1 line
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to
stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.Quoted 2 lines
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
Quoted 8 lines
Quoted 2 lines
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.
I think we are starting to see the stop_machine() approach is really limiting our ability to do even relatively small amount of work without hurting responsiveness significantly.
What's the current showstopper with the breakpoint-bypass-ipi approach that solves this issue properly and makes this batching approach unnecessary ?
Thanks,
Mathieu
Quoted 6 lines
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 57 lines · Mathieu Desnoyers
Quoted 47 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Quoted 4 lines
Quoted 1 line
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to
stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.Quoted 2 lines
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
Quoted 8 lines
Quoted 2 lines
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.
Quoted 6 lines
I think we are starting to see the
stop_machine()approach is really limiting our ability to do even relatively small amount of work without hurting responsiveness significantly.What's the current showstopper with the breakpoint-bypass-ipi approach that solves this issue properly and makes this batching approach unnecessary ?
We still do not have any official answer from chip vendors. As you know, basic implementation has been done.
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
In reply to Mathieu DesnoyersQuoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Quoted 4 lines
Quoted 1 line
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to
stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?
Sorry, my bad. Non tuned kvm guest is so slow...
I've tried to check it again on *bare machine* (4core Xeon 2.33GHz, 4cpu). I found that even without this patch, optimizing 256 probes took 770us in average (min 150us, max 3.3ms.)
With this patch, it went down to 90us in average (min 14us, max 324us!)
Isn't it enough low latency? :)
Quoted 4 lines
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
so, it just takes 90us. I hope it is acceptable.
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch
Quoted 46 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:
Use
text_poke_smp_batch()in optimization path for reducing the number ofstop_machine()issues.Signed-off-by Masami Hiramatsu <[email protected]>Cc Ananth N Mavinakayanahalli <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jim Keniston <[email protected]>Cc Jason Baron <[email protected]>Cc Mathieu Desnoyers <[email protected]>---
arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 13 +------------ 3 files changed, 32 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes.c +0
-0arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388
Quoted 6 lines
arch/x86/kernel/kprobes.c +1
-2@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)return 0 ; } -/* Replace a breakpoint (int3) with a relative jump. */ -int __kprobes arch_optimize_kprobe (struct optimized_kprobe *op) +#define MAX_OPTIMIZE_PROBES 256So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?
Quoted 4 lines
Quoted 1 line
From my experiment on kvm/4cpu, it took about 3 seconds in average.
That's 3 seconds for multiple calls to
stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?Sorry, my bad. Non tuned kvm guest is so slow...
I've tried to check it again on *bare machine* (4core Xeon 2.33GHz, 4cpu). I found that even without this patch, optimizing 256 probes took 770us in average (min 150us, max 3.3ms.)
With this patch, it went down to 90us in average (min 14us, max 324us!)
Quoted 8 lines
Isn't it enough low latency? :)
Quoted 4 lines
Quoted 1 line
With this patch, it went down to 30ms. (x100 faster :))
This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.
so, it just takes 90us. I hope it is acceptable.
Yes, this is far below the scheduler tick, which is much more acceptable.
Thanks,
Mathieu
Quoted 7 lines
Thank you,
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
PATCH 5/5 kprobes: Support delayed unoptimization
In reply to Masami HiramatsuUnoptimization occurs when a probe is unregistered or disabled, and is heavy because it recovers instructions by using stop_machine(). This patch delays unoptimization operations and unoptimize several probes at once by using text_poke_smp_batch().
This can avoid unexpected system slowdown coming from stop_machine().
---
arch/x86/include/asm/kprobes.h | 4 arch/x86/kernel/kprobes.c | 41 ++++ include/linux/kprobes.h | 2 kernel/kprobes.c | 434 +++++++++++++++++++++++++++------------- 4 files changed, 340 insertions(+), 141 deletions(-)
arch/x86/include/asm/kprobes.h +2 -2
@@ -79,12 +79,12 @@ struct arch_specific_insn { };struct arch_optimized_insn { -/* copy of the original instructions */ -kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ];/* detour code buffer */ kprobe_opcode_t *insn;/* the size of instructions copied to detour code buffer */ size_t size; +/* copy of the original instructions */ +kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE ]; };/* Return true (!0) if optinsn is prepared for optimization. */
arch/x86/kernel/kprobes.c +41 -0
59 lines not on this page · raw message
include/linux/kprobes.h +2 -0
@@ -276,6 +276,8 @@ extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);extern int arch_prepare_optimized_kprobe (struct optimized_kprobe *op);extern void arch_remove_optimized_kprobe (struct optimized_kprobe *op);extern void arch_optimize_kprobes (struct list_head *oplist); +extern void arch_unoptimize_kprobes (struct list_head *oplist, +struct list_head *done_list);extern void arch_unoptimize_kprobe (struct optimized_kprobe *op);extern kprobe_opcode_t *get_optinsn_slot (void );extern void free_optinsn_slot (kprobe_opcode_t *slot,int dirty);
kernel/kprobes.c +295 -139
633 lines not on this page · raw message
Signature · 2 lines
Masami Hiramatsu e-mail: [email protected]