Skip to contentSkip to conversation

kprobes: batch (un)optimization support

PATCH Superseded Masami Hiramatsu 2 participants 16 messages May 10, 2010 – May 13, 2010

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover kprobes: batch (un)optimization support

Hi,

Since current kprobes jump optimization calls stop_machine() for each probe, it will almost freeze machine when registering a lot of probes (~1000) at once.
For avoiding this issue, this patch series enhances kprobes and text_poke.

- Introduces text_poke_smp_batch() which modifies multiple codes with one stop_machine().
- Limits how many probes can be optimized at once.
- Introduce delayed unoptimization for batch processing.

text_poke_smp_batch() also helps Jason's Jump label to reduce its overhead coming from text_poke_smp().

Currently, this patch allocates working memory static, however, it is possible to allocate dynamically, because its not in a critical section. it will be done by the next step, or next version.

Thank you,

---

Masami Hiramatsu (5):
kprobes: Support delayed unoptimization
kprobes/x86: Use text_poke_smp_batch
x86: Introduce text_poke_smp_batch() for batch-code modifying
kprobes: Limit maximum number of optimization at once [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize

 arch/x86/include/asm/alternative.h |    7 +
 arch/x86/include/asm/kprobes.h     |    4 
 arch/x86/kernel/alternative.c      |   49 +++-
 arch/x86/kernel/kprobes.c          |   78 ++++++
 include/linux/kprobes.h            |    4 
 kernel/kprobes.c                   |  437 ++++++++++++++++++++++++------------
 6 files changed, 416 insertions(+), 163 deletions(-)
Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#1 lore raw

PATCH 1/5 kprobes: Remove redundant text_mutex lock in optimize

Remove text_mutex locking in optimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 kernel/kprobes.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)
kernel/kprobes.c +0 -2
index 282035f..1d34eef 100644
@@ -606,14 +606,12 @@ static void __kprobes optimize_all_kprobes(void)
 		return;
 
 	kprobes_allow_optimization = true;
-	mutex_lock(&text_mutex);
 	for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
 		head = &kprobe_table[i];
 		hlist_for_each_entry_rcu(p, node, head, hlist)
 			if (!kprobe_disabled(p))
 				optimize_kprobe(p);
 	}
-	mutex_unlock(&text_mutex);
 	printk(KERN_INFO "Kprobes globally optimized\n");
 }
 
Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#2 lore raw

Re: [PATCH -tip 1/5] [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize

Quoted 3 lines · * Masami Hiramatsu ([email protected]) wrote:

Remove text_mutex locking in optimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.

Hi Masami,

A few comments:

1) optimize_all_kprobes/unoptimize_all_kprobes should have comments saying that they are always called with kprobe_mutex held.

2) The sentence above in the changelog could be changed into:

..."because this function doesn't modify text. It simply queues optimizations
for the kprobe_optimizer worker thread."

3)

static DEFINE_MUTEX(kprobe_mutex); /* Protects kprobe_table */

.. should also state that it protects optimizing_list.

Thanks,

Mathieu

Quoted 36 lines
Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 kernel/kprobes.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)
kernel/kprobes.c +0 -2
index 282035f..1d34eef 100644
@@ -606,14 +606,12 @@ static void __kprobes optimize_all_kprobes(void)
 		return;
 
 	kprobes_allow_optimization = true;
-	mutex_lock(&text_mutex);
 	for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
 		head = &kprobe_table[i];
 		hlist_for_each_entry_rcu(p, node, head, hlist)
 			if (!kprobe_disabled(p))
 				optimize_kprobe(p);
 	}
-	mutex_unlock(&text_mutex);
 	printk(KERN_INFO "Kprobes globally optimized\n");
 }
 
Signature · 3 lines
-- 
Masami Hiramatsu
e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#3 lore raw

Re: [PATCH -tip 1/5] [CLEANUP] kprobes: Remove redundant text_mutex lock in optimize

Hi Mathieu,

Quoted 14 lines · Mathieu Desnoyers
Quoted 3 lines · * Masami Hiramatsu ([email protected]) wrote:

Remove text_mutex locking in optimize_all_kprobes, because that function doesn't modify text but just order optimization to worker.

Hi Masami,

A few comments:

1) optimize_all_kprobes/unoptimize_all_kprobes should have comments saying that they are always called with kprobe_mutex held.

2) The sentence above in the changelog could be changed into:

Quoted 9 lines

..."because this function doesn't modify text. It simply queues optimizations
for the kprobe_optimizer worker thread."

3)

static DEFINE_MUTEX(kprobe_mutex); /* Protects kprobe_table */

.. should also state that it protects optimizing_list.

Thanks! all comments are good to me!

Thank you again,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#4 lore raw

PATCH 2/5 kprobes: Limit maximum number of optimization at once

In reply to Masami Hiramatsu

Since text_poke_smp() uses stop_machine(), the machine almost stop a while if we optimize a lot of probes at once. This patch limits the maximum number of probes to optimize (means calling text_poke_smp()) at once, and try it again after a while if there are more probes waiting for optimization than the limitation.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 kernel/kprobes.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/kprobes.c +6 -0
index 1d34eef..aae368a 100644
@@ -424,11 +424,13 @@ static LIST_HEAD(optimizing_list);
 static void kprobe_optimizer(struct work_struct *work);
 static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);
 #define OPTIMIZE_DELAY 5
+#define MAX_OPTIMIZE_PROBES 64
 
 /* Kprobe jump optimizer */
 static __kprobes void kprobe_optimizer(struct work_struct *work)
 {
 	struct optimized_kprobe *op, *tmp;
+	int c = 0;
 
 	/* Lock modules while optimizing kprobes */
 	mutex_lock(&module_mutex);
@@ -462,9 +464,13 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)
 		if (arch_optimize_kprobe(op) < 0)
 			op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED;
 		list_del_init(&op->list);
+		if (++c >= MAX_OPTIMIZE_PROBES)
+			break;
 	}
 	mutex_unlock(&text_mutex);
 	put_online_cpus();
+	if (!list_empty(&optimizing_list))
+		schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
 end:
 	mutex_unlock(&kprobe_mutex);
 	mutex_unlock(&module_mutex);
Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#5 lore raw

PATCH 3/5 x86: Introduce text_poke_smp_batch() for batch-code modifying

In reply to Masami Hiramatsu

Introduce text_poke_smp_batch(). This function modifies several text areas with one stop_machine() on SMPr. Because calling stop_machine() is heavy task, it is better to aggregate text_poke requests.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/include/asm/alternative.h |    7 +++++
 arch/x86/kernel/alternative.c      |   49 +++++++++++++++++++++++++++++-------
 2 files changed, 47 insertions(+), 9 deletions(-)
arch/x86/include/asm/alternative.h +7 -0
index 03b6bb5..9a08773 100644
@@ -176,7 +176,14 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
  * On the local CPU you need to be protected again NMI or MCE handlers seeing an
  * inconsistent instruction while you patch.
  */
+struct text_poke_param {
+	void *addr;
+	const void *opcode;
+	size_t len;
+};
+
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
+extern void text_poke_smp_batch(struct text_poke_param *params, int n);
 
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +40 -9

78 lines not on this page · raw message

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#6 lore raw

PATCH 4/5 kprobes/x86: Use text_poke_smp_batch

In reply to Masami Hiramatsu

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +30 -7
index 345a4b1..63a5c24 100644
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256
+static struct text_poke_param jump_params[MAX_OPTIMIZE_PROBES];
+static char jump_code_buf[MAX_OPTIMIZE_PROBES][RELATIVEJUMP_SIZE];
+
+static void __kprobes setup_optimize_kprobe(struct text_poke_param *tprm,
+					    char *insn_buf,
+					    struct optimized_kprobe *op)
 {
-	unsigned char jmp_code[RELATIVEJUMP_SIZE];
 	s32 rel = (s32)((long)op->optinsn.insn -
 			((long)op->kp.addr + RELATIVEJUMP_SIZE));
 
@@ -1396,16 +1400,35 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
 	memcpy(op->optinsn.copied_insn, op->kp.addr + INT3_SIZE,
 	       RELATIVE_ADDR_SIZE);
 
-	jmp_code[0] = RELATIVEJUMP_OPCODE;
-	*(s32 *)(&jmp_code[1]) = rel;
+	insn_buf[0] = RELATIVEJUMP_OPCODE;
+	*(s32 *)(&insn_buf[1]) = rel;
+
+	tprm->addr = op->kp.addr;
+	tprm->opcode = insn_buf;
+	tprm->len = RELATIVEJUMP_SIZE;
+}
+
+/* Replace a breakpoint (int3) with a relative jump.  */
+void __kprobes arch_optimize_kprobes(struct list_head *oplist)
+{
+	struct optimized_kprobe *op, *tmp;
+	int c = 0;
+
+	list_for_each_entry_safe(op, tmp, oplist, list) {
+		WARN_ON(kprobe_disabled(&op->kp));
+		/* Setup param */
+		setup_optimize_kprobe(&jump_params[c], jump_code_buf[c], op);
+		list_del_init(&op->list);
+		if (++c >= MAX_OPTIMIZE_PROBES)
+			break;
+	}
 
 	/*
 	 * text_poke_smp doesn't support NMI/MCE code modifying.
 	 * However, since kprobes itself also doesn't support NMI/MCE
 	 * code probing, it's not a problem.
 	 */
-	text_poke_smp(op->kp.addr, jmp_code, RELATIVEJUMP_SIZE);
-	return 0;
+	text_poke_smp_batch(jump_params, c);
 }
 
 /* Replace a relative jump with a breakpoint (int3).  */
include/linux/kprobes.h +1 -1
index e7d1b2e..fe157ba 100644
@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);
 extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);
 extern int arch_prepare_optimized_kprobe(struct optimized_kprobe *op);
 extern void arch_remove_optimized_kprobe(struct optimized_kprobe *op);
-extern int  arch_optimize_kprobe(struct optimized_kprobe *op);
+extern void arch_optimize_kprobes(struct list_head *oplist);
 extern void arch_unoptimize_kprobe(struct optimized_kprobe *op);
 extern kprobe_opcode_t *get_optinsn_slot(void);
 extern void free_optinsn_slot(kprobe_opcode_t *slot, int dirty);
kernel/kprobes.c +1 -12
index aae368a..c824c23 100644
@@ -424,14 +424,10 @@ static LIST_HEAD(optimizing_list);
 static void kprobe_optimizer(struct work_struct *work);
 static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);
 #define OPTIMIZE_DELAY 5
-#define MAX_OPTIMIZE_PROBES 64
 
 /* Kprobe jump optimizer */
 static __kprobes void kprobe_optimizer(struct work_struct *work)
 {
-	struct optimized_kprobe *op, *tmp;
-	int c = 0;
-
 	/* Lock modules while optimizing kprobes */
 	mutex_lock(&module_mutex);
 	mutex_lock(&kprobe_mutex);
@@ -459,14 +455,7 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)
 	 */
 	get_online_cpus();
 	mutex_lock(&text_mutex);
-	list_for_each_entry_safe(op, tmp, &optimizing_list, list) {
-		WARN_ON(kprobe_disabled(&op->kp));
-		if (arch_optimize_kprobe(op) < 0)
-			op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED;
-		list_del_init(&op->list);
-		if (++c >= MAX_OPTIMIZE_PROBES)
-			break;
-	}
+	arch_optimize_kprobes(&optimizing_list);
 	mutex_unlock(&text_mutex);
 	put_online_cpus();
 	if (!list_empty(&optimizing_list))
Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#7 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Thanks,

Mathieu

Quoted 104 lines
arch/x86/kernel/kprobes.c +29 -5
+static struct text_poke_param jump_params[MAX_OPTIMIZE_PROBES];
+static char jump_code_buf[MAX_OPTIMIZE_PROBES][RELATIVEJUMP_SIZE];
+
+static void __kprobes setup_optimize_kprobe(struct text_poke_param *tprm,
+					    char *insn_buf,
+					    struct optimized_kprobe *op)
 {
-	unsigned char jmp_code[RELATIVEJUMP_SIZE];
 	s32 rel = (s32)((long)op->optinsn.insn -
 			((long)op->kp.addr + RELATIVEJUMP_SIZE));
 
@@ -1396,16 +1400,35 @@ int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
 	memcpy(op->optinsn.copied_insn, op->kp.addr + INT3_SIZE,
 	       RELATIVE_ADDR_SIZE);
 
-	jmp_code[0] = RELATIVEJUMP_OPCODE;
-	*(s32 *)(&jmp_code[1]) = rel;
+	insn_buf[0] = RELATIVEJUMP_OPCODE;
+	*(s32 *)(&insn_buf[1]) = rel;
+
+	tprm->addr = op->kp.addr;
+	tprm->opcode = insn_buf;
+	tprm->len = RELATIVEJUMP_SIZE;
+}
+
+/* Replace a breakpoint (int3) with a relative jump.  */
+void __kprobes arch_optimize_kprobes(struct list_head *oplist)
+{
+	struct optimized_kprobe *op, *tmp;
+	int c = 0;
+
+	list_for_each_entry_safe(op, tmp, oplist, list) {
+		WARN_ON(kprobe_disabled(&op->kp));
+		/* Setup param */
+		setup_optimize_kprobe(&jump_params[c], jump_code_buf[c], op);
+		list_del_init(&op->list);
+		if (++c >= MAX_OPTIMIZE_PROBES)
+			break;
+	}
 
 	/*
 	 * text_poke_smp doesn't support NMI/MCE code modifying.
 	 * However, since kprobes itself also doesn't support NMI/MCE
 	 * code probing, it's not a problem.
 	 */
-	text_poke_smp(op->kp.addr, jmp_code, RELATIVEJUMP_SIZE);
-	return 0;
+	text_poke_smp_batch(jump_params, c);
 }
 
 /* Replace a relative jump with a breakpoint (int3).  */
include/linux/kprobes.h +1 -1
index e7d1b2e..fe157ba 100644
@@ -275,7 +275,7 @@ extern int arch_prepared_optinsn(struct arch_optimized_insn *optinsn);
 extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);
 extern int arch_prepare_optimized_kprobe(struct optimized_kprobe *op);
 extern void arch_remove_optimized_kprobe(struct optimized_kprobe *op);
-extern int  arch_optimize_kprobe(struct optimized_kprobe *op);
+extern void arch_optimize_kprobes(struct list_head *oplist);
 extern void arch_unoptimize_kprobe(struct optimized_kprobe *op);
 extern kprobe_opcode_t *get_optinsn_slot(void);
 extern void free_optinsn_slot(kprobe_opcode_t *slot, int dirty);
kernel/kprobes.c +1 -12
index aae368a..c824c23 100644
@@ -424,14 +424,10 @@ static LIST_HEAD(optimizing_list);
 static void kprobe_optimizer(struct work_struct *work);
 static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);
 #define OPTIMIZE_DELAY 5
-#define MAX_OPTIMIZE_PROBES 64
 
 /* Kprobe jump optimizer */
 static __kprobes void kprobe_optimizer(struct work_struct *work)
 {
-	struct optimized_kprobe *op, *tmp;
-	int c = 0;
-
 	/* Lock modules while optimizing kprobes */
 	mutex_lock(&module_mutex);
 	mutex_lock(&kprobe_mutex);
@@ -459,14 +455,7 @@ static __kprobes void kprobe_optimizer(struct work_struct *work)
 	 */
 	get_online_cpus();
 	mutex_lock(&text_mutex);
-	list_for_each_entry_safe(op, tmp, &optimizing_list, list) {
-		WARN_ON(kprobe_disabled(&op->kp));
-		if (arch_optimize_kprobe(op) < 0)
-			op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED;
-		list_del_init(&op->list);
-		if (++c >= MAX_OPTIMIZE_PROBES)
-			break;
-	}
+	arch_optimize_kprobes(&optimizing_list);
 	mutex_unlock(&text_mutex);
 	put_online_cpus();
 	if (!list_empty(&optimizing_list))
Signature · 3 lines
-- 
Masami Hiramatsu
e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#8 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
Quoted 8 lines

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 3 lines
arch/x86/kernel/kprobes.c +1 -2
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

From my experiment on kvm/4cpu, it took about 3 seconds in average.

With this patch, it went down to 30ms. (x100 faster :))

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#9 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 26 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
Quoted 8 lines
Quoted 6 lines

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 3 lines
arch/x86/kernel/kprobes.c +1 -2
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

Thanks,

Mathieu

Quoted 6 lines

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#10 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Quoted 4 lines
Quoted 1 line

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.

Quoted 4 lines
Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#11 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 47 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Quoted 4 lines
Quoted 1 line

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.

Quoted 2 lines
Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

Quoted 8 lines
Quoted 2 lines

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.

I think we are starting to see the stop_machine() approach is really limiting our ability to do even relatively small amount of work without hurting responsiveness significantly.

What's the current showstopper with the breakpoint-bypass-ipi approach that solves this issue properly and makes this batching approach unnecessary ?

Thanks,

Mathieu

Quoted 6 lines

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#12 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 57 lines · Mathieu Desnoyers
Quoted 47 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Quoted 4 lines
Quoted 1 line

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Theoretically yes.
But if we register more than 1000 probes at once, it's hard to do anything except optimizing a while(more than 10 sec), because it stops machine so frequently.

Quoted 2 lines
Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

Quoted 8 lines
Quoted 2 lines

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

If it is not acceptable, I can add a knob for control how many probes optimize/unoptimize at once. Anyway, it is expectable latency (after registering/unregistering probes) and it will be small if we put a few probes. (30ms is the worst case)
And if you want, it can be disabled by sysctl.

Quoted 6 lines

I think we are starting to see the stop_machine() approach is really limiting our ability to do even relatively small amount of work without hurting responsiveness significantly.

What's the current showstopper with the breakpoint-bypass-ipi approach that solves this issue properly and makes this batching approach unnecessary ?

We still do not have any official answer from chip vendors. As you know, basic implementation has been done.

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#13 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

In reply to Mathieu Desnoyers
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Quoted 4 lines
Quoted 1 line

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Sorry, my bad. Non tuned kvm guest is so slow...
I've tried to check it again on *bare machine* (4core Xeon 2.33GHz, 4cpu). I found that even without this patch, optimizing 256 probes took 770us in average (min 150us, max 3.3ms.)
With this patch, it went down to 90us in average (min 14us, max 324us!)

Isn't it enough low latency? :)

Quoted 4 lines
Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

so, it just takes 90us. I hope it is acceptable.

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
#14 lore raw

Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch

Quoted 46 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 34 lines · Mathieu Desnoyers
Quoted 33 lines · * Masami Hiramatsu ([email protected]) wrote:
Quoted 31 lines · Mathieu Desnoyers
Quoted 21 lines · * Masami Hiramatsu ([email protected]) wrote:

Use text_poke_smp_batch() in optimization path for reducing the number of stop_machine() issues.

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/kernel/kprobes.c |   37 ++++++++++++++++++++++++++++++-------
 include/linux/kprobes.h   |    2 +-
 kernel/kprobes.c          |   13 +------------
 3 files changed, 32 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes.c +0 -0
index 345a4b1..63a5c24 100644

arch/x86/kernel/kprobes.c int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) · line 1388

Quoted 6 lines
arch/x86/kernel/kprobes.c +1 -2
@@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op)
 	return 0;
 }
 
-/* Replace a breakpoint (int3) with a relative jump.  */
-int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op)
+#define MAX_OPTIMIZE_PROBES 256

So what kind of interrupt latency does a 256-probes batch generate on the system ? Are we talking about a few milliseconds, a few seconds ?

Quoted 4 lines
Quoted 1 line

From my experiment on kvm/4cpu, it took about 3 seconds in average.

That's 3 seconds for multiple calls to stop_machine(). So we can expect latencies in the area of few microseconds for each call, right ?

Sorry, my bad. Non tuned kvm guest is so slow...
I've tried to check it again on *bare machine* (4core Xeon 2.33GHz, 4cpu). I found that even without this patch, optimizing 256 probes took 770us in average (min 150us, max 3.3ms.)
With this patch, it went down to 90us in average (min 14us, max 324us!)

Quoted 8 lines

Isn't it enough low latency? :)

Quoted 4 lines
Quoted 1 line

With this patch, it went down to 30ms. (x100 faster :))

This is beefing up the latency from few microseconds to 30ms. It sounds like a regression rather than a gain to me.

so, it just takes 90us. I hope it is acceptable.

Yes, this is far below the scheduler tick, which is much more acceptable.

Thanks,

Mathieu

Quoted 7 lines

Thank you,

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#15 lore raw

PATCH 5/5 kprobes: Support delayed unoptimization

In reply to Masami Hiramatsu

Unoptimization occurs when a probe is unregistered or disabled, and is heavy because it recovers instructions by using stop_machine(). This patch delays unoptimization operations and unoptimize several probes at once by using text_poke_smp_batch().
This can avoid unexpected system slowdown coming from stop_machine().

Cc Ananth N Mavinakayanahalli <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jim Keniston <[email protected]>
Cc Jason Baron <[email protected]>
Cc Mathieu Desnoyers <[email protected]>

---

 arch/x86/include/asm/kprobes.h |    4 
 arch/x86/kernel/kprobes.c      |   41 ++++
 include/linux/kprobes.h        |    2 
 kernel/kprobes.c               |  434 +++++++++++++++++++++++++++-------------
 4 files changed, 340 insertions(+), 141 deletions(-)
arch/x86/include/asm/kprobes.h +2 -2
index 5478825..e8a864c 100644
@@ -79,12 +79,12 @@ struct arch_specific_insn {
 };
 
 struct arch_optimized_insn {
-	/* copy of the original instructions */
-	kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE];
 	/* detour code buffer */
 	kprobe_opcode_t *insn;
 	/* the size of instructions copied to detour code buffer */
 	size_t size;
+	/* copy of the original instructions */
+	kprobe_opcode_t copied_insn[RELATIVE_ADDR_SIZE];
 };
 
 /* Return true (!0) if optinsn is prepared for optimization. */
arch/x86/kernel/kprobes.c +41 -0

59 lines not on this page · raw message

include/linux/kprobes.h +2 -0
index fe157ba..b78edb5 100644
@@ -276,6 +276,8 @@ extern int arch_check_optimized_kprobe(struct optimized_kprobe *op);
 extern int arch_prepare_optimized_kprobe(struct optimized_kprobe *op);
 extern void arch_remove_optimized_kprobe(struct optimized_kprobe *op);
 extern void arch_optimize_kprobes(struct list_head *oplist);
+extern void arch_unoptimize_kprobes(struct list_head *oplist,
+				    struct list_head *done_list);
 extern void arch_unoptimize_kprobe(struct optimized_kprobe *op);
 extern kprobe_opcode_t *get_optinsn_slot(void);
 extern void free_optinsn_slot(kprobe_opcode_t *slot, int dirty);
kernel/kprobes.c +295 -139

633 lines not on this page · raw message

Signature · 2 lines
Masami Hiramatsu
e-mail: [email protected]