Skip to contentSkip to conversation

bpf: avoid get_kernel_nofault() to fetch kprobe entry IP

PATCH Applied Andrii Nakryiko 5 participants 7 messages Mar 19, 2024 – Mar 25, 2024

Subsystems (4)
lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

PATCH 1/1 bpf: avoid get_kernel_nofault() to fetch kprobe entry IP

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -3
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
 	u32 instr;
 
-	/* Being extra safe in here in case entry ip is on the page-edge. */
-	if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-		return fentry_ip;
+	/* We want to be extra safe in case entry ip is on the page edge,
+	 * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+	 */
+	if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+		if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+			return fentry_ip;
+	} else {
+		instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+	}
 	if (is_endbr(instr))
 		fentry_ip -= ENDBR_INSN_SIZE;
 	return fentry_ip;
Signature · 1 line
2.43.0
#1 lore raw
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Quoted 8 lines

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.

Thanks!

Quoted 34 lines
Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -3
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
 	u32 instr;
 
-	/* Being extra safe in here in case entry ip is on the page-edge. */
-	if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-		return fentry_ip;
+	/* We want to be extra safe in case entry ip is on the page edge,
+	 * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+	 */
+	if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+		if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+			return fentry_ip;
+	} else {
+		instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+	}
 	if (is_endbr(instr))
 		fentry_ip -= ENDBR_INSN_SIZE;
 	return fentry_ip;
Signature · 1 line
2.43.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Acked-by
#2 lore raw
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Quoted 8 lines
Quoted 8 lines

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

I think this is nice improvement

Quoted 4 lines

Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.

I like the idea of new flag saying the address was adjusted for endbr

kprobe adjust the address in arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address in get_entry_ip accordingly

jirka

Quoted 42 lines

Thanks!

Quoted 34 lines
Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -3
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
 	u32 instr;
 
-	/* Being extra safe in here in case entry ip is on the page-edge. */
-	if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-		return fentry_ip;
+	/* We want to be extra safe in case entry ip is on the page edge,
+	 * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+	 */
+	if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+		if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+			return fentry_ip;
+	} else {
+		instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+	}
 	if (is_endbr(instr))
 		fentry_ip -= ENDBR_INSN_SIZE;
 	return fentry_ip;
Signature · 1 line
2.43.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#3 lore raw
Quoted 13 lines · Jiri Olsa
Quoted 11 lines · Masami Hiramatsu
Quoted 8 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

I think this is nice improvement

Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?

Quoted 7 lines
Quoted 4 lines

Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.

I like the idea of new flag saying the address was adjusted for endbr

instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?

Quoted 48 lines

kprobe adjust the address in arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address in get_entry_ip accordingly

jirka

Quoted 42 lines

Thanks!

Quoted 32 lines
Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -4
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
    u32 instr;
 
-   /* Being extra safe in here in case entry ip is on the page-edge. */
-   if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-           return fentry_ip;
+   /* We want to be extra safe in case entry ip is on the page edge,
+    * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+    */
+   if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+           if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+                   return fentry_ip;
+   } else {
+           instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+   }
    if (is_endbr(instr))
            fentry_ip -= ENDBR_INSN_SIZE;
    return fentry_ip;
--
Quoted 2 lines

2.43.0

--
Masami Hiramatsu (Google) <[email protected]>

maintainer Acked-by
#4 lore raw
Quoted 21 lines · On Wed, 20 Mar 2024 10:46:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 20 lines · Jiri Olsa
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Quoted 8 lines
Quoted 8 lines

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

I think this is nice improvement

Quoted 7 lines
Quoted 2 lines

Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?

I'm OK for this change for short term fix. As far as I can see, the kprobe-side change may involve more kprobe internal changes, so

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.

Quoted 6 lines
Quoted 2 lines

I like the idea of new flag saying the address was adjusted for endbr

instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?

It's possible. But it is a bit only for BPF and not fit to kprobe itself. I think we can add it in trace_kprobe instead of kprobe, which can be accessed from struct kprobe *kp.

Thank you,

Quoted 49 lines
Quoted 49 lines

kprobe adjust the address in arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address in get_entry_ip accordingly

jirka

Quoted 42 lines

Thanks!

Quoted 32 lines
Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -4
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
    u32 instr;
 
-   /* Being extra safe in here in case entry ip is on the page-edge. */
-   if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-           return fentry_ip;
+   /* We want to be extra safe in case entry ip is on the page edge,
+    * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+    */
+   if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+           if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+                   return fentry_ip;
+   } else {
+           instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+   }
    if (is_endbr(instr))
            fentry_ip -= ENDBR_INSN_SIZE;
    return fentry_ip;
--
Quoted 2 lines

2.43.0

--
Masami Hiramatsu (Google) <[email protected]>

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#5 lore raw
Quoted 33 lines · Masami Hiramatsu
Quoted 21 lines · On Wed, 20 Mar 2024 10:46:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 20 lines · Jiri Olsa
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Quoted 8 lines
Quoted 8 lines

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.

I think this is nice improvement

Quoted 7 lines
Quoted 2 lines

Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?

Quoted 5 lines

I'm OK for this change for short term fix. As far as I can see, the kprobe-side change may involve more kprobe internal changes, so

Great, thank you!

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.

I like the idea of new flag saying the address was adjusted for endbr

Quoted 7 lines
Quoted 3 lines

instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?

It's possible. But it is a bit only for BPF and not fit to kprobe itself. I think we can add it in trace_kprobe instead of kprobe, which can be accessed from struct kprobe *kp.

sure, if it can be just "endbr64 offset" instead of a true/false flag, it would help to avoid extra conditionals in the hot path (which waste LBR records in some mode, which are important in some applications)

Quoted 56 lines

Thank you,

Quoted 41 lines
Quoted 41 lines

kprobe adjust the address in arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address in get_entry_ip accordingly

jirka

Quoted 34 lines

Thanks!

Quoted 31 lines
Cc Masami Hiramatsu <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/trace/bpf_trace.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -3
index 0a5c4efc73c3..f81adabda38c 100644
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip)
 {
    u32 instr;
 
-   /* Being extra safe in here in case entry ip is on the page-edge. */
-   if (get_kernel_nofault(instr, (u32 *) fentry_ip - 1))
-           return fentry_ip;
+   /* We want to be extra safe in case entry ip is on the page edge,
+    * but otherwise we need to avoid get_kernel_nofault()'s overhead.
+    */
+   if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) {
+           if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE)))
+                   return fentry_ip;
+   } else {
+           instr = *(u32 *)(fentry_ip - ENDBR_INSN_SIZE);
+   }
    if (is_endbr(instr))
            fentry_ip -= ENDBR_INSN_SIZE;
    return fentry_ip;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

--
2.43.0

--
Masami Hiramatsu (Google) <[email protected]>

--
Masami Hiramatsu (Google) <[email protected]>

#6 lore raw
In reply to Andrii Nakryiko
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 9 lines · On Tue, 19 Mar 2024 14:20:13 -0700 you wrote:

get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.

Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are

Quoted 1 line

= 4, in which case we can do direct memory read.

[...]

Here is the summary with links:
- [bpf-next] bpf: avoid get_kernel_nofault() to fetch kprobe entry IP

    https://git.kernel.org/bpf/bpf-next/c/a8497506cd2c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html