bpf: avoid get_kernel_nofault() to fetch kprobe entry IP
Subsystems (4)
Andrii Nakryiko
maintainerPATCH 1/1 bpf: avoid get_kernel_nofault() to fetch kprobe entry IP
get_kernel_nofault() (or, rather, underlying copy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized with CONFIG_X86_KERNEL_IBT=y config.
Let's avoid using it if we know that fentry_ip - 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they are
Quoted 1 line
= 4, in which case we can do direct memory read.
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on copy_from_kernel_nofault() call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.
--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +9 -3
@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip;
Signature · 1 line
2.43.0
Masami Hiramatsu (Google)
maintainerQuoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.
Quoted 8 lines
Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on
copy_from_kernel_nofault()call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.
Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.
Thanks!
Quoted 34 lines
Cc Masami Hiramatsu <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/trace/bpf_trace.c +9
-3@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip;Signature · 1 line
2.43.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Jiri Olsa
Acked-byQuoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.
Quoted 8 lines
Quoted 8 lines
Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on
copy_from_kernel_nofault()call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.
I think this is nice improvement
Quoted 4 lines
Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.
I like the idea of new flag saying the address was adjusted for endbr
kprobe adjust the address in arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address in get_entry_ip accordingly
jirka
Quoted 42 lines
Thanks!
Quoted 34 lines
Cc Masami Hiramatsu <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/trace/bpf_trace.c +9
-3@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip;Signature · 1 line
2.43.0Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 13 lines · Jiri Olsa
Quoted 11 lines · Masami Hiramatsu
Quoted 8 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on
copy_from_kernel_nofault()call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.I think this is nice improvement
Acked-by Jiri Olsa <[email protected]>
Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?
Quoted 7 lines
Quoted 4 lines
Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.
I like the idea of new flag saying the address was adjusted for endbr
instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?
Quoted 48 lines
kprobe adjust the address in
arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address inget_entry_ipaccordinglyjirka
Quoted 42 lines
Thanks!
Quoted 32 lines
Cc Masami Hiramatsu <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/trace/bpf_trace.c +9
-4@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip; --Quoted 2 lines
2.43.0
--
Masami Hiramatsu (Google) <[email protected]>
Quoted 21 lines · On Wed, 20 Mar 2024 10:46:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 20 lines · Jiri Olsa
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.Quoted 8 lines
Quoted 8 lines
Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on
copy_from_kernel_nofault()call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.I think this is nice improvement
Quoted 7 lines
Quoted 2 lines
Acked-by Jiri Olsa <[email protected]>Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?
I'm OK for this change for short term fix. As far as I can see, the kprobe-side change may involve more kprobe internal changes, so
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.
Quoted 6 lines
Quoted 2 lines
I like the idea of new flag saying the address was adjusted for endbr
instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?
It's possible. But it is a bit only for BPF and not fit to kprobe itself. I think we can add it in trace_kprobe instead of kprobe, which can be accessed from struct kprobe *kp.
Thank you,
Quoted 49 lines
Quoted 49 lines
kprobe adjust the address in
arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address inget_entry_ipaccordinglyjirka
Quoted 42 lines
Thanks!
Quoted 32 lines
Cc Masami Hiramatsu <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/trace/bpf_trace.c +9
-4@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip; --Quoted 2 lines
2.43.0
--
Masami Hiramatsu (Google) <[email protected]>
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 33 lines · Masami Hiramatsu
Quoted 21 lines · On Wed, 20 Mar 2024 10:46:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 20 lines · Jiri Olsa
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · On Tue, 19 Mar 2024 14:20:13 -0700 Andrii Nakryiko <[email protected]> wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.Quoted 8 lines
Quoted 8 lines
Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
Another benefit (and actually what caused a closer look at this part of code) is that now LBR record is (typically) not wasted on
copy_from_kernel_nofault()call and code, which helps tools like retsnoop that grab LBR records from inside BPF code in kretprobes.I think this is nice improvement
Quoted 7 lines
Quoted 2 lines
Acked-by Jiri Olsa <[email protected]>Masami, are you ok if we land this rather straightforward fix in bpf-next tree for now, and then you or someone a bit more familiar with ftrace/kprobe internals can generalize this in a more generic way?
Quoted 5 lines
I'm OK for this change for short term fix. As far as I can see, the kprobe-side change may involve more kprobe internal changes, so
Great, thank you!
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Hmm, we may better to have this function in kprobe side and store a flag which such architecture dependent offset is added. That is more natural.
I like the idea of new flag saying the address was adjusted for endbr
Quoted 7 lines
Quoted 3 lines
instead of a flag, can kprobe low-level infrastructure just provide "effective fentry ip" without any flags, so that BPF side of things don't have to care?
It's possible. But it is a bit only for BPF and not fit to kprobe itself. I think we can add it in
trace_kprobeinstead of kprobe, which can be accessed from struct kprobe *kp.
sure, if it can be just "endbr64 offset" instead of a true/false flag, it would help to avoid extra conditionals in the hot path (which waste LBR records in some mode, which are important in some applications)
Quoted 56 lines
Thank you,
Quoted 41 lines
Quoted 41 lines
kprobe adjust the address in
arch_adjust_kprobe_addr, it could be easily added in there and then we'd adjust the address inget_entry_ipaccordinglyjirka
Quoted 34 lines
Thanks!
Quoted 31 lines
Cc Masami Hiramatsu <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/trace/bpf_trace.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/trace/bpf_trace.c +9
-3@@ -1053,9 +1053,15 @@ static unsigned long get_entry_ip(unsigned long fentry_ip) {u32 instr; -/* Being extra safe in here in case entry ip is on the page-edge. */ -if (get_kernel_nofault (instr, (u32 *) fentry_ip -1 )) -return fentry_ip; +/* We want to be extra safe in case entry ip is on the page edge, +* but otherwise we need to avoid get_kernel_nofault()'s overhead. +*/ +if ((fentry_ip & ~PAGE_MASK ) <ENDBR_INSN_SIZE ) { +if (get_kernel_nofault (instr, (u32 *)(fentry_ip -ENDBR_INSN_SIZE ))) +return fentry_ip; + }else { + instr = *(u32 *)(fentry_ip -ENDBR_INSN_SIZE ); + }if (is_endbr (instr)) fentry_ip -=ENDBR_INSN_SIZE ;return fentry_ip;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
--
2.43.0--
Masami Hiramatsu (Google) <[email protected]>--
Masami Hiramatsu (Google) <[email protected]>
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 9 lines · On Tue, 19 Mar 2024 14:20:13 -0700 you wrote:
get_kernel_nofault()(or, rather, underlyingcopy_from_kernel_nofault()) is not free and it does pop up in performance profiles when kprobes are heavily utilized withCONFIG_X86_KERNEL_IBT=yconfig.Let's avoid using it if we know that
fentry_ip- 4 can't cross page boundary. We do that by masking lowest 12 bits and checking if they areQuoted 1 line
= 4, in which case we can do direct memory read.
[...]
Here is the summary with links:
- [bpf-next] bpf: avoid get_kernel_nofault() to fetch kprobe entry IP
https://git.kernel.org/bpf/bpf-next/c/a8497506cd2c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html