Skip to contentSkip to conversation

selftests/bpf: Do not attach kprobe_multi bench to bpf_dispatcher_xdp_func

PATCH Applied Jiri Olsa 6 participants 8 messages Jul 14, 2022 – Jul 15, 2022

Subsystems (3)
lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

PATCH 1/1 selftests/bpf: Do not attach kprobe_multi bench to bpf_dispatcher_xdp_func

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

  WARNING: CPU: 6 PID: 1985 at
  arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
  ...
  ftrace_replace_code+0xa3/0x170
  ftrace_modify_all_code+0xbd/0x150
  ftrace_startup_enable+0x3f/0x50
  ftrace_startup+0x98/0xf0
  register_ftrace_function+0x20/0x60
  register_fprobe_ips+0xbb/0xd0
  bpf_kprobe_multi_link_attach+0x179/0x430
  __sys_bpf+0x18a1/0x2440
  ...
  ------------[ ftrace bug ]------------
  ftrace failed to modify
  [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
   actual:   ffffffe9:7b:ffffff9c:77:1e
  Setting ftrace call site to call ftrace function

It looks like we need some way to way to hide some functions from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

---
 tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
 1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +2 -0
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
 			continue;
 		if (!strncmp(name, "rcu_", 4))
 			continue;
+		if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))
+			continue;
 		if (!strncmp(name, "__ftrace_invalid_address__",
 			     sizeof("__ftrace_invalid_address__") - 1))
 			continue;
Signature · 1 line
2.35.3
#1 lore raw
Quoted 30 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

  WARNING: CPU: 6 PID: 1985 at
  arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
  ...
  ftrace_replace_code+0xa3/0x170
  ftrace_modify_all_code+0xbd/0x150
  ftrace_startup_enable+0x3f/0x50
  ftrace_startup+0x98/0xf0
  register_ftrace_function+0x20/0x60
  register_fprobe_ips+0xbb/0xd0
  bpf_kprobe_multi_link_attach+0x179/0x430
  __sys_bpf+0x18a1/0x2440
  ...
  ------------[ ftrace bug ]------------
  ftrace failed to modify
  [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
   actual:   ffffffe9:7b:ffffff9c:77:1e
  Setting ftrace call site to call ftrace function
Quoted 3 lines

It looks like we need some way to way to hide some functions from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

Tangential: I've seen the same happen on our internal kernel, I thought it's just due to our older ftrace subtree, but now looking at the bpf-next tree I'm not sure. Maybe you can clarify for me?

I think what happens is: we attach a bpf program that uses text_poke and enable ftrace graph and ftrace fails with the same ftrace_verify_code.
I see that on the bpf side, we try to play nicely and use text_poke or modify_ftrace_direct if the location is ftrace-managed, but I don't see something similar on the ftrace side?
How is it supposed to work? Do we have some way to signal to ftrace that we've text_poke'd the location and ftrace shouldn't try to touch it?

Quoted 22 lines
---
 tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
 1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +2 -1
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
                        continue;
                if (!strncmp(name, "rcu_", 4))
                        continue;
+               if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))
+                       continue;
                if (!strncmp(name, "__ftrace_invalid_address__",
                             sizeof("__ftrace_invalid_address__") - 1))
                        continue;
--

2.35.3

#2 lore raw
Quoted 39 lines · Stanislav Fomichev
Quoted 30 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

  WARNING: CPU: 6 PID: 1985 at
  arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
  ...
  ftrace_replace_code+0xa3/0x170
  ftrace_modify_all_code+0xbd/0x150
  ftrace_startup_enable+0x3f/0x50
  ftrace_startup+0x98/0xf0
  register_ftrace_function+0x20/0x60
  register_fprobe_ips+0xbb/0xd0
  bpf_kprobe_multi_link_attach+0x179/0x430
  __sys_bpf+0x18a1/0x2440
  ...
  ------------[ ftrace bug ]------------
  ftrace failed to modify
  [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
   actual:   ffffffe9:7b:ffffff9c:77:1e
  Setting ftrace call site to call ftrace function
Quoted 3 lines

It looks like we need some way to way to hide some functions from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

Tangential: I've seen the same happen on our internal kernel, I thought it's just due to our older ftrace subtree, but now looking at the bpf-next tree I'm not sure. Maybe you can clarify for me?

Quoted 6 lines

I think what happens is: we attach a bpf program that uses text_poke and enable ftrace graph and ftrace fails with the same ftrace_verify_code.
I see that on the bpf side, we try to play nicely and use text_poke or modify_ftrace_direct if the location is ftrace-managed, but I don't see something similar on the ftrace side?

so ftrace keeps track of all the ftrace-able locations and assumes it's the only one that changes them, so when you change some of them with text_poke ftrace won't see expected value there and will think it's a bug

Quoted 3 lines

How is it supposed to work? Do we have some way to signal to ftrace that we've text_poke'd the location and ftrace shouldn't try to touch it?

we discussed to have a way to exclude some functions from ftrace, starting with bpf_dispatcher_xdp_func, I'll send some rfc to start discussion

jirka

Quoted 25 lines
Quoted 25 lines
---
 tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
 1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +2 -1
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
                        continue;
                if (!strncmp(name, "rcu_", 4))
                        continue;
+               if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))
+                       continue;
                if (!strncmp(name, "__ftrace_invalid_address__",
                             sizeof("__ftrace_invalid_address__") - 1))
                        continue;
--

2.35.3

Acked-by on v1 1/1
#3 lore raw
In reply to Jiri Olsa
Quoted 29 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

   WARNING: CPU: 6 PID: 1985 at
   arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
   ...
   ftrace_replace_code+0xa3/0x170
   ftrace_modify_all_code+0xbd/0x150
   ftrace_startup_enable+0x3f/0x50
   ftrace_startup+0x98/0xf0
   register_ftrace_function+0x20/0x60
   register_fprobe_ips+0xbb/0xd0
   bpf_kprobe_multi_link_attach+0x179/0x430
   __sys_bpf+0x18a1/0x2440
   ...
   ------------[ ftrace bug ]------------
   ftrace failed to modify
   [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
    actual:   ffffffe9:7b:ffffff9c:77:1e
   Setting ftrace call site to call ftrace function
Quoted 1 line

It looks like we need some way to way to hide some functions

need some way to hide some functions ...

Quoted 5 lines

from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

I tried with 32cpus on my local qemu/vm but cannot reproduce the crash. But look at the code, your should seem okay as bpf_dispatcher_xdp_func indeed could be poked and simplified. So with a few nits,

tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c static int get_syms(char ***symsp, size_t *cntp) · line 367

Quoted 13 lines
---
  tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
  1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
  			continue;
  		if (!strncmp(name, "rcu_", 4))
  			continue;
+		if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))

ffffffff81b17a90 T bpf_dispatcher_xdp_func

bpf_dispatcher_xdp_func is a full name, you can just use strcmp here. Further,

linux/bpf.h:#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func

Currently, bpf_dispatcher_xdp_func is the ONLY BPF_DISPATCHER_FUNC. So comparing bpf_dispatcher_xdp_func is enough. It would be good to add a comment to explain why not comparing to bpf_dispatcher_*_func.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
+			continue;
  		if (!strncmp(name, "__ftrace_invalid_address__",
  			     sizeof("__ftrace_invalid_address__") - 1))
  			continue;
#4 lore raw
Quoted 40 lines · Yonghong Song
Quoted 29 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

   WARNING: CPU: 6 PID: 1985 at
   arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
   ...
   ftrace_replace_code+0xa3/0x170
   ftrace_modify_all_code+0xbd/0x150
   ftrace_startup_enable+0x3f/0x50
   ftrace_startup+0x98/0xf0
   register_ftrace_function+0x20/0x60
   register_fprobe_ips+0xbb/0xd0
   bpf_kprobe_multi_link_attach+0x179/0x430
   __sys_bpf+0x18a1/0x2440
   ...
   ------------[ ftrace bug ]------------
   ftrace failed to modify
   [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
    actual:   ffffffe9:7b:ffffff9c:77:1e
   Setting ftrace call site to call ftrace function
Quoted 1 line

It looks like we need some way to way to hide some functions

need some way to hide some functions ...

Quoted 3 lines

from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

Quoted 9 lines
Quoted 2 lines

I tried with 32cpus on my local qemu/vm but cannot reproduce the crash. But look at the code, your should seem okay as bpf_dispatcher_xdp_func indeed could be poked and simplified. So with a few nits,

Fixed typo and changed strncmp to strcmp, pushed to bpf-next.

Quoted 29 lines

tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c static int get_syms(char ***symsp, size_t *cntp) · line 367

Quoted 13 lines
---
  tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
  1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
                      continue;
              if (!strncmp(name, "rcu_", 4))
                      continue;
+             if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))

ffffffff81b17a90 T bpf_dispatcher_xdp_func

bpf_dispatcher_xdp_func is a full name, you can just use strcmp here. Further,

linux/bpf.h:#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func

Currently, bpf_dispatcher_xdp_func is the ONLY BPF_DISPATCHER_FUNC. So comparing bpf_dispatcher_xdp_func is enough. It would be good to add a comment to explain why not comparing to bpf_dispatcher_*_func.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
+                     continue;
              if (!strncmp(name, "__ftrace_invalid_address__",
                           sizeof("__ftrace_invalid_address__") - 1))
                      continue;
#5 lore raw
Quoted 44 lines · Andrii Nakryiko
Quoted 43 lines · Yonghong Song
Quoted 29 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

   WARNING: CPU: 6 PID: 1985 at
   arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
   ...
   ftrace_replace_code+0xa3/0x170
   ftrace_modify_all_code+0xbd/0x150
   ftrace_startup_enable+0x3f/0x50
   ftrace_startup+0x98/0xf0
   register_ftrace_function+0x20/0x60
   register_fprobe_ips+0xbb/0xd0
   bpf_kprobe_multi_link_attach+0x179/0x430
   __sys_bpf+0x18a1/0x2440
   ...
   ------------[ ftrace bug ]------------
   ftrace failed to modify
   [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
    actual:   ffffffe9:7b:ffffff9c:77:1e
   Setting ftrace call site to call ftrace function
Quoted 1 line

It looks like we need some way to way to hide some functions

need some way to hide some functions ...

Quoted 5 lines

from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

Quoted 8 lines
Quoted 6 lines

I tried with 32cpus on my local qemu/vm but cannot reproduce the crash. But look at the code, your should seem okay as bpf_dispatcher_xdp_func indeed could be poked and simplified. So with a few nits,

Fixed typo and changed strncmp to strcmp, pushed to bpf-next.

thanks,
jirka

Quoted 30 lines
Quoted 30 lines
Quoted 10 lines
---
  tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
  1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +0 -0
index 5b93d5d0bd93..8c442051f312 100644

tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c static int get_syms(char ***symsp, size_t *cntp) · line 367

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
                      continue;
              if (!strncmp(name, "rcu_", 4))
                      continue;
+             if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))

ffffffff81b17a90 T bpf_dispatcher_xdp_func

bpf_dispatcher_xdp_func is a full name, you can just use strcmp here. Further,

linux/bpf.h:#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func

Currently, bpf_dispatcher_xdp_func is the ONLY BPF_DISPATCHER_FUNC. So comparing bpf_dispatcher_xdp_func is enough. It would be good to add a comment to explain why not comparing to bpf_dispatcher_*_func.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
+                     continue;
              if (!strncmp(name, "__ftrace_invalid_address__",
                           sizeof("__ftrace_invalid_address__") - 1))
                      continue;
#6 lore raw
In reply to Yonghong Song
Quoted 32 lines · Yonghong Song
Quoted 29 lines · Jiri Olsa

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

Ftrace is not aware of this update so this will cause ftrace_bug with:

   WARNING: CPU: 6 PID: 1985 at
   arch/x86/kernel/ftrace.c:94 ftrace_verify_code+0x27/0x50
   ...
   ftrace_replace_code+0xa3/0x170
   ftrace_modify_all_code+0xbd/0x150
   ftrace_startup_enable+0x3f/0x50
   ftrace_startup+0x98/0xf0
   register_ftrace_function+0x20/0x60
   register_fprobe_ips+0xbb/0xd0
   bpf_kprobe_multi_link_attach+0x179/0x430
   __sys_bpf+0x18a1/0x2440
   ...
   ------------[ ftrace bug ]------------
   ftrace failed to modify
   [<ffffffff818d9380>] bpf_dispatcher_xdp_func+0x0/0x10
    actual:   ffffffe9:7b:ffffff9c:77:1e
   Setting ftrace call site to call ftrace function
Quoted 3 lines
Quoted 1 line

It looks like we need some way to way to hide some functions

need some way to hide some functions ...

ugh, right

Quoted 29 lines
Quoted 6 lines

from ftrace, but meanwhile we workaround this by skipping bpf_dispatcher_xdp_func from kprobe_multi bench test.

I tried with 32cpus on my local qemu/vm but cannot reproduce the crash. But look at the code, your should seem okay as bpf_dispatcher_xdp_func indeed could be poked and simplified. So with a few nits,

tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c static int get_syms(char ***symsp, size_t *cntp) · line 367

Quoted 13 lines
---
  tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 2 ++
  1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +1 -0
index 5b93d5d0bd93..8c442051f312 100644
@@ -364,6 +364,8 @@ static int get_syms(char ***symsp, size_t *cntp)
  			continue;
  		if (!strncmp(name, "rcu_", 4))
  			continue;
+		if (!strncmp(name, "bpf_dispatcher_xdp_func", 23))

ffffffff81b17a90 T bpf_dispatcher_xdp_func

Quoted 8 lines

bpf_dispatcher_xdp_func is a full name, you can just use strcmp here. Further,

linux/bpf.h:#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func

Currently, bpf_dispatcher_xdp_func is the ONLY BPF_DISPATCHER_FUNC. So comparing bpf_dispatcher_xdp_func is enough. It would be good to add a comment to explain why not comparing to bpf_dispatcher_*_func.

ok

thanks,
jirka

#7 lore raw
In reply to Jiri Olsa
This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Quoted 9 lines · On Thu, 14 Jul 2022 10:23:16 +0200 you wrote:

Alexei reported crash by running test_progs -j on system with 32 cpus.

It turned out the kprobe_multi bench test that attaches all ftrace-able functions will race with bpf_dispatcher_update, that calls bpf_arch_text_poke on bpf_dispatcher_xdp_func, which is ftrace-able function.

[...]

Here is the summary with links:
- [bpf] selftests/bpf: Do not attach kprobe_multi bench to bpf_dispatcher_xdp_func

    https://git.kernel.org/bpf/bpf-next/c/7fb27a56b9eb

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html