Deadlock triggered by bpfsnoop funcgraph feature
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
Even on the latest bpf-next_base commit 2465bb83e0b4 ("Merge branch 's390-bpf-add-s390-jit-support-for-timed-may_goto'"), the issue still persists.
Reproduction:
1. Build bpfsnoop with Go 1.24 and LLVM 20.
2. Start a VM using vmtest [1].
3. Trigger the deadlock with:
'./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug'
Logs:
[ 126.934205] watchdog: CPU1: Watchdog detected hard LOCKUP on cpu 1 [ 126.934406] Modules linked in: [ 126.934713] irq event stamp: 283284 [ 126.934806] hardirqs last enabled at (283283): [<ffffffffa7fa89f8>] default_idle_call+0xb8/0x1d0 [ 126.934925] hardirqs last disabled at (283284): [<ffffffffa73ac21f>] tick_nohz_idle_exit+0x8f/0x110 [ 126.935026] softirqs last enabled at (283262): [<ffffffffa72a4a06>] __irq_exit_rcu+0xa6/0xd0 [ 126.935124] softirqs last disabled at (283255): [<ffffffffa72a4a06>] __irq_exit_rcu+0xa6/0xd0 [ 126.935518] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.17.0-rc1-gcb708c11617a #23 PREEMPT(full) [ 126.935662] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 126.935865] RIP: 0010:__lock_acquire+0x30f/0x2590 [ 126.935973] Code: 89 f8 45 89 f7 49 89 de 4c 89 e3 41 89 cc 48 89 c1 eb 3e 48 8d 04 80 48 8d 04 80 48 8d 34 c5 40 78 87 a9 0f b6 86 c4 00 00 00 <84> c0 74 12 41 38 c0 44 0f 47 c0 80 be c6 00 00 00 02 44 0f 44 c0 [ 126.936062] RSP: 0018:ffffad20800ab008 EFLAGS: 00000007 [ 126.936219] RAX: 0000000000000003 RBX: ffff97af803f2d18 RCX: 0000000000000000 [ 126.936308] RDX: 0000000000000001 RSI: ffffffffa9877c28 RDI: 0000000000000007 [ 126.936394] RBP: ffffad20800ab080 R08: 0000000000000003 R09: 0000000000000005 [ 126.936480] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000003 [ 126.936566] R13: ffff97af803f2240 R14: ffff97af803f2db8 R15: 0000000000000000 [ 126.936655] FS: 0000000000000000(0000) GS:ffff97b0126d8000(0000) knlGS:0000000000000000 [ 126.936744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 126.936830] CR2: 000000c0285a3000 CR3: 0000000102c62004 CR4: 0000000000770ef0 [ 126.936918] PKRU: 55555554 [ 126.937038] Call Trace: [ 126.937133] <TASK> [ 126.937222] ? __lock_acquire+0x43d/0x2590 [ 126.937620] lock_acquire+0xb1/0x2c0 [ 126.937706] ? __bpf_prog_enter_recur+0x2a/0x110 [ 126.937826] ? lock_release+0xc6/0x280 [ 126.937910] ? lock_release+0xc6/0x280 [ 126.938006] __bpf_prog_enter_recur+0x3e/0x110 [ 126.938090] ? __bpf_prog_enter_recur+0x2a/0x110 [ 126.938204] bpf_trampoline_6442539790+0x88/0x110 [ 126.938301] rcu_lockdep_current_cpu_online+0x9/0x70 [ 126.938392] ? rcu_read_lock_held+0x31/0x60 [ 126.938501] bpf_trampoline_6442539812+0x66/0x110 [ 126.938594] rcu_read_lock_held+0x9/0x60 [ 126.938678] ? __htab_map_lookup_elem+0x25/0xf0 [ 126.938798] bpf_trampoline_6442491246+0x79/0x123 [ 126.938894] __htab_map_lookup_elem+0x9/0xf0 [ 126.938991] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a [ 126.939080] ? __htab_map_lookup_elem+0x9/0xf0 [ 126.939182] bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8 [ 126.939285] ? lock_release+0xc6/0x280 [ 126.939381] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.939473] bpf_trampoline_6442539790+0x4b/0x110 [ 126.939566] rcu_lockdep_current_cpu_online+0x9/0x70 [ 126.939649] ? rcu_read_lock_held+0x31/0x60 [ 126.939737] bpf_trampoline_6442539812+0x66/0x110 [ 126.939829] rcu_read_lock_held+0x9/0x60 [ 126.939913] ? __htab_map_lookup_elem+0x25/0xf0 [ 126.940010] bpf_trampoline_6442491246+0x79/0x123 [ 126.940105] __htab_map_lookup_elem+0x9/0xf0 [ 126.940212] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a [ 126.940300] ? rcu_lockdep_current_cpu_online+0x9/0x70 [ 126.940402] bpf_prog_1ed83077283e3ded_bpfsnoop_fgraph+0x12e/0x423 [ 126.940517] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.940609] bpf_trampoline_6442491246+0xac/0x123 [ 126.940705] __htab_map_lookup_elem+0x9/0xf0 [ 126.940796] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a [ 126.940895] ? bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8 [ 126.940980] ? bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8 [ 126.941080] bpf_prog_8c9f4824b35e5d8e_bpfsnoop_fgraph+0x12e/0x423 [ 126.941181] ? lock_release+0xc6/0x280 [ 126.941265] ? lock_release+0xc6/0x280 [ 126.941360] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.941452] bpf_trampoline_6442539790+0x99/0x110 [ 126.941544] rcu_lockdep_current_cpu_online+0x9/0x70 [ 126.941627] ? rcu_read_lock_held+0x31/0x60 [ 126.941715] bpf_trampoline_6442539812+0x66/0x110 [ 126.941813] rcu_read_lock_held+0x9/0x60 [ 126.941896] ? __htab_map_lookup_elem+0x25/0xf0 [ 126.941993] bpf_trampoline_6442491246+0x79/0x123 [ 126.942089] __htab_map_lookup_elem+0x9/0xf0 [ 126.942186] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a [ 126.942276] ? bpf_trampoline_6442539812+0x4b/0x110 [ 126.942360] ? bpf_trampoline_6442539812+0x4b/0x110 [ 126.942446] bpf_prog_8c9f4824b35e5d8e_bpfsnoop_fgraph+0x12e/0x423 [ 126.942552] ? __bpf_tramp_exit+0x72/0x130 [ 126.942647] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.942739] bpf_trampoline_6442539812+0x99/0x110 [ 126.942832] rcu_read_lock_held+0x9/0x60 [ 126.942915] ? __htab_map_lookup_elem+0x25/0xf0 [ 126.943012] bpf_trampoline_6442491246+0x79/0x123 [ 126.943108] __htab_map_lookup_elem+0x9/0xf0 [ 126.943209] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a [ 126.943299] ? bpf_trampoline_6442491246+0x79/0x123 [ 126.943383] ? bpf_trampoline_6442491246+0x79/0x123 [ 126.943469] bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8 [ 126.943571] ? lock_release+0xc6/0x280 [ 126.943666] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.943758] bpf_trampoline_6442539812+0x4b/0x110 [ 126.943851] rcu_read_lock_held+0x9/0x60 [ 126.943934] ? __htab_map_lookup_elem+0x25/0xf0 [ 126.944031] bpf_trampoline_6442491246+0x79/0x123 [ 126.944126] __htab_map_lookup_elem+0x9/0xf0 [ 126.944235] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a [ 126.944345] ? bpf_prog_5c5e9b8ca18045f2_bpfsnoop_fgraph+0x12e/0x3f2 [ 126.944442] bpf_prog_5c5b59f2388bb72a_bpfsnoop_fgraph+0x12e/0x3f2 [ 126.944545] ? lock_release+0xc6/0x280 [ 126.944640] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.944732] bpf_trampoline_6442491246+0x56/0x123 [ 126.944828] __htab_map_lookup_elem+0x9/0xf0 [ 126.944931] ? bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a [ 126.945019] ? tick_nohz_idle_exit+0xc9/0x110 [ 126.945108] bpf_prog_5c5e9b8ca18045f2_bpfsnoop_fgraph+0x12e/0x3f2 [ 126.945210] ? lock_release+0xc6/0x280 [ 126.945305] ? __bpf_prog_enter_recur+0x43/0x110 [ 126.945411] bpf_trampoline_6442519845+0x5e/0x133 [ 126.945510] hrtimer_start_range_ns+0x9/0x4b0 [ 126.945603] ? tick_nohz_restart_sched_tick+0x89/0xe0 [ 126.945694] tick_nohz_idle_exit+0xc9/0x110 [ 126.945789] do_idle+0x150/0x250 [ 126.945890] cpu_startup_entry+0x2d/0x30 [ 126.945976] start_secondary+0xfc/0x100 [ 126.946069] common_startup_64+0x12c/0x138 [ 126.946197] </TASK>
Full log: [2].
Additional information:
* Kernel version: 6.17.0-rc1-gcb708c11617a
* Config: [3].
Links:
[0] https://github.com/bpfsnoop/bpfsnoop
[1] https://github.com/danobi/vmtest
[2]
https://gist.githubusercontent.com/Asphaltt/88d11c49e62485f4d4f4a7664089c3cd/raw/f26c123c0ec5f3e5ac588844db51bbec0bb0f9c7/deadlock-crash.log
[3]
https://gist.githubusercontent.com/Asphaltt/88d11c49e62485f4d4f4a7664089c3cd/raw/f26c123c0ec5f3e5ac588844db51bbec0bb0f9c7/config
Thanks,
Leon
Quoted 5 lines · Leon Hwang
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
debug it pls.
Sounds like you're implying that the root cause is in bpf, but why do you think so?
You're attaching to things that shouldn't be attached to. Like rcu_lockdep_current_cpu_online()
so effectively you're recursing in that lockdep code. See big lock there. It will dead lock for sure.
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
debug it pls.
It’s quite difficult for me. I’ve tried debugging it but didn’t succeed.
Quoted 7 lines
Sounds like you're implying that the root cause is in bpf, but why do you think so?
You're attaching to things that shouldn't be attached to. Like
rcu_lockdep_current_cpu_online()
so effectively you're recursing in that lockdep code. See big lock there. It will dead lock for sure.
If a function that acquires a lock can be traced by a tracing program, bpfsnoop’s funcgraph will attempt to trace it as well. In such cases, a deadlock is highly likely to occur.
With bpfsnoop I try my best to avoid such deadlock issues. But what about other bpf tracing tools? If they don’t handle this properly, the kernel is very likely to crash.
Thanks,
Leon
Quoted 23 lines · Leon Hwang
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
debug it pls.
It’s quite difficult for me. I’ve tried debugging it but didn’t succeed.
Quoted 7 lines
Sounds like you're implying that the root cause is in bpf, but why do you think so?
You're attaching to things that shouldn't be attached to. Like
rcu_lockdep_current_cpu_online()
so effectively you're recursing in that lockdep code. See big lock there. It will dead lock for sure.
Quoted 7 lines
If a function that acquires a lock can be traced by a tracing program, bpfsnoop’s funcgraph will attempt to trace it as well. In such cases, a deadlock is highly likely to occur.
With bpfsnoop I try my best to avoid such deadlock issues. But what about other bpf tracing tools? If they don’t handle this properly, the kernel is very likely to crash.
bpf infra is trying hard not to crash it, but debug kernel is a different category. rcu_read_lock_held() doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same with rcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.
Quoted 32 lines · Alexei Starovoitov
Quoted 23 lines · Leon Hwang
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
debug it pls.
It’s quite difficult for me. I’ve tried debugging it but didn’t succeed.
Quoted 7 lines
Sounds like you're implying that the root cause is in bpf, but why do you think so?
You're attaching to things that shouldn't be attached to. Like
rcu_lockdep_current_cpu_online()
so effectively you're recursing in that lockdep code. See big lock there. It will dead lock for sure.Quoted 7 lines
If a function that acquires a lock can be traced by a tracing program, bpfsnoop’s funcgraph will attempt to trace it as well. In such cases, a deadlock is highly likely to occur.
With bpfsnoop I try my best to avoid such deadlock issues. But what about other bpf tracing tools? If they don’t handle this properly, the kernel is very likely to crash.
Quoted 5 lines
bpf infra is trying hard not to crash it, but debug kernel is a different category.
rcu_read_lock_held()doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same withrcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.
Indeed, it doesn't scale.
When I run
./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes
unresponsive to key presses.
It seems preferable to avoid running BPF programs continuously in such cases.
Thanks,
Leon
Paul E. McKenney
maintainerQuoted 45 lines · Leon Hwang
Quoted 29 lines · Alexei Starovoitov
Quoted 20 lines · Leon Hwang
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph feature for bpfsnoop [0].
debug it pls.
It’s quite difficult for me. I’ve tried debugging it but didn’t succeed.
Quoted 7 lines
Sounds like you're implying that the root cause is in bpf, but why do you think so?
You're attaching to things that shouldn't be attached to. Like
rcu_lockdep_current_cpu_online()
so effectively you're recursing in that lockdep code. See big lock there. It will dead lock for sure.Quoted 7 lines
If a function that acquires a lock can be traced by a tracing program, bpfsnoop’s funcgraph will attempt to trace it as well. In such cases, a deadlock is highly likely to occur.
With bpfsnoop I try my best to avoid such deadlock issues. But what about other bpf tracing tools? If they don’t handle this properly, the kernel is very likely to crash.
Quoted 5 lines
bpf infra is trying hard not to crash it, but debug kernel is a different category.
rcu_read_lock_held()doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same withrcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.Indeed, it doesn't scale.
When I run
./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes
unresponsive to key presses.
Quoted 2 lines
It seems preferable to avoid running BPF programs continuously in such cases.
Agreed, when adding code to the Linux kernel, whether via a patch, via a BPF program, or by whatever other means, you are taking responsibility for the speed, scalability, and latency effects of that code.
Nevertheless, I am happy to add a few "notrace" modifiers if needed. Do you guys need them for rcu_read_lock_held() and rcu_lockdep_current_cpu_online()?
Thanx, Paul
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov
On Tue, Aug 26, 2025 at 7:58 PM Leon Hwang <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 16 lines
Quoted 6 lines
bpf infra is trying hard not to crash it, but debug kernel is a different category.
rcu_read_lock_held()doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same withrcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.Indeed, it doesn't scale.
When I run
./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes
unresponsive to key presses.Quoted 2 lines
It seems preferable to avoid running BPF programs continuously in such cases.
Quoted 8 lines
Agreed, when adding code to the Linux kernel, whether via a patch, via a BPF program, or by whatever other means, you are taking responsibility for the speed, scalability, and latency effects of that code.
Nevertheless, I am happy to add a few "notrace" modifiers if needed. Do you guys need them for
rcu_read_lock_held()andrcu_lockdep_current_cpu_online()?
I think it would be better to add "notrace" to following functions:
./bpfsnoop -k 'rcu_read_*lock_*held*,rcu_lockdep_*' --show-func-proto bool rcu_lockdep_current_cpu_online(); [traceable]
int rcu_read_lock_any_held(); [traceable]
int rcu_read_lock_bh_held(); [traceable]
int rcu_read_lock_held(); [traceable]
int rcu_read_lock_sched_held(); [traceable]
Thanks,
Leon
Quoted 36 lines · Leon Hwang
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov
On Tue, Aug 26, 2025 at 7:58 PM Leon Hwang <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 16 lines
Quoted 6 lines
bpf infra is trying hard not to crash it, but debug kernel is a different category.
rcu_read_lock_held()doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same withrcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.Indeed, it doesn't scale.
When I run
./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes
unresponsive to key presses.Quoted 2 lines
It seems preferable to avoid running BPF programs continuously in such cases.
Quoted 8 lines
Agreed, when adding code to the Linux kernel, whether via a patch, via a BPF program, or by whatever other means, you are taking responsibility for the speed, scalability, and latency effects of that code.
Nevertheless, I am happy to add a few "notrace" modifiers if needed. Do you guys need them for
rcu_read_lock_held()andrcu_lockdep_current_cpu_online()?
Quoted 8 lines
I think it would be better to add "notrace" to following functions:
./bpfsnoop -k 'rcu_read_*lock_*held*,rcu_lockdep_*' --show-func-proto bool
rcu_lockdep_current_cpu_online(); [traceable]
intrcu_read_lock_any_held(); [traceable]
intrcu_read_lock_bh_held(); [traceable]
intrcu_read_lock_held(); [traceable]
intrcu_read_lock_sched_held(); [traceable]
Agree. Seems like an easy way to remove a footgun.
Independently it would be good to make noinstr/notrace to include __cpuidle functions. I think right now it's allowed to attach to default_idle() which is causing issues.
Paul E. McKenney
maintainerQuoted 39 lines · Alexei Starovoitov
Quoted 38 lines · Leon Hwang
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov
On Tue, Aug 26, 2025 at 7:58 PM Leon Hwang <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 16 lines
Quoted 6 lines
bpf infra is trying hard not to crash it, but debug kernel is a different category.
rcu_read_lock_held()doesn't exist in production kernels. You can propose adding "notrace" for it, but in general that doesn't scale. Same withrcu_lockdep_current_cpu_online().
It probably deserves "notrace" too.Indeed, it doesn't scale.
When I run
./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes
unresponsive to key presses.Quoted 2 lines
It seems preferable to avoid running BPF programs continuously in such cases.
Quoted 8 lines
Agreed, when adding code to the Linux kernel, whether via a patch, via a BPF program, or by whatever other means, you are taking responsibility for the speed, scalability, and latency effects of that code.
Nevertheless, I am happy to add a few "notrace" modifiers if needed. Do you guys need them for
rcu_read_lock_held()andrcu_lockdep_current_cpu_online()?I think it would be better to add "notrace" to following functions:
Quoted 8 lines
Quoted 6 lines
./bpfsnoop -k 'rcu_read_*lock_*held*,rcu_lockdep_*' --show-func-proto bool
rcu_lockdep_current_cpu_online(); [traceable]
intrcu_read_lock_any_held(); [traceable]
intrcu_read_lock_bh_held(); [traceable]
intrcu_read_lock_held(); [traceable]
intrcu_read_lock_sched_held(); [traceable]Agree. Seems like an easy way to remove a footgun.
Very good, and please see below. This might or might not make the next merge window, but if not, it should be good for the one after that.
Quoted 3 lines
Independently it would be good to make noinstr/notrace to include
__cpuidlefunctions. I think right now it's allowed to attach todefault_idle()which is causing issues.
Leon, would you be interested in putting together a patch for these?
Thanx, Paul
------------------------------------------------------------------------
commit dada60c8851f19e54524cc1bcf8ab5938eb909c9
Author: Paul E. McKenney <[email protected]> Date: Thu Aug 28 10:17:10 2025 -0700
rcu: Mark diagnostic functions as notrace
The rcu_lockdep_current_cpu_online(), rcu_read_lock_sched_held(), rcu_read_lock_held(), rcu_read_lock_bh_held(), rcu_read_lock_any_held() are used by tracing-related code paths, so putting traces on them is unlikely to make anyone happy. This commit therefore marks them all "notrace".
kernel/rcu/tree.c +1 -1
@@ -4021,7 +4021,7 @@ bool rcu_cpu_online(int cpu)* RCU on an offline processor during initial boot, hence the check for * rcu_scheduler_fully_active. */ -bool rcu_lockdep_current_cpu_online (void ) +bool notrace rcu_lockdep_current_cpu_online(void) {struct rcu_data *rdp;bool ret = false;
kernel/rcu/update.c +4 -4
@@ -117,7 +117,7 @@ static bool rcu_read_lock_held_common(bool *ret)return false; } -int rcu_read_lock_sched_held (void ) +int notrace rcu_read_lock_sched_held(void) {bool ret;
@@ -342,7 +342,7 @@ EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled);* Note that rcu_read_lock() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_held (void ) +int notrace rcu_read_lock_held(void) {bool ret;
@@ -367,7 +367,7 @@ EXPORT_SYMBOL_GPL(rcu_read_lock_held);* Note that rcu_read_lock_bh() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_bh_held (void ) +int notrace rcu_read_lock_bh_held(void) {bool ret;
@@ -377,7 +377,7 @@ int rcu_read_lock_bh_held(void) }EXPORT_SYMBOL_GPL (rcu_read_lock_bh_held); -int rcu_read_lock_any_held (void ) +int notrace rcu_read_lock_any_held(void) {bool ret;
Quoted 4 lines · Paul E. McKenney
Quoted 3 lines · Alexei Starovoitov
Quoted 2 lines · Leon Hwang
On Thu Aug 28, 2025 at 7:50 PM +08, Paul E. McKenney wrote:
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
I think it would be better to add "notrace" to following functions:
./bpfsnoop -k 'rcu_read_*lock_*held*,rcu_lockdep_*' --show-func-proto bool
rcu_lockdep_current_cpu_online(); [traceable]
intrcu_read_lock_any_held(); [traceable]
intrcu_read_lock_bh_held(); [traceable]
intrcu_read_lock_held(); [traceable]
intrcu_read_lock_sched_held(); [traceable]
Quoted 8 lines
Quoted 1 line
Agree. Seems like an easy way to remove a footgun.
Very good, and please see below. This might or might not make the next merge window, but if not, it should be good for the one after that.
Quoted 3 lines
Independently it would be good to make noinstr/notrace to include
__cpuidlefunctions. I think right now it's allowed to attach todefault_idle()which is causing issues.
Nope.
./bpfsnoop -k 'default_idle' -k '__cpuidle' --show-func-proto void default_idle();
Here are the traced functions of
./bpfsnoop -D -k "htab_*_elem" --output-fgraph --fgraph-debug
--fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack':
htab_lru_map_delete_elemhtab_lru_map_lookup_and_delete_elemhtab_lru_map_lookup_elemhtab_lru_map_update_elemhtab_lru_percpu_map_lookup_and_delete_elemhtab_lru_percpu_map_lookup_elemhtab_lru_percpu_map_lookup_percpu_elemhtab_lru_percpu_map_update_elemhtab_map_delete_elemhtab_map_lookup_and_delete_elemhtab_map_lookup_elemhtab_map_seq_show_elemhtab_map_update_elemhtab_of_map_lookup_elemhtab_percpu_map_lookup_and_delete_elemhtab_percpu_map_lookup_elemhtab_percpu_map_lookup_percpu_elemhtab_percpu_map_seq_show_elemhtab_percpu_map_update_elemadd_taintalloc_htab_elemarch_scale_cpu_capacityarch_stack_walkbpf_list_head_free__bpf_obj_drop_implbpf_obj_free_fields__bpf_prog_putbpf_prog_putbpf_rb_root_freebpf_timer_cancel_and_freebpf_wq_cancel_and_freebtf_find_struct_metabtf_is_kernelbtf_type_seq_showbtf_type_seq_show_flagsbtf_type_showcheck_and_free_fieldscheck_panic_on_warncheck_timeoutclear_pending_if_disabledconsole_verbosecopy_map_value_lockedctx_sched_out__delayacct_blkio_start__do_set_cpus_alloweddo_trace_write_msrenter_lazy_tlb__folio_put__folio_unqueue_deferred_splitfree_frozen_pagesfree_htab_elemfree_huge_folioget_free_pages_noprofget_state_synchronize_rcuget_state_synchronize_rcu_fullget_tainthrtick_start_fairhrtimer_activehrtimer_cancelhrtimer_setuphrtimer_start_range_ns__htab_lru_percpu_map_update_elem__htab_map_lookup_and_delete_elem__htab_map_lookup_elemhtab_map_update_elem_in_placeinvalidate_user_asid__is_kernel_percpu_addressis_kernel_percpu_addressis_module_address__is_module_percpu_addressis_module_percpu_addressis_vmalloc_addrkallsyms_lookupkexec_crash_loadedkvfree_call_rculocal_clock__mem_cgroup_uncharge__might_sleepmm_needs_global_asidmod_node_page_state__msecs_to_jiffiesnbcon_cpu_emergency_enternbcon_cpu_emergency_exitnbcon_get_cpu_emergency_nestingpcpu_copy_valueperf_cgroup_switchperf_clear_dirty_countersperf_ctx_disableperf_ctx_enableperf_ctx_sched_task_cb__perf_event_task_sched_outperf_event_update_timeperf_event_update_userpageperf_exclude_eventperf_pmu_sched_task___perf_sw_eventperf_swevent_eventpick_next_task_fairpick_task_fairpick_task_fairpick_task_idlepreempt_model_str__printk_cpu_sync_put__printk_cpu_sync_try_get__printk_cpu_sync_waitprintk_percpu_data_readyprint_modulesprint_stop_infoprint_taintedprint_tainted_verboseprint_worker_infoprofile_hitsput_prev_entityqueue_delayed_work_onqueued_spin_lock_slowpath__queue_workqueue_work_onrcu_note_context_switchrcu_qsrcu_report_exp_cpu_multrcu_tasks_trace_qs_blkdrcu_trc_cmpxchg_need_qsresilient_queued_spin_lock_slowpathsched_balance_newidle__schedule_bug__schedule_delayed_monitor_workseq_putcseq_writeset_next_entitystack_trace_printstack_trace_saveswitch_ldtswitch_mm_irqs_offsynchronize_rcutask_h_loadtask_work_addthis_cpu_in_panictouch_all_softlockup_watchdogs__traceiter_contention_begin__traceiter_contention_end__traceiter_lock_acquire__traceiter_lock_release__traceiter_rcu_preempt_task__traceiter_sched_entry_tp__traceiter_sched_exit_tp__traceiter_tlb_flushunpin_user_page__update_context_timeupdate_rq_clock
Quoted 3 lines
Leon, would you be interested in putting together a patch for these?
I think we can apply this patch first.
If other patch is required, I would like to send it later.
Thanks,
Leon
Quoted 10 lines · Leon Hwang
Quoted 6 lines
Quoted 6 lines
Independently it would be good to make noinstr/notrace to include
__cpuidlefunctions. I think right now it's allowed to attach todefault_idle()which is causing issues.Nope.
./bpfsnoop -k 'default_idle' -k '__cpuidle' --show-func-proto voiddefault_idle();
hmm. indeed.
Jiri,
why do we still have this in selftest ?
static bool skip_entry(char *name)
{
/*
* We attach to almost all kernel functions and some of them
* will cause 'suspicious RCU usage' when fprobe is attached
* to them. Filter out the current culprits - arch_cpu_idle
* default_idle and rcu_* functions.
*/
if (!strcmp(name, "arch_cpu_idle"))
return true;
if (!strcmp(name, "default_idle"))
return true;Quoted 19 lines · Alexei Starovoitov
Quoted 10 lines · Leon Hwang
Quoted 6 lines
Quoted 6 lines
Independently it would be good to make noinstr/notrace to include
__cpuidlefunctions. I think right now it's allowed to attach todefault_idle()which is causing issues.Nope.
./bpfsnoop -k 'default_idle' -k '__cpuidle' --show-func-proto voiddefault_idle();hmm. indeed.
Jiri,
why do we still have this in selftest ?
static bool
skip_entry(char *name)
Quoted 11 lines
{
/*
* We attach to almost all kernel functions and some of them
* will cause 'suspicious RCU usage' when fprobe is attached
* to them. Filter out the current culprits - arch_cpu_idle
* default_idle and rcu_* functions.
*/
if (!strcmp(name, "arch_cpu_idle"))
return true;
if (!strcmp(name, "default_idle"))
return true;After removing "arch_cpu_idle" and "default_idle", it's OK to run the selftest:
./test_progs -t kprobe_multi_test
#155/1 kprobe_multi_test/skel_api:OK #155/2 kprobe_multi_test/link_api_addrs:OK #155/3 kprobe_multi_test/link_api_syms:OK #155/4 kprobe_multi_test/attach_api_pattern:OK #155/5 kprobe_multi_test/attach_api_addrs:OK #155/6 kprobe_multi_test/attach_api_syms:OK #155/7 kprobe_multi_test/attach_api_fails:OK #155/8 kprobe_multi_test/attach_override:OK #155/9 kprobe_multi_test/session:OK #155/10 kprobe_multi_test/session_cookie:OK #155/11 kprobe_multi_test/unique_match:OK #155/12 kprobe_multi_test/kprobe_session_return_0:OK #155/13 kprobe_multi_test/kprobe_session_return_1:OK #155/14 kprobe_multi_test/kprobe_session_return_2:OK #155 kprobe_multi_test:OK #156/1 kprobe_multi_testmod_test/testmod_attach_api_syms:OK #156/2 kprobe_multi_testmod_test/testmod_attach_api_addrs:OK #156 kprobe_multi_testmod_test:OK
Summary: 2/16 PASSED, 0 SKIPPED, 0 FAILED
Thanks,
Leon