WARNING in bpf_bprintf_prepare (3)
Subsystems (4)
syzbot
syzbotHello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz
The issue was bisected to:
commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000
bpf: Do not disable preemption in bpf_test_run().
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000
IMPORTANT: if you fix the issue, please add the following tag to the commit:
------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834 Modules linked in: CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834 Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8 RSP: 0018:ffffc90003797840 EFLAGS: 00010293 RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80 RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003 RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005 R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000 R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42 FS: 00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0 Call Trace: <TASK> ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline] bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362 bpf_prog_bfbd7bf4bf171090+0x41/0x5a bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline] __bpf_prog_run include/linux/filter.h:721 [inline] bpf_prog_run include/linux/filter.h:728 [inline] bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline] bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline] __se_sys_bpf kernel/bpf/syscall.c:6257 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f25b0f8efc9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9 RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003 </TASK>
---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection
If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.
If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)
If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report
If you want to undo deduplication, reply with:
#syz undup
syzbot
syzbotForwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
For archival purposes, forwarding an incoming command email to [email protected], [email protected].
For archival purposes, forwarding an incoming command email to [email protected], [email protected].
***
Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]
#syz test
net/core/flow_dissector.c +2 -0
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; +preempt_disable (); result =bpf_prog_run_pin_on_cpu (prog, ctx); +preempt_enable (); flow_keys->nhoff =clamp_t (u16, flow_keys->nhoff, nhoff, hlen); flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
--
2.50.1
syzbot
syzbotForwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
In reply to syzbotFor archival purposes, forwarding an incoming command email to [email protected], [email protected].
For archival purposes, forwarding an incoming command email to [email protected], [email protected].
***
Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]
Quoted 26 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().
Quoted 7 lines
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
#syz test
net/bpf/test_run.c +3 -2
@@ -36,7 +36,7 @@ struct bpf_test_timer {static void bpf_test_timer_enter (struct bpf_test_timer *t)__acquires (rcu) { -rcu_read_lock_dont_migrate (); +migrate_disable (); t->time_start =ktime_get_ns (); }
@@ -44,7 +44,8 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t)__releases (rcu) { t->time_start =0 ; -rcu_read_unlock_migrate (); +migrate_enable (); +rcu_read_unlock (); } static bool bpf_test_timer_continue(struct bpf_test_timer *t, int iterations,
Signature · 2 lines
Regards, listout
syzbot
syzbotForwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
In reply to syzbotFor archival purposes, forwarding an incoming command email to [email protected], [email protected].
For archival purposes, forwarding an incoming command email to [email protected], [email protected].
***
Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]
Quoted 26 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().
Quoted 7 lines
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
#syz test
net/bpf/test_run.c +4 -2
@@ -36,7 +36,8 @@ struct bpf_test_timer {static void bpf_test_timer_enter (struct bpf_test_timer *t)__acquires (rcu) { -rcu_read_lock_dont_migrate (); +rcu_read_lock (); +migrate_disable (); t->time_start =ktime_get_ns (); }
@@ -44,7 +45,8 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t)__releases (rcu) { t->time_start =0 ; -rcu_read_unlock_migrate (); +migrate_enable (); +rcu_read_unlock (); } static bool bpf_test_timer_continue(struct bpf_test_timer *t, int iterations,
Signature · 2 lines
Regards, listout
syzbot
syzbotForwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
In reply to syzbotFor archival purposes, forwarding an incoming command email to [email protected], [email protected].
For archival purposes, forwarding an incoming command email to [email protected], [email protected].
***
Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]
Quoted 26 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().
Quoted 7 lines
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
#syz test
net/core/flow_dissector.c +2 -0
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; +preempt_disable (); result =bpf_prog_run_pin_on_cpu (prog, ctx); +preempt_enable (); flow_keys->nhoff =clamp_t (u16, flow_keys->nhoff, nhoff, hlen); flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
Yonghong Song
reviewerQuoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000
Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Okay, the warning is due to the following WARN_ON_ONCE:
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().
There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.
Quoted 70 lines
Modules linked in: CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834 Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8 RSP: 0018:ffffc90003797840 EFLAGS: 00010293 RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80 RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003 RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005 R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000 R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42 FS: 00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0 Call Trace: <TASK> ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline] bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362 bpf_prog_bfbd7bf4bf171090+0x41/0x5a bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline] __bpf_prog_run include/linux/filter.h:721 [inline] bpf_prog_run include/linux/filter.h:728 [inline] bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline] bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline] __se_sys_bpf kernel/bpf/syscall.c:6257 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f25b0f8efc9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9 RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003 </TASK>---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisectionIf the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-titleIf you want syzbot to run the reproducer, reply with:
#syz test:git://repo/address.gitbranch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-reportIf you want to undo deduplication, reply with:
#syz undup
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395Author: Sahil Chandna <[email protected]> Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){ int nest_level; nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); return 0; }
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().
Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.
Sure, thanks!
Quoted 1 line
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
Quoted 71 lines
Quoted 70 lines
Modules linked in: CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834 Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8 RSP: 0018:ffffc90003797840 EFLAGS: 00010293 RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80 RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003 RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005 R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000 R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42 FS: 00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0 Call Trace: <TASK> ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline] bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362 bpf_prog_bfbd7bf4bf171090+0x41/0x5a bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline] __bpf_prog_run include/linux/filter.h:721 [inline] bpf_prog_run include/linux/filter.h:728 [inline] bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline] bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline] __se_sys_bpf kernel/bpf/syscall.c:6257 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f25b0f8efc9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9 RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003 </TASK>---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisectionIf the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-titleIf you want syzbot to run the reproducer, reply with:
#syz test:git://repo/address.gitbranch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-reportIf you want to undo deduplication, reply with:
#syz undup
Yonghong Song
reviewerQuoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().
Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used here
We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.
Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.html
local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.
So I suggest to stick to preempt_disable/enable approach.
Quoted 81 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Quoted 1 line
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
Quoted 71 lines
Quoted 70 lines
Modules linked in: CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834 Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8 RSP: 0018:ffffc90003797840 EFLAGS: 00010293 RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80 RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003 RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005 R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000 R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42 FS: 00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0 Call Trace: <TASK> ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline] bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362 bpf_prog_bfbd7bf4bf171090+0x41/0x5a bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline] __bpf_prog_run include/linux/filter.h:721 [inline] bpf_prog_run include/linux/filter.h:728 [inline] bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline] bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline] __se_sys_bpf kernel/bpf/syscall.c:6257 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f25b0f8efc9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9 RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003 </TASK>---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisectionIf the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-titleIf you want syzbot to run the reproducer, reply with:
#syz test:git://repo/address.gitbranch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-reportIf you want to undo deduplication, reply with:
#syz undup
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.html
local_lockis not suitable to protect against preemption or interrupts on aPREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :
1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()
net/core/flow_dissector.c +2 -0
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; +preempt_disable (); result =bpf_prog_run_pin_on_cpu (prog, ctx); +preempt_enable (); flow_keys->nhoff =clamp_t (u16, flow_keys->nhoff, nhoff, hlen); flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8](https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8)
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) [ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/softirq.c:176 __local_bh_disable_ip+0x3d9/0x540 [ 363.969046][T21257] Modules linked in: [ 363.969176][T21257] Call Trace: [ 363.969181][T21257] <TASK> [ 363.969186][T21257] ? __local_bh_disable_ip+0xa1/0x540 [ 363.969197][T21257] ? sock_map_delete_elem+0xa2/0x170 [ 363.969209][T21257] ? preempt_schedule_common+0x83/0xd0 [ 363.969252][T21257] ? rt_spin_unlock+0x161/0x200 [ 363.969269][T21257] sock_map_delete_elem+0xaf/0x170 [ 363.969280][T21257] bpf_prog_464bc2be3fc7c272+0x43/0x47 [ 363.969289][T21257] bpf_flow_dissect+0x22b/0x750 [ 363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0
2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()
kernel/bpf/helpers.c +5 -0
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {int nest_level; +preempt_disable (); nest_level =this_cpu_inc_return (bpf_bprintf_nest_level);if (WARN_ON_ONCE (nest_level >MAX_BPRINTF_NEST_LEVEL )) {this_cpu_dec (bpf_bprintf_nest_level);return -EBUSY ; } *bufs =this_cpu_ptr (&bpf_bprintf_bufs[nest_level -1 ]); +preempt_enable ();return 0 ; }
@@ -791,7 +793,10 @@ void bpf_put_buffers(void) {if (WARN_ON_ONCE (this_cpu_read (bpf_bprintf_nest_level) ==0 ))return ; + +preempt_disable ();this_cpu_dec (bpf_bprintf_nest_level); +preempt_enable (); }
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():
+struct bpf_cpu_lock {
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+ .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
(int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
flow_keys->flags = flags;+ local_lock(&bpf_cpu_lock.lock);
result = bpf_prog_run_pin_on_cpu(prog, ctx);+ local_unlock(&bpf_cpu_lock.lock);
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.
Let me know if you’d like me to run some more experiments on this.
Yonghong Song
reviewerQuoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8](https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8)
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }
Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().
That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();
Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);
Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.
This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.
Please try my above preempt_disalbe/enable() solution.
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Sebastian Andrzej Siewior
maintainerQuoted 5 lines · Yonghong Song
This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Please try my above preempt_disalbe/enable() solution.
I meant to look into this yesterday but got distracted with other things. I try to take a look.
Sebastian
Quoted 226 lines · Yonghong Song
Quoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.
Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +3 -0
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {int nest_level; +preempt_disable (); nest_level =this_cpu_inc_return (bpf_bprintf_nest_level);if (WARN_ON_ONCE (nest_level >MAX_BPRINTF_NEST_LEVEL )) {this_cpu_dec (bpf_bprintf_nest_level); +preempt_enable ();return -EBUSY ; } *bufs =this_cpu_ptr (&bpf_bprintf_bufs[nest_level -1 ]);
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)if (WARN_ON_ONCE (this_cpu_read (bpf_bprintf_nest_level) ==0 ))return ;this_cpu_dec (bpf_bprintf_nest_level); +preempt_enable (); }
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
Quoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Regards,
Sahil
Sebastian Andrzej Siewior
maintainerQuoted 1 line · Sahil Chandna
Shall I submit a patch with your suggested changes ?
would you mind waiting a bit?
Quoted 2 lines
Regards,
Sahil
Sebastian
Yonghong Song
reviewerQuoted 250 lines · Sahil Chandna
Quoted 226 lines · Yonghong Song
Quoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d..
git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +2
-0@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); return -EBUSY; }*bufs =
this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -792,6 +794,7 @@ void bpf_put_buffers(void) if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
For completeness, we need to add preempt_enable() here as well.
Quoted 11 lines
kernel/bpf/helpers.c +1
-0return ; this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cbQuoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Please. The change looks good to me.
Quoted 3 lines
Regards,
Sahil
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000
Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +2
-0@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); return -EBUSY; }*bufs =
this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -792,6 +794,7 @@ void bpf_put_buffers(void) if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))For completeness, we need to add
preempt_enable()here as well.Quoted 5 lines
kernel/bpf/helpers.c +1
-0return ; this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }
Quoted 14 lines
Quoted 6 lines
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cbQuoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Please. The change looks good to me.
Quoted 3 lines
Regards,
Sahil
Hi Yonghong, Sahil
Previously, I removed preempt_disable from bpf_try_get_buffers, In my understanding, it is safe
to access this_cpu_inc_return(bpf_bprintf_nest_level), can we just remove the WARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?
Signature · 2 lines
Best Regards Tao Chen
Yonghong Song
reviewerQuoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000
Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog,struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +2
-0@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); return -EBUSY; }*bufs =
this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -792,6 +794,7 @@ void bpf_put_buffers(void) if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))For completeness, we need to add
preempt_enable()here as well.Quoted 5 lines
kernel/bpf/helpers.c +1
-0return ; this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 14 lines
Quoted 6 lines
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cbQuoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Please. The change looks good to me.
Quoted 3 lines
Regards,
Sahil
Quoted 7 lines
Hi Yonghong, Sahil
Previously, I removed
preempt_disablefrombpf_try_get_buffers, In my understanding, it is safe
to accessthis_cpu_inc_return(bpf_bprintf_nest_level), can we just remove theWARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?
Yes, even with migration disabled, preemption can be disabled on top of that.
Probably we can remove WARN_ON_ONCE esp. with preemption disabled. But this should be a separate patch.
Quoted 291 lines · Yonghong Song
Quoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000
Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level); return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +2
-0@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); return -EBUSY; }*bufs =
this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -792,6 +794,7 @@ void bpf_put_buffers(void) if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))For completeness, we need to add
preempt_enable()here as well.Quoted 5 lines
kernel/bpf/helpers.c +1
-0return ; this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 14 lines
Quoted 6 lines
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cbQuoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Please. The change looks good to me.
Quoted 3 lines
Regards,
SahilQuoted 7 lines
Hi Yonghong, Sahil
Previously, I removed
preempt_disablefrombpf_try_get_buffers, In my understanding, it is safe
to accessthis_cpu_inc_return(bpf_bprintf_nest_level), can we just remove theWARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?
Quoted 6 lines
Yes, even with migration disabled, preemption can be disabled on top of that.
Probably we can remove
WARN_ON_ONCEesp. with preemption disabled. But this should be a separate patch.
Hi Yonghong, Tao,
I printed nested level with the preempt_disable()/enable() patch and found nested level remains 1 with this patch(below). I tried this with original syzbot reproducer and ran for couple of hours.
kernel/bpf/helpers.c +1 -0
@@ -779,6 +779,7 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)preempt_disable (); nest_level =this_cpu_inc_return (bpf_bprintf_nest_level); +pr_info ("bpf nest inc cpu=%d level=%d\n" ,smp_processor_id (), nest_level);if (WARN_ON_ONCE (nest_level >MAX_BPRINTF_NEST_LEVEL )) {this_cpu_dec (bpf_bprintf_nest_level);preempt_enable ();
I am waiting for Sebastian review on this thread before sending out a patch with preempt_disable(), Shall I also
send out patch after that for removing the WARN_ON_ONCE ?
Yonghong Song
reviewerQuoted 295 lines · Sahil Chandna
Quoted 294 lines · Yonghong Song
Quoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit:
a1e83d4c0361selftests/bpf: Fix redefinition of 'off' as d.. git tree: bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler: Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xzThe issue was bisected to:
commit
7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date: Tue Oct 14 18:56:35 2025 +0000bpf: Do not disable preemption in
bpf_test_run().bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000
Quoted 7 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")------------[ cut here ]------------ WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline] WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834Okay, the warning is due to the following
WARN_ON_ONCE:static
DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); staticDEFINE_PER_CPU(int, bpf_bprintf_nest_level);int
bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs){
intnest_level;
nest_level=this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {this_cpu_dec(bpf_bprintf_nest_level);
return-EBUSY;
}
*bufs =this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);return 0;
}Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().Quoted 2 lines
Right, but using
preempt_disable()may impact builds withCONFIG_PREEMPT_RT=y, similar to bug[1]? Do you thinklocal_lock()could be used hereWe should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any
spin_lockin the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.Quoted 2 lines
as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?
Quoted 16 lines
I am not sure about
local_lock()in RT as for RT,local_lock()could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.htmllocal_lock is not suitable to protect against preemption or interrupts on a
PREEMPT_RTkernel due to thePREEMPT_RTspecificspinlock_tsemantics.So I suggest to stick to preempt_disable/enable approach.
Quoted 7 lines
Quoted 6 lines
There are some kfuncs/helpers need such
preempt_disableprotection, e.g.bpf_stream_printk,bpf_snprintf,bpf_trace_printk,bpf_trace_vprintk,bpf_seq_printf. But please double check.Sure, thanks!
Since these helpers eventually call
bpf_bprintf_prepare(), I figured adding protection aroundbpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :1.
preempt_disable()/preempt_enable()aroundbpf_prog_run_pin_on_cpu()net/core/flow_dissector.c +2
-0@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + preempt_disable(); result = bpf_prog_run_pin_on_cpu(prog, ctx); + preempt_enable(); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff= clamp_t(u16,flow_keys->thoff,
This fixes the originalWARN_ONin bothPREEMPT_FULLand RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from__local_bh_disable_ip(), due to thepreempt_disable()interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[ 363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[ 363.968922][T21257] WARNING: CPU: 0 PID: 21257 atkernel/softirq.c:176__local_bh_disable_ip+0x3d9/0x540
[ 363.969046][T21257] Modules linked in:
[ 363.969176][T21257] Call Trace:
[ 363.969181][T21257] <TASK>
[ 363.969186][T21257] ?__local_bh_disable_ip+0xa1/0x540
[ 363.969197][T21257] ?sock_map_delete_elem+0xa2/0x170
[ 363.969209][T21257] ?preempt_schedule_common+0x83/0xd0
[ 363.969252][T21257] ?rt_spin_unlock+0x161/0x200
[ 363.969269][T21257]sock_map_delete_elem+0xaf/0x170
[ 363.969280][T21257]bpf_prog_464bc2be3fc7c272+0x43/0x47
[ 363.969289][T21257]bpf_flow_dissect+0x22b/0x750
[ 363.969299][T21257]bpf_prog_test_run_flow_dissector+0x37c/0x5c02.
preempt_disable()insidebpf_try_get_buffers()andbpf_put_buffers()kernel/bpf/helpers.c +2
-0@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY; } *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]); + preempt_enable(); return 0;}
kernel/bpf/helpers.c +3
-0@@ -791,7 +793,10 @@ void bpf_put_buffers(void) { if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0)) return; + + preempt_disable(); this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 2 lines
This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.
This does not work. See my earlier suggestions.
Quoted 7 lines
Basically without preempt disable, at process level, it is possible more than one process may trying to take
bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process withbpf_bprintf_bufferswill cause problem in releasing buffers, so we need to havepreempt_disablesurroundingbpf_try_get_buffers()andbpf_put_buffers().That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();Quoted 19 lines
3. Using a per-CPU
local_lock
Finally, I tested with a per-CPUlocal_lockaroundbpf_prog_run_pin_on_cpu():kernel/bpf/helpers.c +9
-0+struct bpf_cpu_lock { + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = { + .lock = INIT_LOCAL_LOCK(), +};@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP); flow_keys->flags = flags; + local_lock(&bpf_cpu_lock.lock); result = bpf_prog_run_pin_on_cpu(prog, ctx); + local_unlock(&bpf_cpu_lock.lock);Quoted 5 lines
This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU
local_lockis to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds,local_lockmaps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for
bpf_bprintf_buffers.Quoted 2 lines
Please try my above preempt_disalbe/enable() solution.
I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.
kernel/bpf/helpers.c +2
-0@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) { int nest_level; + preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); return -EBUSY; }*bufs =
this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -792,6 +794,7 @@ void bpf_put_buffers(void) if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))For completeness, we need to add
preempt_enable()here as well.Quoted 5 lines
kernel/bpf/helpers.c +1
-0return ; this_cpu_dec(bpf_bprintf_nest_level); + preempt_enable(); }Quoted 14 lines
Quoted 6 lines
[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cbQuoted 3 lines
Quoted 2 lines
Let me know if you’d like me to run some more experiments on this.
Shall I submit a patch with your suggested changes ?
Please. The change looks good to me.
Quoted 3 lines
Regards,
SahilQuoted 7 lines
Hi Yonghong, Sahil
Previously, I removed
preempt_disablefrombpf_try_get_buffers, In my understanding, it is safe
to accessthis_cpu_inc_return(bpf_bprintf_nest_level), can we just remove theWARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?Yes, even with migration disabled, preemption can be disabled on top of that.
Quoted 7 lines
Quoted 3 lines
Probably we can remove
WARN_ON_ONCEesp. with preemption disabled. But this should be a separate patch.Hi Yonghong, Tao,
I printed nested level with the preempt_disable()/enable() patch and found nested level remains 1 with this patch(below). I tried this with original syzbot reproducer and ran for couple of hours.
It is not easy to reproduce it with preempt_disable() since the nested bpf_try_get_buffers(...) needs
process context
softirq context
nmi context
but it is hard to predict when nmi/softirq will be nested inside process context where the execution is within
(bpf_try_get_buffers, bpf_put_buffers)
Quoted 15 lines
kernel/bpf/helpers.c +1
-0@@ -779,6 +779,7 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) preempt_disable(); nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); + pr_info("bpf nest inc cpu=%d level=%d\n", smp_processor_id(), nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);preempt_enable();
Quoted 3 lines
I am waiting for Sebastian review on this thread before sending out a patch with
preempt_disable(), Shall I also
send out patch after that for removing theWARN_ON_ONCE?
If everything is correct, WARN_ON_ONCE indeed is not needed (when with preempt_disable()). But the point of WARN_ON_ONCE is to alert something may go wrong.
Not sure whether it is worthwhile to remove them or not. I think this can be a separate patch if you want to do it.