Skip to contentSkip to conversation

WARNING in bpf_bprintf_prepare (3)

syzbot syzbot 5 participants 18 messages Oct 20, 2025 – Nov 3, 2025

Subsystems (4)
lore
Share & tools
lore

syzbot

author syzbot
#0 lore raw

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Modules linked in:
CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8
RSP: 0018:ffffc90003797840 EFLAGS: 00010293
RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005
R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000
R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42
FS:  00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0
Call Trace:
 <TASK>
 ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline]
 bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362
 bpf_prog_bfbd7bf4bf171090+0x41/0x5a
 bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline]
 __bpf_prog_run include/linux/filter.h:721 [inline]
 bpf_prog_run include/linux/filter.h:728 [inline]
 bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline]
 bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024
 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414
 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688
 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167
 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:6257 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f25b0f8efc9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9
RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a
RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003
 </TASK>

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

author syzbot
#1 lore raw

Forwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)

For archival purposes, forwarding an incoming command email to [email protected], [email protected].

For archival purposes, forwarding an incoming command email to [email protected], [email protected].

***

Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]

#syz test

net/core/flow_dissector.c +2 -0
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       preempt_disable();
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       preempt_enable();
 
         flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
         flow_keys->thoff = clamp_t(u16, flow_keys->thoff,

--
2.50.1

author syzbot
#2 lore raw

Forwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)

In reply to syzbot
For archival purposes, forwarding an incoming command email to [email protected], [email protected].

For archival purposes, forwarding an incoming command email to [email protected], [email protected].

***

Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]

Quoted 26 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

Quoted 7 lines
bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")

#syz test

net/bpf/test_run.c +3 -2
index 8b7d0b90fea7..1d0047073eda 100644
@@ -36,7 +36,7 @@ struct bpf_test_timer {
 static void bpf_test_timer_enter(struct bpf_test_timer *t)
 	__acquires(rcu)
 {
-	rcu_read_lock_dont_migrate();
+	migrate_disable();
 	t->time_start = ktime_get_ns();
 }
 
@@ -44,7 +44,8 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t)
 	__releases(rcu)
 {
 	t->time_start = 0;
-	rcu_read_unlock_migrate();
+	migrate_enable();
+	rcu_read_unlock();
 }
 
 static bool bpf_test_timer_continue(struct bpf_test_timer *t, int iterations,
Signature · 2 lines
Regards,
listout
author syzbot
#3 lore raw

Forwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)

In reply to syzbot
For archival purposes, forwarding an incoming command email to [email protected], [email protected].

For archival purposes, forwarding an incoming command email to [email protected], [email protected].

***

Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]

Quoted 26 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

Quoted 7 lines
bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")

#syz test

net/bpf/test_run.c +4 -2
index 8b7d0b90fea7..cc4288f408f2 100644
@@ -36,7 +36,8 @@ struct bpf_test_timer {
 static void bpf_test_timer_enter(struct bpf_test_timer *t)
 	__acquires(rcu)
 {
-	rcu_read_lock_dont_migrate();
+	rcu_read_lock();
+	migrate_disable();
 	t->time_start = ktime_get_ns();
 }
 
@@ -44,7 +45,8 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t)
 	__releases(rcu)
 {
 	t->time_start = 0;
-	rcu_read_unlock_migrate();
+	migrate_enable();
+	rcu_read_unlock();
 }
 
 static bool bpf_test_timer_continue(struct bpf_test_timer *t, int iterations,
Signature · 2 lines
Regards,
listout
author syzbot
#4 lore raw

Forwarded: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)

In reply to syzbot
For archival purposes, forwarding an incoming command email to [email protected], [email protected].

For archival purposes, forwarding an incoming command email to [email protected], [email protected].

***

Subject: Re: [syzbot] [bpf?] WARNING in bpf_bprintf_prepare (3)
Author: [email protected]

Quoted 26 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

Quoted 7 lines
bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")

#syz test

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
  		     (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
  	flow_keys->flags = flags;
  
+	preempt_disable();
  	result = bpf_prog_run_pin_on_cpu(prog, ctx);
+	preempt_enable();
  
  	flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
  	flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
reviewer
#5 lore raw
In reply to syzbot
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000
Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)

{
        int nest_level;

        nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
        if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                this_cpu_dec(bpf_bprintf_nest_level);
                return -EBUSY;
        }
        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

        return 0;
}

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Quoted 70 lines
Modules linked in:
CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8
RSP: 0018:ffffc90003797840 EFLAGS: 00010293
RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005
R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000
R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42
FS:  00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0
Call Trace:
  <TASK>
  ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline]
  bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362
  bpf_prog_bfbd7bf4bf171090+0x41/0x5a
  bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline]
  __bpf_prog_run include/linux/filter.h:721 [inline]
  bpf_prog_run include/linux/filter.h:728 [inline]
  bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline]
  bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024
  bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414
  bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688
  __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167
  __do_sys_bpf kernel/bpf/syscall.c:6259 [inline]
  __se_sys_bpf kernel/bpf/syscall.c:6257 [inline]
  __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257
  do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
  do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
  entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f25b0f8efc9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9
RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a
RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003
  </TASK>

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

#6 lore raw
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395

Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

bpf: Do not disable preemption in bpf_test_run().

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000
Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)

{
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
       if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
               this_cpu_dec(bpf_bprintf_nest_level);
               return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}
Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Quoted 1 line

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8

Quoted 71 lines
Quoted 70 lines
Modules linked in:
CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8
RSP: 0018:ffffc90003797840 EFLAGS: 00010293
RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005
R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000
R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42
FS:  00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0
Call Trace:
 <TASK>
 ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline]
 bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362
 bpf_prog_bfbd7bf4bf171090+0x41/0x5a
 bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline]
 __bpf_prog_run include/linux/filter.h:721 [inline]
 bpf_prog_run include/linux/filter.h:728 [inline]
 bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline]
 bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024
 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414
 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688
 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167
 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:6257 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f25b0f8efc9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9
RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a
RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003
 </TASK>

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

reviewer
#7 lore raw
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 81 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Quoted 1 line

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8

Quoted 71 lines
Quoted 70 lines
Modules linked in:
CPU: 1 UID: 0 PID: 6145 Comm: syz.4.53 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
RIP: 0010:bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
RIP: 0010:bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834
Code: ff e9 ce fe ff ff e8 10 ec e0 ff e9 be fe ff ff e8 06 ec e0 ff e9 b4 fe ff ff e8 fc eb e0 ff e9 aa fe ff ff e8 f2 eb e0 ff 90 <0f> 0b 90 65 ff 0d 27 fd b2 10 b8 f0 ff ff ff e9 17 ff ff ff e8 d8
RSP: 0018:ffffc90003797840 EFLAGS: 00010293
RAX: ffffffff81df57fe RBX: ffffc90003797a10 RCX: ffff888026493c80
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: ffffc90003797970 R08: 0000000000585870 R09: 0000000000000005
R10: dffffc0000000000 R11: fffff520006f2f20 R12: dffffc0000000000
R13: 0000000000000004 R14: 0000000000000003 R15: 1ffff920006f2f42
FS:  00005555805f5500(0000) GS:ffff888125e0c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000007c04e000 CR4: 00000000003526f0
Call Trace:
 <TASK>
 ____bpf_trace_printk kernel/trace/bpf_trace.c:372 [inline]
 bpf_trace_printk+0xdb/0x190 kernel/trace/bpf_trace.c:362
 bpf_prog_bfbd7bf4bf171090+0x41/0x5a
 bpf_dispatcher_nop_func include/linux/bpf.h:1350 [inline]
 __bpf_prog_run include/linux/filter.h:721 [inline]
 bpf_prog_run include/linux/filter.h:728 [inline]
 bpf_prog_run_pin_on_cpu include/linux/filter.h:745 [inline]
 bpf_flow_dissect+0x225/0x720 net/core/flow_dissector.c:1024
 bpf_prog_test_run_flow_dissector+0x37c/0x5c0 net/bpf/test_run.c:1414
 bpf_prog_test_run+0x2c7/0x340 kernel/bpf/syscall.c:4688
 __sys_bpf+0x562/0x860 kernel/bpf/syscall.c:6167
 __do_sys_bpf kernel/bpf/syscall.c:6259 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:6257 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6257
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f25b0f8efc9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe036cd5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f25b11e5fa0 RCX: 00007f25b0f8efc9
RDX: 0000000000000050 RSI: 0000200000000180 RDI: 000000000000000a
RBP: 00007f25b1011f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f25b11e5fa0 R14: 00007f25b11e5fa0 R15: 0000000000000003
 </TASK>

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

#8 lore raw
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
  		     (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
  	flow_keys->flags = flags;
 
+	preempt_disable();
  	result = bpf_prog_run_pin_on_cpu(prog, ctx);
+	preempt_enable();
 
  	flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
  	flow_keys->thoff = clamp_t(u16, flow_keys->thoff,

This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8](https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8)
So this approach avoids the buffer nesting issue, but re-introduces the following issue:

[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257]  bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257]  bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +5 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
 {
        int nest_level;
 
+       preempt_disable();
        nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
        if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                this_cpu_dec(bpf_bprintf_nest_level);
                return -EBUSY;
        }
        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
        return 0;
 }
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
 {
        if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                return;
+
+       preempt_disable();
        this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
 }

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():
+struct bpf_cpu_lock {
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+ .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,

                     (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
        flow_keys->flags = flags;
+       local_lock(&bpf_cpu_lock.lock);
        result = bpf_prog_run_pin_on_cpu(prog, ctx);

+ local_unlock(&bpf_cpu_lock.lock);

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

Let me know if you’d like me to run some more experiments on this.

reviewer
#9 lore raw
Quoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8](https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8)
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257]  bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Please try my above preempt_disalbe/enable() solution.

Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

#10 lore raw
Quoted 5 lines · Yonghong Song

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Please try my above preempt_disalbe/enable() solution.

I meant to look into this yesterday but got distracted with other things. I try to take a look.

Sebastian

#11 lore raw
In reply to Yonghong Song
Quoted 226 lines · Yonghong Song
Quoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257]  bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
preempt_disable();
...
bpf_try_get_buffers()
...
bpf_put_buffers()
...
preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +3 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
 {
        int nest_level;
 
+       preempt_disable();
        nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
        if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                return -EBUSY;
        }
        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
        if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                return;
        this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
 }
 

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Regards,
Sahil

#12 lore raw
Quoted 1 line · Sahil Chandna

Shall I submit a patch with your suggested changes ?

would you mind waiting a bit?

Quoted 2 lines

Regards,
Sahil

Sebastian

reviewer
#13 lore raw
In reply to Sahil Chandna
Quoted 250 lines · Sahil Chandna
Quoted 226 lines · Yonghong Song
Quoted 172 lines · Sahil Chandna
Quoted 83 lines · Yonghong Song
Quoted 68 lines · Sahil Chandna
Quoted 60 lines · Yonghong Song
Quoted 30 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d..
git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt?x=12d21de2580000
kernel config: https://syzkaller.appspot.com/x/.config?x=9ad7b090a18654a7
dashboard link: https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb
compiler:       Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=160cf542580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/873984cfc71e/vmlinux-a1e83d4c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172fe492580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=14afe492580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257]  bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
 preempt_disable();
 ...
 bpf_try_get_buffers()
 ...
 bpf_put_buffers()
 ...
 preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                 return -EBUSY;
         }

        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794

Quoted 1 line
kernel/bpf/helpers.c +0 -0
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))

For completeness, we need to add preempt_enable() here as well.

Quoted 11 lines
kernel/bpf/helpers.c +1 -0
 return;
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
 

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Please. The change looks good to me.

Quoted 3 lines

Regards,
Sahil

#14 lore raw
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler:       Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/ softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257]  bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
 preempt_disable();
 ...
 bpf_try_get_buffers()
 ...
 bpf_put_buffers()
 ...
 preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                 return -EBUSY;
         }

        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794

Quoted 1 line
kernel/bpf/helpers.c +0 -0
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))

For completeness, we need to add preempt_enable() here as well.

Quoted 5 lines
kernel/bpf/helpers.c +1 -0
 return;
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
 
Quoted 14 lines
Quoted 6 lines

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Please. The change looks good to me.

Quoted 3 lines

Regards,
Sahil

Hi Yonghong, Sahil

Previously, I removed preempt_disable from bpf_try_get_buffers, In my understanding, it is safe
to access this_cpu_inc_return(bpf_bprintf_nest_level), can we just remove the WARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=4223bf833c8495e40ae2886acbc0ecbe88fa6306

Signature · 2 lines
Best Regards
Tao Chen
reviewer
#15 lore raw
Quoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler:       Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/ softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257] bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
 preempt_disable();
 ...
 bpf_try_get_buffers()
 ...
 bpf_put_buffers()
 ...
 preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, 
 struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                 return -EBUSY;
         }

        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794

Quoted 1 line
kernel/bpf/helpers.c +0 -0
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))

For completeness, we need to add preempt_enable() here as well.

Quoted 5 lines
kernel/bpf/helpers.c +1 -0
 return;
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
 
Quoted 14 lines
Quoted 6 lines

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Please. The change looks good to me.

Quoted 3 lines

Regards,
Sahil

Quoted 7 lines

Hi Yonghong, Sahil

Previously, I removed preempt_disable from bpf_try_get_buffers, In my understanding, it is safe
to access this_cpu_inc_return(bpf_bprintf_nest_level), can we just remove the WARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?

Yes, even with migration disabled, preemption can be disabled on top of that.

Probably we can remove WARN_ON_ONCE esp. with preemption disabled. But this should be a separate patch.

Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=4223bf833c8495e40ae2886acbc0ecbe88fa6306

#16 lore raw
Quoted 291 lines · Yonghong Song
Quoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler:       Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level); return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog 
 *prog, struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/ softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257]  sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257] bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
 preempt_disable();
 ...
 bpf_try_get_buffers()
 ...
 bpf_put_buffers()
 ...
 preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog 
 *prog, struct bpf_flow_dissector *ctx,
                      (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                 return -EBUSY;
         }

        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794

Quoted 1 line
kernel/bpf/helpers.c +0 -0
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))

For completeness, we need to add preempt_enable() here as well.

Quoted 5 lines
kernel/bpf/helpers.c +1 -0
 return;
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
 
Quoted 14 lines
Quoted 6 lines

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Please. The change looks good to me.

Quoted 3 lines

Regards,
Sahil

Quoted 7 lines

Hi Yonghong, Sahil

Previously, I removed preempt_disable from bpf_try_get_buffers, In my understanding, it is safe
to access this_cpu_inc_return(bpf_bprintf_nest_level), can we just remove the WARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?

Quoted 6 lines

Yes, even with migration disabled, preemption can be disabled on top of that.

Probably we can remove WARN_ON_ONCE esp. with preemption disabled. But this should be a separate patch.

Hi Yonghong, Tao,
I printed nested level with the preempt_disable()/enable() patch and found nested level remains 1 with this patch(below). I tried this with original syzbot reproducer and ran for couple of hours.

kernel/bpf/helpers.c +1 -0
index 4be6dde89d39..657d2100f33c 100644
@@ -779,6 +779,7 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
 
        preempt_disable();
        nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
+       pr_info("bpf nest inc cpu=%d level=%d\n", smp_processor_id(), nest_level);
        if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                this_cpu_dec(bpf_bprintf_nest_level);
                preempt_enable();

I am waiting for Sebastian review on this thread before sending out a patch with preempt_disable(), Shall I also
send out patch after that for removing the WARN_ON_ONCE ?

Quoted 5 lines
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=4223bf833c8495e40ae2886acbc0ecbe88fa6306

reviewer
#17 lore raw
Quoted 295 lines · Sahil Chandna
Quoted 294 lines · Yonghong Song
Quoted 280 lines · Tao Chen
Quoted 264 lines · Yonghong Song
Quoted 252 lines · Sahil Chandna
Quoted 228 lines · Yonghong Song
Quoted 174 lines · Sahil Chandna
Quoted 85 lines · Yonghong Song
Quoted 70 lines · Sahil Chandna
Quoted 62 lines · Yonghong Song
Quoted 32 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    a1e83d4c0361 selftests/bpf: Fix redefinition of 'off' as d.. git tree:       bpf
console output: https://syzkaller.appspot.com/x/log.txt? x=12d21de2580000 kernel config: https://syzkaller.appspot.com/x/.config? x=9ad7b090a18654a7 dashboard link: https://syzkaller.appspot.com/bug? extid=b0cff308140f79a9c4cb compiler:       Debian clang version 20.1.8 (+ +20250708063551+0c9f909b7976-1~exp1~20250708183702.136),
Debian LLD 20.1.8
syz repro: https://syzkaller.appspot.com/x/repro.syz? x=160cf542580000 C reproducer: https://syzkaller.appspot.com/x/repro.c? x=128d5c58580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot- assets/2f6a7a0cd1b7/disk-a1e83d4c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot- assets/873984cfc71e/vmlinux-a1e83d4c.xz kernel image: https://storage.googleapis.com/syzbot- assets/16711d84070c/bzImage-a1e83d4c.xz

The issue was bisected to:

commit 7c33e97a6ef5d84e98b892c3e00c6d1678d20395
Author: Sahil Chandna <[email protected]>
Date:   Tue Oct 14 18:56:35 2025 +0000

    bpf: Do not disable preemption in bpf_test_run().

bisection log: https://syzkaller.appspot.com/x/bisect.txt? x=172fe492580000 final oops: https://syzkaller.appspot.com/x/report.txt? x=14afe492580000 console output: https://syzkaller.appspot.com/x/log.txt? x=10afe492580000

Quoted 7 lines

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes 7c33e97a6ef5 ("bpf: Do not disable preemption in bpf_test_run().")
------------[ cut here ]------------
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_try_get_buffers kernel/bpf/helpers.c:781 [inline]
WARNING: CPU: 1 PID: 6145 at kernel/bpf/helpers.c:781 bpf_bprintf_prepare+0x12cf/0x13a0 kernel/bpf/helpers.c:834

Okay, the warning is due to the following WARN_ON_ONCE:

static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);

int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs) {
       int nest_level;

       nest_level = this_cpu_inc_return(bpf_bprintf_nest_level); if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) { this_cpu_dec(bpf_bprintf_nest_level);
               return -EBUSY;
       }
       *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

       return 0;
}

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

Quoted 2 lines

Right, but using preempt_disable() may impact builds with CONFIG_PREEMPT_RT=y, similar to bug[1]? Do you think local_lock() could be used here

We should be okay. for all the kfuncs/helpers I mentioned below, with the help of AI, I didn't find any spin_lock in the code path and all these helpers although they try to *print* some contents, but the kfuncs/helpers itself is only to deal with buffers and actual print will happen asynchronously.

Quoted 2 lines

as nest level is per cpu variable and local lock semantics can work for both RT and non rt builds ?

Quoted 16 lines

I am not sure about local_lock() in RT as for RT, local_lock() could be nested and the release may not in proper order. See
 https://www.kernel.org/doc/html/v5.8/locking/locktypes.html

 local_lock is not suitable to protect against preemption or interrupts on a PREEMPT_RT kernel due to the PREEMPT_RT specific spinlock_t semantics.

So I suggest to stick to preempt_disable/enable approach.

Quoted 7 lines
Quoted 6 lines

There are some kfuncs/helpers need such preempt_disable protection, e.g. bpf_stream_printk, bpf_snprintf, bpf_trace_printk, bpf_trace_vprintk, bpf_seq_printf. But please double check.

Sure, thanks!

Since these helpers eventually call bpf_bprintf_prepare(), I figured adding protection around bpf_try_get_buffers(), which triggers the original warning, should be sufficient. I tried a few approaches to address the warning as below :

1. preempt_disable() / preempt_enable() around bpf_prog_run_pin_on_cpu()

net/core/flow_dissector.c +2 -0
index 1b61bb25ba0e..6a128179a26f 100644
@@ -1021,7 +1021,9 @@ u32 bpf_flow_dissect(struct bpf_prog 
 *prog, struct bpf_flow_dissector *ctx,
                (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
       flow_keys->flags = flags;
 
+    preempt_disable();
       result = bpf_prog_run_pin_on_cpu(prog, ctx);
+    preempt_enable();
 
       flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);

      flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
This fixes the original WARN_ON in both PREEMPT_FULL and RT builds.
However, when tested with the syz reproducer of the original bug [1], it
still triggers the expected DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) warning from __local_bh_disable_ip(), due to the preempt_disable() interacting with RT spinlock semantics.
[1] [https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
So this approach avoids the buffer nesting issue, but re-introduces the following issue:
[  363.968103][T21257] DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt))
[  363.968922][T21257] WARNING: CPU: 0 PID: 21257 at kernel/ softirq.c:176 __local_bh_disable_ip+0x3d9/0x540
[  363.969046][T21257] Modules linked in:
[  363.969176][T21257] Call Trace:
[  363.969181][T21257]  <TASK>
[  363.969186][T21257]  ? __local_bh_disable_ip+0xa1/0x540
[  363.969197][T21257]  ? sock_map_delete_elem+0xa2/0x170
[  363.969209][T21257]  ? preempt_schedule_common+0x83/0xd0
[  363.969252][T21257]  ? rt_spin_unlock+0x161/0x200
[  363.969269][T21257] sock_map_delete_elem+0xaf/0x170
[  363.969280][T21257] bpf_prog_464bc2be3fc7c272+0x43/0x47
[  363.969289][T21257]  bpf_flow_dissect+0x22b/0x750
[  363.969299][T21257] bpf_prog_test_run_flow_dissector+0x37c/0x5c0

2. preempt_disable() inside bpf_try_get_buffers() and bpf_put_buffers()

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..bc8630833a94 100644
@@ -777,12 +777,14 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
                 return -EBUSY;
         }
         *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
+       preempt_enable();
 
         return 0;

 }

kernel/bpf/helpers.c +3 -0
@@ -791,7 +793,10 @@ void bpf_put_buffers(void)
  {
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
                 return;
+
+       preempt_disable();
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
Quoted 2 lines

This *still* reproduces the original syz issue, so the protection needs to be placed around the entire program run, not inside the helper itself as in above experiment.

This does not work. See my earlier suggestions.

Quoted 7 lines

Basically without preempt disable, at process level, it is possible more than one process may trying to take bpf_bprintf_buffers. Adding softirq and nmi, it is totally likely to have more than 3 level for buffers. Also, more than one process with bpf_bprintf_buffers will cause problem in releasing buffers, so we need to have preempt_disable surrounding bpf_try_get_buffers() and bpf_put_buffers().

That is,
 preempt_disable();
 ...
 bpf_try_get_buffers()
 ...
 bpf_put_buffers()
 ...
 preempt_enable();

Quoted 19 lines

3. Using a per-CPU local_lock
Finally, I tested with a per-CPU local_lock around bpf_prog_run_pin_on_cpu():

kernel/bpf/helpers.c +9 -0
+struct bpf_cpu_lock {
+    local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct bpf_cpu_lock, bpf_cpu_lock) = {
+    .lock = INIT_LOCAL_LOCK(),
+};
@@ -1021,7 +1030,9 @@ u32 bpf_flow_dissect(struct bpf_prog 
 *prog, struct bpf_flow_dissector *ctx,
 (int)FLOW_DISSECTOR_F_STOP_AT_ENCAP);
         flow_keys->flags = flags;
 
+       local_lock(&bpf_cpu_lock.lock);
         result = bpf_prog_run_pin_on_cpu(prog, ctx);
+       local_unlock(&bpf_cpu_lock.lock);
 
Quoted 5 lines

This approach avoid the warning on both RT and non-RT builds, with both the syz reproducer. The intention of introducing the per-CPU local_lock is to maintain consistent per-CPU execution semantics between RT and non-RT kernels. On non-RT builds, local_lock maps to preempt_disable()/enable(),
which provides the same semantics as before.
On RT builds, it maps to an RT-safe per-CPU spinlock, avoiding the
softirq_ctrl.cnt issue.

This should work, but local lock disable interrupts which could have negative side effects on the system. We don't want this. That is the reason we have 3 nested level for bpf_bprintf_buffers.

Quoted 2 lines

Please try my above preempt_disalbe/enable() solution.

I tried following patch with reproducer from both syzbot [1] and [2] and issue *did not reproduce* with them.

kernel/bpf/helpers.c +2 -0
index 8eb117c52817..4be6dde89d39 100644
@@ -777,9 +777,11 @@ int bpf_try_get_buffers(struct 
 bpf_bprintf_buffers **bufs)
  {
         int nest_level;
 
+       preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
                 this_cpu_dec(bpf_bprintf_nest_level);
+               preempt_enable();
                 return -EBUSY;
         }

        *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);

kernel/bpf/helpers.c void bpf_put_buffers(void) · line 794

Quoted 1 line
kernel/bpf/helpers.c +0 -0
@@ -792,6 +794,7 @@ void bpf_put_buffers(void)
         if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))

For completeness, we need to add preempt_enable() here as well.

Quoted 5 lines
kernel/bpf/helpers.c +1 -0
 return;
         this_cpu_dec(bpf_bprintf_nest_level);
+       preempt_enable();
  }
 
Quoted 14 lines
Quoted 6 lines

[1] https://syzkaller.appspot.com/bug?extid=1f1fbecb9413cdbfbef8
[2] https://syzkaller.appspot.com/bug?extid=b0cff308140f79a9c4cb

Quoted 3 lines
Quoted 2 lines

Let me know if you’d like me to run some more experiments on this.

Shall I submit a patch with your suggested changes ?

Please. The change looks good to me.

Quoted 3 lines

Regards,
Sahil

Quoted 7 lines

Hi Yonghong, Sahil

Previously, I removed preempt_disable from bpf_try_get_buffers, In my understanding, it is safe
to access this_cpu_inc_return(bpf_bprintf_nest_level), can we just remove the WARN_ON_ONCE? It seems that BPF allows preemption after run under migration disabled. Is it right?

Yes, even with migration disabled, preemption can be disabled on top of that.

Quoted 7 lines
Quoted 3 lines

Probably we can remove WARN_ON_ONCE esp. with preemption disabled. But this should be a separate patch.

Hi Yonghong, Tao,
I printed nested level with the preempt_disable()/enable() patch and found nested level remains 1 with this patch(below). I tried this with original syzbot reproducer and ran for couple of hours.

It is not easy to reproduce it with preempt_disable() since the nested bpf_try_get_buffers(...) needs
process context
softirq context
nmi context
but it is hard to predict when nmi/softirq will be nested inside process context where the execution is within
(bpf_try_get_buffers, bpf_put_buffers)

Quoted 15 lines
kernel/bpf/helpers.c +1 -0
index 4be6dde89d39..657d2100f33c 100644
@@ -779,6 +779,7 @@ int bpf_try_get_buffers(struct bpf_bprintf_buffers 
 **bufs)
 
         preempt_disable();
         nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
+       pr_info("bpf nest inc cpu=%d level=%d\n", smp_processor_id(), 
 nest_level);
         if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {

                this_cpu_dec(bpf_bprintf_nest_level); preempt_enable();

Quoted 3 lines

I am waiting for Sebastian review on this thread before sending out a patch with preempt_disable(), Shall I also
send out patch after that for removing the WARN_ON_ONCE ?

If everything is correct, WARN_ON_ONCE indeed is not needed (when with preempt_disable()). But the point of WARN_ON_ONCE is to alert something may go wrong.
Not sure whether it is worthwhile to remove them or not. I think this can be a separate patch if you want to do it.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=4223bf833c8495e40ae2886acbc0ecbe88fa6306