Skip to contentSkip to conversation

possible deadlock in nf_tables_dumpreset_obj

syzbot syzbot 4 participants 7 messages Dec 20, 2025 – Dec 23, 2025

Subsystems (3)
Networking Network drivers netfilter
lore
Share & tools
lore

syzbot

author syzbot
#0 lore raw

Hello,

syzbot found the following issue on:

HEAD commit:    8f0b4cce4481 Linux 6.19-rc1
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=104f2d92580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=a11e0f726bfb6765
dashboard link: https://syzkaller.appspot.com/bug?extid=ff16b505ec9152e5f448
compiler:       gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/d900f083ada3/non_bootable_disk-8f0b4cce.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/64c9a36f3f29/vmlinux-8f0b4cce.xz
kernel image: https://storage.googleapis.com/syzbot-assets/27a5e8a8a4b8/bzImage-8f0b4cce.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]

======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
syz.3.970/9330 is trying to acquire lock:
ffff888012d4ccd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491

but task is already holding lock:
ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}:

       __mutex_lock_common kernel/locking/mutex.c:614 [inline]
       __mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
       __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404
       netlink_dump_start include/linux/netlink.h:341 [inline]
       ip_set_dump+0x17f/0x210 net/netfilter/ipset/ip_set_core.c:1717
       nfnetlink_rcv_msg+0x9fc/0x1200 net/netfilter/nfnetlink.c:302
       netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
       nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
       netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
       netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
       netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
       sock_sendmsg_nosec net/socket.c:727 [inline]
       __sock_sendmsg net/socket.c:742 [inline]
       ____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
       ___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
       __sys_sendmsg+0x16d/0x220 net/socket.c:2678
       do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
       do_syscall_64+0xcd/0xf80 arch/x86/entry/syscall_64.c:94
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #1 (nfnl_subsys_ipset){+.+.}-{4:4}:

       __mutex_lock_common kernel/locking/mutex.c:614 [inline]
       __mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
       ip_set_nfnl_get_byindex+0x7c/0x290 net/netfilter/ipset/ip_set_core.c:909
       set_target_v1_checkentry+0x1ac/0x570 net/netfilter/xt_set.c:313
       xt_check_target+0x27c/0xa40 net/netfilter/x_tables.c:1038
       nft_target_init+0x459/0x7d0 net/netfilter/nft_compat.c:267
       nf_tables_newexpr net/netfilter/nf_tables_api.c:3527 [inline]
       nf_tables_newrule+0xedd/0x2910 net/netfilter/nf_tables_api.c:4358
       nfnetlink_rcv_batch+0x190d/0x2350 net/netfilter/nfnetlink.c:526
       nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:649 [inline]
       nfnetlink_rcv+0x3c1/0x430 net/netfilter/nfnetlink.c:667
       netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
       netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
       netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
       sock_sendmsg_nosec net/socket.c:727 [inline]
       __sock_sendmsg net/socket.c:742 [inline]
       ____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
       ___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
       __sys_sendmsg+0x16d/0x220 net/socket.c:2678
       do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
       do_syscall_64+0xcd/0xf80 arch/x86/entry/syscall_64.c:94
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&nft_net->commit_mutex){+.+.}-{4:4}:

       check_prev_add kernel/locking/lockdep.c:3165 [inline]
       check_prevs_add kernel/locking/lockdep.c:3284 [inline]
       validate_chain kernel/locking/lockdep.c:3908 [inline]
       __lock_acquire+0x1669/0x2890 kernel/locking/lockdep.c:5237
       lock_acquire kernel/locking/lockdep.c:5868 [inline]
       lock_acquire+0x179/0x330 kernel/locking/lockdep.c:5825
       __mutex_lock_common kernel/locking/mutex.c:614 [inline]
       __mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
       nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491
       netlink_dump+0x539/0xd30 net/netlink/af_netlink.c:2325
       __netlink_dump_start+0x6d6/0x990 net/netlink/af_netlink.c:2440
       netlink_dump_start include/linux/netlink.h:341 [inline]
       nft_netlink_dump_start_rcu+0x81/0x1f0 net/netfilter/nf_tables_api.c:1286
       nf_tables_getobj_reset+0x56b/0x6b0 net/netfilter/nf_tables_api.c:8626
       nfnetlink_rcv_msg+0x583/0x1200 net/netfilter/nfnetlink.c:290
       netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
       nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
       netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
       netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
       netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
       sock_sendmsg_nosec net/socket.c:727 [inline]
       __sock_sendmsg net/socket.c:742 [inline]
       ____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
       ___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
       __sys_sendmsg+0x16d/0x220 net/socket.c:2678
       do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
       do_syscall_64+0xcd/0xf80 arch/x86/entry/syscall_64.c:94
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

Chain exists of:
  &nft_net->commit_mutex --> nfnl_subsys_ipset --> nlk_cb_mutex-NETFILTER

 Possible unsafe locking scenario:
       CPU0                    CPU1
       ----                    ----
  lock(nlk_cb_mutex-NETFILTER);
                               lock(nfnl_subsys_ipset);
                               lock(nlk_cb_mutex-NETFILTER);

lock(&nft_net->commit_mutex);

*** DEADLOCK ***

1 lock held by syz.3.970/9330:
#0: ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

stack backtrace:
CPU: 0 UID: 0 PID: 9330 Comm: syz.3.970 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:120
 print_circular_bug+0x275/0x340 kernel/locking/lockdep.c:2043
 check_noncircular+0x146/0x160 kernel/locking/lockdep.c:2175
 check_prev_add kernel/locking/lockdep.c:3165 [inline]
 check_prevs_add kernel/locking/lockdep.c:3284 [inline]
 validate_chain kernel/locking/lockdep.c:3908 [inline]
 __lock_acquire+0x1669/0x2890 kernel/locking/lockdep.c:5237
 lock_acquire kernel/locking/lockdep.c:5868 [inline]
 lock_acquire+0x179/0x330 kernel/locking/lockdep.c:5825
 __mutex_lock_common kernel/locking/mutex.c:614 [inline]
 __mutex_lock+0x1aa/0x1ca0 kernel/locking/mutex.c:776
 nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491
 netlink_dump+0x539/0xd30 net/netlink/af_netlink.c:2325
 __netlink_dump_start+0x6d6/0x990 net/netlink/af_netlink.c:2440
 netlink_dump_start include/linux/netlink.h:341 [inline]
 nft_netlink_dump_start_rcu+0x81/0x1f0 net/netfilter/nf_tables_api.c:1286
 nf_tables_getobj_reset+0x56b/0x6b0 net/netfilter/nf_tables_api.c:8626
 nfnetlink_rcv_msg+0x583/0x1200 net/netfilter/nfnetlink.c:290
 netlink_rcv_skb+0x158/0x420 net/netlink/af_netlink.c:2550
 nfnetlink_rcv+0x1b3/0x430 net/netfilter/nfnetlink.c:669
 netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
 netlink_unicast+0x5aa/0x870 net/netlink/af_netlink.c:1344
 netlink_sendmsg+0x8c8/0xdd0 net/netlink/af_netlink.c:1894
 sock_sendmsg_nosec net/socket.c:727 [inline]
 __sock_sendmsg net/socket.c:742 [inline]
 ____sys_sendmsg+0xa5d/0xc30 net/socket.c:2592
 ___sys_sendmsg+0x134/0x1d0 net/socket.c:2646
 __sys_sendmsg+0x16d/0x220 net/socket.c:2678
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xcd/0xf80 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fb7e7b8f7c9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fb7e8a9c038 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007fb7e7de5fa0 RCX: 00007fb7e7b8f7c9
RDX: 0000000004004004 RSI: 0000200000000140 RDI: 0000000000000003
RBP: 00007fb7e7c13f91 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fb7e7de6038 R14: 00007fb7e7de5fa0 R15: 00007fffe518fab8
 </TASK>

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

maintainer
#1 lore raw
Quoted 7 lines · syzbot

syz.3.970/9330 is trying to acquire lock:
ffff888012d4ccd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491

but task is already holding lock:
ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.

I think this is a real bug:

CPU0: 'nft reset'.
CPU1: 'ipset list' (anything in ipset doing a netlink dump op)
CPU2: 'iptables-nft -A ... -m set ...'

... can result in:

CPU0                    CPU1                            CPU2
----                    ----                            ----
lock(nlk_cb_mutex-NETFILTER);
                        lock(nfnl_subsys_ipset);
                                                       lock(&nft_net->commit_mutex);
                        lock(nlk_cb_mutex-NETFILTER);
                                                       lock(nfnl_subsys_ipset);

lock(&nft_net->commit_mutex);

CPU0 is waiting for CPU2 to release transaction mutex. CPU1 is waiting for CPU0 to release the netlink dump mutex CPU2 is waiting for CPU1 to release the ipset subsys mutex

This bug was added when 'nft reset' started to grab the transaction mutex from the dump callback path in nf_tables.

Not yet sure how to avoid it.
Maybe we could get rid of 'lock(nfnl_subsys_ipset);' from the xt_set module call paths.

Or add a new lock (spinlock?) to protect the 'reset' object info instead of using the transaction mutex.

I haven't given it much thought yet and will likely not investigate further for the next two weeks.

maintainer
#2 lore raw
Quoted 10 lines · Florian Westphal
Quoted 7 lines · syzbot

syz.3.970/9330 is trying to acquire lock:
ffff888012d4ccd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491

but task is already holding lock:
ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.

I think this is a real bug:

Yes, I think so too, it was a bad idea to use the commit_mutex for this.

Quoted 31 lines

CPU0: 'nft reset'.
CPU1: 'ipset list' (anything in ipset doing a netlink dump op)
CPU2: 'iptables-nft -A ... -m set ...'

... can result in:

CPU0                    CPU1                            CPU2
----                    ----                            ----
lock(nlk_cb_mutex-NETFILTER);
                        lock(nfnl_subsys_ipset);
                                                       lock(&nft_net->commit_mutex);
                        lock(nlk_cb_mutex-NETFILTER);
                                                       lock(nfnl_subsys_ipset);

lock(&nft_net->commit_mutex);

CPU0 is waiting for CPU2 to release transaction mutex. CPU1 is waiting for CPU0 to release the netlink dump mutex CPU2 is waiting for CPU1 to release the ipset subsys mutex

This bug was added when 'nft reset' started to grab the transaction mutex from the dump callback path in nf_tables.

Not yet sure how to avoid it.
Maybe we could get rid of 'lock(nfnl_subsys_ipset);' from the xt_set module call paths.

Or add a new lock (spinlock?) to protect the 'reset' object info instead of using the transaction mutex.

I haven't given it much thought yet and will likely not investigate further for the next two weeks.

maintainer
#3 lore raw

Sorry, I pressed sent too fast... see below.

Quoted 5 lines · Pablo Neira Ayuso
Quoted 4 lines · Florian Westphal
Quoted 3 lines · syzbot

syz.3.970/9330 is trying to acquire lock:
ffff888012d4ccd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491

Quoted 23 lines
Quoted 6 lines
Quoted 4 lines

but task is already holding lock:
ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.

I think this is a real bug:

Yes, I think so too, it was a bad idea to use the commit_mutex for this.

Quoted 14 lines

CPU0: 'nft reset'.
CPU1: 'ipset list' (anything in ipset doing a netlink dump op)
CPU2: 'iptables-nft -A ... -m set ...'

... can result in:

CPU0                    CPU1                            CPU2
----                    ----                            ----
lock(nlk_cb_mutex-NETFILTER);
                        lock(nfnl_subsys_ipset);
                                                       lock(&nft_net->commit_mutex);
                        lock(nlk_cb_mutex-NETFILTER);
                                                       lock(nfnl_subsys_ipset);

lock(&nft_net->commit_mutex);

Would it work to use a separated mutex for reset itself?

Quoted 18 lines
Quoted 16 lines

CPU0 is waiting for CPU2 to release transaction mutex. CPU1 is waiting for CPU0 to release the netlink dump mutex CPU2 is waiting for CPU1 to release the ipset subsys mutex

This bug was added when 'nft reset' started to grab the transaction mutex from the dump callback path in nf_tables.

Not yet sure how to avoid it.
Maybe we could get rid of 'lock(nfnl_subsys_ipset);' from the xt_set module call paths.

Or add a new lock (spinlock?) to protect the 'reset' object info instead of using the transaction mutex.

I haven't given it much thought yet and will likely not investigate further for the next two weeks.

maintainer
#4 lore raw
Quoted 15 lines · Pablo Neira Ayuso
Quoted 6 lines
Quoted 6 lines

CPU0: 'nft reset'.
CPU1: 'ipset list' (anything in ipset doing a netlink dump op)
CPU2: 'iptables-nft -A ... -m set ...'

... can result in:

Quoted 8 lines
Quoted 8 lines
CPU0                    CPU1                            CPU2
----                    ----                            ----
lock(nlk_cb_mutex-NETFILTER);
                        lock(nfnl_subsys_ipset);
                                                       lock(&nft_net->commit_mutex);
                        lock(nlk_cb_mutex-NETFILTER);
                                                       lock(nfnl_subsys_ipset);

lock(&nft_net->commit_mutex);

Quoted 1 line

Would it work to use a separated mutex for reset itself?

I think so, yes, its only job is to prevent concurrent reset actions, the objects themselves are protected by rcu.

Parallel add/removal should be fine.

#5 lore raw
In reply to Florian Westphal

Hi,

Quoted 30 lines · Florian Westphal
Quoted 7 lines · syzbot

syz.3.970/9330 is trying to acquire lock:
ffff888012d4ccd8 (&nft_net->commit_mutex){+.+.}-{4:4}, at: nf_tables_dumpreset_obj+0x6f/0xa0 net/netfilter/nf_tables_api.c:8491

but task is already holding lock:
ffff88802bce36f0 (nlk_cb_mutex-NETFILTER){+.+.}-{4:4}, at: __netlink_dump_start+0x150/0x990 net/netlink/af_netlink.c:2404

which lock already depends on the new lock.

I think this is a real bug:

CPU0: 'nft reset'.
CPU1: 'ipset list' (anything in ipset doing a netlink dump op)
CPU2: 'iptables-nft -A ... -m set ...'

... can result in:

CPU0                    CPU1                            CPU2
----                    ----                            ----
lock(nlk_cb_mutex-NETFILTER);
                        lock(nfnl_subsys_ipset);
                                                       lock(&nft_net->commit_mutex);
                        lock(nlk_cb_mutex-NETFILTER);
                                                       lock(nfnl_subsys_ipset);

lock(&nft_net->commit_mutex);

CPU0 is waiting for CPU2 to release transaction mutex. CPU1 is waiting for CPU0 to release the netlink dump mutex CPU2 is waiting for CPU1 to release the ipset subsys mutex

Quoted 6 lines

This bug was added when 'nft reset' started to grab the transaction mutex from the dump callback path in nf_tables.

Not yet sure how to avoid it.
Maybe we could get rid of 'lock(nfnl_subsys_ipset);' from the xt_set module call paths.

I don't know how calling it could be avoided: userspace commands (ipset + iptables checkentry using ipset match/target) are serialized by nfnl_subsys_ipset.

Is there a way to force acquiring nlk_cb_mutex-NETFILTER first and then nfnl_subsys_ipset when doing a netlink dump?

Quoted 5 lines

Or add a new lock (spinlock?) to protect the 'reset' object info instead of using the transaction mutex.

I haven't given it much thought yet and will likely not investigate further for the next two weeks.

Best regards,
Jozsef

Signature · 3 lines
E-mail : [email protected], [email protected], [email protected]
Address: Wigner Research Centre for Physics
         H-1525 Budapest 114, POB. 49, Hungary
maintainer
#6 lore raw
Quoted 7 lines · Jozsef Kadlecsik
Quoted 3 lines

Not yet sure how to avoid it.
Maybe we could get rid of 'lock(nfnl_subsys_ipset);' from the xt_set module call paths.

I don't know how calling it could be avoided: userspace commands (ipset + iptables checkentry using ipset match/target) are serialized by nfnl_subsys_ipset.

Ok, thanks Jozsef. In that case its much simpler to leave ipset alone and add a new reset serialization mutex in nf_tables.