WARNING in nf_conntrack_cleanup_net_list
Subsystems (3)
syzbot
syzbotHello,
syzbot found the following issue on:
HEAD commit: 5ce74bc1b7cb Add linux-next specific files for 20251211 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=11a241b4580000 kernel config: https://syzkaller.appspot.com/x/.config?x=a94030c847137a18 dashboard link: https://syzkaller.appspot.com/bug?extid=4393c47753b7808dac7d compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/e1660176a50f/disk-5ce74bc1.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/616ede012bbd/vmlinux-5ce74bc1.xz
kernel image: https://storage.googleapis.com/syzbot-assets/7c0c5bc40fd8/bzImage-5ce74bc1.xz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
------------[ cut here ]------------ conntrack cleanup blocked for 60s WARNING: net/netfilter/nf_conntrack_core.c:2512 at nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511, CPU#1: kworker/u8:0/12 Modules linked in: CPU: 1 UID: 0 PID: 12 Comm: kworker/u8:0 Tainted: G L syzkaller #0 PREEMPT(full) Tainted: [L]=SOFTLOCKUP Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/25/2025 Workqueue: netns cleanup_net RIP: 0010:nf_conntrack_cleanup_net_list+0x234/0x340 net/netfilter/nf_conntrack_core.c:2511 Code: 08 48 89 df e8 fd 78 a3 f8 4c 8b 3b 49 39 df 74 69 e8 20 16 3d f8 45 31 e4 e9 8e fe ff ff e8 13 16 3d f8 48 8d 3d ec 2f 0c 06 <67> 48 0f b9 3a eb c0 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c cd fe ff RSP: 0018:ffffc90000117870 EFLAGS: 00010293 RAX: ffffffff8984a13d RBX: ffffc90000117a00 RCX: ffff88801ce8db80 RDX: 0000000000000000 RSI: ffffffffffffffcb RDI: ffffffff8f90d130 RBP: 0000000000000001 R08: ffff88807ce6b003 R09: 1ffff1100f9cd600 R10: dffffc0000000000 R11: ffffed100f9cd601 R12: 0000000000000001 R13: dffffc0000000000 R14: 00000001000068e3 R15: 0000000100006918 FS: 0000000000000000(0000) GS:ffff888125f32000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000200000020000 CR3: 00000000306ac000 CR4: 00000000003526f0 Call Trace: <TASK> ops_exit_list net/core/net_namespace.c:205 [inline] ops_undo_list+0x525/0x990 net/core/net_namespace.c:252 cleanup_net+0x4d8/0x7a0 net/core/net_namespace.c:696 process_one_work+0x93a/0x15a0 kernel/workqueue.c:3279 process_scheduled_works kernel/workqueue.c:3362 [inline] worker_thread+0x9b0/0xee0 kernel/workqueue.c:3443 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK> ---------------- Code disassembly (best guess): 0: 08 48 89 or %cl,-0x77(%rax) 3: df e8 fucomip %st(0),%st 5: fd std 6: 78 a3 js 0xffffffab 8: f8 clc 9: 4c 8b 3b mov (%rbx),%r15 c: 49 39 df cmp %rbx,%r15 f: 74 69 je 0x7a 11: e8 20 16 3d f8 call 0xf83d1636 16: 45 31 e4 xor %r12d,%r12d 19: e9 8e fe ff ff jmp 0xfffffeac 1e: e8 13 16 3d f8 call 0xf83d1636 23: 48 8d 3d ec 2f 0c 06 lea 0x60c2fec(%rip),%rdi # 0x60c3016 * 2a: 67 48 0f b9 3a ud1 (%edx),%rdi <-- trapping instruction 2f: eb c0 jmp 0xfffffff1 31: 89 e9 mov %ebp,%ecx 33: 80 e1 07 and $0x7,%cl 36: 80 c1 03 add $0x3,%cl 39: 38 c1 cmp %al,%cl 3b: 0f .byte 0xf 3c: 8c cd mov %cs,%ebp 3e: fe (bad) 3f: ff .byte 0xff
---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title
If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)
If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report
If you want to undo deduplication, reply with:
#syz undup
Jakub Kicinski
maintainerQuoted 3 lines · On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
------------[ cut here ]------------ conntrack cleanup blocked for 60s WARNING: net/netfilter/nf_conntrack_core.c:2512 at
Yes, I was about to comment on the patch which added the warning..
There is still a leak somewhere. Running ip_defrag.sh and then load / unload ipvlan repros this (modprobe ipvlan is a quick check if the cleanup thread is wedged, if it is modprobe will hang, if it isn't run ip_defrag.sh, again etc).
I looked around last night but couldn't find an skb stuck anywhere. The nf_conntrack_net->count was == 1
Florian Westphal
maintainerQuoted 7 lines · Jakub Kicinski
Quoted 3 lines · On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
------------[ cut here ]------------ conntrack cleanup blocked for 60s WARNING: net/netfilter/nf_conntrack_core.c:2512 atYes, I was about to comment on the patch which added the warning..
Quoted 7 lines
There is still a leak somewhere. Running
ip_defrag.shand then load / unload ipvlan repros this (modprobe ipvlan is a quick check if the cleanup thread is wedged, if it is modprobe will hang, if it isn't runip_defrag.sh, again etc).I looked around last night but couldn't find an skb stuck anywhere. The
nf_conntrack_net->countwas == 1
Its caused skb skb fraglist skbs that still hold nf_conn references on the softnet data defer lists.
setting net.core.skb_defer_max=0 makes the hang disappear for me.
Quoted 14 lines · Florian Westphal
Quoted 12 lines · Jakub Kicinski
Quoted 3 lines · On Thu, 11 Dec 2025 10:38:31 -0800 syzbot wrote:
------------[ cut here ]------------ conntrack cleanup blocked for 60s WARNING: net/netfilter/nf_conntrack_core.c:2512 atYes, I was about to comment on the patch which added the warning..
There is still a leak somewhere. Running
ip_defrag.shand then load / unload ipvlan repros this (modprobe ipvlan is a quick check if the cleanup thread is wedged, if it is modprobe will hang, if it isn't runip_defrag.sh, again etc).
Quoted 7 lines
Quoted 2 lines
I looked around last night but couldn't find an skb stuck anywhere. The
nf_conntrack_net->countwas == 1Its caused skb skb fraglist skbs that still hold
nf_connreferences on the softnet data defer lists.setting net.core.skb_defer_max=0 makes the hang disappear for me.
What kind of packets ? TCP ones ?
Florian Westphal
maintainerQuoted 9 lines · Eric Dumazet
Quoted 7 lines
Quoted 2 lines
I looked around last night but couldn't find an skb stuck anywhere. The
nf_conntrack_net->countwas == 1Its caused skb skb fraglist skbs that still hold
nf_connreferences on the softnet data defer lists.setting net.core.skb_defer_max=0 makes the hang disappear for me.
What kind of packets ? TCP ones ?
UDP, but I can't say yet if thats an udp specific issue or not. (the packets are generated via ip_defrag.c).
Quoted 8 lines · Florian Westphal
Quoted 6 lines · Eric Dumazet
Quoted 6 lines
Quoted 2 lines
I looked around last night but couldn't find an skb stuck anywhere. The
nf_conntrack_net->countwas == 1Its caused skb skb fraglist skbs that still hold
nf_connreferences on the softnet data defer lists.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
setting net.core.skb_defer_max=0 makes the hang disappear for me.
What kind of packets ? TCP ones ?
UDP, but I can't say yet if thats an udp specific issue or not. (the packets are generated via
ip_defrag.c).
skb_release_head_state() does not follow the fraglist. Oh well.
net/core/skbuff.c +0 -0
100644
net/core/skbuff.c +3 -1
@@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)DEBUG_NET_WARN_ON_ONCE (!in_softirq ()); -if (skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) { +if (skb->alloc_cpu !=smp_processor_id () && + !skb_shared (skb) && + !skb_has_frag_list (skb)) {skb_release_head_state (skb);return skb_attempt_defer_free (skb); }
Florian Westphal
maintainerQuoted 11 lines · Eric Dumazet
Quoted 2 lines
UDP, but I can't say yet if thats an udp specific issue or not. (the packets are generated via
ip_defrag.c).
skb_release_head_state()does not follow the fraglist. Oh well.net/core/skbuff.c +0
-0100644
net/core/skbuff.c void napi_consume_skb(struct sk_buff *skb, int budget) · line 1504
Quoted 9 lines
net/core/skbuff.c +3
-1@@ -1497,7 +1497,9 @@ void napi_consume_skb(struct sk_buff *skb, int budget)DEBUG_NET_WARN_ON_ONCE (!in_softirq ()); -if (skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) { +if (skb->alloc_cpu !=smp_processor_id () && + !skb_shared (skb) && + !skb_has_frag_list (skb)) {skb_release_head_state (skb);return skb_attempt_defer_free (skb);
There is also:skb_attempt_defer_free -> skb_attempt_defer_free
Alternatively we could export skb_defer_free_flush or kick_defer_list_purge() and call that from nf_conntrack net exit path.
I will investigate more closely on monday, I still don't understand why fragments are conntracked in the first place.