Skip to contentSkip to conversation

WARNING in geneve_exit_batch_net (2)

PATCH In review syzbot 4 participants 6 messages Feb 29, 2020 – Mar 15, 2020

Subsystems (3)
lore
Share & tools
lore

syzbot

author syzbot
#0 lore raw

Hello,

syzbot found the following crash on:

HEAD commit:    f8788d86 Linux 5.6-rc3
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=138dd22de00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5d2e033af114153f
dashboard link: https://syzkaller.appspot.com/bug?extid=68a8ed58e3d17c700de5
compiler:       clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=16601d31e00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=14fdf8f9e00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:

Reported-by [email protected]
------------[ cut here ]------------
WARNING: CPU: 0 PID: 304 at drivers/net/geneve.c:1849 geneve_destroy_tunnels drivers/net/geneve.c:1849 [inline]
WARNING: CPU: 0 PID: 304 at drivers/net/geneve.c:1849 geneve_exit_batch_net+0x2b1/0x300 drivers/net/geneve.c:1859
Kernel panic - not syncing: panic_on_warn set ...
CPU: 0 PID: 304 Comm: kworker/u4:4 Not tainted 5.6.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fb/0x318 lib/dump_stack.c:118
 panic+0x264/0x7a9 kernel/panic.c:221
 __warn+0x209/0x210 kernel/panic.c:582
 report_bug+0x1b6/0x2f0 lib/bug.c:195
 fixup_bug arch/x86/kernel/traps.c:174 [inline]
 do_error_trap+0xcf/0x1c0 arch/x86/kernel/traps.c:267
 do_invalid_op+0x36/0x40 arch/x86/kernel/traps.c:286
 invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
RIP: 0010:geneve_destroy_tunnels drivers/net/geneve.c:1849 [inline]
RIP: 0010:geneve_exit_batch_net+0x2b1/0x300 drivers/net/geneve.c:1859
Code: 48 c1 e8 03 42 80 3c 28 00 74 08 48 89 df e8 46 aa d1 fc 48 8b 1b 4c 39 fb 74 13 e8 c9 80 94 fc e9 f4 fd ff ff e8 bf 80 94 fc <0f> 0b eb cf e8 b6 80 94 fc eb 05 e8 af 80 94 fc 48 8d 7d c0 e8 c6
RSP: 0018:ffffc90001917c08 EFLAGS: 00010293
RAX: ffffffff84e288c1 RBX: ffff8880a7bc6120 RCX: ffff8880a88304c0
RDX: 0000000000000000 RSI: ffffc90001917c28 RDI: ffff8880a47da068
RBP: ffffc90001917c68 R08: ffffffff866be459 R09: fffffbfff12b21a9
R10: fffffbfff12b21a9 R11: 0000000000000000 R12: ffffc90001917c28
R13: dffffc0000000000 R14: ffff8880a1ca0dd0 R15: ffffc90001917c98
 ops_exit_list net/core/net_namespace.c:175 [inline]
 cleanup_net+0x78b/0xb80 net/core/net_namespace.c:589
 process_one_work+0x7f5/0x10f0 kernel/workqueue.c:2264
 worker_thread+0xbbc/0x1630 kernel/workqueue.c:2410
 kthread+0x332/0x350 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Kernel Offset: disabled
Rebooting in 86400 seconds..

---
This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

author syzbot
#1 lore raw
syzbot has bisected this bug to:

syzbot has bisected this bug to:

commit 4e645b47c4f000a503b9c90163ad905786b9bc1d

Author: Florian Westphal <[email protected]>
Date:   Thu Nov 30 23:21:02 2017 +0000

netfilter: core: make nf_unregister_net_hooks simple wrapper again

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1594fbfde00000
start commit:   63623fd4 Merge tag 'for-linus' of git://git.kernel.org/pub..
git tree:       upstream
final crash:    https://syzkaller.appspot.com/x/report.txt?x=1794fbfde00000
console output: https://syzkaller.appspot.com/x/log.txt?x=1394fbfde00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9833e26bab355358
dashboard link: https://syzkaller.appspot.com/bug?extid=68a8ed58e3d17c700de5
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14f08165e00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17902329e00000
Reported-by [email protected]
Fixes 4e645b47c4f0 ("netfilter: core: make nf_unregister_net_hooks simple wrapper again")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

maintainer
#2 lore raw
Quoted 7 lines · syzbot

syzbot has bisected this bug to:

commit 4e645b47c4f000a503b9c90163ad905786b9bc1d

Author: Florian Westphal <[email protected]>
Date:   Thu Nov 30 23:21:02 2017 +0000

netfilter: core: make nf_unregister_net_hooks simple wrapper again

No idea why this turns up, the reproducer doesn't hit any of these code paths.

The debug splat is a false-positive; ndo_stop/list_del hasn't run yet. I will send a fix for net tree.

#3 lore raw
Quoted 4 lines · Florian Westphal
Quoted 2 lines · syzbot

syzbot has bisected this bug to:

Quoted 8 lines
Quoted 5 lines

commit 4e645b47c4f000a503b9c90163ad905786b9bc1d

Author: Florian Westphal <[email protected]>
Date:   Thu Nov 30 23:21:02 2017 +0000

netfilter: core: make nf_unregister_net_hooks simple wrapper again

No idea why this turns up, the reproducer doesn't hit any of these code paths.

The attached bisection log usually makes this reasonably transparent. It seems that in this case another kernel bug gets in the way of bisection.

Quoted 7 lines

The debug splat is a false-positive; ndo_stop/list_del hasn't run yet. I will send a fix for net tree.

--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20200313014435.GY979%40breakpoint.cc.

maintainer
#4 lore raw

PATCH 1/1 geneve: move debug check after netdev unregister

In reply to syzbot

The debug check must be done after unregister_netdevice_many() call -- the list_del() for this is done inside .ndo_stop.

Fixes 2843a25348f8 ("geneve: speedup geneve tunnels dismantle")
Reported-and-tested-by <[email protected]>
Cc Haishuang Yan <[email protected]>
---
 drivers/net/geneve.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/net/geneve.c +6 -2
index 75757e9954ba..09f279c0182b 100644
@@ -1845,8 +1845,6 @@ static void geneve_destroy_tunnels(struct net *net, struct list_head *head)
 		if (!net_eq(dev_net(geneve->dev), net))
 			unregister_netdevice_queue(geneve->dev, head);
 	}
-
-	WARN_ON_ONCE(!list_empty(&gn->sock_list));
 }
 
 static void __net_exit geneve_exit_batch_net(struct list_head *net_list)
@@ -1861,6 +1859,12 @@ static void __net_exit geneve_exit_batch_net(struct list_head *net_list)
 	/* unregister the devices gathered above */
 	unregister_netdevice_many(&list);
 	rtnl_unlock();
+
+	list_for_each_entry(net, net_list, exit_list) {
+		const struct geneve_net *gn = net_generic(net, geneve_net_id);
+
+		WARN_ON_ONCE(!list_empty(&gn->sock_list));
+	}
 }
 
 static struct pernet_operations geneve_net_ops = {
Signature · 1 line
2.24.1
maintainer
#5 lore raw

Re: [PATCH net] geneve: move debug check after netdev unregister

From: Florian Westphal <[email protected]>
Date: Sat, 14 Mar 2020 08:18:42 +0100

Quoted 7 lines

The debug check must be done after unregister_netdevice_many() call -- the list_del() for this is done inside .ndo_stop.

Fixes 2843a25348f8 ("geneve: speedup geneve tunnels dismantle")
Reported-and-tested-by <[email protected]>
Cc Haishuang Yan <[email protected]>

Applied and queued up for -stable, thanks Florian.