possible deadlock in dev_set_allmulti
Subsystems (2)
syzbot
syzbotHello,
syzbot found the following issue on:
HEAD commit: 40587f749df2 Merge branch 'enic-enable-32-64-byte-cqes-and.. git tree: net-next console output: https://syzkaller.appspot.com/x/log.txt?x=128b1f8c580000 kernel config: https://syzkaller.appspot.com/x/.config?x=ca99d9d1f4a8ecfa dashboard link: https://syzkaller.appspot.com/bug?extid=b0c03d76056ef6cd12a6 compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/6d02993a9211/disk-40587f74.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/2c8b300bf362/vmlinux-40587f74.xz
kernel image: https://storage.googleapis.com/syzbot-assets/2d5be21882cf/bzImage-40587f74.xz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
netlink: 'syz.4.478': attribute type 10 has an invalid length.
netdevsim netdevsim4 netdevsim0: left allmulticast mode
============================================
WARNING: possible recursive locking detected 6.14.0-rc5-syzkaller-01183-g40587f749df2 #0 Not tainted
--------------------------------------------
syz.4.478/7361 is trying to acquire lock:ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock include/linux/netdevice.h:2731 [inline] ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock_ops include/net/netdev_lock.h:40 [inline] ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: dev_set_allmulti+0x11c/0x270 net/core/dev_api.c:279
but task is already holding lock:ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock include/linux/netdevice.h:2731 [inline] ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock_ops include/net/netdev_lock.h:40 [inline] ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: dev_open+0x11c/0x260 net/core/dev_api.c:190
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
---- lock(&dev->lock);
lock(&dev->lock);
*** DEADLOCK ***
May be due to missing lock nesting notation
2 locks held by syz.4.478/7361:
#0: ffffffff8fed6908 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock net/core/rtnetlink.c:80 [inline] #0: ffffffff8fed6908 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_nets_lock net/core/rtnetlink.c:341 [inline] #0: ffffffff8fed6908 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0xc4c/0x1d90 net/core/rtnetlink.c:4054 #1: ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock include/linux/netdevice.h:2731 [inline] #1: ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: netdev_lock_ops include/net/netdev_lock.h:40 [inline] #1: ffff88807b5e4d28 (&dev->lock){+.+.}-{4:4}, at: dev_open+0x11c/0x260 net/core/dev_api.c:190
stack backtrace: CPU: 1 UID: 0 PID: 7361 Comm: syz.4.478 Not tainted 6.14.0-rc5-syzkaller-01183-g40587f749df2 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025 Call Trace: <TASK> __dump_stack lib/dump_stack.c:94 [inline] dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120 print_deadlock_bug+0x483/0x620 kernel/locking/lockdep.c:3039 check_deadlock kernel/locking/lockdep.c:3091 [inline] validate_chain+0x15e2/0x5920 kernel/locking/lockdep.c:3893 __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5228 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5851 __mutex_lock_common kernel/locking/mutex.c:585 [inline] __mutex_lock+0x19c/0x1010 kernel/locking/mutex.c:730 netdev_lock include/linux/netdevice.h:2731 [inline] netdev_lock_ops include/net/netdev_lock.h:40 [inline] dev_set_allmulti+0x11c/0x270 net/core/dev_api.c:279 vlan_dev_open+0x2be/0x8a0 net/8021q/vlan_dev.c:278 __dev_open+0x45a/0x8a0 net/core/dev.c:1644 __dev_change_flags+0x1e2/0x6f0 net/core/dev.c:9375 netif_change_flags+0x8b/0x1a0 net/core/dev.c:9438 dev_change_flags+0x146/0x270 net/core/dev_api.c:68 vlan_device_event+0x1b81/0x1de0 net/8021q/vlan.c:469 notifier_call_chain+0x1a5/0x3f0 kernel/notifier.c:85 call_netdevice_notifiers_extack net/core/dev.c:2244 [inline] call_netdevice_notifiers net/core/dev.c:2258 [inline] netif_open+0x13a/0x1b0 net/core/dev.c:1672 dev_open+0x13e/0x260 net/core/dev_api.c:191 bond_enslave+0x103c/0x3910 drivers/net/bonding/bond_main.c:2135 do_set_master+0x579/0x730 net/core/rtnetlink.c:2943 do_setlink+0xfee/0x40f0 net/core/rtnetlink.c:3149 rtnl_changelink net/core/rtnetlink.c:3759 [inline] __rtnl_newlink net/core/rtnetlink.c:3918 [inline] rtnl_newlink+0x15a6/0x1d90 net/core/rtnetlink.c:4055 rtnetlink_rcv_msg+0x791/0xcf0 net/core/rtnetlink.c:6945 netlink_rcv_skb+0x206/0x480 net/netlink/af_netlink.c:2534 netlink_unicast_kernel net/netlink/af_netlink.c:1313 [inline] netlink_unicast+0x7f6/0x990 net/netlink/af_netlink.c:1339 netlink_sendmsg+0x8de/0xcb0 net/netlink/af_netlink.c:1883 sock_sendmsg_nosec net/socket.c:709 [inline] __sock_sendmsg+0x221/0x270 net/socket.c:724 ____sys_sendmsg+0x53a/0x860 net/socket.c:2564 ___sys_sendmsg net/socket.c:2618 [inline] __sys_sendmsg+0x269/0x350 net/socket.c:2650 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f191f98d169 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f192081c038 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007f191fba5fa0 RCX: 00007f191f98d169 RDX: 0000000000000000 RSI: 0000400000000680 RDI: 0000000000000003 RBP: 00007f191fa0e2a0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000000 R14: 00007f191fba5fa0 R15: 00007ffc4fecad18 </TASK>
---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title
If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)
If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report
If you want to undo deduplication, reply with:
#syz undup
Quoted 105 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: 40587f749df2 Merge branch 'enic-enable-32-64-byte-cqes-and.. git tree: net-next console output: https://syzkaller.appspot.com/x/log.txt?x=128b1f8c580000 kernel config: https://syzkaller.appspot.com/x/.config?x=ca99d9d1f4a8ecfa dashboard link: https://syzkaller.appspot.com/bug?extid=b0c03d76056ef6cd12a6 compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/6d02993a9211/disk-40587f74.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/2c8b300bf362/vmlinux-40587f74.xz
kernel image: https://storage.googleapis.com/syzbot-assets/2d5be21882cf/bzImage-40587f74.xzIMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]netlink: 'syz.4.478': attribute type 10 has an invalid length.
netdevsim netdevsim4 netdevsim0: left allmulticast mode
============================================
WARNING: possible recursive locking detected 6.14.0-rc5-syzkaller-01183-g40587f749df2 #0 Not tainted
--------------------------------------------
syz.4.478/7361 is trying to acquire lock:ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lockinclude/linux/netdevice.h:2731[inline]ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lock_opsinclude/net/netdev_lock.h:40[inline]ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:dev_set_allmulti+0x11c/0x270net/core/dev_api.c:279but task is already holding lock:
ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lockinclude/linux/netdevice.h:2731[inline]ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lock_opsinclude/net/netdev_lock.h:40[inline]ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:dev_open+0x11c/0x260net/core/dev_api.c:190other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----
lock(&dev->lock);
lock(&dev->lock);*** DEADLOCK ***
May be due to missing lock nesting notation
2 locks held by syz.4.478/7361:
#0:ffffffff8fed6908(rtnl_mutex){+.+.}-{4:4}, at:rtnl_locknet/core/rtnetlink.c:80[inline] #0:ffffffff8fed6908(rtnl_mutex){+.+.}-{4:4}, at:rtnl_nets_locknet/core/rtnetlink.c:341[inline] #0:ffffffff8fed6908(rtnl_mutex){+.+.}-{4:4}, at:rtnl_newlink+0xc4c/0x1d90net/core/rtnetlink.c:4054#1:ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lockinclude/linux/netdevice.h:2731[inline] #1:ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:netdev_lock_opsinclude/net/netdev_lock.h:40[inline] #1:ffff88807b5e4d28(&dev->lock){+.+.}-{4:4}, at:dev_open+0x11c/0x260net/core/dev_api.c:190stack backtrace: CPU: 1 UID: 0 PID: 7361 Comm: syz.4.478 Not tainted 6.14.0-rc5-syzkaller-01183-g40587f749df2 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025 Call Trace: <TASK> __dump_stack lib/dump_stack.c:94 [inline] dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120 print_deadlock_bug+0x483/0x620 kernel/locking/lockdep.c:3039 check_deadlock kernel/locking/lockdep.c:3091 [inline] validate_chain+0x15e2/0x5920 kernel/locking/lockdep.c:3893 __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5228 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5851 __mutex_lock_common kernel/locking/mutex.c:585 [inline] __mutex_lock+0x19c/0x1010 kernel/locking/mutex.c:730 netdev_lock include/linux/netdevice.h:2731 [inline] netdev_lock_ops include/net/netdev_lock.h:40 [inline] dev_set_allmulti+0x11c/0x270 net/core/dev_api.c:279 vlan_dev_open+0x2be/0x8a0 net/8021q/vlan_dev.c:278 __dev_open+0x45a/0x8a0 net/core/dev.c:1644 __dev_change_flags+0x1e2/0x6f0 net/core/dev.c:9375 netif_change_flags+0x8b/0x1a0 net/core/dev.c:9438 dev_change_flags+0x146/0x270 net/core/dev_api.c:68 vlan_device_event+0x1b81/0x1de0 net/8021q/vlan.c:469 notifier_call_chain+0x1a5/0x3f0 kernel/notifier.c:85 call_netdevice_notifiers_extack net/core/dev.c:2244 [inline] call_netdevice_notifiers net/core/dev.c:2258 [inline] netif_open+0x13a/0x1b0 net/core/dev.c:1672 dev_open+0x13e/0x260 net/core/dev_api.c:191 bond_enslave+0x103c/0x3910 drivers/net/bonding/bond_main.c:2135 do_set_master+0x579/0x730 net/core/rtnetlink.c:2943 do_setlink+0xfee/0x40f0 net/core/rtnetlink.c:3149 rtnl_changelink net/core/rtnetlink.c:3759 [inline] __rtnl_newlink net/core/rtnetlink.c:3918 [inline] rtnl_newlink+0x15a6/0x1d90 net/core/rtnetlink.c:4055 rtnetlink_rcv_msg+0x791/0xcf0 net/core/rtnetlink.c:6945 netlink_rcv_skb+0x206/0x480 net/netlink/af_netlink.c:2534 netlink_unicast_kernel net/netlink/af_netlink.c:1313 [inline] netlink_unicast+0x7f6/0x990 net/netlink/af_netlink.c:1339 netlink_sendmsg+0x8de/0xcb0 net/netlink/af_netlink.c:1883 sock_sendmsg_nosec net/socket.c:709 [inline] __sock_sendmsg+0x221/0x270 net/socket.c:724 ____sys_sendmsg+0x53a/0x860 net/socket.c:2564 ___sys_sendmsg net/socket.c:2618 [inline] __sys_sendmsg+0x269/0x350 net/socket.c:2650 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f191f98d169
Quoted 8 lines
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f192081c038 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007f191fba5fa0 RCX: 00007f191f98d169 RDX: 0000000000000000 RSI: 0000400000000680 RDI: 0000000000000003 RBP: 00007f191fa0e2a0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000000 R14: 00007f191fba5fa0 R15: 00007ffc4fecad18 </TASK>
This one is bad :-( If I read the trace correctly, it's a vlan device that's being added to a bonding device. In bond_enslave->dev_open, we lock the vlan device and run the notifiers. And the vlan notifier uses dev_xxx methods that try to lock the vlan device again.
I will try to repro locally tomorrow. We might have to release the netdev locks during the notifier runs :-/