Skip to contentSkip to conversation

net: lockdep warning in ip_mc_msfget (net/ipv4/igmp.c:2400)

DISCUSSION Sasha Levin 3 participants 4 messages Nov 2, 2015 – Nov 3, 2015

Subsystems (3)
Networking Network drivers IPv4/IPv6
lore
Share & tools
lore

Sasha Levin

author
#0 lore raw

Hi all,

While fuzzing with syzkaller inside a KVM tools guest running the latest -next, I saw the following warning:

[ 2391.993558] ======================================================
[ 2391.995441] [ INFO: possible circular locking dependency detected ]
[ 2391.995771] 4.3.0-rc6-next-20151022-sasha-00042-g2b253a1-dirty #2618 Not tainted
[ 2391.995771] -------------------------------------------------------
[ 2391.995771] syzkaller_execu/14105 is trying to acquire lock:
[ 2391.995771] (rtnl_mutex){+.+.+.}, at: rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] Mutex: counter: 1 owner: None
[ 2391.995771]
[ 2391.995771] but task is already holding lock:
[ 2391.995771] (sk_lock-AF_INET){+.+.+.}, at: do_ip_getsockopt (net/ipv4/ip_sockglue.c:1274)
[ 2391.995771]
[ 2391.995771] which lock already depends on the new lock.
[ 2391.995771]
[ 2391.995771]
[ 2391.995771] the existing dependency chain (in reverse order) is:
[ 2391.995771] -> #1 (sk_lock-AF_INET){+.+.+.}:
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] lock_sock_nested (include/linux/bottom_half.h:31 net/core/sock.c:2411)
[ 2391.995771] do_ip_setsockopt.isra.9 (net/ipv4/ip_sockglue.c:623)
[ 2391.995771] ip_setsockopt (net/ipv4/ip_sockglue.c:1202)
[ 2391.995771]        ffffffffffffff, 0x0)
[ 2391.995771] sock_common_setsockopt (net/core/sock.c:2610)
[ 2391.995771] SyS_setsockopt (net/socket.c:1756 net/socket.c:1736)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)
[ 2391.995771] -> #0 (rtnl_mutex){+.+.+.}:
[ 2391.995771] __lock_acquire (kernel/locking/lockdep.c:1877 kernel/locking/lockdep.c:1982 kernel/locking/lockdep.c:2168 kernel/locking/lockdep.c:3239)
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] mutex_lock_nested (kernel/locking/mutex.c:526 kernel/locking/mutex.c:618)
[ 2391.995771] rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] ip_mc_msfget (net/ipv4/igmp.c:2400)
[ 2391.995771] do_ip_getsockopt (net/ipv4/ip_sockglue.c:1401)
[ 2391.995771] ip_getsockopt (net/ipv4/ip_sockglue.c:1498)
[ 2391.995771] raw_getsockopt (net/ipv4/raw.c:851)
[ 2391.995771] sock_common_getsockopt (net/core/sock.c:2569)
[ 2391.995771] SyS_getsockopt (net/socket.c:1787 net/socket.c:1770)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)
[ 2391.995771]
[ 2391.995771] other info that might help us debug this:
[ 2391.995771]
[ 2391.995771]  Possible unsafe locking scenario:
[ 2391.995771]
[ 2391.995771]        CPU0                    CPU1
[ 2391.995771]        ----                    ----
[ 2391.995771]   lock(sk_lock-AF_INET);
[ 2391.995771]                                lock(rtnl_mutex);
[ 2391.995771]                                lock(sk_lock-AF_INET);
[ 2391.995771]   lock(rtnl_mutex);
[ 2391.995771]
[ 2391.995771]  *** DEADLOCK ***
[ 2391.995771]
[ 2391.995771] 1 lock held by syzkaller_execu/14105:
[ 2391.995771] #0: (sk_lock-AF_INET){+.+.+.}, at: do_ip_getsockopt (net/ipv4/ip_sockglue.c:1274)
[ 2391.995771]
[ 2391.995771] stack backtrace:
[ 2391.995771] CPU: 1 PID: 14105 Comm: syzkaller_execu Not tainted 4.3.0-rc6-next-20151022-sasha-00042-g2b253a1-dirty #2618
[ 2391.995771]  0000000000000001 00000000c179c8c9 ffff8800a403f550 ffffffffade32a2b
[ 2391.995771]  ffffffffbb7f5a50 ffffffffbb84a4a0 ffffffffbb7f5a50 ffff8800a403f5a0
[ 2391.995771]  ffffffffac43fca8 ffff8800a403f690 00000000a3e18000 ffff8800a3e18000
[ 2391.995771] Call Trace:
[ 2391.995771] dump_stack (lib/dump_stack.c:52)
[ 2391.995771] print_circular_bug (kernel/locking/lockdep.c:1250)
[ 2391.995771] __lock_acquire (kernel/locking/lockdep.c:1877 kernel/locking/lockdep.c:1982 kernel/locking/lockdep.c:2168 kernel/locking/lockdep.c:3239)
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] mutex_lock_nested (kernel/locking/mutex.c:526 kernel/locking/mutex.c:618)
[ 2391.995771] rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] ip_mc_msfget (net/ipv4/igmp.c:2400)
[ 2391.995771] do_ip_getsockopt (net/ipv4/ip_sockglue.c:1401)
[ 2391.995771] ip_getsockopt (net/ipv4/ip_sockglue.c:1498)
[ 2391.995771] raw_getsockopt (net/ipv4/raw.c:851)
[ 2391.995771] sock_common_getsockopt (net/core/sock.c:2569)
[ 2391.995771] SyS_getsockopt (net/socket.c:1787 net/socket.c:1770)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)

Thanks,
Sasha

#1 lore raw
Quoted 69 lines · Sasha Levin

Hi all,

While fuzzing with syzkaller inside a KVM tools guest running the latest -next, I saw the following warning:

[ 2391.993558] ======================================================
[ 2391.995441] [ INFO: possible circular locking dependency detected ]
[ 2391.995771] 4.3.0-rc6-next-20151022-sasha-00042-g2b253a1-dirty #2618 Not tainted
[ 2391.995771] -------------------------------------------------------
[ 2391.995771] syzkaller_execu/14105 is trying to acquire lock:
[ 2391.995771] (rtnl_mutex){+.+.+.}, at: rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] Mutex: counter: 1 owner: None
[ 2391.995771]
[ 2391.995771] but task is already holding lock:
[ 2391.995771] (sk_lock-AF_INET){+.+.+.}, at: do_ip_getsockopt (net/ipv4/ip_sockglue.c:1274)
[ 2391.995771]
[ 2391.995771] which lock already depends on the new lock.
[ 2391.995771]
[ 2391.995771]
[ 2391.995771] the existing dependency chain (in reverse order) is:
[ 2391.995771] -> #1 (sk_lock-AF_INET){+.+.+.}:
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] lock_sock_nested (include/linux/bottom_half.h:31 net/core/sock.c:2411)
[ 2391.995771] do_ip_setsockopt.isra.9 (net/ipv4/ip_sockglue.c:623)
[ 2391.995771] ip_setsockopt (net/ipv4/ip_sockglue.c:1202)
[ 2391.995771]        ffffffffffffff, 0x0)
[ 2391.995771] sock_common_setsockopt (net/core/sock.c:2610)
[ 2391.995771] SyS_setsockopt (net/socket.c:1756 net/socket.c:1736)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)
[ 2391.995771] -> #0 (rtnl_mutex){+.+.+.}:
[ 2391.995771] __lock_acquire (kernel/locking/lockdep.c:1877 kernel/locking/lockdep.c:1982 kernel/locking/lockdep.c:2168 kernel/locking/lockdep.c:3239)
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] mutex_lock_nested (kernel/locking/mutex.c:526 kernel/locking/mutex.c:618)
[ 2391.995771] rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] ip_mc_msfget (net/ipv4/igmp.c:2400)
[ 2391.995771] do_ip_getsockopt (net/ipv4/ip_sockglue.c:1401)
[ 2391.995771] ip_getsockopt (net/ipv4/ip_sockglue.c:1498)
[ 2391.995771] raw_getsockopt (net/ipv4/raw.c:851)
[ 2391.995771] sock_common_getsockopt (net/core/sock.c:2569)
[ 2391.995771] SyS_getsockopt (net/socket.c:1787 net/socket.c:1770)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)
[ 2391.995771]
[ 2391.995771] other info that might help us debug this:
[ 2391.995771]
[ 2391.995771]  Possible unsafe locking scenario:
[ 2391.995771]
[ 2391.995771]        CPU0                    CPU1
[ 2391.995771]        ----                    ----
[ 2391.995771]   lock(sk_lock-AF_INET);
[ 2391.995771]                                lock(rtnl_mutex);
[ 2391.995771]                                lock(sk_lock-AF_INET);
[ 2391.995771]   lock(rtnl_mutex);
[ 2391.995771]
[ 2391.995771]  *** DEADLOCK ***
[ 2391.995771]
[ 2391.995771] 1 lock held by syzkaller_execu/14105:
[ 2391.995771] #0: (sk_lock-AF_INET){+.+.+.}, at: do_ip_getsockopt (net/ipv4/ip_sockglue.c:1274)
[ 2391.995771]
[ 2391.995771] stack backtrace:
[ 2391.995771] CPU: 1 PID: 14105 Comm: syzkaller_execu Not tainted 4.3.0-rc6-next-20151022-sasha-00042-g2b253a1-dirty #2618
[ 2391.995771]  0000000000000001 00000000c179c8c9 ffff8800a403f550 ffffffffade32a2b
[ 2391.995771]  ffffffffbb7f5a50 ffffffffbb84a4a0 ffffffffbb7f5a50 ffff8800a403f5a0
[ 2391.995771]  ffffffffac43fca8 ffff8800a403f690 00000000a3e18000 ffff8800a3e18000
[ 2391.995771] Call Trace:
[ 2391.995771] dump_stack (lib/dump_stack.c:52)
[ 2391.995771] print_circular_bug (kernel/locking/lockdep.c:1250)
[ 2391.995771] __lock_acquire (kernel/locking/lockdep.c:1877 kernel/locking/lockdep.c:1982 kernel/locking/lockdep.c:2168 kernel/locking/lockdep.c:3239)
[ 2391.995771] lock_acquire (kernel/locking/lockdep.c:3620)
[ 2391.995771] mutex_lock_nested (kernel/locking/mutex.c:526 kernel/locking/mutex.c:618)
Quoted 8 lines
[ 2391.995771] rtnl_lock (net/core/rtnetlink.c:71)
[ 2391.995771] ip_mc_msfget (net/ipv4/igmp.c:2400)
[ 2391.995771] do_ip_getsockopt (net/ipv4/ip_sockglue.c:1401)
[ 2391.995771] ip_getsockopt (net/ipv4/ip_sockglue.c:1498)
[ 2391.995771] raw_getsockopt (net/ipv4/raw.c:851)
[ 2391.995771] sock_common_getsockopt (net/core/sock.c:2569)
[ 2391.995771] SyS_getsockopt (net/socket.c:1787 net/socket.c:1770)
[ 2391.995771] entry_SYSCALL_64_fastpath (arch/x86/entry/entry_64.S:188)

Good catch!

This is probably introduced by:

commit baf606d9c9b12517e47e0d1370e8aa9f7323f210

Author: Marcelo Ricardo Leitner <[email protected]>
Date:   Wed Mar 18 14:50:42 2015 -0300

ipv4,ipv6: grab rtnl before locking the socket

I am thinking what is the right way to fix it...

#2 lore raw
Quoted 5 lines · Cong Wang

Good catch!

This is probably introduced by:

Quoted 7 lines

commit baf606d9c9b12517e47e0d1370e8aa9f7323f210

Author: Marcelo Ricardo Leitner <[email protected]>
Date:   Wed Mar 18 14:50:42 2015 -0300

ipv4,ipv6: grab rtnl before locking the socket

I am thinking what is the right way to fix it...

Please try the attached patch.

Thanks!

net/ipv4/igmp.c +4 -8
index d38b8b6..a2429b7 100644
@@ -2392,11 +2392,11 @@ int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf,
 	struct ip_sf_socklist *psl;
 	struct net *net = sock_net(sk);
 
+	ASSERT_RTNL();
+
 	if (!ipv4_is_multicast(addr))
 		return -EINVAL;
 
-	rtnl_lock();
-
 	imr.imr_multiaddr.s_addr = msf->imsf_multiaddr;
 	imr.imr_address.s_addr = msf->imsf_interface;
 	imr.imr_ifindex = 0;

@@ -2417,7 +2417,6 @@ int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf,
 		goto done;
 	msf->imsf_fmode = pmc->sfmode;
 	psl = rtnl_dereference(pmc->sflist);
-	rtnl_unlock();
 	if (!psl) {
 		len = 0;
 		count = 0;

@@ -2436,7 +2435,6 @@ int ip_mc_msfget(struct sock *sk, struct ip_msfilter *msf,
 		return -EFAULT;
 	return 0;
 done:
-	rtnl_unlock();
 	return err;
 }
 

@@ -2450,6 +2448,8 @@ int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
 	struct inet_sock *inet = inet_sk(sk);
 	struct ip_sf_socklist *psl;
 
+	ASSERT_RTNL();
+
 	psin = (struct sockaddr_in *)&gsf->gf_group;
 	if (psin->sin_family != AF_INET)
 		return -EINVAL;

@@ -2457,8 +2457,6 @@ int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
 	if (!ipv4_is_multicast(addr))
 		return -EINVAL;
 
-	rtnl_lock();
-
 	err = -EADDRNOTAVAIL;
 
 	for_each_pmc_rtnl(inet, pmc) {

@@ -2470,7 +2468,6 @@ int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
 		goto done;
 	gsf->gf_fmode = pmc->sfmode;
 	psl = rtnl_dereference(pmc->sflist);
-	rtnl_unlock();
 	count = psl ? psl->sl_count : 0;
 	copycount = count < gsf->gf_numsrc ? count : gsf->gf_numsrc;
 	gsf->gf_numsrc = count;

@@ -2490,7 +2487,6 @@ int ip_mc_gsfget(struct sock *sk, struct group_filter *gsf,
 	}
 	return 0;
 done:
-	rtnl_unlock();
 	return err;
 }
net/ipv4/ip_sockglue.c +30 -13

88 lines not on this page · raw message

#3 lore raw

Em 02-11-2015 22:38, Cong Wang escreveu:

Quoted 11 lines · Cong Wang
Quoted 10 lines

Good catch!

This is probably introduced by:

commit baf606d9c9b12517e47e0d1370e8aa9f7323f210

Author: Marcelo Ricardo Leitner <[email protected]>
Date:   Wed Mar 18 14:50:42 2015 -0300

ipv4,ipv6: grab rtnl before locking the socket

Yes, that commit introduced this inverse order situation by not fixing the get path of it.

Quoted 5 lines
Quoted 1 line

I am thinking what is the right way to fix it...

Please try the attached patch.

Thanks!

Patch LGTM, same design and covered all rtnl_lock()s I could find on get path now. Thanks!

Marcelo