Skip to contentSkip to conversation

2.6.39-rc6-mmotm0506 - another lockdep splat (networking this time)

DISCUSSION Applied [email protected] 3 participants 6 messages May 10, 2011

Subsystems (3)
Networking Network drivers IPv4/IPv6
lore
Share & tools
lore

[email protected]

author
#0 lore raw

Seen during boot this afternoon:

[   43.309549] 
[   43.309550] ===================================================
[   43.309553] [ INFO: suspicious rcu_dereference_check() usage. ]
[   43.309555] ---------------------------------------------------
[   43.309558] include/linux/inetdevice.h:226 invoked rcu_dereference_check() without protection!
[   43.309560] 
[   43.309561] other info that might help us debug this:
[   43.309561] 
[   43.309563] 
[   43.309564] rcu_scheduler_active = 1, debug_locks = 1
[   43.309567] 2 locks held by ip/1193:
[   43.309568]  #0:  (nlk->cb_mutex){+.+...}, at: [<ffffffff81426c1d>] netlink_dump+0x45/0x1d0
[   43.309579]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff8141b9b9>] rtnl_dump_ifinfo+0x0/0x156
[   43.309587] 
[   43.309587] stack backtrace:
[   43.309590] Pid: 1193, comm: ip Not tainted 2.6.39-rc6-mmotm0506 #1
[   43.309592] Call Trace:
[   43.309599]  [<ffffffff81066b6e>] lockdep_rcu_dereference+0x9a/0xa2
[   43.309604]  [<ffffffff81468d41>] __in_dev_get_rtnl+0x3c/0x47
[   43.309607]  [<ffffffff81468ef0>] inet_fill_link_af+0x12/0x5b
[   43.309611]  [<ffffffff8141b8fc>] rtnl_fill_ifinfo+0x665/0x710
[   43.309616]  [<ffffffff8141ba76>] rtnl_dump_ifinfo+0xbd/0x156
[   43.309620]  [<ffffffff81426c37>] netlink_dump+0x5f/0x1d0
[   43.309624]  [<ffffffff81404e3e>] ? consume_skb+0x8a/0x8f
[   43.309628]  [<ffffffff81426fc8>] netlink_recvmsg+0x1dd/0x31d
[   43.309632]  [<ffffffff813ff21f>] ? rcu_read_unlock+0x21/0x23
[   43.309636]  [<ffffffff813fab57>] sock_recvmsg+0xed/0x112
[   43.309641]  [<ffffffff810d1ebf>] ? might_fault+0x4e/0x9e
[   43.309645]  [<ffffffff81068816>] ? __lock_release+0x93/0x9c
[   43.309649]  [<ffffffff810d1ebf>] ? might_fault+0x4e/0x9e
[   43.309652]  [<ffffffff810d1f08>] ? might_fault+0x97/0x9e
[   43.309656]  [<ffffffff814065f8>] ? copy_from_user+0x3c/0x44
[   43.309660]  [<ffffffff813fc047>] __sys_recvmsg+0x16c/0x224
[   43.309665]  [<ffffffff81059240>] ? up_read+0x23/0x27
[   43.309669]  [<ffffffff81065165>] ? arch_local_irq_save+0x9/0xc
[   43.309673]  [<ffffffff810f736b>] ? fcheck_files+0xb4/0xeb
[   43.309676]  [<ffffffff810f79f3>] ? fget_light+0x35/0x96
[   43.309680]  [<ffffffff813fdded>] sys_recvmsg+0x3d/0x5b
[   43.309686]  [<ffffffff815705bb>] system_call_fastpath+0x16/0x1b
[   44.437679] e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
[   44.488363] e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
[   44.494680] ADDRCONF(NETDEV_UP): eth0: link is not ready
#1 lore raw
Quoted 15 lines · Le lundi 09 mai 2011 à 21:05 -0400, [email protected] a écrit :

Seen during boot this afternoon:

[   43.309549] 
[   43.309550] ===================================================
[   43.309553] [ INFO: suspicious rcu_dereference_check() usage. ]
[   43.309555] ---------------------------------------------------
[   43.309558] include/linux/inetdevice.h:226 invoked rcu_dereference_check() without protection!
[   43.309560] 
[   43.309561] other info that might help us debug this:
[   43.309561] 
[   43.309563] 
[   43.309564] rcu_scheduler_active = 1, debug_locks = 1
[   43.309567] 2 locks held by ip/1193:
[   43.309568]  #0:  (nlk->cb_mutex){+.+...}, at: [<ffffffff81426c1d>] netlink_dump+0x45/0x1d0
[   43.309579]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff8141b9b9>] rtnl_dump_ifinfo+0x0/0x156

strange : rcu_read_lock() is correctly held at this point

Quoted 21 lines
[   43.309587] 
[   43.309587] stack backtrace:
[   43.309590] Pid: 1193, comm: ip Not tainted 2.6.39-rc6-mmotm0506 #1
[   43.309592] Call Trace:
[   43.309599]  [<ffffffff81066b6e>] lockdep_rcu_dereference+0x9a/0xa2
[   43.309604]  [<ffffffff81468d41>] __in_dev_get_rtnl+0x3c/0x47
[   43.309607]  [<ffffffff81468ef0>] inet_fill_link_af+0x12/0x5b
[   43.309611]  [<ffffffff8141b8fc>] rtnl_fill_ifinfo+0x665/0x710
[   43.309616]  [<ffffffff8141ba76>] rtnl_dump_ifinfo+0xbd/0x156
[   43.309620]  [<ffffffff81426c37>] netlink_dump+0x5f/0x1d0
[   43.309624]  [<ffffffff81404e3e>] ? consume_skb+0x8a/0x8f
[   43.309628]  [<ffffffff81426fc8>] netlink_recvmsg+0x1dd/0x31d
[   43.309632]  [<ffffffff813ff21f>] ? rcu_read_unlock+0x21/0x23
[   43.309636]  [<ffffffff813fab57>] sock_recvmsg+0xed/0x112
[   43.309641]  [<ffffffff810d1ebf>] ? might_fault+0x4e/0x9e
[   43.309645]  [<ffffffff81068816>] ? __lock_release+0x93/0x9c
[   43.309649]  [<ffffffff810d1ebf>] ? might_fault+0x4e/0x9e
[   43.309652]  [<ffffffff810d1f08>] ? might_fault+0x97/0x9e
[   43.309656]  [<ffffffff814065f8>] ? copy_from_user+0x3c/0x44
[   43.309660]  [<ffffffff813fc047>] __sys_recvmsg+0x16c/0x224
[   43.309665]  [<ffffffff81059240>] ? up_read+0x23/0x27
Quoted 10 lines
[   43.309669]  [<ffffffff81065165>] ? arch_local_irq_save+0x9/0xc
[   43.309673]  [<ffffffff810f736b>] ? fcheck_files+0xb4/0xeb
[   43.309676]  [<ffffffff810f79f3>] ? fget_light+0x35/0x96
[   43.309680]  [<ffffffff813fdded>] sys_recvmsg+0x3d/0x5b
[   43.309686]  [<ffffffff815705bb>] system_call_fastpath+0x16/0x1b
[   44.437679] e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
[   44.488363] e1000e 0000:00:19.0: irq 46 for MSI/MSI-X
[   44.494680] ADDRCONF(NETDEV_UP): eth0: link is not ready

Thanks for the report, I am taking a look how to fix this.

I am testing following patch, I'll provide an official one when validated.

net/ipv4/devinet.c +2 -2
index cd9ca08..6b1eb92 100644
@@ -1369,7 +1369,7 @@ errout:
 
 static size_t inet_get_link_af_size(const struct net_device *dev)
 {
-	struct in_device *in_dev = __in_dev_get_rtnl(dev);
+	struct in_device *in_dev = __in_dev_get_rcu(dev);
 
 	if (!in_dev)
 		return 0;

@@ -1379,7 +1379,7 @@ static size_t inet_get_link_af_size(const struct net_device *dev)
 
 static int inet_fill_link_af(struct sk_buff *skb, const struct net_device *dev)
 {
-	struct in_device *in_dev = __in_dev_get_rtnl(dev);
+	struct in_device *in_dev = __in_dev_get_rcu(dev);
 	struct nlattr *nla;
 	int i;
#2 lore raw
Quoted 2 lines · Eric Dumazet

Thanks for the report, I am taking a look how to fix this.

OK here is the official patch submission

Thanks Valdis !

[PATCH net-next-2.6] net: fix two lockdep splats

Commit e67f88dd12f6 (net: dont hold rtnl mutex during netlink dump callbacks) switched rtnl protection to RCU, but we forgot to adjust two rcu_dereference() lockdep annotations :

inet_get_link_af_size() or inet_fill_link_af() might be called with rcu_read_lock or rtnl held, so use rcu_dereference_rtnl() instead of rtnl_dereference()

Reported-by Valdis Kletnieks <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/devinet.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv4/devinet.c +2 -2
index cd9ca08..6b1eb92 100644
@@ -1369,7 +1369,7 @@ errout:
 
 static size_t inet_get_link_af_size(const struct net_device *dev)
 {
-	struct in_device *in_dev = __in_dev_get_rtnl(dev);
+	struct in_device *in_dev = rcu_dereference_rtnl(dev->ip_ptr);
 
 	if (!in_dev)
 		return 0;

@@ -1379,7 +1379,7 @@ static size_t inet_get_link_af_size(const struct net_device *dev)
 
 static int inet_fill_link_af(struct sk_buff *skb, const struct net_device *dev)
 {
-	struct in_device *in_dev = __in_dev_get_rtnl(dev);
+	struct in_device *in_dev = rcu_dereference_rtnl(dev->ip_ptr);
 	struct nlattr *nla;
 	int i;
maintainer Applied
#3 lore raw

From: Eric Dumazet <[email protected]>
Date: Tue, 10 May 2011 05:47:51 +0200

Quoted 6 lines

[PATCH net-next-2.6] net: fix two lockdep splats

Commit e67f88dd12f6 (net: dont hold rtnl mutex during netlink dump callbacks) switched rtnl protection to RCU, but we forgot to adjust two rcu_dereference() lockdep annotations :

Quoted 6 lines

inet_get_link_af_size() or inet_fill_link_af() might be called with rcu_read_lock or rtnl held, so use rcu_dereference_rtnl() instead of rtnl_dereference()

Reported-by Valdis Kletnieks <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thanks everyone.

#4 lore raw
Quoted 9 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Tue, 10 May 2011 05:47:51 +0200

Quoted 6 lines

[PATCH net-next-2.6] net: fix two lockdep splats

Commit e67f88dd12f6 (net: dont hold rtnl mutex during netlink dump callbacks) switched rtnl protection to RCU, but we forgot to adjust two rcu_dereference() lockdep annotations :

Quoted 8 lines
Quoted 6 lines

inet_get_link_af_size() or inet_fill_link_af() might be called with rcu_read_lock or rtnl held, so use rcu_dereference_rtnl() instead of rtnl_dereference()

Reported-by Valdis Kletnieks <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thanks everyone.

David, I found you applied this patch for net-2.6, but it was net-next-2.6 material only ...

maintainer
#5 lore raw

From: Eric Dumazet <[email protected]>
Date: Tue, 10 May 2011 23:48:36 +0200

Quoted 14 lines · David Miller
Quoted 13 lines

From: Eric Dumazet <[email protected]>
Date: Tue, 10 May 2011 05:47:51 +0200

Quoted 10 lines

[PATCH net-next-2.6] net: fix two lockdep splats

Commit e67f88dd12f6 (net: dont hold rtnl mutex during netlink dump callbacks) switched rtnl protection to RCU, but we forgot to adjust two rcu_dereference() lockdep annotations :

inet_get_link_af_size() or inet_fill_link_af() might be called with rcu_read_lock or rtnl held, so use rcu_dereference_rtnl() instead of rtnl_dereference()

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Reported-by Valdis Kletnieks <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thanks everyone.

David, I found you applied this patch for net-2.6, but it was net-next-2.6 material only ...

Thanks for catching that, I'll fix this.