Skip to contentSkip to conversation

vxlan: Fix vs->vni_list locking.

PATCH In review Pravin B Shelar 3 participants 13 messages Jul 10, 2013 – Jul 17, 2013

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Pravin B Shelar

author
#0 lore raw

PATCH 1/1 vxlan: Fix vs->vni_list locking.

Use rtnl lock to protect vs->vni_list updates.

Signed-off-by Pravin B Shelar <[email protected]>
---
 drivers/net/vxlan.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)
drivers/net/vxlan.c +9 -4
index 227b54a..deca481 100644
@@ -89,7 +89,7 @@ struct vxlan_sock {
 	struct work_struct del_work;
 	atomic_t	  refcnt;
 	struct socket	  *sock;
-	struct hlist_head vni_list[VNI_HASH_SIZE];
+	struct hlist_head vni_list[VNI_HASH_SIZE]; /* Protected by RTNL. */
 };
 
 /* per-network namespace private data for this module */

@@ -122,7 +122,7 @@ struct vxlan_fdb {
 
 /* Pseudo network device */
 struct vxlan_dev {
-	struct hlist_node hlist;	/* vni hash table */
+	struct hlist_node hlist;	/* vni hash table (vni_list) */
 	struct list_head  next;		/* vxlan's per namespace list */
 	struct vxlan_sock *vn_sock;	/* listening socket */
 	struct net_device *dev;

@@ -1644,16 +1644,21 @@ static void vxlan_sock_work(struct work_struct *work)
 	if (ovs) {
 		atomic_inc(&ovs->refcnt);
 		vxlan->vn_sock = ovs;
-		hlist_add_head_rcu(&vxlan->hlist, vni_head(ovs, vni));
 		spin_unlock(&vn->sock_lock);
 
+		rtnl_lock();
+		hlist_add_head_rcu(&vxlan->hlist, vni_head(ovs, vni));
+		rtnl_unlock();
+
 		sk_release_kernel(nvs->sock->sk);
 		kfree(nvs);
 	} else {
 		vxlan->vn_sock = nvs;
 		hlist_add_head_rcu(&nvs->hlist, vs_head(net, port));
-		hlist_add_head_rcu(&vxlan->hlist, vni_head(nvs, vni));
 		spin_unlock(&vn->sock_lock);
+		rtnl_lock();
+		hlist_add_head_rcu(&vxlan->hlist, vni_head(nvs, vni));
+		rtnl_unlock();
 	}
 out:
 	dev_put(dev);
Signature · 1 line
1.7.1
#1 lore raw
Quoted 4 lines · On Wed, 10 Jul 2013 15:04:44 -0700 Pravin B Shelar <[email protected]> wrote:

Use rtnl lock to protect vs->vni_list updates.

Signed-off-by Pravin B Shelar <[email protected]>

---

I don't think this is necessary. I intentionally changed the locking when socket management was moved to a work queue. The vxlan_net socket lock is held there already, and finer grain.

author
#2 lore raw
Quoted 11 lines · Stephen Hemminger
Quoted 4 lines · On Wed, 10 Jul 2013 15:04:44 -0700 Pravin B Shelar <[email protected]> wrote:

Use rtnl lock to protect vs->vni_list updates.

Signed-off-by Pravin B Shelar <[email protected]>

---

I don't think this is necessary. I intentionally changed the locking when socket management was moved to a work queue. The vxlan_net socket lock is held there already, and finer grain.

what abt vxlan_dellink()?
it is deleting vxlan-dev from hash table without lock.

#3 lore raw
Quoted 10 lines · On Wed, 10 Jul 2013 16:08:38 -0700 Pravin Shelar <[email protected]> wrote:
Quoted 8 lines · Stephen Hemminger
Quoted 4 lines · On Wed, 10 Jul 2013 15:04:44 -0700 Pravin B Shelar <[email protected]> wrote:

Use rtnl lock to protect vs->vni_list updates.

Signed-off-by Pravin B Shelar <[email protected]>

---

Quoted 6 lines
Quoted 3 lines

I don't think this is necessary. I intentionally changed the locking when socket management was moved to a work queue. The vxlan_net socket lock is held there already, and finer grain.

what abt vxlan_dellink()?
it is deleting vxlan-dev from hash table without lock.

Ok, then this should fix it.

drivers/net/vxlan.c +4 -0
@@ -1768,8 +1768,12 @@ static int vxlan_newlink(struct net *net
 static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 {
        struct vxlan_dev *vxlan = netdev_priv(dev);
+       struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
 
+       spin_lock(&vn->sock_lock);
        hlist_del_rcu(&vxlan->hlist);
+       spin_unlock(&vn->sock_lock);
+
        list_del(&vxlan->next);
        unregister_netdevice_queue(dev, head);
 }
maintainer
#4 lore raw

From: Stephen Hemminger <[email protected]>
Date: Wed, 10 Jul 2013 16:25:09 -0700

Quoted 13 lines
Quoted 10 lines · On Wed, 10 Jul 2013 16:08:38 -0700 Pravin Shelar <[email protected]> wrote:
Quoted 8 lines · Stephen Hemminger
Quoted 4 lines · On Wed, 10 Jul 2013 15:04:44 -0700 Pravin B Shelar <[email protected]> wrote:

Use rtnl lock to protect vs->vni_list updates.

Signed-off-by Pravin B Shelar <[email protected]>

---

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't think this is necessary. I intentionally changed the locking when socket management was moved to a work queue. The vxlan_net socket lock is held there already, and finer grain.

what abt vxlan_dellink()?
it is deleting vxlan-dev from hash table without lock.

Ok, then this should fix it.

Looks good, one minor thing:

Quoted 3 lines
 {
        struct vxlan_dev *vxlan = netdev_priv(dev);

+ struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);

Please declare 'vn' on the first line rather than the second.

A formal submission with that fixup and a signoff, and I'll apply this.

Thanks.

#5 lore raw

PATCH 1/1 vxlan: add necessary locking on device removal

In reply to Pravin Shelar

The socket management is now done in workqueue (outside of RTNL) and protected by vn->sock_lock. There were two possible bugs, first the vxlan device was removed from the VNI hash table per socket without holding lock. And there was a race when device is created and the workqueue could run after deletion.

Signed-off-by Stephen Hemminger <[email protected]>
drivers/net/vxlan.c +6 -0
@@ -1767,9 +1767,15 @@ static int vxlan_newlink(struct net *net
 
 static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 {
+	struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
 	struct vxlan_dev *vxlan = netdev_priv(dev);
 
+	flush_workqueue(vxlan_wq);
+
+	spin_lock(&vn->sock_lock);
 	hlist_del_rcu(&vxlan->hlist);
+	spin_unlock(&vn->sock_lock);
+
 	list_del(&vxlan->next);
 	unregister_netdevice_queue(dev, head);
 }
author
#6 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

Quoted 11 lines · Stephen Hemminger

The socket management is now done in workqueue (outside of RTNL) and protected by vn->sock_lock. There were two possible bugs, first the vxlan device was removed from the VNI hash table per socket without holding lock. And there was a race when device is created and the workqueue could run after deletion.

Signed-off-by Stephen Hemminger <[email protected]>

drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 static int vxlan_newlink(struct net *net · line 1774

Quoted 8 lines
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +3 -0
@@ -1767,9 +1767,15 @@ static int vxlan_newlink(struct net *net
 
 static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 {
+       struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
        struct vxlan_dev *vxlan = netdev_priv(dev);
 
+       flush_workqueue(vxlan_wq);
+

Doesn't this create dependency on sock_work thread while holding RTNL. If so it can result in deadlock.

Quoted 7 lines
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +3 -0
+       spin_lock(&vn->sock_lock);
        hlist_del_rcu(&vxlan->hlist);
+       spin_unlock(&vn->sock_lock);
+
        list_del(&vxlan->next);
        unregister_netdevice_queue(dev, head);
 }
maintainer
#7 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

From: Pravin Shelar <[email protected]>
Date: Sat, 13 Jul 2013 12:21:52 -0700

Quoted 19 lines
Quoted 17 lines · Stephen Hemminger

The socket management is now done in workqueue (outside of RTNL) and protected by vn->sock_lock. There were two possible bugs, first the vxlan device was removed from the VNI hash table per socket without holding lock. And there was a race when device is created and the workqueue could run after deletion.

Signed-off-by Stephen Hemminger <[email protected]>
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +1 -0
@@ -1767,9 +1767,15 @@ static int vxlan_newlink(struct net *net
 
 static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 {
+       struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
        struct vxlan_dev *vxlan = netdev_priv(dev);
Quoted 4 lines

drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 static int vxlan_newlink(struct net *net · line 1774

Quoted 2 lines
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +2 -0
+       flush_workqueue(vxlan_wq);
+

Doesn't this create dependency on sock_work thread while holding RTNL. If so it can result in deadlock.

What exact deadlock do you perceive? I don't see any code path in the sock_work handler (vxlan_sock_work) which takes the RTNL mutex.

So we should be able to safely flush any pending sock_work jobs from vxlan_dellink(). The fact that vxlan_dellink() runs with the RTNL mutex shouldn't cause any issues.

author
#8 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

Quoted 27 lines · David Miller

From: Pravin Shelar <[email protected]>
Date: Sat, 13 Jul 2013 12:21:52 -0700

Quoted 19 lines
Quoted 17 lines · Stephen Hemminger

The socket management is now done in workqueue (outside of RTNL) and protected by vn->sock_lock. There were two possible bugs, first the vxlan device was removed from the VNI hash table per socket without holding lock. And there was a race when device is created and the workqueue could run after deletion.

Signed-off-by Stephen Hemminger <[email protected]>
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +1 -0
@@ -1767,9 +1767,15 @@ static int vxlan_newlink(struct net *net
 
 static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 {
+       struct vxlan_net *vn = net_generic(dev_net(dev), vxlan_net_id);
        struct vxlan_dev *vxlan = netdev_priv(dev);
Quoted 4 lines

drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 static int vxlan_newlink(struct net *net · line 1774

Quoted 2 lines
drivers/net/vxlan.c 2013-07-10 20:15:47.337653899 -0700 renamed from drivers/net/vxlan.c 2013-07-08 16:31:50.080744429 -0700 +2 -0
+       flush_workqueue(vxlan_wq);
+

Doesn't this create dependency on sock_work thread while holding RTNL. If so it can result in deadlock.

Quoted 6 lines

What exact deadlock do you perceive? I don't see any code path in the sock_work handler (vxlan_sock_work) which takes the RTNL mutex.

So we should be able to safely flush any pending sock_work jobs from vxlan_dellink(). The fact that vxlan_dellink() runs with the RTNL mutex shouldn't cause any issues.

commit 1c51a9159ddefa51 (vxlan: fix race caused by dropping rtnl_unlock) moved sock-create to sock_work workq. Thats why I thought sock_work takes RTNL. but if it is not taking RTNL in any case, they why vxlan sock create is deferred to sock_work?

maintainer
#9 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

From: Pravin Shelar <[email protected]>
Date: Tue, 16 Jul 2013 13:29:08 -0700

Quoted 4 lines

commit 1c51a9159ddefa51 (vxlan: fix race caused by dropping rtnl_unlock) moved sock-create to sock_work workq. Thats why I thought sock_work takes RTNL. but if it is not taking RTNL in any case, they why vxlan sock create is deferred to sock_work?

That commit is handling the fact that the RTNL mutex is NOT held during the create operation in question.

It defers to a workqueue so that the new ->sock_lock can be taken in the appropriate context.

author
#10 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

Quoted 8 lines · David Miller

From: Pravin Shelar <[email protected]>
Date: Tue, 16 Jul 2013 13:29:08 -0700

Quoted 4 lines

commit 1c51a9159ddefa51 (vxlan: fix race caused by dropping rtnl_unlock) moved sock-create to sock_work workq. Thats why I thought sock_work takes RTNL. but if it is not taking RTNL in any case, they why vxlan sock create is deferred to sock_work?

Quoted 5 lines

That commit is handling the fact that the RTNL mutex is NOT held during the create operation in question.

It defers to a workqueue so that the new ->sock_lock can be taken in the appropriate context.

ok, Thanks for explanation. I do not see any problem with the patch.

maintainer
#11 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

From: Pravin Shelar <[email protected]>
Date: Wed, 17 Jul 2013 08:41:33 -0700

Quoted 9 lines · David Miller
Quoted 8 lines

From: Pravin Shelar <[email protected]>
Date: Tue, 16 Jul 2013 13:29:08 -0700

Quoted 4 lines

commit 1c51a9159ddefa51 (vxlan: fix race caused by dropping rtnl_unlock) moved sock-create to sock_work workq. Thats why I thought sock_work takes RTNL. but if it is not taking RTNL in any case, they why vxlan sock create is deferred to sock_work?

Quoted 7 lines
Quoted 5 lines

That commit is handling the fact that the RTNL mutex is NOT held during the create operation in question.

It defers to a workqueue so that the new ->sock_lock can be taken in the appropriate context.

ok, Thanks for explanation. I do not see any problem with the patch.

Great, I'm going to apply Stephen's patch then.

Thanks.

maintainer Applied
#12 lore raw

Re: [PATCH net] vxlan: add necessary locking on device removal

In reply to Stephen Hemminger

From: Stephen Hemminger <[email protected]>
Date: Sat, 13 Jul 2013 10:18:18 -0700

Quoted 7 lines

The socket management is now done in workqueue (outside of RTNL) and protected by vn->sock_lock. There were two possible bugs, first the vxlan device was removed from the VNI hash table per socket without holding lock. And there was a race when device is created and the workqueue could run after deletion.

Signed-off-by Stephen Hemminger <[email protected]>

Applied.