Skip to contentSkip to conversation

Revert "bonding: remove sysfs before removing devices"

PATCH Applied Nikolay Aleksandrov 3 participants 9 messages Apr 6, 2013 – Apr 8, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 Revert "bonding: remove sysfs before removing devices"

This reverts commit 4de79c737b200492195ebc54a887075327e1ec1d.

This patch introduces a new bug which causes access to freed memory. In bond_uninit: list_del(&bond->bond_list);
bond_list is linked in bond_net's dev_list which is freed by unregister_pernet_subsys.

---
 drivers/net/bonding/bond_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/bonding/bond_main.c +1 -1
index 171b10f..a51241b 100644
@@ -4902,8 +4902,8 @@ static void __exit bonding_exit(void)
 
 	bond_destroy_debugfs();
 
-	unregister_pernet_subsys(&bond_net_ops);
 	rtnl_link_unregister(&bond_link_ops);
+	unregister_pernet_subsys(&bond_net_ops);
 
 #ifdef CONFIG_NET_POLL_CONTROLLER
 	/*
Signature · 1 line
1.8.1.4
#1 lore raw

PATCH 2/2 bonding: fix bonding_masters race condition in bond unloading

While the bonding module is unloading, it is considered that after rtnl_link_unregister all bond devices are destroyed but since no synchronization mechanism exists, a new bond device can be created via bonding_masters before unregister_pernet_subsys which would lead to multiple problems (e.g. NULL pointer dereference, wrong RIP, list corruption).

This patch fixes the issue by removing any bond devices left in the netns after bonding_masters is removed from sysfs.

---
 drivers/net/bonding/bond_main.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/net/bonding/bond_main.c +9 -0
index a51241b..07401a3 100644
@@ -4846,9 +4846,18 @@ static int __net_init bond_net_init(struct net *net)
 static void __net_exit bond_net_exit(struct net *net)
 {
 	struct bond_net *bn = net_generic(net, bond_net_id);
+	struct bonding *bond, *tmp_bond;
+	LIST_HEAD(list);
 
 	bond_destroy_sysfs(bn);
 	bond_destroy_proc_dir(bn);
+
+	/* Kill off any bonds created after unregistering bond rtnl ops */
+	rtnl_lock();
+	list_for_each_entry_safe(bond, tmp_bond, &bn->dev_list, bond_list)
+		unregister_netdevice_queue(bond->dev, &list);
+	unregister_netdevice_many(&list);
+	rtnl_unlock();
 }
 
 static struct pernet_operations bond_net_ops = {
Signature · 1 line
1.8.1.4
#2 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

Quoted 7 lines · Nikolay Aleksandrov

While the bonding module is unloading, it is considered that after rtnl_link_unregister all bond devices are destroyed but since no synchronization mechanism exists, a new bond device can be created via bonding_masters before unregister_pernet_subsys which would lead to multiple problems (e.g. NULL pointer dereference, wrong RIP, list corruption).

Quoted 8 lines

This patch fixes the issue by removing any bond devices left in the netns after bonding_masters is removed from sysfs.

---
drivers/net/bonding/bond_main.c | 9 +++++++++

1 file changed, 9 insertions(+)

I'm still thinking that's it's not the best way of fixing it (remove_devices(); remove_sysfs(); remove_devices()) - but given that I can't come up with anything better and my first fix didn't actually work - I'm ok with your patch.

#3 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

Quoted 17 lines · Veaceslav Falico
Quoted 7 lines · Nikolay Aleksandrov

While the bonding module is unloading, it is considered that after rtnl_link_unregister all bond devices are destroyed but since no synchronization mechanism exists, a new bond device can be created via bonding_masters before unregister_pernet_subsys which would lead to multiple problems (e.g. NULL pointer dereference, wrong RIP, list corruption).

Quoted 8 lines

This patch fixes the issue by removing any bond devices left in the netns after bonding_masters is removed from sysfs.

---
drivers/net/bonding/bond_main.c | 9 +++++++++

1 file changed, 9 insertions(+)
Quoted 4 lines

I'm still thinking that's it's not the best way of fixing it (remove_devices(); remove_sysfs(); remove_devices()) - but given that I can't come up with anything better and my first fix didn't actually work - I'm ok with your patch.

I think I've found a proper way to do it. Even with your approach we still might end up in some kind of race condition with procfs (check bond_net_exit() -> proc removal, it's made without rtnl_lock()). So the best way would be to lock both functions (__rtnl_link_unregister() and unregister_pernet_subsys()) with rtnl_lock(). It wasn't possible because of a possible race with sysfs (we start removing the bonding, lock rtnl(), someone accesses sysfs(), and our sysfs removal code blocks because of this access - deadlock).

However, if we use the rtnl_trylock() mechanism, we will be able to let sysfs go and finish the removal.

What do you think about this approach? A quick-n-dirty patch is below, I'm running rmmod/insmod for an hour already and it seems to work, however there still might be bugs, and the patch definitely needs some cleaning/comments.

From 3a7858ec5d8ef3261dd52fcd35048cb737aec780 Mon Sep 17 00:00:00 2001
From: Veaceslav Falico <[email protected]>
Date: Mon, 8 Apr 2013 10:29:46 +0200
Subject: [PATCH] bonding: properly protect bonding_exit()

We might race with sysfs/procfs on exit, so protect them with rtnl_lock. Also, convert all sysfs code to rtnl_trylock()/restart_syscall(), so that we don't end up in deadlock.

---
 drivers/net/bonding/bond_main.c  |   13 ++++++-------
 drivers/net/bonding/bond_sysfs.c |   11 ++++++++---
 2 files changed, 14 insertions(+), 10 deletions(-)
drivers/net/bonding/bond_main.c +6 -7
index 2aac890..6671f89 100644
@@ -4797,22 +4797,17 @@ static struct rtnl_link_ops bond_link_ops __read_mostly = {
 
 /* Create a new bond based on the specified name and bonding parameters.
  * If name is NULL, obtain a suitable "bond%d" name for us.
- * Caller must NOT hold rtnl_lock; we need to release it here before we
- * set up our sysfs entries.
  */
 int bond_create(struct net *net, const char *name)
 {
 	struct net_device *bond_dev;
 	int res;
 
-	rtnl_lock();
-
 	bond_dev = alloc_netdev_mq(sizeof(struct bonding),
 				   name ? name : "bond%d",
 				   bond_setup, tx_queues);
 	if (!bond_dev) {
 		pr_err("%s: eek! can't alloc netdev!\n", name);
-		rtnl_unlock();
 		return -ENOMEM;
 	}
 
@@ -4823,9 +4818,9 @@ int bond_create(struct net *net, const char *name)
 
 	netif_carrier_off(bond_dev);
 
-	rtnl_unlock();
 	if (res < 0)
 		bond_destructor(bond_dev);
+
 	return res;
 }
 
@@ -4879,7 +4874,9 @@ static int __init bonding_init(void)
 	bond_create_debugfs();
 
 	for (i = 0; i < max_bonds; i++) {
+		rtnl_lock();
 		res = bond_create(&init_net, NULL);
+		rtnl_unlock();
 		if (res)
 			goto err;
 	}
@@ -4901,8 +4898,10 @@ static void __exit bonding_exit(void)
 
 	bond_destroy_debugfs();
 
+	rtnl_lock();
+	__rtnl_link_unregister(&bond_link_ops);
 	unregister_pernet_subsys(&bond_net_ops);
-	rtnl_link_unregister(&bond_link_ops);
+	rtnl_unlock();
 
 #ifdef CONFIG_NET_POLL_CONTROLLER
 	/*
drivers/net/bonding/bond_sysfs.c +8 -3
index ea7a388..cd1d60f 100644
@@ -59,7 +59,8 @@ static ssize_t bonding_show_bonds(struct class *cls,
 	int res = 0;
 	struct bonding *bond;
 
-	rtnl_lock();
+	if (!rtnl_trylock())
+		return restart_syscall();
 
 	list_for_each_entry(bond, &bn->dev_list, bond_list) {
 		if (res > (PAGE_SIZE - IFNAMSIZ)) {
@@ -107,6 +108,9 @@ static ssize_t bonding_store_bonds(struct class *cls,
 	char *ifname;
 	int rv, res = count;
 
+	if (!rtnl_trylock())
+		return restart_syscall();
+
 	sscanf(buffer, "%16s", command); /* IFNAMSIZ*/
 	ifname = command + 1;
 	if ((strlen(command) <= 1) ||
@@ -126,7 +130,6 @@ static ssize_t bonding_store_bonds(struct class *cls,
 	} else if (command[0] == '-') {
 		struct net_device *bond_dev;
 
-		rtnl_lock();
 		bond_dev = bond_get_by_name(bn, ifname);
 		if (bond_dev) {
 			pr_info("%s is being deleted...\n", ifname);
@@ -135,10 +138,11 @@ static ssize_t bonding_store_bonds(struct class *cls,
 			pr_err("unable to delete non-existent %s\n", ifname);
 			res = -ENODEV;
 		}
-		rtnl_unlock();
 	} else
 		goto err_no_cmd;
 
+	rtnl_unlock();
+
 	/* Always return either count or an error.  If you return 0, you'll
 	 * get called forever, which is bad.
 	 */
@@ -146,6 +150,7 @@ static ssize_t bonding_store_bonds(struct class *cls,
 
 err_no_cmd:
 	pr_err("no command found in bonding_masters. Use +ifname or -ifname.\n");
+	rtnl_unlock();
 	return -EPERM;
 }
 
Signature · 1 line
1.7.1
#4 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

Quoted 37 lines
---
 drivers/net/bonding/bond_main.c  |   13 ++++++-------
 drivers/net/bonding/bond_sysfs.c |   11 ++++++++---
 2 files changed, 14 insertions(+), 10 deletions(-)
drivers/net/bonding/bond_main.c +0 -5
index 2aac890..6671f89 100644
@@ -4797,22 +4797,17 @@ static struct rtnl_link_ops bond_link_ops
 __read_mostly = {
 
 /* Create a new bond based on the specified name and bonding parameters.
  * If name is NULL, obtain a suitable "bond%d" name for us.
- * Caller must NOT hold rtnl_lock; we need to release it here before we
- * set up our sysfs entries.
  */
 int bond_create(struct net *net, const char *name)
 {
     struct net_device *bond_dev;
     int res;
 
-    rtnl_lock();
-
     bond_dev = alloc_netdev_mq(sizeof(struct bonding),
                    name ? name : "bond%d",
                    bond_setup, tx_queues);
     if (!bond_dev) {
         pr_err("%s: eek! can't alloc netdev!\n", name);
-        rtnl_unlock();
         return -ENOMEM;
     }

drivers/net/bonding/bond_main.c int bond_create(struct net *net, const char *name) · line 4826

Quoted 10 lines
drivers/net/bonding/bond_main.c +1 -1
@@ -4823,9 +4818,9 @@ int bond_create(struct net *net, const char *name)
 
     netif_carrier_off(bond_dev);
 
-    rtnl_unlock();
     if (res < 0)
         bond_destructor(bond_dev);
+
     return res;
 }
 

bond_destructor calls free_netdev, which is usually called without rtnl after unregister_netdevice is called under rtnl. (net/core/dev.c - free_netdev comments)

Quoted 11 lines
drivers/net/bonding/bond_main.c +2 -0
@@ -4879,7 +4874,9 @@ static int __init bonding_init(void)
     bond_create_debugfs();
 
     for (i = 0; i < max_bonds; i++) {
+        rtnl_lock();
         res = bond_create(&init_net, NULL);
+        rtnl_unlock();
         if (res)
             goto err;
     }

drivers/net/bonding/bond_main.c static void __exit bonding_exit(void) · line 4905

Quoted 9 lines
drivers/net/bonding/bond_main.c +3 -1
@@ -4901,8 +4898,10 @@ static void __exit bonding_exit(void)
 
     bond_destroy_debugfs();
 
+    rtnl_lock();
+    __rtnl_link_unregister(&bond_link_ops);
     unregister_pernet_subsys(&bond_net_ops);
-    rtnl_link_unregister(&bond_link_ops);
+    rtnl_unlock();
 

The usual way is to obtain net_mutex and then rtnl, this reverses it.

Quoted 58 lines
drivers/net/bonding/bond_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
     /*
drivers/net/bonding/bond_sysfs.c +7 -3
index ea7a388..cd1d60f 100644
@@ -59,7 +59,8 @@ static ssize_t bonding_show_bonds(struct class *cls,
     int res = 0;
     struct bonding *bond;
 
-    rtnl_lock();
+    if (!rtnl_trylock())
+        return restart_syscall();
 
     list_for_each_entry(bond, &bn->dev_list, bond_list) {
         if (res > (PAGE_SIZE - IFNAMSIZ)) {
@@ -107,6 +108,9 @@ static ssize_t bonding_store_bonds(struct class *cls,
     char *ifname;
     int rv, res = count;
 
+    if (!rtnl_trylock())
+        return restart_syscall();
+
     sscanf(buffer, "%16s", command); /* IFNAMSIZ*/
     ifname = command + 1;
     if ((strlen(command) <= 1) ||
@@ -126,7 +130,6 @@ static ssize_t bonding_store_bonds(struct class *cls,
     } else if (command[0] == '-') {
         struct net_device *bond_dev;
 
-        rtnl_lock();
         bond_dev = bond_get_by_name(bn, ifname);
         if (bond_dev) {
             pr_info("%s is being deleted...\n", ifname);
@@ -135,10 +138,11 @@ static ssize_t bonding_store_bonds(struct class
 *cls,
             pr_err("unable to delete non-existent %s\n", ifname);
             res = -ENODEV;
         }
-        rtnl_unlock();
     } else
         goto err_no_cmd;
 
+    rtnl_unlock();
+
     /* Always return either count or an error.  If you return 0, you'll
      * get called forever, which is bad.
      */
drivers/net/bonding/bond_sysfs.c +1 -1
@@ -146,6 +150,7 @@ static ssize_t bonding_store_bonds(struct class *cls,
 
 err_no_cmd:
     pr_err("no command found in bonding_masters. Use +ifname or
-ifname.\n");
+    rtnl_unlock();
     return -EPERM;
 }
#5 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

On Mon, Apr 08, 2013 at 11:15:23AM +0200, Nikolay Aleksandrov wrote:
<snip>

Quoted 4 lines
Quoted 4 lines
drivers/net/bonding/bond_main.c +0 -0
@@ -4823,9 +4818,9 @@ int bond_create(struct net *net, const char *name)
 
     netif_carrier_off(bond_dev);
 
Quoted 10 lines

drivers/net/bonding/bond_main.c int bond_create(struct net *net, const char *name) · line 4826

Quoted 7 lines
drivers/net/bonding/bond_main.c +1 -1
-    rtnl_unlock();
     if (res < 0)
         bond_destructor(bond_dev);
+
     return res;
 }
 

bond_destructor calls free_netdev, which is usually called without rtnl after unregister_netdevice is called under rtnl. (net/core/dev.c - free_netdev comments)

It shouldn't be called under rtnl_lock() mainly because of sysfs(), however with this patch we've added rtnl_trylock() to it and should be safe.

It's already used under rtnl_lock() in several places already, so I think it's safe.

Quoted 14 lines
Quoted 14 lines
drivers/net/bonding/bond_main.c +2 -0
@@ -4879,7 +4874,9 @@ static int __init bonding_init(void)
     bond_create_debugfs();
 
     for (i = 0; i < max_bonds; i++) {
+        rtnl_lock();
         res = bond_create(&init_net, NULL);
+        rtnl_unlock();
         if (res)
             goto err;
     }
@@ -4901,8 +4898,10 @@ static void __exit bonding_exit(void)
 
     bond_destroy_debugfs();
 
Quoted 8 lines

drivers/net/bonding/bond_main.c static void __exit bonding_exit(void) · line 4905

Quoted 6 lines
drivers/net/bonding/bond_main.c +3 -1
+    rtnl_lock();
+    __rtnl_link_unregister(&bond_link_ops);
     unregister_pernet_subsys(&bond_net_ops);
-    rtnl_link_unregister(&bond_link_ops);
+    rtnl_unlock();
 

The usual way is to obtain net_mutex and then rtnl, this reverses it.

Good point, we might easily deadlock here. I'll dig and come back if I'll find a way to avoid it...

Quoted 59 lines
Quoted 58 lines
drivers/net/bonding/bond_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
     /*
drivers/net/bonding/bond_sysfs.c +7 -3
index ea7a388..cd1d60f 100644
@@ -59,7 +59,8 @@ static ssize_t bonding_show_bonds(struct class *cls,
     int res = 0;
     struct bonding *bond;
 
-    rtnl_lock();
+    if (!rtnl_trylock())
+        return restart_syscall();
 
     list_for_each_entry(bond, &bn->dev_list, bond_list) {
         if (res > (PAGE_SIZE - IFNAMSIZ)) {
@@ -107,6 +108,9 @@ static ssize_t bonding_store_bonds(struct class *cls,
     char *ifname;
     int rv, res = count;
 
+    if (!rtnl_trylock())
+        return restart_syscall();
+
     sscanf(buffer, "%16s", command); /* IFNAMSIZ*/
     ifname = command + 1;
     if ((strlen(command) <= 1) ||
@@ -126,7 +130,6 @@ static ssize_t bonding_store_bonds(struct class *cls,
     } else if (command[0] == '-') {
         struct net_device *bond_dev;
 
-        rtnl_lock();
         bond_dev = bond_get_by_name(bn, ifname);
         if (bond_dev) {
             pr_info("%s is being deleted...\n", ifname);
@@ -135,10 +138,11 @@ static ssize_t bonding_store_bonds(struct class
 *cls,
             pr_err("unable to delete non-existent %s\n", ifname);
             res = -ENODEV;
         }
-        rtnl_unlock();
     } else
         goto err_no_cmd;
 
+    rtnl_unlock();
+
     /* Always return either count or an error.  If you return 0, you'll
      * get called forever, which is bad.
      */
drivers/net/bonding/bond_sysfs.c +1 -1
@@ -146,6 +150,7 @@ static ssize_t bonding_store_bonds(struct class *cls,
 
 err_no_cmd:
     pr_err("no command found in bonding_masters. Use +ifname or
-ifname.\n");
+    rtnl_unlock();
     return -EPERM;
 }
Acked-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

In reply to Nikolay Aleksandrov
Quoted 7 lines · Nikolay Aleksandrov

While the bonding module is unloading, it is considered that after rtnl_link_unregister all bond devices are destroyed but since no synchronization mechanism exists, a new bond device can be created via bonding_masters before unregister_pernet_subsys which would lead to multiple problems (e.g. NULL pointer dereference, wrong RIP, list corruption).

Quoted 8 lines

This patch fixes the issue by removing any bond devices left in the netns after bonding_masters is removed from sysfs.

---
drivers/net/bonding/bond_main.c | 9 +++++++++

1 file changed, 9 insertions(+)

Given that this bug is in net tree and we really need a fix to continue development, adding a second device removal loop is not a big deal actually, and this can be addressed correctly later in net-next. It's a really rare bug anyway, and the second loop most of the time would be a simple no-op, as Nikolay correctly stated.

maintainer Applied
#7 lore raw

Re: [PATCH 2/2] bonding: fix bonding_masters race condition in bond unloading

In reply to Nikolay Aleksandrov

From: Nikolay Aleksandrov <[email protected]>
Date: Sat, 6 Apr 2013 12:54:38 +0200

Quoted 11 lines

While the bonding module is unloading, it is considered that after rtnl_link_unregister all bond devices are destroyed but since no synchronization mechanism exists, a new bond device can be created via bonding_masters before unregister_pernet_subsys which would lead to multiple problems (e.g. NULL pointer dereference, wrong RIP, list corruption).

This patch fixes the issue by removing any bond devices left in the netns after bonding_masters is removed from sysfs.

Applied.

maintainer Applied
#8 lore raw
In reply to Nikolay Aleksandrov

From: Nikolay Aleksandrov <[email protected]>
Date: Sat, 6 Apr 2013 12:54:37 +0200

Quoted 8 lines

This reverts commit 4de79c737b200492195ebc54a887075327e1ec1d.

This patch introduces a new bug which causes access to freed memory. In bond_uninit: list_del(&bond->bond_list);
bond_list is linked in bond_net's dev_list which is freed by unregister_pernet_subsys.

Applied.