Skip to contentSkip to conversation

netpoll: avoid reference leaks

PATCH Superseded David Decotigny 2 participants 7 messages Jun 30, 2014 – Jul 8, 2014 1 message not in the archive

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 1/2 netpoll: fix use after free

After a bonding master reclaims the netpoll info struct, slaves could still hold a pointer to the reclaimed data. This patch fixes it: as soon as netpoll_async_cleanup is called for a slave (eg. when un-enslaved), we make sure that this slave doesn't point to the data.

The style of this patch is not nice, but it represents the simplest fix. The function is restructured by the next patch anyway ("netpoll:
avoid reference leaks").

Tested:
One way to reproduce the panic is with netconsole: with the script below run in a loop without this patch. Same procedure with this patch can run without panic.
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
Then run in a loop (dual port NIC makes it easier to crash):
ifconfig eth0 192.168.42.4 up
ifconfig eth1 192.168.56.4 up
sleep 10
modprobe -r netconsole
modprobe -r bonding
modprobe netconsole
modprobe bonding mode=4
echo +bond0 > /sys/class/net/bonding_masters
ifconfig bond0 192.168.56.3 up
mkdir /sys/kernel/config/netconsole/blah
echo 0 > /sys/kernel/config/netconsole/blah/enabled
echo bond0 > /sys/kernel/config/netconsole/blah/dev_name echo 192.168.56.42 > /sys/kernel/config/netconsole/blah/remote_ip echo 1 > /sys/kernel/config/netconsole/blah/enabled
# npinfo refcnt ->1
ifenslave bond0 eth1
# npinfo refcnt ->2
ifenslave bond0 eth0
# (this should be optional, preventing ndo_cleanup_nepoll below) # npinfo refcnt ->3
sleep 3
ifenslave -d bond0 eth1
# npinfo refcnt ->2, eth1 keeps ptr to npinfo reclaimed later => garbage sleep 1
echo -bond0 > /sys/class/net/bonding_masters
# netpoll_cleanup(bond0) + dec(refcnt)
# (should be optional: becomes -> 1 (aka. != 0)
# ==> do not call ndo_cleanup_nepoll)
# try to increase chance of writing garbage onto npinfo:
ls -lR / 2>&1 | tail -30
echo +bond0 > /sys/class/net/bonding_masters
ifconfig bond0 192.168.56.3 up
ifenslave bond0 eth1
# dev_open() called --> netpoll_rx_disable() + npinfo is garbage
# -> down(&ni->dev_lock);

---
 net/core/netpoll.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/core/netpoll.c +2 -1
index e33937f..907fb5e 100644
@@ -822,7 +822,8 @@ void __netpoll_cleanup(struct netpoll *np)
 
 		RCU_INIT_POINTER(np->dev->npinfo, NULL);
 		call_rcu_bh(&npinfo->rcu, rcu_cleanup_netpoll_info);
-	}
+	} else
+		RCU_INIT_POINTER(np->dev->npinfo, NULL);
 }
 EXPORT_SYMBOL_GPL(__netpoll_cleanup);
 
Signature · 1 line
2.0.0.526.g5318336
#2 lore raw

PATCH 2/2 netpoll: avoid reference leaks

This ensures that the ndo_netpoll_cleanup callback is called for every device that provides one. Otherwise there is a risk of reference leak with bonding for example, which depends on this callback to cleanup the slaves' references to netpoll info.

Tested:
see patch "netpoll: fix use after free"

---
 net/core/netpoll.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
net/core/netpoll.c +9 -8
index 907fb5e..1e10d5d 100644
@@ -802,6 +802,7 @@ static void rcu_cleanup_netpoll_info(struct rcu_head *rcu_head)
 void __netpoll_cleanup(struct netpoll *np)
 {
 	struct netpoll_info *npinfo;
+	const struct net_device_ops *ops;
 
 	/* rtnl_dereference would be preferable here but
 	 * rcu_cleanup_netpoll path can put us in here safely without
@@ -813,17 +814,17 @@ void __netpoll_cleanup(struct netpoll *np)
 
 	synchronize_srcu(&netpoll_srcu);
 
-	if (atomic_dec_and_test(&npinfo->refcnt)) {
-		const struct net_device_ops *ops;
+	ops = np->dev->netdev_ops;
+	if (ops->ndo_netpoll_cleanup)
+		ops->ndo_netpoll_cleanup(np->dev);
 
-		ops = np->dev->netdev_ops;
-		if (ops->ndo_netpoll_cleanup)
-			ops->ndo_netpoll_cleanup(np->dev);
+	/* before dropping ref count, make sure this device does not
+	 * reference npinfo anymore
+	 */
+	RCU_INIT_POINTER(np->dev->npinfo, NULL);
 
-		RCU_INIT_POINTER(np->dev->npinfo, NULL);
+	if (atomic_dec_and_test(&npinfo->refcnt))
 		call_rcu_bh(&npinfo->rcu, rcu_cleanup_netpoll_info);
-	} else
-		RCU_INIT_POINTER(np->dev->npinfo, NULL);
 }
 EXPORT_SYMBOL_GPL(__netpoll_cleanup);
 
Signature · 1 line
2.0.0.526.g5318336
maintainer
#3 lore raw

From: David Decotigny <[email protected]>
Date: Mon, 30 Jun 2014 16:50:10 -0700

Quoted 9 lines

This ensures that the ndo_netpoll_cleanup callback is called for every device that provides one. Otherwise there is a risk of reference leak with bonding for example, which depends on this callback to cleanup the slaves' references to netpoll info.

Tested:
see patch "netpoll: fix use after free"

I definitely don't understand this.

Why would we call the cleanup function of an object before it's reference count hits zero? It is exactly the act of reaching a zero refcount which should trigger invoking the cleanup callback.

If a refcount is being released in another location without checking if it hits zero and invoking the cleanup if so, _THAT_ is the bug.

#4 lore raw

Hi,

Thanks for the feedback. This patch results from manual inspection of the code. I agree my commit description is abusive: in the case of bonding, I think everything is fine, there should be no ref leak, cleanup paths seem clean.

My point was to make things more predictable: ndo_netpoll_cleanup called anyways to acknowledge actual loss of a ref to npinfo, irrespective of whether it's the last ref or not. Without this patch, calling ndo_netpoll_cleanup would depend on some timing behavior, hard to predict, and users of the API have better be careful to reclaim the refs manually anyways: as a consequence, not sure this callback is actually required in its current inception.

Quoted 21 lines · David Miller

From: David Decotigny <[email protected]>
Date: Mon, 30 Jun 2014 16:50:10 -0700

Quoted 9 lines

This ensures that the ndo_netpoll_cleanup callback is called for every device that provides one. Otherwise there is a risk of reference leak with bonding for example, which depends on this callback to cleanup the slaves' references to netpoll info.

Tested:
see patch "netpoll: fix use after free"

I definitely don't understand this.

Why would we call the cleanup function of an object before it's reference count hits zero? It is exactly the act of reaching a zero refcount which should trigger invoking the cleanup callback.

If a refcount is being released in another location without checking if it hits zero and invoking the cleanup if so, _THAT_ is the bug.

maintainer
#5 lore raw

From: David Decotigny <[email protected]>
Date: Tue, 8 Jul 2014 12:35:14 -0700

Quoted 5 lines

Thanks for the feedback. This patch results from manual inspection of the code. I agree my commit description is abusive: in the case of bonding, I think everything is fine, there should be no ref leak, cleanup paths seem clean.

Quoted 7 lines

My point was to make things more predictable: ndo_netpoll_cleanup called anyways to acknowledge actual loss of a ref to npinfo, irrespective of whether it's the last ref or not. Without this patch, calling ndo_netpoll_cleanup would depend on some timing behavior, hard to predict, and users of the API have better be careful to reclaim the refs manually anyways: as a consequence, not sure this callback is actually required in its current inception.

You've increased my confusion rather than decreased it.

You fail to address the core issue in my feedback:

Whoever drops the refcount to zero must be the one to invoke the cleanup function.

Please address this concisely, and directly.

#6 lore raw

In that case, that's what the original code does: dropping this patch 2/2.

Patch 1/2 "netpoll: fix use after free" is still needed to prevent panics, though.

Quoted 24 lines · David Miller

From: David Decotigny <[email protected]>
Date: Tue, 8 Jul 2014 12:35:14 -0700

Quoted 5 lines

Thanks for the feedback. This patch results from manual inspection of the code. I agree my commit description is abusive: in the case of bonding, I think everything is fine, there should be no ref leak, cleanup paths seem clean.

Quoted 7 lines

My point was to make things more predictable: ndo_netpoll_cleanup called anyways to acknowledge actual loss of a ref to npinfo, irrespective of whether it's the last ref or not. Without this patch, calling ndo_netpoll_cleanup would depend on some timing behavior, hard to predict, and users of the API have better be careful to reclaim the refs manually anyways: as a consequence, not sure this callback is actually required in its current inception.

You've increased my confusion rather than decreased it.

You fail to address the core issue in my feedback:

Whoever drops the refcount to zero must be the one to invoke the cleanup function.

Please address this concisely, and directly.

maintainer
#7 lore raw

From: David Decotigny <[email protected]>
Date: Tue, 8 Jul 2014 14:50:24 -0700

Quoted 4 lines

In that case, that's what the original code does: dropping this patch 2/2.

Patch 1/2 "netpoll: fix use after free" is still needed to prevent panics, though.

Please resubmit it then.