Skip to contentSkip to conversation

ipv4: remove all rt cache entries on UNREGISTER event

PATCH In review Nicolas Dichtel 4 participants 16 messages Sep 28, 2010 – Dec 26, 2010

Subsystems (3)
Networking Network drivers IPv4/IPv6
lore
Share & tools
lore

Nicolas Dichtel

author
#0 lore raw

PATCH 1/1 ipv4: remove all rt cache entries on UNREGISTER event

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Note2: when a.out is executed, tunl1 has no ip address and is down.

Then, I got a serie of "kernel:[1206699.728010] unregister_netdevice: waiting for tunl1 to become free. Usage count = 3" and after some time, interface is removed.

The problem is that route cache entries are only invalidate on UNREGISTER event, and not removed (introduced by commit e2ce146848c81af2f6d42e67990191c284bf0c33). We must wait that rt_check_expire() remove the remaining route cache entries.

To fix the problem, I propose to remove a part of the previous commit.

Regards,
Nicolas

Quoted 1 line

From 3344e2e0431fe803c4dac8757a8746908357d780 Mon Sep 17 00:00:00 2001

From: Nicolas Dichtel <[email protected]>
Date: Tue, 28 Sep 2010 16:38:19 +0200
Subject: [PATCH] ipv4: remove all rt cache entries on UNREGISTER event

Commit e2ce146848c81af2f6d42e67990191c284bf0c33 (ipv4: factorize cache clearing for batched unregister operations) add a new parameter to fib_disable_ip() to only invalidate route cache entries on unregister event.
This is wrong, we should ensure that all cache entries are removed on unregister event, else netdev_wait_allrefs() may complain. A cache entry can be created between event DOWN and UNREGISTER.

So, I revert a part of the patch.

Signed-off-by Nicolas Dichtel <[email protected]>
---
 net/ipv4/fib_frontend.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
net/ipv4/fib_frontend.c +5 -5
index 7d02a9f..377e815 100644
@@ -917,11 +917,11 @@ static void nl_fib_lookup_exit(struct net *net)
 	net->ipv4.fibnl = NULL;
 }
 
-static void fib_disable_ip(struct net_device *dev, int force, int delay)
+static void fib_disable_ip(struct net_device *dev, int force)
 {
 	if (fib_sync_down_dev(dev, force))
 		fib_flush(dev_net(dev));
-	rt_cache_flush(dev_net(dev), delay);
+	rt_cache_flush(dev_net(dev), 0);
 	arp_ifdown(dev);
 }
 

@@ -944,7 +944,7 @@ static int fib_inetaddr_event(struct notifier_block *this, unsigned long event,
 			/* Last address was deleted from this interface.
 			   Disable IP.
 			 */
-			fib_disable_ip(dev, 1, 0);
+			fib_disable_ip(dev, 1);
 		} else {
 			rt_cache_flush(dev_net(dev), -1);
 		}

@@ -959,7 +959,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 	struct in_device *in_dev = __in_dev_get_rtnl(dev);
 
 	if (event == NETDEV_UNREGISTER) {
-		fib_disable_ip(dev, 2, -1);
+		fib_disable_ip(dev, 2);
 		return NOTIFY_DONE;
 	}
 

@@ -977,7 +977,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 		rt_cache_flush(dev_net(dev), -1);
 		break;
 	case NETDEV_DOWN:
-		fib_disable_ip(dev, 0, 0);
+		fib_disable_ip(dev, 0);
 		break;
 	case NETDEV_CHANGEMTU:
 	case NETDEV_CHANGE:
Signature · 1 line
1.5.6.5
#1 lore raw
Quoted 10 lines · Nicolas Dichtel

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1
Quoted 7 lines

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Note2: when a.out is executed, tunl1 has no ip address and is down.

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

Maybe a fix would be to forbid this ?

Some machines have a giant route cache, so its very important to avoid expensive scans.

Quoted 79 lines

Then, I got a serie of "kernel:[1206699.728010] unregister_netdevice:
waiting for tunl1 to become free. Usage count = 3" and after some time, interface is removed.

The problem is that route cache entries are only invalidate on UNREGISTER event, and not removed (introduced by commit e2ce146848c81af2f6d42e67990191c284bf0c33). We must wait that rt_check_expire() remove the remaining route cache entries.

To fix the problem, I propose to remove a part of the previous commit.

Regards,
Nicolas
pièce jointe différences entre fichiers (0001-ipv4-remove-all-rt-cache-entries-on-UNREGISTER-even.patch) From 3344e2e0431fe803c4dac8757a8746908357d780 Mon Sep 17 00:00:00 2001
From: Nicolas Dichtel <[email protected]>
Date: Tue, 28 Sep 2010 16:38:19 +0200
Subject: [PATCH] ipv4: remove all rt cache entries on UNREGISTER event

Commit e2ce146848c81af2f6d42e67990191c284bf0c33 (ipv4: factorize cache clearing for batched unregister operations) add a new parameter to fib_disable_ip() to only invalidate route cache entries on unregister event.
This is wrong, we should ensure that all cache entries are removed on unregister event, else netdev_wait_allrefs() may complain. A cache entry can be created between event DOWN and UNREGISTER.

So, I revert a part of the patch.

Signed-off-by Nicolas Dichtel <[email protected]>
---
 net/ipv4/fib_frontend.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
net/ipv4/fib_frontend.c +5 -5
index 7d02a9f..377e815 100644
@@ -917,11 +917,11 @@ static void nl_fib_lookup_exit(struct net *net)
 	net->ipv4.fibnl = NULL;
 }
 
-static void fib_disable_ip(struct net_device *dev, int force, int delay)
+static void fib_disable_ip(struct net_device *dev, int force)
 {
 	if (fib_sync_down_dev(dev, force))
 		fib_flush(dev_net(dev));
-	rt_cache_flush(dev_net(dev), delay);
+	rt_cache_flush(dev_net(dev), 0);
 	arp_ifdown(dev);
 }
 

@@ -944,7 +944,7 @@ static int fib_inetaddr_event(struct notifier_block *this, unsigned long event,
 			/* Last address was deleted from this interface.
 			   Disable IP.
 			 */
-			fib_disable_ip(dev, 1, 0);
+			fib_disable_ip(dev, 1);
 		} else {
 			rt_cache_flush(dev_net(dev), -1);
 		}

@@ -959,7 +959,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 	struct in_device *in_dev = __in_dev_get_rtnl(dev);
 
 	if (event == NETDEV_UNREGISTER) {
-		fib_disable_ip(dev, 2, -1);
+		fib_disable_ip(dev, 2);
 		return NOTIFY_DONE;
 	}
 

@@ -977,7 +977,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 		rt_cache_flush(dev_net(dev), -1);
 		break;
 	case NETDEV_DOWN:
-		fib_disable_ip(dev, 0, 0);
+		fib_disable_ip(dev, 0);
 		break;
 	case NETDEV_CHANGEMTU:
 	case NETDEV_CHANGE:
author
#2 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Nicolas Dichtel

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1
Quoted 8 lines
Quoted 4 lines

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Note2: when a.out is executed, tunl1 has no ip address and is down.

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

At first, I asked myself the same question, but it seems that this is allowed to send a packet through this kind of socket, even if interface is down. Packet will be destroyed by the noop qdisk. But I agree that it is strange to perform route lookup and everything to
destroy the packet at the end ...
Maybe raw_sendmsg() can delete it directly ;-) ... or maybe ip_route_output_flow().

Any suggestions welcome.

Regards,
Nicolas

Quoted 82 lines

Maybe a fix would be to forbid this ?

Some machines have a giant route cache, so its very important to avoid expensive scans.

Quoted 74 lines

Then, I got a serie of "kernel:[1206699.728010] unregister_netdevice: waiting for tunl1 to become free. Usage count = 3" and after some time, interface is removed.

The problem is that route cache entries are only invalidate on UNREGISTER event, and not removed (introduced by commit e2ce146848c81af2f6d42e67990191c284bf0c33). We must wait that rt_check_expire() remove the remaining route cache entries.

To fix the problem, I propose to remove a part of the previous commit.

Regards,
Nicolas
pièce jointe différences entre fichiers (0001-ipv4-remove-all-rt-cache-entries-on-UNREGISTER-even.patch) From 3344e2e0431fe803c4dac8757a8746908357d780 Mon Sep 17 00:00:00 2001
From: Nicolas Dichtel <[email protected]>
Date: Tue, 28 Sep 2010 16:38:19 +0200
Subject: [PATCH] ipv4: remove all rt cache entries on UNREGISTER event

Commit e2ce146848c81af2f6d42e67990191c284bf0c33 (ipv4: factorize cache clearing for batched unregister operations) add a new parameter to fib_disable_ip() to only invalidate route cache entries on unregister event.
This is wrong, we should ensure that all cache entries are removed on unregister event, else netdev_wait_allrefs() may complain. A cache entry can be created between event DOWN and UNREGISTER.

So, I revert a part of the patch.

Signed-off-by Nicolas Dichtel <[email protected]>
---
 net/ipv4/fib_frontend.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
net/ipv4/fib_frontend.c +5 -5
index 7d02a9f..377e815 100644
@@ -917,11 +917,11 @@ static void nl_fib_lookup_exit(struct net *net)
 	net->ipv4.fibnl = NULL;
 }
 
-static void fib_disable_ip(struct net_device *dev, int force, int delay)
+static void fib_disable_ip(struct net_device *dev, int force)
 {
 	if (fib_sync_down_dev(dev, force))
 		fib_flush(dev_net(dev));
-	rt_cache_flush(dev_net(dev), delay);
+	rt_cache_flush(dev_net(dev), 0);
 	arp_ifdown(dev);
 }
 

@@ -944,7 +944,7 @@ static int fib_inetaddr_event(struct notifier_block *this, unsigned long event,
 			/* Last address was deleted from this interface.
 			   Disable IP.
 			 */
-			fib_disable_ip(dev, 1, 0);
+			fib_disable_ip(dev, 1);
 		} else {
 			rt_cache_flush(dev_net(dev), -1);
 		}

@@ -959,7 +959,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 	struct in_device *in_dev = __in_dev_get_rtnl(dev);
 
 	if (event == NETDEV_UNREGISTER) {
-		fib_disable_ip(dev, 2, -1);
+		fib_disable_ip(dev, 2);
 		return NOTIFY_DONE;
 	}
 

@@ -977,7 +977,7 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
 		rt_cache_flush(dev_net(dev), -1);
 		break;
 	case NETDEV_DOWN:
-		fib_disable_ip(dev, 0, 0);
+		fib_disable_ip(dev, 0);
 		break;
 	case NETDEV_CHANGEMTU:
 	case NETDEV_CHANGE:
#3 lore raw
Quoted 31 lines · Nicolas Dichtel
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Nicolas Dichtel

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Quoted 6 lines
Quoted 2 lines

Note2: when a.out is executed, tunl1 has no ip address and is down.

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

At first, I asked myself the same question, but it seems that this is allowed to send a packet through this kind of socket, even if interface is down. Packet will be destroyed by the noop qdisk.
But I agree that it is strange to perform route lookup and everything to
destroy the packet at the end ...
Maybe raw_sendmsg() can delete it directly ;-) ... or maybe ip_route_output_flow().

Quoted 2 lines

Any suggestions welcome.

Hmm...

One way to track this kind of problem would be to add a WARN_ON() in dev_hold()

-> Check that when a reference on dev is taken, we are in a known state.

Something like this ?

include/linux/netdevice.h +1 -0
index 83de0eb..54bef78 100644
@@ -1773,6 +1774,7 @@ static inline void dev_put(struct net_device *dev)
  */
 static inline void dev_hold(struct net_device *dev)
 {
+	WARN_ON(dev->reg_state != NETREG_REGISTERED);
 	atomic_inc(&dev->refcnt);
 }
author
#4 lore raw
Quoted 12 lines · Eric Dumazet
Quoted 11 lines · Nicolas Dichtel
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Nicolas Dichtel

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Note2: when a.out is executed, tunl1 has no ip address and is down.

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

The route is created because no function will check interface status (up and running or down). Just at the end, the packet will be enqueued in the noop qdisc.

Quoted 8 lines
Quoted 7 lines

At first, I asked myself the same question, but it seems that this is allowed to send a packet through this kind of socket, even if interface is down. Packet will be destroyed by the noop qdisk. But I agree that it is strange to perform route lookup and everything to
destroy the packet at the end ...
Maybe raw_sendmsg() can delete it directly ;-) ... or maybe ip_route_output_flow().

Any suggestions welcome.

Quoted 8 lines

Hmm...

One way to track this kind of problem would be to add a WARN_ON() in dev_hold()

-> Check that when a reference on dev is taken, we are in a known state.

Something like this ?

dev_hold() is done when interface is down, but before unregistering process start.

Regards,
Nicolas

Quoted 15 lines
include/linux/netdevice.h +1 -0
index 83de0eb..54bef78 100644
@@ -1773,6 +1774,7 @@ static inline void dev_put(struct net_device *dev)
  */
 static inline void dev_hold(struct net_device *dev)
 {
+	WARN_ON(dev->reg_state != NETREG_REGISTERED);
 	atomic_inc(&dev->refcnt);
 }
#5 lore raw
Quoted 19 lines · Nicolas Dichtel
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Nicolas Dichtel
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Nicolas Dichtel

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Note2: when a.out is executed, tunl1 has no ip address and is down.

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

The route is created because no function will check interface status (up and running or down). Just at the end, the packet will be enqueued in the noop qdisc.

In your case maybe, but I think there is another point where we can call dev_hold() while device is in dismantle phase.

Quoted 13 lines
Quoted 13 lines
Quoted 10 lines

At first, I asked myself the same question, but it seems that this is allowed to send a packet through this kind of socket, even if interface is down. Packet will be destroyed by the noop qdisk.
But I agree that it is strange to perform route lookup and everything to
destroy the packet at the end ...
Maybe raw_sendmsg() can delete it directly ;-) ... or maybe ip_route_output_flow().

Any suggestions welcome.

Hmm...

Quoted 8 lines
Quoted 6 lines

One way to track this kind of problem would be to add a WARN_ON() in dev_hold()

-> Check that when a reference on dev is taken, we are in a known state.

Something like this ?

dev_hold() is done when interface is down, but before unregistering process start.

Not on my machine. I can see the backtrace sometimes.

There is a race somewhere (maybe several), and your patch only reduce the window of this race.

I am working on it.

#6 lore raw

I found following patch was enough to avoid route being created if device is down. This is still racy and needs more thinking.

net/ipv4/route.c +3 -2
index ac6559c..1ee0b1a 100644
@@ -2586,9 +2586,10 @@ static int ip_route_output_slow(struct net *net, struct rtable **rp,
 			goto out;
 
 		/* RACE: Check return value of inet_select_addr instead. */
-		if (__in_dev_get_rtnl(dev_out) == NULL) {
+		if (!(dev_out->flags & IFF_UP) || __in_dev_get_rtnl(dev_out) == NULL) {
 			dev_put(dev_out);
-			goto out;	/* Wrong error code */
+			err = -ENETUNREACH;
+			goto out;
 		}
 
 		if (ipv4_is_local_multicast(oldflp->fl4_dst) ||
author Acked-by
#7 lore raw

Patch works well with my case.
In fact, it's more proper to returns an error to the daemon to let it knows that packet was not sent.

Acked-by Nicolas Dichtel <[email protected]>

Regards,
Nicolas

Quoted 23 lines · Eric Dumazet

I found following patch was enough to avoid route being created if device is down. This is still racy and needs more thinking.

net/ipv4/route.c +3 -2
index ac6559c..1ee0b1a 100644
@@ -2586,9 +2586,10 @@ static int ip_route_output_slow(struct net *net, struct rtable **rp,
 			goto out;
 
 		/* RACE: Check return value of inet_select_addr instead. */
-		if (__in_dev_get_rtnl(dev_out) == NULL) {
+		if (!(dev_out->flags & IFF_UP) || __in_dev_get_rtnl(dev_out) == NULL) {
 			dev_put(dev_out);
-			goto out;	/* Wrong error code */
+			err = -ENETUNREACH;
+			goto out;
 		}
 
 		if (ipv4_is_local_multicast(oldflp->fl4_dst) ||
author
#8 lore raw
In reply to Eric Dumazet

What is the status of this patch? The initial problem is still here in 2.6.37-rc5+

Regards,
Nicolas

Quoted 23 lines · Eric Dumazet

I found following patch was enough to avoid route being created if device is down. This is still racy and needs more thinking.

net/ipv4/route.c +3 -2
index ac6559c..1ee0b1a 100644
@@ -2586,9 +2586,10 @@ static int ip_route_output_slow(struct net *net, struct rtable **rp,
 			goto out;
 
 		/* RACE: Check return value of inet_select_addr instead. */
-		if (__in_dev_get_rtnl(dev_out) == NULL) {
+		if (!(dev_out->flags & IFF_UP) || __in_dev_get_rtnl(dev_out) == NULL) {
 			dev_put(dev_out);
-			goto out;	/* Wrong error code */
+			err = -ENETUNREACH;
+			goto out;
 		}
 
 		if (ipv4_is_local_multicast(oldflp->fl4_dst) ||
#9 lore raw
Quoted 2 lines · Nicolas Dichtel

What is the status of this patch? The initial problem is still here in 2.6.37-rc5+

I cannot reproduce the problem on net-next-2.6, are you sure we still need a new patch ?

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ip link show dev tunl1
19: tunl1: <POINTOPOINT,NOARP> mtu 1480 qdisc noop state DOWN 
    link/ipip 10.16.0.72 peer 10.16.0.164
# /root/vrrp tunl1
# ip tunnel del tunl1
#10 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 2 lines · Nicolas Dichtel

What is the status of this patch? The initial problem is still here in 2.6.37-rc5+

I cannot reproduce the problem on net-next-2.6, are you sure we still need a new patch ?

Quoted 8 lines
# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ip link show dev tunl1
19: tunl1: <POINTOPOINT,NOARP> mtu 1480 qdisc noop state DOWN 
    link/ipip 10.16.0.72 peer 10.16.0.164
# /root/vrrp tunl1
# ip tunnel del tunl1

(I thought commit 332dd96f7ac15e fixed the problem for you, as it did for me)

net/dst: dst_dev_event() called after other notifiers

author
#11 lore raw

Yes, I saw this commit, but I still got the problem:

shelby:/home/root/src# uname -a
Linux shelby 2.6.37-rc5+ #10 SMP Wed Dec 22 05:02:53 EST 2010 i686 GNU/Linux
shelby:/home/root/src# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
shelby:/home/root/src# ./a.out tunl1
shelby:/home/root/src# ip tunnel del tunl1

Message from syslogd@shelby at Dec 22 10:12:08 ...
kernel:[18459.828011] unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3

Message from syslogd@shelby at Dec 22 10:12:19 ...
kernel:[18470.072017] unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3

Message from syslogd@shelby at Dec 22 10:12:29 ...
kernel:[18480.316011] unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3

Message from syslogd@shelby at Dec 22 10:12:39 ...
kernel:[18490.560010] unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3
shelby:/home/root/src#

I don't know if I will have time to investigate more before next year.

Regards,
Nicolas

Quoted 23 lines · Eric Dumazet
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Nicolas Dichtel

What is the status of this patch? The initial problem is still here in 2.6.37-rc5+

I cannot reproduce the problem on net-next-2.6, are you sure we still need a new patch ?

Quoted 8 lines
# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ip link show dev tunl1
19: tunl1: <POINTOPOINT,NOARP> mtu 1480 qdisc noop state DOWN 
    link/ipip 10.16.0.72 peer 10.16.0.164
# /root/vrrp tunl1
# ip tunnel del tunl1

(I thought commit 332dd96f7ac15e fixed the problem for you, as it did for me)

net/dst: dst_dev_event() called after other notifiers

Signature · 13 lines
Nicolas DICHTEL
6WIND
R&D Engineer

Tel: +33 1 39 30 92 10
Fax: +33 1 39 30 92 11
[email protected]
www.6wind.com
Join the Multicore Packet Processing Forum: www.multicorepacketprocessing.com

Ce courriel ainsi que toutes les pièces jointes, est uniquement destiné à son ou ses destinataires. Il contient des informations confidentielles qui sont la propriété de 6WIND. Toute révélation, distribution ou copie des informations qu'il contient est strictement interdite. Si vous avez reçu ce message par erreur, veuillez immédiatement le signaler à l'émetteur et détruire toutes les données reçues.

This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and contains information that is confidential and proprietary to 6WIND. All unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
#12 lore raw

PATCH 1/1 ipv4: dont create routes on down devices

Quoted 9 lines · Nicolas Dichtel

Yes, I saw this commit, but I still got the problem:

shelby:/home/root/src# uname -a
Linux shelby 2.6.37-rc5+ #10 SMP Wed Dec 22 05:02:53 EST 2010 i686 GNU/Linux
shelby:/home/root/src# ip tunnel add mode ipip remote 10.16.0.164 local 
10.16.0.72 dev eth0
shelby:/home/root/src# ./a.out tunl1
shelby:/home/root/src# ip tunnel del tunl1
Quoted 4 lines

Message from syslogd@shelby at Dec 22 10:12:08 ...
kernel:[18459.828011] unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3

On another machine, and net-next-2.6, I reproduced the problem, so we need the patch after all, sorry for the delay.

Thanks

[PATCH] ipv4: dont create routes on down devices

In ip_route_output_slow(), instead of allowing a route to be created on a not UPed device, report -ENETUNREACH immediately.

# ip tunnel add mode ipip remote 10.16.0.164 local 
10.16.0.72 dev eth0
# (Note : tunl1 is down)
# ping -I tunl1 10.1.2.3
PING 10.1.2.3 (10.1.2.3) from 192.168.18.5 tunl1: 56(84) bytes of data.
(nothing)
# ./a.out tunl1
# ip tunnel del tunl1
Message from syslogd@shelby at Dec 22 10:12:08 ...
  kernel: unregister_netdevice: waiting for tunl1 to become free. 
Usage count = 3

After patch:

# ping -I tunl1 10.1.2.3
connect: Network is unreachable
Reported-by Nicolas Dichtel <[email protected]>
Cc Octavian Purdila <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/route.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
net/ipv4/route.c +4 -3
index d8b4f4d..f1defb7 100644
@@ -2562,9 +2562,10 @@ static int ip_route_output_slow(struct net *net, struct rtable **rp,
 			goto out;
 
 		/* RACE: Check return value of inet_select_addr instead. */
-		if (rcu_dereference(dev_out->ip_ptr) == NULL)
-			goto out;	/* Wrong error code */
-
+		if (!(dev_out->flags & IFF_UP) || !__in_dev_get_rcu(dev_out)) {
+			err = -ENETUNREACH;
+			goto out;
+		}
 		if (ipv4_is_local_multicast(oldflp->fl4_dst) ||
 		    ipv4_is_lbcast(oldflp->fl4_dst)) {
 			if (!fl.fl4_src)
Reviewed-by on v1 1/1
#13 lore raw

Re: [PATCH] ipv4: dont create routes on down devices

From: Eric Dumazet <[email protected]>
Date: Wednesday 22 December 2010, 16:39:39

Quoted 17 lines

[PATCH] ipv4: dont create routes on down devices

In ip_route_output_slow(), instead of allowing a route to be created on a not UPed device, report -ENETUNREACH immediately.

# ip tunnel add mode ipip remote 10.16.0.164 local
10.16.0.72 dev eth0
# (Note : tunl1 is down)
# ping -I tunl1 10.1.2.3
PING 10.1.2.3 (10.1.2.3) from 192.168.18.5 tunl1: 56(84) bytes of data.
(nothing)
# ./a.out tunl1
# ip tunnel del tunl1
Message from syslogd@shelby at Dec 22 10:12:08 ...
  kernel: unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3
Quoted 8 lines

After patch:

# ping -I tunl1 10.1.2.3
connect: Network is unreachable
Reported-by Nicolas Dichtel <[email protected]>
Cc Octavian Purdila <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Thanks Eric !

Reviewed-by Octavian Purdila <[email protected]>
maintainer Applied
#14 lore raw

Re: [PATCH] ipv4: dont create routes on down devices

From: Octavian Purdila <[email protected]>
Date: Thu, 23 Dec 2010 10:50:25 +0200

Quoted 25 lines

From: Eric Dumazet <[email protected]>
Date: Wednesday 22 December 2010, 16:39:39

Quoted 22 lines

[PATCH] ipv4: dont create routes on down devices

In ip_route_output_slow(), instead of allowing a route to be created on a not UPed device, report -ENETUNREACH immediately.

# ip tunnel add mode ipip remote 10.16.0.164 local
10.16.0.72 dev eth0
# (Note : tunl1 is down)
# ping -I tunl1 10.1.2.3
PING 10.1.2.3 (10.1.2.3) from 192.168.18.5 tunl1: 56(84) bytes of data.
(nothing)
# ./a.out tunl1
# ip tunnel del tunl1
Message from syslogd@shelby at Dec 22 10:12:08 ...
  kernel: unregister_netdevice: waiting for tunl1 to become free.
Usage count = 3

After patch:

# ping -I tunl1 10.1.2.3
connect: Network is unreachable
Quoted 7 lines
Quoted 3 lines
Reported-by Nicolas Dichtel <[email protected]>
Cc Octavian Purdila <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Thanks Eric !

Reviewed-by Octavian Purdila <[email protected]>

Applied, thanks everyone.

#15 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Tuesday 28 September 2010, 19:33:49

Quoted 18 lines · Nicolas Dichtel
Quoted 10 lines

Hi,

I face a problem when I try to remove an interface, netdev_wait_allrefs() complains about refcount.

Here is a trivial scenario to reproduce the problem:

# ip tunnel add mode ipip remote 10.16.0.164 local 10.16.0.72 dev eth0
# ./a.out tunl1
# ip tunnel del tunl1
Quoted 6 lines

Note: a.out binary create an IPv4 raw socket, attach it to tunl1 (SO_BINDTODEVICE), set it as multicast (IP_MULTICAST_LOOP), set the multicast interface to tunl1 (IP_MULTICAST_IF), build the IP header (IP_HDRINCL) and then send a single packet (192.168.6.1 -> 224.0.0.18).

Note2: when a.out is executed, tunl1 has no ip address and is down.

Quoted 20 lines

CC Octavian Purdila, the patch author.

I am just wondering why this route is created in the first place.

Maybe a fix would be to forbid this ?

Some machines have a giant route cache, so its very important to avoid expensive scans.

Quoted 11 lines

Then, I got a serie of "kernel:[1206699.728010] unregister_netdevice:
waiting for tunl1 to become free. Usage count = 3" and after some time, interface is removed.

The problem is that route cache entries are only invalidate on UNREGISTER event, and not removed (introduced by commit e2ce146848c81af2f6d42e67990191c284bf0c33). We must wait that rt_check_expire() remove the remaining route cache entries.

To fix the problem, I propose to remove a part of the previous commit.

Hi Nicolas,

The purpose of my original patch was to speed up interfaces deregistration even more after Eric's batch work. Reverting it might slow things down again, but since this is breaking things we should probably revert it and think a proper optimization afterward. I know that Eric B has done some more work in this area, for batch namespace cleanup, maybe the issue is not even there anymore.

So, Ack from me.

We might even fully revert the patch, since the bit that is left doesn't have any value anymore.

Thanks,
tavi