Fix Network namespace shutdown take 2
Subsystems (4)
cover Fix Network namespace shutdown take 2
6 months ago when I introduced net_alive I fixed the symptoms but I failed to properly fix network namespace shutdown.
I realized this when I received a bug report on Tuesday about a failure in icmp_send caused by packets in the arp_gueue.
It turns out that the net_alive check in netif_receive_skb is completely unnecessary and just masked the real problem.
If we remove all network devices from a network namespace before we shutdown network subsystems and protocols then as designed we cannot have packets in flight causing problems.
It turns out that the root cause of these problems is that the icmp code was calling register_pernet_device instead of register_pernet_subsys and so it's cleanup was happening much too early.
The following patchset which should work against both 2.6.29-rcX and net-next fixes the registration problems and removes the unncessary net_alive check, making the code simpler and hopefully more comprehensible.
Eric
Re: [PATCH 1/3] netns: Fix icmp shutdown.
Recently I had a kernel panic in icmp_send during a network namespace cleanup. There were packets in the arp queue that failed to be sent and we attempted to generate an ICMP host unreachable message, but failed because icmp_sk_exit had already been called.
The network devices are removed from a network namespace and their arp queues are flushed before we do attempt to shutdown subsystems so this error should have been impossible.
It turns out icmp_init is using register_pernet_device instead of register_pernet_subsys. Which resulted in icmp being shut down while we still had the possibility of packets in flight, making a nasty NULL pointer deference in interrupt context possible.
Changing this to register_pernet_subsys fixes the problem in my testing.
--- net/ipv4/icmp.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/ipv4/icmp.c +1 -1
@@ -1207,7 +1207,7 @@ static struct pernet_operations __net_initdata icmp_sk_ops = {int __init icmp_init (void ) { -return register_pernet_device (&icmp_sk_ops); +return register_pernet_subsys (&icmp_sk_ops); }EXPORT_SYMBOL (icmp_err_convert);
Signature · 1 line
1.6.1.2.350.g88cc
PATCH 2/3 tcp: Like icmp use register_pernet_subsys
To remove the possibility of packets flying around when network devices are being cleaned up use reisger_pernet_subsys instead of register_pernet_device.
--- net/ipv4/tcp_ipv4.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/ipv4/tcp_ipv4.c +1 -1
@@ -2443,7 +2443,7 @@ static struct pernet_operations __net_initdata tcp_sk_ops = {void __init tcp_v4_init (void ) {inet_hashinfo_init (&tcp_hashinfo); -if (register_pernet_device (&tcp_sk_ops)) +if (register_pernet_subsys (&tcp_sk_ops))panic ("Failed to create the TCP control socket.\n" ); }
Signature · 1 line
1.6.1.2.350.g88cc
Denis V. Lunev
Acked-byRe: [PATCH 1/3] netns: Fix icmp shutdown.
In reply to a message not in the archiveQuoted 35 lines · Eric W. Biederman
Recently I had a kernel panic in
icmp_sendduring a network namespace cleanup. There were packets in the arp queue that failed to be sent and we attempted to generate an ICMP host unreachable message, but failed becauseicmp_sk_exithad already been called.The network devices are removed from a network namespace and their arp queues are flushed before we do attempt to shutdown subsystems so this error should have been impossible.
It turns out
icmp_initis usingregister_pernet_deviceinstead ofregister_pernet_subsys. Which resulted in icmp being shut down while we still had the possibility of packets in flight, making a nasty NULL pointer deference in interrupt context possible.Changing this to
register_pernet_subsysfixes the problem in my testing.Signed-off-by Eric W. Biederman <[email protected]>--- net/ipv4/icmp.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/ipv4/icmp.c +1
-1@@ -1207,7 +1207,7 @@ static struct pernet_operations __net_initdata icmp_sk_ops = {int __init icmp_init (void ) { -return register_pernet_device (&icmp_sk_ops); +return register_pernet_subsys (&icmp_sk_ops); }EXPORT_SYMBOL (icmp_err_convert);
Denis V. Lunev
Acked-byRe: [PATCH 2/3] tcp: Like icmp use register_pernet_subsys
In reply to a message not in the archiveQuoted 22 lines · Eric W. Biederman
To remove the possibility of packets flying around when network devices are being cleaned up use
reisger_pernet_subsysinstead ofregister_pernet_device.Signed-off-by Eric W. Biederman <[email protected]>--- net/ipv4/tcp_ipv4.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/ipv4/tcp_ipv4.c +1
-1@@ -2443,7 +2443,7 @@ static struct pernet_operations __net_initdata tcp_sk_ops = {void __init tcp_v4_init (void ) {inet_hashinfo_init (&tcp_hashinfo); -if (register_pernet_device (&tcp_sk_ops)) +if (register_pernet_subsys (&tcp_sk_ops))panic ("Failed to create the TCP control socket.\n" ); }
PATCH 3/3 netns: Remove net_alive
In reply to Eric W. BiedermanIt turns out that net_alive is unnecessary, and the original problem that led to it being added was simply that the icmp code thought it was a network device and wound up being unable to handle packets while there were still packets in the network namespace.
Now that icmp and tcp have been fixed to properly register themselves this problem is no longer present and we have a stronger guarantee that packets will not arrive in a network namespace then that provided by net_alive in netif_receive_skb. So remove net_alive allowing packet reception run a little faster.
Additionally document the strong reason why network namespace cleanup is safe so that if something happens again someone else will have a chance of figuring it out.
--- include/net/net_namespace.h | 27 +++++++++++++++++---------- net/core/dev.c | 6 ------ net/core/net_namespace.c | 3 --- 3 files changed, 17 insertions(+), 19 deletions(-)
include/net/net_namespace.h +17 -10
@@ -109,11 +109,6 @@ extern struct list_head net_namespace_list;#ifdef CONFIG_NET_NS extern void __put_net (struct net *net); -static inline int net_alive (struct net *net) -{ -return net &&atomic_read (&net->count); -} -static inline struct net *get_net (struct net *net) {atomic_inc (&net->count);
@@ -145,11 +140,6 @@ int net_eq(const struct net *net1, const struct net *net2) }#else -static inline int net_alive (struct net *net) -{ -return 1 ; -} -static inline struct net *get_net (struct net *net) {return net;
@@ -234,6 +224,23 @@ struct pernet_operations {void (*exit)(struct net *net); }; +/* +* Use these carefully. If you implement a network device and it +* needs per network namespace operations use device pernet operations, +* otherwise use pernet subsys operations. +* +* This is critically important. Most of the network code cleanup +* runs with the assumption that dev_remove_pack has been called so no +* new packets will arrive during and after the cleanup functions have +* been called. dev_remove_pack is not per namespace so instead the +* guarantee of no more packets arriving in a network namespace is +* provided by ensuring that all network devices and all sockets have +* left the network namespace before the cleanup methods are called. +* +* For the longest time the ipv4 icmp code was registered as a pernet +* device which caused kernel oops, and panics during network +* namespace cleanup. So please don't get this wrong. +*/ extern int register_pernet_subsys (struct pernet_operations *);extern void unregister_pernet_subsys (struct pernet_operations *);extern int register_pernet_gen_subsys (int *id,struct pernet_operations *);
net/core/dev.c +0 -6
@@ -2267,12 +2267,6 @@ int netif_receive_skb(struct sk_buff *skb)rcu_read_lock (); -/* Don't receive packets in an exiting network namespace */ -if (!net_alive (dev_net (skb->dev))) { -kfree_skb (skb); - goto out; - } -#ifdef CONFIG_NET_CLS_ACT if (skb->tc_verd &TC_NCLS ) { skb->tc_verd =CLR_TC_NCLS (skb->tc_verd);
net/core/net_namespace.c +0 -3
@@ -140,9 +140,6 @@ static void cleanup_net(struct work_struct *work)struct pernet_operations *ops;struct net *net; -/* Be very certain incoming network packets will not find us */ -rcu_barrier (); - net = container_of(work, struct net, work);mutex_lock (&net_mutex);
Signature · 1 line
1.6.1.2.350.g88cc
Re: [PATCH 1/3] netns: Fix icmp shutdown.
In reply to Denis V. LunevFrom: "Denis V. Lunev" <[email protected]>
Date: Fri, 20 Feb 2009 18:57:02 +0300
Quoted 1 line
Acked-by Denis V. Lunev <[email protected]>
Applied.
Re: [PATCH 2/3] tcp: Like icmp use register_pernet_subsys
In reply to Denis V. LunevFrom: "Denis V. Lunev" <[email protected]>
Date: Fri, 20 Feb 2009 18:57:23 +0300
Quoted 1 line
Acked-by Denis V. Lunev <[email protected]>
Applied.
David Miller
maintainerRe: [PATCH 3/3] netns: Remove net_alive
In reply to Eric W. BiedermanFrom: [email protected] (Eric W. Biederman)
Date: Fri, 20 Feb 2009 08:02:57 -0800
Quoted 12 lines
It turns out that
net_aliveis unnecessary, and the original problem that led to it being added was simply that the icmp code thought it was a network device and wound up being unable to handle packets while there were still packets in the network namespace.Now that icmp and tcp have been fixed to properly register themselves this problem is no longer present and we have a stronger guarantee that packets will not arrive in a network namespace then that provided by
net_aliveinnetif_receive_skb. So removenet_aliveallowing packet reception run a little faster.
Quoted 5 lines
Additionally document the strong reason why network namespace cleanup is safe so that if something happens again someone else will have a chance of figuring it out.
Signed-off-by Eric W. Biederman <[email protected]>
Also applied, thanks Eric.
Quoted 18 lines · Eric W. Biederman
6 months ago when I introduced
net_aliveI fixed the symptoms but I failed to properly fix network namespace shutdown.I realized this when I received a bug report on Tuesday about a failure in
icmp_sendcaused by packets in thearp_gueue.It turns out that the
net_alivecheck innetif_receive_skbis completely unnecessary and just masked the real problem.If we remove all network devices from a network namespace before we shutdown network subsystems and protocols then as designed we cannot have packets in flight causing problems.
It turns out that the root cause of these problems is that the icmp code was calling
register_pernet_deviceinstead ofregister_pernet_subsysand so it's cleanup was happening much too early.
Quoted 5 lines
The following patchset which should work against both 2.6.29-rcX and net-next fixes the registration problems and removes the unncessary
net_alivecheck, making the code simpler and hopefully more comprehensible.
Hi Dave,
I don't see these patches in the net-2.6 tree. Shouldn't they be in net-2.6 too ?
-- Daniel
David Miller
maintainerFrom: Daniel Lezcano <[email protected]>
Date: Wed, 25 Feb 2009 13:43:29 +0100
Quoted 2 lines
I don't see these patches in the net-2.6 tree. Shouldn't they be in net-2.6 too ?
Ok, I'll think about cherry picking them into net-2.6...
Quoted 10 lines · David Miller
From: Daniel Lezcano <[email protected]>
Date: Wed, 25 Feb 2009 13:43:29 +0100Quoted 3 lines
I don't see these patches in the net-2.6 tree. Shouldn't they be in net-2.6 too ?
Ok, I'll think about cherry picking them into net-2.6...
Thanks Dave.