Longstanding networking / SMP issue? (duplextest)
Subsystems (2)
Quoted 5 lines · Simon Kirby
Baffled by this, I did a bunch of testing and tried to narrow down which servers were experiencing this problem. A sweep of some of our server blocks showed that only SMP boxes were having the problem; however, not all SMP boxes were affected.
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())
There is also a similar problem with ICMPs getting passed to an TCP socket. When the socket is already locked it is dropped. You can check for the later by looking at the LockDroppedIcmps counter in netstat -s
I guess it would be useful to cover the first drop case in that counter too, try this untested patch and then check the counter in netstat -s.
net/ipv4/icmp.c renamed from net/ipv4/icmp.c-o +3 -1
@@ -196,8 +196,10 @@static int icmp_xmit_lock_bh (void ) {if (!spin_trylock (&icmp_socket->sk->lock.slock)) { -if (icmp_xmit_holder ==smp_processor_id ()) +if (icmp_xmit_holder ==smp_processor_id ()) { +NET_INC_STATS_BH (LockDroppedIcmps) return -EAGAIN; + }spin_lock (&icmp_socket->sk->lock.slock); } icmp_xmit_holder =smp_processor_id ();
-Andi
From: Andi Kleen <[email protected]>
Date: 20 Feb 2003 08:52:46 +0100
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())
Good spotting Andi.
Quoted 4 lines · Andi Kleen
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())
Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
This is most likely the cause, but I will test tomorrow to confirm.
Thanks,
Simon-
[ Simon Kirby ][ Network Operations ] [ [email protected] ][ NetNation Communications ]
[ Opinions expressed are not necessarily those of my employer. ]
Quoted 9 lines · Simon Kirby
Quoted 4 lines · Andi Kleen
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?
-Andi
Quoted 6 lines · Simon Kirby
Quoted 4 lines · Andi Kleen
Quoted 3 lines
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP
triggered
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?
Yes IP is best-effort. But this argument cant explain why IP on linux works better if we disable SMP on linux...
I'm sorry Andy but "IP is best effort" sounds very lazy in this case.
SMP should give more power to the machine, not drop some frames because 'another CPU' is busy and has a lock we *need*.
Eric
Quoted 2 lines
Yes IP is best-effort. But this argument cant explain why IP on linux works better if we disable SMP on linux...
It has nothing to do with SMP. The lazy locking dropping packets can happen on UP kernels too in extreme cases. Also with preempt.
-Andi
Quoted 3 lines
Quoted 3 lines
Yes IP is best-effort. But this argument cant explain why IP on linux
works
Quoted 3 lines
Quoted 1 line
better if we disable SMP on linux...
It has nothing to do with SMP. The lazy locking dropping packets can
happen
Quoted 2 lines
on UP kernels too in extreme cases. Also with preempt.
Well, I too noticed that binding NIC IRQS one one CPU
echo 1 > /proc/irq/18/smp_affinity
helps a lot in normal cases.
Some of us want SMP machine because application needs a lot of CPU, but we also need to not drop frames to save the limited network bandwith.
Thanks
From: Andi Kleen <[email protected]>
Date: Thu, 20 Feb 2003 10:34:22 +0100
Quoted 2 lines · Simon Kirby
Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?
True.
But this is a quality of implementation issue and I doubt the kernel would be slower if we fixed this silly behavior.
Frankly, the locking is due to lazyness, rather than a specific design decision. So let's fix it.
Quoted 10 lines · David S. Miller
From: Andi Kleen <[email protected]>
Date: Thu, 20 Feb 2003 10:34:22 +0100Quoted 2 lines · Simon Kirby
Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?
Quoted 7 lines
True.
But this is a quality of implementation issue and I doubt the kernel would be slower if we fixed this silly behavior.
Frankly, the locking is due to lazyness, rather than a specific design decision. So let's fix it.
For icmp_xmit_lock it can be only done in a limited fashion - you are always restricted by the buffer size of the ICMP socket. Also I don't know how to lock the socket from BH context nicely - the only simple way probably is the trick from the retransmit timer to just try again in a jiffie, but could have nasty queueing up under high load.
Fixing the error drop behaviour of TCP will be somewhat nasty too.
In both cases you'll need a retry timer (unreliable) or an dedicated ICMP backlog (complicated)
-Andi
From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 08:27:19 +0100
For icmp_xmit_lock it can be only done in a limited fashion - you are always restricted by the buffer size of the ICMP socket. Also I don't know how to lock the socket from BH context nicely - the only simple way probably is the trick from the retransmit timer to just try again in a jiffie, but could have nasty queueing up under high load.
Fixing the error drop behaviour of TCP will be somewhat nasty too.
In both cases you'll need a retry timer (unreliable) or an dedicated ICMP backlog (complicated)
The big problem is that we have one ICMP socket for UP and only one for SMP too. That's just dumb, we should make this be a per-cpu thing.
I suspect this will fix the original bug report.
I don't think the TCP case is much of an issue. TCP retries things etc.
Quoted 3 lines
The big problem is that we have one ICMP socket for UP and only one for SMP too. That's just dumb, we should make this be a per-cpu thing.
Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?
-Andi
From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:22:58 +0100
Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?
What runs in process context? ICMP responses are generated from BH's.
Quoted 7 lines · David S. Miller
From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:22:58 +0100Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?
What runs in process context? ICMP responses are generated from BH's.
hmm, i was thinking about tcp process context, but you're right it can only generate icmps for dead ports and that happens while still in tcp_v4_rcv().
at least ipip.c will send icmps from process context (ipip_tunnel_xmit)
netfilter will probably do too.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 7 lines · Simon Kirby
Quoted 4 lines · Andi Kleen
That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())
Quoted 5 lines
Quoted 2 lines
Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)
IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?
Software is not a zero sum exercise. Therefore "fast" and "correct" are
not mutually exclusive.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:45:41 +0100
at least ipip.c will send icmps from process context (ipip_tunnel_xmit)
netfilter will probably do too.
We can easily make them all run in BH (protected) context :-)
This makes the locking much simpler. Add to this the per-cpu socket idea I proposed and we arrive at the patch below (against 2.5.x, easily ported to 2.4.x just remove the cpu_possible() stuff).
Comments?
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +40 -57
@@ -223,57 +223,29 @@static struct icmp_control icmp_pointers[NR_ICMP_TYPES +1 ];/* -* The ICMP socket. This is the most convenient way to flow control +* The ICMP socket(s). This is the most convenient way to flow control * our ICMP output as well as maintain a clean interface throughout * all layers. All Socketless IP sends will soon be gone. +* +* On SMP we have one ICMP socket per-cpu. */ -struct socket *icmp_socket; - -/* ICMPv4 socket is only a bit non-reenterable (unlike ICMPv6, -which is strongly non-reenterable). A bit later it will be made -reenterable and the lock may be removed then. -*/ - -static int icmp_xmit_holder =-1 ; - -static int icmp_xmit_lock_bh (void ) -{ -int rc; -if (!spin_trylock (&icmp_socket->sk->lock.slock)) { - rc = -EAGAIN ; -if (icmp_xmit_holder ==smp_processor_id ()) - goto out; -spin_lock (&icmp_socket->sk->lock.slock); - } - rc =0 ; - icmp_xmit_holder =smp_processor_id (); -out: -return rc; -} +static struct socket *__icmp_socket[NR_CPUS ]; +#define icmp_socket __icmp_socket[smp_processor_id()] -static __inline__int icmp_xmit_lock (void ) +static __inline__void icmp_xmit_lock (void ) { -int ret;local_bh_disable (); - ret =icmp_xmit_lock_bh (); -if (ret) -local_bh_enable (); -return ret; -} -static void icmp_xmit_unlock_bh (void ) -{ - icmp_xmit_holder =-1 ; -spin_unlock (&icmp_socket->sk->lock.slock); +if (!spin_trylock (&icmp_socket->sk->lock.slock)) +BUG (); } -static __inline__void icmp_xmit_unlock (void ) +static void icmp_xmit_unlock (void ) { -icmp_xmit_unlock_bh (); +spin_unlock (&icmp_socket->sk->lock.slock);local_bh_enable (); } -/* * Send an ICMP frame. */
@@ -404,10 +376,11 @@struct rtable *rt = (struct rtable *)skb->dst;u32 daddr; -if (ip_options_echo (&icmp_param->replyopts, skb) || -icmp_xmit_lock_bh ()) +if (ip_options_echo (&icmp_param->replyopts, skb)) goto out; +icmp_xmit_lock (); + icmp_param->data.icmph.checksum =0 ;icmp_out_count (icmp_param->data.icmph.type);
@@ -434,7 +407,7 @@icmp_push_reply (icmp_param, &ipc, rt);ip_rt_put (rt); out_unlock: -icmp_xmit_unlock_bh (); +icmp_xmit_unlock (); out:; }
@@ -519,8 +492,7 @@ } } -if (icmp_xmit_lock ()) - goto out; +icmp_xmit_lock ();/* * Construct source address and options.
@@ -1141,19 +1113,30 @@void __init icmp_init (struct net_proto_family *ops) {struct inet_opt *inet; -int err =sock_create (PF_INET ,SOCK_RAW ,IPPROTO_ICMP , &icmp_socket); +int i; -if (err <0 ) -panic ("Failed to create the ICMP control socket.\n" ); - icmp_socket->sk->allocation =GFP_ATOMIC ; - icmp_socket->sk->sndbuf =SK_WMEM_MAX *2 ; - inet =inet_sk (icmp_socket->sk); - inet->ttl =MAXTTL ; - inet->pmtudisc =IP_PMTUDISC_DONT ; - -/* Unhash it so that IP input processing does not even -* see it, we do not wish this socket to see incoming -* packets. -*/ - icmp_socket->sk->prot->unhash (icmp_socket->sk); +for (i =0 ; i <NR_CPUS ; i++) { +int err; + +if (!cpu_possible (i)) +continue ; + + err =sock_create (PF_INET ,SOCK_RAW ,IPPROTO_ICMP , + &__icmp_socket[i]); + +if (err <0 ) +panic ("Failed to create the ICMP control socket.\n" ); + + __icmp_socket[i]->sk->allocation =GFP_ATOMIC ; + __icmp_socket[i]->sk->sndbuf =SK_WMEM_MAX *2 ; + inet =inet_sk (__icmp_socket[i]->sk); + inet->ttl =MAXTTL ; + inet->pmtudisc =IP_PMTUDISC_DONT ; + +/* Unhash it so that IP input processing does not even +* see it, we do not wish this socket to see incoming +* packets. +*/ + __icmp_socket[i]->sk->prot->unhash (__icmp_socket[i]->sk); + } }
ipv6/icmp.c renamed from ipv6/icmp.c.~1~ +53 -50
159 lines not on this page · raw message
ipv4/icmp.c line 233
Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2
-0+static struct socket *__icmp_socket[NR_CPUS ]; +#define icmp_socket __icmp_socket[smp_processor_id()]
This should be per-cpu data
Quoted 7 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +1
-4-static __inline__int icmp_xmit_lock (void ) +static __inline__void icmp_xmit_lock (void ) { -int ret;local_bh_disable (); - ret =icmp_xmit_lock_bh (); -if (ret)
ipv4/icmp.c line 240
Quoted 10 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2
-7-local_bh_enable (); -return ret; -} -static void icmp_xmit_unlock_bh (void ) -{ - icmp_xmit_holder =-1 ; -spin_unlock (&icmp_socket->sk->lock.slock); +if (!spin_trylock (&icmp_socket->sk->lock.slock)) +BUG ();
unlikely()?
ipv4/icmp.c line 246
Quoted 6 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2
-2-static __inline__void icmp_xmit_unlock (void ) +static void icmp_xmit_unlock (void ) { -icmp_xmit_unlock_bh (); +spin_unlock (&icmp_socket->sk->lock.slock);local_bh_enable ();
spin_unlock_bh
Review
Quoted 1 line
(quoted diff) +1
-0+ icmp_xmit_lock();
Hmm, and I guess the code would be much more readable if you used the spin_lock call directly. The impliclit icmp_socket doesn't really help readability either.
From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000
ipv4/icmp.c line 233
Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2
-0+static struct socket *__icmp_socket[NR_CPUS ]; +#define icmp_socket __icmp_socket[smp_processor_id()]
This should be per-cpu data
My bad. Thanks for spotting this.
From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000
Review
Quoted 1 line
(quoted diff) +1
-0+ icmp_xmit_lock();
Hmm, and I guess the code would be much more readable if you used the spin_lock call directly. The impliclit icmp_socket doesn't really help readability either.
Sure it does, it encapsulates the icmp_socket->sk->foo->far->fum dereferencing into one place instead of 4.
Quoted 10 lines · David S. Miller
From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000ipv4/icmp.c line 233
Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2
-0+static struct socket *__icmp_socket[NR_CPUS ]; +#define icmp_socket __icmp_socket[smp_processor_id()]This should be per-cpu data
My bad. Thanks for spotting this.
Won't work if the IPv4 code is ever made modular.
-Andi
Quoted 3 lines · Andi Kleen
Quoted 1 line
My bad. Thanks for spotting this.
Won't work if the IPv4 code is ever made modular.
I think Rusty is working on making it working for modules. Anyway I think that would be one of the minor problem when making ipv4 modular :)
From: Andi Kleen <[email protected]>
Date: Sun, 23 Feb 2003 11:30:39 +0100
Quoted 1 line · David S. Miller
My bad. Thanks for spotting this.
Won't work if the IPv4 code is ever made modular.
Right, which is why I can't use per_cpu on the ipv6 side.