Skip to contentSkip to conversation

Longstanding networking / SMP issue? (duplextest)

DISCUSSION Andi Kleen 6 participants 21 messages Feb 20, 2003 – Feb 23, 2003 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

Andi Kleen

author
#1 lore raw
In reply to a message not in the archive
Quoted 5 lines · Simon Kirby

Baffled by this, I did a bunch of testing and tried to narrow down which servers were experiencing this problem. A sweep of some of our server blocks showed that only SMP boxes were having the problem; however, not all SMP boxes were affected.

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

There is also a similar problem with ICMPs getting passed to an TCP socket. When the socket is already locked it is dropped. You can check for the later by looking at the LockDroppedIcmps counter in netstat -s

I guess it would be useful to cover the first drop case in that counter too, try this untested patch and then check the counter in netstat -s.

net/ipv4/icmp.c renamed from net/ipv4/icmp.c-o +3 -1
@@ -196,8 +196,10 @@
 static int icmp_xmit_lock_bh(void)
 {
 	if (!spin_trylock(&icmp_socket->sk->lock.slock)) {
-		if (icmp_xmit_holder == smp_processor_id())
+		if (icmp_xmit_holder == smp_processor_id()) {
+			NET_INC_STATS_BH(LockDroppedIcmps)
 			return -EAGAIN;
+		}
 		spin_lock(&icmp_socket->sk->lock.slock);
 	}
 	icmp_xmit_holder = smp_processor_id();

-Andi

#2 lore raw

From: Andi Kleen <[email protected]>
Date: 20 Feb 2003 08:52:46 +0100

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

Good spotting Andi.

#3 lore raw
In reply to Andi Kleen
Quoted 4 lines · Andi Kleen

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

This is most likely the cause, but I will test tomorrow to confirm.

Thanks,

Simon-

[        Simon Kirby        ][        Network Operations        ]
[     [email protected]     ][     NetNation Communications     ]

[ Opinions expressed are not necessarily those of my employer. ]

author
#4 lore raw
Quoted 9 lines · Simon Kirby
Quoted 4 lines · Andi Kleen

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?

-Andi

#5 lore raw
Quoted 6 lines · Simon Kirby
Quoted 4 lines · Andi Kleen
Quoted 3 lines

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP

triggered

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?

Yes IP is best-effort. But this argument cant explain why IP on linux works better if we disable SMP on linux...

I'm sorry Andy but "IP is best effort" sounds very lazy in this case.

SMP should give more power to the machine, not drop some frames because 'another CPU' is busy and has a lock we *need*.

Eric

author
#6 lore raw
Quoted 2 lines

Yes IP is best-effort. But this argument cant explain why IP on linux works better if we disable SMP on linux...

It has nothing to do with SMP. The lazy locking dropping packets can happen on UP kernels too in extreme cases. Also with preempt.

-Andi

#7 lore raw
Quoted 3 lines
Quoted 3 lines

Yes IP is best-effort. But this argument cant explain why IP on linux

works

Quoted 3 lines
Quoted 1 line

better if we disable SMP on linux...

It has nothing to do with SMP. The lazy locking dropping packets can

happen

Quoted 2 lines

on UP kernels too in extreme cases. Also with preempt.

Well, I too noticed that binding NIC IRQS one one CPU

echo 1 > /proc/irq/18/smp_affinity

helps a lot in normal cases.

Some of us want SMP machine because application needs a lot of CPU, but we also need to not drop frames to save the limited network bandwith.

Thanks

#8 lore raw
In reply to Andi Kleen

From: Andi Kleen <[email protected]>
Date: Thu, 20 Feb 2003 10:34:22 +0100

Quoted 2 lines · Simon Kirby

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?

True.

But this is a quality of implementation issue and I doubt the kernel would be slower if we fixed this silly behavior.

Frankly, the locking is due to lazyness, rather than a specific design decision. So let's fix it.

author
#9 lore raw
Quoted 10 lines · David S. Miller

From: Andi Kleen <[email protected]>
Date: Thu, 20 Feb 2003 10:34:22 +0100

Quoted 2 lines · Simon Kirby

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?

Quoted 7 lines

True.

But this is a quality of implementation issue and I doubt the kernel would be slower if we fixed this silly behavior.

Frankly, the locking is due to lazyness, rather than a specific design decision. So let's fix it.

For icmp_xmit_lock it can be only done in a limited fashion - you are always restricted by the buffer size of the ICMP socket. Also I don't know how to lock the socket from BH context nicely - the only simple way probably is the trick from the retransmit timer to just try again in a jiffie, but could have nasty queueing up under high load.

Fixing the error drop behaviour of TCP will be somewhat nasty too.

In both cases you'll need a retry timer (unreliable) or an dedicated ICMP backlog (complicated)

-Andi

#10 lore raw

From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 08:27:19 +0100

For icmp_xmit_lock it can be only done in a limited fashion - you are always restricted by the buffer size of the ICMP socket. Also I don't know how to lock the socket from BH context nicely - the only simple way probably is the trick from the retransmit timer to just try again in a jiffie, but could have nasty queueing up under high load.

Fixing the error drop behaviour of TCP will be somewhat nasty too.

In both cases you'll need a retry timer (unreliable) or an dedicated ICMP backlog (complicated)

The big problem is that we have one ICMP socket for UP and only one for SMP too. That's just dumb, we should make this be a per-cpu thing.

I suspect this will fix the original bug report.

I don't think the TCP case is much of an issue. TCP retries things etc.

author
#11 lore raw
Quoted 3 lines

The big problem is that we have one ICMP socket for UP and only one for SMP too. That's just dumb, we should make this be a per-cpu thing.

Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?

-Andi

#12 lore raw

From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:22:58 +0100

Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?

What runs in process context? ICMP responses are generated from BH's.

author
#13 lore raw
Quoted 7 lines · David S. Miller

From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:22:58 +0100

Doesn't work on preemptive, does it? How do you keep it on a single CPU when it runs in process context ?

What runs in process context? ICMP responses are generated from BH's.

hmm, i was thinking about tcp process context, but you're right it can only generate icmps for dead ports and that happens while still in tcp_v4_rcv().

at least ipip.c will send icmps from process context (ipip_tunnel_xmit)

netfilter will probably do too.

-Andi

#21 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 7 lines · Simon Kirby
Quoted 4 lines · Andi Kleen

That's probably because of the lazy ICMP socket locking used for the ICMP socket. When an ICMP is already in process the next ICMP triggered from a softirq (e.g. ECHO-REQUEST) is dropped
(see net/ipv4/icmp_xmit_lock_bh())

Quoted 5 lines
Quoted 2 lines

Hmm...and this is considered desired behavior? It seems like an odd way of handling packets intended to test latency and reliability. :)

IP is best-effort. Dropping packets in odd cases to make locking simpler is not unreasonable. Would you prefer an slower kernel?

Software is not a zero sum exercise. Therefore "fast" and "correct" are
not mutually exclusive.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#14 lore raw
In reply to Andi Kleen

From: Andi Kleen <[email protected]>
Date: Fri, 21 Feb 2003 11:45:41 +0100

at least ipip.c will send icmps from process context (ipip_tunnel_xmit)

netfilter will probably do too.

We can easily make them all run in BH (protected) context :-)

This makes the locking much simpler. Add to this the per-cpu socket idea I proposed and we arrive at the patch below (against 2.5.x, easily ported to 2.4.x just remove the cpu_possible() stuff).

Comments?

ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +40 -57
@@ -223,57 +223,29 @@
 static struct icmp_control icmp_pointers[NR_ICMP_TYPES+1];
 
 /*
- *	The ICMP socket. This is the most convenient way to flow control
+ *	The ICMP socket(s). This is the most convenient way to flow control
  *	our ICMP output as well as maintain a clean interface throughout
  *	all layers. All Socketless IP sends will soon be gone.
+ *
+ *	On SMP we have one ICMP socket per-cpu.
  */
-struct socket *icmp_socket;
-
-/* ICMPv4 socket is only a bit non-reenterable (unlike ICMPv6,
-   which is strongly non-reenterable). A bit later it will be made
-   reenterable and the lock may be removed then.
- */
-
-static int icmp_xmit_holder = -1;
-
-static int icmp_xmit_lock_bh(void)
-{
-	int rc;
-	if (!spin_trylock(&icmp_socket->sk->lock.slock)) {
-		rc = -EAGAIN;
-		if (icmp_xmit_holder == smp_processor_id())
-			goto out;
-		spin_lock(&icmp_socket->sk->lock.slock);
-	}
-	rc = 0;
-	icmp_xmit_holder = smp_processor_id();
-out:
-	return rc;
-}
+static struct socket *__icmp_socket[NR_CPUS];
+#define icmp_socket	__icmp_socket[smp_processor_id()]
 
-static __inline__ int icmp_xmit_lock(void)
+static __inline__ void icmp_xmit_lock(void)
 {
-	int ret;
 	local_bh_disable();
-	ret = icmp_xmit_lock_bh();
-	if (ret)
-		local_bh_enable();
-	return ret;
-}
 
-static void icmp_xmit_unlock_bh(void)
-{
-	icmp_xmit_holder = -1;
-	spin_unlock(&icmp_socket->sk->lock.slock);
+	if (!spin_trylock(&icmp_socket->sk->lock.slock))
+		BUG();
 }
 
-static __inline__ void icmp_xmit_unlock(void)
+static void icmp_xmit_unlock(void)
 {
-	icmp_xmit_unlock_bh();
+	spin_unlock(&icmp_socket->sk->lock.slock);
 	local_bh_enable();
 }
 
-
 /*
  *	Send an ICMP frame.
  */
@@ -404,10 +376,11 @@
 	struct rtable *rt = (struct rtable *)skb->dst;
 	u32 daddr;
 
-	if (ip_options_echo(&icmp_param->replyopts, skb) ||
-	    icmp_xmit_lock_bh())
+	if (ip_options_echo(&icmp_param->replyopts, skb))
 		goto out;
 
+	icmp_xmit_lock();
+
 	icmp_param->data.icmph.checksum = 0;
 	icmp_out_count(icmp_param->data.icmph.type);
 
@@ -434,7 +407,7 @@
 		icmp_push_reply(icmp_param, &ipc, rt);
 	ip_rt_put(rt);
 out_unlock:
-	icmp_xmit_unlock_bh();
+	icmp_xmit_unlock();
 out:;
 }
 
@@ -519,8 +492,7 @@
 		}
 	}
 
-	if (icmp_xmit_lock())
-		goto out;
+	icmp_xmit_lock();
 
 	/*
 	 *	Construct source address and options.
@@ -1141,19 +1113,30 @@
 void __init icmp_init(struct net_proto_family *ops)
 {
 	struct inet_opt *inet;
-	int err = sock_create(PF_INET, SOCK_RAW, IPPROTO_ICMP, &icmp_socket);
+	int i;
 
-	if (err < 0)
-		panic("Failed to create the ICMP control socket.\n");
-	icmp_socket->sk->allocation = GFP_ATOMIC;
-	icmp_socket->sk->sndbuf	    = SK_WMEM_MAX * 2;
-	inet	       = inet_sk(icmp_socket->sk);
-	inet->ttl      = MAXTTL;
-	inet->pmtudisc = IP_PMTUDISC_DONT;
-
-	/* Unhash it so that IP input processing does not even
-	 * see it, we do not wish this socket to see incoming
-	 * packets.
-	 */
-	icmp_socket->sk->prot->unhash(icmp_socket->sk);
+	for (i = 0; i < NR_CPUS; i++) {
+		int err;
+
+		if (!cpu_possible(i))
+			continue;
+
+		err = sock_create(PF_INET, SOCK_RAW, IPPROTO_ICMP,
+				  &__icmp_socket[i]);
+
+		if (err < 0)
+			panic("Failed to create the ICMP control socket.\n");
+
+		__icmp_socket[i]->sk->allocation = GFP_ATOMIC;
+		__icmp_socket[i]->sk->sndbuf = SK_WMEM_MAX * 2;
+		inet = inet_sk(__icmp_socket[i]->sk);
+		inet->ttl = MAXTTL;
+		inet->pmtudisc = IP_PMTUDISC_DONT;
+
+		/* Unhash it so that IP input processing does not even
+		 * see it, we do not wish this socket to see incoming
+		 * packets.
+		 */
+		__icmp_socket[i]->sk->prot->unhash(__icmp_socket[i]->sk);
+	}
 }
ipv6/icmp.c renamed from ipv6/icmp.c.~1~ +53 -50

159 lines not on this page · raw message

#15 lore raw

ipv4/icmp.c line 233

Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2 -0
+static struct socket *__icmp_socket[NR_CPUS];
+#define icmp_socket	__icmp_socket[smp_processor_id()]

This should be per-cpu data

Quoted 7 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +1 -4
-static __inline__ int icmp_xmit_lock(void)
+static __inline__ void icmp_xmit_lock(void)
 {
-	int ret;
 	local_bh_disable();
-	ret = icmp_xmit_lock_bh();
-	if (ret)

ipv4/icmp.c line 240

Quoted 10 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2 -7
-		local_bh_enable();
-	return ret;
-}
 
-static void icmp_xmit_unlock_bh(void)
-{
-	icmp_xmit_holder = -1;
-	spin_unlock(&icmp_socket->sk->lock.slock);
+	if (!spin_trylock(&icmp_socket->sk->lock.slock))
+		BUG();

unlikely()?

ipv4/icmp.c line 246

Quoted 6 lines
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2 -2
-static __inline__ void icmp_xmit_unlock(void)
+static void icmp_xmit_unlock(void)
 {
-	icmp_xmit_unlock_bh();
+	spin_unlock(&icmp_socket->sk->lock.slock);
 	local_bh_enable();

spin_unlock_bh

Review

Quoted 1 line
(quoted diff) +1 -0
+	icmp_xmit_lock();

Hmm, and I guess the code would be much more readable if you used the spin_lock call directly. The impliclit icmp_socket doesn't really help readability either.

#16 lore raw

From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000

ipv4/icmp.c line 233

Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2 -0
+static struct socket *__icmp_socket[NR_CPUS];
+#define icmp_socket	__icmp_socket[smp_processor_id()]

This should be per-cpu data

My bad. Thanks for spotting this.

#20 lore raw
In reply to Christoph Hellwig

From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000

Review

Quoted 1 line
(quoted diff) +1 -0
+	icmp_xmit_lock();

Hmm, and I guess the code would be much more readable if you used the spin_lock call directly. The impliclit icmp_socket doesn't really help readability either.

Sure it does, it encapsulates the icmp_socket->sk->foo->far->fum dereferencing into one place instead of 4.

author
#17 lore raw
In reply to David S. Miller
Quoted 10 lines · David S. Miller

From: Christoph Hellwig <[email protected]>
Date: Sun, 23 Feb 2003 10:02:34 +0000

ipv4/icmp.c line 233

Quoted 2 lines · David S. Miller
ipv4/icmp.c renamed from ipv4/icmp.c.~1~ +2 -0
+static struct socket *__icmp_socket[NR_CPUS];
+#define icmp_socket	__icmp_socket[smp_processor_id()]

This should be per-cpu data

My bad. Thanks for spotting this.

Won't work if the IPv4 code is ever made modular.

-Andi

#18 lore raw
Quoted 3 lines · Andi Kleen
Quoted 1 line

My bad. Thanks for spotting this.

Won't work if the IPv4 code is ever made modular.

I think Rusty is working on making it working for modules. Anyway I think that would be one of the minor problem when making ipv4 modular :)