Skip to contentSkip to conversation

net: frag performance followup

PATCH Applied Jesper Dangaard Brouer 5 participants 29 messages Mar 27, 2013 – Apr 4, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover net: frag performance followup

This patchset is a followup to my previously accepted fragmentation patchset:
http://thread.gmane.org/gmane.linux.network/257155

This patchset is not my entire patch queue, as I have left out the patch I mentioned in:
http://thread.gmane.org/gmane.linux.network/261924
"RFC crap-patch [PATCH] net: Per CPU separate frag mem accounting"

Because I'm working on another "replacement" patch which removes the LRU list, which I discussed with Eric Dumazet during Netfilter Workshop. I have some preliminary results of that later in this mail.

I'm uncertain if this is net-next or net material? (for now it's based on net-next on top of commit f5a03cf461)

Patch list:
Patch-01: avoid several CPUs grabbing same frag queue during LRU evictor loop
Patch-02: use the frag lru_lock to protect netns_frags.nqueues update
Patch-03: frag queue per hash bucket locking
(below not-included)
Patch-XX: Try Impl. Eric's idea, no LRU and direct hash cleaning

Notice, I have changed the frag DoS generator script to be more efficient/deadly. Before it would only hit one RX queue, now its sending packets causing multi-queue RX, due to "better" RX hashing.

Same test setup:
Two 10G interfaces, on seperate NUMA nodes, are under-test, and uses Ethernet flow-control. A third interface is used for generating the DoS attack (with trafgen).

Test types summary (netperf UDP_STREAM):
Test-20G64K == 2x10G with 65K fragments
Test-20G3F == 2x10G with 3x fragments (3*1472 bytes)
Test-20G64K+DoS == Same as 20G64K with frag DoS

 Test-20G3F+DoS  == Same as 20G3F  with frag DoS
 Test-20G64K+MQ  == Same as 20G64K with Multi-Queue frag DoS
 Test-20G3F+MQ   == Same as 20G3F  with Multi-Queue frag DoS

Performance table summary (in Mbit/s):

 Test-type:  20G64K    20G3F    20G64K+DoS  20G3F+DoS  20G64K+MQ 20G3F+MQ
 ----------  -------   -------  ----------  ---------  --------  -------
  net-next:  18486.7   10723.2   3657.85     4560.64      99.9    189.1
  Patch-01:  18830.8   13388.4   4054.96     5377.27     127.9    433.4
  Patch-02:  18848.7   13230.1   4103.04     5310.36     130.0    440.2
  Patch-03:  18838.0   13490.5   4405.11     6814.72     196.6    461.6

(below work-in-progress)
Patch-XX: 18800.0 15698.4 10012.90 12039.00 4257.39 3305.8

After his patchset, the LRU list is the major bottleneck. As can also be seen by my preliminary results of removing the LRU list.

---

Jesper Dangaard Brouer (3):
net: frag queue per hash bucket locking
net: use the frag lru_lock to protect netns_frags.nqueues update
net: frag, avoid several CPUs grabbing same frag queue during LRU evictor loop

 include/net/inet_frag.h  |   11 +++++++-
 net/ipv4/inet_fragment.c |   65 +++++++++++++++++++++++++++++++++++-----------
 2 files changed, 60 insertions(+), 16 deletions(-)

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Sr. Network Kernel Developer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

#1 lore raw

PATCH 1/3 net: frag, avoid several CPUs grabbing same frag queue during LRU evictor loop

The LRU list is protected by its own lock, since commit 3ef0eb0db4 (net: frag, move LRU list maintenance outside of rwlock), and no-longer by a read_lock.

This makes it possible, to remove the inet_frag_queue, which is about to be "evicted", from the LRU list head. This avoids the problem, of several CPUs grabbing the same frag queue.

Note, cannot remove the inet_frag_lru_del() call in fq_unlink() called by inet_frag_kill(), because inet_frag_kill() is also used in other situations. Thus, we use list_del_init() to allow this double list_del to work.

---

 net/ipv4/inet_fragment.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
net/ipv4/inet_fragment.c +3 -0
index 2bff045..8ba548a 100644
@@ -204,6 +204,9 @@ int inet_frag_evictor(struct netns_frags *nf, struct inet_frags *f, bool force)
 		q = list_first_entry(&nf->lru_list,
 				struct inet_frag_queue, lru_list);
 		atomic_inc(&q->refcnt);
+		/* Remove q from list to avoid several CPUs grabbing it */
+		list_del_init(&q->lru_list);
+
 		spin_unlock(&nf->lru_lock);
 
 		spin_lock(&q->lock);
Acked-by on v1 1/3
#2 lore raw

Re: [net-next PATCH 1/3] net: frag, avoid several CPUs grabbing same frag queue during LRU evictor loop

Quoted 13 lines · Jesper Dangaard Brouer

The LRU list is protected by its own lock, since commit 3ef0eb0db4 (net: frag, move LRU list maintenance outside of rwlock), and no-longer by a read_lock.

This makes it possible, to remove the inet_frag_queue, which is about to be "evicted", from the LRU list head. This avoids the problem, of several CPUs grabbing the same frag queue.

Note, cannot remove the inet_frag_lru_del() call in fq_unlink() called by inet_frag_kill(), because inet_frag_kill() is also used in other situations. Thus, we use list_del_init() to allow this double list_del to work.

Quoted 6 lines

---

 net/ipv4/inet_fragment.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
maintainer Applied
#3 lore raw

Re: [net-next PATCH 1/3] net: frag, avoid several CPUs grabbing same frag queue during LRU evictor loop

From: Eric Dumazet <[email protected]>
Date: Wed, 27 Mar 2013 09:14:04 -0700

Quoted 14 lines · Jesper Dangaard Brouer
Quoted 13 lines

The LRU list is protected by its own lock, since commit 3ef0eb0db4 (net: frag, move LRU list maintenance outside of rwlock), and no-longer by a read_lock.

This makes it possible, to remove the inet_frag_queue, which is about to be "evicted", from the LRU list head. This avoids the problem, of several CPUs grabbing the same frag queue.

Note, cannot remove the inet_frag_lru_del() call in fq_unlink() called by inet_frag_kill(), because inet_frag_kill() is also used in other situations. Thus, we use list_del_init() to allow this double list_del to work.

Quoted 8 lines
Quoted 6 lines

---

 net/ipv4/inet_fragment.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

Applied.

#4 lore raw

PATCH 2/3 net: use the frag lru_lock to protect netns_frags.nqueues update

In reply to Jesper Dangaard Brouer

Move the protection of netns_frags.nqueues updates under the LRU_lock, instead of the write lock. As they are located on the same cacheline, and this is also needed when transitioning to use per hash bucket locking.

---

 include/net/inet_frag.h  |    2 ++
 net/ipv4/inet_fragment.c |    2 --
 2 files changed, 2 insertions(+), 2 deletions(-)
include/net/inet_frag.h +2 -0
index 64b4e7d..7cac9c5 100644
@@ -143,6 +143,7 @@ static inline void inet_frag_lru_del(struct inet_frag_queue *q)
 {
 	spin_lock(&q->net->lru_lock);
 	list_del(&q->lru_list);
+	q->net->nqueues--;
 	spin_unlock(&q->net->lru_lock);
 }
 
@@ -151,6 +152,7 @@ static inline void inet_frag_lru_add(struct netns_frags *nf,
 {
 	spin_lock(&nf->lru_lock);
 	list_add_tail(&q->lru_list, &nf->lru_list);
+	q->net->nqueues++;
 	spin_unlock(&nf->lru_lock);
 }
 
net/ipv4/inet_fragment.c +0 -2
index 8ba548a..1206ca6 100644
@@ -124,7 +124,6 @@ static inline void fq_unlink(struct inet_frag_queue *fq, struct inet_frags *f)
 {
 	write_lock(&f->lock);
 	hlist_del(&fq->list);
-	fq->net->nqueues--;
 	write_unlock(&f->lock);
 	inet_frag_lru_del(fq);
 }
@@ -260,7 +259,6 @@ static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
 
 	atomic_inc(&qp->refcnt);
 	hlist_add_head(&qp->list, &f->hash[hash]);
-	nf->nqueues++;
 	write_unlock(&f->lock);
 	inet_frag_lru_add(nf, qp);
 	return qp;
Acked-by on v1 2/3
#5 lore raw

Re: [net-next PATCH 2/3] net: use the frag lru_lock to protect netns_frags.nqueues update

Quoted 6 lines · Jesper Dangaard Brouer

Move the protection of netns_frags.nqueues updates under the LRU_lock, instead of the write lock. As they are located on the same cacheline, and this is also needed when transitioning to use per hash bucket locking.

---

maintainer Applied
#6 lore raw

Re: [net-next PATCH 2/3] net: use the frag lru_lock to protect netns_frags.nqueues update

From: Eric Dumazet <[email protected]>
Date: Wed, 27 Mar 2013 09:21:48 -0700

Quoted 9 lines · Jesper Dangaard Brouer
Quoted 6 lines

Move the protection of netns_frags.nqueues updates under the LRU_lock, instead of the write lock. As they are located on the same cacheline, and this is also needed when transitioning to use per hash bucket locking.

---

Applied.

#7 lore raw

PATCH 3/3 net: frag queue per hash bucket locking

In reply to Jesper Dangaard Brouer

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

---

 include/net/inet_frag.h  |    9 ++++++-
 net/ipv4/inet_fragment.c |   60 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 55 insertions(+), 14 deletions(-)
include/net/inet_frag.h +8 -1
index 7cac9c5..c4f5183 100644
@@ -50,10 +50,17 @@ struct inet_frag_queue {
  */
 #define INETFRAGS_MAXDEPTH		128
 
+struct inet_frag_bucket {
+	struct hlist_head	chain;
+	spinlock_t		chain_lock;
+	u16			chain_len;
+};
+
 struct inet_frags {
-	struct hlist_head	hash[INETFRAGS_HASHSZ];
+	struct inet_frag_bucket	hash[INETFRAGS_HASHSZ];
 	/* This rwlock is a global lock (seperate per IPv4, IPv6 and
 	 * netfilter). Important to keep this on a seperate cacheline.
+	 * Its primarily a rebuild protection rwlock.
 	 */
 	rwlock_t		lock ____cacheline_aligned_in_smp;
 	int			secret_interval;
net/ipv4/inet_fragment.c +47 -13

137 lines not on this page · raw message

#8 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 46 lines · Jesper Dangaard Brouer

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

net/ipv4/inet_fragment.c +9 -4
@@ -226,27 +247,32 @@ static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
 		struct inet_frag_queue *qp_in, struct inet_frags *f,
 		void *arg)
 {
+	struct inet_frag_bucket *hb;
 	struct inet_frag_queue *qp;
 #ifdef CONFIG_SMP
 #endif
 	unsigned int hash;
 
-	write_lock(&f->lock);
+	read_lock(&f->lock); /* Protects against hash rebuild */
 	/*
 	 * While we stayed w/o the lock other CPU could update
 	 * the rnd seed, so we need to re-calculate the hash
 	 * chain. Fortunatelly the qp_in can be used to get one.
 	 */
 	hash = f->hashfn(qp_in);
+	hb = &f->hash[hash];
+	spin_lock_bh(&hb->chain_lock);
+
 #ifdef CONFIG_SMP
 	/* With SMP race we have to recheck hash table, because
 	 * such entry could be created on other cpu, while we
-	 * promoted read lock to write lock.
+	 * released the hash bucket lock.
 	 */
-	hlist_for_each_entry(qp, &f->hash[hash], list) {
+	hlist_for_each_entry(qp, &hb->chain, list) {
 		if (qp->net == nf && f->match(qp, arg)) {
 			atomic_inc(&qp->refcnt);
-			write_unlock(&f->lock);
+			spin_unlock_bh(&hb->chain_lock);
+			read_unlock(&f->lock);
 			qp_in->last_in |= INET_FRAG_COMPLETE;
 			inet_frag_put(qp_in, f);
 			return qp;
@@ -258,8 +284,10 @@ static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
 		atomic_inc(&qp->refcnt);
 

net/ipv4/inet_fragment.c static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf, · line 293

Quoted 10 lines
net/ipv4/inet_fragment.c +4 -2
 	atomic_inc(&qp->refcnt);
-	hlist_add_head(&qp->list, &f->hash[hash]);
-	write_unlock(&f->lock);
+	hlist_add_head(&qp->list, &hb->chain);
+	hb->chain_len++;
+	spin_unlock_bh(&hb->chain_lock);
+	read_unlock(&f->lock);
 	inet_frag_lru_add(nf, qp);
 	return qp;
 }

I am not sure why you added _bh suffix to spin_lock()/spin_unlock() here ?

(Please check in other functions as well)

#9 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 47 lines · Eric Dumazet
Quoted 46 lines · Jesper Dangaard Brouer

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

net/ipv4/inet_fragment.c +9 -4
@@ -226,27 +247,32 @@ static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
 		struct inet_frag_queue *qp_in, struct inet_frags *f,
 		void *arg)
 {
+	struct inet_frag_bucket *hb;
 	struct inet_frag_queue *qp;
 #ifdef CONFIG_SMP
 #endif
 	unsigned int hash;
 
-	write_lock(&f->lock);
+	read_lock(&f->lock); /* Protects against hash rebuild */
 	/*
 	 * While we stayed w/o the lock other CPU could update
 	 * the rnd seed, so we need to re-calculate the hash
 	 * chain. Fortunatelly the qp_in can be used to get one.
 	 */
 	hash = f->hashfn(qp_in);
+	hb = &f->hash[hash];
+	spin_lock_bh(&hb->chain_lock);
+
 #ifdef CONFIG_SMP
 	/* With SMP race we have to recheck hash table, because
 	 * such entry could be created on other cpu, while we
-	 * promoted read lock to write lock.
+	 * released the hash bucket lock.
 	 */
-	hlist_for_each_entry(qp, &f->hash[hash], list) {
+	hlist_for_each_entry(qp, &hb->chain, list) {
 		if (qp->net == nf && f->match(qp, arg)) {
 			atomic_inc(&qp->refcnt);
-			write_unlock(&f->lock);
+			spin_unlock_bh(&hb->chain_lock);
+			read_unlock(&f->lock);
 			qp_in->last_in |= INET_FRAG_COMPLETE;
 			inet_frag_put(qp_in, f);
 			return qp;
@@ -258,8 +284,10 @@ static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
 		atomic_inc(&qp->refcnt);
 
Quoted 14 lines

net/ipv4/inet_fragment.c static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf, · line 293

Quoted 10 lines
net/ipv4/inet_fragment.c +4 -2
 	atomic_inc(&qp->refcnt);
-	hlist_add_head(&qp->list, &f->hash[hash]);
-	write_unlock(&f->lock);
+	hlist_add_head(&qp->list, &hb->chain);
+	hb->chain_len++;
+	spin_unlock_bh(&hb->chain_lock);
+	read_unlock(&f->lock);
 	inet_frag_lru_add(nf, qp);
 	return qp;
 }

I am not sure why you added _bh suffix to spin_lock()/spin_unlock() here ?

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

maintainer
#10 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

From: Hannes Frederic Sowa <[email protected]>
Date: Thu, 28 Mar 2013 19:57:21 +0100

Quoted 6 lines · Eric Dumazet
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 4 lines

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

...

Quoted 6 lines
Quoted 2 lines

I am not sure why you added _bh suffix to spin_lock()/spin_unlock() here ?

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

That appears to be the case yes.

That's an odd environment for these routines to be invoked from, so longer term we should probably make the nf conntrack code do the BH disabling around the inet frag calls, rather than make the inet frag code eat the extra overhead for the more common invocations.

#11 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 17 lines · David Miller

From: Hannes Frederic Sowa <[email protected]>
Date: Thu, 28 Mar 2013 19:57:21 +0100

Quoted 6 lines · Eric Dumazet
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 4 lines

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

...

Quoted 6 lines
Quoted 2 lines

I am not sure why you added _bh suffix to spin_lock()/spin_unlock() here ?

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Quoted 6 lines

That appears to be the case yes.

That's an odd environment for these routines to be invoked from, so longer term we should probably make the nf conntrack code do the BH disabling around the inet frag calls, rather than make the inet frag code eat the extra overhead for the more common invocations.

My idea was a bh-safe flag in struct inet_frags and conditionally use spin_lock and spin_unlock_bh (these could be wrapped in inline functions just for inet_fragment.c).

maintainer
#12 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

From: Hannes Frederic Sowa <[email protected]>
Date: Thu, 28 Mar 2013 20:10:50 +0100

Quoted 20 lines · David Miller
Quoted 19 lines

From: Hannes Frederic Sowa <[email protected]>
Date: Thu, 28 Mar 2013 19:57:21 +0100

Quoted 6 lines · Eric Dumazet
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 4 lines

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

...

Quoted 6 lines
Quoted 2 lines

I am not sure why you added _bh suffix to spin_lock()/spin_unlock() here ?

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

That appears to be the case yes.

Quoted 8 lines
Quoted 4 lines

That's an odd environment for these routines to be invoked from, so longer term we should probably make the nf conntrack code do the BH disabling around the inet frag calls, rather than make the inet frag code eat the extra overhead for the more common invocations.

My idea was a bh-safe flag in struct inet_frags and conditionally use spin_lock and spin_unlock_bh (these could be wrapped in inline functions just for inet_fragment.c).

That's an extra check eaten by all users. Really, put the overhead into the call sites that need it, and nowhere else.

#13 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

In reply to Hannes Frederic Sowa
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Then there would be a possible deadlock in current code.

#14 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

#15 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 12 lines · Hannes Frederic Sowa
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

Good, so no need for the _bh() as I suspected.

#16 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 8 lines · Eric Dumazet
Quoted 7 lines · Hannes Frederic Sowa
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

Good, so no need for the _bh() as I suspected.

Ack.

I replaced the _bh spin_locks with plain spinlocks and tested the code with sending fragments and receiving fragments (netfilter and reassmbly logic) with lockdep and didn't get any splats. Looks good so far.

#17 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 15 lines · Hannes Frederic Sowa
Quoted 14 lines · Eric Dumazet
Quoted 12 lines · Hannes Frederic Sowa
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

Quoted 7 lines
Quoted 1 line

Good, so no need for the _bh() as I suspected.

Ack.

I replaced the _bh spin_locks with plain spinlocks and tested the code with sending fragments and receiving fragments (netfilter and reassmbly logic) with lockdep and didn't get any splats. Looks good so far.

Well, it's great to see, that you are working on solving my patch proposal. While I'm on Easter vacation ;-) Much appreciated. I'm officially back from vacation Tuesday, and I'll repost then (after testing it on my 10G testlab).

--Jesper

#18 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

Quoted 4 lines · Jesper Dangaard Brouer

Well, it's great to see, that you are working on solving my patch proposal. While I'm on Easter vacation ;-) Much appreciated. I'm officially back from vacation Tuesday, and I'll repost then (after testing it on my 10G testlab).

Have a nice Easter break !

maintainer
#19 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

In reply to Jesper Dangaard Brouer

From: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 29 Mar 2013 20:01:33 +0100

Quoted 20 lines · Hannes Frederic Sowa
Quoted 19 lines · Eric Dumazet
Quoted 8 lines · Hannes Frederic Sowa
Quoted 7 lines · Eric Dumazet
Quoted 6 lines · Hannes Frederic Sowa
Quoted 4 lines

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

Good, so no need for the _bh() as I suspected.

Ack.

Quoted 8 lines
Quoted 3 lines

I replaced the _bh spin_locks with plain spinlocks and tested the code with sending fragments and receiving fragments (netfilter and reassmbly logic) with lockdep and didn't get any splats. Looks good so far.

Well, it's great to see, that you are working on solving my patch proposal. While I'm on Easter vacation ;-) Much appreciated. I'm officially back from vacation Tuesday, and I'll repost then (after testing it on my 10G testlab).

Jesper, when you do this, please just put the whole of the excellent description text you had in the "0/3" posting from this series into the commit message for the respun patch #3.

Thanks.

#20 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

In reply to Jesper Dangaard Brouer
Quoted 2 lines · Jesper Dangaard Brouer

I'm officially back from vacation Tuesday, and I'll repost then (after testing it on my 10G testlab).

Argh- just deleted a long mail with a lot of test results, showing strange performance results ... as I just realized that /boot ran dry during my testing procedure, thus invalidating all the test I spend most of the day performing, argh!

--Jesper

#21 lore raw

Re: [net-next PATCH 3/3] net: frag queue per hash bucket locking

In reply to Jesper Dangaard Brouer
Quoted 20 lines · Jesper Dangaard Brouer
Quoted 19 lines · Hannes Frederic Sowa
Quoted 8 lines · Eric Dumazet
Quoted 7 lines · Hannes Frederic Sowa
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Hannes Frederic Sowa

I assume that it has to do with the usage of this code in ipv6/netfilter/nf_conntrack_reasm.c, which could be invoked from process context, if I read it correctly.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Then there would be a possible deadlock in current code.

Netfilter currently does a local_bh_disable() before entering inet_fragment (and later enables it, again).

Good, so no need for the _bh() as I suspected.

Ack.

Quoted 8 lines
Quoted 3 lines

I replaced the _bh spin_locks with plain spinlocks and tested the code with sending fragments and receiving fragments (netfilter and reassmbly logic) with lockdep and didn't get any splats. Looks good so far.

Well, it's great to see, that you are working on solving my patch proposal. While I'm on Easter vacation ;-) Much appreciated. I'm officially back from vacation Tuesday, and I'll repost then (after testing it on my 10G testlab).

When I rebased patch-03 (on top of net-next commit a210576c) and removed the _bh spinlock, I saw a performance regression. BUT this was caused by some unrelated change in-between. See tests below.

Test (A) is what I reported before for patch-02, accepted in commit 1b5ab0de. Test (B) verifying-retest of commit 1b5ab0de correspond to patch-02. Test (C) is what I reported before for patch-03

Test (D) is net-next master HEAD (commit a210576c), which reveals some (unknown) performance regression (compared against test (B)). And (D) function as a new base-test.

(#) Test-type:  20G64K    20G3F    20G64K+DoS  20G3F+DoS  20G64K+MQ 20G3F+MQ
    ----------  -------   -------  ----------  ---------  --------  -------
(A) Patch-02  : 18848.7   13230.1   4103.04     5310.36     130.0    440.2
(B) 1b5ab0de  : 18841.5   13156.8   4101.08     5314.57     129.0    424.2
(C) Patch-03v1: 18838.0   13490.5   4405.11     6814.72     196.6    461.6
(D) a210576c  : 18321.5   11250.4   3635.34     5160.13     119.1    405.2
(E) with _bh  : 17247.3   11492.6   3994.74     6405.29     166.7    413.6
(F) without bh: 17471.3   11298.7   3818.05     6102.11     165.7    406.3

Test (E) and (F) is patch-03, with and without the _bh spinlocks.

I cannot explain the slow down for 20G64K (but its an artificial "labtest" so I'm not worried). But the other results does show improvements. And test (E) "with _bh" version is slightly better.

p.s. Damm, it too a bit longer, than expected, to test this fairly small correction to the patch...

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#22 lore raw

PATCH v2 1/1 net: frag queue per hash bucket locking

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

V2:
- By analysis from Hannes Frederic Sowa and Eric Dumazet, we don't need the spinlock _bh versions, as Netfilter currently does a local_bh_disable() before entering inet_fragment.
- Fold-in desc from cover-mail

This patch is part of "net: frag performance followup"
http://thread.gmane.org/gmane.linux.network/263644
of which two patches have already been accepted:

Same test setup as previous:
(http://thread.gmane.org/gmane.linux.network/257155)
Two 10G interfaces, on seperate NUMA nodes, are under-test, and uses Ethernet flow-control. A third interface is used for generating the DoS attack (with trafgen).

Notice, I have changed the frag DoS generator script to be more efficient/deadly. Before it would only hit one RX queue, now its sending packets causing multi-queue RX, due to "better" RX hashing.

Test types summary (netperf UDP_STREAM):
Test-20G64K == 2x10G with 65K fragments
Test-20G3F == 2x10G with 3x fragments (3*1472 bytes)
Test-20G64K+DoS == Same as 20G64K with frag DoS

 Test-20G3F+DoS  == Same as 20G3F  with frag DoS
 Test-20G64K+MQ  == Same as 20G64K with Multi-Queue frag DoS
 Test-20G3F+MQ   == Same as 20G3F  with Multi-Queue frag DoS

When I rebased this-patch(03) (on top of net-next commit a210576c) and removed the _bh spinlock, I saw a performance regression. BUT this was caused by some unrelated change in-between. See tests below.

Test (A) is what I reported before for patch-02, accepted in commit 1b5ab0de. Test (B) verifying-retest of commit 1b5ab0de corrospond to patch-02. Test (C) is what I reported before for this-patch

Test (D) is net-next master HEAD (commit a210576c), which reveals some (unknown) performance regression (compared against test (B)). Test (D) function as a new base-test.

Performance table summary (in Mbit/s):

(#) Test-type:  20G64K    20G3F    20G64K+DoS  20G3F+DoS  20G64K+MQ 20G3F+MQ
    ----------  -------   -------  ----------  ---------  --------  -------
(A) Patch-02  : 18848.7   13230.1   4103.04     5310.36     130.0    440.2
(B) 1b5ab0de  : 18841.5   13156.8   4101.08     5314.57     129.0    424.2
(C) Patch-03v1: 18838.0   13490.5   4405.11     6814.72     196.6    461.6
(D) a210576c  : 18321.5   11250.4   3635.34     5160.13     119.1    405.2
(E) with _bh  : 17247.3   11492.6   3994.74     6405.29     166.7    413.6
(F) without bh: 17471.3   11298.7   3818.05     6102.11     165.7    406.3

Test (E) and (F) is this-patch(03), with(V1) and without(V2) the _bh spinlocks.

I cannot explain the slow down for 20G64K (but its an artificial "lab-test" so I'm not worried). But the other results does show improvements. And test (E) "with _bh" version is slightly better.

---

 include/net/inet_frag.h  |    9 ++++++-
 net/ipv4/inet_fragment.c |   60 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 55 insertions(+), 14 deletions(-)
include/net/inet_frag.h +8 -1
index 7cac9c5..c4f5183 100644
@@ -50,10 +50,17 @@ struct inet_frag_queue {
  */
 #define INETFRAGS_MAXDEPTH		128
 
+struct inet_frag_bucket {
+	struct hlist_head	chain;
+	spinlock_t		chain_lock;
+	u16			chain_len;
+};
+
 struct inet_frags {
-	struct hlist_head	hash[INETFRAGS_HASHSZ];
+	struct inet_frag_bucket	hash[INETFRAGS_HASHSZ];
 	/* This rwlock is a global lock (seperate per IPv4, IPv6 and
 	 * netfilter). Important to keep this on a seperate cacheline.
+	 * Its primarily a rebuild protection rwlock.
 	 */
 	rwlock_t		lock ____cacheline_aligned_in_smp;
 	int			secret_interval;
net/ipv4/inet_fragment.c +47 -13

137 lines not on this page · raw message

#23 lore raw

Re: [net-next PATCH V2] net: frag queue per hash bucket locking

include/net/inet_frag.h struct inet_frag_queue { · line 58

Quoted 6 lines · Jesper Dangaard Brouer
include/net/inet_frag.h +6 -0
+struct inet_frag_bucket {
+	struct hlist_head	chain;
+	spinlock_t		chain_lock;
+	u16			chain_len;
+};
+

I just noticed and wanted to ask for what chain_len is needed? Could it be dropped?

If the elements are swapped between the hash buckets in inet_frag_secret_rebuild it seems you forgot to update chain_len correctly.

Thanks,

Hannes

#24 lore raw

Re: [net-next PATCH V2] net: frag queue per hash bucket locking

Quoted 10 lines · Hannes Frederic Sowa

include/net/inet_frag.h struct inet_frag_queue { · line 58

Quoted 6 lines · Jesper Dangaard Brouer
include/net/inet_frag.h +6 -0
+struct inet_frag_bucket {
+	struct hlist_head	chain;
+	spinlock_t		chain_lock;
+	u16			chain_len;
+};
+

I just noticed and wanted to ask for what chain_len is needed? Could it be dropped?

It could be dropped from this patch. Its part of my future hash cleanup strategy.
I also wanted to use it to replace the nqueues counter, but its would not be correct, because nqueues counter is maintained per netns (network namespace).

Its currently the netns separation, which is causing "headaches" for my removal of LRU and direct-hash-cleaning solution...

Quoted 3 lines

If the elements are swapped between the hash buckets in inet_frag_secret_rebuild it seems you forgot to update chain_len correctly.

Ah, good catch.
Given its not even correct, I'll remove the chain_len and repost a V3.

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#25 lore raw

PATCH v3 1/1 net: frag queue per hash bucket locking

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

This patch is part of "net: frag performance followup"
http://thread.gmane.org/gmane.linux.network/263644
of which two patches have already been accepted:

Same test setup as previous:
(http://thread.gmane.org/gmane.linux.network/257155)
Two 10G interfaces, on seperate NUMA nodes, are under-test, and uses Ethernet flow-control. A third interface is used for generating the DoS attack (with trafgen).

Notice, I have changed the frag DoS generator script to be more efficient/deadly. Before it would only hit one RX queue, now its sending packets causing multi-queue RX, due to "better" RX hashing.

Test types summary (netperf UDP_STREAM):
Test-20G64K == 2x10G with 65K fragments
Test-20G3F == 2x10G with 3x fragments (3*1472 bytes)
Test-20G64K+DoS == Same as 20G64K with frag DoS

 Test-20G3F+DoS  == Same as 20G3F  with frag DoS
 Test-20G64K+MQ  == Same as 20G64K with Multi-Queue frag DoS
 Test-20G3F+MQ   == Same as 20G3F  with Multi-Queue frag DoS

When I rebased this-patch(03) (on top of net-next commit a210576c) and removed the _bh spinlock, I saw a performance regression. BUT this was caused by some unrelated change in-between. See tests below.

Test (A) is what I reported before for patch-02, accepted in commit 1b5ab0de. Test (B) verifying-retest of commit 1b5ab0de corrospond to patch-02. Test (C) is what I reported before for this-patch

Test (D) is net-next master HEAD (commit a210576c), which reveals some (unknown) performance regression (compared against test (B)). Test (D) function as a new base-test.

Performance table summary (in Mbit/s):

(#) Test-type:  20G64K    20G3F    20G64K+DoS  20G3F+DoS  20G64K+MQ 20G3F+MQ
    ----------  -------   -------  ----------  ---------  --------  -------
(A) Patch-02  : 18848.7   13230.1   4103.04     5310.36     130.0    440.2
(B) 1b5ab0de  : 18841.5   13156.8   4101.08     5314.57     129.0    424.2
(C) Patch-03v1: 18838.0   13490.5   4405.11     6814.72     196.6    461.6
(D) a210576c  : 18321.5   11250.4   3635.34     5160.13     119.1    405.2
(E) with _bh  : 17247.3   11492.6   3994.74     6405.29     166.7    413.6
(F) without bh: 17471.3   11298.7   3818.05     6102.11     165.7    406.3

Test (E) and (F) is this-patch(03), with(V1) and without(V2) the _bh spinlocks.

I cannot explain the slow down for 20G64K (but its an artificial "lab-test" so I'm not worried). But the other results does show improvements. And test (E) "with _bh" version is slightly better.

----
V2:
- By analysis from Hannes Frederic Sowa and Eric Dumazet, we don't need the spinlock _bh versions, as Netfilter currently does a local_bh_disable() before entering inet_fragment.
- Fold-in desc from cover-mail
V3:
- Drop the chain_len counter per hash bucket.
---

 include/net/inet_frag.h  |    8 ++++++
 net/ipv4/inet_fragment.c |   57 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 51 insertions(+), 14 deletions(-)
include/net/inet_frag.h +7 -1
index 7cac9c5..6f41b45 100644
@@ -50,10 +50,16 @@ struct inet_frag_queue {
  */
 #define INETFRAGS_MAXDEPTH		128
 
+struct inet_frag_bucket {
+	struct hlist_head	chain;
+	spinlock_t		chain_lock;
+};
+
 struct inet_frags {
-	struct hlist_head	hash[INETFRAGS_HASHSZ];
+	struct inet_frag_bucket	hash[INETFRAGS_HASHSZ];
 	/* This rwlock is a global lock (seperate per IPv4, IPv6 and
 	 * netfilter). Important to keep this on a seperate cacheline.
+	 * Its primarily a rebuild protection rwlock.
 	 */
 	rwlock_t		lock ____cacheline_aligned_in_smp;
 	int			secret_interval;
net/ipv4/inet_fragment.c +44 -13

134 lines not on this page · raw message

Acked-by on v3 1/1
#26 lore raw

Re: [net-next PATCH V3] net: frag queue per hash bucket locking

Quoted 10 lines · Jesper Dangaard Brouer

----
V2:
- By analysis from Hannes Frederic Sowa and Eric Dumazet, we don't need the spinlock _bh versions, as Netfilter currently does a local_bh_disable() before entering inet_fragment.
- Fold-in desc from cover-mail
V3:
- Drop the chain_len counter per hash bucket.

Acked-by on v3 1/1
#27 lore raw

Re: [net-next PATCH V3] net: frag queue per hash bucket locking

In reply to Jesper Dangaard Brouer
Quoted 4 lines · Jesper Dangaard Brouer

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

maintainer Applied
#28 lore raw

Re: [net-next PATCH V3] net: frag queue per hash bucket locking

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Apr 2013 09:24:15 -0700

Quoted 7 lines · Jesper Dangaard Brouer
Quoted 4 lines

This patch implements per hash bucket locking for the frag queue hash. This removes two write locks, and the only remaining write lock is for protecting hash rebuild. This essentially reduce the readers-writer lock to a rebuild lock.

Applied, thanks everyone.