udp: reduce cache pressure

Since UDP no more uses sk->destructor, we can clear completely the skb head state before enqueuing. Amend and use skb_release_head_state() for that.

All head states share a single cacheline, which is not normally used/accesses on dequeue. We can avoid entirely accessing such cacheline implementing and using in the UDP code a specialized skb free helper which ignores the skb head state.

This saves a cacheline miss at skb deallocation time.

v1 -> v2:
replaced secpath_reset() with skb_release_head_state()

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 include/linux/skbuff.h |  2 ++
 net/core/skbuff.c      | 24 ++++++++++++++++++++----
 net/ipv4/udp.c         |  6 +++++-
 3 files changed, 27 insertions(+), 5 deletions(-)

include/linux/skbuff.h +2 -0

index decce36..d66d4fe 100644
@@ -880,10 +880,12 @@ static inline bool skb_unref(struct sk_buff *skb)
 	return true;
 }
 
+void skb_release_head_state(struct sk_buff *skb);
 void kfree_skb(struct sk_buff *skb);
 void kfree_skb_list(struct sk_buff *segs);
 void skb_tx_error(struct sk_buff *skb);
 void consume_skb(struct sk_buff *skb);
+void consume_stateless_skb(struct sk_buff *skb);
 void  __kfree_skb(struct sk_buff *skb);
 extern struct kmem_cache *skbuff_head_cache;
 

net/core/skbuff.c +20 -4

index 747263c..3046027 100644
@@ -643,12 +643,10 @@ static void kfree_skbmem(struct sk_buff *skb)
 	kmem_cache_free(skbuff_fclone_cache, fclones);
 }
 
-static void skb_release_head_state(struct sk_buff *skb)
+void skb_release_head_state(struct sk_buff *skb)
 {
 	skb_dst_drop(skb);
-#ifdef CONFIG_XFRM
-	secpath_put(skb->sp);
-#endif
+	secpath_reset(skb);
 	if (skb->destructor) {
 		WARN_ON(in_irq());
 		skb->destructor(skb);

@@ -751,6 +749,24 @@ void consume_skb(struct sk_buff *skb)
 }
 EXPORT_SYMBOL(consume_skb);
 
+/**
+ *	consume_stateless_skb - free an skbuff, assuming it is stateless
+ *	@skb: buffer to free
+ *
+ *	Works like consume_skb(), but this variant assumes that all the head
+ *	states have been already dropped.
+ */
+void consume_stateless_skb(struct sk_buff *skb)
+{
+	if (!skb_unref(skb))
+		return;
+
+	trace_consume_skb(skb);
+	if (likely(skb->head))
+		skb_release_data(skb);
+	kfree_skbmem(skb);
+}
+
 void __kfree_skb_flush(void)
 {
 	struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache);

net/ipv4/udp.c +2 -1

index fdcb743..d8b265f 100644
@@ -1359,7 +1359,8 @@ void skb_consume_udp(struct sock *sk, struct sk_buff *skb, int len)
 		sk_peek_offset_bwd(sk, len);
 		unlock_sock_fast(sk, slow);
 	}
-	consume_skb(skb);
+
+	consume_stateless_skb(skb);
 }
 EXPORT_SYMBOL_GPL(skb_consume_udp);
 

Eric noticed that in udp_recvmsg() we still need to access skb->dst while processing the IP options.
Since commit 0a463c78d25b ("udp: avoid a cache miss on dequeue") skb->dst is no more available at recvmsg() time and bad things will happen if we enter the relevant code path.

This commit address the issue, avoid clearing skb->dst if any IP options are present into the relevant skb. Since the IP CB is contained in the first skb cacheline, we can test it to decide to leverage the consume_stateless_skb() optimization, without measurable additional cost in the faster path.

Fixes b65ac44674dd ("udp: try to avoid 2 cache miss on dequeue")

Reported-by Eric Dumazet <[email protected]>

Signed-off-by Paolo Abeni <[email protected]>

---
 net/ipv4/udp.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

net/ipv4/udp.c +11 -2

index 25294d4..b057653 100644
@@ -1388,6 +1388,11 @@ void skb_consume_udp(struct sock *sk, struct sk_buff *skb, int len)
 		unlock_sock_fast(sk, slow);
 	}
 
+	/* we cleared the head states previously only if the skb lacks any IP
+	 * options, see __udp_queue_rcv_skb().
+	 */
+	if (unlikely(IPCB(skb)->opt.optlen > 0))
+		skb_release_head_state(skb);
 	consume_stateless_skb(skb);
 }
 EXPORT_SYMBOL_GPL(skb_consume_udp);

@@ -1779,8 +1784,12 @@ static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 		sk_mark_napi_id_once(sk, skb);
 	}
 
-	/* clear all pending head states while they are hot in the cache */
-	skb_release_head_state(skb);
+	/* At recvmsg() time we need skb->dst to process IP options-related
+	 * cmsg, elsewhere can we clear all pending head states while they are
+	 * hot in the cache
+	 */
+	if (likely(IPCB(skb)->opt.optlen == 0))
+		skb_release_head_state(skb);
 
 	rc = __udp_enqueue_schedule_skb(sk, skb);
 	if (rc < 0) {

Quoted 13 lines · Paolo Abeni

Eric noticed that in udp_recvmsg() we still need to access skb->dst while processing the IP options.
Since commit 0a463c78d25b ("udp: avoid a cache miss on dequeue") skb->dst is no more available at recvmsg() time and bad things will happen if we enter the relevant code path.

This commit address the issue, avoid clearing skb->dst if any IP options are present into the relevant skb. Since the IP CB is contained in the first skb cacheline, we can test it to decide to leverage the consume_stateless_skb() optimization, without measurable additional cost in the faster path.

Quoted 1 line

Fixes b65ac44674dd ("udp: try to avoid 2 cache miss on dequeue")

Strange.... I thought bug was coming from
commit 0a463c78d25b ("udp: avoid a cache miss on dequeue")

Please also add original reporter, from syzkaller team.

Reported-by Andrey Konovalov <[email protected]>

Eric noticed that in udp_recvmsg() we still need to access skb->dst while processing the IP options.
Since commit 0a463c78d25b ("udp: avoid a cache miss on dequeue") skb->dst is no more available at recvmsg() time and bad things will happen if we enter the relevant code path.

This commit address the issue, avoid clearing skb->dst if any IP options are present into the relevant skb. Since the IP CB is contained in the first skb cacheline, we can test it to decide to leverage the consume_stateless_skb() optimization, without measurable additional cost in the faster path.

Quoted 6 lines · Paolo Abeni

Eric noticed that in udp_recvmsg() we still need to access skb->dst while processing the IP options.
Since commit 0a463c78d25b ("udp: avoid a cache miss on dequeue") skb->dst is no more available at recvmsg() time and bad things will happen if we enter the relevant code path.

Quoted 6 lines

This commit address the issue, avoid clearing skb->dst if any IP options are present into the relevant skb. Since the IP CB is contained in the first skb cacheline, we can test it to decide to leverage the consume_stateless_skb() optimization, without measurable additional cost in the faster path.

Also the problem was not skb->dst at all, but fact that skb->head can not be freed if IP options need later to be fetched.

( __ip_options_echo() ->
sptr = skb_network_header(skb);

... -> use after free

when udp_recvmsg() is executed, on x86_64 and other archs, most skb fields are on cold cachelines.
If the skb are linear and the kernel don't need to compute the udp csum, only a handful of skb fields are required by udp_recvmsg(). Since we already use skb->dev_scratch to cache hot data, and there are 32 bits unused on 64 bit archs, use such field to cache as much data as we can, and try to prefetch on dequeue the relevant fields that are left out.

This can save up to 2 cache miss per packet.

v1 -> v2:
- changed udp_dev_scratch fields types to u{32,16} variant, replaced bitfiled with bool

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 net/ipv4/udp.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 103 insertions(+), 11 deletions(-)

Quoted 15 lines · Paolo

when udp_recvmsg() is executed, on x86_64 and other archs, most skb fields are on cold cachelines.
If the skb are linear and the kernel don't need to compute the udp csum, only a handful of skb fields are required by udp_recvmsg(). Since we already use skb->dev_scratch to cache hot data, and there are 32 bits unused on 64 bit archs, use such field to cache as much data as we can, and try to prefetch on dequeue the relevant fields that are left out.

This can save up to 2 cache miss per packet.

v1 -> v2:
- changed udp_dev_scratch fields types to u{32,16} variant, replaced bitfiled with bool

Quoted 5 lines

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 net/ipv4/udp.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 103 insertions(+), 11 deletions(-)

This appears to break wget on one of my machines.

Networking in general is working, I'm able to SSH in, but then I can't do a wget.

eg:

$ wget google.com
--2017-06-22 22:45:39--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

$ host proxy.pmdw.com
proxy.pmdw.com is an alias for raven.pmdw.com.
raven.pmdw.com has address 10.1.2.3

$ wget google.com
--2017-06-22 22:52:08--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

Maybe host is using TCP but the man page says it doesn't?

Everything is OK if I boot back to the previous commit 0a463c78d25b ("udp: avoid a cache miss on dequeue"):

$ wget google.com
--2017-06-22 23:00:01--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... 10.1.2.3
Connecting to proxy.pmdw.com (proxy.pmdw.com)|10.1.2.3|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE [following]
--2017-06-22 23:00:01--  http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE
Reusing existing connection to proxy.pmdw.com:3128.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html [ <=> ] 11.37K --.-KB/s in 0.001s

2017-06-22 23:00:01 (22.0 MB/s) - ‘index.html’ saved [11640]

$ uname -a
Linux 4.12.0-rc4-gcc6-00988-g0a463c7 #88 SMP Thu Jun 22 22:55:12 AEST 2017 ppc64 GNU/Linux

Haven't had time to debug any further. Any ideas?

Quoted 69 lines · Michael Ellerman

Quoted 15 lines · Paolo

when udp_recvmsg() is executed, on x86_64 and other archs, most skb fields are on cold cachelines.
If the skb are linear and the kernel don't need to compute the udp csum, only a handful of skb fields are required by udp_recvmsg(). Since we already use skb->dev_scratch to cache hot data, and there are 32 bits unused on 64 bit archs, use such field to cache as much data as we can, and try to prefetch on dequeue the relevant fields that are left out.

This can save up to 2 cache miss per packet.

v1 -> v2:
- changed udp_dev_scratch fields types to u{32,16} variant, replaced bitfiled with bool

Quoted 5 lines

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 net/ipv4/udp.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 103 insertions(+), 11 deletions(-)

This appears to break wget on one of my machines.

Networking in general is working, I'm able to SSH in, but then I can't do a wget.

eg:

$ wget google.com
--2017-06-22 22:45:39--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

$ host proxy.pmdw.com
proxy.pmdw.com is an alias for raven.pmdw.com.
raven.pmdw.com has address 10.1.2.3

$ wget google.com
--2017-06-22 22:52:08--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

Maybe host is using TCP but the man page says it doesn't?

Everything is OK if I boot back to the previous commit 0a463c78d25b ("udp: avoid a cache miss on dequeue"):

$ wget google.com
--2017-06-22 23:00:01--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... 10.1.2.3
Connecting to proxy.pmdw.com (proxy.pmdw.com)|10.1.2.3|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE [following]
--2017-06-22 23:00:01--  http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE
Reusing existing connection to proxy.pmdw.com:3128.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html [ <=> ] 11.37K --.-KB/s in 0.001s

2017-06-22 23:00:01 (22.0 MB/s) - ‘index.html’ saved [11640]

$ uname -a
Linux 4.12.0-rc4-gcc6-00988-g0a463c7 #88 SMP Thu Jun 22 22:55:12 AEST 2017 ppc64 GNU/Linux

Quoted 1 line

Haven't had time to debug any further. Any ideas?

Thank you for this report.

Can you please specify features of the relevant NIC ? (ethtool -k <name>)

I'll try to replicate the issue as soon I'll get hands on suitable HW,

Quoted 69 lines · Michael Ellerman

Quoted 15 lines · Paolo

when udp_recvmsg() is executed, on x86_64 and other archs, most skb fields are on cold cachelines.
If the skb are linear and the kernel don't need to compute the udp csum, only a handful of skb fields are required by udp_recvmsg(). Since we already use skb->dev_scratch to cache hot data, and there are 32 bits unused on 64 bit archs, use such field to cache as much data as we can, and try to prefetch on dequeue the relevant fields that are left out.

This can save up to 2 cache miss per packet.

v1 -> v2:
- changed udp_dev_scratch fields types to u{32,16} variant, replaced bitfiled with bool

Quoted 5 lines

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 net/ipv4/udp.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 103 insertions(+), 11 deletions(-)

This appears to break wget on one of my machines.

Networking in general is working, I'm able to SSH in, but then I can't do a wget.

eg:

$ wget google.com
--2017-06-22 22:45:39--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

$ host proxy.pmdw.com
proxy.pmdw.com is an alias for raven.pmdw.com.
raven.pmdw.com has address 10.1.2.3

$ wget google.com
--2017-06-22 22:52:08--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘proxy.pmdw.com’

Maybe host is using TCP but the man page says it doesn't?

Everything is OK if I boot back to the previous commit 0a463c78d25b ("udp: avoid a cache miss on dequeue"):

$ wget google.com
--2017-06-22 23:00:01--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... 10.1.2.3
Connecting to proxy.pmdw.com (proxy.pmdw.com)|10.1.2.3|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE [following]
--2017-06-22 23:00:01--  http://www.google.com.au/?gfe_rd=cr&ei=Ub9LWbPbLujDXrH1uPgE
Reusing existing connection to proxy.pmdw.com:3128.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html [ <=> ] 11.37K --.-KB/s in 0.001s

2017-06-22 23:00:01 (22.0 MB/s) - ‘index.html’ saved [11640]

$ uname -a
Linux 4.12.0-rc4-gcc6-00988-g0a463c7 #88 SMP Thu Jun 22 22:55:12 AEST 2017 ppc64 GNU/Linux

Quoted 1 line

Haven't had time to debug any further. Any ideas?

Thank you for this report.

Can you please specify features of the relevant NIC ? (ethtool -k <name>)

I'll try to replicate the issue as soon I'll get hands on suitable HW,

Quoted 15 lines · Paolo

when udp_recvmsg() is executed, on x86_64 and other archs, most skb fields are on cold cachelines.
If the skb are linear and the kernel don't need to compute the udp csum, only a handful of skb fields are required by udp_recvmsg(). Since we already use skb->dev_scratch to cache hot data, and there are 32 bits unused on 64 bit archs, use such field to cache as much data as we can, and try to prefetch on dequeue the relevant fields that are left out.

This can save up to 2 cache miss per packet.

v1 -> v2:
- changed udp_dev_scratch fields types to u{32,16} variant, replaced bitfiled with bool

Quoted 5 lines

Signed-off-by Paolo Abeni <[email protected]>

Acked-by Eric Dumazet <[email protected]>

---
 net/ipv4/udp.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 103 insertions(+), 11 deletions(-)

This appears to break wget on one of my machines.

Networking in general is working, I'm able to SSH in, but then I can't do a wget.

Can you please check if the following patch fixes the issue? Only compiled tested here.

Thanks!!!
---

net/ipv4/udp.c +0 -0

index 067a607..80d89fe 100644
@@ -1446,16 +1446,19 @@ static struct sk_buff
 *__first_packet_length(struct sock *sk,
  {
  	struct sk_buff *skb;

net/ipv4/udp.c +2 -10

-       while ((skb = skb_peek(rcvq)) != NULL &&
-              udp_lib_checksum_complete(skb)) {
-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
-                               IS_UDPLITE(sk));
-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
-                               IS_UDPLITE(sk));
-               atomic_inc(&sk->sk_drops);
-               __skb_unlink(skb, rcvq);
-               *total += skb->truesize;
-               kfree_skb(skb);
+       while ((skb = skb_peek(rcvq)) != NULL) {
+               if (udp_lib_checksum_complete(skb)) {

+ }
  }
  return skb;
 }

Quoted 9 lines · Paolo Abeni

=20
Can you please check if the following patch fixes the issue? Only compiled tested here.
=20
Thanks!!!
---

net/ipv4/udp.c +0 -1

index 067a607..80d89fe 100644
--- a/net/ipv4/udp.c

net/ipv4/udp.c

Quoted 8 lines

+++ b/net/ipv4/udp.c

net/ipv4/udp.c +0 -0

@@ -1446,16 +1446,19 @@ static struct sk_buff
 *__first_packet_length(struct sock *sk,
 =C2=A0{
 =C2=A0	struct sk_buff *skb;

=C2=A0

net/ipv4/udp.c +0 -2

-       while ((skb =3D skb_peek(rcvq)) !=3D NULL &&
-       =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0udp_lib_checksum_compl=

Quoted 10 lines

net/ipv4/udp.c +2 -8

-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
-                               IS_UDPLITE(sk));
-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
-                               IS_UDPLITE(sk));
-               atomic_inc(&sk->sk_drops);
-               __skb_unlink(skb, rcvq);
-               *total +=3D skb->truesize;
-               kfree_skb(skb);
+       while ((skb =3D skb_peek(rcvq)) !=3D NULL) {
+               if (udp_lib_checksum_complete(skb)) {

net/ipv4/udp.c

Quoted 10 lines

net/ipv4/udp.c +10 -0

+                       __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
+                                       IS_UDPLITE(sk));
+                       __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
+                                       IS_UDPLITE(sk));
+                       atomic_inc(&sk->sk_drops);
+                       __skb_unlink(skb, rcvq);
+                       *total +=3D skb->truesize;
+                       kfree_skb(skb);
+               } else {
+                       udp_set_dev_scratch(skb);

It needs a "break;" here.

Quoted 4 lines

+ }
=C2=A0 }
=C2=A0 return skb;
=C2=A0}

Quoted 32 lines · Hannes Frederic Sowa

Quoted 31 lines · Paolo Abeni

Can you please check if the following patch fixes the issue? Only compiled tested here.

Thanks!!!
---

net/ipv4/udp.c +0 -0

index 067a607..80d89fe 100644
@@ -1446,16 +1446,19 @@ static struct sk_buff
 *__first_packet_length(struct sock *sk,
  {
  	struct sk_buff *skb;

net/ipv4/udp.c +6 -10

-       while ((skb = skb_peek(rcvq)) != NULL &&
-              udp_lib_checksum_complete(skb)) {
-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
-                               IS_UDPLITE(sk));
-               __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
-                               IS_UDPLITE(sk));
-               atomic_inc(&sk->sk_drops);
-               __skb_unlink(skb, rcvq);
-               *total += skb->truesize;
-               kfree_skb(skb);
+       while ((skb = skb_peek(rcvq)) != NULL) {
+               if (udp_lib_checksum_complete(skb)) {
+                       __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
+                                       IS_UDPLITE(sk));
+                       __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
+                                       IS_UDPLITE(sk));

Quoted 13 lines

net/ipv4/udp.c

Quoted 6 lines

net/ipv4/udp.c +6 -0

+                       atomic_inc(&sk->sk_drops);
+                       __skb_unlink(skb, rcvq);
+                       *total += skb->truesize;
+                       kfree_skb(skb);
+               } else {
+                       udp_set_dev_scratch(skb);

It needs a "break;" here.

Quoted 4 lines

+ }
  }
  return skb;
 }

That works!

$ wget google.com
--2017-06-23 16:56:31--  http://google.com/
Resolving proxy.pmdw.com (proxy.pmdw.com)... 10.1.2.3
Connecting to proxy.pmdw.com (proxy.pmdw.com)|10.1.2.3|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.google.com.au/?gfe_rd=cr&ei=n7tMWeb9JYPr8wfg4LXYAQ [following]
--2017-06-23 16:56:31--  http://www.google.com.au/?gfe_rd=cr&ei=n7tMWeb9JYPr8wfg4LXYAQ
Reusing existing connection to proxy.pmdw.com:3128.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

The patch had whitespace issues or something and I had to apply it by hand, here's what I actually tested.

Thank you!

I'll submit formally the patch after some more testing.

I noticed this version has entered the ppc patchwork, but I think that the formal submission should go towards the net-next tree.

In the most common use case, many skb fields are not used by recvmsg(), and the few ones actually accessed lays on cold cachelines, which leads to several cache miss per packet.

This patch series attempts to reduce such misses with different strategies:
* caching the interesting fields in the scratched space
* avoid accessing at all uninteresting fields
* prefetching

Tested using the udp_sink program by Jesper[1] as the receiver, an h/w l4 rx hash on the ingress nic, so that the number of ingress nic rx queues hit by the udp traffic could be controlled via ethtool -L.

The udp_sink program was bound to the first idle cpu, to get more stable numbers.

On a single numa node receiver:

nic rx queues           vanilla                 patched kernel      delta
1                       1850 kpps               1850 kpps           0%
2                       2370 kpps               2700 kpps           13.9%
16                      2000 kpps               2220 kpps           11%

[1] https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c