IPoIB: Leave space in skb linear buffer for IP headers
PATCH v4 1/1 IPoIB: Leave space in skb linear buffer for IP headers
From: Roland Dreier <[email protected]>
Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing memcpy() in __pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.
We're already allocating extra tailroom for the skb linear part, so we might as well use it. In fact, we might as well allocate a big enough linear part so that all the data fits there in the relatively common case of a 2K IB MTU, and only use a fragment page for 4K IB MTU.
---
v4: Leave enough space in linear part of skb so that all data ends up there with 2K IB MTU. Still not sure how this affects perf with a 4K IB MTU (should be better since we avoid pulling IP headers out of first fragment).
drivers/infiniband/ulp/ipoib/ipoib.h | 6 ++- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 78 +++++++++++++++++++-------------- 2 files changed, 49 insertions(+), 35 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +4 -2
@@ -64,8 +64,9 @@ enum ipoib_flush_level {enum {IPOIB_ENCAP_LEN =4 , -IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +IPOIB_ENCAP_LEN , -IPOIB_UD_RX_SG =2 ,/* max buffer needed for 4K mtu */ +/* enough so w/ 2K mtu, everything is in linear part of skb */ +IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +2048 , +IPOIB_UD_RX_SG =2 ,/* max buffer needed for 4K mtu w/ 4K pages */ IPOIB_CM_MTU =0x10000 -0x10 ,/* padding to align header to 16 */ IPOIB_CM_BUF_SIZE =IPOIB_CM_MTU +IPOIB_ENCAP_LEN ,
@@ -155,6 +156,7 @@ struct ipoib_mcast {struct ipoib_rx_buf {struct sk_buff *skb; +struct page *page;u64 mapping[IPOIB_UD_RX_SG ]; };
drivers/infiniband/ulp/ipoib/ipoib_ib.c +45 -33
173 lines not on this page · raw message
Signature · 6 lines
1.8.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1366129801-29234-1-git-send-email-roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Quoted 13 lines · Roland Dreier
From: Roland Dreier <[email protected]>
Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing
memcpy()in__pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.We're already allocating extra tailroom for the skb linear part, so we might as well use it. In fact, we might as well allocate a big enough linear part so that all the data fits there in the relatively common case of a 2K IB MTU, and only use a fragment page for 4K IB MTU.
Quoted 9 lines
Cc Eric Dumazet <[email protected]>Reported-by Markus Stockhausen <[email protected]>Signed-off-by Roland Dreier <[email protected]>---
v4: Leave enough space in linear part of skb so that all data ends up there with 2K IB MTU. Still not sure how this affects perf with a 4K IB MTU (should be better since we avoid pulling IP headers out of first fragment).
I am afraid I don't understand what the issue is.
the pull_tail() in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers in skb->head , and nobody noticed.
Real cost is the cache line miss.
Now, if you pull too many bytes in skb->head, say part of TCP payload, you lose opportunities in TCP coalescing or splice().
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CANn89iK9Z7i59+pHXAXJzgYZxKB6OptR-+TRwfjTqNwHMLDOow-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 7 lines · Eric Dumazet
I am afraid I don't understand what the issue is.
the
pull_tail()in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers inskb->head, and nobody noticed.Real cost is the cache line miss.
According to Markus there is a pretty huge performance penalty if the whole packet (TCP/IP headers + payload) ends up in the fragment, and stack has to memcpy headers into the linear part.
Quoted 2 lines
Now, if you pull too many bytes in
skb->head, say part of TCP payload, you lose opportunities in TCP coalescing orsplice().
So it sounds like this patch is a bad change from that POV, since it would cause us to have lots of TCP payload in the linear part and then second half of the payload in the fragments (if we are running with 4K IPoIB MTU).
Seems like the real fix is to enable the header separation feature in adapters that support it (ConnectX-2 and -3) so that exactly TCP/IP headers go in linear part and payload goes in the fragment. That's a bit more involved, but I'll look into doing it.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines
I am afraid I don't understand what the issue is.
the
pull_tail()in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers inskb->head, and nobody noticed.Real cost is the cache line miss.
Now, if you pull too many bytes in
skb->head, say part of TCP payload, you lose opportunities in TCP coalescing orsplice().
With patch v4 netperf and NFS receive performance raises to the expected values. As I'm no expert in this I can only repost the initial performance report that started the whole discussion. __pskb_pull_tail consumes a lot time on our XEON L5420 test server.
...
- server side: netserver -p 12345
- client side: netperf -H <server_ip> -p 12345 -l 120
Analysis was performed on the server side with
- perf record -a -g sleep 10
- perf report
# Overhead Symbol
# ........ .............................................
#
19.67% [k] copy_user_generic_string
|
|--99.74%-- skb_copy_datagram_iovec
| tcp_recvmsg
| inet_recvmsg
| sock_recvmsg
| sys_recvfrom
| system_call_fastpath
| recv
| |
| |--50.17%-- 0x7074656e00667265
| |
| --49.83%-- 0x6672657074656e
--0.26%-- [...]
7.38% [k] memcpy
|
|--84.56%-- __pskb_pull_tail
| |
| |--81.88%-- pskb_may_pull.part.6
| | skb_gro_header_slow
| | inet_gro_receive
| | dev_gro_receive
| | napi_gro_receive
| | ipoib_ib_handle_rx_wc
| | ipoib_poll
| | net_rx_action
| | __do_softirqMarkus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CD935857.5873%[email protected]
Quoted 9 lines · Markus Stockhausen
Quoted 9 lines
I am afraid I don't understand what the issue is.
the
pull_tail()in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers inskb->head, and nobody noticed.Real cost is the cache line miss.
Quoted 9 lines
Quoted 2 lines
Now, if you pull too many bytes in
skb->head, say part of TCP payload, you lose opportunities in TCP coalescing orsplice().With patch v4 netperf and NFS receive performance raises to the expected values. As I'm no expert in this I can only repost the initial performance report that started the whole discussion.
__pskb_pull_tailconsumes a lot time on our XEON L5420 test server.
That's probably because of a cache line miss.
The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.
So what does this adapter in this respect ?
I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.
Many drivers use prefetch() to make sure cpu starts to bring this cache line into cache as soon as possible.
A single prefetch() call at the right place might help a lot. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CANn89iLopRHCMhYc+BcPvhZJtfL53H3T6JdP=ssfdg=oxh+Waw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
IPoIB header prefetch for fragmented SKB
Quoted 10 lines
That's probably because of a cache line miss.
The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.
So what does this adapter in this respect ?
Quoted 7 lines
I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.
Many drivers use
prefetch()to make sure cpu starts to bring this cache line into cache as soon as possible.A single
prefetch()call at the right place might help a lot.
Hello,
@Eric: Thanks for the tip.
In the 4K MAX MTU IPoIB driver path ipoib_ib_handle_rx_wc() will produce an empty skb linear part with the whole data placed into the first fragment. napi_gro_receive() finally pulls the IP header out of the fragment into the linear part.
As far as I understand the pull out of the fragment should come without additional cost when one calls a prefetch "long" before the skb_pull().
I'm willing to check this out but I'm unsure if the IP header is aligned to a cache line of 64 bytes. As a simple guess I would implement the prefetch here:
static void ipoib_ib_handle_rx_wc();
...
skb_pull (skb, IB_GRH_BYTES);
skb->protocol = ((struct ipoib_header *) skb->data)->proto; skb_reset_mac_header(skb);
skb_pull(skb, IPOIB_ENCAP_LEN);
+
+ if (ipoib_ud_need_sg(priv->max_ib_mtu))
+ prefetch(<whatever address>);
...
Can you give me a hint what address one should put into the call?
Thanks in advance.
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CD94B1E6.588B%[email protected]
Re: IPoIB header prefetch for fragmented SKB
Please try :
(quoted diff) +0 -0
b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 2cfa76f..9bdff21 100644
drivers/infiniband/ulp/ipoib/ipoib_ib.c +0 -0
@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct
ipoib_dev_priv *priv,
if (ipoib_ud_need_sg(priv->max_ib_mtu)) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
unsigned int size;+
+ prefetch(page_address(frag->page.p));
/*
* There is only two buffers needed for max_payload = 4K,
* first buf size is IPOIB_UD_HEAD_SIZE
Quoted 47 lines · Markus Stockhausen
Quoted 10 lines
That's probably because of a cache line miss.
The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.
So what does this adapter in this respect ?
Quoted 7 lines
I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.
Many drivers use
prefetch()to make sure cpu starts to bring this cache line into cache as soon as possible.A single
prefetch()call at the right place might help a lot.Hello,
@Eric: Thanks for the tip.
In the 4K MAX MTU IPoIB driver path
ipoib_ib_handle_rx_wc()will produce an empty skb linear part with the whole data placed into the first fragment.napi_gro_receive()finally pulls the IP header out of the fragment into the linear part.As far as I understand the pull out of the fragment should come without additional cost when one calls a prefetch "long" before the
skb_pull().I'm willing to check this out but I'm unsure if the IP header is aligned to a cache line of 64 bytes. As a simple guess I would implement the prefetch here:
static void
ipoib_ib_handle_rx_wc();
...
skb_pull(skb,IB_GRH_BYTES);skb->protocol = ((struct ipoib_header *) skb->data)->proto; skb_reset_mac_header(skb); skb_pull(skb, IPOIB_ENCAP_LEN);(quoted diff) +3
-0+ + if (ipoib_ud_need_sg(priv->max_ib_mtu)) + prefetch(<whatever address>);
Quoted 7 lines
Can you give me a hint what address one should put into the call?
Thanks in advance.
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CANn89iLwF9LUohz_c765c4b+6DC0gTH77bN5QzbAkR-4LCcUog-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: IPoIB header prefetch for fragmented SKB
Quoted 1 line · Eric Dumazet
+ prefetch(page_address(frag->page.p));
I guess we would have to change our allocation to __get_free_page() (instead of alloc_page()), so that we make sure never to get a highmem page?
Other than that, seems fine.
By the way, is there any value to header splitting -- having the adapter put the TCP/IP headers in the linear part and the payload in the fragment page? How about in the case where we have a small enough MTU that we could put the whole packet in the linear part?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAL1RGDWVOPsEe+92sgcz+fDWj8=02CJt40889_cgsQKBX6XJ4A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: IPoIB header prefetch for fragmented SKB
alloc_page(GFP_ATOMIC) is safe. it doesnt give you a highmempage
Not sure about header splitting. If done, it probably should be smart as you said (allowing small frames to be in the first part)
Quoted 7 lines · Roland Dreier
Quoted 1 line · Eric Dumazet
+ prefetch(page_address(frag->page.p));
I guess we would have to change our allocation to
__get_free_page()(instead ofalloc_page()), so that we make sure never to get a highmem page?
Quoted 8 lines
Other than that, seems fine.
By the way, is there any value to header splitting -- having the adapter put the TCP/IP headers in the linear part and the payload in the fragment page? How about in the case where we have a small enough MTU that we could put the whole packet in the linear part?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: IPoIB header prefetch for fragmented SKB
In reply to a message not in the archiveAm 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:
Quoted 16 lines
Please try :
drivers/infiniband/ulp/ipoib/ipoib_ib.c +2
-0@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv, if (ipoib_ud_need_sg(priv->max_ib_mtu)) {skb_frag_t *frag = &skb_shinfo (skb)->frags[0 ];unsigned int size; + +prefetch (page_address (frag->page.p));/* * There is only two buffers needed for max_payload = 4K,
Quoted 1 line
* first buf size is
IPOIB_UD_HEAD_SIZE
Thanks to both of you for spending time on this.
Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside pskb_may_pull() just in front of __pskb_pull_tail() to give an idea what happens.
During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:
[ 52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes [ 52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
Now I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.
Any ideas?
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CD975158.5900%[email protected]
Re: IPoIB header prefetch for fragmented SKB
I dont think so : a total of 52 bytes are pulled.
(20 bytes for IPv4 header, 32 bytes for TCP header (TS))
Quoted 39 lines · Markus Stockhausen
Am 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:
Quoted 16 lines
Please try :
drivers/infiniband/ulp/ipoib/ipoib_ib.c +2
-0@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv, if (ipoib_ud_need_sg(priv->max_ib_mtu)) {skb_frag_t *frag = &skb_shinfo (skb)->frags[0 ];unsigned int size; + +prefetch (page_address (frag->page.p));/* * There is only two buffers needed for max_payload = 4K, Quoted 1 line
* first buf size is
IPOIB_UD_HEAD_SIZEThanks to both of you for spending time on this.
Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside
pskb_may_pull()just in front of__pskb_pull_tail()to give an idea what happens.During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:
[ 52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes [ 52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
Quoted 10 lines
Now I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.
Any ideas?
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CANn89iL8_sD5m-DcrwVuJ6-Bi=tCOof4g2DLi-5Wxcz62rv0iA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: IPoIB header prefetch for fragmented SKB
This pulls cannot explain a major performance problem.
Its a memcpy( ... 20 bytes). No more no less.
I have NICs here, reaching 20Gbp on a single TCP flow, were the pull is done.
Something else is going on.
Quoted 55 lines · Eric Dumazet
I dont think so : a total of 52 bytes are pulled.
(20 bytes for IPv4 header, 32 bytes for TCP header (TS))
Quoted 49 lines · Markus Stockhausen
Am 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:
Quoted 16 lines
Please try :
drivers/infiniband/ulp/ipoib/ipoib_ib.c +2
-0@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv, if (ipoib_ud_need_sg(priv->max_ib_mtu)) {skb_frag_t *frag = &skb_shinfo (skb)->frags[0 ];unsigned int size; + +prefetch (page_address (frag->page.p));/* * There is only two buffers needed for max_payload = 4K, Quoted 1 line
* first buf size is
IPOIB_UD_HEAD_SIZEThanks to both of you for spending time on this.
Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside
pskb_may_pull()just in front of__pskb_pull_tail()to give an idea what happens.During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:
[ 52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes [ 52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes [ 52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes [ 52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytesNow I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.
Any ideas?
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: IPoIB header prefetch for fragmented SKB
In reply to a message not in the archiveQuoted 1 line · Eric Dumazet
+ prefetch(page_address(frag->page.p));
Would it also make sense to do a
prefetchw(skb->data);
here? (That's where we'll copy the header, right?)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAL1RGDVcH666eAkyeYrReg_xY2+L0GVDzvwiVrrrNp9Vba2jvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: IPoIB header prefetch for fragmented SKB
Almost no driver care to do that, but you could try anyway ;)
Quoted 8 lines · Roland Dreier
Quoted 1 line · Eric Dumazet
+ prefetch(page_address(frag->page.p));
Would it also make sense to do a
prefetchw(skb->data);here? (That's where we'll copy the header, right?)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html