Skip to contentSkip to conversation

IPoIB: Leave space in skb linear buffer for IP headers

PATCH In review Roland Dreier 3 participants 14 messages Apr 16, 2013 – Apr 20, 2013 10 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 IPoIB: Leave space in skb linear buffer for IP headers

From: Roland Dreier <[email protected]>

Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing memcpy() in __pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.

We're already allocating extra tailroom for the skb linear part, so we might as well use it. In fact, we might as well allocate a big enough linear part so that all the data fits there in the relatively common case of a 2K IB MTU, and only use a fragment page for 4K IB MTU.

---
v4: Leave enough space in linear part of skb so that all data ends up there with 2K IB MTU. Still not sure how this affects perf with a 4K IB MTU (should be better since we avoid pulling IP headers out of first fragment).

 drivers/infiniband/ulp/ipoib/ipoib.h    |  6 ++-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c | 78 +++++++++++++++++++--------------
 2 files changed, 49 insertions(+), 35 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +4 -2
index eb71aaa..7a56a8e 100644
@@ -64,8 +64,9 @@ enum ipoib_flush_level {
 enum {
 	IPOIB_ENCAP_LEN		  = 4,
 
-	IPOIB_UD_HEAD_SIZE	  = IB_GRH_BYTES + IPOIB_ENCAP_LEN,
-	IPOIB_UD_RX_SG		  = 2, /* max buffer needed for 4K mtu */
+	/* enough so w/ 2K mtu, everything is in linear part of skb */
+	IPOIB_UD_HEAD_SIZE	  = IB_GRH_BYTES + 2048,
+	IPOIB_UD_RX_SG		  = 2, /* max buffer needed for 4K mtu w/ 4K pages */
 
 	IPOIB_CM_MTU		  = 0x10000 - 0x10, /* padding to align header to 16 */
 	IPOIB_CM_BUF_SIZE	  = IPOIB_CM_MTU  + IPOIB_ENCAP_LEN,
@@ -155,6 +156,7 @@ struct ipoib_mcast {
 
 struct ipoib_rx_buf {
 	struct sk_buff *skb;
+	struct page    *page;
 	u64		mapping[IPOIB_UD_RX_SG];
 };
 
drivers/infiniband/ulp/ipoib/ipoib_ib.c +45 -33

173 lines not on this page · raw message

Signature · 6 lines
1.8.1.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1366129801-29234-1-git-send-email-roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#2 lore raw
Quoted 13 lines · Roland Dreier

From: Roland Dreier <[email protected]>

Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing memcpy() in __pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.

We're already allocating extra tailroom for the skb linear part, so we might as well use it. In fact, we might as well allocate a big enough linear part so that all the data fits there in the relatively common case of a 2K IB MTU, and only use a fragment page for 4K IB MTU.

Quoted 9 lines

---
v4: Leave enough space in linear part of skb so that all data ends up there with 2K IB MTU. Still not sure how this affects perf with a 4K IB MTU (should be better since we avoid pulling IP headers out of first fragment).

I am afraid I don't understand what the issue is.

the pull_tail() in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers in skb->head , and nobody noticed.

Real cost is the cache line miss.

Now, if you pull too many bytes in skb->head, say part of TCP payload, you lose opportunities in TCP coalescing or splice().
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CANn89iK9Z7i59+pHXAXJzgYZxKB6OptR-+TRwfjTqNwHMLDOow-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#4 lore raw
Quoted 7 lines · Eric Dumazet

I am afraid I don't understand what the issue is.

the pull_tail() in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers in skb->head , and nobody noticed.

Real cost is the cache line miss.

According to Markus there is a pretty huge performance penalty if the whole packet (TCP/IP headers + payload) ends up in the fragment, and stack has to memcpy headers into the linear part.

Quoted 2 lines

Now, if you pull too many bytes in skb->head, say part of TCP payload, you lose opportunities in TCP coalescing or splice().

So it sounds like this patch is a bad change from that POV, since it would cause us to have lots of TCP payload in the linear part and then second half of the payload in the fragments (if we are running with 4K IPoIB MTU).

Seems like the real fix is to enable the header separation feature in adapters that support it (ConnectX-2 and -3) so that exactly TCP/IP headers go in linear part and payload goes in the fragment. That's a bit more involved, but I'll look into doing it.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to a message not in the archive
Quoted 11 lines

I am afraid I don't understand what the issue is.

the pull_tail() in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers in skb->head , and nobody noticed.

Real cost is the cache line miss.

Now, if you pull too many bytes in skb->head, say part of TCP payload, you lose opportunities in TCP coalescing or splice().

With patch v4 netperf and NFS receive performance raises to the expected values. As I'm no expert in this I can only repost the initial performance report that started the whole discussion. __pskb_pull_tail consumes a lot time on our XEON L5420 test server.

...

- server side: netserver -p 12345
- client side: netperf -H <server_ip> -p 12345 -l 120

Analysis was performed on the server side with
- perf record -a -g sleep 10
- perf report

# Overhead Symbol

# ........  .............................................
#
    19.67%  [k] copy_user_generic_string
            |
            |--99.74%-- skb_copy_datagram_iovec
            |          tcp_recvmsg
            |          inet_recvmsg
            |          sock_recvmsg
            |          sys_recvfrom
            |          system_call_fastpath
            |          recv
            |          |
            |          |--50.17%-- 0x7074656e00667265
            |          |
            |           --49.83%-- 0x6672657074656e
             --0.26%-- [...]
     7.38%  [k] memcpy
            |
            |--84.56%-- __pskb_pull_tail
            |          |
            |          |--81.88%-- pskb_may_pull.part.6
            |          |          skb_gro_header_slow
            |          |          inet_gro_receive
            |          |          dev_gro_receive
            |          |          napi_gro_receive
            |          |          ipoib_ib_handle_rx_wc
            |          |          ipoib_poll
            |          |          net_rx_action
            |          |          __do_softirq

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CD935857.5873%[email protected]

#7 lore raw
Quoted 9 lines · Markus Stockhausen
Quoted 9 lines

I am afraid I don't understand what the issue is.

the pull_tail() in itself is not a performance issue : Intel guys only fixed last gays ago fact that IGB/IXGBE drivers were not pulling tcp headers in skb->head , and nobody noticed.

Real cost is the cache line miss.

Quoted 9 lines
Quoted 2 lines

Now, if you pull too many bytes in skb->head, say part of TCP payload, you lose opportunities in TCP coalescing or splice().

With patch v4 netperf and NFS receive performance raises to the expected values. As I'm no expert in this I can only repost the initial performance report that started the whole discussion. __pskb_pull_tail consumes a lot time on our XEON L5420 test server.

That's probably because of a cache line miss.

The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.

So what does this adapter in this respect ?

I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.

Many drivers use prefetch() to make sure cpu starts to bring this cache line into cache as soon as possible.

A single prefetch() call at the right place might help a lot. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CANn89iLopRHCMhYc+BcPvhZJtfL53H3T6JdP=ssfdg=oxh+Waw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#9 lore raw

IPoIB header prefetch for fragmented SKB

Quoted 10 lines

That's probably because of a cache line miss.

The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.

So what does this adapter in this respect ?

Quoted 7 lines

I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.

Many drivers use prefetch() to make sure cpu starts to bring this cache line into cache as soon as possible.

A single prefetch() call at the right place might help a lot.

Hello,

@Eric: Thanks for the tip.

In the 4K MAX MTU IPoIB driver path ipoib_ib_handle_rx_wc() will produce an empty skb linear part with the whole data placed into the first fragment. napi_gro_receive() finally pulls the IP header out of the fragment into the linear part.

As far as I understand the pull out of the fragment should come without additional cost when one calls a prefetch "long" before the skb_pull().

I'm willing to check this out but I'm unsure if the IP header is aligned to a cache line of 64 bytes. As a simple guess I would implement the prefetch here:

static void ipoib_ib_handle_rx_wc();
...
skb_pull (skb, IB_GRH_BYTES);

skb->protocol = ((struct ipoib_header *) skb->data)->proto; skb_reset_mac_header(skb);
skb_pull(skb, IPOIB_ENCAP_LEN);
+
+ if (ipoib_ud_need_sg(priv->max_ib_mtu))
+ prefetch(<whatever address>);
...

Can you give me a hint what address one should put into the call?

Thanks in advance.

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CD94B1E6.588B%[email protected]

#11 lore raw

Re: IPoIB header prefetch for fragmented SKB

Please try :

(quoted diff) +0 -0

b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 2cfa76f..9bdff21 100644

drivers/infiniband/ulp/ipoib/ipoib_ib.c +0 -0
@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct

ipoib_dev_priv *priv,

        if (ipoib_ud_need_sg(priv->max_ib_mtu)) {
                skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
                unsigned int size;

+
+ prefetch(page_address(frag->page.p));
/*
* There is only two buffers needed for max_payload = 4K,
* first buf size is IPOIB_UD_HEAD_SIZE

Quoted 47 lines · Markus Stockhausen
Quoted 10 lines

That's probably because of a cache line miss.

The thing I don't really understand is that normally, the first cache line (64 bytes) contains both the Ethernet header and IPv4 header.

So what does this adapter in this respect ?

Quoted 7 lines

I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole cache line.

Many drivers use prefetch() to make sure cpu starts to bring this cache line into cache as soon as possible.

A single prefetch() call at the right place might help a lot.

Hello,

@Eric: Thanks for the tip.

In the 4K MAX MTU IPoIB driver path ipoib_ib_handle_rx_wc() will produce an empty skb linear part with the whole data placed into the first fragment. napi_gro_receive() finally pulls the IP header out of the fragment into the linear part.

As far as I understand the pull out of the fragment should come without additional cost when one calls a prefetch "long" before the skb_pull().

I'm willing to check this out but I'm unsure if the IP header is aligned to a cache line of 64 bytes. As a simple guess I would implement the prefetch here:

static void ipoib_ib_handle_rx_wc();
...
skb_pull (skb, IB_GRH_BYTES);

  skb->protocol = ((struct ipoib_header *) skb->data)->proto;
  skb_reset_mac_header(skb);
  skb_pull(skb, IPOIB_ENCAP_LEN);
(quoted diff) +3 -0
+
+ if (ipoib_ud_need_sg(priv->max_ib_mtu))
+   prefetch(<whatever address>);
...
Quoted 7 lines

Can you give me a hint what address one should put into the call?

Thanks in advance.

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CANn89iLwF9LUohz_c765c4b+6DC0gTH77bN5QzbAkR-4LCcUog-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#13 lore raw

Re: IPoIB header prefetch for fragmented SKB

Quoted 1 line · Eric Dumazet

+ prefetch(page_address(frag->page.p));

I guess we would have to change our allocation to __get_free_page() (instead of alloc_page()), so that we make sure never to get a highmem page?

Other than that, seems fine.

By the way, is there any value to header splitting -- having the adapter put the TCP/IP headers in the linear part and the payload in the fragment page? How about in the case where we have a small enough MTU that we could put the whole packet in the linear part?

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAL1RGDWVOPsEe+92sgcz+fDWj8=02CJt40889_cgsQKBX6XJ4A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#15 lore raw

Re: IPoIB header prefetch for fragmented SKB

alloc_page(GFP_ATOMIC) is safe. it doesnt give you a highmempage

Not sure about header splitting. If done, it probably should be smart as you said (allowing small frames to be in the first part)

Quoted 7 lines · Roland Dreier
Quoted 1 line · Eric Dumazet

+ prefetch(page_address(frag->page.p));

I guess we would have to change our allocation to __get_free_page() (instead of alloc_page()), so that we make sure never to get a highmem page?

Quoted 8 lines

Other than that, seems fine.

By the way, is there any value to header splitting -- having the adapter put the TCP/IP headers in the linear part and the payload in the fragment page? How about in the case where we have a small enough MTU that we could put the whole packet in the linear part?

- R.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw

Re: IPoIB header prefetch for fragmented SKB

In reply to a message not in the archive

Am 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:

Quoted 16 lines

Please try :

drivers/infiniband/ulp/ipoib/ipoib_ib.c +2 -0
index 2cfa76f..9bdff21 100644
@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct
 ipoib_dev_priv *priv,
       if (ipoib_ud_need_sg(priv->max_ib_mtu)) {
               skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
               unsigned int size;
+
+               prefetch(page_address(frag->page.p));
               /*
                * There is only two buffers needed for max_payload = 4K,
Quoted 1 line

* first buf size is IPOIB_UD_HEAD_SIZE

Thanks to both of you for spending time on this.

Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside pskb_may_pull() just in front of __pskb_pull_tail() to give an idea what happens.

During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:

[   52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
[   52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes

Now I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.

Any ideas?

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CD975158.5900%[email protected]

#18 lore raw

Re: IPoIB header prefetch for fragmented SKB

I dont think so : a total of 52 bytes are pulled.

(20 bytes for IPv4 header, 32 bytes for TCP header (TS))

Quoted 39 lines · Markus Stockhausen

Am 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:

Quoted 16 lines

Please try :

drivers/infiniband/ulp/ipoib/ipoib_ib.c +2 -0
index 2cfa76f..9bdff21 100644
@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct
 ipoib_dev_priv *priv,
       if (ipoib_ud_need_sg(priv->max_ib_mtu)) {
               skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
               unsigned int size;
+
+               prefetch(page_address(frag->page.p));
               /*
                * There is only two buffers needed for max_payload = 4K,
Quoted 1 line

* first buf size is IPOIB_UD_HEAD_SIZE

Thanks to both of you for spending time on this.

Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside pskb_may_pull() just in front of __pskb_pull_tail() to give an idea what happens.

During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:

[   52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
[   52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
Quoted 10 lines

Now I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.

Any ideas?

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CANn89iL8_sD5m-DcrwVuJ6-Bi=tCOof4g2DLi-5Wxcz62rv0iA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#20 lore raw

Re: IPoIB header prefetch for fragmented SKB

This pulls cannot explain a major performance problem.

Its a memcpy( ... 20 bytes). No more no less.

I have NICs here, reaching 20Gbp on a single TCP flow, were the pull is done.

Something else is going on.

Quoted 55 lines · Eric Dumazet

I dont think so : a total of 52 bytes are pulled.

(20 bytes for IPv4 header, 32 bytes for TCP header (TS))

Quoted 49 lines · Markus Stockhausen

Am 17.04.13 21:10 schrieb "Eric Dumazet" unter <[email protected]>:

Quoted 16 lines

Please try :

drivers/infiniband/ulp/ipoib/ipoib_ib.c +2 -0
index 2cfa76f..9bdff21 100644
@@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct
 ipoib_dev_priv *priv,
       if (ipoib_ud_need_sg(priv->max_ib_mtu)) {
               skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
               unsigned int size;
+
+               prefetch(page_address(frag->page.p));
               /*
                * There is only two buffers needed for max_payload = 4K,
Quoted 1 line

* first buf size is IPOIB_UD_HEAD_SIZE

Thanks to both of you for spending time on this.

Sorry to tell you that the above patch does not give any performance improvements. So I put a debug message inside inside pskb_may_pull() just in front of __pskb_pull_tail() to give an idea what happens.

During a netserver/netperf run with an MTU of 2044 bytes I can read the following repeating messages:

[   52.909805] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909807] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909809] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes
[   52.909812] 0 bytes inline, 1988 bytes in fragment, pull 20 bytes
[   52.909814] 20 bytes inline, 1968 bytes in fragment, pull 40 bytes
[   52.909816] 40 bytes inline, 1948 bytes in fragment, pull 52 bytes

Now I'm getting lost somehow. From 2044 bytes 1988 are left in the fragment and 112 have to be pulled into the linear part. This is achieved by 3 steps. So we are far beyond the cache line size of 64 bytes.

Any ideas?

Markus

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAL1RGDVcH666eAkyeYrReg_xY2+L0GVDzvwiVrrrNp9Vba2jvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#23 lore raw

Re: IPoIB header prefetch for fragmented SKB

Almost no driver care to do that, but you could try anyway ;)

Quoted 8 lines · Roland Dreier
Quoted 1 line · Eric Dumazet

+ prefetch(page_address(frag->page.p));

Would it also make sense to do a

    prefetchw(skb->data);

here? (That's where we'll copy the header, right?)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html