IPoIB: Leave space in skb linear buffer for IP headers
From: Roland Dreier <[email protected]>
Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing memcpy() in __pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.
We're already allocating extra tailroom for the skb linear part, so we might as well use it.
--- drivers/infiniband/ulp/ipoib/ipoib.h | 3 ++- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 12 +++--------- 2 files changed, 5 insertions(+), 10 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +2 -1
@@ -64,7 +64,8 @@ enum ipoib_flush_level {enum {IPOIB_ENCAP_LEN =4 , -IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +IPOIB_ENCAP_LEN , +/* add 128 bytes of tailroom for IP/TCP headers */ +IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +IPOIB_ENCAP_LEN +128 ,IPOIB_UD_RX_SG =2 ,/* max buffer needed for 4K mtu */ IPOIB_CM_MTU =0x10000 -0x10 ,/* padding to align header to 16 */
drivers/infiniband/ulp/ipoib/ipoib_ib.c +3 -9
@@ -156,18 +156,12 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)struct ipoib_dev_priv *priv =netdev_priv (dev);struct sk_buff *skb;int buf_size; -int tailroom;u64 *mapping; -if (ipoib_ud_need_sg (priv->max_ib_mtu)) { - buf_size =IPOIB_UD_HEAD_SIZE ; - tailroom =128 ;/* reserve some tailroom for IP/TCP headers */ - }else { - buf_size =IPOIB_UD_BUF_SIZE (priv->max_ib_mtu); - tailroom =0 ; - } + buf_size =ipoib_ud_need_sg (priv->max_ib_mtu) ? +IPOIB_UD_HEAD_SIZE :IPOIB_UD_BUF_SIZE (priv->max_ib_mtu); - skb =dev_alloc_skb (buf_size + tailroom +4 ); + skb =dev_alloc_skb (buf_size +4 );if (unlikely (!skb))return NULL ;
Signature · 6 lines
1.8.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1365099482-30287-1-git-send-email-roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Quoted 26 lines
From: Roland Dreier <[email protected]>
Markus Stockhausen <[email protected]> noticed that IPoIB was spending significant time doing
memcpy()in__pskb_pull_tail(). He
found that this is because his adapter reports a maximum MTU of 4K,
which causes IPoIB datagram mode to receive all the actual data in a separate page in the fragment list.We're already allocating extra tailroom for the skb linear part, so we might as well use it.
Cc Eric Dumazet <[email protected]>Reported-by Markus Stockhausen <[email protected]>Signed-off-by Roland Dreier <[email protected]>---
drivers/infiniband/ulp/ipoib/ipoib.h| 3 ++-drivers/infiniband/ulp/ipoib/ipoib_ib.c| 12 +++---------2 files changed, 5 insertions(+), 10 deletions(-)drivers/infiniband/ulp/ipoib/ipoib.h +0
-0
drivers/infiniband/ulp/ipoib/ipoib.h enum ipoib_flush_level { · line 68
Quoted 7 lines
drivers/infiniband/ulp/ipoib/ipoib.h +2
-1@@ -64,7 +64,8 @@ enum ipoib_flush_level {enum {IPOIB_ENCAP_LEN =4 , -IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +IPOIB_ENCAP_LEN , +/* add 128 bytes of tailroom for IP/TCP headers */ +IPOIB_UD_HEAD_SIZE =IB_GRH_BYTES +IPOIB_ENCAP_LEN +128 ,
Thanks for the help but I guess the patch is not yet perfect.
My (remote) test machine stopped responding after loading the new ipoib module. Tomorrow I can check the console. Having a look at the source code I guess we now have some major problems when receiving small packets:
...ipoib_ud_skb_put_frags(..., unsigned int length)
...
size = length - IPOIB_UD_HEAD_SIZE; /* may be less than zero! */ skb_frag_size_set(frag, size);
...
Markus
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html