getting warn once around skb_try_coalesce
Subsystems (3)
Hi Dave, Eric,
Another trace that I see here with net-next is this one-time warning. I get it always on the passive side of TCP, something that seems related to GRO, it happens only with IPoIB, not with mlx4_en and igb (when igb get to work on net-next...)
The latest commit in this area is bad43ca8325f493dcaa0896c2f036276af059c7e "net: introduce skb_try_coalesce()" from Eric.
Or.
-----------[ cut here ]------------
WARNING: at net/core/skbuff.c:3413 skb_try_coalesce+0x1f8/0x31d() Hardware name: X7DWU Modules linked in: drbd lru_cache cn autofs4 sunrpc 8021q ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash dm_log dm_round_robin dm_multipath uinput mlx4_ib ib_mad ib_core mlx4_en mlx4_core igb sg joydev kvm microcode pcspkr rng_core ioatdma dm_mod dca floppy shpchp button sr_mod ext3 jbd usb_storage sd_mod ata_piix libata scsi_mod ehci_hcd uhci_hcd [last unloaded: scsi_wait_scan] Pid: 0, comm: swapper/1 Tainted: G I 3.5.0-rc1-00107-gf5bae8a-dirty #57 Call Trace: <IRQ> [<ffffffff8102ab65>] warn_slowpath_common+0x80/0x98 [<ffffffff8102ab92>] warn_slowpath_null+0x15/0x17 [<ffffffff812c5a73>] skb_try_coalesce+0x1f8/0x31d [<ffffffff8130a6ad>] tcp_try_coalesce+0x4c/0xa0 [<ffffffff8130a759>] tcp_queue_rcv+0x58/0xe1 [<ffffffff8130d4ca>] tcp_data_queue+0x1bd/0xa8d [<ffffffff8130ecba>] tcp_rcv_established+0x646/0x6fc [<ffffffff81314fd7>] ? tcp_v4_rcv+0x427/0xa1b [<ffffffff81314892>] tcp_v4_do_rcv+0xd8/0x3f6 [<ffffffff8136aefb>] ? _raw_spin_lock_nested+0x41/0x48 [<ffffffff813151a5>] tcp_v4_rcv+0x5f5/0xa1b [<ffffffff812f8626>] ip_local_deliver_finish+0x1a1/0x2b2 [<ffffffff812f84ba>] ? ip_local_deliver_finish+0x35/0x2b2 [<ffffffff812f87a9>] ip_local_deliver+0x72/0x79 [<ffffffff812f820d>] ip_rcv_finish+0x399/0x3b1 [<ffffffff812f845f>] ip_rcv+0x23a/0x260 [<ffffffff812cd086>] __netif_receive_skb+0x3b2/0x41b [<ffffffff812cce0e>] ? __netif_receive_skb+0x13a/0x41b [<ffffffff812ce93c>] netif_receive_skb+0xee/0xf7 [<ffffffff81322512>] ? inet_compat_ioctl+0x1e/0x1e [<ffffffff812ceb90>] napi_gro_complete+0x133/0x140 [<ffffffff812ceaab>] ? napi_gro_complete+0x4e/0x140 [<ffffffff812ced3d>] dev_gro_receive+0x1a0/0x2fb [<ffffffff812cec19>] ? dev_gro_receive+0x7c/0x2fb [<ffffffff812cf1c5>] napi_gro_receive+0x105/0x11e [<ffffffffa02ed6d4>] ipoib_ib_handle_rx_wc+0x243/0x277 [ib_ipoib] [<ffffffffa02ee84e>] ipoib_poll+0xa9/0x12d [ib_ipoib] [<ffffffff812cf355>] net_rx_action+0xc1/0x1ee [<ffffffff81031e4a>] __do_softirq+0xff/0x1de [<ffffffff813735cc>] call_softirq+0x1c/0x30 [<ffffffff81003174>] do_softirq+0x38/0x80 [<ffffffff81031b23>] irq_exit+0x4e/0x83 [<ffffffff810029dd>] do_IRQ+0x98/0xaf [<ffffffff8136b92c>] common_interrupt+0x6c/0x6c <EOI> [<ffffffff8100850c>] ? mwait_idle+0x13c/0x208 [<ffffffff81008503>] ? mwait_idle+0x133/0x208 [<ffffffff810089f1>] cpu_idle+0x6e/0xab [<ffffffff81363763>] start_secondary+0x1b9/0x1bd ---[ end trace fdf1b0e917b37732 ]---
Quoted 8 lines · Or Gerlitz
Hi Dave, Eric,
Another trace that I see here with net-next is this one-time warning. I get it always
on the passive side of TCP, something that seems related to GRO, it happens only with
IPoIB, not withmlx4_enand igb (when igb get to work on net-next...)
Quoted 7 lines
The latest commit in this area is
bad43ca8325f493dcaa0896c2f036276af059c7e"net: introduceskb_try_coalesce()" from Eric.Or.
-----------[ cut here ]------------
WARNING: at net/core/skbuff.c:3413 skb_try_coalesce+0x1f8/0x31d()
This warning catch skb truesize offenders, most probably its a driver issue.
Quoted 12 lines · Eric Dumazet
Quoted 11 lines · Or Gerlitz
Hi Dave, Eric,
Another trace that I see here with net-next is this one-time warning. I get it always
on the passive side of TCP, something that seems related to GRO, it happens only with
IPoIB, not withmlx4_enand igb (when igb get to work on net-next...)The latest commit in this area is
bad43ca8325f493dcaa0896c2f036276af059c7e"net: introduceskb_try_coalesce()" from Eric.
Quoted 8 lines
Quoted 4 lines
Or.
-----------[ cut here ]------------
WARNING: at net/core/skbuff.c:3413 skb_try_coalesce+0x1f8/0x31d()This warning catch skb truesize offenders, most probably its a driver issue.
By the way, this driver allocates not enough tailroom in skbs, so IP/TCP stacks need to reallocate skb head to pull IP/TCP headers. Thats not efficient.
I suggest using following patch :
drivers/infiniband/ulp/ipoib/ipoib_ib.c +1 -1
@@ -159,7 +159,7 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)u64 *mapping;if (ipoib_ud_need_sg (priv->max_ib_mtu)) - buf_size =IPOIB_UD_HEAD_SIZE ; + buf_size =IPOIB_UD_HEAD_SIZE +128 ;/* reserve some tailroom for IP/TCP headers */ else buf_size =IPOIB_UD_BUF_SIZE (priv->max_ib_mtu);
Quoted 22 lines · Eric Dumazet
Quoted 12 lines · Eric Dumazet
Quoted 11 lines · Or Gerlitz
Hi Dave, Eric,
Another trace that I see here with net-next is this one-time warning. I get it always
on the passive side of TCP, something that seems related to GRO, it happens only with
IPoIB, not withmlx4_enand igb (when igb get to work on net-next...)The latest commit in this area is
bad43ca8325f493dcaa0896c2f036276af059c7e"net: introduceskb_try_coalesce()" from Eric.Quoted 8 lines
Quoted 4 lines
Or.
-----------[ cut here ]------------
WARNING: at net/core/skbuff.c:3413 skb_try_coalesce+0x1f8/0x31d()This warning catch skb truesize offenders, most probably its a driver issue.
Quoted 5 lines
By the way, this driver allocates not enough tailroom in skbs, so IP/TCP stacks need to reallocate skb head to pull IP/TCP headers. Thats not efficient.
I suggest using following patch :
And of course we also can fix the truesize bug. (Not sure it will fix the warning, but worth trying)
Since this driver allocates a full page, it must use the PAGE_SIZE, not the used part in the fragment
drivers/infiniband/ulp/ipoib/ipoib_ib.c +2 -2
@@ -123,7 +123,7 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv,skb_frag_size_set (frag, size); skb->data_len += size; - skb->truesize += size; + skb->truesize +=PAGE_SIZE ; }else skb_put (skb, length);
@@ -159,7 +159,7 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)u64 *mapping;if (ipoib_ud_need_sg (priv->max_ib_mtu)) - buf_size =IPOIB_UD_HEAD_SIZE ; + buf_size =IPOIB_UD_HEAD_SIZE +128 ;/* reserve some tailroom for IP/TCP headers */ else buf_size =IPOIB_UD_BUF_SIZE (priv->max_ib_mtu);
Quoted 1 line · Eric Dumazet
And of course we also can fix the truesize bug. (Not sure it will fix the warning, but worth trying)
thanks, we will give that a try and let you know
Shlomo Pongartz
AppliedQuoted 57 lines · Eric Dumazet
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Or Gerlitz
Hi Dave, Eric,
Another trace that I see here with net-next is this one-time warning. I get it always
on the passive side of TCP, something that seems related to GRO, it happens only with
IPoIB, not withmlx4_enand igb (when igb get to work on net-next...)The latest commit in this area is
bad43ca8325f493dcaa0896c2f036276af059c7e"net: introduceskb_try_coalesce()" from Eric.Or.
-----------[ cut here ]------------
WARNING: at net/core/skbuff.c:3413 skb_try_coalesce+0x1f8/0x31d()Quoted 8 lines
Quoted 3 lines
This warning catch skb truesize offenders, most probably its a driver issue.
By the way, this driver allocates not enough tailroom in skbs, so IP/TCP stacks need to reallocate skb head to pull IP/TCP headers. Thats not efficient.
I suggest using following patch :
And of course we also can fix the truesize bug. (Not sure it will fix the warning, but worth trying)
Since this driver allocates a full page, it must use the
PAGE_SIZE, not the used part in the fragmentdrivers/infiniband/ulp/ipoib/ipoib_ib.c +2
-2@@ -123,7 +123,7 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv,skb_frag_size_set (frag, size); skb->data_len += size; - skb->truesize += size; + skb->truesize +=PAGE_SIZE ; }else skb_put (skb, length);@@ -159,7 +159,7 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)u64 *mapping;if (ipoib_ud_need_sg (priv->max_ib_mtu)) - buf_size =IPOIB_UD_HEAD_SIZE ; + buf_size =IPOIB_UD_HEAD_SIZE +128 ;/* reserve some tailroom for IP/TCP headers */ else buf_size =IPOIB_UD_BUF_SIZE (priv->max_ib_mtu);
Quoted 2 lines
.
Hi,
I've applied the patch and there are no more warnings. Thanks.
Can you please elaborate on this issue which was there from day one and AFAIK never manifested itself.
Best regards,
S.P.
Quoted 3 lines · Shlomo Pongartz
I've applied the patch and there are no more warnings. Thanks. Can you please elaborate on this issue which was there from day one and AFAIK never manifested itself.
two problems :
1) truesize underestimation
Well, I posted at least 50 patches related to various skb->truesize mismatches in the past year.
skb->truesize is/should_be the true size of skb, that is the memory allocated for sk_buff, skb->head and all fragments
Check commit e1ac50f64691de9a (bnx2x: fix skb truesize underestimation) for a similar fix done on bnx2x
Its very important to do so to avoid OOM.
If you account few bytes per fragment but allocate PAGE_SIZE bytes, its pretty easy to allocate far more memory than allowed by various socket/tcp/udp/... limits, and exhaust kernel memory.
commit 924a4c7d2e962b (myri10ge: fix truesize underestimation) commit 7b8b59617ead5acc (igbvf: fix truesize underestimation)
I probably missed your driver because it was not on drivers/net tree but drivers/infiniband
2) Not enough tailroom
Invisible but performance suffers, because IP/TCP will need
to call pskb_may_pull() and the expensive __pskb_pull_tail().
So each incoming IP packet needs at least one pskb_expand_head().
I'll send an official patch, but I believe I should refine the tailroom allocation like that :
drivers/infiniband/ulp/ipoib/ipoib_ib.c +8 -4
@@ -123,7 +123,7 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv,skb_frag_size_set (frag, size); skb->data_len += size; - skb->truesize += size; + skb->truesize +=PAGE_SIZE ; }else skb_put (skb, length);
@@ -156,14 +156,18 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)struct ipoib_dev_priv *priv =netdev_priv (dev);struct sk_buff *skb;int buf_size; +int tailroom;u64 *mapping; -if (ipoib_ud_need_sg (priv->max_ib_mtu)) +if (ipoib_ud_need_sg (priv->max_ib_mtu)) { buf_size =IPOIB_UD_HEAD_SIZE ; -else + tailroom =128 ;/* reserve some tailroom for IP/TCP headers */ + }else { buf_size =IPOIB_UD_BUF_SIZE (priv->max_ib_mtu); + tailroom =0 ; + } - skb =dev_alloc_skb (buf_size +4 ); + skb =dev_alloc_skb (buf_size + tailroom +4 );if (unlikely (!skb))return NULL ;