tcp: Don't collapse if resulting skb could overflow skb->csum_start
Subsystems (3)
PATCH 1/1 tcp: Don't collapse if resulting skb could overflow skb->csum_start
If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start which is based on the start of the headroom. It has been observed rarely in the wild with IPoIB due to the 64K MTU.
With this patch, the overflow has not been observed for over a week while previously it would occur within ~ 1 day.
A big thank you to Jim Foraker <[email protected]> and the team at LLNL for helping out with the investigation and testing.
--- net/ipv4/tcp_output.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/ipv4/tcp_output.c +6 -0
@@ -2305,6 +2305,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,if (after (TCP_SKB_CB (skb)->end_seq,tcp_wnd_end (tp)))break ; +/* Never collapse if the resulting headroom + data exceeds +* 64K as that is the maximum csum_start can cover. +*/ +if (skb_headroom (to) + to->len + skb->len >0xFFFF ) +break ; +tcp_collapse_retrans (sk, to); } }
Quoted 25 lines · Thomas Graf
If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit
skb->csum_startwhich is based on the start of the headroom. It has been observed rarely in the wild with IPoIB due to the 64K MTU.With this patch, the overflow has not been observed for over a week while previously it would occur within ~ 1 day.
A big thank you to Jim Foraker <[email protected]> and the team at LLNL for helping out with the investigation and testing.
Reported-by Jim Foraker <[email protected]>Signed-off-by Thomas Graf <[email protected]>--- net/ipv4/tcp_output.c | 6 ++++++ 1 file changed, 6 insertions(+)net/ipv4/tcp_output.c +0
-0@@ -2305,6 +2305,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,if (after (TCP_SKB_CB (skb)->end_seq,tcp_wnd_end (tp)))break ;
net/ipv4/tcp_output.c static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to, · line 2316
Quoted 10 lines
net/ipv4/tcp_output.c +6
-0+/* Never collapse if the resulting headroom + data exceeds +* 64K as that is the maximum csum_start can cover. +*/ +if (skb_headroom (to) + to->len + skb->len >0xFFFF ) +break ; +tcp_collapse_retrans (sk, to); } }
but.... what is the value of skb_availroom(to) ?
The earlier test at line 2302 should already guard this case ?
/* Punt if not enough space exists in the first SKB for
* the data in the second
*/
if (skb->len > skb_availroom(to))
break;Quoted 9 lines · Eric Dumazet
but.... what is the value of
skb_availroom(to)?The earlier test at line 2302 should already guard this case ?
/* Punt if not enough space exists in the first SKB for * the data in the second */ if (skb->len > skb_availroom(to)) break;
Only if it is guaranteed that we never see an MSS > 64K.
Assuming that is true forever, then a21d45726 (tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra room kmalloc() might have given us in __alloc_skb() for collapsing.
Was that an intentional side effect of a21d45726?
Quoted 11 lines · Thomas Graf
Quoted 9 lines · Eric Dumazet
but.... what is the value of
skb_availroom(to)?The earlier test at line 2302 should already guard this case ?
/* Punt if not enough space exists in the first SKB for * the data in the second */ if (skb->len > skb_availroom(to)) break;
Quoted 8 lines
Only if it is guaranteed that we never see an MSS > 64K.
Assuming that is true forever, then
a21d45726(tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra roomkmalloc()might have given us in__alloc_skb()for collapsing.Was that an intentional side effect of
a21d45726?
This had a followup with 22b4a4f22da4
(tcp: fix retransmit of partially acked frames)
I also wonder if there is not another similar potential problem in __tcp_retransmit_skb) after call to
tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)
csum_start can overflow again because of MAX_TCP_HEADER headroom reserve.
So maybe we should limit TCP MTU to (64K - MAX_TCP_HEADER)
Quoted 27 lines · Eric Dumazet
Quoted 11 lines · Thomas Graf
Quoted 9 lines · Eric Dumazet
but.... what is the value of
skb_availroom(to)?The earlier test at line 2302 should already guard this case ?
/* Punt if not enough space exists in the first SKB for * the data in the second */ if (skb->len > skb_availroom(to)) break;Quoted 8 lines
Only if it is guaranteed that we never see an MSS > 64K.
Assuming that is true forever, then
a21d45726(tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra roomkmalloc()might have given us in__alloc_skb()for collapsing.Was that an intentional side effect of
a21d45726?This had a followup with
22b4a4f22da4
(tcp: fix retransmit of partially acked frames)I also wonder if there is not another similar potential problem in
__tcp_retransmit_skb) after call to
Quoted 7 lines
tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)
csum_startcan overflow again because ofMAX_TCP_HEADERheadroom reserve.So maybe we should limit TCP MTU to (64K -
MAX_TCP_HEADER)
Or adapt the test at line 2390 in net/ipv4/tcp_output.c
to force in the case there could be an overflow a :
struct sk_buff *nskb = __pskb_copy()
So that we have a new skb with minimal headroom.
Quoted 15 lines · Eric Dumazet
Quoted 6 lines · Eric Dumazet
This had a followup with
22b4a4f22da4
(tcp: fix retransmit of partially acked frames)I also wonder if there is not another similar potential problem in
__tcp_retransmit_skb) after call toQuoted 7 lines
tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)
csum_startcan overflow again because ofMAX_TCP_HEADERheadroom reserve.So maybe we should limit TCP MTU to (64K -
MAX_TCP_HEADER)
Quoted 7 lines
Or adapt the test at line 2390 in
net/ipv4/tcp_output.cto force in the case there could be an overflow a :
struct
sk_buff*nskb =__pskb_copy()So that we have a new skb with minimal headroom.
I would favour the __pskb_copy() variant as that would be more generic. I'll redo the patch.