Skip to contentSkip to conversation

tcp: Don't collapse if resulting skb could overflow skb->csum_start

PATCH In review Thomas Graf 2 participants 6 messages Feb 28, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tcp: Don't collapse if resulting skb could overflow skb->csum_start

If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start which is based on the start of the headroom. It has been observed rarely in the wild with IPoIB due to the 64K MTU.

With this patch, the overflow has not been observed for over a week while previously it would occur within ~ 1 day.

A big thank you to Jim Foraker <[email protected]> and the team at LLNL for helping out with the investigation and testing.

---
 net/ipv4/tcp_output.c | 6 ++++++
 1 file changed, 6 insertions(+)
net/ipv4/tcp_output.c +6 -0
index e2b4461..1902fee 100644
@@ -2305,6 +2305,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,
 		if (after(TCP_SKB_CB(skb)->end_seq, tcp_wnd_end(tp)))
 			break;
 
+		/* Never collapse if the resulting headroom + data exceeds
+		 * 64K as that is the maximum csum_start can cover.
+		 */
+		if (skb_headroom(to) + to->len + skb->len > 0xFFFF)
+			break;
+
 		tcp_collapse_retrans(sk, to);
 	}
 }
#1 lore raw
Quoted 25 lines · Thomas Graf

If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start which is based on the start of the headroom. It has been observed rarely in the wild with IPoIB due to the 64K MTU.

With this patch, the overflow has not been observed for over a week while previously it would occur within ~ 1 day.

A big thank you to Jim Foraker <[email protected]> and the team at LLNL for helping out with the investigation and testing.

---
 net/ipv4/tcp_output.c | 6 ++++++
 1 file changed, 6 insertions(+)
net/ipv4/tcp_output.c +0 -0
index e2b4461..1902fee 100644
@@ -2305,6 +2305,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,
 		if (after(TCP_SKB_CB(skb)->end_seq, tcp_wnd_end(tp)))
 			break;

net/ipv4/tcp_output.c static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to, · line 2316

Quoted 10 lines
net/ipv4/tcp_output.c +6 -0
 
+		/* Never collapse if the resulting headroom + data exceeds
+		 * 64K as that is the maximum csum_start can cover.
+		 */
+		if (skb_headroom(to) + to->len + skb->len > 0xFFFF)
+			break;
+
 		tcp_collapse_retrans(sk, to);
 	}
 }

but.... what is the value of skb_availroom(to) ?

The earlier test at line 2302 should already guard this case ?

               /* Punt if not enough space exists in the first SKB for
                 * the data in the second
                 */
                if (skb->len > skb_availroom(to))
                        break;
#2 lore raw
Quoted 9 lines · Eric Dumazet

but.... what is the value of skb_availroom(to) ?

The earlier test at line 2302 should already guard this case ?

               /* Punt if not enough space exists in the first SKB for
                 * the data in the second
                 */
                if (skb->len > skb_availroom(to))
                        break;

Only if it is guaranteed that we never see an MSS > 64K.

Assuming that is true forever, then a21d45726 (tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra room kmalloc() might have given us in __alloc_skb() for collapsing.

Was that an intentional side effect of a21d45726?

#3 lore raw
Quoted 11 lines · Thomas Graf
Quoted 9 lines · Eric Dumazet

but.... what is the value of skb_availroom(to) ?

The earlier test at line 2302 should already guard this case ?

               /* Punt if not enough space exists in the first SKB for
                 * the data in the second
                 */
                if (skb->len > skb_availroom(to))
                        break;
Quoted 8 lines

Only if it is guaranteed that we never see an MSS > 64K.

Assuming that is true forever, then a21d45726 (tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra room kmalloc() might have given us in __alloc_skb() for collapsing.

Was that an intentional side effect of a21d45726?

This had a followup with 22b4a4f22da4
(tcp: fix retransmit of partially acked frames)

I also wonder if there is not another similar potential problem in __tcp_retransmit_skb) after call to

tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)

csum_start can overflow again because of MAX_TCP_HEADER headroom reserve.

So maybe we should limit TCP MTU to (64K - MAX_TCP_HEADER)

#4 lore raw
Quoted 27 lines · Eric Dumazet
Quoted 11 lines · Thomas Graf
Quoted 9 lines · Eric Dumazet

but.... what is the value of skb_availroom(to) ?

The earlier test at line 2302 should already guard this case ?

               /* Punt if not enough space exists in the first SKB for
                 * the data in the second
                 */
                if (skb->len > skb_availroom(to))
                        break;
Quoted 8 lines

Only if it is guaranteed that we never see an MSS > 64K.

Assuming that is true forever, then a21d45726 (tcp: avoid order-1 allocations on wifi and tx path) does in fact resolve this issue at the cost of not being able to use the extra room kmalloc() might have given us in __alloc_skb() for collapsing.

Was that an intentional side effect of a21d45726?

This had a followup with 22b4a4f22da4
(tcp: fix retransmit of partially acked frames)

I also wonder if there is not another similar potential problem in __tcp_retransmit_skb) after call to

Quoted 7 lines

tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)

csum_start can overflow again because of MAX_TCP_HEADER headroom reserve.

So maybe we should limit TCP MTU to (64K - MAX_TCP_HEADER)

Or adapt the test at line 2390 in net/ipv4/tcp_output.c

to force in the case there could be an overflow a :

struct sk_buff *nskb = __pskb_copy()

So that we have a new skb with minimal headroom.

#5 lore raw
Quoted 15 lines · Eric Dumazet
Quoted 6 lines · Eric Dumazet

This had a followup with 22b4a4f22da4
(tcp: fix retransmit of partially acked frames)

I also wonder if there is not another similar potential problem in __tcp_retransmit_skb) after call to

Quoted 7 lines

tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq)

csum_start can overflow again because of MAX_TCP_HEADER headroom reserve.

So maybe we should limit TCP MTU to (64K - MAX_TCP_HEADER)

Quoted 7 lines

Or adapt the test at line 2390 in net/ipv4/tcp_output.c

to force in the case there could be an overflow a :

struct sk_buff *nskb = __pskb_copy()

So that we have a new skb with minimal headroom.

I would favour the __pskb_copy() variant as that would be more generic. I'll redo the patch.