Skip to contentSkip to conversation

net: Convert skb->csum_(start|offset) integrity BUG_ON() to WARN_ON() & drop

PATCH In review Thomas Graf 3 participants 8 messages Feb 13, 2013 – Feb 14, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: Convert skb->csum_(start|offset) integrity BUG_ON() to WARN_ON() & drop

skb_checksum_help() verifies the integrity of skb->csum_start and skb->csum_offset with BUG_ON()s.

They have been hit with IPoIB which uses a 64K MTU. If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start.

This in turn will trigger the BUG_ON() in skb_checksum_help(). Convert these to WARN_ON() and drop the packet.

---
 net/core/dev.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/core/dev.c +5 -2
index f64e439..629d22e 100644
@@ -2047,11 +2047,14 @@ int skb_checksum_help(struct sk_buff *skb)
 	}
 
 	offset = skb_checksum_start_offset(skb);
-	BUG_ON(offset >= skb_headlen(skb));
+	if (WARN_ON(offset >= skb_headlen(skb)))
+		return -ERANGE;
+
 	csum = skb_checksum(skb, offset, skb->len - offset, 0);
 
 	offset += skb->csum_offset;
-	BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));
+	if (WARN_ON(offset + sizeof(__sum16) > skb_headlen(skb)))
+		return -ERANGE;
 
 	if (skb_cloned(skb) &&
 	    !skb_clone_writable(skb, offset + sizeof(__sum16))) {
#1 lore raw
Quoted 4 lines · Thomas Graf

They have been hit with IPoIB which uses a 64K MTU. If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start.

On the subject of fixing this, I considered:

a) Reallocate the headroom in tcp_trim_head() if it would overflow. I disregarded this idea because replacing the old skb with the new skb on the write queue for such a rare situation seems overkill.

b) No longer collapse if the new skb would result in a a headroom + data that exceeds 64K. This seems to be the most trivial fix.

c) Increase size of csum_start or store checksum_start_offset differently.

Other ideas?

maintainer
#2 lore raw

From: Thomas Graf <[email protected]>
Date: Wed, 13 Feb 2013 23:48:43 +0000

Quoted 13 lines · Thomas Graf
Quoted 4 lines

They have been hit with IPoIB which uses a 64K MTU. If a TCP retransmission gets partially ACKed and collapsed multiple times it is possible for the headroom to grow beyond 64K which will overflow the 16bit skb->csum_start.

On the subject of fixing this, I considered:

a) Reallocate the headroom in tcp_trim_head() if it would overflow. I disregarded this idea because replacing the old skb with the new skb on the write queue for such a rare situation seems overkill.

Quoted 8 lines

b) No longer collapse if the new skb would result in a a headroom + data that exceeds 64K. This seems to be the most trivial fix.

c) Increase size of csum_start or store checksum_start_offset differently.

Other ideas?

"b" is a good idea.

Let's not paper over this, this BUG_ON() is really a BUG_ON() meaning "FIX ME NOW" :-)

#3 lore raw
Quoted 2 lines · David Miller

From: Thomas Graf <[email protected]>
Date: Wed, 13 Feb 2013 23:48:43 +0000

[...]

Quoted 3 lines
Quoted 3 lines

b) No longer collapse if the new skb would result in a a headroom + data that exceeds 64K. This seems to be the most trivial fix.

[...]

Quoted 3 lines
Quoted 1 line

Other ideas?

"b" is a good idea.

OK, patch to do so being tested by original reporter.

Quoted 2 lines

Let's not paper over this, this BUG_ON() is really a BUG_ON() meaning "FIX ME NOW" :-)

Maybe it's my general dislike of BUG_ON() in the processing path, especially if the bug condition can be influenced remotely. It looks absolutely doable to trigger the previously mentioned partial acking & collapsing on purpose by a malicious receiver even with an MTU of 1500. I believe we should avoid total DoS in future similar situations that we don't think of yet.

#4 lore raw
Quoted 6 lines · Thomas Graf

Maybe it's my general dislike of BUG_ON() in the processing path, especially if the bug condition can be influenced remotely. It looks absolutely doable to trigger the previously mentioned partial acking & collapsing on purpose by a malicious receiver even with an MTU of 1500. I believe we should avoid total DoS in future similar situations that we don't think of yet.

It seems not possible to avoid bugs, being a BUG_ON() or a out of bound memory access or whatever. We must fix them eventually.

In this case, it seems we must limit payload to

65535 - MAX_TCP_HEADER

It would make tcp_xmit_size_goal() a bit shorter.

net/ipv4/tcp.c +1 -4
index 2c7e596..2f6c8e5 100644
@@ -793,10 +793,7 @@ static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now,
 	xmit_size_goal = mss_now;
 
 	if (large_allowed && sk_can_gso(sk)) {
-		xmit_size_goal = ((sk->sk_gso_max_size - 1) -
-				  inet_csk(sk)->icsk_af_ops->net_header_len -
-				  inet_csk(sk)->icsk_ext_hdr_len -
-				  tp->tcp_header_len);
+		xmit_size_goal = sk->sk_gso_max_size - 1 - MAX_TCP_HEADER;
 
 		/* TSQ : try to have two TSO segments in flight */
 		xmit_size_goal = min_t(u32, xmit_size_goal,
#5 lore raw
Quoted 2 lines · Eric Dumazet

It seems not possible to avoid bugs, being a BUG_ON() or a out of bound memory access or whatever. We must fix them eventually.

Of course, I never intended to not fix this but I still think leaving the BUG_ON() is wrong ;-)

Quoted 12 lines

In this case, it seems we must limit payload to

65535 - MAX_TCP_HEADER

It would make tcp_xmit_size_goal() a bit shorter.

net/ipv4/tcp.c +0 -0
index 2c7e596..2f6c8e5 100644
@@ -793,10 +793,7 @@ static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now,
 	xmit_size_goal = mss_now;

net/ipv4/tcp.c static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, · line 799

Quoted 10 lines
net/ipv4/tcp.c +1 -4
 
 	if (large_allowed && sk_can_gso(sk)) {
-		xmit_size_goal = ((sk->sk_gso_max_size - 1) -
-				  inet_csk(sk)->icsk_af_ops->net_header_len -
-				  inet_csk(sk)->icsk_ext_hdr_len -
-				  tp->tcp_header_len);
+		xmit_size_goal = sk->sk_gso_max_size - 1 - MAX_TCP_HEADER;
 
 		/* TSQ : try to have two TSO segments in flight */
 		xmit_size_goal = min_t(u32, xmit_size_goal,

I don't think this would help. The allocated skb data will still exceed 64K and thus after trimming the acked data and collapsing the header might be stored after the 64K mark.

We would have to limit the skb tailroom to 64K upon allocation. That would mean we would waste some of the additional space that kmalloc() might have given us:

net/core/skbuff.c +2 -0
index 32443eb..c8f9850 100644
@@ -241,6 +241,8 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mas
         * to allow max possible filling before reallocation.
         */
        size = SKB_WITH_OVERHEAD(ksize(data));
+       /* ensure that all offsets based on skb->head fit into 16bits */
+       size = min_t(int, size, 65535);
        prefetchw(data + size);
 
        /*

Or if that is not ideal, avoiding the collapse that causes the overflow would also help:

net/ipv4/tcp_output.c +6 -0
index 5d45159..e9111b4 100644
@@ -2301,6 +2301,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,
                if (after(TCP_SKB_CB(skb)->end_seq, tcp_wnd_end(tp)))
                        break;
 
+               /* Never collapse if the resulting headroom + data exceeds
+                * 64K as that is the maximum csum_start can cover.
+                */
+               if (skb_headroom(to) + to->len + skb->len > 65535)
+                       break;
+
                tcp_collapse_retrans(sk, to);
        }
 }
#6 lore raw
Quoted 27 lines · Thomas Graf
Quoted 2 lines · Eric Dumazet

It seems not possible to avoid bugs, being a BUG_ON() or a out of bound memory access or whatever. We must fix them eventually.

Of course, I never intended to not fix this but I still think leaving the BUG_ON() is wrong ;-)

Quoted 20 lines

In this case, it seems we must limit payload to

65535 - MAX_TCP_HEADER

It would make tcp_xmit_size_goal() a bit shorter.

net/ipv4/tcp.c +1 -4
index 2c7e596..2f6c8e5 100644
@@ -793,10 +793,7 @@ static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now,
 	xmit_size_goal = mss_now;
 
 	if (large_allowed && sk_can_gso(sk)) {
-		xmit_size_goal = ((sk->sk_gso_max_size - 1) -
-				  inet_csk(sk)->icsk_af_ops->net_header_len -
-				  inet_csk(sk)->icsk_ext_hdr_len -
-				  tp->tcp_header_len);
+		xmit_size_goal = sk->sk_gso_max_size - 1 - MAX_TCP_HEADER;
 
Quoted 7 lines

net/ipv4/tcp.c static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, · line 799

Quoted 2 lines
net/ipv4/tcp.c +0 -0
 		/* TSQ : try to have two TSO segments in flight */
 		xmit_size_goal = min_t(u32, xmit_size_goal,

I don't think this would help. The allocated skb data will still exceed 64K and thus after trimming the acked data and collapsing the header might be stored after the 64K mark.

OK, I now understand your issue.

Quoted 9 lines

We would have to limit the skb tailroom to 64K upon allocation. That would mean we would waste some of the additional space that kmalloc() might have given us:

net/core/skbuff.c +0 -0
index 32443eb..c8f9850 100644

net/core/skbuff.c struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mas · line 247

Quoted 7 lines
net/core/skbuff.c +2 -0
@@ -241,6 +241,8 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mas
         * to allow max possible filling before reallocation.
         */
        size = SKB_WITH_OVERHEAD(ksize(data));
+       /* ensure that all offsets based on skb->head fit into 16bits */
+       size = min_t(int, size, 65535);
        prefetchw(data + size);
 

This part is certainly not good.

alloc_skb() is generic and some users really want more than 64K

Quoted 13 lines
net/core/skbuff.c +0 -0
        /*

Or if that is not ideal, avoiding the collapse that causes the overflow would also help:

net/ipv4/tcp_output.c +0 -0
index 5d45159..e9111b4 100644
@@ -2301,6 +2301,12 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to,
                if (after(TCP_SKB_CB(skb)->end_seq, tcp_wnd_end(tp)))

net/ipv4/tcp_output.c static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *to, · line 2311

Quoted 10 lines
net/ipv4/tcp_output.c +6 -0
                        break;
 
+               /* Never collapse if the resulting headroom + data exceeds
+                * 64K as that is the maximum csum_start can cover.
+                */
+               if (skb_headroom(to) + to->len + skb->len > 65535)
+                       break;
+
                tcp_collapse_retrans(sk, to);
 

Definitely better.

maintainer
#7 lore raw
In reply to Thomas Graf

From: Thomas Graf <[email protected]>
Date: Thu, 14 Feb 2013 10:18:53 +0000

Quoted 7 lines · David Miller

Maybe it's my general dislike of BUG_ON() in the processing path, especially if the bug condition can be influenced remotely. It looks absolutely doable to trigger the previously mentioned partial acking & collapsing on purpose by a malicious receiver even with an MTU of 1500. I believe we should avoid total DoS in future similar situations that we don't think of yet.

I heard that people can very effectively protect themselves from DoS's by not using Infiniband. :-)