Skip to contentSkip to conversation

net: check skb partial checksum offset after trim

PATCH In review Vasily Averin 4 participants 12 messages Dec 11, 2020 – Dec 15, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: check skb partial checksum offset after trim

syzkaller reproduces BUG_ON in skb_checksum_help():
tun creates skb with big partial checksum area and small ip packet inside, then ip_rcv() decreases skb size of below length of checksummed area, then checksum_tg() called via netfilter hook detects incorrect skb:

        offset = skb_checksum_start_offset(skb);
        BUG_ON(offset >= skb_headlen(skb));

This patch drops CHEKSUM_PARTIAL mark when skb is trimmed below size of checksummed area.

---
 include/linux/skbuff.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
include/linux/skbuff.h +10 -1
index a828cf9..0a9545d 100644
@@ -3419,9 +3419,18 @@ static inline void *skb_push_rcsum(struct sk_buff *skb, unsigned int len)
 
 static inline int pskb_trim_rcsum(struct sk_buff *skb, unsigned int len)
 {
+	int ret;
+
 	if (likely(len >= skb->len))
 		return 0;
-	return pskb_trim_rcsum_slow(skb, len);
+	ret = pskb_trim_rcsum_slow(skb, len);
+	if (!ret && (skb->ip_summed == CHECKSUM_PARTIAL)) {
+		int offset = skb_checksum_start_offset(skb) + skb->csum_offset;
+
+		if (offset + sizeof(__sum16) > skb_headlen(skb))
+			skb->ip_summed = CHECKSUM_NONE;
+	}
+	return ret;
 }
 
 static inline int __skb_trim_rcsum(struct sk_buff *skb, unsigned int len)
Signature · 1 line
1.8.3.1
#1 lore raw

Originally it was reported on Ubuntu 4.14 kernel,
then I've reproduced it on upstream 5.10-rc7.
If I'm right the problem is quite old and should
affect all maintained stable kernels too.

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

Thank you,
Vasily Averin

Quoted 42 lines · Vasily Averin

syzkaller reproduces BUG_ON in skb_checksum_help():
tun creates skb with big partial checksum area and small ip packet inside, then ip_rcv() decreases skb size of below length of checksummed area, then checksum_tg() called via netfilter hook detects incorrect skb:

        offset = skb_checksum_start_offset(skb);
        BUG_ON(offset >= skb_headlen(skb));

This patch drops CHEKSUM_PARTIAL mark when skb is trimmed below size of checksummed area.

---
 include/linux/skbuff.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
include/linux/skbuff.h +10 -1
index a828cf9..0a9545d 100644
@@ -3419,9 +3419,18 @@ static inline void *skb_push_rcsum(struct sk_buff *skb, unsigned int len)
 
 static inline int pskb_trim_rcsum(struct sk_buff *skb, unsigned int len)
 {
+	int ret;
+
 	if (likely(len >= skb->len))
 		return 0;
-	return pskb_trim_rcsum_slow(skb, len);
+	ret = pskb_trim_rcsum_slow(skb, len);
+	if (!ret && (skb->ip_summed == CHECKSUM_PARTIAL)) {
+		int offset = skb_checksum_start_offset(skb) + skb->csum_offset;
+
+		if (offset + sizeof(__sum16) > skb_headlen(skb))
+			skb->ip_summed = CHECKSUM_NONE;
+	}
+	return ret;
 }
 
 static inline int __skb_trim_rcsum(struct sk_buff *skb, unsigned int len)
#2 lore raw
Quoted 5 lines · Vasily Averin

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Thank you,
Vasily Averin

#3 lore raw
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

It is not ideal to have to add branches in the common path for these obscure bad packets from virtio/tuntap/af_packet. We try to avoid that with more strict validation at the source in virtio_net_hdr_to_skb. Evidently syzbot again found a way past again.

If this is a packet with gso_type and checksum offload, we know the accepted protocols and can validate the offset. If gso_type is none, however, no such assumptions can be made. All we could do is try to dissect and if a known protocol and valid th_off, compare that to the checksum fields passed by userspace.

So that path is certainly more complex than your fix, which works as well.

#4 lore raw
Quoted 12 lines · Willem de Bruijn
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Quoted 6 lines

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?

Thank you,
Vasily Averin

#5 lore raw
Quoted 14 lines · Vasily Averin
Quoted 12 lines · Willem de Bruijn
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Quoted 8 lines
Quoted 6 lines

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?

I would prefer to have more strict input validation in tun/virtio/packet (virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.

Instead of adding a workaround in the not path, I thought about converting the two checks in skb_checksum_help

  BUG_ON(offset >= skb_headlen(skb));
  BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));

to normal error paths and return EINVAL. But most callers, including this one (checksum_tg), don't check the return value to drop the packet.

Given that, your approach sounds the most reasonable. I would still drop these packets, as they are clearly bad and the only source of badness we know is untrusted user input.

In that case, perhaps the test can move into pskb_trim_rcsum_slow, below the CHECKSUM_COMPLETE branch.

#6 lore raw

PATCH v2 1/1 net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet

syzbot reproduces BUG_ON in skb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Then ip_rcv trims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Then checksum_tg() called via netfilter hook triggers BUG_ON:

        offset = skb_checksum_start_offset(skb);
        BUG_ON(offset >= skb_headlen(skb));

To work around the problem this patch forces pskb_trim_rcsum_slow() to return -EINVAL in described scenario. It allows its callers to drop such kind of packets.

---
v2: drop bogus packets instead change its CHECKSUM_PARTIAL to CHECKSUM_NONE

 net/core/skbuff.c | 6 ++++++
 1 file changed, 6 insertions(+)
net/core/skbuff.c +6 -0
index e578544..fbadd93 100644
@@ -2011,6 +2011,12 @@ int pskb_trim_rcsum_slow(struct sk_buff *skb, unsigned int len)
 		skb->csum = csum_block_sub(skb->csum,
 					   skb_checksum(skb, len, delta, 0),
 					   len);
+	} else if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		int hdlen = (len > skb_headlen(skb)) ? skb_headlen(skb) : len;
+		int offset = skb_checksum_start_offset(skb) + skb->csum_offset;
+
+		if (offset + sizeof(__sum16) > hdlen)
+			return -EINVAL;
 	}
 	return __pskb_trim(skb, len);
 }
Signature · 1 line
1.8.3.1
Acked-by on v2 1/1
#7 lore raw

Re: [PATCH v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet

Quoted 15 lines · Vasily Averin

syzbot reproduces BUG_ON in skb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Then ip_rcv trims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Then checksum_tg() called via netfilter hook triggers BUG_ON:

        offset = skb_checksum_start_offset(skb);
        BUG_ON(offset >= skb_headlen(skb));

To work around the problem this patch forces pskb_trim_rcsum_slow() to return -EINVAL in described scenario. It allows its callers to drop such kind of packets.

Quoted 5 lines

---
v2: drop bogus packets instead change its CHECKSUM_PARTIAL to CHECKSUM_NONE

Thanks for revising.

As far as I can tell, this goes back to the original introduction of that user interface to set checksum offload, so

Fixes 296f96fcfc16 ("Net driver using virtio")

For next time, please also mark network fixes as [PATCH net]. With that

#8 lore raw

Re: [PATCH v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet

In reply to Vasily Averin
This patch was applied to netdev/net-next.git (refs/heads/master):

Hello:

This patch was applied to netdev/net-next.git (refs/heads/master):

Quoted 8 lines · On Mon, 14 Dec 2020 22:07:39 +0300 you wrote:

syzbot reproduces BUG_ON in skb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Then ip_rcv trims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Then checksum_tg() called via netfilter hook triggers BUG_ON:

[...]

Here is the summary with links:
- [v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet

    https://git.kernel.org/netdev/net-next/c/54970a2fbb67

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

maintainer
#9 lore raw
In reply to Willem de Bruijn
Quoted 18 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 17 lines · Vasily Averin
Quoted 16 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

Quoted 7 lines
Quoted 1 line

Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?

I would prefer to have more strict input validation in tun/virtio/packet (virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.

Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this? Otherwise the validation at the source would work as well, no?

Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.

Quoted 16 lines

Instead of adding a workaround in the not path, I thought about converting the two checks in skb_checksum_help

  BUG_ON(offset >= skb_headlen(skb));
  BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));

to normal error paths and return EINVAL. But most callers, including this one (checksum_tg), don't check the return value to drop the packet.

Given that, your approach sounds the most reasonable. I would still drop these packets, as they are clearly bad and the only source of badness we know is untrusted user input.

In that case, perhaps the test can move into pskb_trim_rcsum_slow, below the CHECKSUM_COMPLETE branch.

#10 lore raw
Quoted 22 lines · Jakub Kicinski
Quoted 20 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 18 lines · Vasily Averin
Quoted 10 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Quoted 6 lines

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?

Quoted 8 lines
Quoted 5 lines

I would prefer to have more strict input validation in tun/virtio/packet (virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.

Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this?

Somewhat convoluted, yes. A packet with a checksum offset beyond the end of the ip packet.

skb_partial_csum_set (called from virtio_net_hdr_to_skb) verifies that the offsets are within the linear buffer passed from userspace, but without protocol parsing we don't know at that time that the offset is beyond the end of the packet.

Quoted 2 lines

Otherwise the validation
at the source would work as well, no?

The problem with validation is two fold: it may add noticeable cost to the hot path and it may have false positives: packets that the flow dissector cannot fully dissect, but which are harmless and were previously accepted.

I do want to add such strict source validation based on flow dissection, but as an opt-in (sysctl) feature.

Quoted 3 lines

Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.

From the description, I would assume so, too. Haven't tested.

#11 lore raw
Quoted 43 lines · Willem de Bruijn
Quoted 22 lines · Jakub Kicinski
Quoted 20 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 18 lines · Vasily Averin
Quoted 10 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines

It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?

I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.

Quoted 6 lines

Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...

Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.

Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?

Quoted 8 lines
Quoted 5 lines

I would prefer to have more strict input validation in tun/virtio/packet (virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.

Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this?

Somewhat convoluted, yes. A packet with a checksum offset beyond the end of the ip packet.

skb_partial_csum_set (called from virtio_net_hdr_to_skb) verifies that the offsets are within the linear buffer passed from userspace, but without protocol parsing we don't know at that time that the offset is beyond the end of the packet.

Quoted 2 lines

Otherwise the validation
at the source would work as well, no?

Quoted 13 lines

The problem with validation is two fold: it may add noticeable cost to the hot path and it may have false positives: packets that the flow dissector cannot fully dissect, but which are harmless and were previously accepted.

I do want to add such strict source validation based on flow dissection, but as an opt-in (sysctl) feature.

Quoted 5 lines

Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.

From the description, I would assume so, too. Haven't tested.

Original syzkaller reproducer fails on upstream because it prepares invalid iptable ruleset, new kernels have more strict validation of iptable rules. I've commented this call im originsl reproducer and set the CHECKSUM rule manually, then run of corrected reproducer triggered BUG_ON in skb_checksum_help(). I've crashed upstream 5.10-rc7 kernel by this way and then validated patched kernel. originally we got such problem on RHEL7-based kernel, so I think the problem affects all stable and actual distribution kernels.

Thank you,
Vasily Averin