net: check skb partial checksum offset after trim
Subsystems (2)
PATCH 1/1 net: check skb partial checksum offset after trim
syzkaller reproduces BUG_ON in skb_checksum_help():
tun creates skb with big partial checksum area and small ip packet inside, then ip_rcv() decreases skb size of below length of checksummed area, then checksum_tg() called via netfilter hook detects incorrect skb:
offset = skb_checksum_start_offset(skb);
BUG_ON(offset >= skb_headlen(skb));This patch drops CHEKSUM_PARTIAL mark when skb is trimmed below size of checksummed area.
--- include/linux/skbuff.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
include/linux/skbuff.h +10 -1
@@ -3419,9 +3419,18 @@ static inline void *skb_push_rcsum(struct sk_buff *skb, unsigned int len)static inline int pskb_trim_rcsum (struct sk_buff *skb,unsigned int len) { +int ret; +if (likely (len >= skb->len))return 0 ; -return pskb_trim_rcsum_slow (skb, len); + ret =pskb_trim_rcsum_slow (skb, len); +if (!ret && (skb->ip_summed ==CHECKSUM_PARTIAL )) { +int offset =skb_checksum_start_offset (skb) + skb->csum_offset; + +if (offset +sizeof (__sum16) >skb_headlen (skb)) + skb->ip_summed =CHECKSUM_NONE ; + } +return ret; }static inline int __skb_trim_rcsum (struct sk_buff *skb,unsigned int len)
Signature · 1 line
1.8.3.1
Originally it was reported on Ubuntu 4.14 kernel,
then I've reproduced it on upstream 5.10-rc7.
If I'm right the problem is quite old and should
affect all maintained stable kernels too.
It seems for me the similar problem can happen in __skb_trim_rcsum(). Also I doubt that that skb_checksum_start_offset(skb) checks in __skb_postpull_rcsum() and skb_csum_unnecessary() are correct, becasue they do not guarantee that skb have correct CHECKSUM_PARTIAL. Could somebody confirm it?
Thank you,
Vasily Averin
Quoted 42 lines · Vasily Averin
syzkaller reproduces
BUG_ONinskb_checksum_help():
tun creates skb with big partial checksum area and small ip packet inside, thenip_rcv()decreases skb size of below length of checksummed area, thenchecksum_tg()called via netfilter hook detects incorrect skb:offset = skb_checksum_start_offset(skb); BUG_ON(offset >= skb_headlen(skb));This patch drops
CHEKSUM_PARTIALmark when skb is trimmed below size of checksummed area.Reported-by [email protected]Signed-off-by Vasily Averin <[email protected]>--- include/linux/skbuff.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)include/linux/skbuff.h +10
-1@@ -3419,9 +3419,18 @@ static inline void *skb_push_rcsum(struct sk_buff *skb, unsigned int len)static inline int pskb_trim_rcsum (struct sk_buff *skb,unsigned int len) { +int ret; +if (likely (len >= skb->len))return 0 ; -return pskb_trim_rcsum_slow (skb, len); + ret =pskb_trim_rcsum_slow (skb, len); +if (!ret && (skb->ip_summed ==CHECKSUM_PARTIAL )) { +int offset =skb_checksum_start_offset (skb) + skb->csum_offset; + +if (offset +sizeof (__sum16) >skb_headlen (skb)) + skb->ip_summed =CHECKSUM_NONE ; + } +return ret; }static inline int __skb_trim_rcsum (struct sk_buff *skb,unsigned int len)
Quoted 5 lines · Vasily Averin
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?
I've rechecked the code and I think now that other places are not affected, i.e. skb_push_rcsum() only should be patched.
Thank you,
Vasily Averin
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.
Thanks for investigating this. So tun was able to insert a packet with csum_start + csum_off + 2 beyond the packet after trimming, using virtio_net_hdr.csum_...
Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to CHECKSUM_NONE.
It is not ideal to have to add branches in the common path for these obscure bad packets from virtio/tuntap/af_packet. We try to avoid that with more strict validation at the source in virtio_net_hdr_to_skb. Evidently syzbot again found a way past again.
If this is a packet with gso_type and checksum offload, we know the accepted protocols and can validate the offset. If gso_type is none, however, no such assumptions can be made. All we could do is try to dissect and if a known protocol and valid th_off, compare that to the checksum fields passed by userspace.
So that path is certainly more complex than your fix, which works as well.
Quoted 12 lines · Willem de Bruijn
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.
Quoted 6 lines
Thanks for investigating this. So tun was able to insert a packet with
csum_start+csum_off+ 2 beyond the packet after trimming, using virtio_net_hdr.csum_...Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to
CHECKSUM_NONE.
Do you mean it's better to force pskb_trim_rcsum() to return -EINVAL instead?
Thank you,
Vasily Averin
Quoted 14 lines · Vasily Averin
Quoted 12 lines · Willem de Bruijn
Quoted 10 lines · Vasily Averin
Quoted 5 lines · Vasily Averin
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.
Quoted 8 lines
Quoted 6 lines
Thanks for investigating this. So tun was able to insert a packet with
csum_start+csum_off+ 2 beyond the packet after trimming, using virtio_net_hdr.csum_...Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to
CHECKSUM_NONE.Do you mean it's better to force
pskb_trim_rcsum()to return-EINVALinstead?
I would prefer to have more strict input validation in tun/virtio/packet (virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.
Instead of adding a workaround in the not path, I thought about converting the two checks in skb_checksum_help
BUG_ON(offset >= skb_headlen(skb)); BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));
to normal error paths and return EINVAL. But most callers, including this one (checksum_tg), don't check the return value to drop the packet.
Given that, your approach sounds the most reasonable. I would still drop these packets, as they are clearly bad and the only source of badness we know is untrusted user input.
In that case, perhaps the test can move into pskb_trim_rcsum_slow, below the CHECKSUM_COMPLETE branch.
PATCH v2 1/1 net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet
syzbot reproduces BUG_ON in skb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Then ip_rcv trims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Then checksum_tg() called via netfilter hook triggers BUG_ON:
offset = skb_checksum_start_offset(skb);
BUG_ON(offset >= skb_headlen(skb));To work around the problem this patch forces pskb_trim_rcsum_slow() to return -EINVAL in described scenario. It allows its callers to drop such kind of packets.
---
v2: drop bogus packets instead change its CHECKSUM_PARTIAL to CHECKSUM_NONE
net/core/skbuff.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/core/skbuff.c +6 -0
@@ -2011,6 +2011,12 @@ int pskb_trim_rcsum_slow(struct sk_buff *skb, unsigned int len) skb->csum =csum_block_sub (skb->csum,skb_checksum (skb, len, delta,0 ), len); + }else if (skb->ip_summed ==CHECKSUM_PARTIAL ) { +int hdlen = (len >skb_headlen (skb)) ?skb_headlen (skb) : len; +int offset =skb_checksum_start_offset (skb) + skb->csum_offset; + +if (offset +sizeof (__sum16) > hdlen) +return -EINVAL ; }return __pskb_trim (skb, len); }
Signature · 1 line
1.8.3.1
Re: [PATCH v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet
Quoted 15 lines · Vasily Averin
syzbot reproduces
BUG_ONinskb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Thenip_rcvtrims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Thenchecksum_tg()called via netfilter hook triggersBUG_ON:offset = skb_checksum_start_offset(skb); BUG_ON(offset >= skb_headlen(skb));To work around the problem this patch forces
pskb_trim_rcsum_slow()to return-EINVALin described scenario. It allows its callers to drop such kind of packets.
Quoted 5 lines
Reported-by [email protected]Signed-off-by Vasily Averin <[email protected]>---
v2: drop bogus packets instead change itsCHECKSUM_PARTIALtoCHECKSUM_NONE
Thanks for revising.
As far as I can tell, this goes back to the original introduction of that user interface to set checksum offload, so
For next time, please also mark network fixes as [PATCH net]. With that
Re: [PATCH v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet
In reply to Vasily AverinThis patch was applied to netdev/net-next.git (refs/heads/master):
Hello:
This patch was applied to netdev/net-next.git (refs/heads/master):
Quoted 8 lines · On Mon, 14 Dec 2020 22:07:39 +0300 you wrote:
syzbot reproduces
BUG_ONinskb_checksum_help():
tun creates (bogus) skb with huge partial-checksummed area and small ip packet inside. Thenip_rcvtrims the skb based on size of internal ip packet, after that csum offset points beyond of trimmed skb. Thenchecksum_tg()called via netfilter hook triggersBUG_ON:[...]
Here is the summary with links:
- [v2] net: drop bogus skb with CHECKSUM_PARTIAL and offset beyond end of trimmed packet
https://git.kernel.org/netdev/net-next/c/54970a2fbb67
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Jakub Kicinski
maintainerQuoted 18 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 17 lines · Vasily Averin
Quoted 16 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.Thanks for investigating this. So tun was able to insert a packet with
csum_start+csum_off+ 2 beyond the packet after trimming, using virtio_net_hdr.csum_...Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to
CHECKSUM_NONE.
Quoted 7 lines
Quoted 1 line
Do you mean it's better to force
pskb_trim_rcsum()to return-EINVALinstead?I would prefer to have more strict input validation in tun/virtio/packet (
virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.
Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this? Otherwise the validation at the source would work as well, no?
Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.
Quoted 16 lines
Instead of adding a workaround in the not path, I thought about converting the two checks in
skb_checksum_helpBUG_ON(offset >= skb_headlen(skb)); BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));to normal error paths and return EINVAL. But most callers, including this one (
checksum_tg), don't check the return value to drop the packet.Given that, your approach sounds the most reasonable. I would still drop these packets, as they are clearly bad and the only source of badness we know is untrusted user input.
In that case, perhaps the test can move into
pskb_trim_rcsum_slow, below theCHECKSUM_COMPLETEbranch.
Quoted 22 lines · Jakub Kicinski
Quoted 20 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 18 lines · Vasily Averin
Quoted 10 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.Quoted 6 lines
Thanks for investigating this. So tun was able to insert a packet with
csum_start+csum_off+ 2 beyond the packet after trimming, using virtio_net_hdr.csum_...Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to
CHECKSUM_NONE.Do you mean it's better to force
pskb_trim_rcsum()to return-EINVALinstead?
Quoted 8 lines
Quoted 5 lines
I would prefer to have more strict input validation in tun/virtio/packet (
virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this?
Somewhat convoluted, yes. A packet with a checksum offset beyond the end of the ip packet.
skb_partial_csum_set (called from virtio_net_hdr_to_skb) verifies that the offsets are within the linear buffer passed from userspace, but without protocol parsing we don't know at that time that the offset is beyond the end of the packet.
Quoted 2 lines
Otherwise the validation
at the source would work as well, no?
The problem with validation is two fold: it may add noticeable cost to the hot path and it may have false positives: packets that the flow dissector cannot fully dissect, but which are harmless and were previously accepted.
I do want to add such strict source validation based on flow dissection, but as an opt-in (sysctl) feature.
Quoted 3 lines
Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.
From the description, I would assume so, too. Haven't tested.
Quoted 43 lines · Willem de Bruijn
Quoted 22 lines · Jakub Kicinski
Quoted 20 lines · On Sun, 13 Dec 2020 20:59:54 -0500 Willem de Bruijn wrote:
Quoted 18 lines · Vasily Averin
Quoted 10 lines
Quoted 9 lines · Vasily Averin
Quoted 5 lines
It seems for me the similar problem can happen in
__skb_trim_rcsum(). Also I doubt that thatskb_checksum_start_offset(skb)checks in__skb_postpull_rcsum()andskb_csum_unnecessary()are correct, becasue they do not guarantee that skb have correctCHECKSUM_PARTIAL. Could somebody confirm it?I've rechecked the code and I think now that other places are not affected, i.e.
skb_push_rcsum()only should be patched.Quoted 6 lines
Thanks for investigating this. So tun was able to insert a packet with
csum_start+csum_off+ 2 beyond the packet after trimming, using virtio_net_hdr.csum_...Any packet with an offset beyond the end of the packet is bogus really. No need to try to accept it by downgrading to
CHECKSUM_NONE.Do you mean it's better to force
pskb_trim_rcsum()to return-EINVALinstead?Quoted 8 lines
Quoted 5 lines
I would prefer to have more strict input validation in tun/virtio/packet (
virtio_net_hdr_to_skb), rather than new checks in the hot path. But that is a larger change and not feasible unconditionally due to performance impact and likely some false positive drops. So out of scope here.Could you please elaborate? Is it the case that syzbot constructed some extremely convoluted frame to trigger this?
Somewhat convoluted, yes. A packet with a checksum offset beyond the end of the ip packet.
skb_partial_csum_set(called fromvirtio_net_hdr_to_skb) verifies that the offsets are within the linear buffer passed from userspace, but without protocol parsing we don't know at that time that the offset is beyond the end of the packet.Quoted 2 lines
Otherwise the validation
at the source would work as well, no?
Quoted 13 lines
The problem with validation is two fold: it may add noticeable cost to the hot path and it may have false positives: packets that the flow dissector cannot fully dissect, but which are harmless and were previously accepted.
I do want to add such strict source validation based on flow dissection, but as an opt-in (sysctl) feature.
Quoted 5 lines
Does it actually trigger upstream? The linked syzbot report is for 4.14 but from the commit description it sounds like the problem should repro rather reliably.
From the description, I would assume so, too. Haven't tested.
Original syzkaller reproducer fails on upstream because it prepares invalid iptable ruleset, new kernels have more strict validation of iptable rules. I've commented this call im originsl reproducer and set the CHECKSUM rule manually, then run of corrected reproducer triggered BUG_ON in skb_checksum_help(). I've crashed upstream 5.10-rc7 kernel by this way and then validated patched kernel. originally we got such problem on RHEL7-based kernel, so I think the problem affects all stable and actual distribution kernels.
Thank you,
Vasily Averin