bpf: make sure skb->len != 0 when redirecting to a tunneling device
Subsystems (3)
PATCH 1/1 bpf: make sure skb->len != 0 when redirecting to a tunneling device
syzkaller managed to trigger another case where skb->len == 0 when we enter __dev_queue_xmit:
WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we __skb_pull l2 header and arrive again at skb->len == 0. Doesn't seem like we can do anything better than having an explicit check after __skb_pull?
--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)
net/core/filter.c +4 -0
@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }/* At ingress, the mac header has already been pulled once. * At egress, skb_pospull_rcsum has to be done in case that
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
Martin KaFai Lau
reviewerQuoted 27 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6
Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?
hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in the bpf_prog_test_run (?),
how about ensure that the skb will at least have some header after l2 header in bpf_prog_test_run_skb(). Adding some headers/bytes if the data_size_in does not have it. This may break some external test cases that somehow has no l3/4? test_progs should be mostly fine considering they are using the pkt_v[46] in network_helpers.h.
Quoted 23 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)net/core/filter.c +4
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }/* At ingress, the mac header has already been pulled once. * At egress, skb_pospull_rcsum has to be done in case that
Quoted 36 lines · Martin KaFai Lau
Quoted 27 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?
Quoted 8 lines
hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in thebpf_prog_test_run(?),
how about ensure that the skb will at least have some header after l2 header inbpf_prog_test_run_skb(). Adding some headers/bytes if thedata_size_indoes not have it. This may break some external test cases that somehow has no l3/4?test_progsshould be mostly fine considering they are using the pkt_v[46] innetwork_helpers.h.
For the previous issue we've added "skb->len != 0" check which works for the cases that remove l2.
For the ones that don't, I think you're right, and checking at the time of bpf_prog_test_run_skb can probably be enough, lemme try (require ETH_HLEN+1 vs ETH_HLEN).
For some reason I was under the impression that Lorenz changed the size from 0 to 14 [0], but he went from 14 to 15, so we won't break at least cilium again..
CC'd him just in case.
0: https://github.com/cilium/ebpf/pull/788
Quoted 2 lines
Adding some headers/bytes if the
data_size_indoes not have it. This may break some external test cases that somehow has no l3/4?
Yeah, idk, this seems like a last resort? I'd prefer to explicitly fail and communicate it back to the user than slap some extra byte and then fail in some other place unpredictably?
Quoted 23 lines
Quoted 22 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)net/core/filter.c +4
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }/* At ingress, the mac header has already been pulled once. * At egress, skb_pospull_rcsum has to be done in case that
Martin KaFai Lau
reviewerQuoted 56 lines · Stanislav Fomichev
Quoted 36 lines · Martin KaFai Lau
Quoted 27 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?Quoted 8 lines
hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in thebpf_prog_test_run(?),
how about ensure that the skb will at least have some header after l2 header inbpf_prog_test_run_skb(). Adding some headers/bytes if thedata_size_indoes not have it. This may break some external test cases that somehow has no l3/4?test_progsshould be mostly fine considering they are using the pkt_v[46] innetwork_helpers.h.For the previous issue we've added "
skb->len!= 0" check which works for the cases that remove l2.
For the ones that don't, I think you're right, and checking at the time ofbpf_prog_test_run_skbcan probably be enough, lemme try (require ETH_HLEN+1 vsETH_HLEN).
For some reason I was under the impression that Lorenz changed the size from 0 to 14 [0], but he went from 14 to 15, so we won't break at least cilium again..
CC'd him just in case.
Quoted 1 line
Thanks for the pointer.
The cilium's prog is SOCKET_FILTER (not l2). It is why the new "skb->len != 0" test broke it.
Quoted 7 lines
Quoted 3 lines
Adding some headers/bytes if the
data_size_indoes not have it. This may break some external test cases that somehow has no l3/4?Yeah, idk, this seems like a last resort? I'd prefer to explicitly fail and communicate it back to the user than slap some extra byte and then fail in some other place unpredictably?
If fixing in the fast path in filter.c, is __bpf_redirect_no_mac the only place that needs this check? bpf_redirect_neigh() looks ok to me since the neigh should have filled the mac header.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)net/core/filter.c +4
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }/* At ingress, the mac header has already been pulled once. * At egress, skb_pospull_rcsum has to be done in case that
This patch was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Quoted 27 lines · On Thu, 27 Oct 2022 15:55:37 -0700 you wrote:
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next] bpf: make sure skb->len != 0 when redirecting to a tunneling device
https://git.kernel.org/bpf/bpf-next/c/0ed041b1dd33
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Martin KaFai Lau
reviewerQuoted 47 lines · Martin KaFai Lau
Quoted 46 lines · Stanislav Fomichev
Quoted 44 lines · Martin KaFai Lau
Quoted 27 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in thebpf_prog_test_run(?),
how about ensure that the skb will at least have some header after l2 header in bpf_prog_test_run_skb(). Adding some headers/bytes if thedata_size_indoes not have it. This may break some external test cases that somehow has no l3/4?test_progsshould be mostly fine considering they are using the pkt_v[46] innetwork_helpers.h.
Quoted 2 lines
Quoted 2 lines
For the previous issue we've added "
skb->len!= 0" check which works for the cases that remove l2.
Yeah, I replied on the "bpf: Don't redirect packets with invalid pkt_len" thread which is hitting the same syzbot report afaict. I don't think that patch is actually fixing it.
Quoted 15 lines
Quoted 9 lines
For the ones that don't, I think you're right, and checking at the time of
bpf_prog_test_run_skbcan probably be enough, lemme try (require ETH_HLEN+1 vsETH_HLEN).
For some reason I was under the impression that Lorenz changed the size from 0 to 14 [0], but he went from 14 to 15, so we won't break at least cilium again..
CC'd him just in case.Thanks for the pointer.
The cilium's prog is
SOCKET_FILTER(not l2). It is why the new "skb->len!= 0" test broke it.Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Adding some headers/bytes if the
data_size_indoes not have it. This may break some external test cases that somehow has no l3/4?Yeah, idk, this seems like a last resort? I'd prefer to explicitly fail and communicate it back to the user than slap some extra byte and then fail in some other place unpredictably?
If fixing in the fast path in
filter.c, is__bpf_redirect_no_macthe only place that needs this check?bpf_redirect_neigh()looks ok to me since the neigh should have filled the mac header.
I took a closer look. This seems to be the only place needed the check, so applied. If it turns out there are other cases caused by test-run generated skb, we will revisit a fix in test_run.c and the existing tests have to adjust.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>---
net/core/filter.c| 4 ++++1 file changed, 4 insertions(+)net/core/filter.c +0
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/core/filter.c static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev, · line 2132
Quoted 6 lines
net/core/filter.c +4
-0if (mlen) { __skb_pull(skb, mlen); + if (unlikely(!skb->len)) { + kfree_skb(skb); + return -ERANGE; + }
One question, if the "!skb->len" check is deleted from convert___skb_to_skb(), this "unlikely(!skb->len)" block here has to be moved out of the "if (mlen)"?
Quoted 105 lines · Martin KaFai Lau
Quoted 49 lines · Martin KaFai Lau
Quoted 48 lines · Stanislav Fomichev
Quoted 46 lines · Martin KaFai Lau
Quoted 29 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in thebpf_prog_test_run(?),
how about ensure that the skb will at least have some header after l2 header inbpf_prog_test_run_skb(). Adding some headers/bytes if thedata_size_indoes not have it. This may break some external test cases that somehow has no l3/4?test_progsshould be mostly fine considering they are using the pkt_v[46] innetwork_helpers.h.Quoted 2 lines
Quoted 2 lines
For the previous issue we've added "
skb->len!= 0" check which works for the cases that remove l2.Yeah, I replied on the "bpf: Don't redirect packets with invalid
pkt_len" thread which is hitting the same syzbot report afaict. I don't think that patch is actually fixing it.Quoted 19 lines
Quoted 9 lines
For the ones that don't, I think you're right, and checking at the time of
bpf_prog_test_run_skbcan probably be enough, lemme try (require ETH_HLEN+1 vsETH_HLEN).
For some reason I was under the impression that Lorenz changed the size from 0 to 14 [0], but he went from 14 to 15, so we won't break at least cilium again..
CC'd him just in case.Thanks for the pointer.
The cilium's prog is
SOCKET_FILTER(not l2). It is why the new "skb->len!= 0" test broke it.Quoted 4 lines
Quoted 3 lines
Adding some headers/bytes if the
data_size_indoes not have it. This may break some external test cases that somehow has no l3/4?Quoted 7 lines
Quoted 3 lines
Yeah, idk, this seems like a last resort? I'd prefer to explicitly fail and communicate it back to the user than slap some extra byte and then fail in some other place unpredictably?
If fixing in the fast path in
filter.c, is__bpf_redirect_no_macthe only place that needs this check?bpf_redirect_neigh()looks ok to me since the neigh should have filled the mac header.I took a closer look. This seems to be the only place needed the check, so applied. If it turns out there are other cases caused by test-run generated skb, we will revisit a fix in
test_run.cand the existing tests have to adjust.Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)net/core/filter.c +0
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/core/filter.c static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev, · line 2132
Quoted 6 lines
net/core/filter.c +4
-0if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }One question, if the "!skb->len" check is deleted from
convert___skb_to_skb(), this "unlikely(!skb->len)" block here has to be moved out of the "if (mlen)"?
I see, yeah, that might be the alternative. I'm assuming __bpf_redirect_common is covered by "skb->mac_header >= skb->network_header" check?
Martin KaFai Lau
reviewerQuoted 116 lines · Stanislav Fomichev
Quoted 105 lines · Martin KaFai Lau
Quoted 49 lines · Martin KaFai Lau
Quoted 48 lines · Stanislav Fomichev
Quoted 46 lines · Martin KaFai Lau
Quoted 29 lines · Stanislav Fomichev
syzkaller managed to trigger another case where
skb->len== 0 when we enter__dev_queue_xmit:WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 skb_assert_len include/linux/skbuff.h:2576 [inline] WARNING: CPU: 0 PID: 2470 at include/linux/skbuff.h:2576 __dev_queue_xmit+0x2069/0x35e0 net/core/dev.c:4295 Call Trace: dev_queue_xmit+0x17/0x20 net/core/dev.c:4406 __bpf_tx_skb net/core/filter.c:2115 [inline] __bpf_redirect_no_mac net/core/filter.c:2140 [inline] __bpf_redirect+0x5fb/0xda0 net/core/filter.c:2163 ____bpf_clone_redirect net/core/filter.c:2447 [inline] bpf_clone_redirect+0x247/0x390 net/core/filter.c:2419 bpf_prog_48159a89cb4a9a16+0x59/0x5e bpf_dispatcher_nop_func include/linux/bpf.h:897 [inline] __bpf_prog_run include/linux/filter.h:596 [inline] bpf_prog_run include/linux/filter.h:603 [inline] bpf_test_run+0x46c/0x890 net/bpf/test_run.c:402 bpf_prog_test_run_skb+0xbdc/0x14c0 net/bpf/test_run.c:1170 bpf_prog_test_run+0x345/0x3c0 kernel/bpf/syscall.c:3648 __sys_bpf+0x43a/0x6c0 kernel/bpf/syscall.c:5005 __do_sys_bpf kernel/bpf/syscall.c:5091 [inline] __se_sys_bpf kernel/bpf/syscall.c:5089 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5089 do_syscall_64+0x54/0x70 arch/x86/entry/common.c:48 entry_SYSCALL_64_after_hwframe+0x61/0xc6Quoted 7 lines
The reproducer doesn't really reproduce outside of syzkaller environment, so I'm taking a guess here. It looks like we do generate correct ETH_HLEN-sized packet, but we redirect the packet to the tunneling device. Before we do so, we
__skb_pulll2 header and arrive again atskb->len== 0. Doesn't seem like we can do anything better than having an explicit check after__skb_pull?hmm... I recall there was similar report but I didn't follow those earlier fixes and discussion. Not sure if this has been considered:
If this skb can only happen in thebpf_prog_test_run(?),
how about ensure that the skb will at least have some header after l2 header inbpf_prog_test_run_skb(). Adding some headers/bytes if thedata_size_indoes not have it. This may break some external test cases that somehow has no l3/4?test_progsshould be mostly fine considering they are using the pkt_v[46] innetwork_helpers.h.Quoted 2 lines
Quoted 2 lines
For the previous issue we've added "
skb->len!= 0" check which works for the cases that remove l2.Yeah, I replied on the "bpf: Don't redirect packets with invalid
pkt_len" thread which is hitting the same syzbot report afaict. I don't think that patch is actually fixing it.Quoted 19 lines
Quoted 9 lines
For the ones that don't, I think you're right, and checking at the time of
bpf_prog_test_run_skbcan probably be enough, lemme try (require ETH_HLEN+1 vsETH_HLEN).
For some reason I was under the impression that Lorenz changed the size from 0 to 14 [0], but he went from 14 to 15, so we won't break at least cilium again..
CC'd him just in case.Thanks for the pointer.
The cilium's prog is
SOCKET_FILTER(not l2). It is why the new "skb->len!= 0" test broke it.Quoted 4 lines
Quoted 3 lines
Adding some headers/bytes if the
data_size_indoes not have it. This may break some external test cases that somehow has no l3/4?Quoted 7 lines
Quoted 3 lines
Yeah, idk, this seems like a last resort? I'd prefer to explicitly fail and communicate it back to the user than slap some extra byte and then fail in some other place unpredictably?
If fixing in the fast path in
filter.c, is__bpf_redirect_no_macthe only place that needs this check?bpf_redirect_neigh()looks ok to me since the neigh should have filled the mac header.I took a closer look. This seems to be the only place needed the check, so applied. If it turns out there are other cases caused by test-run generated skb, we will revisit a fix in
test_run.cand the existing tests have to adjust.Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Cc Eric Dumazet <[email protected]>Reported-by [email protected]Signed-off-by Stanislav Fomichev <[email protected]>--- net/core/filter.c | 4 ++++ 1 file changed, 4 insertions(+)net/core/filter.c +0
-0@@ -2126,6 +2126,10 @@ static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev,Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/core/filter.c static int __bpf_redirect_no_mac(struct sk_buff *skb, struct net_device *dev, · line 2132
Quoted 6 lines
net/core/filter.c +4
-0if (mlen) {__skb_pull (skb, mlen); +if (unlikely (!skb->len)) { +kfree_skb (skb); +return -ERANGE ; + }One question, if the "!skb->len" check is deleted from
convert___skb_to_skb(), this "unlikely(!skb->len)" block here has to be moved out of the "if (mlen)"?
Quoted 3 lines
I see, yeah, that might be the alternative. I'm assuming
__bpf_redirect_commonis covered by "skb->mac_header>=skb->network_header" check?
It is my understanding also. The same goes for __bpf_redirect_neigh. afaict, __bpf_redirect_no_mac is the only exception that does not have len check.