bpf: fix ktls panic with sockmap and add tests
Subsystems (6)
v2 cover bpf: fix ktls panic with sockmap and add tests
We can reproduce the issue using the existing test program:
'./test_sockmap --ktls'
Or use the selftest I provided, which will cause a panic:
------------[ cut here ]------------ kernel BUG at lib/iov_iter.c:629!
PKRU: 55555554
Call Trace: <TASK> ? die+0x36/0x90 ? do_trap+0xdd/0x100 ? iov_iter_revert+0x178/0x180 ? iov_iter_revert+0x178/0x180 ? do_error_trap+0x7d/0x110 ? iov_iter_revert+0x178/0x180 ? exc_invalid_op+0x50/0x70 ? iov_iter_revert+0x178/0x180 ? asm_exc_invalid_op+0x1a/0x20 ? iov_iter_revert+0x178/0x180 ? iov_iter_revert+0x5c/0x180 tls_sw_sendmsg_locked.isra.0+0x794/0x840 tls_sw_sendmsg+0x52/0x80 ? inet_sendmsg+0x1f/0x70 __sys_sendto+0x1cd/0x200 ? find_held_lock+0x2b/0x80 ? syscall_trace_enter+0x140/0x270 ? __lock_release.isra.0+0x5e/0x170 ? find_held_lock+0x2b/0x80 ? syscall_trace_enter+0x140/0x270 ? lockdep_hardirqs_on_prepare+0xda/0x190 ? ktime_get_coarse_real_ts64+0xc2/0xd0 __x64_sys_sendto+0x24/0x30 do_syscall_64+0x90/0x170
1. It looks like the issue started occurring after bpf being introduced to ktls and later the addition of assertions to iov_iter has caused a panic. If my fix tag is incorrect, please assist me in correcting the fix tag.
2. I make minimal changes for now, it's enough to make ktls work correctly.
---v1->v2: Added more content to the commit message
https://lore.kernel.org/all/[email protected]/#r
---
Jiayuan Chen (2):
bpf: fix ktls panic with sockmap
selftests/bpf: add ktls selftest
net/tls/tls_sw.c | 8 +- .../selftests/bpf/prog_tests/sockmap_ktls.c | 174 +++++++++++++++++- .../selftests/bpf/progs/test_sockmap_ktls.c | 26 +++ 3 files changed, 205 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/test_sockmap_ktls.c
Signature · 1 line
2.47.1
PATCH v2 1/2 bpf: fix ktls panic with sockmap
[ 2172.936997] ------------[ cut here ]------------ [ 2172.936999] kernel BUG at lib/iov_iter.c:629! ...... [ 2172.944996] PKRU: 55555554 [ 2172.945155] Call Trace: [ 2172.945299] <TASK> [ 2172.945428] ? die+0x36/0x90 [ 2172.945601] ? do_trap+0xdd/0x100 [ 2172.945795] ? iov_iter_revert+0x178/0x180 [ 2172.946031] ? iov_iter_revert+0x178/0x180 [ 2172.946267] ? do_error_trap+0x7d/0x110 [ 2172.946499] ? iov_iter_revert+0x178/0x180 [ 2172.946736] ? exc_invalid_op+0x50/0x70 [ 2172.946961] ? iov_iter_revert+0x178/0x180 [ 2172.947197] ? asm_exc_invalid_op+0x1a/0x20 [ 2172.947446] ? iov_iter_revert+0x178/0x180 [ 2172.947683] ? iov_iter_revert+0x5c/0x180 [ 2172.947913] tls_sw_sendmsg_locked.isra.0+0x794/0x840 [ 2172.948206] tls_sw_sendmsg+0x52/0x80 [ 2172.948420] ? inet_sendmsg+0x1f/0x70 [ 2172.948634] __sys_sendto+0x1cd/0x200 [ 2172.948848] ? find_held_lock+0x2b/0x80 [ 2172.949072] ? syscall_trace_enter+0x140/0x270 [ 2172.949330] ? __lock_release.isra.0+0x5e/0x170 [ 2172.949595] ? find_held_lock+0x2b/0x80 [ 2172.949817] ? syscall_trace_enter+0x140/0x270 [ 2172.950211] ? lockdep_hardirqs_on_prepare+0xda/0x190 [ 2172.950632] ? ktime_get_coarse_real_ts64+0xc2/0xd0 [ 2172.951036] __x64_sys_sendto+0x24/0x30 [ 2172.951382] do_syscall_64+0x90/0x170
......
After calling bpf_exec_tx_verdict(), the size of msg_pl->sg may increase, e.g., when the BPF program executes bpf_msg_push_data().
If the BPF program sets cork_bytes and sg.size is smaller than cork_bytes, it will return -ENOSPC and attempt to roll back to the non-zero copy logic. However, during rollback, msg->msg_iter is reset, but since msg_pl->sg.size has been increased, subsequent executions will exceed the actual size of msg_iter.
'''iov_iter_revert(&msg->msg_iter, msg_pl->sg.size - orig_size); '''
The changes in this commit are based on the following considerations:
1. When cork_bytes is set, rolling back to non-zero copy logic is pointless and can directly go to zero-copy logic.
2. We can not calculate the correct number of bytes to revert msg_iter.
Assume the original data is "abcdefgh" (8 bytes), and after 3 pushes by the BPF program, it becomes 11-byte data: "abc?de?fgh?". Then, we set cork_bytes to 6, which means the first 6 bytes have been processed, and the remaining 5 bytes "?fgh?" will be cached until the length meets the cork_bytes requirement.
However, some data in "?fgh?" is not within 'sg->msg_iter' (but in msg_pl instead), especially the data "?" we pushed.
So it doesn't seem as simple as just reverting through an offset of msg_iter.
3. For non-TLS sockets in tcp_bpf_sendmsg, when a "cork" situation occurs, the user-space send() doesn't return an error, and the returned length is the same as the input length parameter, even if some data is cached.
Additionally, I saw that the current non-zero-copy logic for handling corking is written as:
'''
line 1177
else if (ret != -EAGAIN) {
if (ret == -ENOSPC)
ret = 0;
goto send_end;'''
So it's ok to just return 'copied' without error when a "cork" situation occurs.
--- net/tls/tls_sw.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
net/tls/tls_sw.c +6 -2
@@ -1120,9 +1120,13 @@ static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, num_async++; else if (ret == -ENOMEM) goto wait_for_memory; - else if (ctx->open_rec && ret == -ENOSPC) + else if (ctx->open_rec && ret == -ENOSPC) { +if (msg_pl->cork_bytes) { + ret =0 ; + goto send_end; + } goto rollback_iter; - else if (ret != -EAGAIN) + } else if (ret != -EAGAIN) goto send_end; }continue ;
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v2 1/2] bpf: fix ktls panic with sockmap
Quoted 32 lines · John Fastabend
From: Jiayuan Chen <[email protected]>
[ 2172.936997] ------------[ cut here ]------------ [ 2172.936999] kernel BUG at lib/iov_iter.c:629! ...... [ 2172.944996] PKRU: 55555554 [ 2172.945155] Call Trace: [ 2172.945299] <TASK> [ 2172.945428] ? die+0x36/0x90 [ 2172.945601] ? do_trap+0xdd/0x100 [ 2172.945795] ? iov_iter_revert+0x178/0x180 [ 2172.946031] ? iov_iter_revert+0x178/0x180 [ 2172.946267] ? do_error_trap+0x7d/0x110 [ 2172.946499] ? iov_iter_revert+0x178/0x180 [ 2172.946736] ? exc_invalid_op+0x50/0x70 [ 2172.946961] ? iov_iter_revert+0x178/0x180 [ 2172.947197] ? asm_exc_invalid_op+0x1a/0x20 [ 2172.947446] ? iov_iter_revert+0x178/0x180 [ 2172.947683] ? iov_iter_revert+0x5c/0x180 [ 2172.947913] tls_sw_sendmsg_locked.isra.0+0x794/0x840 [ 2172.948206] tls_sw_sendmsg+0x52/0x80 [ 2172.948420] ? inet_sendmsg+0x1f/0x70 [ 2172.948634] __sys_sendto+0x1cd/0x200 [ 2172.948848] ? find_held_lock+0x2b/0x80 [ 2172.949072] ? syscall_trace_enter+0x140/0x270 [ 2172.949330] ? __lock_release.isra.0+0x5e/0x170 [ 2172.949595] ? find_held_lock+0x2b/0x80 [ 2172.949817] ? syscall_trace_enter+0x140/0x270 [ 2172.950211] ? lockdep_hardirqs_on_prepare+0xda/0x190 [ 2172.950632] ? ktime_get_coarse_real_ts64+0xc2/0xd0 [ 2172.951036] __x64_sys_sendto+0x24/0x30 [ 2172.951382] do_syscall_64+0x90/0x170
Quoted 1 line
......
Sorry for the broken send there I hit send on accident. New laptop doesn't have all the right config yet.
Quoted 7 lines
Fixes fcb14cb1bdac ("new iov_iter flavour - ITER_UBUF")Fixes d3b18ad31f93 ("tls: add bpf support to sk_msg handling")Signed-off-by Jiayuan Chen <[email protected]>--- net/tls/tls_sw.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
I tend to agree this is not a good situation. Returning 0 is probably suspect as well and likely breaks some applications. But considering we already have this behavior I think its best not to change here if its not causing trouble.
And not panic'ing is clearly better. So...
Quoted 25 lines
net/tls/tls_sw.c +6
-2@@ -1120,9 +1120,13 @@ static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, num_async++; else if (ret == -ENOMEM) goto wait_for_memory; - else if (ctx->open_rec && ret == -ENOSPC) + else if (ctx->open_rec && ret == -ENOSPC) { +if (msg_pl->cork_bytes) { + ret =0 ; + goto send_end; + } goto rollback_iter; - else if (ret != -EAGAIN) + } else if (ret != -EAGAIN) goto send_end; }continue ;Signature · 1 line
2.47.1
Re: [PATCH bpf-next v2 1/2] bpf: fix ktls panic with sockmap
Quoted 5 lines · John Fastabend
I tend to agree this is not a good situation. Returning 0 is probably
suspect as well and likely breaks some applications. But considering
Quoted 8 lines
we already have this behavior I think its best not to change here
if its not causing trouble.
And not panic'ing is clearly better. So...
Acked-by John Fastabend <[email protected]>
Thank you, John, for your review, we're glad to have finally resolved this issue!
Thank.
PATCH v2 2/2 selftests/bpf: add ktls selftest
In reply to Jiayuan Chenadd ktls selftest for sockmap
Test results:
sockmap_ktls/sockmap_ktls disconnect_after_delete IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls update_fails_when_sock_has_ulp IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls disconnect_after_delete IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls update_fails_when_sock_has_ulp IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls disconnect_after_delete IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls update_fails_when_sock_has_ulp IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls disconnect_after_delete IPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktls update_fails_when_sock_has_ulp IPv4 SOCKMAP:OK sockmap_ktls/tls simple offload:OK
sockmap_ktls/tls tx cork:OK
sockmap_ktls/tls tx cork with push:OK
sockmap_ktls/tls simple offload:OK
sockmap_ktls/tls tx cork:OK
sockmap_ktls/tls tx cork with push:OK
sockmap_ktls:OK
--- .../selftests/bpf/prog_tests/sockmap_ktls.c | 174 +++++++++++++++++- .../selftests/bpf/progs/test_sockmap_ktls.c | 26 +++ 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/test_sockmap_ktls.c
tools/testing/selftests/bpf/prog_tests/sockmap_ktls.c +173 -1
@@ -3,13 +3,64 @@/* * Tests for sockmap/sockhash holding kTLS sockets. */ - +#include <error.h> #include <netinet/tcp.h> +#include <linux/tls.h> #include "test_progs.h" +#include "sockmap_helpers.h" +#include "test_skmsg_load_helpers.skel.h" +#include "test_sockmap_ktls.skel.h" #define MAX_TEST_NAME 80#define TCP_ULP 31 +static int init_ktls_pairs (int c,int p) +{ +int err; +struct tls12_crypto_info_aes_gcm_128 crypto_rx; +struct tls12_crypto_info_aes_gcm_128 crypto_tx; + + err =setsockopt (c,IPPROTO_TCP ,TCP_ULP ,"tls" ,strlen ("tls" )); +if (!ASSERT_OK (err,"setsockopt(TCP_ULP)" )) + goto out; + + err =setsockopt (p,IPPROTO_TCP ,TCP_ULP ,"tls" ,strlen ("tls" )); +if (!ASSERT_OK (err,"setsockopt(TCP_ULP)" )) + goto out; + +memset (&crypto_rx,0 ,sizeof (crypto_rx)); +memset (&crypto_tx,0 ,sizeof (crypto_tx)); + crypto_rx.info.version =TLS_1_2_VERSION ; + crypto_tx.info.version =TLS_1_2_VERSION ; + crypto_rx.info.cipher_type =TLS_CIPHER_AES_GCM_128 ; + crypto_tx.info.cipher_type =TLS_CIPHER_AES_GCM_128 ; + + err =setsockopt (c,SOL_TLS ,TLS_TX , &crypto_tx,sizeof (crypto_tx)); +if (!ASSERT_OK (err,"setsockopt(TLS_TX)" )) + goto out; + + err =setsockopt (p,SOL_TLS ,TLS_RX , &crypto_rx,sizeof (crypto_rx)); +if (!ASSERT_OK (err,"setsockopt(TLS_RX)" )) + goto out; +return 0 ; +out: +return -1 ; +} + +static int create_ktls_pairs (int family,int sotype,int *c,int *p) +{ +int err; + + err =create_pair (family, sotype, c, p); +if (!ASSERT_OK (err,"create_pair()" )) +return -1 ; + + err =init_ktls_pairs (*c, *p); +if (!ASSERT_OK (err,"init_ktls_pairs(c, p)" )) +return -1 ; +return 0 ; +} +static int tcp_server (int family) {int err, s;
@@ -146,6 +197,115 @@ static const char *fmt_test_name(const char *subtest_name, int family,return test_name; } +static void test_sockmap_ktls_offload (int family,int sotype) +{ +int err; +int c =0 , p =0 , sent, recvd; +char msg[12 ] ="hello world\0" ; +char rcv[13 ]; + + err =create_ktls_pairs (family, sotype, &c, &p); +if (!ASSERT_OK (err,"create_ktls_pairs()" )) + goto out; + + sent =send (c, msg,sizeof (msg),0 ); +if (!ASSERT_OK (err,"send(msg)" )) + goto out; + + recvd =recv (p, rcv,sizeof (rcv),0 ); +if (!ASSERT_OK (err,"recv(msg)" ) || + !ASSERT_EQ (recvd, sent,"length mismatch" )) + goto out; + +ASSERT_OK (memcmp (msg, rcv,sizeof (msg)),"data mismatch" ); + +out: +if (c) +close (c); +if (p) +close (p); +} + +static void test_sockmap_ktls_tx_cork (int family,int sotype,bool push) +{ +int err, off; +int i, j; +int start_push =0 , push_len =0 ; +int c =0 , p =0 , one =1 , sent, recvd; +int prog_fd, map_fd; +char msg[12 ] ="hello world\0" ; +char rcv[20 ] = {0 }; +struct test_sockmap_ktls *skel; + + skel =test_sockmap_ktls__open_and_load (); +if (!ASSERT_TRUE (skel,"open ktls skel" )) +return ; + + err =create_pair (family, sotype, &c, &p); +if (!ASSERT_OK (err,"create_pair()" )) + goto out; + + prog_fd =bpf_program__fd (skel->progs.prog_sk_policy); + map_fd =bpf_map__fd (skel->maps.sock_map); + + err =bpf_prog_attach (prog_fd, map_fd,BPF_SK_MSG_VERDICT ,0 ); +if (!ASSERT_OK (err,"bpf_prog_attach sk msg" )) + goto out; + + err =bpf_map_update_elem (map_fd, &one, &c,BPF_NOEXIST ); +if (!ASSERT_OK (err,"bpf_map_update_elem(c)" )) + goto out; + + err =init_ktls_pairs (c, p); +if (!ASSERT_OK (err,"init_ktls_pairs(c, p)" )) + goto out; + + skel->bss->cork_byte =sizeof (msg); +if (push) { + start_push =1 ; + push_len =2 ; + } + skel->bss->push_start = start_push; + skel->bss->push_end = push_len; + + off =sizeof (msg) /2 ; + sent =send (c, msg, off,0 ); +if (!ASSERT_EQ (sent, off,"send(msg)" )) + goto out; + + recvd =recv_timeout (p, rcv,sizeof (rcv),MSG_DONTWAIT ,1 ); +if (!ASSERT_EQ (-1 , recvd,"expected no data" )) + goto out; + +/* send remaining msg */ + sent =send (c, msg + off,sizeof (msg) - off,0 ); +if (!ASSERT_EQ (sent,sizeof (msg) - off,"send remaining data" )) + goto out; + + recvd =recv_timeout (p, rcv,sizeof (rcv),MSG_DONTWAIT ,1 ); +if (!ASSERT_OK (err,"recv(msg)" ) || + !ASSERT_EQ (recvd,sizeof (msg) + push_len,"check length mismatch" )) + goto out; + +for (i =0 , j =0 ; i < recvd;) { +/* skip checking the data that has been pushed in */ +if (i >= start_push && i <= start_push + push_len -1 ) { + i++; +continue ; + } +if (!ASSERT_EQ (rcv[i], msg[j],"data mismatch" )) + goto out; + i++; + j++; + } +out: +if (c) +close (c); +if (p) +close (p); +test_sockmap_ktls__destroy (skel); +} +static void run_tests (int family,enum bpf_map_type map_type) {int map;
@@ -162,10 +322,22 @@ static void run_tests(int family, enum bpf_map_type map_type)close (map); } +static void run_ktls_test (int family,int sotype) +{ +if (test__start_subtest ("tls simple offload" )) +test_sockmap_ktls_offload (family, sotype); +if (test__start_subtest ("tls tx cork" )) +test_sockmap_ktls_tx_cork (family, sotype, false); +if (test__start_subtest ("tls tx cork with push" )) +test_sockmap_ktls_tx_cork (family, sotype, true); +} +void test_sockmap_ktls (void ) {run_tests (AF_INET ,BPF_MAP_TYPE_SOCKMAP );run_tests (AF_INET ,BPF_MAP_TYPE_SOCKHASH );run_tests (AF_INET6 ,BPF_MAP_TYPE_SOCKMAP );run_tests (AF_INET6 ,BPF_MAP_TYPE_SOCKHASH ); +run_ktls_test (AF_INET ,SOCK_STREAM ); +run_ktls_test (AF_INET6 ,SOCK_STREAM ); }
tools/testing/selftests/bpf/progs/test_sockmap_ktls.c new file +26 -0
@@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +int cork_byte; +int push_start; +int push_end; + +struct { + __uint(type, BPF_MAP_TYPE_SOCKMAP); + __uint(max_entries, 20); + __type(key, int); + __type(value, int); +} sock_map SEC(".maps"); + +SEC("sk_msg") +int prog_sk_policy (struct sk_msg_md *msg) +{ +if (cork_byte >0 ) +bpf_msg_cork_bytes (msg, cork_byte); +if (push_start >0 && push_end >0 ) +bpf_msg_push_data (msg, push_start, push_end,0 ); + +return SK_PASS ; +}
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v2 2/2] selftests/bpf: add ktls selftest
Quoted 19 lines · Jiayuan Chen
add ktls selftest for sockmap
Test results:
sockmap_ktls/sockmap_ktlsdisconnect_after_deleteIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsupdate_fails_when_sock_has_ulpIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsdisconnect_after_deleteIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsupdate_fails_when_sock_has_ulpIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsdisconnect_after_deleteIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsupdate_fails_when_sock_has_ulpIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsdisconnect_after_deleteIPv4 SOCKMAP:OK sockmap_ktls/sockmap_ktlsupdate_fails_when_sock_has_ulpIPv4 SOCKMAP:OK sockmap_ktls/tls simple offload:OK
sockmap_ktls/tls tx cork:OK
sockmap_ktls/tls tx cork with push:OK
sockmap_ktls/tls simple offload:OK
sockmap_ktls/tls tx cork:OK
sockmap_ktls/tls tx cork with push:OK
sockmap_ktls:OK
Quoted 2 lines
Signed-off-by Jiayuan Chen <[email protected]>---
Thanks Jiayuan sorry for the _very_ long delay this fell of my list and I missed it. Thanks again!
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 8 lines · On Wed, 19 Feb 2025 13:20:13 +0800 you wrote:
We can reproduce the issue using the existing test program:
'./test_sockmap --ktls'
Or use the selftest I provided, which will cause a panic:------------[ cut here ]------------ kernel BUG at lib/iov_iter.c:629![...]
Here is the summary with links:
- [bpf-next,v2,1/2] bpf: fix ktls panic with sockmap
https://git.kernel.org/bpf/bpf-next/c/54a3ecaeeeae - [bpf-next,v2,2/2] selftests/bpf: add ktls selftest https://git.kernel.org/bpf/bpf-next/c/05ebde1bcb50
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html