bpf, sockmap: Fix some issues with using apply_bytes
Subsystems (8)
cover bpf, sockmap: Fix some issues with using apply_bytes
Patch 0001~0003 fixes three issues with using apply_bytes when redirecting. Patch 0004 adds ingress tests for txmsg with apply_bytes in selftests.
Pengcheng Yang (4):
bpf, sockmap: Fix repeated calls to sock_put() when msg has more_data bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes bpf, sockmap: Fix data loss caused by using apply_bytes on ingress
redirect
selftests/bpf: Add ingress tests for txmsg with apply_bytes
include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 9 +++++++-- net/tls/tls_sw.c | 1 + tools/testing/selftests/bpf/test_sockmap.c | 18 ++++++++++++++++++ 5 files changed, 28 insertions(+), 2 deletions(-)
Signature · 1 line
1.8.3.1
PATCH RESEND 1/4 bpf, sockmap: Fix repeated calls to sock_put() when msg has more_data
In tcp_bpf_send_verdict() redirection, the eval variable is assigned to __SK_REDIRECT after the apply_bytes data is sent, if msg has more_data, sock_put() will be called multiple times.
We should reset the eval variable to __SK_NONE every time more_data starts.
This causes:
IPv4: Attempt to release TCP socket in state 1 00000000b4c925d7
------------[ cut here ]------------ refcount_t: addition on 0; use-after-free. WARNING: CPU: 5 PID: 4482 at lib/refcount.c:25 refcount_warn_saturate+0x7d/0x110 Modules linked in: CPU: 5 PID: 4482 Comm: sockhash_bypass Kdump: loaded Not tainted 6.0.0 #1 Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014 Call Trace: <TASK> __tcp_transmit_skb+0xa1b/0xb90 ? __alloc_skb+0x8c/0x1a0 ? __kmalloc_node_track_caller+0x184/0x320 tcp_write_xmit+0x22a/0x1110 __tcp_push_pending_frames+0x32/0xf0 do_tcp_sendpages+0x62d/0x640 tcp_bpf_push+0xae/0x2c0 tcp_bpf_sendmsg_redir+0x260/0x410 ? preempt_count_add+0x70/0xa0 tcp_bpf_send_verdict+0x386/0x4b0 tcp_bpf_sendmsg+0x21b/0x3b0 sock_sendmsg+0x58/0x70 __sys_sendto+0xfa/0x170 ? xfd_validate_state+0x1d/0x80 ? switch_fpu_return+0x59/0xe0 __x64_sys_sendto+0x24/0x30 do_syscall_64+0x37/0x90 entry_SYSCALL_64_after_hwframe+0x63/0xcd
--- net/ipv4/tcp_bpf.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_bpf.c +2 -1
@@ -279,7 +279,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,bool cork = false, enospc =sk_msg_full (msg);struct sock *sk_redir;u32 tosend, origsize, sent, delta =0 ; -u32 eval = __SK_NONE; +u32 eval;int ret; more_data:
@@ -310,6 +310,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, tosend = msg->sg.size;if (psock->apply_bytes && psock->apply_bytes < tosend) tosend = psock->apply_bytes; + eval = __SK_NONE;switch (psock->eval) {case __SK_PASS:
Signature · 1 line
1.8.3.1
RE: [PATCH RESEND bpf 1/4] bpf, sockmap: Fix repeated calls to sock_put() when msg has more_data
Quoted 36 lines · Pengcheng Yang
In
tcp_bpf_send_verdict()redirection, the eval variable is assigned to__SK_REDIRECTafter theapply_bytesdata is sent, if msg hasmore_data,sock_put()will be called multiple times.
We should reset the eval variable to__SK_NONEevery timemore_datastarts.This causes:
IPv4: Attempt to release TCP socket in state 1
00000000b4c925d7------------[ cut here ]------------ refcount_t: addition on 0; use-after-free. WARNING: CPU: 5 PID: 4482 at lib/refcount.c:25 refcount_warn_saturate+0x7d/0x110 Modules linked in: CPU: 5 PID: 4482 Comm: sockhash_bypass Kdump: loaded Not tainted 6.0.0 #1 Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014 Call Trace: <TASK> __tcp_transmit_skb+0xa1b/0xb90 ? __alloc_skb+0x8c/0x1a0 ? __kmalloc_node_track_caller+0x184/0x320 tcp_write_xmit+0x22a/0x1110 __tcp_push_pending_frames+0x32/0xf0 do_tcp_sendpages+0x62d/0x640 tcp_bpf_push+0xae/0x2c0 tcp_bpf_sendmsg_redir+0x260/0x410 ? preempt_count_add+0x70/0xa0 tcp_bpf_send_verdict+0x386/0x4b0 tcp_bpf_sendmsg+0x21b/0x3b0 sock_sendmsg+0x58/0x70 __sys_sendto+0xfa/0x170 ? xfd_validate_state+0x1d/0x80 ? switch_fpu_return+0x59/0xe0 __x64_sys_sendto+0x24/0x30 do_syscall_64+0x37/0x90 entry_SYSCALL_64_after_hwframe+0x63/0xcd
Quoted 2 lines
Fixes cd9733f5d75c ("tcp_bpf: Fix one concurrency problem in the tcp_bpf_send_verdict function")Signed-off-by Pengcheng Yang <[email protected]>
Thanks.
PATCH RESEND 2/4 bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
In reply to Pengcheng YangWhen redirecting, we use sk_msg_to_ingress() to get the BPF_F_INGRESS flag from the msg->flags. If apply_bytes is used and it is larger than the current data being processed, sk_psock_msg_verdict() will not be called when sendmsg() is called again. At this time, the msg->flags is 0, and we lost the BPF_F_INGRESS flag.
So we need to save the BPF_F_INGRESS flag in sk_psock and assign it to msg->flags before redirection.
--- include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 1 + net/tls/tls_sw.c | 1 + 4 files changed, 4 insertions(+)
include/linux/skmsg.h +1 -0
@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +u32 flags;struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )
net/core/skmsg.c +1 -0
@@ -888,6 +888,7 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,if (psock->sk_redir)sock_put (psock->sk_redir); psock->sk_redir = msg->sk_redir; + psock->flags = msg->flags;if (!psock->sk_redir) { ret = __SK_DROP; goto out;
net/ipv4/tcp_bpf.c +1 -0
@@ -323,6 +323,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;sk_msg_apply_bytes (psock, tosend);if (!psock->apply_bytes) {/* Clean up before releasing the sock lock. */
net/tls/tls_sw.c +1 -0
@@ -838,6 +838,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;memcpy (&msg_redir, msg,sizeof (*msg));if (msg->apply_bytes < send) msg->apply_bytes =0 ;
Signature · 1 line
1.8.3.1
John Fastabend
maintainerRE: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
Quoted 47 lines · Pengcheng Yang
When redirecting, we use
sk_msg_to_ingress()to get theBPF_F_INGRESSflag from themsg->flags. Ifapply_bytesis used and it is larger than the current data being processed,sk_psock_msg_verdict()will not be called whensendmsg()is called again. At this time, themsg->flagsis 0, and we lost theBPF_F_INGRESSflag.So we need to save the
BPF_F_INGRESSflag insk_psockand assign it tomsg->flagsbefore redirection.Fixes 8934ce2fd081 ("bpf: sockmap redirect ingress support")Signed-off-by Pengcheng Yang <[email protected]>--- include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 1 + net/tls/tls_sw.c | 1 + 4 files changed, 4 insertions(+)include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +u32 flags;struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )net/core/skmsg.c +1
-0@@ -888,6 +888,7 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,if (psock->sk_redir)sock_put (psock->sk_redir); psock->sk_redir = msg->sk_redir; + psock->flags = msg->flags;if (!psock->sk_redir) { ret = __SK_DROP; goto out;net/ipv4/tcp_bpf.c +0
-0
net/ipv4/tcp_bpf.c static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, · line 329
Quoted 7 lines
net/ipv4/tcp_bpf.c +1
-0@@ -323,6 +323,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;sk_msg_apply_bytes (psock, tosend);if (!psock->apply_bytes) {/* Clean up before releasing the sock lock. */
^^^^^^^^^^^^^^^
In this block reposted here with the rest of the block
if (!psock->apply_bytes) {
/* Clean up before releasing the sock lock. */
eval = psock->eval;
psock->eval = __SK_NONE;
psock->sk_redir = NULL;
}Now that we have a psock->flags we should clera that as well right?
Re: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
Quoted 67 lines · John Fastabend
Quoted 47 lines · Pengcheng Yang
When redirecting, we use
sk_msg_to_ingress()to get theBPF_F_INGRESSflag from themsg->flags. Ifapply_bytesis used and it is larger than the current data being processed,sk_psock_msg_verdict()will not be called whensendmsg()is called again. At this time, themsg->flagsis 0, and we lost theBPF_F_INGRESSflag.So we need to save the
BPF_F_INGRESSflag insk_psockand assign it tomsg->flagsbefore redirection.Fixes 8934ce2fd081 ("bpf: sockmap redirect ingress support")Signed-off-by Pengcheng Yang <[email protected]>--- include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 1 + net/tls/tls_sw.c | 1 + 4 files changed, 4 insertions(+)include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +u32 flags;struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )net/core/skmsg.c +1
-0@@ -888,6 +888,7 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,if (psock->sk_redir)sock_put (psock->sk_redir); psock->sk_redir = msg->sk_redir; + psock->flags = msg->flags;if (!psock->sk_redir) { ret = __SK_DROP; goto out;net/ipv4/tcp_bpf.c +0
-0net/ipv4/tcp_bpf.c static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, · line 329
Quoted 7 lines
net/ipv4/tcp_bpf.c +1
-0@@ -323,6 +323,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;sk_msg_apply_bytes (psock, tosend);if (!psock->apply_bytes) {/* Clean up before releasing the sock lock. */ ^^^^^^^^^^^^^^^In this block reposted here with the rest of the block
if (!psock->apply_bytes) { /* Clean up before releasing the sock lock. */ eval = psock->eval; psock->eval = __SK_NONE; psock->sk_redir = NULL; }
Quoted 2 lines
Now that we have a
psock->flagswe should clera that as well right?
According to my understanding, it is not necessary (but can) to clear psock->flags here, because psock->flags will be overwritten by msg->flags at the beginning of each redirection (in sk_psock_msg_verdict()).
Jakub Sitnicki
maintainerRe: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
Quoted 68 lines · Pengcheng Yang
Quoted 67 lines · John Fastabend
Quoted 47 lines · Pengcheng Yang
When redirecting, we use
sk_msg_to_ingress()to get theBPF_F_INGRESSflag from themsg->flags. Ifapply_bytesis used and it is larger than the current data being processed,sk_psock_msg_verdict()will not be called whensendmsg()is called again. At this time, themsg->flagsis 0, and we lost theBPF_F_INGRESSflag.So we need to save the
BPF_F_INGRESSflag insk_psockand assign it tomsg->flagsbefore redirection.Fixes 8934ce2fd081 ("bpf: sockmap redirect ingress support")Signed-off-by Pengcheng Yang <[email protected]>--- include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 1 + net/tls/tls_sw.c | 1 + 4 files changed, 4 insertions(+)include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +u32 flags;struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )net/core/skmsg.c +1
-0@@ -888,6 +888,7 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,if (psock->sk_redir)sock_put (psock->sk_redir); psock->sk_redir = msg->sk_redir; + psock->flags = msg->flags;if (!psock->sk_redir) { ret = __SK_DROP; goto out;net/ipv4/tcp_bpf.c +0
-0net/ipv4/tcp_bpf.c static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, · line 329
Quoted 7 lines
net/ipv4/tcp_bpf.c +1
-0@@ -323,6 +323,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;sk_msg_apply_bytes (psock, tosend);if (!psock->apply_bytes) {/* Clean up before releasing the sock lock. */ ^^^^^^^^^^^^^^^In this block reposted here with the rest of the block
if (!psock->apply_bytes) { /* Clean up before releasing the sock lock. */ eval = psock->eval; psock->eval = __SK_NONE; psock->sk_redir = NULL; }
Quoted 6 lines
Quoted 2 lines
Now that we have a
psock->flagswe should clera that as well right?According to my understanding, it is not necessary (but can) to clear
psock->flagshere, becausepsock->flagswill be overwritten bymsg->flagsat the beginning of each redirection (insk_psock_msg_verdict()).
1. We should at least document that psock->flags value can be garbage (undefined) if psock->sk_redir is null.
2. 'flags' is amiguous (flags for what?). I'd suggest to rename to something like redir_flags.
Also, since we don't care about all flags, but just the ingress bit, we should store just that. It's not about size. Less state passed around is easier to reason about. See patch below.
3. Alternatively, I'd turn psock->sk_redir into a tagged pointer, like skb->_sk_redir is. This way all state (pointer & flags) is bundled and managed together. It would be a bigger change. Would have to be split out from this patch set.
--8<--
include/linux/skmsg.h +1 -0
@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +bool redir_ingress;/* undefined if sk_redir is null */ struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )
include/net/tcp.h +2 -2
@@ -2291,8 +2291,8 @@ int tcp_bpf_update_proto(struct sock *sk, struct sk_psock *psock, bool restore);void tcp_bpf_clone (const struct sock *sk,struct sock *newsk);#endif /* CONFIG_BPF_SYSCALL */ -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg,u32 bytes, -int flags); +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags);#endif /* CONFIG_NET_SOCK_MSG */ #if !defined(CONFIG_BPF_SYSCALL ) || !defined(CONFIG_NET_SOCK_MSG )
net/core/skmsg.c +6 -3
@@ -886,13 +886,16 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock, ret =sk_psock_map_verd (ret, msg->sk_redir); psock->apply_bytes = msg->apply_bytes;if (ret == __SK_REDIRECT) { -if (psock->sk_redir) +if (psock->sk_redir) {sock_put (psock->sk_redir); - psock->sk_redir = msg->sk_redir; -if (!psock->sk_redir) { + psock->sk_redir =NULL ; + } +if (!msg->sk_redir) { ret = __SK_DROP; goto out; } + psock->redir_ingress =sk_msg_to_ingress (msg); + psock->sk_redir = msg->sk_redir;sock_hold (psock->sk_redir); } out:
net/ipv4/tcp_bpf.c +4 -4
@@ -131,10 +131,9 @@ static int tcp_bpf_push_locked(struct sock *sk, struct sk_msg *msg,return ret; } -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg, -u32 bytes,int flags) +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags) { -bool ingress =sk_msg_to_ingress (msg);struct sk_psock *psock =sk_psock_get (sk);int ret;
@@ -337,7 +336,8 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,release_sock (sk); origsize = msg->sg.size; - ret =tcp_bpf_sendmsg_redir (sk_redir, msg, tosend, flags); + ret =tcp_bpf_sendmsg_redir (sk_redir, psock->redir_ingress, + msg, tosend, flags); sent = origsize - msg->sg.size;if (eval == __SK_REDIRECT)
net/tls/tls_sw.c +2 -1
@@ -846,7 +846,8 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,sk_msg_return_zero (sk, msg, send); msg->sg.size -= send;release_sock (sk); - err =tcp_bpf_sendmsg_redir (sk_redir, &msg_redir, send, flags); + err =tcp_bpf_sendmsg_redir (sk_redir, psock->redir_ingress, + &msg_redir, send, flags);lock_sock (sk);if (err <0 ) { *copied -=sk_msg_free_nocharge (sk, &msg_redir);
John Fastabend
maintainerRe: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
Quoted 72 lines · Jakub Sitnicki
Quoted 71 lines · Pengcheng Yang
Quoted 69 lines · John Fastabend
Quoted 47 lines · Pengcheng Yang
When redirecting, we use
sk_msg_to_ingress()to get theBPF_F_INGRESSflag from themsg->flags. Ifapply_bytesis used and it is larger than the current data being processed,sk_psock_msg_verdict()will not be called whensendmsg()is called again. At this time, themsg->flagsis 0, and we lost theBPF_F_INGRESSflag.So we need to save the
BPF_F_INGRESSflag insk_psockand assign it tomsg->flagsbefore redirection.Fixes 8934ce2fd081 ("bpf: sockmap redirect ingress support")Signed-off-by Pengcheng Yang <[email protected]>--- include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/ipv4/tcp_bpf.c | 1 + net/tls/tls_sw.c | 1 + 4 files changed, 4 insertions(+)include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +u32 flags;struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )net/core/skmsg.c +1
-0@@ -888,6 +888,7 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,if (psock->sk_redir)sock_put (psock->sk_redir); psock->sk_redir = msg->sk_redir; + psock->flags = msg->flags;if (!psock->sk_redir) { ret = __SK_DROP; goto out;net/ipv4/tcp_bpf.c +0
-0net/ipv4/tcp_bpf.c static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, · line 329
Quoted 7 lines
net/ipv4/tcp_bpf.c +1
-0@@ -323,6 +323,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,break ;case __SK_REDIRECT: sk_redir = psock->sk_redir; + msg->flags = psock->flags;sk_msg_apply_bytes (psock, tosend);if (!psock->apply_bytes) {/* Clean up before releasing the sock lock. */ ^^^^^^^^^^^^^^^In this block reposted here with the rest of the block
if (!psock->apply_bytes) { /* Clean up before releasing the sock lock. */ eval = psock->eval; psock->eval = __SK_NONE; psock->sk_redir = NULL; }Now that we have a
psock->flagswe should clera that as well right?
Quoted 6 lines
Quoted 3 lines
According to my understanding, it is not necessary (but can) to clear
psock->flagshere, becausepsock->flagswill be overwritten bymsg->flagsat the beginning of each redirection (insk_psock_msg_verdict()).1. We should at least document that
psock->flagsvalue can be garbage (undefined) ifpsock->sk_rediris null.
Per v2 I think we should not have garbage flags. Just zero the flags field no point in saving a single insn here IMO.
Quoted 7 lines
2. 'flags' is amiguous (flags for what?). I'd suggest to rename to something like
redir_flags.Also, since we don't care about all flags, but just the ingress bit, we should store just that. It's not about size. Less state passed around is easier to reason about. See patch below.
rename makes sense to me.
Quoted 100 lines
3. Alternatively, I'd turn
psock->sk_redirinto a tagged pointer, likeskb->_sk_rediris. This way all state (pointer & flags) is bundled and managed together. It would be a bigger change. Would have to be split out from this patch set.--8<--
include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +bool redir_ingress;/* undefined if sk_redir is null */ struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )include/net/tcp.h +2
-2@@ -2291,8 +2291,8 @@ int tcp_bpf_update_proto(struct sock *sk, struct sk_psock *psock, bool restore);void tcp_bpf_clone (const struct sock *sk,struct sock *newsk);#endif /* CONFIG_BPF_SYSCALL */ -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg,u32 bytes, -int flags); +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags);#endif /* CONFIG_NET_SOCK_MSG */ #if !defined(CONFIG_BPF_SYSCALL ) || !defined(CONFIG_NET_SOCK_MSG )net/core/skmsg.c +6
-3@@ -886,13 +886,16 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock, ret =sk_psock_map_verd (ret, msg->sk_redir); psock->apply_bytes = msg->apply_bytes;if (ret == __SK_REDIRECT) { -if (psock->sk_redir) +if (psock->sk_redir) {sock_put (psock->sk_redir); - psock->sk_redir = msg->sk_redir; -if (!psock->sk_redir) { + psock->sk_redir =NULL ; + } +if (!msg->sk_redir) { ret = __SK_DROP; goto out; } + psock->redir_ingress =sk_msg_to_ingress (msg); + psock->sk_redir = msg->sk_redir;sock_hold (psock->sk_redir); } out:net/ipv4/tcp_bpf.c +4
-4@@ -131,10 +131,9 @@ static int tcp_bpf_push_locked(struct sock *sk, struct sk_msg *msg,return ret; } -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg, -u32 bytes,int flags) +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags) { -bool ingress =sk_msg_to_ingress (msg);struct sk_psock *psock =sk_psock_get (sk);int ret;@@ -337,7 +336,8 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,release_sock (sk); origsize = msg->sg.size; - ret =tcp_bpf_sendmsg_redir (sk_redir, msg, tosend, flags); + ret =tcp_bpf_sendmsg_redir (sk_redir, psock->redir_ingress, + msg, tosend, flags); sent = origsize - msg->sg.size;if (eval == __SK_REDIRECT)net/tls/tls_sw.c +2
-1@@ -846,7 +846,8 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,sk_msg_return_zero (sk, msg, send); msg->sg.size -= send;release_sock (sk); - err =tcp_bpf_sendmsg_redir (sk_redir, &msg_redir, send, flags); + err =tcp_bpf_sendmsg_redir (sk_redir, psock->redir_ingress, + &msg_redir, send, flags);lock_sock (sk);if (err <0 ) { *copied -=sk_msg_free_nocharge (sk, &msg_redir);
Jakub Sitnicki
maintainerRe: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
Quoted 3 lines · John Fastabend
Quoted 2 lines · Jakub Sitnicki
Quoted 1 line · Pengcheng Yang
John Fastabend <[email protected]> wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Now that we have a
psock->flagswe should clera that as well right?According to my understanding, it is not necessary (but can) to clear
psock->flagshere, becausepsock->flagswill be overwritten bymsg->flagsat the beginning of each redirection (insk_psock_msg_verdict()).
Quoted 5 lines
Quoted 2 lines
1. We should at least document that
psock->flagsvalue can be garbage (undefined) ifpsock->sk_rediris null.Per v2 I think we should not have garbage flags. Just zero the flags field no point in saving a single insn here IMO.
It would make sense to me if zero was not a valid value here. But since it signifies "redirect to egress", we won't be able to tell if it has been reset anyway.
Re: [PATCH RESEND bpf 2/4] bpf, sockmap: Fix missing BPF_F_INGRESS flag when using apply_bytes
In reply to Jakub SitnickiQuoted 103 lines · Jakub Sitnicki
Quoted 9 lines · Pengcheng Yang
Quoted 8 lines · John Fastabend
if (!psock->apply_bytes) { /* Clean up before releasing the sock lock. */ eval = psock->eval; psock->eval = __SK_NONE; psock->sk_redir = NULL; }Quoted 6 lines
Quoted 2 lines
Now that we have a
psock->flagswe should clera that as well right?According to my understanding, it is not necessary (but can) to clear
psock->flagshere, becausepsock->flagswill be overwritten bymsg->flagsat the beginning of each redirection (insk_psock_msg_verdict()).1. We should at least document that
psock->flagsvalue can be garbage (undefined) ifpsock->sk_rediris null.2. 'flags' is amiguous (flags for what?). I'd suggest to rename to something like
redir_flags.Also, since we don't care about all flags, but just the ingress bit, we should store just that. It's not about size. Less state passed around is easier to reason about. See patch below.
3. Alternatively, I'd turn
psock->sk_redirinto a tagged pointer, likeskb->_sk_rediris. This way all state (pointer & flags) is bundled and managed together. It would be a bigger change. Would have to be split out from this patch set.--8<--
include/linux/skmsg.h +1
-0@@ -82,6 +82,7 @@ struct sk_psock {u32 apply_bytes;u32 cork_bytes;u32 eval; +bool redir_ingress;/* undefined if sk_redir is null */ struct sk_msg *cork;struct sk_psock_progs progs;#if IS_ENABLED (CONFIG_BPF_STREAM_PARSER )include/net/tcp.h +2
-2@@ -2291,8 +2291,8 @@ int tcp_bpf_update_proto(struct sock *sk, struct sk_psock *psock, bool restore);void tcp_bpf_clone (const struct sock *sk,struct sock *newsk);#endif /* CONFIG_BPF_SYSCALL */ -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg,u32 bytes, -int flags); +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags);#endif /* CONFIG_NET_SOCK_MSG */ #if !defined(CONFIG_BPF_SYSCALL ) || !defined(CONFIG_NET_SOCK_MSG )net/core/skmsg.c +6
-3@@ -886,13 +886,16 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock, ret =sk_psock_map_verd (ret, msg->sk_redir); psock->apply_bytes = msg->apply_bytes;if (ret == __SK_REDIRECT) { -if (psock->sk_redir) +if (psock->sk_redir) {sock_put (psock->sk_redir); - psock->sk_redir = msg->sk_redir; -if (!psock->sk_redir) { + psock->sk_redir =NULL ; + } +if (!msg->sk_redir) { ret = __SK_DROP; goto out; } + psock->redir_ingress =sk_msg_to_ingress (msg); + psock->sk_redir = msg->sk_redir;sock_hold (psock->sk_redir); } out:net/ipv4/tcp_bpf.c +2
-3@@ -131,10 +131,9 @@ static int tcp_bpf_push_locked(struct sock *sk, struct sk_msg *msg,return ret; } -int tcp_bpf_sendmsg_redir (struct sock *sk,struct sk_msg *msg, -u32 bytes,int flags) +int tcp_bpf_sendmsg_redir (struct sock *sk,bool ingress, +struct sk_msg *msg,u32 bytes,int flags) { -bool ingress =sk_msg_to_ingress (msg);struct sk_psock *psock =sk_psock_get (sk);int ret;
net/ipv4/tcp_bpf.c static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock, · line 339
Quoted 5 lines
net/ipv4/tcp_bpf.c +1
-1@@ -337,7 +336,8 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,release_sock (sk); origsize = msg->sg.size; - ret =tcp_bpf_sendmsg_redir (sk_redir, msg, tosend, flags); + ret = tcp_bpf_sendmsg_redir(sk_redir, psock->redir_ingress,
^^^^^^^
Thanks for such detailed advice.
Here it looks like we need to pre-save the redir_ingress before setting psock->sk_redir to NULL and release_sock.
Quoted 18 lines
net/ipv4/tcp_bpf.c +1
-0+ msg, tosend, flags); sent = origsize - msg->sg.size;if (eval == __SK_REDIRECT)net/tls/tls_sw.c +2
-1@@ -846,7 +846,8 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,sk_msg_return_zero (sk, msg, send); msg->sg.size -= send;release_sock (sk); - err =tcp_bpf_sendmsg_redir (sk_redir, &msg_redir, send, flags); + err =tcp_bpf_sendmsg_redir (sk_redir, psock->redir_ingress, + &msg_redir, send, flags);lock_sock (sk);if (err <0 ) { *copied -=sk_msg_free_nocharge (sk, &msg_redir);
PATCH RESEND 3/4 bpf, sockmap: Fix data loss caused by using apply_bytes on ingress redirect
In reply to Pengcheng YangUse apply_bytes on ingress redirect, when apply_bytes is less than the length of msg data, some data may be skipped and lost in bpf_tcp_ingress().
If there is still data in the scatterlist that has not been consumed, we cannot move the msg iter.
--- net/ipv4/tcp_bpf.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
net/ipv4/tcp_bpf.c +4 -1
@@ -45,8 +45,11 @@ static int bpf_tcp_ingress(struct sock *sk, struct sk_psock *psock, tmp->sg.end = i;if (apply) { apply_bytes -= size; -if (!apply_bytes) +if (!apply_bytes) { +if (sge->length) +sk_msg_iter_var_prev (i);break ; + } } }while (i != msg->sg.end);
Signature · 1 line
1.8.3.1
PATCH RESEND 4/4 selftests/bpf: Add ingress tests for txmsg with apply_bytes
In reply to Pengcheng YangCurrently, the ingress redirect is not covered in "txmsg test apply".
--- tools/testing/selftests/bpf/test_sockmap.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
tools/testing/selftests/bpf/test_sockmap.c +18 -0
@@ -1649,24 +1649,42 @@ static void test_txmsg_apply(int cgrp, struct sockmap_options *opt) { txmsg_pass =1 ; txmsg_redir =0 ; + txmsg_ingress =0 ; txmsg_apply =1 ; txmsg_cork =0 ;test_send_one (opt, cgrp); txmsg_pass =0 ; txmsg_redir =1 ; + txmsg_ingress =0 ; + txmsg_apply =1 ; + txmsg_cork =0 ; +test_send_one (opt, cgrp); + + txmsg_pass =0 ; + txmsg_redir =1 ; + txmsg_ingress =1 ; txmsg_apply =1 ; txmsg_cork =0 ;test_send_one (opt, cgrp); txmsg_pass =1 ; txmsg_redir =0 ; + txmsg_ingress =0 ; + txmsg_apply =1024 ; + txmsg_cork =0 ; +test_send_large (opt, cgrp); + + txmsg_pass =0 ; + txmsg_redir =1 ; + txmsg_ingress =0 ; txmsg_apply =1024 ; txmsg_cork =0 ;test_send_large (opt, cgrp); txmsg_pass =0 ; txmsg_redir =1 ; + txmsg_ingress =1 ; txmsg_apply =1024 ; txmsg_cork =0 ;test_send_large (opt, cgrp);
Signature · 1 line
1.8.3.1
RE: [PATCH RESEND bpf 4/4] selftests/bpf: Add ingress tests for txmsg with apply_bytes
Quoted 4 lines · Pengcheng Yang
Currently, the ingress redirect is not covered in "txmsg test apply".
Signed-off-by Pengcheng Yang <[email protected]>---
Thanks for adding extra test.