bpf, sockmap: Fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
PATCH 1/1 bpf, sockmap: Fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
From: Wang Yufen <[email protected]>
[ Upstream commit d8616ee2affcff37c5d315310da557a694a3303d ]
During TCP sockmap redirect pressure test, the following warning is triggered:
WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0 CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9 Call Trace: inet_csk_destroy_sock+0x55/0x110 inet_csk_listen_stop+0xbb/0x380 tcp_close+0x41b/0x480 inet_release+0x42/0x80 __sock_release+0x3d/0xa0 sock_close+0x11/0x20 __fput+0x9d/0x240 task_work_run+0x62/0x90 exit_to_user_mode_prepare+0x110/0x120 syscall_exit_to_user_mode+0x27/0x190 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The reason we observed is that:
When the listener is closing, a connection may have completed the three-way handshake but not accepted, and the client has sent some packets. The child sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(), but psocks of child sks have not released.
To fix, add sock_map_destroy to release psocks.
Stable-dep-of: 8bbabb3fddcd ("bpf, sock_map: Move cancel_work_sync() out of sock lock")
[Conflict in include/linux/bpf.h due to function declaration position and remove non-existed sk_psock_stop helper from sock_map_destroy.]
---
background:
@stable team:
This backport has 2 changes compared to the original patch:
- fix conflict due to sock_map_destroy declaration position in include/linux/bpf.h;
- remove the non-existed sk_psock_stop helper from sock_map_destroy. This helper is introduced by 799aa7f98d53 ("skmsg: Avoid lock_sock() in sk_psock_backlog()") after v5.10, it is not a fix and hard to backport. Considering that what did in sk_psock_stop is done in sk_psock_drop and neither sock_map_close nor sock_map_unhash in v5.10 introduces sk_psock_stop, I removed it from sock_map_destroy too.
I tested it in my environment, the regression was gone.
Cc: Wang Yufen <[email protected]>
@Yufen, if I missed anything, please point it out, thanks!
--- include/linux/bpf.h | 1 + include/linux/skmsg.h | 1 + net/core/skmsg.c | 1 + net/core/sock_map.c | 22 ++++++++++++++++++++++ net/ipv4/tcp_bpf.c | 1 + 5 files changed, 26 insertions(+)
include/linux/bpf.h +1 -0
@@ -1800,6 +1800,7 @@ int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog);int sock_map_prog_detach (const union bpf_attr *attr,enum bpf_prog_type ptype);int sock_map_update_elem_sys (struct bpf_map *map,void *key,void *value,u64 flags);void sock_map_unhash (struct sock *sk); +void sock_map_destroy (struct sock *sk);void sock_map_close (struct sock *sk,long timeout);#else static inline int sock_map_prog_update(struct bpf_map *map,
include/linux/skmsg.h +1 -0
@@ -98,6 +98,7 @@ struct sk_psock {spinlock_t link_lock;refcount_t refcnt;void (*saved_unhash)(struct sock *sk); +void (*saved_destroy)(struct sock *sk);void (*saved_close)(struct sock *sk,long timeout);void (*saved_write_space)(struct sock *sk);struct proto *sk_proto;
net/core/skmsg.c +1 -0
@@ -599,6 +599,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node) psock->eval = __SK_NONE; psock->sk_proto = prot; psock->saved_unhash = prot->unhash; + psock->saved_destroy = prot->destroy; psock->saved_close = prot->close; psock->saved_write_space = sk->sk_write_space;
net/core/sock_map.c +22 -0
@@ -1566,6 +1566,28 @@ void sock_map_unhash(struct sock *sk)saved_unhash (sk); } +void sock_map_destroy (struct sock *sk) +{ +void (*saved_destroy)(struct sock *sk); +struct sk_psock *psock; + +rcu_read_lock (); + psock =sk_psock_get (sk); +if (unlikely (!psock)) { +rcu_read_unlock (); +if (sk->sk_prot->destroy) + sk->sk_prot->destroy (sk); +return ; + } + + saved_destroy = psock->saved_destroy; +sock_map_remove_links (sk, psock); +rcu_read_unlock (); +sk_psock_put (sk, psock); +saved_destroy (sk); +} +EXPORT_SYMBOL_GPL (sock_map_destroy); +void sock_map_close (struct sock *sk,long timeout) {void (*saved_close)(struct sock *sk,long timeout);
net/ipv4/tcp_bpf.c +1 -0
@@ -582,6 +582,7 @@ static void tcp_bpf_rebuild_protos(struct proto prot[TCP_BPF_NUM_CFGS], struct proto *base) { prot[TCP_BPF_BASE ] = *base; + prot[TCP_BPF_BASE ].destroy = sock_map_destroy; prot[TCP_BPF_BASE ].close = sock_map_close; prot[TCP_BPF_BASE ].recvmsg = tcp_bpf_recvmsg; prot[TCP_BPF_BASE ].stream_memory_read = tcp_bpf_stream_read;
Signature · 1 line
2.32.0.3.g01195cf9f
Greg KH
maintainerQuoted 45 lines · Wen Gu
From: Wang Yufen <[email protected]>
[ Upstream commit
d8616ee2affcff37c5d315310da557a694a3303d]During TCP sockmap redirect pressure test, the following warning is triggered:
WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0 CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9 Call Trace: inet_csk_destroy_sock+0x55/0x110 inet_csk_listen_stop+0xbb/0x380 tcp_close+0x41b/0x480 inet_release+0x42/0x80 __sock_release+0x3d/0xa0 sock_close+0x11/0x20 __fput+0x9d/0x240 task_work_run+0x62/0x90 exit_to_user_mode_prepare+0x110/0x120 syscall_exit_to_user_mode+0x27/0x190 entry_SYSCALL_64_after_hwframe+0x44/0xa9The reason we observed is that:
When the listener is closing, a connection may have completed the three-way handshake but not accepted, and the client has sent some packets. The child sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(), but psocks of child sks have not released.
To fix, add
sock_map_destroyto release psocks.Signed-off-by Wang Yufen <[email protected]>Signed-off-by Daniel Borkmann <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Acked-by Jakub Sitnicki <[email protected]>Acked-by John Fastabend <[email protected]>Stable-dep-of:
8bbabb3fddcd("bpf,sock_map: Movecancel_work_sync()out of sock lock")Signed-off-by Sasha Levin <[email protected]>[Conflict in
include/linux/bpf.hdue to function declaration position and remove non-existedsk_psock_stophelper fromsock_map_destroy.]Signed-off-by Wen Gu <[email protected]>---
background:
Quoted 9 lines
@stable team:
This backport has 2 changes compared to the original patch:
- fix conflict due tosock_map_destroydeclaration position ininclude/linux/bpf.h;
- remove the non-existedsk_psock_stophelper fromsock_map_destroy. This helper is introduced by799aa7f98d53("skmsg: Avoidlock_sock()insk_psock_backlog()") after v5.10, it is not a fix and hard to backport. Considering that what did insk_psock_stopis done insk_psock_dropand neithersock_map_closenorsock_map_unhashin v5.10 introducessk_psock_stop, I removed it fromsock_map_destroytoo.
I tested it in my environment, the regression was gone.
Now queued up, thanks.
greg k-h