udp: avoid false sharing on receive
Subsystems (5)
Paolo Abeni
maintainerv2 cover udp: avoid false sharing on receive
Under high UDP load, the BH processing and the user-space receiver can run on different cores.
The UDP implementation does a lot of effort to avoid false sharing in the receive path, but recent changes to the struct sock layout moved the sk_forward_alloc and the sk_rcvbuf fields on the same cacheline:
/* --- cacheline 4 boundary (256 bytes) --- */
struct sk_buff * tail;
} sk_backlog;
int sk_forward_alloc;
unsigned int sk_reserved_mem;
unsigned int sk_ll_usec;
unsigned int sk_napi_id;
int sk_rcvbuf;sk_forward_alloc is updated by the BH, while sk_rcvbuf is accessed by udp_recvmsg(), causing false sharing.
A possible solution would be to re-order the struct sock fields to avoid the false sharing. Such change is subject to being invalidated by future changes and could have negative side effects on other workload.
Instead this series uses a different approach, touching only the UDP socket layout.
The first patch generalizes the custom setsockopt infrastructure, to allow UDP tracking the buffer size, and the second patch addresses the issue, copying the relevant buffer information into an already hot cacheline.
Overall the above gives a 10% peek throughput increase under UDP flood.
v1 -> v2:
- introduce and use a common helper to initialize the UDP v4/v6 sockets (Kuniyuki)
Paolo Abeni (2):
net: introduce and use custom sockopt socket flag
udp: track the forward memory release threshold in an hot cacheline
include/linux/net.h | 1 + include/linux/udp.h | 3 +++ include/net/udp.h | 9 +++++++++ net/ipv4/udp.c | 18 +++++++++++++++--- net/ipv6/udp.c | 4 ++-- net/mptcp/protocol.c | 4 ++++ net/socket.c | 8 +------- 7 files changed, 35 insertions(+), 12 deletions(-)
Signature · 1 line
2.37.3
Paolo Abeni
maintainerPATCH v2 1/2 net: introduce and use custom sockopt socket flag
We will soon introduce custom setsockopt for UDP sockets, too. Instead of doing even more complex arbitrary checks inside sock_use_custom_sol_socket(), add a new socket flag and set it for the relevant socket types (currently only MPTCP).
--- include/linux/net.h | 1 + net/mptcp/protocol.c | 4 ++++ net/socket.c | 8 +------- 3 files changed, 6 insertions(+), 7 deletions(-)
include/linux/net.h +1 -0
@@ -41,6 +41,7 @@ struct net;#define SOCK_NOSPACE 2#define SOCK_PASSCRED 3#define SOCK_PASSSEC 4 +#define SOCK_CUSTOM_SOCKOPT 5#ifndef ARCH_HAS_SOCKET_TYPES /**
net/mptcp/protocol.c +4 -0
@@ -2708,6 +2708,8 @@ static int mptcp_init_sock(struct sock *sk)if (ret)return ret; +set_bit (SOCK_CUSTOM_SOCKOPT , &sk->sk_socket->flags); +/* fetch the ca name; do it outside __mptcp_init_sock(), so that clone will * propagate the correct value */
@@ -3684,6 +3686,8 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,struct mptcp_subflow_context *subflow;struct sock *newsk = newsock->sk; +set_bit (SOCK_CUSTOM_SOCKOPT , &newsock->flags); +lock_sock (newsk);/* PM/worker can now acquire the first subflow socket
net/socket.c +1 -7
@@ -2199,13 +2199,7 @@ SYSCALL_DEFINE4(recv, int, fd, void __user *, ubuf, size_t, size,static bool sock_use_custom_sol_socket (const struct socket *sock) { -const struct sock *sk = sock->sk; - -/* Use sock->ops->setsockopt() for MPTCP */ -return IS_ENABLED (CONFIG_MPTCP ) && - sk->sk_protocol ==IPPROTO_MPTCP && - sk->sk_type ==SOCK_STREAM && - (sk->sk_family ==AF_INET || sk->sk_family ==AF_INET6 ); +return test_bit (SOCK_CUSTOM_SOCKOPT , &sock->flags); }/*
Signature · 1 line
2.37.3
Re: [PATCH net-next v2 1/2] net: introduce and use custom sockopt socket flag
Quoted 10 lines · Paolo Abeni
We will soon introduce custom setsockopt for UDP sockets, too. Instead of doing even more complex arbitrary checks inside
sock_use_custom_sol_socket(), add a new socket flag and set it for the relevant socket types (currently only MPTCP).Reviewed-by Matthieu Baerts <[email protected]>Signed-off-by Paolo Abeni <[email protected]>---
Re: [PATCH net-next v2 1/2] net: introduce and use custom sockopt socket flag
In reply to Paolo AbeniFrom: Paolo Abeni <[email protected]> Date: Thu, 20 Oct 2022 19:48:51 +0200
Quoted 7 lines
We will soon introduce custom setsockopt for UDP sockets, too. Instead of doing even more complex arbitrary checks inside
sock_use_custom_sol_socket(), add a new socket flag and set it for the relevant socket types (currently only MPTCP).Reviewed-by Matthieu Baerts <[email protected]>Signed-off-by Paolo Abeni <[email protected]>
Quoted 61 lines
--- include/linux/net.h | 1 + net/mptcp/protocol.c | 4 ++++ net/socket.c | 8 +------- 3 files changed, 6 insertions(+), 7 deletions(-)include/linux/net.h +1
-0@@ -41,6 +41,7 @@ struct net;#define SOCK_NOSPACE 2#define SOCK_PASSCRED 3#define SOCK_PASSSEC 4 +#define SOCK_CUSTOM_SOCKOPT 5#ifndef ARCH_HAS_SOCKET_TYPES /** net/mptcp/protocol.c +4
-0@@ -2708,6 +2708,8 @@ static int mptcp_init_sock(struct sock *sk)if (ret)return ret; +set_bit (SOCK_CUSTOM_SOCKOPT , &sk->sk_socket->flags); +/* fetch the ca name; do it outside __mptcp_init_sock(), so that clone will * propagate the correct value */ @@ -3684,6 +3686,8 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,struct mptcp_subflow_context *subflow;struct sock *newsk = newsock->sk; +set_bit (SOCK_CUSTOM_SOCKOPT , &newsock->flags); +lock_sock (newsk);/* PM/worker can now acquire the first subflow socket net/socket.c +1
-7@@ -2199,13 +2199,7 @@ SYSCALL_DEFINE4(recv, int, fd, void __user *, ubuf, size_t, size,static bool sock_use_custom_sol_socket (const struct socket *sock) { -const struct sock *sk = sock->sk; - -/* Use sock->ops->setsockopt() for MPTCP */ -return IS_ENABLED (CONFIG_MPTCP ) && - sk->sk_protocol ==IPPROTO_MPTCP && - sk->sk_type ==SOCK_STREAM && - (sk->sk_family ==AF_INET || sk->sk_family ==AF_INET6 ); +return test_bit (SOCK_CUSTOM_SOCKOPT , &sock->flags); }/* Signature · 1 line
2.37.3
Paolo Abeni
maintainerPATCH v2 2/2 udp: track the forward memory release threshold in an hot cacheline
In reply to Paolo AbeniWhen the receiver process and the BH runs on different cores, udp_rmem_release() experience a cache miss while accessing sk_rcvbuf, as the latter shares the same cacheline with sk_forward_alloc, written by the BH.
With this patch, UDP tracks the rcvbuf value and its update via custom SOL_SOCKET socket options, and copies the forward memory threshold value used by udp_rmem_release() in a different cacheline, already accessed by the above function and uncontended.
Since the UDP socket init operation grown a bit, factor out the common code between v4 and v6 in a shared helper.
Overall the above give a 10% peek throughput increase under UDP flood.
---
v1 -> v2:
- factor out common init helper for udp && udpv6 sock (Kuniyuki)
--- include/linux/udp.h | 3 +++ include/net/udp.h | 9 +++++++++ net/ipv4/udp.c | 18 +++++++++++++++--- net/ipv6/udp.c | 4 ++-- 4 files changed, 29 insertions(+), 5 deletions(-)
include/linux/udp.h +3 -0
@@ -87,6 +87,9 @@ struct udp_sock {/* This field is dirtied by udp_recvmsg() */ int forward_deficit; + +/* This fields follows rcvbuf value, and is touched by udp_recvmsg */ +int forward_threshold; };#define UDP_MAX_SEGMENTS (1 << 6UL)
include/net/udp.h +9 -0
@@ -174,6 +174,15 @@ INDIRECT_CALLABLE_DECLARE(int udpv6_rcv(struct sk_buff *));struct sk_buff *__udp_gso_segment (struct sk_buff *gso_skb,netdev_features_t features,bool is_ipv6); +static inline void udp_lib_init_sock (struct sock *sk) +{ +struct udp_sock *up =udp_sk (sk); + +skb_queue_head_init (&up->reader_queue); + up->forward_threshold = sk->sk_rcvbuf >>2 ; +set_bit (SOCK_CUSTOM_SOCKOPT , &sk->sk_socket->flags); +} +/* hash routines shared between UDPv4/6 and UDP-Litev4/6 */ static inline int udp_lib_hash (struct sock *sk) {
net/ipv4/udp.c +15 -3
@@ -1448,7 +1448,7 @@ static void udp_rmem_release(struct sock *sk, int size, int partial,if (likely (partial)) { up->forward_deficit += size; size = up->forward_deficit; -if (size < (sk->sk_rcvbuf >>2 ) && +if (size <READ_ONCE (up->forward_threshold) && !skb_queue_empty (&up->reader_queue))return ; }else {
@@ -1622,7 +1622,7 @@ static void udp_destruct_sock(struct sock *sk)int udp_init_sock (struct sock *sk) { -skb_queue_head_init (&udp_sk (sk)->reader_queue); +udp_lib_init_sock (sk); sk->sk_destruct = udp_destruct_sock;return 0 ; }
@@ -2671,6 +2671,18 @@ int udp_lib_setsockopt(struct sock *sk, int level, int optname,int err =0 ;int is_udplite =IS_UDPLITE (sk); +if (level ==SOL_SOCKET ) { + err =sk_setsockopt (sk, level, optname, optval, optlen); + +if (optname ==SO_RCVBUF || optname ==SO_RCVBUFFORCE ) { +sockopt_lock_sock (sk); +/* paired with READ_ONCE in udp_rmem_release() */ +WRITE_ONCE (up->forward_threshold, sk->sk_rcvbuf >>2 ); +sockopt_release_sock (sk); + } +return err; + } +if (optlen <sizeof (int ))return -EINVAL ;
@@ -2784,7 +2796,7 @@ EXPORT_SYMBOL(udp_lib_setsockopt);int udp_setsockopt (struct sock *sk,int level,int optname,sockptr_t optval,unsigned int optlen) { -if (level ==SOL_UDP || level ==SOL_UDPLITE ) +if (level ==SOL_UDP || level ==SOL_UDPLITE || level ==SOL_SOCKET )return udp_lib_setsockopt (sk, level, optname, optval, optlen, udp_push_pending_frames);
net/ipv6/udp.c +2 -2
@@ -64,7 +64,7 @@ static void udpv6_destruct_sock(struct sock *sk)int udpv6_init_sock (struct sock *sk) { -skb_queue_head_init (&udp_sk (sk)->reader_queue); +udp_lib_init_sock (sk); sk->sk_destruct = udpv6_destruct_sock;return 0 ; }
@@ -1671,7 +1671,7 @@ void udpv6_destroy_sock(struct sock *sk)int udpv6_setsockopt (struct sock *sk,int level,int optname,sockptr_t optval,unsigned int optlen) { -if (level ==SOL_UDP || level ==SOL_UDPLITE ) +if (level ==SOL_UDP || level ==SOL_UDPLITE || level ==SOL_SOCKET )return udp_lib_setsockopt (sk, level, optname, optval, optlen, udp_v6_push_pending_frames);
Signature · 1 line
2.37.3
Re: [PATCH net-next v2 2/2] udp: track the forward memory release threshold in an hot cacheline
Quoted 11 lines · Paolo Abeni
When the receiver process and the BH runs on different cores,
udp_rmem_release()experience a cache miss while accessingsk_rcvbuf, as the latter shares the same cacheline withsk_forward_alloc, written by the BH.With this patch, UDP tracks the rcvbuf value and its update via custom
SOL_SOCKETsocket options, and copies the forward memory threshold value used byudp_rmem_release()in a different cacheline, already accessed by the above function and uncontended.
Quoted 6 lines
Since the UDP socket init operation grown a bit, factor out the common code between v4 and v6 in a shared helper.
Overall the above give a 10% peek throughput increase under UDP flood.
Signed-off-by Paolo Abeni <[email protected]>
Re: [PATCH net-next v2 2/2] udp: track the forward memory release threshold in an hot cacheline
In reply to Paolo AbeniFrom: Paolo Abeni <[email protected]> Date: Thu, 20 Oct 2022 19:48:52 +0200
Quoted 10 lines
When the receiver process and the BH runs on different cores,
udp_rmem_release()experience a cache miss while accessingsk_rcvbuf, as the latter shares the same cacheline withsk_forward_alloc, written by the BH.With this patch, UDP tracks the rcvbuf value and its update via custom
SOL_SOCKETsocket options, and copies the forward memory threshold value used byudp_rmem_release()in a different cacheline, already accessed by the above function and uncontended.
Quoted 6 lines
Since the UDP socket init operation grown a bit, factor out the common code between v4 and v6 in a shared helper.
Overall the above give a 10% peek throughput increase under UDP flood.
Signed-off-by Paolo Abeni <[email protected]>
Thank you!
Quoted 118 lines
---
v1 -> v2:
- factor out common init helper for udp && udpv6 sock (Kuniyuki)--- include/linux/udp.h | 3 +++ include/net/udp.h | 9 +++++++++ net/ipv4/udp.c | 18 +++++++++++++++--- net/ipv6/udp.c | 4 ++-- 4 files changed, 29 insertions(+), 5 deletions(-)include/linux/udp.h +3
-0@@ -87,6 +87,9 @@ struct udp_sock {/* This field is dirtied by udp_recvmsg() */ int forward_deficit; + +/* This fields follows rcvbuf value, and is touched by udp_recvmsg */ +int forward_threshold; };#define UDP_MAX_SEGMENTS (1 << 6UL)include/net/udp.h +9
-0@@ -174,6 +174,15 @@ INDIRECT_CALLABLE_DECLARE(int udpv6_rcv(struct sk_buff *));struct sk_buff *__udp_gso_segment (struct sk_buff *gso_skb,netdev_features_t features,bool is_ipv6); +static inline void udp_lib_init_sock (struct sock *sk) +{ +struct udp_sock *up =udp_sk (sk); + +skb_queue_head_init (&up->reader_queue); + up->forward_threshold = sk->sk_rcvbuf >>2 ; +set_bit (SOCK_CUSTOM_SOCKOPT , &sk->sk_socket->flags); +} +/* hash routines shared between UDPv4/6 and UDP-Litev4/6 */ static inline int udp_lib_hash (struct sock *sk) {net/ipv4/udp.c +15
-3@@ -1448,7 +1448,7 @@ static void udp_rmem_release(struct sock *sk, int size, int partial,if (likely (partial)) { up->forward_deficit += size; size = up->forward_deficit; -if (size < (sk->sk_rcvbuf >>2 ) && +if (size <READ_ONCE (up->forward_threshold) && !skb_queue_empty (&up->reader_queue))return ; }else {@@ -1622,7 +1622,7 @@ static void udp_destruct_sock(struct sock *sk)int udp_init_sock (struct sock *sk) { -skb_queue_head_init (&udp_sk (sk)->reader_queue); +udp_lib_init_sock (sk); sk->sk_destruct = udp_destruct_sock;return 0 ; }@@ -2671,6 +2671,18 @@ int udp_lib_setsockopt(struct sock *sk, int level, int optname,int err =0 ;int is_udplite =IS_UDPLITE (sk); +if (level ==SOL_SOCKET ) { + err =sk_setsockopt (sk, level, optname, optval, optlen); + +if (optname ==SO_RCVBUF || optname ==SO_RCVBUFFORCE ) { +sockopt_lock_sock (sk); +/* paired with READ_ONCE in udp_rmem_release() */ +WRITE_ONCE (up->forward_threshold, sk->sk_rcvbuf >>2 ); +sockopt_release_sock (sk); + } +return err; + } +if (optlen <sizeof (int ))return -EINVAL ;@@ -2784,7 +2796,7 @@ EXPORT_SYMBOL(udp_lib_setsockopt);int udp_setsockopt (struct sock *sk,int level,int optname,sockptr_t optval,unsigned int optlen) { -if (level ==SOL_UDP || level ==SOL_UDPLITE ) +if (level ==SOL_UDP || level ==SOL_UDPLITE || level ==SOL_SOCKET )return udp_lib_setsockopt (sk, level, optname, optval, optlen, udp_push_pending_frames);net/ipv6/udp.c +2
-2@@ -64,7 +64,7 @@ static void udpv6_destruct_sock(struct sock *sk)int udpv6_init_sock (struct sock *sk) { -skb_queue_head_init (&udp_sk (sk)->reader_queue); +udp_lib_init_sock (sk); sk->sk_destruct = udpv6_destruct_sock;return 0 ; }@@ -1671,7 +1671,7 @@ void udpv6_destroy_sock(struct sock *sk)int udpv6_setsockopt (struct sock *sk,int level,int optname,sockptr_t optval,unsigned int optlen) { -if (level ==SOL_UDP || level ==SOL_UDPLITE ) +if (level ==SOL_UDP || level ==SOL_UDPLITE || level ==SOL_SOCKET )return udp_lib_setsockopt (sk, level, optname, optval, optlen, udp_v6_push_pending_frames);Signature · 1 line
2.37.3
Re: udp: track the forward memory release threshold in an hot cacheline: Tests Results
In reply to Paolo AbeniHi Paolo,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal:
- Success! ✅:
- Task: https://cirrus-ci.com/task/5317274872578048
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5317274872578048/summary/summary.txt
- {"code":404,"message":
- "Can't find artifacts containing file conclusion.txt"}:
- Task: https://cirrus-ci.com/task/6443174779420672
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6443174779420672/summary/summary.txt
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/31ed4dad9833
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-debugFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 8 lines · On Thu, 20 Oct 2022 19:48:50 +0200 you wrote:
Under high UDP load, the BH processing and the user-space receiver can run on different cores.
The UDP implementation does a lot of effort to avoid false sharing in the receive path, but recent changes to the struct sock layout moved the
sk_forward_allocand thesk_rcvbuffields on the same cacheline:[...]
Here is the summary with links:
- [net-next,v2,1/2] net: introduce and use custom sockopt socket flag
https://git.kernel.org/netdev/net-next/c/a5ef058dc4d9 - [net-next,v2,2/2] udp: track the forward memory release threshold in an hot cacheline https://git.kernel.org/netdev/net-next/c/8a3854c7b8e4
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html