net: use ICW for sk_proto->{send,recv}msg
Subsystems (4)
Paolo Abeni
maintainercover net: use ICW for sk_proto->{send,recv}msg
This series extends ICW usage to one of the few remaining spots in fast-path still hitting per packet retpoline overhead, namely the sk_proto->{send,recv}msg calls.
The first 3 patches in this series refactor the existing code so that applying the ICW macros is straight-forward: we demux inet_{recv,send}msg in ipv4 and ipv6 variants so that each of them can easily select the appropriate TCP or UDP direct call. While at it, a new helper is created to avoid excessive code duplication, and the current ICWs for inet_{recv,send}msg are adjusted accordingly.
The last 2 patches really introduce the new ICW use-case, respectively for the ipv6 and the ipv4 code path.
This gives up to 5% performance improvement under UDP flood, and smaller but measurable gains for TCP RR workloads.
Paolo Abeni (5):
inet: factor out inet_send_prepare()
ipv6: provide and use ipv6 specific version for {recv,send}msg
net: adjust socket level ICW to cope with ipv6 variant of
{recv,send}msg
ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
include/net/inet_common.h | 1 + include/net/ipv6.h | 3 +++ net/ipv4/af_inet.c | 33 ++++++++++++++++++++----------- net/ipv6/af_inet6.c | 41 +++++++++++++++++++++++++++++++++++---- net/socket.c | 17 ++++++---------- 5 files changed, 69 insertions(+), 26 deletions(-)
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH 2/5 ipv6: provide and use ipv6 specific version for {recv,send}msg
This will simplify indirect call wrapper invocation in the following patch.
No functional change intended, any - out-of-tree - IPv6 user of inet_{recv,send}msg can keep using the existing functions.
SCTP code still uses the existing version even for ipv6: as this series will not add ICW for SCTP, moving to the new helper would not give any benefit.
The only other in-kernel user of inet_{recv,send}msg is pvcalls_conn_back_read(), but psvcalls explicitly creates only IPv4 socket, so no need to update that code path, too.
--- include/net/ipv6.h | 3 +++ net/ipv6/af_inet6.c | 35 +++++++++++++++++++++++++++++++---- 2 files changed, 34 insertions(+), 4 deletions(-)
include/net/ipv6.h +3 -0
@@ -1089,6 +1089,9 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg);int inet6_hash_connect (struct inet_timewait_death_row *death_row,struct sock *sk); +int inet6_sendmsg (struct socket *sock,struct msghdr *msg,size_t size); +int inet6_recvmsg (struct socket *sock,struct msghdr *msg,size_t size, +int flags);/* * reassembly.c
net/ipv6/af_inet6.c +31 -4
@@ -564,6 +564,33 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) }EXPORT_SYMBOL (inet6_ioctl); +int inet6_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +{ +struct sock *sk = sock->sk; + +if (unlikely (inet_send_prepare (sk))) +return -EAGAIN ; + +return sk->sk_prot->sendmsg (sk, msg, size); +} + +int inet6_recvmsg (struct socket *sock,struct msghdr *msg,size_t size, +int flags) +{ +struct sock *sk = sock->sk; +int addr_len =0 ; +int err; + +if (likely (!(flags &MSG_ERRQUEUE ))) +sock_rps_record_flow (sk); + + err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len); +if (err >=0 ) + msg->msg_namelen = addr_len; +return err; +} +const struct proto_ops inet6_stream_ops = { .family =PF_INET6 , .owner =THIS_MODULE ,
@@ -580,8 +607,8 @@ const struct proto_ops inet6_stream_ops = { .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ .getsockopt = sock_common_getsockopt, /* ok */ - .sendmsg = inet_sendmsg, /* ok */ - .recvmsg = inet_recvmsg, /* ok */ + .sendmsg = inet6_sendmsg, /* retpoline's sake */ + .recvmsg = inet6_recvmsg, /* retpoline's sake */#ifdef CONFIG_MMU .mmap = tcp_mmap,#endif
@@ -614,8 +641,8 @@ const struct proto_ops inet6_dgram_ops = { .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ .getsockopt = sock_common_getsockopt, /* ok */ - .sendmsg = inet_sendmsg, /* ok */ - .recvmsg = inet_recvmsg, /* ok */ + .sendmsg = inet6_sendmsg, /* retpoline's sake */ + .recvmsg = inet6_recvmsg, /* retpoline's sake */ .mmap = sock_no_mmap, .sendpage = sock_no_sendpage, .set_peek_off = sk_set_peek_off,
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH 3/5 net: adjust socket level ICW to cope with ipv6 variant of {recv,send}msg
In reply to Paolo AbeniAfter the previous patch we have ipv{6,4} variants for {recv,send}msg, we should use the generic _INET ICW variant to call into the proper build-in.
This also allows dropping the now unused and rather ugly _INET4 ICW macro
--- net/socket.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
net/socket.c +6 -11
@@ -103,13 +103,6 @@#include <net/busy_poll.h> #include <linux/errqueue.h> -/* proto_ops for ipv4 and ipv6 use the same {recv,send}msg function */ -#if IS_ENABLED (CONFIG_INET ) -#define INDIRECT_CALL_INET4 (f, f1, ...) INDIRECT_CALL_1(f, f1, __VA_ARGS__) -#else -#define INDIRECT_CALL_INET4 (f, f1, ...) f(__VA_ARGS__) -#endif -#ifdef CONFIG_NET_RX_BUSY_POLL unsigned int sysctl_net_busy_read__read_mostly ;unsigned int sysctl_net_busy_poll__read_mostly ;
@@ -643,8 +636,9 @@ INDIRECT_CALLABLE_DECLARE(int inet_sendmsg(struct socket *, struct msghdr *, size_t));static inline int sock_sendmsg_nosec (struct socket *sock,struct msghdr *msg) { -int ret =INDIRECT_CALL_INET4 (sock->ops->sendmsg, inet_sendmsg, sock, - msg,msg_data_left (msg)); +int ret =INDIRECT_CALL_INET (sock->ops->sendmsg, inet6_sendmsg, + inet_sendmsg, sock, msg, +msg_data_left (msg));BUG_ON (ret == -EIOCBQUEUED );return ret; }
@@ -874,8 +868,9 @@ INDIRECT_CALLABLE_DECLARE(int inet_recvmsg(struct socket *, struct msghdr *,static inline int sock_recvmsg_nosec (struct socket *sock,struct msghdr *msg,int flags) { -return INDIRECT_CALL_INET4 (sock->ops->recvmsg, inet_recvmsg, sock, msg, -msg_data_left (msg), flags); +return INDIRECT_CALL_INET (sock->ops->recvmsg, inet6_recvmsg, + inet_recvmsg, sock, msg,msg_data_left (msg), + flags); }/**
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH 1/5 inet: factor out inet_send_prepare()
In reply to Paolo AbeniThe same code is replicated verbatim in multiple places, and the next patches will introduce an additional user for it. Factor out a helper and use it where appropriate. No functional change intended.
--- include/net/inet_common.h | 1 + net/ipv4/af_inet.c | 21 +++++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-)
include/net/inet_common.h +1 -0
@@ -25,6 +25,7 @@ int inet_dgram_connect(struct socket *sock, struct sockaddr *uaddr, int addr_len, int flags);int inet_accept (struct socket *sock,struct socket *newsock,int flags,bool kern); +int inet_send_prepare (struct sock *sk);int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size);ssize_t inet_sendpage (struct socket *sock,struct page *page,int offset,size_t size,int flags);
net/ipv4/af_inet.c +13 -8
@@ -784,10 +784,8 @@ int inet_getname(struct socket *sock, struct sockaddr *uaddr, }EXPORT_SYMBOL (inet_getname); -int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +int inet_send_prepare (struct sock *sk) { -struct sock *sk = sock->sk; -sock_rps_record_flow (sk);/* We may need to bind the socket. */
@@ -795,6 +793,17 @@ int inet_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) inet_autobind(sk))return -EAGAIN ; +return 0 ; +} +EXPORT_SYMBOL_GPL (inet_send_prepare); + +int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +{ +struct sock *sk = sock->sk; + +if (unlikely (inet_send_prepare (sk))) +return -EAGAIN ; +return sk->sk_prot->sendmsg (sk, msg, size); }EXPORT_SYMBOL (inet_sendmsg);
@@ -804,11 +813,7 @@ ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset, {struct sock *sk = sock->sk; -sock_rps_record_flow (sk); - -/* We may need to bind the socket. */ -if (!inet_sk (sk)->inet_num && !sk->sk_prot->no_autobind && -inet_autobind (sk)) +if (unlikely (inet_send_prepare (sk)))return -EAGAIN ;if (sk->sk_prot->sendpage)
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH 4/5 ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
In reply to Paolo AbeniThis avoids an indirect call per syscall for common ipv6 transports
--- net/ipv6/af_inet6.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
net/ipv6/af_inet6.c +9 -3
@@ -564,6 +564,8 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) }EXPORT_SYMBOL (inet6_ioctl); +INDIRECT_CALLABLE_DECLARE(int udpv6_sendmsg(struct sock *, struct msghdr *, +size_t ));int inet6_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) {struct sock *sk = sock->sk;
@@ -571,9 +573,12 @@ int inet6_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)if (unlikely (inet_send_prepare (sk)))return -EAGAIN ; -return sk->sk_prot->sendmsg (sk, msg, size); +return INDIRECT_CALL_2 (sk->sk_prot->sendmsg, tcp_sendmsg, udpv6_sendmsg, + sk, msg, size); } +INDIRECT_CALLABLE_DECLARE(int udpv6_recvmsg(struct sock *, struct msghdr *, +size_t ,int ,int ,int *));int inet6_recvmsg (struct socket *sock,struct msghdr *msg,size_t size,int flags) {
@@ -584,8 +589,9 @@ int inet6_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,if (likely (!(flags &MSG_ERRQUEUE )))sock_rps_record_flow (sk); - err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , - flags & ~MSG_DONTWAIT , &addr_len); + err =INDIRECT_CALL_2 (sk->sk_prot->recvmsg, tcp_recvmsg, udpv6_recvmsg, + sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len);if (err >=0 ) msg->msg_namelen = addr_len;return err;
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH 5/5 ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
In reply to Paolo AbeniThis avoids an indirect call per syscall for common ipv4 transports
--- net/ipv4/af_inet.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
net/ipv4/af_inet.c +9 -3
@@ -797,6 +797,8 @@ int inet_send_prepare(struct sock *sk) }EXPORT_SYMBOL_GPL (inet_send_prepare); +INDIRECT_CALLABLE_DECLARE(int udp_sendmsg(struct sock *, struct msghdr *, +size_t ));int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) {struct sock *sk = sock->sk;
@@ -804,7 +806,8 @@ int inet_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)if (unlikely (inet_send_prepare (sk)))return -EAGAIN ; -return sk->sk_prot->sendmsg (sk, msg, size); +return INDIRECT_CALL_2 (sk->sk_prot->sendmsg, tcp_sendmsg, udp_sendmsg, + sk, msg, size); }EXPORT_SYMBOL (inet_sendmsg);
@@ -822,6 +825,8 @@ ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset, }EXPORT_SYMBOL (inet_sendpage); +INDIRECT_CALLABLE_DECLARE(int udp_recvmsg(struct sock *, struct msghdr *, +size_t ,int ,int ,int *));int inet_recvmsg (struct socket *sock,struct msghdr *msg,size_t size,int flags) {
@@ -832,8 +837,9 @@ int inet_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,if (likely (!(flags &MSG_ERRQUEUE )))sock_rps_record_flow (sk); - err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , - flags & ~MSG_DONTWAIT , &addr_len); + err =INDIRECT_CALL_2 (sk->sk_prot->recvmsg, tcp_recvmsg, udp_recvmsg, + sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len);if (err >=0 ) msg->msg_namelen = addr_len;return err;
Signature · 1 line
2.20.1
Re: [PATCH net-next 5/5] ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
Quoted 13 lines · Paolo Abeni
This avoids an indirect call per syscall for common ipv4 transports
Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/af_inet.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)net/ipv4/af_inet.c +0
-0
net/ipv4/af_inet.c int inet_send_prepare(struct sock *sk) · line 801
Quoted 5 lines
net/ipv4/af_inet.c +2
-0@@ -797,6 +797,8 @@ int inet_send_prepare(struct sock *sk) }EXPORT_SYMBOL_GPL (inet_send_prepare); +INDIRECT_CALLABLE_DECLARE(int udp_sendmsg(struct sock *, struct msghdr *, +size_t ));
Small nit: this is already defined in include/net/udp.h, which is included. So like tcp_sendmsg, probably no need to declare.
If defining inet6_sendmsg and inet6_recvmsg in include/net/ipv6.h, perhaps do the same for the other missing functions, instead of these indirect declarations at the callsite?
Aside from that small point, patch set looks great to me. Thanks Paolo.
Paolo Abeni
maintainerRe: [PATCH net-next 5/5] ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
Quoted 16 lines · Willem de Bruijn
Quoted 15 lines · Paolo Abeni
This avoids an indirect call per syscall for common ipv4 transports
Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/af_inet.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)net/ipv4/af_inet.c +0
-0@@ -797,6 +797,8 @@ int inet_send_prepare(struct sock *sk) }EXPORT_SYMBOL_GPL (inet_send_prepare);
Quoted 5 lines
net/ipv4/af_inet.c int inet_send_prepare(struct sock *sk) · line 801
Quoted 2 lines
net/ipv4/af_inet.c +2
-0+INDIRECT_CALLABLE_DECLARE(int udp_sendmsg(struct sock *, struct msghdr *, +size_t ));Small nit: this is already defined in
include/net/udp.h, which is included. So liketcp_sendmsg, probably no need to declare.
Thank you for the review!
You are right, that declaration can be dropped.
Quoted 4 lines
If defining
inet6_sendmsgandinet6_recvmsgininclude/net/ipv6.h, perhaps do the same for the other missing functions, instead of these indirect declarations at the callsite?
Uhm... since inet6_{send,recv}msg exists only for retpoline sake and are not exported, I think is probably better move their declaration to socket.c via INDIRECT_CALLABLE_DECLARE(), to that ICWs are all self- contained.
Unless there are objections about spamming, I can repost the series with the above changes.
Cheers,
Paolo
Re: [PATCH net-next 5/5] ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
Quoted 32 lines · Paolo Abeni
Quoted 16 lines · Willem de Bruijn
Quoted 15 lines · Paolo Abeni
This avoids an indirect call per syscall for common ipv4 transports
Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/af_inet.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)net/ipv4/af_inet.c +0
-0@@ -797,6 +797,8 @@ int inet_send_prepare(struct sock *sk) }EXPORT_SYMBOL_GPL (inet_send_prepare);Quoted 5 lines
net/ipv4/af_inet.c int inet_send_prepare(struct sock *sk) · line 801
Quoted 2 lines
net/ipv4/af_inet.c +2
-0+INDIRECT_CALLABLE_DECLARE(int udp_sendmsg(struct sock *, struct msghdr *, +size_t ));Small nit: this is already defined in
include/net/udp.h, which is included. So liketcp_sendmsg, probably no need to declare.Thank you for the review!
You are right, that declaration can be dropped.
Quoted 4 lines
If defining
inet6_sendmsgandinet6_recvmsgininclude/net/ipv6.h, perhaps do the same for the other missing functions, instead of these indirect declarations at the callsite?
Quoted 7 lines
Uhm... since inet6_{send,recv}msg exists only for retpoline sake and are not exported, I think is probably better move their declaration to
socket.cviaINDIRECT_CALLABLE_DECLARE(), to that ICWs are all self- contained.Unless there are objections about spamming, I can repost the series with the above changes.
If just for the spurious declaration, can be merged as is, too. Either way. Not spammy at all to resend after a day.