net: use ICW for sk_proto->{send,recv}msg
Subsystems (3)
Paolo Abeni
maintainerv2 cover net: use ICW for sk_proto->{send,recv}msg
This series extends ICW usage to one of the few remaining spots in fast-path still hitting per packet retpoline overhead, namely the sk_proto->{send,recv}msg calls.
The first 3 patches in this series refactor the existing code so that applying the ICW macros is straight-forward: we demux inet_{recv,send}msg in ipv4 and ipv6 variants so that each of them can easily select the appropriate TCP or UDP direct call. While at it, a new helper is created to avoid excessive code duplication, and the current ICWs for inet_{recv,send}msg are adjusted accordingly.
The last 2 patches really introduce the new ICW use-case, respectively for the ipv6 and the ipv4 code path.
This gives up to 5% performance improvement under UDP flood, and smaller but measurable gains for TCP RR workloads.
v1 -> v2:
- drop inet6_{recv,send}msg declaration from header file, prefer ICW macro instead
- avoid unneeded reclaration for udp_sendmsg, as suggested by Willem
Paolo Abeni (5):
inet: factor out inet_send_prepare()
ipv6: provide and use ipv6 specific version for {recv,send}msg
net: adjust socket level ICW to cope with ipv6 variant of
{recv,send}msg
ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
include/net/inet_common.h | 1 + net/ipv4/af_inet.c | 31 ++++++++++++++++++----------- net/ipv6/af_inet6.c | 41 +++++++++++++++++++++++++++++++++++---- net/socket.c | 23 +++++++++++----------- 4 files changed, 69 insertions(+), 27 deletions(-)
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v2 1/5 inet: factor out inet_send_prepare()
The same code is replicated verbatim in multiple places, and the next patches will introduce an additional user for it. Factor out a helper and use it where appropriate. No functional change intended.
--- include/net/inet_common.h | 1 + net/ipv4/af_inet.c | 21 +++++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-)
include/net/inet_common.h +1 -0
@@ -25,6 +25,7 @@ int inet_dgram_connect(struct socket *sock, struct sockaddr *uaddr, int addr_len, int flags);int inet_accept (struct socket *sock,struct socket *newsock,int flags,bool kern); +int inet_send_prepare (struct sock *sk);int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size);ssize_t inet_sendpage (struct socket *sock,struct page *page,int offset,size_t size,int flags);
net/ipv4/af_inet.c +13 -8
@@ -784,10 +784,8 @@ int inet_getname(struct socket *sock, struct sockaddr *uaddr, }EXPORT_SYMBOL (inet_getname); -int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +int inet_send_prepare (struct sock *sk) { -struct sock *sk = sock->sk; -sock_rps_record_flow (sk);/* We may need to bind the socket. */
@@ -795,6 +793,17 @@ int inet_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) inet_autobind(sk))return -EAGAIN ; +return 0 ; +} +EXPORT_SYMBOL_GPL (inet_send_prepare); + +int inet_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +{ +struct sock *sk = sock->sk; + +if (unlikely (inet_send_prepare (sk))) +return -EAGAIN ; +return sk->sk_prot->sendmsg (sk, msg, size); }EXPORT_SYMBOL (inet_sendmsg);
@@ -804,11 +813,7 @@ ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset, {struct sock *sk = sock->sk; -sock_rps_record_flow (sk); - -/* We may need to bind the socket. */ -if (!inet_sk (sk)->inet_num && !sk->sk_prot->no_autobind && -inet_autobind (sk)) +if (unlikely (inet_send_prepare (sk)))return -EAGAIN ;if (sk->sk_prot->sendpage)
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v2 2/5 ipv6: provide and use ipv6 specific version for {recv,send}msg
In reply to Paolo AbeniThis will simplify indirect call wrapper invocation in the following patch.
No functional change intended, any - out-of-tree - IPv6 user of inet_{recv,send}msg can keep using the existing functions.
SCTP code still uses the existing version even for ipv6: as this series will not add ICW for SCTP, moving to the new helper would not give any benefit.
The only other in-kernel user of inet_{recv,send}msg is pvcalls_conn_back_read(), but psvcalls explicitly creates only IPv4 socket, so no need to update that code path, too.
v1 -> v2: drop inet6_{recv,send}msg declaration from header file,
prefer ICW macro instead
--- net/ipv6/af_inet6.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-)
net/ipv6/af_inet6.c +31 -4
@@ -564,6 +564,33 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) }EXPORT_SYMBOL (inet6_ioctl); +int inet6_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) +{ +struct sock *sk = sock->sk; + +if (unlikely (inet_send_prepare (sk))) +return -EAGAIN ; + +return sk->sk_prot->sendmsg (sk, msg, size); +} + +int inet6_recvmsg (struct socket *sock,struct msghdr *msg,size_t size, +int flags) +{ +struct sock *sk = sock->sk; +int addr_len =0 ; +int err; + +if (likely (!(flags &MSG_ERRQUEUE ))) +sock_rps_record_flow (sk); + + err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len); +if (err >=0 ) + msg->msg_namelen = addr_len; +return err; +} +const struct proto_ops inet6_stream_ops = { .family =PF_INET6 , .owner =THIS_MODULE ,
@@ -580,8 +607,8 @@ const struct proto_ops inet6_stream_ops = { .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ .getsockopt = sock_common_getsockopt, /* ok */ - .sendmsg = inet_sendmsg, /* ok */ - .recvmsg = inet_recvmsg, /* ok */ + .sendmsg = inet6_sendmsg, /* retpoline's sake */ + .recvmsg = inet6_recvmsg, /* retpoline's sake */#ifdef CONFIG_MMU .mmap = tcp_mmap,#endif
@@ -614,8 +641,8 @@ const struct proto_ops inet6_dgram_ops = { .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ .getsockopt = sock_common_getsockopt, /* ok */ - .sendmsg = inet_sendmsg, /* ok */ - .recvmsg = inet_recvmsg, /* ok */ + .sendmsg = inet6_sendmsg, /* retpoline's sake */ + .recvmsg = inet6_recvmsg, /* retpoline's sake */ .mmap = sock_no_mmap, .sendpage = sock_no_sendpage, .set_peek_off = sk_set_peek_off,
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v2 3/5 net: adjust socket level ICW to cope with ipv6 variant of {recv,send}msg
In reply to Paolo AbeniAfter the previous patch we have ipv{6,4} variants for {recv,send}msg, we should use the generic _INET ICW variant to call into the proper build-in.
This also allows dropping the now unused and rather ugly _INET4 ICW macro
v1 -> v2:
- use ICW macro to declare inet6_{recv,send}msg
- fix a couple of checkpatch offender in the code context
--- net/socket.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
net/socket.c +11 -12
@@ -103,13 +103,6 @@#include <net/busy_poll.h> #include <linux/errqueue.h> -/* proto_ops for ipv4 and ipv6 use the same {recv,send}msg function */ -#if IS_ENABLED (CONFIG_INET ) -#define INDIRECT_CALL_INET4 (f, f1, ...) INDIRECT_CALL_1(f, f1, __VA_ARGS__) -#else -#define INDIRECT_CALL_INET4 (f, f1, ...) f(__VA_ARGS__) -#endif -#ifdef CONFIG_NET_RX_BUSY_POLL unsigned int sysctl_net_busy_read__read_mostly ;unsigned int sysctl_net_busy_poll__read_mostly ;
@@ -641,10 +634,13 @@ EXPORT_SYMBOL(__sock_tx_timestamp); INDIRECT_CALLABLE_DECLARE(int inet_sendmsg(struct socket *, struct msghdr *,size_t )); +INDIRECT_CALLABLE_DECLARE(int inet6_sendmsg(struct socket *, struct msghdr *, +size_t ));static inline int sock_sendmsg_nosec (struct socket *sock,struct msghdr *msg) { -int ret =INDIRECT_CALL_INET4 (sock->ops->sendmsg, inet_sendmsg, sock, - msg,msg_data_left (msg)); +int ret =INDIRECT_CALL_INET (sock->ops->sendmsg, inet6_sendmsg, + inet_sendmsg, sock, msg, +msg_data_left (msg));BUG_ON (ret == -EIOCBQUEUED );return ret; }
@@ -870,12 +866,15 @@ void __sock_recv_ts_and_drops(struct msghdr *msg, struct sock *sk,EXPORT_SYMBOL_GPL (__sock_recv_ts_and_drops); INDIRECT_CALLABLE_DECLARE(int inet_recvmsg(struct socket *, struct msghdr *, -size_t ,int )); +size_t ,int )); +INDIRECT_CALLABLE_DECLARE(int inet6_recvmsg(struct socket *, struct msghdr *, +size_t ,int ));static inline int sock_recvmsg_nosec (struct socket *sock,struct msghdr *msg,int flags) { -return INDIRECT_CALL_INET4 (sock->ops->recvmsg, inet_recvmsg, sock, msg, -msg_data_left (msg), flags); +return INDIRECT_CALL_INET (sock->ops->recvmsg, inet6_recvmsg, + inet_recvmsg, sock, msg,msg_data_left (msg), + flags); }/**
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v2 5/5 ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
In reply to Paolo AbeniThis avoids an indirect call per syscall for common ipv4 transports
v1 -> v2:
- avoid unneeded reclaration for udp_sendmsg, as suggested by Willem
--- net/ipv4/af_inet.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
net/ipv4/af_inet.c +7 -3
@@ -804,7 +804,8 @@ int inet_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)if (unlikely (inet_send_prepare (sk)))return -EAGAIN ; -return sk->sk_prot->sendmsg (sk, msg, size); +return INDIRECT_CALL_2 (sk->sk_prot->sendmsg, tcp_sendmsg, udp_sendmsg, + sk, msg, size); }EXPORT_SYMBOL (inet_sendmsg);
@@ -822,6 +823,8 @@ ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset, }EXPORT_SYMBOL (inet_sendpage); +INDIRECT_CALLABLE_DECLARE(int udp_recvmsg(struct sock *, struct msghdr *, +size_t ,int ,int ,int *));int inet_recvmsg (struct socket *sock,struct msghdr *msg,size_t size,int flags) {
@@ -832,8 +835,9 @@ int inet_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,if (likely (!(flags &MSG_ERRQUEUE )))sock_rps_record_flow (sk); - err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , - flags & ~MSG_DONTWAIT , &addr_len); + err =INDIRECT_CALL_2 (sk->sk_prot->recvmsg, tcp_recvmsg, udp_recvmsg, + sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len);if (err >=0 ) msg->msg_namelen = addr_len;return err;
Signature · 1 line
2.20.1
Paolo Abeni
maintainerPATCH v2 4/5 ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
In reply to Paolo AbeniThis avoids an indirect call per syscall for common ipv6 transports
--- net/ipv6/af_inet6.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
net/ipv6/af_inet6.c +9 -3
@@ -564,6 +564,8 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) }EXPORT_SYMBOL (inet6_ioctl); +INDIRECT_CALLABLE_DECLARE(int udpv6_sendmsg(struct sock *, struct msghdr *, +size_t ));int inet6_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) {struct sock *sk = sock->sk;
@@ -571,9 +573,12 @@ int inet6_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)if (unlikely (inet_send_prepare (sk)))return -EAGAIN ; -return sk->sk_prot->sendmsg (sk, msg, size); +return INDIRECT_CALL_2 (sk->sk_prot->sendmsg, tcp_sendmsg, udpv6_sendmsg, + sk, msg, size); } +INDIRECT_CALLABLE_DECLARE(int udpv6_recvmsg(struct sock *, struct msghdr *, +size_t ,int ,int ,int *));int inet6_recvmsg (struct socket *sock,struct msghdr *msg,size_t size,int flags) {
@@ -584,8 +589,9 @@ int inet6_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,if (likely (!(flags &MSG_ERRQUEUE )))sock_rps_record_flow (sk); - err = sk->sk_prot->recvmsg (sk, msg, size, flags &MSG_DONTWAIT , - flags & ~MSG_DONTWAIT , &addr_len); + err =INDIRECT_CALL_2 (sk->sk_prot->recvmsg, tcp_recvmsg, udpv6_recvmsg, + sk, msg, size, flags &MSG_DONTWAIT , + flags & ~MSG_DONTWAIT , &addr_len);if (err >=0 ) msg->msg_namelen = addr_len;return err;
Signature · 1 line
2.20.1
Quoted 23 lines · Paolo Abeni
This series extends ICW usage to one of the few remaining spots in fast-path still hitting per packet retpoline overhead, namely the sk_proto->{send,recv}msg calls.
The first 3 patches in this series refactor the existing code so that applying the ICW macros is straight-forward: we demux inet_{recv,send}msg in ipv4 and ipv6 variants so that each of them can easily select the appropriate TCP or UDP direct call. While at it, a new helper is created to avoid excessive code duplication, and the current ICWs for inet_{recv,send}msg are adjusted accordingly.
The last 2 patches really introduce the new ICW use-case, respectively for the ipv6 and the ipv4 code path.
This gives up to 5% performance improvement under UDP flood, and smaller but measurable gains for TCP RR workloads.
v1 -> v2:
- drop inet6_{recv,send}msg declaration from header file, prefer ICW macro instead
- avoid unneeded reclaration forudp_sendmsg, as suggested by Willem
Quoted 7 lines
Paolo Abeni (5):
inet: factor outinet_send_prepare()
ipv6: provide and use ipv6 specific version for {recv,send}msg
net: adjust socket level ICW to cope with ipv6 variant of
{recv,send}msg
ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
David Miller
maintainerFrom: Willem de Bruijn <[email protected]>
Date: Wed, 3 Jul 2019 10:45:13 -0400
Quoted 32 lines · Paolo Abeni
Quoted 23 lines
This series extends ICW usage to one of the few remaining spots in fast-path still hitting per packet retpoline overhead, namely the sk_proto->{send,recv}msg calls.
The first 3 patches in this series refactor the existing code so that applying the ICW macros is straight-forward: we demux inet_{recv,send}msg in ipv4 and ipv6 variants so that each of them can easily select the appropriate TCP or UDP direct call. While at it, a new helper is created to avoid excessive code duplication, and the current ICWs for inet_{recv,send}msg are adjusted accordingly.
The last 2 patches really introduce the new ICW use-case, respectively for the ipv6 and the ipv4 code path.
This gives up to 5% performance improvement under UDP flood, and smaller but measurable gains for TCP RR workloads.
v1 -> v2:
- drop inet6_{recv,send}msg declaration from header file, prefer ICW macro instead
- avoid unneeded reclaration forudp_sendmsg, as suggested by WillemQuoted 7 lines
Paolo Abeni (5):
inet: factor outinet_send_prepare()
ipv6: provide and use ipv6 specific version for {recv,send}msg
net: adjust socket level ICW to cope with ipv6 variant of
{recv,send}msg
ipv6: use indirect call wrappers for {tcp,udpv6}_{recv,send}msg()
ipv4: use indirect call wrappers for {tcp,udp}_{recv,send}msg()
Quoted 1 line
Acked-by Willem de Bruijn <[email protected]>
Series applied, thanks.