udp gso
Subsystems (6)
Willem de Bruijn
maintainerv2 cover udp gso
From: Willem de Bruijn <[email protected]>
Segmentation offload reduces cycles/byte for large packets by amortizing the cost of protocol stack traversal.
This patchset implements GSO for UDP. A process can concatenate and submit multiple datagrams to the same destination in one send call by setting socket option SOL_UDP/UDP_SEGMENT with the segment size, or passing an analogous cmsg at send time.
The stack will send the entire large (up to network layer max size) datagram through the protocol layer. At the GSO layer, it is broken up in individual segments. All receive the same network layer header and UDP src and dst port. All but the last segment have the same UDP header, but the last may differ in length and checksum.
Initial results show a significant reduction in UDP cycles/byte. See the main patch for more details and benchmark results.
udp
876 MB/s 14873 msg/s 624666 calls/s
11,205,777,429 cycles
udp gso
2139 MB/s 36282 msg/s 36282 calls/s
11,204,374,561 cyclesThe patch set is broken down as follows:
- patch 1 is a prerequisite: code rearrangement, noop otherwise
- patch 2 implements the gso logic
- patch 3 adds protocol stack support for UDP_SEGMENT
- patch 4,5,7 are refinements
- patch 6 adds the cmsg interface
- patch 8..11 are tests
This idea was presented previously at netconf 2017-2
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf
Changes v1 -> v2
- Convert __udp_gso_segment to modify headers after skb_segment
- Split main patch into two, one for gso logic, one for UDP_SEGMENT
Changes RFC -> v1
- MSG_MORE:
fixed, by allowing checksum offload with corking if gso
- SKB_GSO_UDP_L4:
made independent from SKB_GSO_UDP
and removed skb_is_ufo() wrapper
- NETIF_F_GSO_UDP_L4:
add to netdev_features_string
and to netdev-features.txt
add BUILD_BUG_ON to match SKB_GSO_UDP_L4 value
- UDP_MAX_SEGMENTS:
introduce limit on number of segments per gso skb to avoid extreme cases like IP_MAX_MTU/IPV4_MIN_MTU
- CHECKSUM_PARTIAL:
test against missing feature after ndo_features_check if not supported return error, analogous to udp_send_check
- MSG_ZEROCOPY: removed, deferred for now
Willem de Bruijn (11):
udp: expose inet cork to udp
udp: add udp gso
udp: generate gso with UDP_SEGMENT
udp: better wmem accounting on gso
udp: paged allocation with gso
udp: add gso segment cmsg
udp: add gso support to virtual devices
selftests: udp gso
selftests: udp gso with connected sockets
selftests: udp gso with corking
selftests: udp gso benchmark
Documentation/networking/netdev-features.txt | 7 + include/linux/netdev_features.h | 5 +- include/linux/netdevice.h | 1 + include/linux/skbuff.h | 2 + include/linux/udp.h | 3 + include/net/inet_sock.h | 1 + include/net/ip.h | 3 +- include/net/ipv6.h | 2 + include/net/udp.h | 5 + include/uapi/linux/udp.h | 1 + net/core/ethtool.c | 1 + net/core/skbuff.c | 2 + net/ipv4/ip_output.c | 41 +- net/ipv4/udp.c | 80 ++- net/ipv4/udp_offload.c | 66 +- net/ipv6/ip6_offload.c | 6 +- net/ipv6/ip6_output.c | 45 +- net/ipv6/udp.c | 31 +- net/ipv6/udp_offload.c | 19 +- tools/testing/selftests/net/.gitignore | 3 + tools/testing/selftests/net/Makefile | 4 +- tools/testing/selftests/net/udpgso.c | 621 ++++++++++++++++++ tools/testing/selftests/net/udpgso.sh | 29 + tools/testing/selftests/net/udpgso_bench.sh | 74 +++ tools/testing/selftests/net/udpgso_bench_rx.c | 265 ++++++++ tools/testing/selftests/net/udpgso_bench_tx.c | 420 ++++++++++++ 26 files changed, 1686 insertions(+), 51 deletions(-) create mode 100644 tools/testing/selftests/net/udpgso.c create mode 100755 tools/testing/selftests/net/udpgso.sh create mode 100755 tools/testing/selftests/net/udpgso_bench.sh create mode 100644 tools/testing/selftests/net/udpgso_bench_rx.c create mode 100644 tools/testing/selftests/net/udpgso_bench_tx.c
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 1/11 udp: expose inet cork to udp
From: Willem de Bruijn <[email protected]>
UDP segmentation offload needs access to inet_cork in the udp layer. Pass the struct to ip(6)_make_skb instead of allocating it on the stack in that function itself.
This patch is a noop otherwise.
--- include/net/ip.h | 2 +- include/net/ipv6.h | 1 + net/ipv4/ip_output.c | 17 ++++++++--------- net/ipv4/udp.c | 4 +++- net/ipv6/ip6_output.c | 20 ++++++++++---------- net/ipv6/udp.c | 3 ++- 6 files changed, 25 insertions(+), 22 deletions(-)
include/net/ip.h +1 -1
@@ -171,7 +171,7 @@ struct sk_buff *ip_make_skb(struct sock *sk, struct flowi4 *fl4, int len, int odd, struct sk_buff *skb), void *from, int length, int transhdrlen, struct ipcm_cookie *ipc, struct rtable **rtp, - unsigned int flags); + struct inet_cork *cork, unsigned int flags);static inline struct sk_buff *ip_finish_skb (struct sock *sk,struct flowi4 *fl4) {
include/net/ipv6.h +1 -0
@@ -950,6 +950,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, void *from, int length, int transhdrlen, struct ipcm6_cookie *ipc6, struct flowi6 *fl6, struct rt6_info *rt, unsigned int flags, + struct inet_cork_full *cork, const struct sockcm_cookie *sockc);static inline struct sk_buff *ip6_finish_skb (struct sock *sk)
net/ipv4/ip_output.c +8 -9
@@ -1470,9 +1470,8 @@ struct sk_buff *ip_make_skb(struct sock *sk, int len, int odd, struct sk_buff *skb), void *from, int length, int transhdrlen, struct ipcm_cookie *ipc, struct rtable **rtp, - unsigned int flags) + struct inet_cork *cork, unsigned int flags) { - struct inet_cork cork; struct sk_buff_head queue;int err;
@@ -1481,22 +1480,22 @@ struct sk_buff *ip_make_skb(struct sock *sk,__skb_queue_head_init (&queue); - cork.flags =0 ; - cork.addr =0 ; - cork.opt =NULL ; - err =ip_setup_cork (sk, &cork, ipc, rtp); + cork->flags =0 ; + cork->addr =0 ; + cork->opt =NULL ; + err =ip_setup_cork (sk, cork, ipc, rtp);if (err)return ERR_PTR (err); - err =__ip_append_data (sk, fl4, &queue, &cork, + err =__ip_append_data (sk, fl4, &queue, cork, ¤t->task_frag, getfrag, from, length, transhdrlen, flags);if (err) { -__ip_flush_pending_frames (sk, &queue, &cork); +__ip_flush_pending_frames (sk, &queue, cork);return ERR_PTR (err); } -return __ip_make_skb (sk, fl4, &queue, &cork); +return __ip_make_skb (sk, fl4, &queue, cork); }/*
net/ipv4/udp.c +3 -1
@@ -1030,9 +1030,11 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)/* Lockless fast path for the non-corking case. */ if (!corkreq) { +struct inet_cork cork; + skb =ip_make_skb (sk, fl4, getfrag, msg, ulen,sizeof (struct udphdr ), &ipc, &rt, - msg->msg_flags); + &cork, msg->msg_flags); err =PTR_ERR (skb);if (!IS_ERR_OR_NULL (skb)) err =udp_send_skb (skb, fl4);
net/ipv6/ip6_output.c +10 -10
@@ -1755,9 +1755,9 @@ struct sk_buff *ip6_make_skb(struct sock *sk, void *from, int length, int transhdrlen, struct ipcm6_cookie *ipc6, struct flowi6 *fl6, struct rt6_info *rt, unsigned int flags, + struct inet_cork_full *cork, const struct sockcm_cookie *sockc) { - struct inet_cork_full cork; struct inet6_cork v6_cork;struct sk_buff_head queue;int exthdrlen = (ipc6->opt ? ipc6->opt->opt_flen :0 );
@@ -1768,27 +1768,27 @@ struct sk_buff *ip6_make_skb(struct sock *sk,__skb_queue_head_init (&queue); - cork.base.flags =0 ; - cork.base.addr =0 ; - cork.base.opt =NULL ; - cork.base.dst =NULL ; + cork->base.flags =0 ; + cork->base.addr =0 ; + cork->base.opt =NULL ; + cork->base.dst =NULL ; v6_cork.opt =NULL ; - err =ip6_setup_cork (sk, &cork, &v6_cork, ipc6, rt, fl6); + err =ip6_setup_cork (sk, cork, &v6_cork, ipc6, rt, fl6);if (err) { -ip6_cork_release (&cork, &v6_cork); +ip6_cork_release (cork, &v6_cork);return ERR_PTR (err); }if (ipc6->dontfrag <0 ) ipc6->dontfrag =inet6_sk (sk)->dontfrag; - err =__ip6_append_data (sk, fl6, &queue, &cork.base, &v6_cork, + err =__ip6_append_data (sk, fl6, &queue, &cork->base, &v6_cork, ¤t->task_frag, getfrag, from, length + exthdrlen, transhdrlen + exthdrlen, flags, ipc6, sockc);if (err) { -__ip6_flush_pending_frames (sk, &queue, &cork, &v6_cork); +__ip6_flush_pending_frames (sk, &queue, cork, &v6_cork);return ERR_PTR (err); } -return __ip6_make_skb (sk, &queue, &cork, &v6_cork); +return __ip6_make_skb (sk, &queue, cork, &v6_cork); }
net/ipv6/udp.c +2 -1
@@ -1324,12 +1324,13 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)/* Lockless fast path for the non-corking case */ if (!corkreq) { +struct inet_cork_full cork;struct sk_buff *skb; skb =ip6_make_skb (sk, getfrag, msg, ulen,sizeof (struct udphdr ), &ipc6, &fl6, (struct rt6_info *)dst, - msg->msg_flags, &sockc); + msg->msg_flags, &cork, &sockc); err =PTR_ERR (skb);if (!IS_ERR_OR_NULL (skb)) err =udp_v6_send_skb (skb, &fl6);
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 2/11 udp: add udp gso
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Implement generic segmentation offload support for udp datagrams. A follow-up patch adds support to the protocol stack to generate such packets.
UDP GSO is not UFO. UFO fragments a single large datagram. GSO splits a large payload into a number of discrete UDP datagrams.
The implementation adds a GSO type SKB_UDP_GSO_L4 to differentiate it from UFO (SKB_UDP_GSO).
IPPROTO_UDPLITE is excluded, as that protocol has no gso handler registered.
--- include/linux/skbuff.h | 2 ++ include/net/udp.h | 4 ++++ net/core/skbuff.c | 2 ++ net/ipv4/udp_offload.c | 52 +++++++++++++++++++++++++++++++++++++++++- net/ipv6/ip6_offload.c | 6 +++-- net/ipv6/udp_offload.c | 19 ++++++++++++++- 6 files changed, 81 insertions(+), 4 deletions(-)
include/linux/skbuff.h +2 -0
@@ -573,6 +573,8 @@ enum {SKB_GSO_ESP =1 <<15 ,SKB_GSO_UDP =1 <<16 , + + SKB_GSO_UDP_L4 = 1 << 17, };#if BITS_PER_LONG >32
include/net/udp.h +4 -0
@@ -174,6 +174,10 @@ struct sk_buff **udp_gro_receive(struct sk_buff **head, struct sk_buff *skb, struct udphdr *uh, udp_lookup_t lookup);int udp_gro_complete (struct sk_buff *skb,int nhoff,udp_lookup_t lookup); +struct sk_buff *__udp_gso_segment (struct sk_buff *gso_skb, +netdev_features_t features, +unsigned int mss,__sum16 check); +static inline struct udphdr *udp_gro_udphdr (struct sk_buff *skb) {struct udphdr *uh;
net/core/skbuff.c +2 -0
@@ -4940,6 +4940,8 @@ static unsigned int skb_gso_transport_seglen(const struct sk_buff *skb) thlen =tcp_hdrlen (skb); }else if (unlikely (skb_is_gso_sctp (skb))) { thlen =sizeof (struct sctphdr ); + }else if (shinfo->gso_type &SKB_GSO_UDP_L4 ) { + thlen =sizeof (struct udphdr ); }/* UFO sets gso_size to the size of the fragmentation * payload, i.e. the size of the L4 (UDP) header is already
net/ipv4/udp_offload.c +51 -1
69 lines not on this page · raw message
net/ipv6/ip6_offload.c +4 -2
@@ -88,9 +88,11 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,if (skb->encapsulation &&skb_shinfo (skb)->gso_type & (SKB_GSO_IPXIP4 |SKB_GSO_IPXIP6 )) - udpfrag = proto ==IPPROTO_UDP && encap; + udpfrag = proto ==IPPROTO_UDP && encap && + (skb_shinfo (skb)->gso_type &SKB_GSO_UDP );else - udpfrag = proto ==IPPROTO_UDP && !skb->encapsulation; + udpfrag = proto ==IPPROTO_UDP && !skb->encapsulation && + (skb_shinfo (skb)->gso_type &SKB_GSO_UDP ); ops =rcu_dereference (inet6_offloads[proto]);if (likely (ops && ops->callbacks.gso_segment)) {
net/ipv6/udp_offload.c +18 -1
@@ -17,6 +17,20 @@#include <net/ip6_checksum.h> #include "ip6_offload.h" +static struct sk_buff *__udp6_gso_segment (struct sk_buff *gso_skb, +netdev_features_t features) +{ +const struct ipv6hdr *ip6h =ipv6_hdr (gso_skb); +unsigned int mss =skb_shinfo (gso_skb)->gso_size; + +if (!can_checksum_protocol (features,htons (ETH_P_IPV6 ))) +return ERR_PTR (-EIO ); + +return __udp_gso_segment (gso_skb, features, mss, +udp_v6_check (sizeof (struct udphdr ) + mss, + &ip6h->saddr, &ip6h->daddr,0 )); +} +static struct sk_buff *udp6_ufo_fragment (struct sk_buff *skb,netdev_features_t features) {
@@ -42,12 +56,15 @@ static struct sk_buff *udp6_ufo_fragment(struct sk_buff *skb,const struct ipv6hdr *ipv6h;struct udphdr *uh; -if (!(skb_shinfo (skb)->gso_type &SKB_GSO_UDP )) +if (!(skb_shinfo (skb)->gso_type & (SKB_GSO_UDP |SKB_GSO_UDP_L4 ))) goto out;if (!pskb_may_pull (skb,sizeof (struct udphdr ))) goto out; +if (skb_shinfo (skb)->gso_type &SKB_GSO_UDP_L4 ) +return __udp6_gso_segment (skb, features); +/* Do software UFO. Complete and fill in the UDP checksum as HW cannot * do checksum of UDP packets sent as multiple IP fragments. */
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 3/11 udp: generate gso with UDP_SEGMENT
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Support generic segmentation offload for udp datagrams. Callers can concatenate and send at once the payload of multiple datagrams with the same destination.
To set segment size, the caller sets socket option UDP_SEGMENT to the length of each discrete payload. This value must be smaller than or equal to the relevant MTU.
A follow-up patch adds cmsg UDP_SEGMENT to specify segment size on a per send call basis.
Total byte length may then exceed MTU. If not an exact multiple of segment size, the last segment will be shorter.
The implementation adds a gso_size field to the udp socket, ip(v6) cmsg cookie and inet_cork structure to be able to set the value at setsockopt or cmsg time and to work with both lockless and corked paths.
Initial benchmark numbers show UDP GSO about as expensive as TCP GSO.
tcp tso
3197 MB/s 54232 msg/s 54232 calls/s
6,457,754,262 cycles
tcp gso
1765 MB/s 29939 msg/s 29939 calls/s
11,203,021,806 cycles
tcp without tso/gso *
739 MB/s 12548 msg/s 12548 calls/s
11,205,483,630 cycles
udp
876 MB/s 14873 msg/s 624666 calls/s
11,205,777,429 cycles
udp gso
2139 MB/s 36282 msg/s 36282 calls/s
11,204,374,561 cycles [*] after reverting commit 0a6b2a1dc2a2
("tcp: switch to GSO being always on")
Measured total system cycles ('-a') for one core while pinning both the network receive path and benchmark process to that core:
perf stat -a -C 12 -e cycles \
./udpgso_bench_tx -C 12 -4 -D "$DST" -l 4Note the reduction in calls/s with GSO. Bytes per syscall drops increases from 1470 to 61818.
--- include/linux/udp.h | 3 +++ include/net/inet_sock.h | 1 + include/net/ip.h | 1 + include/net/ipv6.h | 1 + include/uapi/linux/udp.h | 1 + net/ipv4/ip_output.c | 9 ++++++--- net/ipv4/udp.c | 33 ++++++++++++++++++++++++++++++--- net/ipv6/ip6_output.c | 6 ++++-- net/ipv6/udp.c | 23 ++++++++++++++++++++--- 9 files changed, 67 insertions(+), 11 deletions(-)
include/linux/udp.h +3 -0
@@ -55,6 +55,7 @@ struct udp_sock {* when the socket is uncorked. */ __u16 len;/* total length of pending frames */ +__u16 gso_size;/* * Fields specific to UDP-Lite. */
@@ -87,6 +88,8 @@ struct udp_sock {int forward_deficit; }; +#define UDP_MAX_SEGMENTS (1 << 6UL) +static inline struct udp_sock *udp_sk (const struct sock *sk) {return (struct udp_sock *)sk;
include/net/inet_sock.h +1 -0
@@ -147,6 +147,7 @@ struct inet_cork {__u8 ttl;__s16 tos;char priority; +__u16 gso_size; };struct inet_cork_full {
include/net/ip.h +1 -0
@@ -76,6 +76,7 @@ struct ipcm_cookie {__u8 ttl;__s16 tos;char priority; +__u16 gso_size; };#define IPCB (skb) ((struct inet_skb_parm*)((skb)->cb))
include/net/ipv6.h +1 -0
@@ -298,6 +298,7 @@ struct ipcm6_cookie {__s16 tclass;__s8 dontfrag;struct ipv6_txoptions *opt; +__u16 gso_size; };static inline struct ipv6_txoptions *txopt_get (const struct ipv6_pinfo *np)
include/uapi/linux/udp.h +1 -0
@@ -32,6 +32,7 @@ struct udphdr {#define UDP_ENCAP 100/* Set the socket to accept encapsulated packets */ #define UDP_NO_CHECK6_TX 101/* Disable sending checksum for UDP6X */ #define UDP_NO_CHECK6_RX 102/* Disable accpeting checksum for UDP6 */ +#define UDP_SEGMENT 103/* Set GSO segmentation size */ /* UDP encapsulation types */ #define UDP_ENCAP_ESPINUDP_NON_IKE 1/* draft-ietf-ipsec-nat-t-ike-00/01 */
net/ipv4/ip_output.c +6 -3
@@ -882,7 +882,8 @@ static int __ip_append_data(struct sock *sk, skb =skb_peek_tail (queue); exthdrlen = !skb ? rt->dst.header_len :0 ; - mtu = cork->fragsize; + mtu = cork->gso_size ?IP_MAX_MTU : cork->fragsize; +if (cork->tx_flags &SKBTX_ANY_SW_TSTAMP && sk->sk_tsflags &SOF_TIMESTAMPING_OPT_ID ) tskey = sk->sk_tskey++;
@@ -906,7 +907,7 @@ static int __ip_append_data(struct sock *sk,if (transhdrlen && length + fragheaderlen <= mtu && rt->dst.dev->features & (NETIF_F_HW_CSUM |NETIF_F_IP_CSUM ) && - !(flags &MSG_MORE ) && + (!(flags &MSG_MORE ) || cork->gso_size) && !exthdrlen) csummode =CHECKSUM_PARTIAL ;
@@ -1135,6 +1136,8 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, *rtp =NULL ; cork->fragsize =ip_sk_use_pmtu (sk) ?dst_mtu (&rt->dst) : rt->dst.dev->mtu; + + cork->gso_size = sk->sk_type ==SOCK_DGRAM ? ipc->gso_size :0 ; cork->dst = &rt->dst; cork->length =0 ; cork->ttl = ipc->ttl;
@@ -1214,7 +1217,7 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,return -EOPNOTSUPP ; hh_len =LL_RESERVED_SPACE (rt->dst.dev); - mtu = cork->fragsize; + mtu = cork->gso_size ?IP_MAX_MTU : cork->fragsize; fragheaderlen =sizeof (struct iphdr ) + (opt ? opt->optlen :0 ); maxfraglen = ((mtu - fragheaderlen) & ~7 ) + fragheaderlen;
net/ipv4/udp.c +30 -3
@@ -757,7 +757,8 @@ void udp_set_csum(bool nocheck, struct sk_buff *skb, }EXPORT_SYMBOL (udp_set_csum); -static int udp_send_skb (struct sk_buff *skb,struct flowi4 *fl4) +static int udp_send_skb (struct sk_buff *skb,struct flowi4 *fl4, +struct inet_cork *cork) {struct sock *sk = skb->sk;struct inet_sock *inet =inet_sk (sk);
@@ -777,6 +778,21 @@ static int udp_send_skb(struct sk_buff *skb, struct flowi4 *fl4) uh->len =htons (len); uh->check =0 ; +if (cork->gso_size) { +const int hlen =skb_network_header_len (skb) + +sizeof (struct udphdr ); + +if (hlen + cork->gso_size > cork->fragsize) +return -EINVAL ; +if (skb->len > cork->gso_size *UDP_MAX_SEGMENTS ) +return -EINVAL ; +if (skb->ip_summed !=CHECKSUM_PARTIAL || is_udplite) +return -EIO ; + +skb_shinfo (skb)->gso_size = cork->gso_size; +skb_shinfo (skb)->gso_type =SKB_GSO_UDP_L4 ; + } +if (is_udplite)/* UDP-Lite */ csum =udplite_csum (skb);
@@ -828,7 +844,7 @@ int udp_push_pending_frames(struct sock *sk)if (!skb) goto out; - err =udp_send_skb (skb, fl4); + err =udp_send_skb (skb, fl4, &inet->cork.base); out: up->len =0 ;
@@ -922,6 +938,7 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) ipc.sockc.tsflags = sk->sk_tsflags; ipc.addr = inet->inet_saddr; ipc.oif = sk->sk_bound_dev_if; + ipc.gso_size = up->gso_size;if (msg->msg_controllen) { err =ip_cmsg_send (sk, msg, &ipc, sk->sk_family ==AF_INET6 );
@@ -1037,7 +1054,7 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) &cork, msg->msg_flags); err =PTR_ERR (skb);if (!IS_ERR_OR_NULL (skb)) - err =udp_send_skb (skb, fl4); + err =udp_send_skb (skb, fl4, &cork); goto out; }
@@ -2367,6 +2384,12 @@ int udp_lib_setsockopt(struct sock *sk, int level, int optname, up->no_check6_rx = valbool;break ; +case UDP_SEGMENT : +if (val <0 || val >USHRT_MAX ) +return -EINVAL ; + up->gso_size = val; +break ; +/* * UDP-Lite's partial checksum coverage (RFC 3828). */
@@ -2457,6 +2480,10 @@ int udp_lib_getsockopt(struct sock *sk, int level, int optname, val = up->no_check6_rx;break ; +case UDP_SEGMENT : + val = up->gso_size; +break ; +/* The following two cannot be changed on UDP sockets, the return is * always 0 (which corresponds to the full checksum coverage of UDP). */ case UDPLITE_SEND_CSCOV :
net/ipv6/ip6_output.c +4 -2
@@ -1240,6 +1240,8 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork,if (mtu <IPV6_MIN_MTU )return -EINVAL ; cork->base.fragsize = mtu; + cork->base.gso_size = sk->sk_type ==SOCK_DGRAM ? ipc6->gso_size :0 ; +if (dst_allfrag (xfrm_dst_path (&rt->dst))) cork->base.flags |=IPCORK_ALLFRAG ; cork->base.length =0 ;
@@ -1281,7 +1283,7 @@ static int __ip6_append_data(struct sock *sk, dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len; } - mtu = cork->fragsize; + mtu = cork->gso_size ?IP6_MAX_MTU : cork->fragsize; orig_mtu = mtu; hh_len =LL_RESERVED_SPACE (rt->dst.dev);
@@ -1329,7 +1331,7 @@ static int __ip6_append_data(struct sock *sk,if (transhdrlen && sk->sk_protocol ==IPPROTO_UDP && headersize ==sizeof (struct ipv6hdr ) && length <= mtu - headersize && - !(flags &MSG_MORE ) && + (!(flags &MSG_MORE ) || cork->gso_size) && rt->dst.dev->features & (NETIF_F_IPV6_CSUM |NETIF_F_HW_CSUM )) csummode =CHECKSUM_PARTIAL ;
net/ipv6/udp.c +20 -3
@@ -1023,7 +1023,8 @@ static void udp6_hwcsum_outgoing(struct sock *sk, struct sk_buff *skb,* Sending */ -static int udp_v6_send_skb (struct sk_buff *skb,struct flowi6 *fl6) +static int udp_v6_send_skb (struct sk_buff *skb,struct flowi6 *fl6, +struct inet_cork *cork) {struct sock *sk = skb->sk;struct udphdr *uh;
@@ -1042,6 +1043,21 @@ static int udp_v6_send_skb(struct sk_buff *skb, struct flowi6 *fl6) uh->len =htons (len); uh->check =0 ; +if (cork->gso_size) { +const int hlen =skb_network_header_len (skb) + +sizeof (struct udphdr ); + +if (hlen + cork->gso_size > cork->fragsize) +return -EINVAL ; +if (skb->len > cork->gso_size *UDP_MAX_SEGMENTS ) +return -EINVAL ; +if (skb->ip_summed !=CHECKSUM_PARTIAL || is_udplite) +return -EIO ; + +skb_shinfo (skb)->gso_size = cork->gso_size; +skb_shinfo (skb)->gso_type =SKB_GSO_UDP_L4 ; + } +if (is_udplite) csum =udplite_csum (skb);else if (udp_sk (sk)->no_check6_tx) {/* UDP csum disabled */
@@ -1093,7 +1109,7 @@ static int udp_v6_push_pending_frames(struct sock *sk)if (!skb) goto out; - err =udp_v6_send_skb (skb, &fl6); + err =udp_v6_send_skb (skb, &fl6, &inet_sk (sk)->cork.base); out: up->len =0 ;
@@ -1127,6 +1143,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) ipc6.hlimit =-1 ; ipc6.tclass =-1 ; ipc6.dontfrag =-1 ; + ipc6.gso_size = up->gso_size; sockc.tsflags = sk->sk_tsflags;/* destination address check */
@@ -1333,7 +1350,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) msg->msg_flags, &cork, &sockc); err =PTR_ERR (skb);if (!IS_ERR_OR_NULL (skb)) - err =udp_v6_send_skb (skb, &fl6); + err =udp_v6_send_skb (skb, &fl6, &cork.base); goto out; }
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 4/11 udp: better wmem accounting on gso
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
skb_segment by default transfers allocated wmem from the gso skb to the tail of the segment list. This underreports real truesize of the list, especially if the tail might be dropped.
Similar to tcp_gso_segment, update wmem_alloc with the aggregate list truesize and make each segment responsible for its own share by setting skb->destructor.
Clear gso_skb->destructor prior to calling skb_segment to skip the default assignment to tail.
--- net/ipv4/udp_offload.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
net/ipv4/udp_offload.c +15 -1
@@ -191,6 +191,8 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,netdev_features_t features, unsigned int mss, __sum16 check) { + struct sock *sk = gso_skb->sk; +unsigned int sum_truesize =0 ;struct sk_buff *segs, *seg;unsigned int hdrlen;struct udphdr *uh;
@@ -201,9 +203,15 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb, hdrlen = gso_skb->data -skb_mac_header (gso_skb);skb_pull (gso_skb,sizeof (*uh)); +/* clear destructor to avoid skb_segment assigning it to tail */ +WARN_ON_ONCE (gso_skb->destructor != sock_wfree); + gso_skb->destructor =NULL ; + segs =skb_segment (gso_skb, features); -if (unlikely (IS_ERR_OR_NULL (segs))) +if (unlikely (IS_ERR_OR_NULL (segs))) { + gso_skb->destructor = sock_wfree;return segs; + }for (seg = segs; seg; seg = seg->next) { uh =udp_hdr (seg);
@@ -214,8 +222,14 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,if (!seg->next)csum_replace2 (&uh->check,htons (mss),htons (seg->len - hdrlen -sizeof (*uh))); + + seg->destructor = sock_wfree; + seg->sk = sk; + sum_truesize += seg->truesize; } +refcount_add (sum_truesize - gso_skb->truesize, &sk->sk_wmem_alloc); +return segs; }
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 5/11 udp: paged allocation with gso
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
When sending large datagrams that are later segmented, store data in page frags to avoid copying from linear in skb_segment.
--- net/ipv4/ip_output.c | 15 +++++++++++---- net/ipv6/ip6_output.c | 19 ++++++++++++++----- 2 files changed, 25 insertions(+), 9 deletions(-)
net/ipv4/ip_output.c +11 -4
@@ -878,11 +878,13 @@ static int __ip_append_data(struct sock *sk,struct rtable *rt = (struct rtable *)cork->dst;unsigned int wmem_alloc_delta =0 ;u32 tskey =0 ; +bool paged; skb =skb_peek_tail (queue); exthdrlen = !skb ? rt->dst.header_len :0 ; mtu = cork->gso_size ?IP_MAX_MTU : cork->fragsize; + paged = !!cork->gso_size;if (cork->tx_flags &SKBTX_ANY_SW_TSTAMP && sk->sk_tsflags &SOF_TIMESTAMPING_OPT_ID )
@@ -934,6 +936,7 @@ static int __ip_append_data(struct sock *sk,unsigned int fraglen;unsigned int fraggap;unsigned int alloclen; +unsigned int pagedlen =0 ;struct sk_buff *skb_prev; alloc_new_skb: skb_prev = skb;
@@ -954,8 +957,12 @@ static int __ip_append_data(struct sock *sk,if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else +else if (!paged) alloclen = fraglen; +else { + alloclen =min_t (int, fraglen,MAX_HEADER ); + pagedlen = fraglen - alloclen; + } alloclen += exthdrlen;
@@ -999,7 +1006,7 @@ static int __ip_append_data(struct sock *sk,/* * Find where to start putting bytes. */ - data =skb_put (skb, fraglen + exthdrlen); + data =skb_put (skb, fraglen + exthdrlen - pagedlen);skb_set_network_header (skb, exthdrlen); skb->transport_header = (skb->network_header + fragheaderlen);
@@ -1015,7 +1022,7 @@ static int __ip_append_data(struct sock *sk,pskb_trim_unique (skb_prev, maxfraglen); } - copy = datalen - transhdrlen - fraggap; + copy = datalen - transhdrlen - fraggap - pagedlen;if (copy >0 &&getfrag (from, data + transhdrlen, offset, copy, fraggap, skb) <0 ) { err = -EFAULT ;kfree_skb (skb);
@@ -1023,7 +1030,7 @@ static int __ip_append_data(struct sock *sk, } offset += copy; - length -= datalen - fraggap; + length -= copy + transhdrlen; transhdrlen =0 ; exthdrlen =0 ; csummode =CHECKSUM_NONE ;
net/ipv6/ip6_output.c +14 -5
@@ -1276,6 +1276,7 @@ static int __ip6_append_data(struct sock *sk,int csummode =CHECKSUM_NONE ;unsigned int maxnonfragsize, headersize;unsigned int wmem_alloc_delta =0 ; +bool paged; skb =skb_peek_tail (queue);if (!skb) {
@@ -1283,6 +1284,7 @@ static int __ip6_append_data(struct sock *sk, dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len; } + paged = !!cork->gso_size; mtu = cork->gso_size ?IP6_MAX_MTU : cork->fragsize; orig_mtu = mtu;
@@ -1374,6 +1376,7 @@ static int __ip6_append_data(struct sock *sk,unsigned int fraglen;unsigned int fraggap;unsigned int alloclen; +unsigned int pagedlen =0 ; alloc_new_skb:/* There's no room in the current skb */ if (skb)
@@ -1396,11 +1399,17 @@ static int __ip6_append_data(struct sock *sk,if (datalen > (cork->length <= mtu && !(cork->flags &IPCORK_ALLFRAG ) ? mtu : maxfraglen) - fragheaderlen) datalen = maxfraglen - fragheaderlen - rt->dst.trailer_len; + fraglen = datalen + fragheaderlen; +if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; -else - alloclen = datalen + fragheaderlen; +else if (!paged) + alloclen = fraglen; +else { + alloclen =min_t (int, fraglen,MAX_HEADER ); + pagedlen = fraglen - alloclen; + } alloclen += dst_exthdrlen;
@@ -1422,7 +1431,7 @@ static int __ip6_append_data(struct sock *sk,*/ alloclen +=sizeof (struct frag_hdr ); - copy = datalen - transhdrlen - fraggap; + copy = datalen - transhdrlen - fraggap - pagedlen;if (copy <0 ) { err = -EINVAL ; goto error;
@@ -1461,7 +1470,7 @@ static int __ip6_append_data(struct sock *sk,/* * Find where to start putting bytes */ - data =skb_put (skb, fraglen); + data =skb_put (skb, fraglen - pagedlen);skb_set_network_header (skb, exthdrlen); data += fragheaderlen; skb->transport_header = (skb->network_header +
@@ -1484,7 +1493,7 @@ static int __ip6_append_data(struct sock *sk, } offset += copy; - length -= datalen - fraggap; + length -= copy + transhdrlen; transhdrlen =0 ; exthdrlen =0 ; dst_exthdrlen =0 ;
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 6/11 udp: add gso segment cmsg
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Allow specifying segment size in the send call.
The new control message performs the same function as socket option UDP_SEGMENT while avoiding the extra system call.
--- include/net/udp.h | 1 + net/ipv4/udp.c | 43 +++++++++++++++++++++++++++++++++++++++++-- net/ipv6/udp.c | 5 ++++- 3 files changed, 46 insertions(+), 3 deletions(-)
include/net/udp.h +1 -0
@@ -273,6 +273,7 @@ int udp_abort(struct sock *sk, int err);int udp_sendmsg (struct sock *sk,struct msghdr *msg,size_t len);int udp_push_pending_frames (struct sock *sk);void udp_flush_pending_frames (struct sock *sk); +int udp_cmsg_send (struct sock *sk,struct msghdr *msg,u16 *gso_size);void udp4_hwcsum (struct sk_buff *skb,__be32 src,__be32 dst);int udp_rcv (struct sk_buff *skb);int udp_ioctl (struct sock *sk,int cmd,unsigned long arg);
net/ipv4/udp.c +41 -2
55 lines not on this page · raw message
net/ipv6/udp.c +4 -1
@@ -1276,7 +1276,10 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) opt->tot_len =sizeof (*opt); ipc6.opt = opt; - err =ip6_datagram_send_ctl (sock_net (sk), sk, msg, &fl6, &ipc6, &sockc); + err =udp_cmsg_send (sk, msg, &ipc6.gso_size); +if (err >0 ) + err =ip6_datagram_send_ctl (sock_net (sk), sk, msg, &fl6, + &ipc6, &sockc);if (err <0 ) {fl6_sock_release (flowlabel);return err;
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 7/11 udp: add gso support to virtual devices
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Virtual devices such as tunnels and bonding can handle large packets. Only segment packets when reaching a physical or loopback device.
--- Documentation/networking/netdev-features.txt | 7 +++++++ include/linux/netdev_features.h | 5 ++++- include/linux/netdevice.h | 1 + net/core/ethtool.c | 1 + 4 files changed, 13 insertions(+), 1 deletion(-)
Documentation/networking/netdev-features.txt +7 -0
@@ -113,6 +113,13 @@ whatever headers there might be. NETIF_F_TSO_ECN means that hardware can properly split packets with CWR bit set, be it TCPv4 (when NETIF_F_TSO is enabled) or TCPv6 (NETIF_F_TSO6). + * Transmit UDP segmentation offload + +NETIF_F_GSO_UDP_GSO_L4 accepts a single UDP header with a payload that exceeds +gso_size. On segmentation, it segments the payload on gso_size boundaries and +replicates the network and UDP headers (fixing up the last one if less than +gso_size). + * Transmit DMA from high memory On platforms where this is relevant, NETIF_F_HIGHDMA signals that
include/linux/netdev_features.h +4 -1
@@ -55,8 +55,9 @@ enum {NETIF_F_GSO_SCTP_BIT ,/* ... SCTP fragmentation */ NETIF_F_GSO_ESP_BIT ,/* ... ESP with TSO */ NETIF_F_GSO_UDP_BIT ,/* ... UFO, deprecated except tuntap */ +NETIF_F_GSO_UDP_L4_BIT ,/* ... UDP payload GSO (not UFO) */ /**/ NETIF_F_GSO_LAST =/* last bit, see GSO_MASK */ -NETIF_F_GSO_UDP_BIT , +NETIF_F_GSO_UDP_L4_BIT ,NETIF_F_FCOE_CRC_BIT ,/* FCoE CRC32 */ NETIF_F_SCTP_CRC_BIT, /* SCTP checksum offload */
@@ -147,6 +148,7 @@ enum {#define NETIF_F_HW_ESP_TX_CSUM __NETIF_F(HW_ESP_TX_CSUM)#define NETIF_F_RX_UDP_TUNNEL_PORT __NETIF_F(RX_UDP_TUNNEL_PORT)#define NETIF_F_HW_TLS_RECORD __NETIF_F(HW_TLS_RECORD) +#define NETIF_F_GSO_UDP_L4 __NETIF_F(GSO_UDP_L4)#define for_each_netdev_feature (mask_addr, bit) \ for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)
@@ -216,6 +218,7 @@ enum {NETIF_F_GSO_GRE_CSUM | \NETIF_F_GSO_IPXIP4 | \NETIF_F_GSO_IPXIP6 | \ +NETIF_F_GSO_UDP_L4 | \NETIF_F_GSO_UDP_TUNNEL | \ NETIF_F_GSO_UDP_TUNNEL_CSUM)
include/linux/netdevice.h +1 -0
@@ -4186,6 +4186,7 @@ static inline bool net_gso_ok(netdev_features_t features, int gso_type)BUILD_BUG_ON (SKB_GSO_SCTP != (NETIF_F_GSO_SCTP >>NETIF_F_GSO_SHIFT ));BUILD_BUG_ON (SKB_GSO_ESP != (NETIF_F_GSO_ESP >>NETIF_F_GSO_SHIFT ));BUILD_BUG_ON (SKB_GSO_UDP != (NETIF_F_GSO_UDP >>NETIF_F_GSO_SHIFT )); +BUILD_BUG_ON (SKB_GSO_UDP_L4 != (NETIF_F_GSO_UDP_L4 >>NETIF_F_GSO_SHIFT ));return (features & feature) == feature; }
net/core/ethtool.c +1 -0
@@ -92,6 +92,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] [NETIF_F_GSO_PARTIAL_BIT] = "tx-gso-partial", [NETIF_F_GSO_SCTP_BIT] = "tx-sctp-segmentation", [NETIF_F_GSO_ESP_BIT] = "tx-esp-segmentation", + [NETIF_F_GSO_UDP_L4_BIT] = "tx-udp-segmentation", [NETIF_F_FCOE_CRC_BIT] = "tx-checksum-fcoe-crc", [NETIF_F_SCTP_CRC_BIT] = "tx-checksum-sctp",
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 8/11 selftests: udp gso
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Validate udp gso, including edge cases (such as min/max gso sizes).
--- tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 3 +- tools/testing/selftests/net/udpgso.c | 486 +++++++++++++++++++++++++ tools/testing/selftests/net/udpgso.sh | 16 + 4 files changed, 505 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/net/udpgso.c create mode 100755 tools/testing/selftests/net/udpgso.sh
tools/testing/selftests/net/.gitignore +1 -0
@@ -8,3 +8,4 @@ reuseport_bpf_numa reuseport_dualstack reuseaddr_conflict tcp_mmap +udpgso
tools/testing/selftests/net/Makefile +2 -1
@@ -5,12 +5,13 @@ CFLAGS = -Wall -Wl,--no-as-needed -O2 -g CFLAGS += -I../../../../usr/include/ TEST_PROGS := run_netsocktests run_afpackettests test_bpf.sh netdevice.sh rtnetlink.sh -TEST_PROGS += fib_tests.sh fib-onlink-tests.sh in_netns.sh pmtu.sh +TEST_PROGS += fib_tests.sh fib-onlink-tests.sh in_netns.sh pmtu.sh udpgso.sh TEST_GEN_FILES = socket TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy TEST_GEN_FILES += tcp_mmap TEST_GEN_PROGS = reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa TEST_GEN_PROGS += reuseport_dualstack reuseaddr_conflict +TEST_GEN_PROGS += udpgso include ../lib.mk
tools/testing/selftests/net/udpgso.c new file +486 -0
486 lines not on this page · raw message
tools/testing/selftests/net/udpgso.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Run a series of udpgso regression tests + +echo "ipv4 cmsg" +./in_netns.sh ./udpgso -4 -C + +echo "ipv4 setsockopt" +./in_netns.sh ./udpgso -4 -C -s + +echo "ipv6 cmsg" +./in_netns.sh ./udpgso -6 -C + +echo "ipv6 setsockopt" +./in_netns.sh ./udpgso -6 -C -s
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 9/11 selftests: udp gso with connected sockets
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Connected sockets use path mtu instead of device mtu.
Test this path by inserting a route mtu that is lower than the device mtu. Verify that the path mtu for the connection matches this lower number, then run the same test as in the connectionless case.
--- tools/testing/selftests/net/udpgso.c | 117 +++++++++++++++++++++++++- tools/testing/selftests/net/udpgso.sh | 7 ++ 2 files changed, 122 insertions(+), 2 deletions(-)
tools/testing/selftests/net/udpgso.c +115 -2
@@ -47,6 +47,7 @@static bool cfg_do_ipv4;static bool cfg_do_ipv6; +static bool cfg_do_connected;static bool cfg_do_connectionless;static bool cfg_do_setsockopt;static int cfg_specific_test_id =-1 ;
@@ -273,6 +274,101 @@ static void set_pmtu_discover(int fd, bool is_ipv4)error (1 , errno,"setsockopt path mtu" ); } +static unsigned int get_path_mtu (int fd,bool is_ipv4) +{ +socklen_t vallen; +unsigned int mtu; +int ret; + + vallen =sizeof (mtu); +if (is_ipv4) + ret =getsockopt (fd,SOL_IP ,IP_MTU , &mtu, &vallen); +else + ret =getsockopt (fd,SOL_IPV6 ,IPV6_MTU , &mtu, &vallen); + +if (ret) +error (1 , errno,"getsockopt mtu" ); + + +fprintf (stderr,"path mtu (read): %u\n" , mtu); +return mtu; +} + +/* very wordy version of system("ip route add dev lo mtu 1500 127.0.0.3/32") */ +static void set_route_mtu (int mtu,bool is_ipv4) +{ +struct sockaddr_nl nladdr = { .nl_family =AF_NETLINK }; +struct nlmsghdr *nh; +struct rtattr *rta; +struct rtmsg *rt; +char data[NLMSG_ALIGN (sizeof (*nh)) + +NLMSG_ALIGN (sizeof (*rt)) + +NLMSG_ALIGN (RTA_LENGTH (sizeof (addr6))) + +NLMSG_ALIGN (RTA_LENGTH (sizeof (int ))) + +NLMSG_ALIGN (RTA_LENGTH (0 ) +RTA_LENGTH (sizeof (int )))]; +int fd, ret, alen, off =0 ; + + alen = is_ipv4 ?sizeof (addr4) :sizeof (addr6); + + fd =socket (AF_NETLINK ,SOCK_RAW ,NETLINK_ROUTE ); +if (fd ==-1 ) +error (1 , errno,"socket netlink" ); + +memset (data,0 ,sizeof (data)); + + nh = (void *)data; + nh->nlmsg_type =RTM_NEWROUTE ; + nh->nlmsg_flags =NLM_F_REQUEST |NLM_F_CREATE ; + off +=NLMSG_ALIGN (sizeof (*nh)); + + rt = (void *)(data + off); + rt->rtm_family = is_ipv4 ?AF_INET :AF_INET6 ; + rt->rtm_table =RT_TABLE_MAIN ; + rt->rtm_dst_len = alen <<3 ; + rt->rtm_protocol =RTPROT_BOOT ; + rt->rtm_scope =RT_SCOPE_UNIVERSE ; + rt->rtm_type =RTN_UNICAST ; + off +=NLMSG_ALIGN (sizeof (*rt)); + + rta = (void *)(data + off); + rta->rta_type =RTA_DST ; + rta->rta_len =RTA_LENGTH (alen); +if (is_ipv4) +memcpy (RTA_DATA (rta), &addr4, alen); +else +memcpy (RTA_DATA (rta), &addr6, alen); + off +=NLMSG_ALIGN (rta->rta_len); + + rta = (void *)(data + off); + rta->rta_type =RTA_OIF ; + rta->rta_len =RTA_LENGTH (sizeof (int )); + *((int *)(RTA_DATA (rta))) =1 ;//if_nametoindex("lo"); + off +=NLMSG_ALIGN (rta->rta_len); + +/* MTU is a subtype in a metrics type */ + rta = (void *)(data + off); + rta->rta_type =RTA_METRICS ; + rta->rta_len =RTA_LENGTH (0 ) +RTA_LENGTH (sizeof (int )); + off +=NLMSG_ALIGN (rta->rta_len); + +/* now fill MTU subtype. Note that it fits within above rta_len */ + rta = (void *)(((char *) rta) +RTA_LENGTH (0 )); + rta->rta_type =RTAX_MTU ; + rta->rta_len =RTA_LENGTH (sizeof (int )); + *((int *)(RTA_DATA (rta))) = mtu; + + nh->nlmsg_len = off; + + ret =sendto (fd, data, off,0 , (void *)&nladdr,sizeof (nladdr)); +if (ret != off) +error (1 , errno,"send netlink: %uB != %uB\n" , ret, off); + +if (close (fd)) +error (1 , errno,"close netlink" ); + +fprintf (stderr,"route mtu (test): %u\n" , mtu); +} +static bool send_one (int fd,int len,int gso_len,struct sockaddr *addr,socklen_t alen) {
@@ -391,7 +487,7 @@ static void run_all(int fdt, int fdr, struct sockaddr *addr, socklen_t alen)static void run_test (struct sockaddr *addr,socklen_t alen) {struct timeval tv = { .tv_usec =100 *1000 }; -int fdr, fdt; +int fdr, fdt, val; fdr =socket (addr->sa_family,SOCK_DGRAM ,0 );if (fdr ==-1 )
@@ -416,6 +512,20 @@ static void run_test(struct sockaddr *addr, socklen_t alen)run_all (fdt, fdr, addr, alen); } +if (cfg_do_connected) { +set_device_mtu (fdt,CONST_MTU_TEST +100 ); +set_route_mtu (CONST_MTU_TEST , addr->sa_family ==AF_INET ); + +if (connect (fdt, addr, alen)) +error (1 , errno,"connect" ); + + val =get_path_mtu (fdt, addr->sa_family ==AF_INET ); +if (val !=CONST_MTU_TEST ) +error (1 ,0 ,"bad path mtu %u\n" , val); + +run_all (fdt, fdr, addr,0 /* use connected addr */ ); + } +if (close (fdt))error (1 , errno,"close t" );if (close (fdr))
@@ -448,7 +558,7 @@ static void parse_opts(int argc, char **argv) {int c; -while ((c =getopt (argc, argv,"46Cst:" )) !=-1 ) { +while ((c =getopt (argc, argv,"46cCst:" )) !=-1 ) {switch (c) {case '4' : cfg_do_ipv4 = true;
@@ -456,6 +566,9 @@ static void parse_opts(int argc, char **argv)case '6' : cfg_do_ipv6 = true;break ; +case 'c' : + cfg_do_connected = true; +break ;case 'C' : cfg_do_connectionless = true;break ;
tools/testing/selftests/net/udpgso.sh +7 -0
@@ -14,3 +14,10 @@ echo "ipv6 cmsg" echo "ipv6 setsockopt" ./in_netns.sh ./udpgso -6 -C -s + +echo "ipv4 connected" +./in_netns.sh ./udpgso -4 -c + +# blocked on 2nd loopback address +# echo "ipv6 connected" +# ./in_netns.sh ./udpgso -6 -c
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 10/11 selftests: udp gso with corking
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Corked sockets take a different path to construct a udp datagram than the lockless fast path. Test this alternate path.
--- tools/testing/selftests/net/udpgso.c | 42 ++++++++++++++++++++------- tools/testing/selftests/net/udpgso.sh | 6 ++++ 2 files changed, 38 insertions(+), 10 deletions(-)
tools/testing/selftests/net/udpgso.c +32 -10
@@ -49,6 +49,7 @@ static bool cfg_do_ipv4;static bool cfg_do_ipv6;static bool cfg_do_connected;static bool cfg_do_connectionless; +static bool cfg_do_msgmore;static bool cfg_do_setsockopt;static int cfg_specific_test_id =-1 ;
@@ -369,6 +370,23 @@ static void set_route_mtu(int mtu, bool is_ipv4)fprintf (stderr,"route mtu (test): %u\n" , mtu); } +static bool __send_one (int fd,struct msghdr *msg,int flags) +{ +int ret; + + ret =sendmsg (fd, msg, flags); +if (ret ==-1 && (errno ==EMSGSIZE || errno ==ENOMEM )) +return false; +if (ret ==-1 ) +error (1 , errno,"sendmsg" ); +if (ret != msg->msg_iov->iov_len) +error (1 ,0 ,"sendto: %d != %lu" , ret, msg->msg_iov->iov_len); +if (msg->msg_flags) +error (1 ,0 ,"sendmsg: return flags 0x%x\n" , msg->msg_flags); + +return true; +} +static bool send_one (int fd,int len,int gso_len,struct sockaddr *addr,socklen_t alen) {
@@ -376,7 +394,6 @@ static bool send_one(int fd, int len, int gso_len,struct msghdr msg = {0 };struct iovec iov = {0 };struct cmsghdr *cm; -int ret; iov.iov_base = buf; iov.iov_len = len;
@@ -398,15 +415,17 @@ static bool send_one(int fd, int len, int gso_len, *((uint16_t *)CMSG_DATA (cm)) = gso_len; } - ret =sendmsg (fd, &msg,0 ); -if (ret ==-1 && (errno ==EMSGSIZE || errno ==ENOMEM )) -return false; -if (ret ==-1 ) -error (1 , errno,"sendmsg" ); -if (ret != len) -error (1 ,0 ,"sendto: %d != %u" , ret, len); +/* If MSG_MORE, send 1 byte followed by remainder */ +if (cfg_do_msgmore && len >1 ) { + iov.iov_len =1 ; +if (!__send_one (fd, &msg,MSG_MORE )) +error (1 ,0 ,"send 1B failed" ); -return true; + iov.iov_base++; + iov.iov_len = len -1 ; + } + +return __send_one (fd, &msg,0 ); }static int recv_one (int fd,int flags)
@@ -558,7 +577,7 @@ static void parse_opts(int argc, char **argv) {int c; -while ((c =getopt (argc, argv,"46cCst:" )) !=-1 ) { +while ((c =getopt (argc, argv,"46cCmst:" )) !=-1 ) {switch (c) {case '4' : cfg_do_ipv4 = true;
@@ -572,6 +591,9 @@ static void parse_opts(int argc, char **argv)case 'C' : cfg_do_connectionless = true;break ; +case 'm' : + cfg_do_msgmore = true; +break ;case 's' : cfg_do_setsockopt = true;break ;
tools/testing/selftests/net/udpgso.sh +6 -0
@@ -21,3 +21,9 @@ echo "ipv4 connected" # blocked on 2nd loopback address # echo "ipv6 connected" # ./in_netns.sh ./udpgso -6 -c + +echo "ipv4 msg_more" +./in_netns.sh ./udpgso -4 -C -m + +echo "ipv6 msg_more" +./in_netns.sh ./udpgso -6 -C -m
Signature · 1 line
2.17.0.484.g0c8726318c-goog
Willem de Bruijn
maintainerPATCH v2 11/11 selftests: udp gso benchmark
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Send udp data between a source and sink, optionally with udp gso. The two processes are expected to be run on separate hosts.
A script is included that runs them together over loopback in a single namespace for functionality testing.
--- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 3 +- tools/testing/selftests/net/udpgso_bench.sh | 74 +++ tools/testing/selftests/net/udpgso_bench_rx.c | 265 +++++++++++ tools/testing/selftests/net/udpgso_bench_tx.c | 420 ++++++++++++++++++ 5 files changed, 763 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/net/udpgso_bench.sh create mode 100644 tools/testing/selftests/net/udpgso_bench_rx.c create mode 100644 tools/testing/selftests/net/udpgso_bench_tx.c
tools/testing/selftests/net/.gitignore +2 -0
@@ -9,3 +9,5 @@ reuseport_dualstack reuseaddr_conflict tcp_mmap udpgso +udpgso_bench_rx +udpgso_bench_tx
tools/testing/selftests/net/Makefile +2 -1
@@ -6,12 +6,13 @@ CFLAGS += -I../../../../usr/include/ TEST_PROGS := run_netsocktests run_afpackettests test_bpf.sh netdevice.sh rtnetlink.sh TEST_PROGS += fib_tests.sh fib-onlink-tests.sh in_netns.sh pmtu.sh udpgso.sh +TEST_PROGS += udpgso_bench.sh TEST_GEN_FILES = socket TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy TEST_GEN_FILES += tcp_mmap TEST_GEN_PROGS = reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa TEST_GEN_PROGS += reuseport_dualstack reuseaddr_conflict -TEST_GEN_PROGS += udpgso +TEST_GEN_PROGS += udpgso udpgso_bench_tx udpgso_bench_rx include ../lib.mk
tools/testing/selftests/net/udpgso_bench.sh new file +74 -0
74 lines not on this page · raw message
tools/testing/selftests/net/udpgso_bench_rx.c new file +265 -0
265 lines not on this page · raw message
tools/testing/selftests/net/udpgso_bench_tx.c new file +420 -0
420 lines not on this page · raw message
Signature · 1 line
2.17.0.484.g0c8726318c-goog
David Miller
maintainerFrom: Willem de Bruijn <[email protected]>
Date: Thu, 26 Apr 2018 13:42:14 -0400
Quoted 7 lines
Segmentation offload reduces cycles/byte for large packets by amortizing the cost of protocol stack traversal.
This patchset implements GSO for UDP. A process can concatenate and submit multiple datagrams to the same destination in one send call by setting socket option SOL_UDP/UDP_SEGMENT with the segment size, or passing an analogous cmsg at send time.
Looks great.
Build testing revealed that with ipv6=m we have to export __udp_gso_segment (patch #2) and udp_cmsg_send (patch #6).
I added the exports while applying this series.
Nice work, thanks Willem!
Willem de Bruijn
maintainerQuoted 11 lines · David Miller
From: Willem de Bruijn <[email protected]>
Date: Thu, 26 Apr 2018 13:42:14 -0400Quoted 7 lines
Segmentation offload reduces cycles/byte for large packets by amortizing the cost of protocol stack traversal.
This patchset implements GSO for UDP. A process can concatenate and submit multiple datagrams to the same destination in one send call by setting socket option SOL_UDP/UDP_SEGMENT with the segment size, or passing an analogous cmsg at send time.
Quoted 4 lines
Looks great.
Build testing revealed that with ipv6=m we have to export
__udp_gso_segment(patch #2) andudp_cmsg_send(patch #6).
Oops :/
Thanks for fixing up this breakage, David.