Support fraglist GRO/GSO
Subsystems (4)
v3 cover Support fraglist GRO/GSO
This patchset adds support to do GRO/GSO by chaining packets of the same flow at the SKB frag_list pointer. This avoids the overhead to merge payloads into one big packet, and on the other end, if GSO is needed it avoids the overhead of splitting the big packet back to the native form.
Patch 1 adds netdev feature flags to enable fraglist GRO, this implements one of the configuration options discussed at netconf 2019.
Patch 2 adds a netdev software feature set that defaults to off and assigns the new fraglist GRO feature flag to it.
Patch 3 adds the core infrastructure to do fraglist GRO/GSO.
Patch 4 enables UDP to use fraglist GRO/GSO if configured.
I have only meaningful forwarding performance measurements. I did some tests for the local receive path with netperf and iperf, but in this case the sender that generates the packets is the bottleneck. So the benchmarks are not that meaningful for the receive path.
Paolo Abeni did some benchmarks of the local receive path for the RFC v2 version of this pachset, results can be found here:
https://www.spinics.net/lists/netdev/msg551158.html
I used my IPsec forwarding test setup for the performance measurements:
------------ ------------
-->| router 1 |-------->| router 2 |--
| ------------ ------------ |
| |
| -------------------- |
--------|Spirent Testcenter|<----------
--------------------net-next (September 7th 2019):
Single stream UDP frame size 1460 Bytes: 1.161.000 fps (13.5 Gbps).
----------------------------------------------------------------------
net-next (September 7th 2019) + standard UDP GRO/GSO (not implemented in this patchset):
Single stream UDP frame size 1460 Bytes: 1.801.000 fps (21 Gbps).
----------------------------------------------------------------------
net-next (September 7th 2019) + fraglist UDP GRO/GSO:
Single stream UDP frame size 1460 Bytes: 2.860.000 fps (33.4 Gbps).
=======================================================================
net-next (January 23th 2020):
Single stream UDP frame size 1460 Bytes: 919.000 fps (10.73 Gbps).
----------------------------------------------------------------------
net-next (January 23th 2020) + fraglist UDP GRO/GSO:
Single stream UDP frame size 1460 Bytes: 2.430.000 fps (28.38 Gbps).
-----------------------------------------------------------------------
Changes from RFC v1:
- Add IPv6 support.
- Split patchset to enable UDP GRO by default before adding fraglist GRO support.
- Mark fraglist GRO packets as CHECKSUM_NONE.
- Take a refcount on the first segment skb when doing fraglist segmentation. With this we can use the same error handling path as with standard segmentation.
Changes from RFC v2:
- Add a netdev feature flag to configure listifyed GRO.
- Fix UDP GRO enabling for IPv6.
- Fix a rcu_read_lock() imbalance.
- Fix error path in skb_segment_list().
Changes from RFC v3:
- Rename NETIF_F_GRO_LIST to NETIF_F_GRO_FRAGLIST and add NETIF_F_GSO_FRAGLIST.
- Move introduction of SKB_GSO_FRAGLIST to patch 2.
- Use udpv6_encap_needed_key instead of udp_encap_needed_key in IPv6.
- Move some missplaced code from patch 5 to patch 1 where it belongs to.
Changes from RFC v4:
- Drop the 'UDP: enable GRO by default' patch for now. Standard UDP GRO is not changed with this patchset.
- Rebase to net-next current.
Changes fom v1 (December 18th):
- Do a full __copy_skb_header instead of tryng to find the really needed subset header fields. Thisa can be done later.
- Mark all fraglist GRO packets with CHECKSUM_UNNECESSARY.
- Rebase to net-next current.
Changes fom v2 (January 24th):
- Do the CHECKSUM_UNNECESSARY setting from IPv4 for IPv6 too.
PATCH v3 1/4 net: Add fraglist GRO/GSO feature flags
This adds new Fraglist GRO/GSO feature flags. They will be used to configure fraglist GRO/GSO what will be implemented with some followup paches.
--- include/linux/netdev_features.h | 6 +++++- include/linux/netdevice.h | 1 + include/linux/skbuff.h | 2 ++ net/ethtool/common.c | 1 + 4 files changed, 9 insertions(+), 1 deletion(-)
include/linux/netdev_features.h +5 -1
@@ -53,8 +53,9 @@ enum {NETIF_F_GSO_ESP_BIT ,/* ... ESP with TSO */ NETIF_F_GSO_UDP_BIT ,/* ... UFO, deprecated except tuntap */ NETIF_F_GSO_UDP_L4_BIT ,/* ... UDP payload GSO (not UFO) */ +NETIF_F_GSO_FRAGLIST_BIT ,/* ... Fraglist GSO */ /**/ NETIF_F_GSO_LAST =/* last bit, see GSO_MASK */ -NETIF_F_GSO_UDP_L4_BIT , +NETIF_F_GSO_FRAGLIST_BIT ,NETIF_F_FCOE_CRC_BIT ,/* FCoE CRC32 */ NETIF_F_SCTP_CRC_BIT, /* SCTP checksum offload */
@@ -80,6 +81,7 @@ enum {NETIF_F_GRO_HW_BIT ,/* Hardware Generic receive offload */ NETIF_F_HW_TLS_RECORD_BIT ,/* Offload TLS record */ + NETIF_F_GRO_FRAGLIST_BIT, /* Fraglist GRO *//* * Add your fresh new feature above and remember to update
@@ -150,6 +152,8 @@ enum {#define NETIF_F_GSO_UDP_L4 __NETIF_F(GSO_UDP_L4)#define NETIF_F_HW_TLS_TX __NETIF_F(HW_TLS_TX)#define NETIF_F_HW_TLS_RX __NETIF_F(HW_TLS_RX) +#define NETIF_F_GRO_FRAGLIST __NETIF_F(GRO_FRAGLIST) +#define NETIF_F_GSO_FRAGLIST __NETIF_F(GSO_FRAGLIST)/* Finds the next feature with the highest number of the range of start till 0. */
include/linux/netdevice.h +1 -0
@@ -4564,6 +4564,7 @@ static inline bool net_gso_ok(netdev_features_t features, int gso_type)BUILD_BUG_ON (SKB_GSO_ESP != (NETIF_F_GSO_ESP >>NETIF_F_GSO_SHIFT ));BUILD_BUG_ON (SKB_GSO_UDP != (NETIF_F_GSO_UDP >>NETIF_F_GSO_SHIFT ));BUILD_BUG_ON (SKB_GSO_UDP_L4 != (NETIF_F_GSO_UDP_L4 >>NETIF_F_GSO_SHIFT )); +BUILD_BUG_ON (SKB_GSO_FRAGLIST != (NETIF_F_GSO_FRAGLIST >>NETIF_F_GSO_SHIFT ));return (features & feature) == feature; }
include/linux/skbuff.h +2 -0
@@ -592,6 +592,8 @@ enum {SKB_GSO_UDP =1 <<16 ,SKB_GSO_UDP_L4 =1 <<17 , + + SKB_GSO_FRAGLIST = 1 << 18, };#if BITS_PER_LONG >32
net/ethtool/common.c +1 -0
@@ -59,6 +59,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_HW_TLS_RECORD_BIT] = "tls-hw-record", [NETIF_F_HW_TLS_TX_BIT] = "tls-hw-tx-offload", [NETIF_F_HW_TLS_RX_BIT] = "tls-hw-rx-offload", + [NETIF_F_GRO_FRAGLIST_BIT] = "rx-gro-list", }; const char
Signature · 1 line
2.17.1
PATCH v3 2/4 net: Add a netdev software feature set that defaults to off.
In reply to Steffen KlassertThe previous patch added the NETIF_F_GRO_FRAGLIST feature. This is a software feature that should default to off. Current software features default to on, so add a new feature set that defaults to off.
--- include/linux/netdev_features.h | 3 +++ net/core/dev.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/netdev_features.h +3 -0
@@ -230,6 +230,9 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)/* changeable features with no special hardware requirements */ #define NETIF_F_SOFT_FEATURES (NETIF_F_GSO | NETIF_F_GRO) +/* Changeable features with no special hardware requirements that defaults to off. */ +#define NETIF_F_SOFT_FEATURES_OFF NETIF_F_GRO_FRAGLIST +#define NETIF_F_VLAN_FEATURES (NETIF_F_HW_VLAN_CTAG_FILTER | \ NETIF_F_HW_VLAN_CTAG_RX | \ NETIF_F_HW_VLAN_CTAG_TX | \
net/core/dev.c +1 -1
@@ -9272,7 +9272,7 @@ int register_netdevice(struct net_device *dev)/* Transfer changeable features to wanted_features and enable * software offloads (GSO and GRO). */ - dev->hw_features |=NETIF_F_SOFT_FEATURES ; + dev->hw_features |= (NETIF_F_SOFT_FEATURES |NETIF_F_SOFT_FEATURES_OFF ); dev->features |=NETIF_F_SOFT_FEATURES ;if (dev->netdev_ops->ndo_udp_tunnel_add) {
Signature · 1 line
2.17.1
PATCH v3 3/4 net: Support GRO/GSO fraglist chaining.
In reply to Steffen KlassertThis patch adds the core functions to chain/unchain GSO skbs at the frag_list pointer. This also adds a new GSO type SKB_GSO_FRAGLIST and a is_flist flag to napi_gro_cb which indicates that this flow will be GROed by fraglist chaining.
--- include/linux/netdevice.h | 4 +- include/linux/skbuff.h | 2 + net/core/dev.c | 2 +- net/core/skbuff.c | 91 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 97 insertions(+), 2 deletions(-)
include/linux/netdevice.h +3 -1
@@ -2322,7 +2322,8 @@ struct napi_gro_cb {/* Number of gro_receive callbacks this packet already went through */ u8 recursion_counter:4; -/* 1 bit hole */ +/* GRO is done by frag_list pointer chaining. */ + u8 is_flist:1;/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum;
@@ -2690,6 +2691,7 @@ struct net_device *dev_get_by_napi_id(unsigned int napi_id);int netdev_get_name (struct net *net,char *name,int ifindex);int dev_restart (struct net_device *dev);int skb_gro_receive (struct sk_buff *p,struct sk_buff *skb); +int skb_gro_receive_list (struct sk_buff *p,struct sk_buff *skb);static inline unsigned int skb_gro_offset (const struct sk_buff *skb) {
include/linux/skbuff.h +2 -0
@@ -3530,6 +3530,8 @@ void skb_scrub_packet(struct sk_buff *skb, bool xnet);bool skb_gso_validate_network_len (const struct sk_buff *skb,unsigned int mtu);bool skb_gso_validate_mac_len (const struct sk_buff *skb,unsigned int len);struct sk_buff *skb_segment (struct sk_buff *skb,netdev_features_t features); +struct sk_buff *skb_segment_list (struct sk_buff *skb,netdev_features_t features, +unsigned int offset);struct sk_buff *skb_vlan_untag (struct sk_buff *skb);int skb_ensure_writable (struct sk_buff *skb,int write_len);int __skb_vlan_pop (struct sk_buff *skb,u16 *vlan_tci);
net/core/dev.c +1 -1
@@ -3249,7 +3249,7 @@ struct sk_buff *__skb_gso_segment(struct sk_buff *skb, segs =skb_mac_gso_segment (skb, features); -if (unlikely (skb_needs_check (skb, tx_path) && !IS_ERR (segs))) +if (segs != skb &&unlikely (skb_needs_check (skb, tx_path) && !IS_ERR (segs)))skb_warn_bad_offload (skb);return segs;
net/core/skbuff.c +91 -0
97 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 4/4 udp: Support UDP fraglist GRO/GSO.
In reply to Steffen KlassertThis patch extends UDP GRO to support fraglist GRO/GSO by using the previously introduced infrastructure. If the feature is enabled, all UDP packets are going to fraglist GRO (local input and forward).
After validating the csum, we mark ip_summed as CHECKSUM_UNNECESSARY for fraglist GRO packets to make sure that the csum is not touched.
--- include/net/udp.h | 2 +- net/ipv4/udp_offload.c | 104 ++++++++++++++++++++++++++++++++--------- net/ipv6/udp_offload.c | 27 ++++++++++- 3 files changed, 107 insertions(+), 26 deletions(-)
include/net/udp.h +1 -1
@@ -167,7 +167,7 @@ typedef struct sock *(*udp_lookup_t)(struct sk_buff *skb, __be16 sport, __be16 dport);struct sk_buff *udp_gro_receive (struct list_head *head,struct sk_buff *skb, -struct udphdr *uh,udp_lookup_t lookup); +struct udphdr *uh,struct sock *sk);int udp_gro_complete (struct sk_buff *skb,int nhoff,udp_lookup_t lookup); struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,
net/ipv4/udp_offload.c +81 -23
200 lines not on this page · raw message
net/ipv6/udp_offload.c +25 -2
@@ -115,8 +115,10 @@ INDIRECT_CALLABLE_SCOPEstruct sk_buff *udp6_gro_receive (struct list_head *head,struct sk_buff *skb) {struct udphdr *uh =udp_gro_udphdr (skb); +struct sk_buff *pp; +struct sock *sk; -if (unlikely (!uh) || !static_branch_unlikely (&udpv6_encap_needed_key)) +if (unlikely (!uh)) goto flush;/* Don't bother verifying checksum if we're going to flush anyway. */
@@ -132,7 +134,11 @@ struct sk_buff *udp6_gro_receive(struct list_head *head, struct sk_buff *skb) skip:NAPI_GRO_CB (skb)->is_ipv6 =1 ; -return udp_gro_receive (head, skb, uh, udp6_lib_lookup_skb); +rcu_read_lock (); + sk =static_branch_unlikely (&udpv6_encap_needed_key) ?udp6_lib_lookup_skb (skb, uh->source, uh->dest) :NULL ; + pp =udp_gro_receive (head, skb, uh, sk); +rcu_read_unlock (); +return pp; flush:NAPI_GRO_CB (skb)->flush =1 ;
@@ -144,6 +150,23 @@ INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff)const struct ipv6hdr *ipv6h =ipv6_hdr (skb);struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); +if (NAPI_GRO_CB (skb)->is_flist) { + uh->len =htons (skb->len - nhoff); + +skb_shinfo (skb)->gso_type |= (SKB_GSO_FRAGLIST |SKB_GSO_UDP_L4 ); +skb_shinfo (skb)->gso_segs =NAPI_GRO_CB (skb)->count; + +if (skb->ip_summed ==CHECKSUM_UNNECESSARY ) { +if (skb->csum_level <SKB_MAX_CSUM_LEVEL ) + skb->csum_level++; + }else { + skb->ip_summed =CHECKSUM_UNNECESSARY ; + skb->csum_level =0 ; + } + +return 0 ; + } +if (uh->check) uh->check = ~udp_v6_check (skb->len - nhoff, &ipv6h->saddr, &ipv6h->daddr,0 );
Signature · 1 line
2.17.1
Re: [PATCH net-next v3 4/4] udp: Support UDP fraglist GRO/GSO.
Quoted 11 lines · Steffen Klassert
This patch extends UDP GRO to support fraglist GRO/GSO by using the previously introduced infrastructure. If the feature is enabled, all UDP packets are going to fraglist GRO (local input and forward).
After validating the csum, we mark
ip_summedasCHECKSUM_UNNECESSARYfor fraglist GRO packets to make sure that the csum is not touched.Signed-off-by Steffen Klassert <[email protected]>
David Miller
maintainerFrom: Steffen Klassert <[email protected]>
Date: Sat, 25 Jan 2020 11:26:41 +0100
Quoted 5 lines
This patchset adds support to do GRO/GSO by chaining packets of the same flow at the SKB
frag_listpointer. This avoids the overhead to merge payloads into one big packet, and on the other end, if GSO is needed it avoids the overhead of splitting the big packet back to the native form.
...
Series applied, thank you Steffen.