Skip to contentSkip to conversation

udp and configurable gro

RFC PATCH In review Willem de Bruijn 5 participants 31 messages Sep 14, 2018 – Oct 8, 2018

Subsystems (8)
lore
Share & tools
lore

Willem de Bruijn

author maintainer
#0 lore raw

cover udp and configurable gro

From: Willem de Bruijn <[email protected]>

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

Reduce UDP receive cost for bulk traffic by enabling datagram coalescing with GRO.

Before adding more GRO callbacks, make GRO configurable by the administrator to optionally reduce the attack surface of this early receive path. See also [2].

Introduce sysctls net.(core|ipv4|ipv6).gro that expose the table of protocols for which GRO is support. Allow the administrator to disable individual entries in the table.

To have a single infrastructure, convert dev_offloads to the table-based approach to existing inet(6)_offloads. Additional small benefit is that ipv6 will no longer take two list lookups to find.

Patch 1 converts dev_offloads to the infra of inet(6)_offloads Patch 2 deduplicates gro_complete logic now that all share infra Patch 3 does the same for gro_receive, in anticipation of adding
a branch to check whether gro_receive is enabled
Patch 4 harmonizes ipv6 header opts, so that those, too can be
optionally disabled.
Patch 5 makes inet(6)_offloads non-const to allow disabling a flag Patch 6 introduces the administrative sysctl
Patch 7 avoids udp gro cost if no udp gro callback is register Patch 8 introduces udp gro

[1] http://patchwork.ozlabs.org/project/netdev/list/?series=65741
[2] http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf

Willem de Bruijn (8):
gro: convert device offloads to net_offload
gro: deduplicate gro_complete
gro: add net_gro_receive
ipv6: remove offload exception for hopopts
net: deconstify net_offload
net: make gro configurable
udp: gro behind static key
udp: add gro

 drivers/net/geneve.c       |  11 +---
 drivers/net/vxlan.c        |   8 +++
 include/linux/netdevice.h  |  64 +++++++++++++++++++--
 include/net/protocol.h     |  19 ++-----
 include/net/udp.h          |   2 +
 include/uapi/linux/udp.h   |   1 +
 net/8021q/vlan.c           |  12 +---
 net/core/dev.c             | 112 ++++++++-----------------------------
 net/core/sysctl_net_core.c |  60 ++++++++++++++++++++
 net/ethernet/eth.c         |  13 +----
 net/ipv4/af_inet.c         |  21 ++-----
 net/ipv4/esp4_offload.c    |   2 +-
 net/ipv4/fou.c             |  41 ++++----------
 net/ipv4/gre_offload.c     |  26 ++++-----
 net/ipv4/protocol.c        |  10 ++--
 net/ipv4/sysctl_net_ipv4.c |   7 +++
 net/ipv4/tcp_offload.c     |   2 +-
 net/ipv4/udp.c             |  73 +++++++++++++++++++++++-
 net/ipv4/udp_offload.c     |  19 +++----
 net/ipv6/esp6_offload.c    |   2 +-
 net/ipv6/exthdrs_offload.c |  17 +++++-
 net/ipv6/ip6_offload.c     |  69 +++++++++--------------
 net/ipv6/protocol.c        |  10 ++--
 net/ipv6/sysctl_net_ipv6.c |   8 +++
 net/ipv6/tcpv6_offload.c   |   2 +-
 net/ipv6/udp.c             |   2 +-
 net/ipv6/udp_offload.c     |   4 +-
 net/sctp/offload.c         |   2 +-
 28 files changed, 344 insertions(+), 275 deletions(-)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#1 lore raw

RFC PATCH 1/8 gro: convert device offloads to net_offload

From: Willem de Bruijn <[email protected]>

In preparation of making GRO receive configurable, have all offloads share the same infrastructure.

---
 include/linux/netdevice.h |  17 +++++-
 include/net/protocol.h    |   7 ---
 net/core/dev.c            | 105 +++++++++++++-------------------------
 3 files changed, 51 insertions(+), 78 deletions(-)
include/linux/netdevice.h +15 -2
index e2b3bd750c98..7425068fa249 100644
@@ -2366,13 +2366,18 @@ struct offload_callbacks {
 	int			(*gro_complete)(struct sk_buff *skb, int nhoff);
 };
 
-struct packet_offload {
+struct net_offload {
 	__be16			 type;	/* This is really htons(ether_type). */
 	u16			 priority;
 	struct offload_callbacks callbacks;
-	struct list_head	 list;
+	unsigned int		 flags;	/* Flags used by IPv6 for now */
 };
 
+#define packet_offload	net_offload
+
+/* This should be set for any extension header which is compatible with GSO. */
+#define INET6_PROTO_GSO_EXTHDR	0x1
+
 /* often modified stats are per-CPU, other are shared (netdev->stats) */
 struct pcpu_sw_netstats {
 	u64     rx_packets;
@@ -3554,6 +3559,14 @@ gro_result_t napi_gro_frags(struct napi_struct *napi);
 struct packet_offload *gro_find_receive_by_type(__be16 type);
 struct packet_offload *gro_find_complete_by_type(__be16 type);
 
+static inline u8 net_offload_from_type(u16 type)
+{
+	/* Do not bother handling collisions. There are none.
+	 * If they do occur with new offloads, add a mapping function here.
+	 */
+	return type & 0xFF;
+}
+
 static inline void napi_free_frags(struct napi_struct *napi)
 {
 	kfree_skb(napi->skb);
include/net/protocol.h +0 -7
index 4fc75f7ae23b..53a0322ee545 100644
@@ -69,13 +69,6 @@ struct inet6_protocol {
 #define INET6_PROTO_FINAL	0x2
 #endif
 
-struct net_offload {
-	struct offload_callbacks callbacks;
-	unsigned int		 flags;	/* Flags used by IPv6 for now */
-};
-/* This should be set for any extension header which is compatible with GSO. */
-#define INET6_PROTO_GSO_EXTHDR	0x1
-
 /* This is used to register socket interfaces for IP protocols.  */
 struct inet_protosw {
 	struct list_head list;
net/core/dev.c +36 -69

191 lines not on this page · raw message

Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#2 lore raw

RFC PATCH 2/8 gro: deduplicate gro_complete

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

The gro completion datapath is open coded for all protocols. Deduplicate with new helper function net_gro_complete.

---
 drivers/net/geneve.c      |  9 +--------
 include/linux/netdevice.h | 19 ++++++++++++++++++-
 net/8021q/vlan.c          | 10 +---------
 net/core/dev.c            | 24 +-----------------------
 net/ethernet/eth.c        | 11 +----------
 net/ipv4/af_inet.c        | 15 ++-------------
 net/ipv4/fou.c            | 25 +++----------------------
 net/ipv4/gre_offload.c    | 12 +++---------
 net/ipv6/ip6_offload.c    | 13 +------------
 9 files changed, 31 insertions(+), 107 deletions(-)
drivers/net/geneve.c +1 -8
index 6625fabe2c88..a3a4621d9bee 100644
@@ -488,7 +488,6 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb,
 			       int nhoff)
 {
 	struct genevehdr *gh;
-	struct packet_offload *ptype;
 	__be16 type;
 	int gh_len;
 	int err = -ENOSYS;
@@ -497,13 +496,7 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb,
 	gh_len = geneve_hlen(gh);
 	type = gh->proto_type;
 
-	rcu_read_lock();
-	ptype = gro_find_complete_by_type(type);
-	if (ptype)
-		err = ptype->callbacks.gro_complete(skb, nhoff + gh_len);
-
-	rcu_read_unlock();
-
+	err = net_gro_complete(dev_offloads, type, skb, nhoff + gh_len);
 	skb_set_inner_mac_header(skb, nhoff + gh_len);
 
 	return err;
include/linux/netdevice.h +18 -1
index 7425068fa249..0d292ea6716e 100644
@@ -3557,7 +3557,8 @@ void napi_gro_flush(struct napi_struct *napi, bool flush_old);
 struct sk_buff *napi_get_frags(struct napi_struct *napi);
 gro_result_t napi_gro_frags(struct napi_struct *napi);
 struct packet_offload *gro_find_receive_by_type(__be16 type);
-struct packet_offload *gro_find_complete_by_type(__be16 type);
+
+extern const struct net_offload __rcu *dev_offloads[256];
 
 static inline u8 net_offload_from_type(u16 type)
 {
@@ -3567,6 +3568,22 @@ static inline u8 net_offload_from_type(u16 type)
 	return type & 0xFF;
 }
 
+static inline int net_gro_complete(const struct net_offload __rcu **offs,
+				   u16 type, struct sk_buff *skb, int nhoff)
+{
+	const struct net_offload *off;
+	int ret = -ENOENT;
+
+	rcu_read_lock();
+	off = rcu_dereference(offs[net_offload_from_type(type)]);
+	if (off && off->callbacks.gro_complete &&
+	    (!off->type || off->type == type))
+		ret = off->callbacks.gro_complete(skb, nhoff);
+	rcu_read_unlock();
+
+	return ret;
+}
+
 static inline void napi_free_frags(struct napi_struct *napi)
 {
 	kfree_skb(napi->skb);
net/8021q/vlan.c +1 -9
index 5e9950453955..6ac27aa9f158 100644
@@ -703,16 +703,8 @@ static int vlan_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);
 	__be16 type = vhdr->h_vlan_encapsulated_proto;
-	struct packet_offload *ptype;
-	int err = -ENOENT;
 
-	rcu_read_lock();
-	ptype = gro_find_complete_by_type(type);
-	if (ptype)
-		err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(*vhdr));
-
-	rcu_read_unlock();
-	return err;
+	return net_gro_complete(dev_offloads, type, skb, nhoff + sizeof(*vhdr));
 }
 
 static struct packet_offload vlan_packet_offloads[] __read_mostly = {
net/core/dev.c +1 -23
index 55f86b6d3182..2c21e507291f 100644
@@ -5235,10 +5235,6 @@ static void flush_all_backlogs(void)
 
 static int napi_gro_complete(struct sk_buff *skb)
 {
-	const struct packet_offload *ptype;
-	__be16 type = skb->protocol;
-	int err = -ENOENT;
-
 	BUILD_BUG_ON(sizeof(struct napi_gro_cb) > sizeof(skb->cb));
 
 	if (NAPI_GRO_CB(skb)->count == 1) {
@@ -5246,13 +5242,7 @@ static int napi_gro_complete(struct sk_buff *skb)
 		goto out;
 	}
 
-	rcu_read_lock();
-	ptype = dev_offloads[net_offload_from_type(type)];
-	if (ptype && ptype->callbacks.gro_complete)
-		err = ptype->callbacks.gro_complete(skb, 0);
-	rcu_read_unlock();
-
-	if (err) {
+	if (net_gro_complete(dev_offloads, skb->protocol, skb, 0)) {
 		kfree_skb(skb);
 		return NET_RX_SUCCESS;
 	}
@@ -5505,18 +5495,6 @@ struct packet_offload *gro_find_receive_by_type(__be16 type)
 }
 EXPORT_SYMBOL(gro_find_receive_by_type);
 
-struct packet_offload *gro_find_complete_by_type(__be16 type)
-{
-	struct net_offload *off;
-
-	off = (struct net_offload *) rcu_dereference(dev_offloads[type & 0xFF]);
-	if (off && off->type == type && off->callbacks.gro_complete)
-		return off;
-	else
-		return NULL;
-}
-EXPORT_SYMBOL(gro_find_complete_by_type);
-
 static void napi_skb_free_stolen_head(struct sk_buff *skb)
 {
 	skb_dst_drop(skb);
net/ethernet/eth.c +1 -10
index fd8faa0dfa61..fb17a13722e8 100644
@@ -485,20 +485,11 @@ int eth_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff);
 	__be16 type = eh->h_proto;
-	struct packet_offload *ptype;
-	int err = -ENOSYS;
 
 	if (skb->encapsulation)
 		skb_set_inner_mac_header(skb, nhoff);
 
-	rcu_read_lock();
-	ptype = gro_find_complete_by_type(type);
-	if (ptype != NULL)
-		err = ptype->callbacks.gro_complete(skb, nhoff +
-						    sizeof(struct ethhdr));
-
-	rcu_read_unlock();
-	return err;
+	return net_gro_complete(dev_offloads, type, skb, nhoff + sizeof(*eh));
 }
 EXPORT_SYMBOL(eth_gro_complete);
 
net/ipv4/af_inet.c +2 -13
index 1fbe2f815474..1b72ee4a7811 100644
@@ -1560,9 +1560,7 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	__be16 newlen = htons(skb->len - nhoff);
 	struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);
-	const struct net_offload *ops;
 	int proto = iph->protocol;
-	int err = -ENOSYS;
 
 	if (skb->encapsulation) {
 		skb_set_inner_protocol(skb, cpu_to_be16(ETH_P_IP));
@@ -1572,21 +1570,12 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)
 	csum_replace2(&iph->check, iph->tot_len, newlen);
 	iph->tot_len = newlen;
 
-	rcu_read_lock();
-	ops = rcu_dereference(inet_offloads[proto]);
-	if (WARN_ON(!ops || !ops->callbacks.gro_complete))
-		goto out_unlock;
-
 	/* Only need to add sizeof(*iph) to get to the next hdr below
 	 * because any hdr with option will have been flushed in
 	 * inet_gro_receive().
 	 */
-	err = ops->callbacks.gro_complete(skb, nhoff + sizeof(*iph));
-
-out_unlock:
-	rcu_read_unlock();
-
-	return err;
+	return net_gro_complete(inet_offloads, proto, skb,
+				nhoff + sizeof(*iph));
 }
 EXPORT_SYMBOL(inet_gro_complete);
 
net/ipv4/fou.c +3 -22
index 500a59906b87..c42a3ef17864 100644
@@ -261,24 +261,14 @@ static struct sk_buff *fou_gro_receive(struct sock *sk,
 static int fou_gro_complete(struct sock *sk, struct sk_buff *skb,
 			    int nhoff)
 {
-	const struct net_offload *ops;
 	u8 proto = fou_from_sock(sk)->protocol;
-	int err = -ENOSYS;
 	const struct net_offload **offloads;
+	int err;
 
-	rcu_read_lock();
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
-	ops = rcu_dereference(offloads[proto]);
-	if (WARN_ON(!ops || !ops->callbacks.gro_complete))
-		goto out_unlock;
-
-	err = ops->callbacks.gro_complete(skb, nhoff);
-
+	err = net_gro_complete(offloads, proto, skb, nhoff);
 	skb_set_inner_mac_header(skb, nhoff);
 
-out_unlock:
-	rcu_read_unlock();
-
 	return err;
 }
 
@@ -457,7 +447,6 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff)
 {
 	const struct net_offload **offloads;
 	struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff);
-	const struct net_offload *ops;
 	unsigned int guehlen = 0;
 	u8 proto;
 	int err = -ENOENT;
@@ -483,18 +472,10 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff)
 		return err;
 	}
 
-	rcu_read_lock();
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
-	ops = rcu_dereference(offloads[proto]);
-	if (WARN_ON(!ops || !ops->callbacks.gro_complete))
-		goto out_unlock;
-
-	err = ops->callbacks.gro_complete(skb, nhoff + guehlen);
-
+	err = net_gro_complete(offloads, proto, skb, nhoff + guehlen);
 	skb_set_inner_mac_header(skb, nhoff + guehlen);
 
-out_unlock:
-	rcu_read_unlock();
 	return err;
 }
 
net/ipv4/gre_offload.c +3 -9
index 6c63524f598a..fc8c99e4a058 100644
@@ -231,10 +231,9 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,
 static int gre_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	struct gre_base_hdr *greh = (struct gre_base_hdr *)(skb->data + nhoff);
-	struct packet_offload *ptype;
 	unsigned int grehlen = sizeof(*greh);
-	int err = -ENOENT;
 	__be16 type;
+	int err;
 
 	skb->encapsulation = 1;
 	skb_shinfo(skb)->gso_type = SKB_GSO_GRE;
@@ -246,13 +245,8 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff)
 	if (greh->flags & GRE_CSUM)
 		grehlen += GRE_HEADER_SECTION;
 
-	rcu_read_lock();
-	ptype = gro_find_complete_by_type(type);
-	if (ptype)
-		err = ptype->callbacks.gro_complete(skb, nhoff + grehlen);
-
-	rcu_read_unlock();
-
+	err = net_gro_complete(dev_offloads, type, skb,
+				nhoff + sizeof(*greh));
 	skb_set_inner_mac_header(skb, nhoff + grehlen);
 
 	return err;
net/ipv6/ip6_offload.c +1 -12
index c7e495f12011..e8bf554ae611 100644
@@ -298,7 +298,6 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 {
 	const struct net_offload *ops;
 	struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
-	int err = -ENOSYS;
 
 	if (skb->encapsulation) {
 		skb_set_inner_protocol(skb, cpu_to_be16(ETH_P_IPV6));
@@ -307,18 +306,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 
 	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
 
-	rcu_read_lock();
-
 	nhoff += sizeof(*iph) + ipv6_exthdrs_len(iph, &ops);
-	if (WARN_ON(!ops || !ops->callbacks.gro_complete))
-		goto out_unlock;
-
-	err = ops->callbacks.gro_complete(skb, nhoff);
-
-out_unlock:
-	rcu_read_unlock();
-
-	return err;
+	return net_gro_complete(inet6_offloads, ops->type, skb, nhoff);
 }
 
 static int sit_gro_complete(struct sk_buff *skb, int nhoff)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#3 lore raw

RFC PATCH 3/8 gro: add net_gro_receive

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

For configurable gro_receive all callsites need to be updated. Similar to gro_complete, introduce a single shared helper, net_gro_receive.

---
 drivers/net/geneve.c      |  2 +-
 include/linux/netdevice.h | 14 +++++++++++++-
 net/8021q/vlan.c          |  2 +-
 net/core/dev.c            | 20 ++++----------------
 net/ethernet/eth.c        |  2 +-
 net/ipv4/af_inet.c        |  4 ++--
 net/ipv4/fou.c            |  8 ++++----
 net/ipv4/gre_offload.c    | 12 ++++++------
 net/ipv6/ip6_offload.c    |  8 ++++----
 9 files changed, 36 insertions(+), 36 deletions(-)
drivers/net/geneve.c +1 -1
index a3a4621d9bee..a812a774e5fd 100644
@@ -467,7 +467,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk,
 	type = gh->proto_type;
 
 	rcu_read_lock();
-	ptype = gro_find_receive_by_type(type);
+	ptype = net_gro_receive(dev_offloads, type);
 	if (!ptype)
 		goto out_unlock;
 
include/linux/netdevice.h +13 -1
index 0d292ea6716e..0be594f8d1ce 100644
@@ -3556,7 +3556,6 @@ gro_result_t napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb);
 void napi_gro_flush(struct napi_struct *napi, bool flush_old);
 struct sk_buff *napi_get_frags(struct napi_struct *napi);
 gro_result_t napi_gro_frags(struct napi_struct *napi);
-struct packet_offload *gro_find_receive_by_type(__be16 type);
 
 extern const struct net_offload __rcu *dev_offloads[256];
 
@@ -3568,6 +3567,19 @@ static inline u8 net_offload_from_type(u16 type)
 	return type & 0xFF;
 }
 
+static inline const struct net_offload *
+net_gro_receive(const struct net_offload __rcu **offs, u16 type)
+{
+	const struct net_offload *off;
+
+	off = rcu_dereference(offs[net_offload_from_type(type)]);
+	if (off && off->callbacks.gro_receive &&
+	    (!off->type || off->type == type))
+		return off;
+	else
+		return NULL;
+}
+
 static inline int net_gro_complete(const struct net_offload __rcu **offs,
 				   u16 type, struct sk_buff *skb, int nhoff)
 {
net/8021q/vlan.c +1 -1
index 6ac27aa9f158..a106c5373b1d 100644
@@ -670,7 +670,7 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head,
 	type = vhdr->h_vlan_encapsulated_proto;
 
 	rcu_read_lock();
-	ptype = gro_find_receive_by_type(type);
+	ptype = net_gro_receive(dev_offloads, type);
 	if (!ptype)
 		goto out_unlock;
 
net/core/dev.c +4 -16
index 2c21e507291f..ae5fbd4114d2 100644
@@ -5382,7 +5382,7 @@ static void gro_flush_oldest(struct list_head *head)
 static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
 {
 	u32 hash = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1);
-	const struct packet_offload *ptype;
+	const struct net_offload *ops;
 	__be16 type = skb->protocol;
 	struct list_head *gro_head;
 	struct sk_buff *pp = NULL;
@@ -5396,8 +5396,8 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	gro_head = gro_list_prepare(napi, skb);
 
 	rcu_read_lock();
-	ptype = dev_offloads[net_offload_from_type(type)];
-	if (ptype && ptype->callbacks.gro_receive) {
+	ops = net_gro_receive(dev_offloads, type);
+	if (ops) {
 		skb_set_network_header(skb, skb_gro_offset(skb));
 		skb_reset_mac_len(skb);
 		NAPI_GRO_CB(skb)->same_flow = 0;
@@ -5425,7 +5425,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 			NAPI_GRO_CB(skb)->csum_valid = 0;
 		}
 
-		pp = ptype->callbacks.gro_receive(gro_head, skb);
+		pp = ops->callbacks.gro_receive(gro_head, skb);
 		rcu_read_unlock();
 	} else {
 		rcu_read_unlock();
@@ -5483,18 +5483,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	goto pull;
 }
 
-struct packet_offload *gro_find_receive_by_type(__be16 type)
-{
-	struct net_offload *off;
-
-	off = (struct net_offload *) rcu_dereference(dev_offloads[type & 0xFF]);
-	if (off && off->type == type && off->callbacks.gro_receive)
-		return off;
-	else
-		return NULL;
-}
-EXPORT_SYMBOL(gro_find_receive_by_type);
-
 static void napi_skb_free_stolen_head(struct sk_buff *skb)
 {
 	skb_dst_drop(skb);
net/ethernet/eth.c +1 -1
index fb17a13722e8..542dbc2ec956 100644
@@ -462,7 +462,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb)
 	type = eh->h_proto;
 
 	rcu_read_lock();
-	ptype = gro_find_receive_by_type(type);
+	ptype = net_gro_receive(dev_offloads, type);
 	if (ptype == NULL) {
 		flush = 1;
 		goto out_unlock;
net/ipv4/af_inet.c +2 -2
index 1b72ee4a7811..28b7c7671789 100644
@@ -1409,8 +1409,8 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
 	proto = iph->protocol;
 
 	rcu_read_lock();
-	ops = rcu_dereference(inet_offloads[proto]);
-	if (!ops || !ops->callbacks.gro_receive)
+	ops = net_gro_receive(inet_offloads, proto);
+	if (!ops)
 		goto out_unlock;
 
 	if (*(u8 *)iph != 0x45)
net/ipv4/fou.c +4 -4
index c42a3ef17864..13401cb2e7a4 100644
@@ -246,8 +246,8 @@ static struct sk_buff *fou_gro_receive(struct sock *sk,
 
 	rcu_read_lock();
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
-	ops = rcu_dereference(offloads[proto]);
-	if (!ops || !ops->callbacks.gro_receive)
+	ops = net_gro_receive(offloads, proto);
+	if (!ops)
 		goto out_unlock;
 
 	pp = call_gro_receive(ops->callbacks.gro_receive, head, skb);
@@ -428,8 +428,8 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,
 
 	rcu_read_lock();
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
-	ops = rcu_dereference(offloads[proto]);
-	if (WARN_ON_ONCE(!ops || !ops->callbacks.gro_receive))
+	ops = net_gro_receive(offloads, proto);
+	if (WARN_ON_ONCE(!ops))
 		goto out_unlock;
 
 	pp = call_gro_receive(ops->callbacks.gro_receive, head, skb);
net/ipv4/gre_offload.c +6 -6
index fc8c99e4a058..4f9237a4bea1 100644
@@ -111,13 +111,13 @@ static struct sk_buff *gre_gso_segment(struct sk_buff *skb,
 static struct sk_buff *gre_gro_receive(struct list_head *head,
 				       struct sk_buff *skb)
 {
-	struct sk_buff *pp = NULL;
-	struct sk_buff *p;
 	const struct gre_base_hdr *greh;
+	const struct net_offload *ops;
 	unsigned int hlen, grehlen;
+	struct sk_buff *pp = NULL;
+	struct sk_buff *p;
 	unsigned int off;
 	int flush = 1;
-	struct packet_offload *ptype;
 	__be16 type;
 
 	if (NAPI_GRO_CB(skb)->encap_mark)
@@ -154,8 +154,8 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,
 	type = greh->protocol;
 
 	rcu_read_lock();
-	ptype = gro_find_receive_by_type(type);
-	if (!ptype)
+	ops = net_gro_receive(dev_offloads, type);
+	if (!ops)
 		goto out_unlock;
 
 	grehlen = GRE_HEADER_SECTION;
@@ -217,7 +217,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,
 	/* Adjusted NAPI_GRO_CB(skb)->csum after skb_gro_pull()*/
 	skb_gro_postpull_rcsum(skb, greh, grehlen);
 
-	pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
+	pp = call_gro_receive(ops->callbacks.gro_receive, head, skb);
 	flush = 0;
 
 out_unlock:
net/ipv6/ip6_offload.c +4 -4
index e8bf554ae611..9d301bef0e23 100644
@@ -194,8 +194,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 
 	rcu_read_lock();
 	proto = iph->nexthdr;
-	ops = rcu_dereference(inet6_offloads[proto]);
-	if (!ops || !ops->callbacks.gro_receive) {
+	ops = net_gro_receive(inet6_offloads, proto);
+	if (!ops) {
 		__pskb_pull(skb, skb_gro_offset(skb));
 		skb_gro_frag0_invalidate(skb);
 		proto = ipv6_gso_pull_exthdrs(skb, proto);
@@ -203,8 +203,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 		skb_reset_transport_header(skb);
 		__skb_push(skb, skb_gro_offset(skb));
 
-		ops = rcu_dereference(inet6_offloads[proto]);
-		if (!ops || !ops->callbacks.gro_receive)
+		ops = net_gro_receive(inet6_offloads, proto);
+		if (!ops)
 			goto out_unlock;
 
 		iph = ipv6_hdr(skb);
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#4 lore raw

RFC PATCH 4/8 ipv6: remove offload exception for hopopts

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Extension headers in ipv6 are pulled without calling a callback function. An inet6_offload signals this feature with flag INET6_PROTO_GSO_EXTHDR.

Add net_has_flag helper to hide implementation details and in prepartion for configurable gro.

Convert NEXTHDR_HOP from a special case branch to a standard extension header offload.

---
 include/linux/netdevice.h  |  9 +++++++++
 net/ipv6/exthdrs_offload.c | 17 ++++++++++++++---
 net/ipv6/ip6_offload.c     | 36 +++++++++++++-----------------------
 3 files changed, 36 insertions(+), 26 deletions(-)
include/linux/netdevice.h +9 -0
index 0be594f8d1ce..1c97a048506f 100644
@@ -3567,6 +3567,15 @@ static inline u8 net_offload_from_type(u16 type)
 	return type & 0xFF;
 }
 
+static inline bool net_offload_has_flag(const struct net_offload __rcu **offs,
+					u16 type, u16 flag)
+{
+	const struct net_offload *off;
+
+	off = offs ? rcu_dereference(offs[net_offload_from_type(type)]) : NULL;
+	return off && off->flags & flag;
+}
+
 static inline const struct net_offload *
 net_gro_receive(const struct net_offload __rcu **offs, u16 type)
 {
net/ipv6/exthdrs_offload.c +14 -3
index f5e2ba1c18bf..2230331c6012 100644
@@ -12,11 +12,15 @@
 #include <net/protocol.h>
 #include "ip6_offload.h"
 
-static const struct net_offload rthdr_offload = {
+static struct net_offload hophdr_offload = {
 	.flags		=	INET6_PROTO_GSO_EXTHDR,
 };
 
-static const struct net_offload dstopt_offload = {
+static struct net_offload rthdr_offload = {
+	.flags		=	INET6_PROTO_GSO_EXTHDR,
+};
+
+static struct net_offload dstopt_offload = {
 	.flags		=	INET6_PROTO_GSO_EXTHDR,
 };
 
@@ -24,10 +28,14 @@ int __init ipv6_exthdrs_offload_init(void)
 {
 	int ret;
 
-	ret = inet6_add_offload(&rthdr_offload, IPPROTO_ROUTING);
+	ret = inet6_add_offload(&hophdr_offload, IPPROTO_HOPOPTS);
 	if (ret)
 		goto out;
 
+	ret = inet6_add_offload(&rthdr_offload, IPPROTO_ROUTING);
+	if (ret)
+		goto out_hop;
+
 	ret = inet6_add_offload(&dstopt_offload, IPPROTO_DSTOPTS);
 	if (ret)
 		goto out_rt;
@@ -37,5 +45,8 @@ int __init ipv6_exthdrs_offload_init(void)
 
 out_rt:
 	inet6_del_offload(&rthdr_offload, IPPROTO_ROUTING);
+
+out_hop:
+	inet6_del_offload(&rthdr_offload, IPPROTO_HOPOPTS);
 	goto out;
 }
net/ipv6/ip6_offload.c +13 -23
index 9d301bef0e23..4854509a2c5d 100644
@@ -22,21 +22,13 @@
 
 static int ipv6_gso_pull_exthdrs(struct sk_buff *skb, int proto)
 {
-	const struct net_offload *ops = NULL;
-
 	for (;;) {
 		struct ipv6_opt_hdr *opth;
 		int len;
 
-		if (proto != NEXTHDR_HOP) {
-			ops = rcu_dereference(inet6_offloads[proto]);
-
-			if (unlikely(!ops))
-				break;
-
-			if (!(ops->flags & INET6_PROTO_GSO_EXTHDR))
-				break;
-		}
+		if (!net_offload_has_flag(inet6_offloads, proto,
+					  INET6_PROTO_GSO_EXTHDR))
+			break;
 
 		if (unlikely(!pskb_may_pull(skb, 8)))
 			break;
@@ -141,26 +133,24 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,
 /* Return the total length of all the extension hdrs, following the same
  * logic in ipv6_gso_pull_exthdrs() when parsing ext-hdrs.
  */
-static int ipv6_exthdrs_len(struct ipv6hdr *iph,
-			    const struct net_offload **opps)
+static int ipv6_exthdrs_len(struct ipv6hdr *iph, u8 *pproto)
 {
 	struct ipv6_opt_hdr *opth = (void *)iph;
 	int len = 0, proto, optlen = sizeof(*iph);
 
 	proto = iph->nexthdr;
 	for (;;) {
-		if (proto != NEXTHDR_HOP) {
-			*opps = rcu_dereference(inet6_offloads[proto]);
-			if (unlikely(!(*opps)))
-				break;
-			if (!((*opps)->flags & INET6_PROTO_GSO_EXTHDR))
-				break;
-		}
+		if (!net_offload_has_flag(inet6_offloads, proto,
+					  INET6_PROTO_GSO_EXTHDR))
+			break;
+
 		opth = (void *)opth + optlen;
 		optlen = ipv6_optlen(opth);
 		len += optlen;
 		proto = opth->nexthdr;
 	}
+
+	*pproto = proto;
 	return len;
 }
 
@@ -296,8 +286,8 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head,
 
 static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 {
-	const struct net_offload *ops;
 	struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
+	u8 proto;
 
 	if (skb->encapsulation) {
 		skb_set_inner_protocol(skb, cpu_to_be16(ETH_P_IPV6));
@@ -306,8 +296,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 
 	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
 
-	nhoff += sizeof(*iph) + ipv6_exthdrs_len(iph, &ops);
-	return net_gro_complete(inet6_offloads, ops->type, skb, nhoff);
+	nhoff += sizeof(*iph) + ipv6_exthdrs_len(iph, &proto);
+	return net_gro_complete(inet6_offloads, proto, skb, nhoff);
 }
 
 static int sit_gro_complete(struct sk_buff *skb, int nhoff)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#5 lore raw

RFC PATCH 5/8 net: deconstify net_offload

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

With configurable gro, the flags field in net_offloads may be changed.

Remove the const keyword. This is a noop otherwise.

---
 include/linux/netdevice.h | 14 +++++++-------
 include/net/protocol.h    | 12 ++++++------
 net/core/dev.c            |  8 +++-----
 net/ipv4/af_inet.c        |  2 +-
 net/ipv4/esp4_offload.c   |  2 +-
 net/ipv4/fou.c            |  8 ++++----
 net/ipv4/gre_offload.c    |  2 +-
 net/ipv4/protocol.c       | 10 +++++-----
 net/ipv4/tcp_offload.c    |  2 +-
 net/ipv4/udp_offload.c    |  6 +++---
 net/ipv6/esp6_offload.c   |  2 +-
 net/ipv6/ip6_offload.c    |  6 +++---
 net/ipv6/protocol.c       | 10 +++++-----
 net/ipv6/tcpv6_offload.c  |  2 +-
 net/ipv6/udp_offload.c    |  2 +-
 net/sctp/offload.c        |  2 +-
 16 files changed, 44 insertions(+), 46 deletions(-)
include/linux/netdevice.h +7 -7
index 1c97a048506f..b9e671887fc2 100644
@@ -3557,7 +3557,7 @@ void napi_gro_flush(struct napi_struct *napi, bool flush_old);
 struct sk_buff *napi_get_frags(struct napi_struct *napi);
 gro_result_t napi_gro_frags(struct napi_struct *napi);
 
-extern const struct net_offload __rcu *dev_offloads[256];
+extern struct net_offload __rcu *dev_offloads[256];
 
 static inline u8 net_offload_from_type(u16 type)
 {
@@ -3567,19 +3567,19 @@ static inline u8 net_offload_from_type(u16 type)
 	return type & 0xFF;
 }
 
-static inline bool net_offload_has_flag(const struct net_offload __rcu **offs,
+static inline bool net_offload_has_flag(struct net_offload __rcu **offs,
 					u16 type, u16 flag)
 {
-	const struct net_offload *off;
+	struct net_offload *off;
 
 	off = offs ? rcu_dereference(offs[net_offload_from_type(type)]) : NULL;
 	return off && off->flags & flag;
 }
 
 static inline const struct net_offload *
-net_gro_receive(const struct net_offload __rcu **offs, u16 type)
+net_gro_receive(struct net_offload __rcu **offs, u16 type)
 {
-	const struct net_offload *off;
+	struct net_offload *off;
 
 	off = rcu_dereference(offs[net_offload_from_type(type)]);
 	if (off && off->callbacks.gro_receive &&
@@ -3589,10 +3589,10 @@ net_gro_receive(const struct net_offload __rcu **offs, u16 type)
 		return NULL;
 }
 
-static inline int net_gro_complete(const struct net_offload __rcu **offs,
+static inline int net_gro_complete(struct net_offload __rcu **offs,
 				   u16 type, struct sk_buff *skb, int nhoff)
 {
-	const struct net_offload *off;
+	struct net_offload *off;
 	int ret = -ENOENT;
 
 	rcu_read_lock();
include/net/protocol.h +6 -6
index 53a0322ee545..5e2c20b662d1 100644
@@ -87,8 +87,8 @@ struct inet_protosw {
 #define INET_PROTOSW_ICSK      0x04  /* Is this an inet_connection_sock? */
 
 extern struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS];
-extern const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS];
-extern const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS];
+extern struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS];
+extern struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS];
 
 #if IS_ENABLED(CONFIG_IPV6)
 extern struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS];
@@ -96,8 +96,8 @@ extern struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS];
 
 int inet_add_protocol(const struct net_protocol *prot, unsigned char num);
 int inet_del_protocol(const struct net_protocol *prot, unsigned char num);
-int inet_add_offload(const struct net_offload *prot, unsigned char num);
-int inet_del_offload(const struct net_offload *prot, unsigned char num);
+int inet_add_offload(struct net_offload *prot, unsigned char num);
+int inet_del_offload(struct net_offload *prot, unsigned char num);
 void inet_register_protosw(struct inet_protosw *p);
 void inet_unregister_protosw(struct inet_protosw *p);
 
@@ -107,7 +107,7 @@ int inet6_del_protocol(const struct inet6_protocol *prot, unsigned char num);
 int inet6_register_protosw(struct inet_protosw *p);
 void inet6_unregister_protosw(struct inet_protosw *p);
 #endif
-int inet6_add_offload(const struct net_offload *prot, unsigned char num);
-int inet6_del_offload(const struct net_offload *prot, unsigned char num);
+int inet6_add_offload(struct net_offload *prot, unsigned char num);
+int inet6_del_offload(struct net_offload *prot, unsigned char num);
 
 #endif	/* _PROTOCOL_H */
net/core/dev.c +3 -5
index ae5fbd4114d2..20d9552afd38 100644
@@ -466,7 +466,7 @@ void dev_remove_pack(struct packet_type *pt)
 EXPORT_SYMBOL(dev_remove_pack);
 
 
-const struct net_offload __rcu *dev_offloads[256] __read_mostly;
+struct net_offload __rcu *dev_offloads[256] __read_mostly;
 EXPORT_SYMBOL(dev_offloads);
 
 /**
@@ -483,8 +483,7 @@ EXPORT_SYMBOL(dev_offloads);
  */
 void dev_add_offload(struct packet_offload *po)
 {
-	cmpxchg((const struct net_offload **)
-		&dev_offloads[net_offload_from_type(po->type)],
+	cmpxchg(&dev_offloads[net_offload_from_type(po->type)],
 			NULL, po);
 }
 EXPORT_SYMBOL(dev_add_offload);
@@ -504,8 +503,7 @@ EXPORT_SYMBOL(dev_add_offload);
  */
 static int __dev_remove_offload(struct packet_offload *po)
 {
-	return (cmpxchg((const struct net_offload **)
-			&dev_offloads[net_offload_from_type(po->type)],
+	return (cmpxchg(&dev_offloads[net_offload_from_type(po->type)],
 		       po, NULL) == po) ? 0 : -1;
 }
 
net/ipv4/af_inet.c +1 -1
index 28b7c7671789..f3ee6f4dfc0f 100644
@@ -1839,7 +1839,7 @@ static struct packet_offload ip_packet_offload __read_mostly = {
 	},
 };
 
-static const struct net_offload ipip_offload = {
+static struct net_offload ipip_offload = {
 	.callbacks = {
 		.gso_segment	= inet_gso_segment,
 		.gro_receive	= ipip_gro_receive,
net/ipv4/esp4_offload.c +1 -1
index 58834a10c0be..e6d7a9be9244 100644
@@ -240,7 +240,7 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb,  netdev_features_
 	return 0;
 }
 
-static const struct net_offload esp4_offload = {
+static struct net_offload esp4_offload = {
 	.callbacks = {
 		.gro_receive = esp4_gro_receive,
 		.gso_segment = esp4_gso_segment,
net/ipv4/fou.c +4 -4
index 13401cb2e7a4..52e01dcaa417 100644
@@ -229,7 +229,7 @@ static struct sk_buff *fou_gro_receive(struct sock *sk,
 				       struct sk_buff *skb)
 {
 	u8 proto = fou_from_sock(sk)->protocol;
-	const struct net_offload **offloads;
+	struct net_offload **offloads;
 	const struct net_offload *ops;
 	struct sk_buff *pp = NULL;
 
@@ -262,7 +262,7 @@ static int fou_gro_complete(struct sock *sk, struct sk_buff *skb,
 			    int nhoff)
 {
 	u8 proto = fou_from_sock(sk)->protocol;
-	const struct net_offload **offloads;
+	struct net_offload **offloads;
 	int err;
 
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
@@ -299,7 +299,7 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,
 				       struct list_head *head,
 				       struct sk_buff *skb)
 {
-	const struct net_offload **offloads;
+	struct net_offload **offloads;
 	const struct net_offload *ops;
 	struct sk_buff *pp = NULL;
 	struct sk_buff *p;
@@ -445,7 +445,7 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,
 
 static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff)
 {
-	const struct net_offload **offloads;
+	struct net_offload **offloads;
 	struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff);
 	unsigned int guehlen = 0;
 	u8 proto;
net/ipv4/gre_offload.c +1 -1
index 4f9237a4bea1..70910650d322 100644
@@ -252,7 +252,7 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff)
 	return err;
 }
 
-static const struct net_offload gre_offload = {
+static struct net_offload gre_offload = {
 	.callbacks = {
 		.gso_segment = gre_gso_segment,
 		.gro_receive = gre_gro_receive,
net/ipv4/protocol.c +5 -5
index 32a691b7ce2c..66948d77672e 100644
@@ -29,7 +29,7 @@
 #include <net/protocol.h>
 
 struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS] __read_mostly;
-const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS] __read_mostly;
+struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS] __read_mostly;
 EXPORT_SYMBOL(inet_offloads);
 
 int inet_add_protocol(const struct net_protocol *prot, unsigned char protocol)
@@ -45,9 +45,9 @@ int inet_add_protocol(const struct net_protocol *prot, unsigned char protocol)
 }
 EXPORT_SYMBOL(inet_add_protocol);
 
-int inet_add_offload(const struct net_offload *prot, unsigned char protocol)
+int inet_add_offload(struct net_offload *prot, unsigned char protocol)
 {
-	return !cmpxchg((const struct net_offload **)&inet_offloads[protocol],
+	return !cmpxchg((struct net_offload **)&inet_offloads[protocol],
 			NULL, prot) ? 0 : -1;
 }
 EXPORT_SYMBOL(inet_add_offload);
@@ -65,11 +65,11 @@ int inet_del_protocol(const struct net_protocol *prot, unsigned char protocol)
 }
 EXPORT_SYMBOL(inet_del_protocol);
 
-int inet_del_offload(const struct net_offload *prot, unsigned char protocol)
+int inet_del_offload(struct net_offload *prot, unsigned char protocol)
 {
 	int ret;
 
-	ret = (cmpxchg((const struct net_offload **)&inet_offloads[protocol],
+	ret = (cmpxchg((struct net_offload **)&inet_offloads[protocol],
 		       prot, NULL) == prot) ? 0 : -1;
 
 	synchronize_net();
net/ipv4/tcp_offload.c +1 -1
index 870b0a335061..d670f2d008bc 100644
@@ -333,7 +333,7 @@ static int tcp4_gro_complete(struct sk_buff *skb, int thoff)
 	return tcp_gro_complete(skb);
 }
 
-static const struct net_offload tcpv4_offload = {
+static struct net_offload tcpv4_offload = {
 	.callbacks = {
 		.gso_segment	=	tcp4_gso_segment,
 		.gro_receive	=	tcp4_gro_receive,
net/ipv4/udp_offload.c +3 -3
index 0c0522b79b43..4f6aa95a9b12 100644
@@ -153,8 +153,8 @@ struct sk_buff *skb_udp_tunnel_segment(struct sk_buff *skb,
 				       bool is_ipv6)
 {
 	__be16 protocol = skb->protocol;
-	const struct net_offload **offloads;
-	const struct net_offload *ops;
+	struct net_offload **offloads;
+	struct net_offload *ops;
 	struct sk_buff *segs = ERR_PTR(-EINVAL);
 	struct sk_buff *(*gso_inner_segment)(struct sk_buff *skb,
 					     netdev_features_t features);
@@ -472,7 +472,7 @@ static int udp4_gro_complete(struct sk_buff *skb, int nhoff)
 	return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb);
 }
 
-static const struct net_offload udpv4_offload = {
+static struct net_offload udpv4_offload = {
 	.callbacks = {
 		.gso_segment = udp4_ufo_fragment,
 		.gro_receive  =	udp4_gro_receive,
net/ipv6/esp6_offload.c +1 -1
index 6177e2171171..169dcd5c7135 100644
@@ -268,7 +268,7 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb,  netdev_features
 	return 0;
 }
 
-static const struct net_offload esp6_offload = {
+static struct net_offload esp6_offload = {
 	.callbacks = {
 		.gro_receive = esp6_gro_receive,
 		.gso_segment = esp6_gso_segment,
net/ipv6/ip6_offload.c +3 -3
index 4854509a2c5d..2d0ea3f453f2 100644
@@ -330,7 +330,7 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
 	},
 };
 
-static const struct net_offload sit_offload = {
+static struct net_offload sit_offload = {
 	.callbacks = {
 		.gso_segment	= ipv6_gso_segment,
 		.gro_receive    = sit_ip6ip6_gro_receive,
@@ -338,7 +338,7 @@ static const struct net_offload sit_offload = {
 	},
 };
 
-static const struct net_offload ip4ip6_offload = {
+static struct net_offload ip4ip6_offload = {
 	.callbacks = {
 		.gso_segment	= inet_gso_segment,
 		.gro_receive    = ip4ip6_gro_receive,
@@ -346,7 +346,7 @@ static const struct net_offload ip4ip6_offload = {
 	},
 };
 
-static const struct net_offload ip6ip6_offload = {
+static struct net_offload ip6ip6_offload = {
 	.callbacks = {
 		.gso_segment	= ipv6_gso_segment,
 		.gro_receive    = sit_ip6ip6_gro_receive,
net/ipv6/protocol.c +5 -5
index b5d54d4f995c..06efcfc6d02b 100644
@@ -50,21 +50,21 @@ int inet6_del_protocol(const struct inet6_protocol *prot, unsigned char protocol
 EXPORT_SYMBOL(inet6_del_protocol);
 #endif
 
-const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS] __read_mostly;
+struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS] __read_mostly;
 EXPORT_SYMBOL(inet6_offloads);
 
-int inet6_add_offload(const struct net_offload *prot, unsigned char protocol)
+int inet6_add_offload(struct net_offload *prot, unsigned char protocol)
 {
-	return !cmpxchg((const struct net_offload **)&inet6_offloads[protocol],
+	return !cmpxchg((struct net_offload **)&inet6_offloads[protocol],
 			NULL, prot) ? 0 : -1;
 }
 EXPORT_SYMBOL(inet6_add_offload);
 
-int inet6_del_offload(const struct net_offload *prot, unsigned char protocol)
+int inet6_del_offload(struct net_offload *prot, unsigned char protocol)
 {
 	int ret;
 
-	ret = (cmpxchg((const struct net_offload **)&inet6_offloads[protocol],
+	ret = (cmpxchg((struct net_offload **)&inet6_offloads[protocol],
 		       prot, NULL) == prot) ? 0 : -1;
 
 	synchronize_net();
net/ipv6/tcpv6_offload.c +1 -1
index e72947c99454..a3c5010e1361 100644
@@ -67,7 +67,7 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb,
 
 	return tcp_gso_segment(skb, features);
 }
-static const struct net_offload tcpv6_offload = {
+static struct net_offload tcpv6_offload = {
 	.callbacks = {
 		.gso_segment	=	tcp6_gso_segment,
 		.gro_receive	=	tcp6_gro_receive,
net/ipv6/udp_offload.c +1 -1
index 95dee9ca8d22..2a41da0dd33f 100644
@@ -158,7 +158,7 @@ static int udp6_gro_complete(struct sk_buff *skb, int nhoff)
 	return udp_gro_complete(skb, nhoff, udp6_lib_lookup_skb);
 }
 
-static const struct net_offload udpv6_offload = {
+static struct net_offload udpv6_offload = {
 	.callbacks = {
 		.gso_segment	=	udp6_ufo_fragment,
 		.gro_receive	=	udp6_gro_receive,
net/sctp/offload.c +1 -1
index 123e9f2dc226..ad504b83245d 100644
@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff *skb,
 	return segs;
 }
 
-static const struct net_offload sctp_offload = {
+static struct net_offload sctp_offload = {
 	.callbacks = {
 		.gso_segment = sctp_gso_segment,
 	},
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#8 lore raw

RFC PATCH 6/8 net: make gro configurable

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Add net_offload flag NET_OFF_FLAG_GRO_OFF. If set, a net_offload will not be used for gro receive processing.

Also add sysctl helper proc_do_net_offload that toggles this flag and register sysctls net.{core,ipv4,ipv6}.gro

---
 drivers/net/vxlan.c        |  8 +++++
 include/linux/netdevice.h  |  7 ++++-
 net/core/dev.c             |  1 +
 net/core/sysctl_net_core.c | 60 ++++++++++++++++++++++++++++++++++++++
 net/ipv4/sysctl_net_ipv4.c |  7 +++++
 net/ipv6/ip6_offload.c     | 10 +++++--
 net/ipv6/sysctl_net_ipv6.c |  8 +++++
 7 files changed, 97 insertions(+), 4 deletions(-)
drivers/net/vxlan.c +8 -0
index e5d236595206..8cb8e02c8ab6 100644
@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 					 struct list_head *head,
 					 struct sk_buff *skb)
 {
+	const struct net_offload *ops;
 	struct sk_buff *pp = NULL;
 	struct sk_buff *p;
 	struct vxlanhdr *vh, *vh2;
@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 			goto out;
 	}
 
+	rcu_read_lock();
+	ops = net_gro_receive(dev_offloads, ETH_P_TEB);
+	rcu_read_unlock();
+	if (!ops)
+		goto out;
+
 	skb_gro_pull(skb, sizeof(struct vxlanhdr)); /* pull vxlan header */
 
 	list_for_each_entry(p, head, list) {
@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 	}
 
 	pp = call_gro_receive(eth_gro_receive, head, skb);
+
 	flush = 0;
 
 out:
include/linux/netdevice.h +6 -1
index b9e671887fc2..93e8c9ade593 100644
@@ -2377,6 +2377,10 @@ struct net_offload {
 
 /* This should be set for any extension header which is compatible with GSO. */
 #define INET6_PROTO_GSO_EXTHDR	0x1
+#define NET_OFF_FLAG_GRO_OFF	0x2
+
+int proc_do_net_offload(struct ctl_table *ctl, int write, void __user *buffer,
+			size_t *lenp, loff_t *ppos);
 
 /* often modified stats are per-CPU, other are shared (netdev->stats) */
 struct pcpu_sw_netstats {
@@ -3583,7 +3587,8 @@ net_gro_receive(struct net_offload __rcu **offs, u16 type)
 
 	off = rcu_dereference(offs[net_offload_from_type(type)]);
 	if (off && off->callbacks.gro_receive &&
-	    (!off->type || off->type == type))
+	    (!off->type || off->type == type) &&
+	    !(off->flags & NET_OFF_FLAG_GRO_OFF))
 		return off;
 	else
 		return NULL;
net/core/dev.c +1 -0
index 20d9552afd38..0fd5273bc931 100644
@@ -154,6 +154,7 @@
 #define GRO_MAX_HEAD (MAX_HEADER + 128)
 
 static DEFINE_SPINLOCK(ptype_lock);
+DEFINE_SPINLOCK(offload_lock);
 struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
 struct list_head ptype_all __read_mostly;	/* Taps */
 static struct list_head offload_base __read_mostly;
net/core/sysctl_net_core.c +60 -0

78 lines not on this page · raw message

net/ipv4/sysctl_net_ipv4.c +7 -0
index b92f422f2fa8..7a525039afb2 100644
@@ -477,6 +477,13 @@ static struct ctl_table ipv4_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec
 	},
+	{
+		.procname	= "gro",
+		.mode		= 0644,
+		.proc_handler	= proc_do_net_offload,
+		.extra1		= inet_offloads,
+		.extra2		= (void *) NET_OFF_FLAG_GRO_OFF,
+	},
 #ifdef CONFIG_NETLABEL
 	{
 		.procname	= "cipso_cache_enable",
net/ipv6/ip6_offload.c +7 -3
index 2d0ea3f453f2..6be5adbd2ce7 100644
@@ -20,7 +20,7 @@
 
 #include "ip6_offload.h"
 
-static int ipv6_gso_pull_exthdrs(struct sk_buff *skb, int proto)
+static int ipv6_gso_pull_exthdrs(struct sk_buff *skb, int proto, bool is_gro)
 {
 	for (;;) {
 		struct ipv6_opt_hdr *opth;
@@ -30,6 +30,10 @@ static int ipv6_gso_pull_exthdrs(struct sk_buff *skb, int proto)
 					  INET6_PROTO_GSO_EXTHDR))
 			break;
 
+		if (is_gro && !net_offload_has_flag(inet6_offloads, proto,
+						    NET_OFF_FLAG_GRO_OFF))
+			break;
+
 		if (unlikely(!pskb_may_pull(skb, 8)))
 			break;
 
@@ -76,7 +80,7 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,
 	__skb_pull(skb, sizeof(*ipv6h));
 	segs = ERR_PTR(-EPROTONOSUPPORT);
 
-	proto = ipv6_gso_pull_exthdrs(skb, ipv6h->nexthdr);
+	proto = ipv6_gso_pull_exthdrs(skb, ipv6h->nexthdr, false);
 
 	if (skb->encapsulation &&
 	    skb_shinfo(skb)->gso_type & (SKB_GSO_IPXIP4 | SKB_GSO_IPXIP6))
@@ -188,7 +192,7 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,
 	if (!ops) {
 		__pskb_pull(skb, skb_gro_offset(skb));
 		skb_gro_frag0_invalidate(skb);
-		proto = ipv6_gso_pull_exthdrs(skb, proto);
+		proto = ipv6_gso_pull_exthdrs(skb, proto, true);
 		skb_gro_pull(skb, -skb_transport_offset(skb));
 		skb_reset_transport_header(skb);
 		__skb_push(skb, skb_gro_offset(skb));
net/ipv6/sysctl_net_ipv6.c +8 -0
index e15cd37024fd..83f14962a909 100644
@@ -20,6 +20,7 @@
 #ifdef CONFIG_NETLABEL
 #include <net/calipso.h>
 #endif
+#include <net/protocol.h>
 
 static int zero;
 static int one = 1;
@@ -178,6 +179,13 @@ static struct ctl_table ipv6_rotable[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= &one
 	},
+	{
+		.procname	= "gro",
+		.mode		= 0644,
+		.proc_handler	= proc_do_net_offload,
+		.extra1		= inet6_offloads,
+		.extra2		= (void *) NET_OFF_FLAG_GRO_OFF,
+	},
 #ifdef CONFIG_NETLABEL
 	{
 		.procname	= "calipso_cache_enable",
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#13 lore raw

RFC PATCH 7/8 udp: gro behind static key

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

---
 include/net/udp.h      | 2 ++
 net/ipv4/udp.c         | 2 +-
 net/ipv4/udp_offload.c | 2 +-
 net/ipv6/udp.c         | 2 +-
 net/ipv6/udp_offload.c | 2 +-
 5 files changed, 6 insertions(+), 4 deletions(-)
include/net/udp.h +2 -0
index 8482a990b0bb..9e82cb391dea 100644
@@ -443,8 +443,10 @@ int udpv4_offload_init(void);
 
 void udp_init(void);
 
+DECLARE_STATIC_KEY_FALSE(udp_encap_needed_key);
 void udp_encap_enable(void);
 #if IS_ENABLED(CONFIG_IPV6)
+DECLARE_STATIC_KEY_FALSE(udpv6_encap_needed_key);
 void udpv6_encap_enable(void);
 #endif
 
net/ipv4/udp.c +1 -1
index f4e35b2ff8b8..bd873a5b8a86 100644
@@ -1889,7 +1889,7 @@ static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 	return 0;
 }
 
-static DEFINE_STATIC_KEY_FALSE(udp_encap_needed_key);
+DEFINE_STATIC_KEY_FALSE(udp_encap_needed_key);
 void udp_encap_enable(void)
 {
 	static_branch_enable(&udp_encap_needed_key);
net/ipv4/udp_offload.c +1 -1
index 4f6aa95a9b12..f44fe328aa0f 100644
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
 	struct udphdr *uh = udp_gro_udphdr(skb);
 
-	if (unlikely(!uh))
+	if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
 		goto flush;
 
 	/* Don't bother verifying checksum if we're going to flush anyway. */
net/ipv6/udp.c +1 -1
index 83f4c77c79d8..d84672959f10 100644
@@ -548,7 +548,7 @@ static __inline__ void udpv6_err(struct sk_buff *skb,
 	__udp6_lib_err(skb, opt, type, code, offset, info, &udp_table);
 }
 
-static DEFINE_STATIC_KEY_FALSE(udpv6_encap_needed_key);
+DEFINE_STATIC_KEY_FALSE(udpv6_encap_needed_key);
 void udpv6_encap_enable(void)
 {
 	static_branch_enable(&udpv6_encap_needed_key);
net/ipv6/udp_offload.c +1 -1
index 2a41da0dd33f..e00f19c4a939 100644
@@ -119,7 +119,7 @@ static struct sk_buff *udp6_gro_receive(struct list_head *head,
 {
 	struct udphdr *uh = udp_gro_udphdr(skb);
 
-	if (unlikely(!uh))
+	if (unlikely(!uh) || !static_branch_unlikely(&udpv6_encap_needed_key))
 		goto flush;
 
 	/* Don't bother verifying checksum if we're going to flush anyway. */
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
author maintainer
#23 lore raw

RFC PATCH 8/8 udp: add gro

In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Very rough initial version of udp gro, for discussion purpose only at this point.

Among others it
- lacks the cmsg UDP_SEGMENT to return gso_size
- probably breaks udp tunnels
- hard breaks at 40 segments
- does not allow a last segment of unequal size

---
 include/uapi/linux/udp.h |  1 +
 net/ipv4/udp.c           | 71 ++++++++++++++++++++++++++++++++++++++++
 net/ipv4/udp_offload.c   | 11 +++----
 3 files changed, 76 insertions(+), 7 deletions(-)
include/uapi/linux/udp.h +1 -0
index 09d00f8c442b..7fda3e8c7fcf 100644
@@ -33,6 +33,7 @@ struct udphdr {
 #define UDP_NO_CHECK6_TX 101	/* Disable sending checksum for UDP6X */
 #define UDP_NO_CHECK6_RX 102	/* Disable accpeting checksum for UDP6 */
 #define UDP_SEGMENT	103	/* Set GSO segmentation size */
+#define UDP_GRO		104	/* Enable GRO */
 
 /* UDP encapsulation types */
 #define UDP_ENCAP_ESPINUDP_NON_IKE	1 /* draft-ietf-ipsec-nat-t-ike-00/01 */
net/ipv4/udp.c +71 -0

83 lines not on this page · raw message

net/ipv4/udp_offload.c +4 -7
index f44fe328aa0f..6dd3f0a28b5e 100644
@@ -386,6 +386,8 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,
 			NAPI_GRO_CB(p)->same_flow = 0;
 			continue;
 		}
+
+		/* TODO: for UDP_GRO: match size */
 	}
 
 	skb_gro_pull(skb, sizeof(struct udphdr)); /* pull encapsulating udp header */
@@ -437,11 +439,6 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff,
 
 	uh->len = newlen;
 
-	/* Set encapsulation before calling into inner gro_complete() functions
-	 * to make them set up the inner offsets.
-	 */
-	skb->encapsulation = 1;
-
 	rcu_read_lock();
 	sk = (*lookup)(skb, uh->source, uh->dest);
 	if (sk && udp_sk(sk)->gro_complete)
@@ -462,11 +459,11 @@ static int udp4_gro_complete(struct sk_buff *skb, int nhoff)
 	struct udphdr *uh = (struct udphdr *)(skb->data + nhoff);
 
 	if (uh->check) {
-		skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
+		skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_L4;
 		uh->check = ~udp_v4_check(skb->len - nhoff, iph->saddr,
 					  iph->daddr, 0);
 	} else {
-		skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
+		skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_L4;
 	}
 
 	return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb);
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
#9 lore raw

Re: [PATCH net-next RFC 6/8] net: make gro configurable

In reply to Willem de Bruijn
Quoted 13 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1 -0
index e5d236595206..8cb8e02c8ab6 100644
@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 					 struct list_head *head,
 					 struct sk_buff *skb)
 {
+	const struct net_offload *ops;
 	struct sk_buff *pp = NULL;
 	struct sk_buff *p;
 	struct vxlanhdr *vh, *vh2;

drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614

Quoted 8 lines
drivers/net/vxlan.c +5 -0
@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 			goto out;
 	}
 
+	rcu_read_lock();
+	ops = net_gro_receive(dev_offloads, ETH_P_TEB);
+	rcu_read_unlock();
+	if (!ops)
+		goto out;

Isn't rcu_read_lock already held here?
RCU read lock is always held in the receive handler path

drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 632

Quoted 10 lines
drivers/net/vxlan.c +2 -0
+
 	skb_gro_pull(skb, sizeof(struct vxlanhdr)); /* pull vxlan header */
 
 	list_for_each_entry(p, head, list) {
@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
 	}
 
 	pp = call_gro_receive(eth_gro_receive, head, skb);
+
 	flush = 0;

whitespace change crept into this patch.

author maintainer
#10 lore raw

Re: [PATCH net-next RFC 6/8] net: make gro configurable

Quoted 20 lines · Stephen Hemminger
Quoted 16 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1 -0
index e5d236595206..8cb8e02c8ab6 100644
@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
                                       struct list_head *head,
                                       struct sk_buff *skb)
 {
+     const struct net_offload *ops;
      struct sk_buff *pp = NULL;
      struct sk_buff *p;
      struct vxlanhdr *vh, *vh2;
@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
                      goto out;
      }
 
Quoted 8 lines

drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614

Quoted 5 lines
drivers/net/vxlan.c +5 -0
+     rcu_read_lock();
+     ops = net_gro_receive(dev_offloads, ETH_P_TEB);
+     rcu_read_unlock();
+     if (!ops)
+             goto out;

Isn't rcu_read_lock already held here?
RCU read lock is always held in the receive handler path

There is a critical section on receive, taken in netif_receive_skb_core, but gro code runs before that. All the existing gro handlers call rcu_read_lock.

Quoted 7 lines
Quoted 7 lines
drivers/net/vxlan.c +1 -0
+
      skb_gro_pull(skb, sizeof(struct vxlanhdr)); /* pull vxlan header */
 
      list_for_each_entry(p, head, list) {
@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
      }
 
Quoted 5 lines

drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 632

Quoted 3 lines
drivers/net/vxlan.c +1 -0
      pp = call_gro_receive(eth_gro_receive, head, skb);
+
      flush = 0;

whitespace change crept into this patch.

Oops, thanks.

author maintainer
#11 lore raw

Re: [PATCH net-next RFC 6/8] net: make gro configurable

Quoted 32 lines · Willem de Bruijn
Quoted 20 lines · Stephen Hemminger
Quoted 16 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1 -0
index e5d236595206..8cb8e02c8ab6 100644
@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
                                       struct list_head *head,
                                       struct sk_buff *skb)
 {
+     const struct net_offload *ops;
      struct sk_buff *pp = NULL;
      struct sk_buff *p;
      struct vxlanhdr *vh, *vh2;
@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,
                      goto out;
      }
 
Quoted 8 lines

drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614

Quoted 5 lines
drivers/net/vxlan.c +5 -0
+     rcu_read_lock();
+     ops = net_gro_receive(dev_offloads, ETH_P_TEB);
+     rcu_read_unlock();
+     if (!ops)
+             goto out;

Isn't rcu_read_lock already held here?
RCU read lock is always held in the receive handler path

Quoted 3 lines

There is a critical section on receive, taken in netif_receive_skb_core, but gro code runs before that. All the existing gro handlers call rcu_read_lock.

Though if dev_gro_receive is the entry point for all of gro, then all other handlers are ensured to be executed within its rcu readside section.

author maintainer
#12 lore raw

Re: [PATCH net-next RFC 6/8] net: make gro configurable

In reply to Willem de Bruijn
Quoted 50 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Add net_offload flag NET_OFF_FLAG_GRO_OFF. If set, a net_offload will not be used for gro receive processing.

Also add sysctl helper proc_do_net_offload that toggles this flag and register sysctls net.{core,ipv4,ipv6}.gro

---

net/core/dev.c +1 -0
index 20d9552afd38..0fd5273bc931 100644
@@ -154,6 +154,7 @@
 #define GRO_MAX_HEAD (MAX_HEADER + 128)
 
 static DEFINE_SPINLOCK(ptype_lock);
+DEFINE_SPINLOCK(offload_lock);
 struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
 struct list_head ptype_all __read_mostly;      /* Taps */
 static struct list_head offload_base __read_mostly;
net/core/sysctl_net_core.c +12 -0
index b1a2c5e38530..d2d72afdd9eb 100644
@@ -15,6 +15,7 @@
 #include <linux/vmalloc.h>
 #include <linux/init.h>
 #include <linux/slab.h>
+#include <linux/bitmap.h>
 
 #include <net/ip.h>
 #include <net/sock.h>
@@ -34,6 +35,58 @@ static int net_msg_warn;     /* Unused, but still a sysctl */
 int sysctl_fb_tunnels_only_for_init_net __read_mostly = 0;
 EXPORT_SYMBOL(sysctl_fb_tunnels_only_for_init_net);
 
+extern spinlock_t offload_lock;
+
+#define NET_OFF_TBL_LEN        256
+
+int proc_do_net_offload(struct ctl_table *ctl, int write, void __user *buffer,
+                       size_t *lenp, loff_t *ppos)
+{
+       unsigned long bitmap[NET_OFF_TBL_LEN / (sizeof(unsigned long) << 3)];
+       struct ctl_table tbl = { .maxlen = NET_OFF_TBL_LEN, .data = bitmap };
+       unsigned long flag = (unsigned long) ctl->extra2;
+       struct net_offload __rcu **offs = ctl->extra1;

net/core/sysctl_net_core.c static int net_msg_warn; /* Unused, but still a sysctl */ · line 58

Quoted 10 lines
net/core/sysctl_net_core.c +10 -0
+       struct net_offload *off;
+       int i, ret;
+
+       memset(bitmap, 0, sizeof(bitmap));
+
+       spin_lock(&offload_lock);
+
+       for (i = 0; i < tbl.maxlen; i++) {
+               off = rcu_dereference_protected(offs[i], lockdep_is_held(&offload_lock));
+               if (off && off->flags & flag) {

This does not actually work as is. No protocol will have this flag set out of the box.

I was in the middle of rewriting some of this when it became topical, so I sent it out for discussion. It's bound not to be the only bug of the patchset as is. I'll work through them to get it back in shape.

#6 lore raw

Re: [PATCH net-next RFC 5/8] net: deconstify net_offload

In reply to Willem de Bruijn
Quoted 14 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

With configurable gro, the flags field in net_offloads may be changed.

Remove the const keyword. This is a noop otherwise.

net/sctp/offload.c +0 -0
index 123e9f2dc226..ad504b83245d 100644
@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct 

sk_buff
*skb,

net/sctp/offload.c

Quoted 8 lines
net/sctp/offload.c +1 -1
 	return segs;
 }
 
-static const struct net_offload sctp_offload = {
+static struct net_offload sctp_offload = {
 	.callbacks = {
 		.gso_segment = sctp_gso_segment,
 	},

Hi Willem

sctp6 also needs to be deconstified.

net/sctp/offload.c +1 -1
index ad504b8..4be7794 100644
@@ -96,7 +96,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff *skb,
        },
 };
 
-static const struct net_offload sctp6_offload = {
+static struct net_offload sctp6_offload = {
        .callbacks = {
                .gso_segment = sctp_gso_segment,
        },
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#14 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Willem de Bruijn
Quoted 12 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

net/ipv4/udp_offload.c +0 -0
index 4f6aa95a9b12..f44fe328aa0f 100644

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct · line 411

Quoted 8 lines
net/ipv4/udp_offload.c +1 -1
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct
 list_head *head,
 {
 	struct udphdr *uh = udp_gro_udphdr(skb);
 
-	if (unlikely(!uh))
+	if (unlikely(!uh) ||
 !static_branch_unlikely(&udp_encap_needed_key))
 		goto flush;

Hi Willem

Does this need to be

net/ipv4/udp_offload.c +1 -1
index 6dd3f0a..fcd5589 100644
@@ -407,7 +407,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
        struct udphdr *uh = udp_gro_udphdr(skb);
 
-       if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
+       if (unlikely(!uh) || static_branch_unlikely(&udp_encap_needed_key))
                goto flush;
 
        /* Don't bother verifying checksum if we're going to flush 

anyway. */

I tried setting UDP_GRO socket option and I had to make this change to exercise the udp_gro_receive_cb code path.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
author maintainer
#15 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

Quoted 43 lines · Subash Abhinov Kasiviswanathan
Quoted 12 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

net/ipv4/udp_offload.c +0 -0
index 4f6aa95a9b12..f44fe328aa0f 100644

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct · line 411

Quoted 8 lines
net/ipv4/udp_offload.c +1 -1
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct
 list_head *head,
 {
      struct udphdr *uh = udp_gro_udphdr(skb);
 
-     if (unlikely(!uh))
+     if (unlikely(!uh) ||
 !static_branch_unlikely(&udp_encap_needed_key))
              goto flush;

Hi Willem

Does this need to be

net/ipv4/udp_offload.c +1 -1
index 6dd3f0a..fcd5589 100644
@@ -407,7 +407,7 @@ static struct sk_buff *udp4_gro_receive(struct
 list_head *head,
  {
         struct udphdr *uh = udp_gro_udphdr(skb);
 
-       if (unlikely(!uh) ||
 !static_branch_unlikely(&udp_encap_needed_key))
+       if (unlikely(!uh) ||
 static_branch_unlikely(&udp_encap_needed_key))
                 goto flush;

         /* Don't bother verifying checksum if we're going to flush
Quoted 4 lines

anyway. */

I tried setting UDP_GRO socket option and I had to make this change to exercise the udp_gro_receive_cb code path.

Thanks for trying it out, Subash.

The static_branch logic is correct as is. It skips the gro logic if the static key is not enabled.

But there is indeed a bug. the UDP_GRO setsockopt has to enable the static key. A full patch is more complete, but this should fix it for now:

net/ipv4/udp.c +0 -0
@@ -2506,6 +2506,7 @@ int udp_lib_setsockopt(struct sock *sk, int

level, int optname,

                        if (!udp_sk(sk)->gro_receive) {
                                udp_sk(sk)->gro_complete = udp_gro_complete_cb;
                                udp_sk(sk)->gro_receive = udp_gro_receive_cb;

+ udp_encap_enable();

sorry about that. I had tested the two patches independently, but apparently not on top of each other.

Independent of udp gro, I tested the static key by running udp_rr with perf record -a -g and looking for __udp4_lib_lookup. With the patch, all calls are from __udp4_lib_rcv else udp_gro_receive also shows up. I had to stress the porttable by opening ~32K ports.

Also, the simple patch to udpgso_bench_rx.c that I used to test gro:

(quoted diff) +0 -0

b/tools/testing/selftests/net/udpgso_bench_rx.c
index 727cf67a3f75..35ba8567fc56 100644

tools/testing/selftests/net/udpgso_bench_rx.c +14 -1
@@ -31,6 +31,11 @@
 #include <sys/wait.h>
 #include <unistd.h>
 
+#ifndef UDP_GRO
+#define UDP_GRO                104
+#endif
+
+static bool cfg_do_gro;
 static int  cfg_port           = 8000;
 static bool cfg_tcp;
 static bool cfg_verify;
@@ -89,6 +94,11 @@ static int do_socket(bool do_tcp)
        if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &val, sizeof(val)))
                error(1, errno, "setsockopt reuseport");
 
+       if (cfg_do_gro) {
+               if (setsockopt(fd, SOL_UDP, UDP_GRO, &val, sizeof(val)))
+                       error(1, errno, "setsockopt gro");
+       }
+
        addr.sin6_family =      PF_INET6;
        addr.sin6_port =        htons(cfg_port);
        addr.sin6_addr =        in6addr_any;
@@ -199,8 +209,11 @@ static void parse_opts(int argc, char **argv)
 {
        int c;
 
-       while ((c = getopt(argc, argv, "ptv")) != -1) {
+       while ((c = getopt(argc, argv, "gptv")) != -1) {
                switch (c) {
+               case 'g':
+                       cfg_do_gro = true;
+                       break;
                case 'p':
                        cfg_port = htons(strtoul(optarg, NULL, 0));
                        break;
author maintainer
#7 lore raw

Re: [PATCH net-next RFC 5/8] net: deconstify net_offload

In reply to Subash Abhinov Kasiviswanathan
Quoted 42 lines · Subash Abhinov Kasiviswanathan
Quoted 17 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

With configurable gro, the flags field in net_offloads may be changed.

Remove the const keyword. This is a noop otherwise.

net/sctp/offload.c +0 -0
index 123e9f2dc226..ad504b83245d 100644
@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct

sk_buff
*skb,

      return segs;

}

net/sctp/offload.c static struct sk_buff *sctp_gso_segment(struct sk_buff *skb, · line 102

Quoted 5 lines
net/sctp/offload.c +1 -1
-static const struct net_offload sctp_offload = {
+static struct net_offload sctp_offload = {
      .callbacks = {
              .gso_segment = sctp_gso_segment,
      },

Hi Willem

sctp6 also needs to be deconstified.

net/sctp/offload.c +1 -1
index ad504b8..4be7794 100644
@@ -96,7 +96,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff
 *skb,
         },
  };
 
-static const struct net_offload sctp6_offload = {
+static struct net_offload sctp6_offload = {
         .callbacks = {
                 .gso_segment = sctp_gso_segment,
Quoted 1 line
         },

Thanks! I'll update that.

maintainer
#16 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Willem de Bruijn
Quoted 6 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

...

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 409

Quoted 11 lines
net/ipv4/udp_offload.c +1 -1
index 4f6aa95a9b12..f44fe328aa0f 100644
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
 	struct udphdr *uh = udp_gro_udphdr(skb);
 
-	if (unlikely(!uh))
+	if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
 		goto flush;

If you use udp_encap_needed_key to enalbe UDP GRO, then a UDP encapsulation socket will enable it too. Not sure if this is intentional.

That said, enabling UDP GRO on a UDP encapsulation socket (ESP in UPD etc.) will fail badly as then encrypted ESP packets might be merged together. So we somehow should make sure that this does not happen.

Anyway, this reminds me that we can support GRO for UDP encapsulation. It just requires separate GRO callbacks for the different encapsulation types.

maintainer
#18 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Willem de Bruijn

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 411

Quoted 13 lines · Willem de Bruijn
net/ipv4/udp_offload.c +1 -1
index 4f6aa95a9b12..f44fe328aa0f 100644
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
 	struct udphdr *uh = udp_gro_udphdr(skb);
 
-	if (unlikely(!uh))
+	if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
 		goto flush;
 
 	/* Don't bother verifying checksum if we're going to flush anyway. */

If I read this correctly, once udp_encap_needed_key is enabled, it will never be turned off, because the tunnel and encap socket shut down does not cope with udp_encap_needed_key.

Perhaps we should take care of that, too.

Cheers,

Paolo

maintainer
#20 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Willem de Bruijn
Quoted 4 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

It would be nice if we could do GRO not just for GRO configured sockets, but also for flows that are going to be IPsec transformed or directly forwarded.

Maybe in case that forwarding is enabled on the receiving device, inet_gro_receive() could do a route lookup and allow GRO if the route lookup returned at forwarding route.

For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting. This could be done similar to what I proposed for the list receive case:

https://www.spinics.net/lists/netdev/msg522706.html

How GRO should be done could be even configured
by replacing the net_offload pointer similar
to what Paolo propsed in his pachset with
the inet_update_offload() function.

author maintainer
#17 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Steffen Klassert
Quoted 19 lines · Steffen Klassert
Quoted 6 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

...

Quoted 8 lines
net/ipv4/udp_offload.c +0 -0
index 4f6aa95a9b12..f44fe328aa0f 100644
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
      struct udphdr *uh = udp_gro_udphdr(skb);
 
Quoted 7 lines

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 409

Quoted 3 lines
net/ipv4/udp_offload.c +1 -1
-     if (unlikely(!uh))
+     if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
              goto flush;

If you use udp_encap_needed_key to enalbe UDP GRO, then a UDP encapsulation socket will enable it too. Not sure if this is intentional.

Yes. That is already the case to a certain point. The function was introduced with tunnels and is enabled by tunnels, but so far only compiles out the encap_rcv() branch in udp_qeueue_rcv_skb.

With patch 7/8 it also toggles the GRO path. Critically, both are enabled as soon as a tunnel is registered.

Quoted 5 lines

That said, enabling UDP GRO on a UDP encapsulation socket (ESP in UPD etc.) will fail badly as then encrypted ESP packets might be merged together. So we somehow should make sure that this does not happen.

Absolutely. This initial implementation probably breaks UDP tunnels badly. That needs to be addressed.

Quoted 4 lines

Anyway, this reminds me that we can support GRO for UDP encapsulation. It just requires separate GRO callbacks for the different encapsulation types.

author maintainer
#19 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Paolo Abeni
Quoted 14 lines · Paolo Abeni
Quoted 12 lines · Willem de Bruijn
net/ipv4/udp_offload.c +1 -1
index 4f6aa95a9b12..f44fe328aa0f 100644
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head,
 {
      struct udphdr *uh = udp_gro_udphdr(skb);
 
-     if (unlikely(!uh))
+     if (unlikely(!uh) || !static_branch_unlikely(&udp_encap_needed_key))
              goto flush;
 
Quoted 7 lines

net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 411

Quoted 1 line
net/ipv4/udp_offload.c +0 -0
      /* Don't bother verifying checksum if we're going to flush anyway. */

If I read this correctly, once udp_encap_needed_key is enabled, it will never be turned off, because the tunnel and encap socket shut down does not cope with udp_encap_needed_key.

Perhaps we should take care of that, too.

Agreed. For now I reused what's already there, but I can extend that with refcounting using static_branch_inc/static_branch_dec.

author maintainer
#21 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

In reply to Steffen Klassert
Quoted 10 lines · Steffen Klassert
Quoted 4 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.

It would be nice if we could do GRO not just for GRO configured sockets, but also for flows that are going to be IPsec transformed or directly forwarded.

I thought about that, as we have GSO. An egregious hack enables GRO for all registered local sockets that support it and for any flow for which no local socket is registered:

@@ -365,11 +369,13 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,

        rcu_read_lock();
        sk = (*lookup)(skb, uh->source, uh->dest);

- if (sk && udp_sk(sk)->gro_receive)
- goto unflush;
- goto out_unlock;
+ if (!sk)
+ gro_receive_cb = udp_gro_receive_cb;
+ else if (!udp_sk(sk)->gro_receive)
+ goto out_unlock;
+ else
+ gro_receive_cb = udp_sk(sk)->gro_receive;

@@ -392,7 +398,7 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,

skb_gro_pull(skb, sizeof(struct udphdr)); /* pull
encapsulating udp header */

        skb_gro_postpull_rcsum(skb, uh, sizeof(struct udphdr));

- pp = call_gro_receive_sk(udp_sk(sk)->gro_receive, sk, head, skb);
+ pp = call_gro_receive_sk(gro_receive_cb, sk, head, skb);

But not having a local socket does not imply forwarding path, of course.

Quoted 4 lines

Maybe in case that forwarding is enabled on the receiving device, inet_gro_receive() could do a route lookup and allow GRO if the route lookup returned at forwarding route.

That's a better solution, if the cost is acceptable. We do have to be careful against increasing per packet cycle cost in this path given that it's a possible vector for DoS attempts.

Quoted 4 lines

For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting.

With software GSO that is faster, as it would have to allocate all the separate segment skbs in skb_segment later. Though there is some complexity if MTUs differ.

With hardware UDP GSO, having a single skb will be cheaper in the forwarding path. Using napi_gro_frags, device drivers really do only end up allocating one skb for the GSO packet.

Quoted 9 lines

This could be done similar to what I proposed for the list receive case:

https://www.spinics.net/lists/netdev/msg522706.html

How GRO should be done could be even configured
by replacing the net_offload pointer similar
to what Paolo propsed in his pachset with
the inet_update_offload() function.

Right. The above hack also already has to use two distinct callback assignments.

maintainer
#22 lore raw

Re: [PATCH net-next RFC 7/8] udp: gro behind static key

Quoted 10 lines · Willem de Bruijn
Quoted 4 lines · Steffen Klassert

Maybe in case that forwarding is enabled on the receiving device, inet_gro_receive() could do a route lookup and allow GRO if the route lookup returned at forwarding route.

That's a better solution, if the cost is acceptable. We do have to be careful against increasing per packet cycle cost in this path given that it's a possible vector for DoS attempts.

I played with this already and I have not seen any significant overhead when doing a route lookup in GRO. We have to do a route lookup anyway, so it should not matter too much if we do it here or later in the IP layer.

Quoted 9 lines
Quoted 5 lines

For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting.

With software GSO that is faster, as it would have to allocate all the separate segment skbs in skb_segment later. Though there is some complexity if MTUs differ.

This can be handled the same way as it is done for TCP GSO packets. If the size of the original packets is bigger than the outgoing MTU, we either signal ICMP_FRAG_NEEDED to the sender, or split the chain back into single packets and do fragmentation on them.

Quoted 4 lines

With hardware UDP GSO, having a single skb will be cheaper in the forwarding path. Using napi_gro_frags, device drivers really do only end up allocating one skb for the GSO packet.

Right, this is why it would be nice to have this
configurable.

maintainer
#24 lore raw
In reply to Willem de Bruijn

Hi all,

Quoted 2 lines · Willem de Bruijn

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

I'm wondering how we go on from this ? I'm fine with either approaches.

Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?

Cheers,

Paolo

author maintainer
#25 lore raw
Quoted 8 lines · Paolo Abeni

Hi all,

Quoted 2 lines · Willem de Bruijn

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

I'm wondering how we go on from this ? I'm fine with either approaches.

Let me send the udp gro static_key patch. Then we don't need the enable udp on demand logic (patch 2/4).

Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate UDP_GRO socket option instead of adding this to the existing UDP_SEGMENT.

Sounds good?

Quoted 3 lines

Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?

Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).

maintainer
#26 lore raw

Hi,

Thank you for the prompt reply!

Quoted 11 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni

Hi all,

Quoted 2 lines · Willem de Bruijn

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

I'm wondering how we go on from this ? I'm fine with either approaches.

Let me send the udp gro static_key patch.

Would love that. We need to care of key decr, too (and possibly don't be fooled by encap_rcv() users).

Quoted 1 line

Then we don't need the enable udp on demand logic (patch 2/4).

ok.

Quoted 6 lines

Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate UDP_GRO socket option instead of adding this to the existing UDP_SEGMENT.

Sounds good?

Indeed!
I need also to add a cmsg to expose to the user the skb gro_size, and some test cases. Locally I'm [ab-]using the GRO functionality introduced recently on veth to test the code in a namespace pair (attaching a dummy XDP program to the RX-side veth). I'm not sure if that could fit a selftest.

Quoted 10 lines
Quoted 3 lines

Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?

Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).

Agreed.

Let's try the first step first ;)

Final but relevant note: I'll try my best to avoid delaying this, but lately I tend to be pre-empted by other tasks, it's difficult for me to assure a deadline here.

Cheers,

Paolo

author maintainer
#27 lore raw
Quoted 5 lines · Paolo Abeni

Hi,

Thank you for the prompt reply!

Not at all. Thanks for moving this forward :)

Quoted 9 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni
Quoted 7 lines

Hi all,

Quoted 2 lines · Willem de Bruijn

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I'm wondering how we go on from this ? I'm fine with either approaches.

Let me send the udp gro static_key patch.

Would love that. We need to care of key decr, too (and possibly don't be fooled by encap_rcv() users).

I just sent http://patchwork.ozlabs.org/patch/979525/

Right now all users are those that call setup_udp_tunnel_sock to register encap_rcv.

If accepted, I'll add a separate patch to decrement the key. That's probably in udp_tunnel_sock_release, but I need to take a closer look.

Quoted 9 lines
Quoted 1 line

Then we don't need the enable udp on demand logic (patch 2/4).

ok.

Quoted 5 lines

Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate UDP_GRO socket option instead of adding this to the existing UDP_SEGMENT.

Quoted 8 lines
Quoted 1 line

Sounds good?

Indeed!
I need also to add a cmsg to expose to the user the skb gro_size, and some test cases. Locally I'm [ab-]using the GRO functionality introduced recently on veth to test the code in a namespace pair (attaching a dummy XDP program to the RX-side veth). I'm not sure if that could fit a selftest.

Very nice. Yes, veth only implements napi in xdp mode.

Quoted 24 lines
Quoted 11 lines
Quoted 4 lines

Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?

Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).

Agreed.

Let's try the first step first ;)

Final but relevant note: I'll try my best to avoid delaying this, but lately I tend to be pre-empted by other tasks, it's difficult for me to assure a deadline here.

Cheers,

Paolo

maintainer
#28 lore raw
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni

Would love that. We need to care of key decr, too (and possibly don't be fooled by encap_rcv() users).

I just sent http://patchwork.ozlabs.org/patch/979525/

Right now all users are those that call setup_udp_tunnel_sock to register encap_rcv.

plus setsockopt(UDP_ENCAP)

Quoted 3 lines

If accepted, I'll add a separate patch to decrement the key. That's probably in udp_tunnel_sock_release, but I need to take a closer look.

l2tp calls setup_udp_tunnel_sock() but don't use udp_tunnel_sock_release(). Possibly it would be safer checking for:

up->encap_type || up(sk)->gro_receive

in udp_destroy_sock()

Cheers,

Paolo

author maintainer
#29 lore raw
Quoted 17 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni

Would love that. We need to care of key decr, too (and possibly don't be fooled by encap_rcv() users).

I just sent http://patchwork.ozlabs.org/patch/979525/

Right now all users are those that call setup_udp_tunnel_sock to register encap_rcv.

plus setsockopt(UDP_ENCAP)

Quoted 3 lines

If accepted, I'll add a separate patch to decrement the key. That's probably in udp_tunnel_sock_release, but I need to take a closer look.

Quoted 6 lines

l2tp calls setup_udp_tunnel_sock() but don't use udp_tunnel_sock_release(). Possibly it would be safer checking for:

up->encap_type || up(sk)->gro_receive

in udp_destroy_sock()

Ah indeed. And gtp might be another example.

Thanks!

maintainer
#30 lore raw
In reply to Willem de Bruijn
Quoted 24 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni

Hi all,

Quoted 2 lines · Willem de Bruijn

This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].

I'm wondering how we go on from this ? I'm fine with either approaches.

Let me send the udp gro static_key patch. Then we don't need the enable udp on demand logic (patch 2/4).

Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate UDP_GRO socket option instead of adding this to the existing UDP_SEGMENT.

Sounds good?

Quoted 3 lines

Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?

Quoted 6 lines

Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).

I think it depends a bit on the usecase and hardware etc. if the first or the second approach is faster. So it would be good if we can choose which one to use depending on that.

For local socket receiving, building big GSO packets is likely faster than the chaining method.

But on forwarding the chaining method might be faster because we don't have the overhead of creating GSO packets and of segmenting them back to their native form (at least as long as we don't have NICs that support hardware UDP GSO). Same applies to packets that undergo IPsec transformation.

Another thing where the chaining method could be intersting is when we receive already big LRO or HW GRO packets from the NIC. Packets of the same flow could still travel together through the stack with the chaining method. I've never tried this, though. For now it is just an idea.

I have the code for the chaining mehthod here, I'd just need some method to hook it in. Maybe it could be done with some sort of an inet_update_offload() as Paolo already propsed in his pachset, or we could make it configurable per device...