udp and configurable gro
Subsystems (8)
Willem de Bruijn
maintainercover udp and configurable gro
From: Willem de Bruijn <[email protected]>
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
Reduce UDP receive cost for bulk traffic by enabling datagram coalescing with GRO.
Before adding more GRO callbacks, make GRO configurable by the administrator to optionally reduce the attack surface of this early receive path. See also [2].
Introduce sysctls net.(core|ipv4|ipv6).gro that expose the table of protocols for which GRO is support. Allow the administrator to disable individual entries in the table.
To have a single infrastructure, convert dev_offloads to the table-based approach to existing inet(6)_offloads. Additional small benefit is that ipv6 will no longer take two list lookups to find.
Patch 1 converts dev_offloads to the infra of inet(6)_offloads Patch 2 deduplicates gro_complete logic now that all share infra Patch 3 does the same for gro_receive, in anticipation of adding
a branch to check whether gro_receive is enabled
Patch 4 harmonizes ipv6 header opts, so that those, too can be
optionally disabled.
Patch 5 makes inet(6)_offloads non-const to allow disabling a flag Patch 6 introduces the administrative sysctl
Patch 7 avoids udp gro cost if no udp gro callback is register Patch 8 introduces udp gro
[1] http://patchwork.ozlabs.org/project/netdev/list/?series=65741
[2] http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf
Willem de Bruijn (8):
gro: convert device offloads to net_offload
gro: deduplicate gro_complete
gro: add net_gro_receive
ipv6: remove offload exception for hopopts
net: deconstify net_offload
net: make gro configurable
udp: gro behind static key
udp: add gro
drivers/net/geneve.c | 11 +--- drivers/net/vxlan.c | 8 +++ include/linux/netdevice.h | 64 +++++++++++++++++++-- include/net/protocol.h | 19 ++----- include/net/udp.h | 2 + include/uapi/linux/udp.h | 1 + net/8021q/vlan.c | 12 +--- net/core/dev.c | 112 ++++++++----------------------------- net/core/sysctl_net_core.c | 60 ++++++++++++++++++++ net/ethernet/eth.c | 13 +---- net/ipv4/af_inet.c | 21 ++----- net/ipv4/esp4_offload.c | 2 +- net/ipv4/fou.c | 41 ++++---------- net/ipv4/gre_offload.c | 26 ++++----- net/ipv4/protocol.c | 10 ++-- net/ipv4/sysctl_net_ipv4.c | 7 +++ net/ipv4/tcp_offload.c | 2 +- net/ipv4/udp.c | 73 +++++++++++++++++++++++- net/ipv4/udp_offload.c | 19 +++---- net/ipv6/esp6_offload.c | 2 +- net/ipv6/exthdrs_offload.c | 17 +++++- net/ipv6/ip6_offload.c | 69 +++++++++-------------- net/ipv6/protocol.c | 10 ++-- net/ipv6/sysctl_net_ipv6.c | 8 +++ net/ipv6/tcpv6_offload.c | 2 +- net/ipv6/udp.c | 2 +- net/ipv6/udp_offload.c | 4 +- net/sctp/offload.c | 2 +- 28 files changed, 344 insertions(+), 275 deletions(-)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Willem de Bruijn
maintainerRFC PATCH 1/8 gro: convert device offloads to net_offload
From: Willem de Bruijn <[email protected]>
In preparation of making GRO receive configurable, have all offloads share the same infrastructure.
--- include/linux/netdevice.h | 17 +++++- include/net/protocol.h | 7 --- net/core/dev.c | 105 +++++++++++++------------------------- 3 files changed, 51 insertions(+), 78 deletions(-)
include/linux/netdevice.h +15 -2
@@ -2366,13 +2366,18 @@ struct offload_callbacks {int (*gro_complete)(struct sk_buff *skb,int nhoff); }; -struct packet_offload { +struct net_offload {__be16 type;/* This is really htons(ether_type). */ u16 priority;struct offload_callbacks callbacks; -struct list_head list; +unsigned int flags;/* Flags used by IPv6 for now */ }; +#define packet_offload net_offload + +/* This should be set for any extension header which is compatible with GSO. */ +#define INET6_PROTO_GSO_EXTHDR 0x1 +/* often modified stats are per-CPU, other are shared (netdev->stats) */ struct pcpu_sw_netstats {u64 rx_packets;
@@ -3554,6 +3559,14 @@ gro_result_t napi_gro_frags(struct napi_struct *napi);struct packet_offload *gro_find_receive_by_type (__be16 type);struct packet_offload *gro_find_complete_by_type (__be16 type); +static inline u8 net_offload_from_type (u16 type) +{ +/* Do not bother handling collisions. There are none. +* If they do occur with new offloads, add a mapping function here. +*/ +return type &0xFF ; +} +static inline void napi_free_frags (struct napi_struct *napi) {kfree_skb (napi->skb);
include/net/protocol.h +0 -7
@@ -69,13 +69,6 @@ struct inet6_protocol {#define INET6_PROTO_FINAL 0x2#endif -struct net_offload { -struct offload_callbacks callbacks; -unsigned int flags;/* Flags used by IPv6 for now */ -}; -/* This should be set for any extension header which is compatible with GSO. */ -#define INET6_PROTO_GSO_EXTHDR 0x1 -/* This is used to register socket interfaces for IP protocols. */ struct inet_protosw {struct list_head list;
net/core/dev.c +36 -69
191 lines not on this page · raw message
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Willem de Bruijn
maintainerRFC PATCH 2/8 gro: deduplicate gro_complete
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
The gro completion datapath is open coded for all protocols. Deduplicate with new helper function net_gro_complete.
--- drivers/net/geneve.c | 9 +-------- include/linux/netdevice.h | 19 ++++++++++++++++++- net/8021q/vlan.c | 10 +--------- net/core/dev.c | 24 +----------------------- net/ethernet/eth.c | 11 +---------- net/ipv4/af_inet.c | 15 ++------------- net/ipv4/fou.c | 25 +++---------------------- net/ipv4/gre_offload.c | 12 +++--------- net/ipv6/ip6_offload.c | 13 +------------ 9 files changed, 31 insertions(+), 107 deletions(-)
drivers/net/geneve.c +1 -8
@@ -488,7 +488,6 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) {struct genevehdr *gh; -struct packet_offload *ptype;__be16 type;int gh_len;int err = -ENOSYS ;
@@ -497,13 +496,7 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb, gh_len =geneve_hlen (gh); type = gh->proto_type; -rcu_read_lock (); - ptype =gro_find_complete_by_type (type); -if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff + gh_len); - -rcu_read_unlock (); - + err =net_gro_complete (dev_offloads, type, skb, nhoff + gh_len);skb_set_inner_mac_header (skb, nhoff + gh_len);return err;
include/linux/netdevice.h +18 -1
@@ -3557,7 +3557,8 @@ void napi_gro_flush(struct napi_struct *napi, bool flush_old);struct sk_buff *napi_get_frags (struct napi_struct *napi);gro_result_t napi_gro_frags (struct napi_struct *napi);struct packet_offload *gro_find_receive_by_type (__be16 type); -struct packet_offload *gro_find_complete_by_type (__be16 type); + +extern const struct net_offload __rcu *dev_offloads[256 ];static inline u8 net_offload_from_type (u16 type) {
@@ -3567,6 +3568,22 @@ static inline u8 net_offload_from_type(u16 type)return type &0xFF ; } +static inline int net_gro_complete (const struct net_offload __rcu **offs, +u16 type,struct sk_buff *skb,int nhoff) +{ +const struct net_offload *off; +int ret = -ENOENT ; + +rcu_read_lock (); + off =rcu_dereference (offs[net_offload_from_type (type)]); +if (off && off->callbacks.gro_complete && + (!off->type || off->type == type)) + ret = off->callbacks.gro_complete (skb, nhoff); +rcu_read_unlock (); + +return ret; +} +static inline void napi_free_frags (struct napi_struct *napi) {kfree_skb (napi->skb);
net/8021q/vlan.c +1 -9
@@ -703,16 +703,8 @@ static int vlan_gro_complete(struct sk_buff *skb, int nhoff) {struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);__be16 type = vhdr->h_vlan_encapsulated_proto; -struct packet_offload *ptype; -int err = -ENOENT ; -rcu_read_lock (); - ptype =gro_find_complete_by_type (type); -if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff +sizeof (*vhdr)); - -rcu_read_unlock (); -return err; +return net_gro_complete (dev_offloads, type, skb, nhoff +sizeof (*vhdr)); }static struct packet_offload vlan_packet_offloads[]__read_mostly = {
net/core/dev.c +1 -23
@@ -5235,10 +5235,6 @@ static void flush_all_backlogs(void)static int napi_gro_complete (struct sk_buff *skb) { -const struct packet_offload *ptype; -__be16 type = skb->protocol; -int err = -ENOENT ; -BUILD_BUG_ON (sizeof (struct napi_gro_cb ) >sizeof (skb->cb));if (NAPI_GRO_CB (skb)->count ==1 ) {
@@ -5246,13 +5242,7 @@ static int napi_gro_complete(struct sk_buff *skb) goto out; } -rcu_read_lock (); - ptype = dev_offloads[net_offload_from_type (type)]; -if (ptype && ptype->callbacks.gro_complete) - err = ptype->callbacks.gro_complete (skb,0 ); -rcu_read_unlock (); - -if (err) { +if (net_gro_complete (dev_offloads, skb->protocol, skb,0 )) {kfree_skb (skb);return NET_RX_SUCCESS ; }
@@ -5505,18 +5495,6 @@ struct packet_offload *gro_find_receive_by_type(__be16 type) }EXPORT_SYMBOL (gro_find_receive_by_type); -struct packet_offload *gro_find_complete_by_type (__be16 type) -{ -struct net_offload *off; - - off = (struct net_offload *)rcu_dereference (dev_offloads[type &0xFF ]); -if (off && off->type == type && off->callbacks.gro_complete) -return off; -else -return NULL ; -} -EXPORT_SYMBOL (gro_find_complete_by_type); -static void napi_skb_free_stolen_head (struct sk_buff *skb) {skb_dst_drop (skb);
net/ethernet/eth.c +1 -10
@@ -485,20 +485,11 @@ int eth_gro_complete(struct sk_buff *skb, int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff);__be16 type = eh->h_proto; -struct packet_offload *ptype; -int err = -ENOSYS ;if (skb->encapsulation)skb_set_inner_mac_header (skb, nhoff); -rcu_read_lock (); - ptype =gro_find_complete_by_type (type); -if (ptype !=NULL ) - err = ptype->callbacks.gro_complete (skb, nhoff + -sizeof (struct ethhdr )); - -rcu_read_unlock (); -return err; +return net_gro_complete (dev_offloads, type, skb, nhoff +sizeof (*eh)); }EXPORT_SYMBOL (eth_gro_complete);
net/ipv4/af_inet.c +2 -13
@@ -1560,9 +1560,7 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff) {__be16 newlen =htons (skb->len - nhoff);struct iphdr *iph = (struct iphdr *)(skb->data + nhoff); -const struct net_offload *ops;int proto = iph->protocol; -int err = -ENOSYS ;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IP ));
@@ -1572,21 +1570,12 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)csum_replace2 (&iph->check, iph->tot_len, newlen); iph->tot_len = newlen; -rcu_read_lock (); - ops =rcu_dereference (inet_offloads[proto]); -if (WARN_ON (!ops || !ops->callbacks.gro_complete)) - goto out_unlock; -/* Only need to add sizeof(*iph) to get to the next hdr below * because any hdr with option will have been flushed in * inet_gro_receive(). */ - err = ops->callbacks.gro_complete (skb, nhoff +sizeof (*iph)); - -out_unlock: -rcu_read_unlock (); - -return err; +return net_gro_complete (inet_offloads, proto, skb, + nhoff +sizeof (*iph)); }EXPORT_SYMBOL (inet_gro_complete);
net/ipv4/fou.c +3 -22
@@ -261,24 +261,14 @@ static struct sk_buff *fou_gro_receive(struct sock *sk,static int fou_gro_complete (struct sock *sk,struct sk_buff *skb,int nhoff) { -const struct net_offload *ops;u8 proto =fou_from_sock (sk)->protocol; -int err = -ENOSYS ;const struct net_offload **offloads; +int err; -rcu_read_lock (); offloads =NAPI_GRO_CB (skb)->is_ipv6 ? inet6_offloads : inet_offloads; - ops =rcu_dereference (offloads[proto]); -if (WARN_ON (!ops || !ops->callbacks.gro_complete)) - goto out_unlock; - - err = ops->callbacks.gro_complete (skb, nhoff); - + err =net_gro_complete (offloads, proto, skb, nhoff);skb_set_inner_mac_header (skb, nhoff); -out_unlock: -rcu_read_unlock (); -return err; }
@@ -457,7 +447,6 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) {const struct net_offload **offloads;struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff); -const struct net_offload *ops;unsigned int guehlen =0 ;u8 proto;int err = -ENOENT ;
@@ -483,18 +472,10 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff)return err; } -rcu_read_lock (); offloads =NAPI_GRO_CB (skb)->is_ipv6 ? inet6_offloads : inet_offloads; - ops =rcu_dereference (offloads[proto]); -if (WARN_ON (!ops || !ops->callbacks.gro_complete)) - goto out_unlock; - - err = ops->callbacks.gro_complete (skb, nhoff + guehlen); - + err =net_gro_complete (offloads, proto, skb, nhoff + guehlen);skb_set_inner_mac_header (skb, nhoff + guehlen); -out_unlock: -rcu_read_unlock ();return err; }
net/ipv4/gre_offload.c +3 -9
@@ -231,10 +231,9 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,static int gre_gro_complete (struct sk_buff *skb,int nhoff) {struct gre_base_hdr *greh = (struct gre_base_hdr *)(skb->data + nhoff); -struct packet_offload *ptype;unsigned int grehlen =sizeof (*greh); -int err = -ENOENT ;__be16 type; +int err; skb->encapsulation =1 ;skb_shinfo (skb)->gso_type =SKB_GSO_GRE ;
@@ -246,13 +245,8 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff)if (greh->flags &GRE_CSUM ) grehlen +=GRE_HEADER_SECTION ; -rcu_read_lock (); - ptype =gro_find_complete_by_type (type); -if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff + grehlen); - -rcu_read_unlock (); - + err =net_gro_complete (dev_offloads, type, skb, + nhoff +sizeof (*greh));skb_set_inner_mac_header (skb, nhoff + grehlen);return err;
net/ipv6/ip6_offload.c +1 -12
@@ -298,7 +298,6 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff) {const struct net_offload *ops;struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff); -int err = -ENOSYS ;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IPV6 ));
@@ -307,18 +306,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (skb->len - nhoff -sizeof (*iph)); -rcu_read_lock (); - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); -if (WARN_ON (!ops || !ops->callbacks.gro_complete)) - goto out_unlock; - - err = ops->callbacks.gro_complete (skb, nhoff); - -out_unlock: -rcu_read_unlock (); - -return err; +return net_gro_complete (inet6_offloads, ops->type, skb, nhoff); }static int sit_gro_complete (struct sk_buff *skb,int nhoff)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Willem de Bruijn
maintainerRFC PATCH 3/8 gro: add net_gro_receive
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
For configurable gro_receive all callsites need to be updated. Similar to gro_complete, introduce a single shared helper, net_gro_receive.
--- drivers/net/geneve.c | 2 +- include/linux/netdevice.h | 14 +++++++++++++- net/8021q/vlan.c | 2 +- net/core/dev.c | 20 ++++---------------- net/ethernet/eth.c | 2 +- net/ipv4/af_inet.c | 4 ++-- net/ipv4/fou.c | 8 ++++---- net/ipv4/gre_offload.c | 12 ++++++------ net/ipv6/ip6_offload.c | 8 ++++---- 9 files changed, 36 insertions(+), 36 deletions(-)
drivers/net/geneve.c +1 -1
@@ -467,7 +467,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk, type = gh->proto_type;rcu_read_lock (); - ptype =gro_find_receive_by_type (type); + ptype =net_gro_receive (dev_offloads, type);if (!ptype) goto out_unlock;
include/linux/netdevice.h +13 -1
@@ -3556,7 +3556,6 @@ gro_result_t napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb);void napi_gro_flush (struct napi_struct *napi,bool flush_old);struct sk_buff *napi_get_frags (struct napi_struct *napi);gro_result_t napi_gro_frags (struct napi_struct *napi); -struct packet_offload *gro_find_receive_by_type (__be16 type);extern const struct net_offload __rcu *dev_offloads[256 ];
@@ -3568,6 +3567,19 @@ static inline u8 net_offload_from_type(u16 type)return type &0xFF ; } +static inline const struct net_offload * +net_gro_receive (const struct net_offload __rcu **offs,u16 type) +{ +const struct net_offload *off; + + off =rcu_dereference (offs[net_offload_from_type (type)]); +if (off && off->callbacks.gro_receive && + (!off->type || off->type == type)) +return off; +else +return NULL ; +} +static inline int net_gro_complete (const struct net_offload __rcu **offs,u16 type,struct sk_buff *skb,int nhoff) {
net/8021q/vlan.c +1 -1
@@ -670,7 +670,7 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head, type = vhdr->h_vlan_encapsulated_proto;rcu_read_lock (); - ptype =gro_find_receive_by_type (type); + ptype =net_gro_receive (dev_offloads, type);if (!ptype) goto out_unlock;
net/core/dev.c +4 -16
@@ -5382,7 +5382,7 @@ static void gro_flush_oldest(struct list_head *head)static enum gro_result dev_gro_receive (struct napi_struct *napi,struct sk_buff *skb) {u32 hash =skb_get_hash_raw (skb) & (GRO_HASH_BUCKETS -1 ); -const struct packet_offload *ptype; +const struct net_offload *ops;__be16 type = skb->protocol;struct list_head *gro_head;struct sk_buff *pp =NULL ;
@@ -5396,8 +5396,8 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff gro_head =gro_list_prepare (napi, skb);rcu_read_lock (); - ptype = dev_offloads[net_offload_from_type (type)]; -if (ptype && ptype->callbacks.gro_receive) { + ops =net_gro_receive (dev_offloads, type); +if (ops) {skb_set_network_header (skb,skb_gro_offset (skb));skb_reset_mac_len (skb);NAPI_GRO_CB (skb)->same_flow =0 ;
@@ -5425,7 +5425,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buffNAPI_GRO_CB (skb)->csum_valid =0 ; } - pp = ptype->callbacks.gro_receive (gro_head, skb); + pp = ops->callbacks.gro_receive (gro_head, skb);rcu_read_unlock (); }else {rcu_read_unlock ();
@@ -5483,18 +5483,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff goto pull; } -struct packet_offload *gro_find_receive_by_type (__be16 type) -{ -struct net_offload *off; - - off = (struct net_offload *)rcu_dereference (dev_offloads[type &0xFF ]); -if (off && off->type == type && off->callbacks.gro_receive) -return off; -else -return NULL ; -} -EXPORT_SYMBOL (gro_find_receive_by_type); -static void napi_skb_free_stolen_head (struct sk_buff *skb) {skb_dst_drop (skb);
net/ethernet/eth.c +1 -1
@@ -462,7 +462,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) type = eh->h_proto;rcu_read_lock (); - ptype =gro_find_receive_by_type (type); + ptype =net_gro_receive (dev_offloads, type);if (ptype ==NULL ) { flush =1 ; goto out_unlock;
net/ipv4/af_inet.c +2 -2
@@ -1409,8 +1409,8 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb) proto = iph->protocol;rcu_read_lock (); - ops =rcu_dereference (inet_offloads[proto]); -if (!ops || !ops->callbacks.gro_receive) + ops =net_gro_receive (inet_offloads, proto); +if (!ops) goto out_unlock;if (*(u8 *)iph !=0x45 )
net/ipv4/fou.c +4 -4
@@ -246,8 +246,8 @@ static struct sk_buff *fou_gro_receive(struct sock *sk,rcu_read_lock (); offloads =NAPI_GRO_CB (skb)->is_ipv6 ? inet6_offloads : inet_offloads; - ops =rcu_dereference (offloads[proto]); -if (!ops || !ops->callbacks.gro_receive) + ops =net_gro_receive (offloads, proto); +if (!ops) goto out_unlock; pp =call_gro_receive (ops->callbacks.gro_receive, head, skb);
@@ -428,8 +428,8 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,rcu_read_lock (); offloads =NAPI_GRO_CB (skb)->is_ipv6 ? inet6_offloads : inet_offloads; - ops =rcu_dereference (offloads[proto]); -if (WARN_ON_ONCE (!ops || !ops->callbacks.gro_receive)) + ops =net_gro_receive (offloads, proto); +if (WARN_ON_ONCE (!ops)) goto out_unlock; pp =call_gro_receive (ops->callbacks.gro_receive, head, skb);
net/ipv4/gre_offload.c +6 -6
@@ -111,13 +111,13 @@ static struct sk_buff *gre_gso_segment(struct sk_buff *skb,static struct sk_buff *gre_gro_receive (struct list_head *head,struct sk_buff *skb) { -struct sk_buff *pp =NULL ; -struct sk_buff *p;const struct gre_base_hdr *greh; +const struct net_offload *ops;unsigned int hlen, grehlen; +struct sk_buff *pp =NULL ; +struct sk_buff *p;unsigned int off;int flush =1 ; -struct packet_offload *ptype;__be16 type;if (NAPI_GRO_CB (skb)->encap_mark)
@@ -154,8 +154,8 @@ static struct sk_buff *gre_gro_receive(struct list_head *head, type = greh->protocol;rcu_read_lock (); - ptype =gro_find_receive_by_type (type); -if (!ptype) + ops =net_gro_receive (dev_offloads, type); +if (!ops) goto out_unlock; grehlen =GRE_HEADER_SECTION ;
@@ -217,7 +217,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,/* Adjusted NAPI_GRO_CB(skb)->csum after skb_gro_pull()*/ skb_gro_postpull_rcsum (skb, greh, grehlen); - pp =call_gro_receive (ptype->callbacks.gro_receive, head, skb); + pp =call_gro_receive (ops->callbacks.gro_receive, head, skb); flush =0 ; out_unlock:
net/ipv6/ip6_offload.c +4 -4
@@ -194,8 +194,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,rcu_read_lock (); proto = iph->nexthdr; - ops =rcu_dereference (inet6_offloads[proto]); -if (!ops || !ops->callbacks.gro_receive) { + ops =net_gro_receive (inet6_offloads, proto); +if (!ops) {__pskb_pull (skb,skb_gro_offset (skb));skb_gro_frag0_invalidate (skb); proto =ipv6_gso_pull_exthdrs (skb, proto);
@@ -203,8 +203,8 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,skb_reset_transport_header (skb);__skb_push (skb,skb_gro_offset (skb)); - ops =rcu_dereference (inet6_offloads[proto]); -if (!ops || !ops->callbacks.gro_receive) + ops =net_gro_receive (inet6_offloads, proto); +if (!ops) goto out_unlock; iph =ipv6_hdr (skb);
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Willem de Bruijn
maintainerRFC PATCH 4/8 ipv6: remove offload exception for hopopts
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Extension headers in ipv6 are pulled without calling a callback function. An inet6_offload signals this feature with flag INET6_PROTO_GSO_EXTHDR.
Add net_has_flag helper to hide implementation details and in prepartion for configurable gro.
Convert NEXTHDR_HOP from a special case branch to a standard extension header offload.
--- include/linux/netdevice.h | 9 +++++++++ net/ipv6/exthdrs_offload.c | 17 ++++++++++++++--- net/ipv6/ip6_offload.c | 36 +++++++++++++----------------------- 3 files changed, 36 insertions(+), 26 deletions(-)
include/linux/netdevice.h +9 -0
@@ -3567,6 +3567,15 @@ static inline u8 net_offload_from_type(u16 type)return type &0xFF ; } +static inline bool net_offload_has_flag (const struct net_offload __rcu **offs, +u16 type,u16 flag) +{ +const struct net_offload *off; + + off = offs ?rcu_dereference (offs[net_offload_from_type (type)]) :NULL ; +return off && off->flags & flag; +} +static inline const struct net_offload *net_gro_receive (const struct net_offload __rcu **offs,u16 type) {
net/ipv6/exthdrs_offload.c +14 -3
@@ -12,11 +12,15 @@#include <net/protocol.h> #include "ip6_offload.h" -static const struct net_offload rthdr_offload = { +static struct net_offload hophdr_offload = { .flags =INET6_PROTO_GSO_EXTHDR , }; -static const struct net_offload dstopt_offload = { +static struct net_offload rthdr_offload = { + .flags =INET6_PROTO_GSO_EXTHDR , +}; + +static struct net_offload dstopt_offload = { .flags =INET6_PROTO_GSO_EXTHDR , };
@@ -24,10 +28,14 @@ int __init ipv6_exthdrs_offload_init(void) {int ret; - ret =inet6_add_offload (&rthdr_offload,IPPROTO_ROUTING ); + ret =inet6_add_offload (&hophdr_offload,IPPROTO_HOPOPTS );if (ret) goto out; + ret =inet6_add_offload (&rthdr_offload,IPPROTO_ROUTING ); +if (ret) + goto out_hop; + ret =inet6_add_offload (&dstopt_offload,IPPROTO_DSTOPTS );if (ret) goto out_rt;
@@ -37,5 +45,8 @@ int __init ipv6_exthdrs_offload_init(void) out_rt:inet6_del_offload (&rthdr_offload,IPPROTO_ROUTING ); + +out_hop: +inet6_del_offload (&rthdr_offload,IPPROTO_HOPOPTS ); goto out; }
net/ipv6/ip6_offload.c +13 -23
@@ -22,21 +22,13 @@static int ipv6_gso_pull_exthdrs (struct sk_buff *skb,int proto) { -const struct net_offload *ops =NULL ; -for (;;) {struct ipv6_opt_hdr *opth;int len; -if (proto !=NEXTHDR_HOP ) { - ops =rcu_dereference (inet6_offloads[proto]); - -if (unlikely (!ops)) -break ; - -if (!(ops->flags &INET6_PROTO_GSO_EXTHDR )) -break ; - } +if (!net_offload_has_flag (inet6_offloads, proto, +INET6_PROTO_GSO_EXTHDR )) +break ;if (unlikely (!pskb_may_pull (skb,8 )))break ;
@@ -141,26 +133,24 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,/* Return the total length of all the extension hdrs, following the same * logic in ipv6_gso_pull_exthdrs() when parsing ext-hdrs. */ -static int ipv6_exthdrs_len (struct ipv6hdr *iph, -const struct net_offload **opps) +static int ipv6_exthdrs_len (struct ipv6hdr *iph,u8 *pproto) {struct ipv6_opt_hdr *opth = (void *)iph;int len =0 , proto, optlen =sizeof (*iph); proto = iph->nexthdr;for (;;) { -if (proto !=NEXTHDR_HOP ) { - *opps =rcu_dereference (inet6_offloads[proto]); -if (unlikely (!(*opps))) -break ; -if (!((*opps)->flags &INET6_PROTO_GSO_EXTHDR )) -break ; - } +if (!net_offload_has_flag (inet6_offloads, proto, +INET6_PROTO_GSO_EXTHDR )) +break ; + opth = (void *)opth + optlen; optlen =ipv6_optlen (opth); len += optlen; proto = opth->nexthdr; } + + *pproto = proto;return len; }
@@ -296,8 +286,8 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head,static int ipv6_gro_complete (struct sk_buff *skb,int nhoff) { -const struct net_offload *ops;struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff); +u8 proto;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IPV6 ));
@@ -306,8 +296,8 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (skb->len - nhoff -sizeof (*iph)); - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); -return net_gro_complete (inet6_offloads, ops->type, skb, nhoff); + nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &proto); +return net_gro_complete (inet6_offloads, proto, skb, nhoff); }static int sit_gro_complete (struct sk_buff *skb,int nhoff)
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Willem de Bruijn
maintainerRFC PATCH 5/8 net: deconstify net_offload
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
With configurable gro, the flags field in net_offloads may be changed.
Remove the const keyword. This is a noop otherwise.
--- include/linux/netdevice.h | 14 +++++++------- include/net/protocol.h | 12 ++++++------ net/core/dev.c | 8 +++----- net/ipv4/af_inet.c | 2 +- net/ipv4/esp4_offload.c | 2 +- net/ipv4/fou.c | 8 ++++---- net/ipv4/gre_offload.c | 2 +- net/ipv4/protocol.c | 10 +++++----- net/ipv4/tcp_offload.c | 2 +- net/ipv4/udp_offload.c | 6 +++--- net/ipv6/esp6_offload.c | 2 +- net/ipv6/ip6_offload.c | 6 +++--- net/ipv6/protocol.c | 10 +++++----- net/ipv6/tcpv6_offload.c | 2 +- net/ipv6/udp_offload.c | 2 +- net/sctp/offload.c | 2 +- 16 files changed, 44 insertions(+), 46 deletions(-)
include/linux/netdevice.h +7 -7
@@ -3557,7 +3557,7 @@ void napi_gro_flush(struct napi_struct *napi, bool flush_old);struct sk_buff *napi_get_frags (struct napi_struct *napi);gro_result_t napi_gro_frags (struct napi_struct *napi); -extern const struct net_offload __rcu *dev_offloads[256 ]; +extern struct net_offload __rcu *dev_offloads[256 ];static inline u8 net_offload_from_type (u16 type) {
@@ -3567,19 +3567,19 @@ static inline u8 net_offload_from_type(u16 type)return type &0xFF ; } -static inline bool net_offload_has_flag (const struct net_offload __rcu **offs, +static inline bool net_offload_has_flag (struct net_offload __rcu **offs,u16 type,u16 flag) { -const struct net_offload *off; +struct net_offload *off; off = offs ?rcu_dereference (offs[net_offload_from_type (type)]) :NULL ;return off && off->flags & flag; }static inline const struct net_offload * -net_gro_receive (const struct net_offload __rcu **offs,u16 type) +net_gro_receive (struct net_offload __rcu **offs,u16 type) { -const struct net_offload *off; +struct net_offload *off; off =rcu_dereference (offs[net_offload_from_type (type)]); if (off && off->callbacks.gro_receive &&
@@ -3589,10 +3589,10 @@ net_gro_receive(const struct net_offload __rcu **offs, u16 type)return NULL ; } -static inline int net_gro_complete (const struct net_offload __rcu **offs, +static inline int net_gro_complete (struct net_offload __rcu **offs,u16 type,struct sk_buff *skb,int nhoff) { -const struct net_offload *off; +struct net_offload *off;int ret = -ENOENT ;rcu_read_lock ();
include/net/protocol.h +6 -6
@@ -87,8 +87,8 @@ struct inet_protosw {#define INET_PROTOSW_ICSK 0x04/* Is this an inet_connection_sock? */ extern struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS ]; -extern const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]; -extern const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]; +extern struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]; +extern struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ];#if IS_ENABLED (CONFIG_IPV6 )extern struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS ];
@@ -96,8 +96,8 @@ extern struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS];int inet_add_protocol (const struct net_protocol *prot,unsigned char num);int inet_del_protocol (const struct net_protocol *prot,unsigned char num); -int inet_add_offload (const struct net_offload *prot,unsigned char num); -int inet_del_offload (const struct net_offload *prot,unsigned char num); +int inet_add_offload (struct net_offload *prot,unsigned char num); +int inet_del_offload (struct net_offload *prot,unsigned char num);void inet_register_protosw (struct inet_protosw *p);void inet_unregister_protosw (struct inet_protosw *p);
@@ -107,7 +107,7 @@ int inet6_del_protocol(const struct inet6_protocol *prot, unsigned char num);int inet6_register_protosw (struct inet_protosw *p);void inet6_unregister_protosw (struct inet_protosw *p);#endif -int inet6_add_offload (const struct net_offload *prot,unsigned char num); -int inet6_del_offload (const struct net_offload *prot,unsigned char num); +int inet6_add_offload (struct net_offload *prot,unsigned char num); +int inet6_del_offload (struct net_offload *prot,unsigned char num);#endif /* _PROTOCOL_H */
net/core/dev.c +3 -5
@@ -466,7 +466,7 @@ void dev_remove_pack(struct packet_type *pt)EXPORT_SYMBOL (dev_remove_pack); -const struct net_offload __rcu *dev_offloads[256 ]__read_mostly ; +struct net_offload __rcu *dev_offloads[256 ]__read_mostly ;EXPORT_SYMBOL (dev_offloads);/**
@@ -483,8 +483,7 @@ EXPORT_SYMBOL(dev_offloads);*/ void dev_add_offload (struct packet_offload *po) { -cmpxchg ((const struct net_offload **) - &dev_offloads[net_offload_from_type (po->type)], +cmpxchg (&dev_offloads[net_offload_from_type (po->type)],NULL , po); }EXPORT_SYMBOL (dev_add_offload);
@@ -504,8 +503,7 @@ EXPORT_SYMBOL(dev_add_offload);*/ static int __dev_remove_offload (struct packet_offload *po) { -return (cmpxchg ((const struct net_offload **) - &dev_offloads[net_offload_from_type (po->type)], +return (cmpxchg (&dev_offloads[net_offload_from_type (po->type)], po,NULL ) == po) ?0 :-1 ; }
net/ipv4/af_inet.c +1 -1
@@ -1839,7 +1839,7 @@ static struct packet_offload ip_packet_offload __read_mostly = { }, }; -static const struct net_offload ipip_offload = { +static struct net_offload ipip_offload = { .callbacks = { .gso_segment = inet_gso_segment, .gro_receive = ipip_gro_receive,
net/ipv4/esp4_offload.c +1 -1
@@ -240,7 +240,7 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_return 0 ; } -static const struct net_offload esp4_offload = { +static struct net_offload esp4_offload = { .callbacks = { .gro_receive = esp4_gro_receive, .gso_segment = esp4_gso_segment,
net/ipv4/fou.c +4 -4
@@ -229,7 +229,7 @@ static struct sk_buff *fou_gro_receive(struct sock *sk, struct sk_buff *skb) {u8 proto =fou_from_sock (sk)->protocol; -const struct net_offload **offloads; +struct net_offload **offloads;const struct net_offload *ops;struct sk_buff *pp =NULL ;
@@ -262,7 +262,7 @@ static int fou_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) {u8 proto =fou_from_sock (sk)->protocol; -const struct net_offload **offloads; +struct net_offload **offloads;int err; offloads =NAPI_GRO_CB (skb)->is_ipv6 ? inet6_offloads : inet_offloads;
@@ -299,7 +299,7 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,struct list_head *head, struct sk_buff *skb) { - const struct net_offload **offloads; + struct net_offload **offloads;const struct net_offload *ops;struct sk_buff *pp =NULL ;struct sk_buff *p;
@@ -445,7 +445,7 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,static int gue_gro_complete (struct sock *sk,struct sk_buff *skb,int nhoff) { -const struct net_offload **offloads; +struct net_offload **offloads;struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff);unsigned int guehlen =0 ;u8 proto;
net/ipv4/gre_offload.c +1 -1
@@ -252,7 +252,7 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff)return err; } -static const struct net_offload gre_offload = { +static struct net_offload gre_offload = { .callbacks = { .gso_segment = gre_gso_segment, .gro_receive = gre_gro_receive,
net/ipv4/protocol.c +5 -5
@@ -29,7 +29,7 @@#include <net/protocol.h> struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS ]__read_mostly ; -const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]__read_mostly ; +struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]__read_mostly ;EXPORT_SYMBOL (inet_offloads);int inet_add_protocol (const struct net_protocol *prot,unsigned char protocol)
@@ -45,9 +45,9 @@ int inet_add_protocol(const struct net_protocol *prot, unsigned char protocol) }EXPORT_SYMBOL (inet_add_protocol); -int inet_add_offload (const struct net_offload *prot,unsigned char protocol) +int inet_add_offload (struct net_offload *prot,unsigned char protocol) { -return !cmpxchg ((const struct net_offload **)&inet_offloads[protocol], +return !cmpxchg ((struct net_offload **)&inet_offloads[protocol],NULL , prot) ?0 :-1 ; }EXPORT_SYMBOL (inet_add_offload);
@@ -65,11 +65,11 @@ int inet_del_protocol(const struct net_protocol *prot, unsigned char protocol) }EXPORT_SYMBOL (inet_del_protocol); -int inet_del_offload (const struct net_offload *prot,unsigned char protocol) +int inet_del_offload (struct net_offload *prot,unsigned char protocol) {int ret; - ret = (cmpxchg ((const struct net_offload **)&inet_offloads[protocol], + ret = (cmpxchg ((struct net_offload **)&inet_offloads[protocol], prot,NULL ) == prot) ?0 :-1 ;synchronize_net ();
net/ipv4/tcp_offload.c +1 -1
@@ -333,7 +333,7 @@ static int tcp4_gro_complete(struct sk_buff *skb, int thoff)return tcp_gro_complete (skb); } -static const struct net_offload tcpv4_offload = { +static struct net_offload tcpv4_offload = { .callbacks = { .gso_segment = tcp4_gso_segment, .gro_receive = tcp4_gro_receive,
net/ipv4/udp_offload.c +3 -3
@@ -153,8 +153,8 @@ struct sk_buff *skb_udp_tunnel_segment(struct sk_buff *skb, bool is_ipv6) {__be16 protocol = skb->protocol; -const struct net_offload **offloads; -const struct net_offload *ops; +struct net_offload **offloads; +struct net_offload *ops;struct sk_buff *segs =ERR_PTR (-EINVAL );struct sk_buff *(*gso_inner_segment)(struct sk_buff *skb,netdev_features_t features);
@@ -472,7 +472,7 @@ static int udp4_gro_complete(struct sk_buff *skb, int nhoff)return udp_gro_complete (skb, nhoff, udp4_lib_lookup_skb); } -static const struct net_offload udpv4_offload = { +static struct net_offload udpv4_offload = { .callbacks = { .gso_segment = udp4_ufo_fragment, .gro_receive = udp4_gro_receive,
net/ipv6/esp6_offload.c +1 -1
@@ -268,7 +268,7 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_featuresreturn 0 ; } -static const struct net_offload esp6_offload = { +static struct net_offload esp6_offload = { .callbacks = { .gro_receive = esp6_gro_receive, .gso_segment = esp6_gso_segment,
net/ipv6/ip6_offload.c +3 -3
@@ -330,7 +330,7 @@ static struct packet_offload ipv6_packet_offload __read_mostly = { }, }; -static const struct net_offload sit_offload = { +static struct net_offload sit_offload = { .callbacks = { .gso_segment = ipv6_gso_segment, .gro_receive = sit_ip6ip6_gro_receive,
@@ -338,7 +338,7 @@ static const struct net_offload sit_offload = { }, }; -static const struct net_offload ip4ip6_offload = { +static struct net_offload ip4ip6_offload = { .callbacks = { .gso_segment = inet_gso_segment, .gro_receive = ip4ip6_gro_receive,
@@ -346,7 +346,7 @@ static const struct net_offload ip4ip6_offload = { }, }; -static const struct net_offload ip6ip6_offload = { +static struct net_offload ip6ip6_offload = { .callbacks = { .gso_segment = ipv6_gso_segment, .gro_receive = sit_ip6ip6_gro_receive,
net/ipv6/protocol.c +5 -5
@@ -50,21 +50,21 @@ int inet6_del_protocol(const struct inet6_protocol *prot, unsigned char protocolEXPORT_SYMBOL (inet6_del_protocol);#endif -const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]__read_mostly ; +struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]__read_mostly ;EXPORT_SYMBOL (inet6_offloads); -int inet6_add_offload (const struct net_offload *prot,unsigned char protocol) +int inet6_add_offload (struct net_offload *prot,unsigned char protocol) { -return !cmpxchg ((const struct net_offload **)&inet6_offloads[protocol], +return !cmpxchg ((struct net_offload **)&inet6_offloads[protocol],NULL , prot) ?0 :-1 ; }EXPORT_SYMBOL (inet6_add_offload); -int inet6_del_offload (const struct net_offload *prot,unsigned char protocol) +int inet6_del_offload (struct net_offload *prot,unsigned char protocol) {int ret; - ret = (cmpxchg ((const struct net_offload **)&inet6_offloads[protocol], + ret = (cmpxchg ((struct net_offload **)&inet6_offloads[protocol], prot,NULL ) == prot) ?0 :-1 ;synchronize_net ();
net/ipv6/tcpv6_offload.c +1 -1
@@ -67,7 +67,7 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb,return tcp_gso_segment (skb, features); } -static const struct net_offload tcpv6_offload = { +static struct net_offload tcpv6_offload = { .callbacks = { .gso_segment = tcp6_gso_segment, .gro_receive = tcp6_gro_receive,
net/ipv6/udp_offload.c +1 -1
@@ -158,7 +158,7 @@ static int udp6_gro_complete(struct sk_buff *skb, int nhoff)return udp_gro_complete (skb, nhoff, udp6_lib_lookup_skb); } -static const struct net_offload udpv6_offload = { +static struct net_offload udpv6_offload = { .callbacks = { .gso_segment = udp6_ufo_fragment, .gro_receive = udp6_gro_receive,
net/sctp/offload.c +1 -1
@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff *skb,return segs; } -static const struct net_offload sctp_offload = { +static struct net_offload sctp_offload = { .callbacks = { .gso_segment = sctp_gso_segment, },
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH net-next RFC 5/8] net: deconstify net_offload
Quoted 14 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
With configurable gro, the flags field in
net_offloadsmay be changed.Remove the const keyword. This is a noop otherwise.
Signed-off-by Willem de Bruijn <[email protected]>net/sctp/offload.c +0
-0@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct
sk_buff
*skb,
net/sctp/offload.c
Quoted 8 lines
net/sctp/offload.c +1
-1return segs; } -static const struct net_offload sctp_offload = { +static struct net_offload sctp_offload = { .callbacks = { .gso_segment = sctp_gso_segment, },
Hi Willem
sctp6 also needs to be deconstified.
net/sctp/offload.c +1 -1
@@ -96,7 +96,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff *skb, }, }; -static const struct net_offload sctp6_offload = { +static struct net_offload sctp6_offload = { .callbacks = { .gso_segment = sctp_gso_segment, },
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 5/8] net: deconstify net_offload
Quoted 42 lines · Subash Abhinov Kasiviswanathan
Quoted 17 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
With configurable gro, the flags field in
net_offloadsmay be changed.Remove the const keyword. This is a noop otherwise.
Signed-off-by Willem de Bruijn <[email protected]>net/sctp/offload.c +0
-0@@ -90,7 +90,7 @@ static struct sk_buff *sctp_gso_segment(struct
sk_buff
*skb,return segs;}
net/sctp/offload.c static struct sk_buff *sctp_gso_segment(struct sk_buff *skb, · line 102
Quoted 5 lines
net/sctp/offload.c +1
-1-static const struct net_offload sctp_offload = { +static struct net_offload sctp_offload = { .callbacks = { .gso_segment = sctp_gso_segment, },Hi Willem
sctp6 also needs to be deconstified.
net/sctp/offload.c +1
-1@@ -96,7 +96,7 @@ static struct sk_buff *sctp_gso_segment(struct sk_buff *skb, }, }; -static const struct net_offload sctp6_offload = { +static struct net_offload sctp6_offload = { .callbacks = { .gso_segment = sctp_gso_segment,
Quoted 1 line
},
Thanks! I'll update that.
Willem de Bruijn
maintainerRFC PATCH 6/8 net: make gro configurable
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Add net_offload flag NET_OFF_FLAG_GRO_OFF. If set, a net_offload will not be used for gro receive processing.
Also add sysctl helper proc_do_net_offload that toggles this flag and register sysctls net.{core,ipv4,ipv6}.gro
--- drivers/net/vxlan.c | 8 +++++ include/linux/netdevice.h | 7 ++++- net/core/dev.c | 1 + net/core/sysctl_net_core.c | 60 ++++++++++++++++++++++++++++++++++++++ net/ipv4/sysctl_net_ipv4.c | 7 +++++ net/ipv6/ip6_offload.c | 10 +++++-- net/ipv6/sysctl_net_ipv6.c | 8 +++++ 7 files changed, 97 insertions(+), 4 deletions(-)
drivers/net/vxlan.c +8 -0
@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,struct list_head *head, struct sk_buff *skb) { + const struct net_offload *ops;struct sk_buff *pp =NULL ;struct sk_buff *p;struct vxlanhdr *vh, *vh2;
@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, goto out; } +rcu_read_lock (); + ops =net_gro_receive (dev_offloads,ETH_P_TEB ); +rcu_read_unlock (); +if (!ops) + goto out; +skb_gro_pull (skb,sizeof (struct vxlanhdr ));/* pull vxlan header */ list_for_each_entry (p, head, list) {
@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, } pp =call_gro_receive (eth_gro_receive, head, skb); + flush =0 ; out:
include/linux/netdevice.h +6 -1
@@ -2377,6 +2377,10 @@ struct net_offload {/* This should be set for any extension header which is compatible with GSO. */ #define INET6_PROTO_GSO_EXTHDR 0x1 +#define NET_OFF_FLAG_GRO_OFF 0x2 + +int proc_do_net_offload (struct ctl_table *ctl,int write,void __user *buffer, +size_t *lenp,loff_t *ppos);/* often modified stats are per-CPU, other are shared (netdev->stats) */ struct pcpu_sw_netstats {
@@ -3583,7 +3587,8 @@ net_gro_receive(struct net_offload __rcu **offs, u16 type) off =rcu_dereference (offs[net_offload_from_type (type)]);if (off && off->callbacks.gro_receive && - (!off->type || off->type == type)) + (!off->type || off->type == type) && + !(off->flags &NET_OFF_FLAG_GRO_OFF ))return off;else return NULL ;
net/core/dev.c +1 -0
@@ -154,6 +154,7 @@#define GRO_MAX_HEAD (MAX_HEADER + 128)static DEFINE_SPINLOCK (ptype_lock); +DEFINE_SPINLOCK (offload_lock);struct list_head ptype_base[PTYPE_HASH_SIZE ]__read_mostly ;struct list_head ptype_all__read_mostly ;/* Taps */ static struct list_head offload_base__read_mostly ;
net/core/sysctl_net_core.c +60 -0
78 lines not on this page · raw message
net/ipv4/sysctl_net_ipv4.c +7 -0
@@ -477,6 +477,13 @@ static struct ctl_table ipv4_table[] = { .mode = 0644, .proc_handler = proc_dointvec }, + { + .procname = "gro", + .mode = 0644, + .proc_handler = proc_do_net_offload, + .extra1 = inet_offloads, + .extra2 = (void *) NET_OFF_FLAG_GRO_OFF, + },#ifdef CONFIG_NETLABEL { .procname = "cipso_cache_enable",
net/ipv6/ip6_offload.c +7 -3
@@ -20,7 +20,7 @@#include "ip6_offload.h" -static int ipv6_gso_pull_exthdrs (struct sk_buff *skb,int proto) +static int ipv6_gso_pull_exthdrs (struct sk_buff *skb,int proto,bool is_gro) {for (;;) {struct ipv6_opt_hdr *opth;
@@ -30,6 +30,10 @@ static int ipv6_gso_pull_exthdrs(struct sk_buff *skb, int proto) INET6_PROTO_GSO_EXTHDR))break ; +if (is_gro && !net_offload_has_flag (inet6_offloads, proto, +NET_OFF_FLAG_GRO_OFF )) +break ; +if (unlikely (!pskb_may_pull (skb,8 )))break ;
@@ -76,7 +80,7 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,__skb_pull (skb,sizeof (*ipv6h)); segs =ERR_PTR (-EPROTONOSUPPORT ); - proto =ipv6_gso_pull_exthdrs (skb, ipv6h->nexthdr); + proto =ipv6_gso_pull_exthdrs (skb, ipv6h->nexthdr, false);if (skb->encapsulation &&skb_shinfo (skb)->gso_type & (SKB_GSO_IPXIP4 |SKB_GSO_IPXIP6 ))
@@ -188,7 +192,7 @@ static struct sk_buff *ipv6_gro_receive(struct list_head *head,if (!ops) {__pskb_pull (skb,skb_gro_offset (skb));skb_gro_frag0_invalidate (skb); - proto =ipv6_gso_pull_exthdrs (skb, proto); + proto =ipv6_gso_pull_exthdrs (skb, proto, true);skb_gro_pull (skb, -skb_transport_offset (skb));skb_reset_transport_header (skb);__skb_push (skb,skb_gro_offset (skb));
net/ipv6/sysctl_net_ipv6.c +8 -0
@@ -20,6 +20,7 @@#ifdef CONFIG_NETLABEL #include <net/calipso.h> #endif +#include <net/protocol.h> static int zero;static int one =1 ;
@@ -178,6 +179,13 @@ static struct ctl_table ipv6_rotable[] = { .proc_handler = proc_dointvec_minmax, .extra1 = &one }, + { + .procname = "gro", + .mode = 0644, + .proc_handler = proc_do_net_offload, + .extra1 = inet6_offloads, + .extra2 = (void *) NET_OFF_FLAG_GRO_OFF, + },#ifdef CONFIG_NETLABEL { .procname = "calipso_cache_enable",
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH net-next RFC 6/8] net: make gro configurable
Quoted 13 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1
-0@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,struct list_head *head, struct sk_buff *skb) { + const struct net_offload *ops;struct sk_buff *pp =NULL ;struct sk_buff *p;struct vxlanhdr *vh, *vh2;
drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614
Quoted 8 lines
drivers/net/vxlan.c +5
-0@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, goto out; } +rcu_read_lock (); + ops =net_gro_receive (dev_offloads,ETH_P_TEB ); +rcu_read_unlock (); +if (!ops) + goto out;
Isn't rcu_read_lock already held here?
RCU read lock is always held in the receive handler path
drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 632
Quoted 10 lines
drivers/net/vxlan.c +2
-0+skb_gro_pull (skb,sizeof (struct vxlanhdr ));/* pull vxlan header */ list_for_each_entry (p, head, list) {@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, } pp =call_gro_receive (eth_gro_receive, head, skb); + flush =0 ;
whitespace change crept into this patch.
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 6/8] net: make gro configurable
Quoted 20 lines · Stephen Hemminger
Quoted 16 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1
-0@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,struct list_head *head, struct sk_buff *skb) { + const struct net_offload *ops;struct sk_buff *pp =NULL ;struct sk_buff *p;struct vxlanhdr *vh, *vh2;@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, goto out; }
Quoted 8 lines
drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614
Quoted 5 lines
drivers/net/vxlan.c +5
-0+rcu_read_lock (); + ops =net_gro_receive (dev_offloads,ETH_P_TEB ); +rcu_read_unlock (); +if (!ops) + goto out;Isn't
rcu_read_lockalready held here?
RCU read lock is always held in the receive handler path
There is a critical section on receive, taken in netif_receive_skb_core, but gro code runs before that. All the existing gro handlers call rcu_read_lock.
Quoted 7 lines
Quoted 7 lines
drivers/net/vxlan.c +1
-0+skb_gro_pull (skb,sizeof (struct vxlanhdr ));/* pull vxlan header */ list_for_each_entry (p, head, list) {@@ -621,6 +628,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, }
Quoted 5 lines
drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 632
Quoted 3 lines
drivers/net/vxlan.c +1
-0pp =call_gro_receive (eth_gro_receive, head, skb); + flush =0 ;whitespace change crept into this patch.
Oops, thanks.
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 6/8] net: make gro configurable
Quoted 32 lines · Willem de Bruijn
Quoted 20 lines · Stephen Hemminger
Quoted 16 lines · On Fri, 14 Sep 2018 13:59:39 -0400 Willem de Bruijn…
drivers/net/vxlan.c +1
-0@@ -572,6 +572,7 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk,struct list_head *head, struct sk_buff *skb) { + const struct net_offload *ops;struct sk_buff *pp =NULL ;struct sk_buff *p;struct vxlanhdr *vh, *vh2;@@ -606,6 +607,12 @@ static struct sk_buff *vxlan_gro_receive(struct sock *sk, goto out; }Quoted 8 lines
drivers/net/vxlan.c static struct sk_buff *vxlan_gro_receive(struct sock *sk, · line 614
Quoted 5 lines
drivers/net/vxlan.c +5
-0+rcu_read_lock (); + ops =net_gro_receive (dev_offloads,ETH_P_TEB ); +rcu_read_unlock (); +if (!ops) + goto out;Isn't
rcu_read_lockalready held here?
RCU read lock is always held in the receive handler path
Quoted 3 lines
There is a critical section on receive, taken in
netif_receive_skb_core, but gro code runs before that. All the existing gro handlers callrcu_read_lock.
Though if dev_gro_receive is the entry point for all of gro, then all other handlers are ensured to be executed within its rcu readside section.
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 6/8] net: make gro configurable
In reply to Willem de BruijnQuoted 50 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Add
net_offloadflagNET_OFF_FLAG_GRO_OFF. If set, anet_offloadwill not be used for gro receive processing.Also add sysctl helper
proc_do_net_offloadthat toggles this flag and register sysctls net.{core,ipv4,ipv6}.groSigned-off-by Willem de Bruijn <[email protected]>---
net/core/dev.c +1
-0@@ -154,6 +154,7 @@#define GRO_MAX_HEAD (MAX_HEADER + 128)static DEFINE_SPINLOCK (ptype_lock); +DEFINE_SPINLOCK (offload_lock);struct list_head ptype_base[PTYPE_HASH_SIZE ]__read_mostly ;struct list_head ptype_all__read_mostly ;/* Taps */ static struct list_head offload_base__read_mostly ;net/core/sysctl_net_core.c +12
-0@@ -15,6 +15,7 @@#include <linux/vmalloc.h> #include <linux/init.h> #include <linux/slab.h> +#include <linux/bitmap.h> #include <net/ip.h> #include <net/sock.h> @@ -34,6 +35,58 @@ static int net_msg_warn; /* Unused, but still a sysctl */int sysctl_fb_tunnels_only_for_init_net__read_mostly =0 ;EXPORT_SYMBOL (sysctl_fb_tunnels_only_for_init_net); +extern spinlock_t offload_lock; + +#define NET_OFF_TBL_LEN 256 + +int proc_do_net_offload (struct ctl_table *ctl,int write,void __user *buffer, +size_t *lenp,loff_t *ppos) +{ +unsigned long bitmap[NET_OFF_TBL_LEN / (sizeof (unsigned long ) <<3 )]; +struct ctl_table tbl = { .maxlen =NET_OFF_TBL_LEN , .data = bitmap }; +unsigned long flag = (unsigned long ) ctl->extra2; +struct net_offload __rcu **offs = ctl->extra1;
net/core/sysctl_net_core.c static int net_msg_warn; /* Unused, but still a sysctl */ · line 58
Quoted 10 lines
net/core/sysctl_net_core.c +10
-0+struct net_offload *off; +int i, ret; + +memset (bitmap,0 ,sizeof (bitmap)); + +spin_lock (&offload_lock); + +for (i =0 ; i < tbl.maxlen; i++) { + off =rcu_dereference_protected (offs[i],lockdep_is_held (&offload_lock)); +if (off && off->flags & flag) {
This does not actually work as is. No protocol will have this flag set out of the box.
I was in the middle of rewriting some of this when it became topical, so I sent it out for discussion. It's bound not to be the only bug of the patchset as is. I'll work through them to get it back in shape.
Willem de Bruijn
maintainerRFC PATCH 7/8 udp: gro behind static key
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in udp_gro_receive if no socket has a gro callback configured.
--- include/net/udp.h | 2 ++ net/ipv4/udp.c | 2 +- net/ipv4/udp_offload.c | 2 +- net/ipv6/udp.c | 2 +- net/ipv6/udp_offload.c | 2 +- 5 files changed, 6 insertions(+), 4 deletions(-)
include/net/udp.h +2 -0
@@ -443,8 +443,10 @@ int udpv4_offload_init(void);void udp_init (void ); +DECLARE_STATIC_KEY_FALSE (udp_encap_needed_key);void udp_encap_enable (void );#if IS_ENABLED (CONFIG_IPV6 ) +DECLARE_STATIC_KEY_FALSE (udpv6_encap_needed_key);void udpv6_encap_enable (void );#endif
net/ipv4/udp.c +1 -1
@@ -1889,7 +1889,7 @@ static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)return 0 ; } -static DEFINE_STATIC_KEY_FALSE (udp_encap_needed_key); +DEFINE_STATIC_KEY_FALSE (udp_encap_needed_key);void udp_encap_enable (void ) {static_branch_enable (&udp_encap_needed_key);
net/ipv4/udp_offload.c +1 -1
@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;/* Don't bother verifying checksum if we're going to flush anyway. */
net/ipv6/udp.c +1 -1
@@ -548,7 +548,7 @@ static __inline__ void udpv6_err(struct sk_buff *skb,__udp6_lib_err (skb, opt, type, code, offset, info, &udp_table); } -static DEFINE_STATIC_KEY_FALSE (udpv6_encap_needed_key); +DEFINE_STATIC_KEY_FALSE (udpv6_encap_needed_key);void udpv6_encap_enable (void ) {static_branch_enable (&udpv6_encap_needed_key);
net/ipv6/udp_offload.c +1 -1
@@ -119,7 +119,7 @@ static struct sk_buff *udp6_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udpv6_encap_needed_key)) goto flush;/* Don't bother verifying checksum if we're going to flush anyway. */
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 12 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.Signed-off-by Willem de Bruijn <[email protected]>net/ipv4/udp_offload.c +0
-0
net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct · line 411
Quoted 8 lines
net/ipv4/udp_offload.c +1
-1@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;
Hi Willem
Does this need to be
net/ipv4/udp_offload.c +1 -1
@@ -407,7 +407,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) +if (unlikely (!uh) ||static_branch_unlikely (&udp_encap_needed_key)) goto flush;/* Don't bother verifying checksum if we're going to flush
anyway. */
I tried setting UDP_GRO socket option and I had to make this change to exercise the udp_gro_receive_cb code path.
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 43 lines · Subash Abhinov Kasiviswanathan
Quoted 12 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.Signed-off-by Willem de Bruijn <[email protected]>net/ipv4/udp_offload.c +0
-0net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct · line 411
Quoted 8 lines
net/ipv4/udp_offload.c +1
-1@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;Hi Willem
Does this need to be
net/ipv4/udp_offload.c +1
-1@@ -407,7 +407,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh) || !static_branch_unlikely(&udp_encap_needed_key)) + if (unlikely(!uh) ||static_branch_unlikely (&udp_encap_needed_key))goto flush; /* Don't bother verifying checksum if we're going to flush
Quoted 4 lines
anyway. */
I tried setting
UDP_GROsocket option and I had to make this change to exercise theudp_gro_receive_cbcode path.
Thanks for trying it out, Subash.
The static_branch logic is correct as is. It skips the gro logic if the static key is not enabled.
But there is indeed a bug. the UDP_GRO setsockopt has to enable the static key. A full patch is more complete, but this should fix it for now:
net/ipv4/udp.c +0 -0
@@ -2506,6 +2506,7 @@ int udp_lib_setsockopt(struct sock *sk, int
level, int optname,
if (!udp_sk(sk)->gro_receive) {
udp_sk(sk)->gro_complete = udp_gro_complete_cb;
udp_sk(sk)->gro_receive = udp_gro_receive_cb;+ udp_encap_enable();
sorry about that. I had tested the two patches independently, but apparently not on top of each other.
Independent of udp gro, I tested the static key by running udp_rr with perf record -a -g and looking for __udp4_lib_lookup. With the patch, all calls are from __udp4_lib_rcv else udp_gro_receive also shows up. I had to stress the porttable by opening ~32K ports.
Also, the simple patch to udpgso_bench_rx.c that I used to test gro:
(quoted diff) +0 -0
b/tools/testing/selftests/net/udpgso_bench_rx.c
index 727cf67a3f75..35ba8567fc56 100644
tools/testing/selftests/net/udpgso_bench_rx.c +14 -1
@@ -31,6 +31,11 @@#include <sys/wait.h> #include <unistd.h> +#ifndef UDP_GRO +#define UDP_GRO 104 +#endif + +static bool cfg_do_gro;static int cfg_port =8000 ;static bool cfg_tcp;static bool cfg_verify;
@@ -89,6 +94,11 @@ static int do_socket(bool do_tcp)if (setsockopt (fd,SOL_SOCKET ,SO_REUSEPORT , &val,sizeof (val)))error (1 , errno,"setsockopt reuseport" ); +if (cfg_do_gro) { +if (setsockopt (fd,SOL_UDP ,UDP_GRO , &val,sizeof (val))) +error (1 , errno,"setsockopt gro" ); + } + addr.sin6_family =PF_INET6 ; addr.sin6_port =htons (cfg_port); addr.sin6_addr = in6addr_any;
@@ -199,8 +209,11 @@ static void parse_opts(int argc, char **argv) {int c; -while ((c =getopt (argc, argv,"ptv" )) !=-1 ) { +while ((c =getopt (argc, argv,"gptv" )) !=-1 ) {switch (c) { +case 'g' : + cfg_do_gro = true; +break ;case 'p' : cfg_port =htons (strtoul (optarg,NULL ,0 ));break ;
Steffen Klassert
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
In reply to Willem de BruijnQuoted 6 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.Signed-off-by Willem de Bruijn <[email protected]>
...
net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 409
Quoted 11 lines
net/ipv4/udp_offload.c +1
-1@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;
If you use udp_encap_needed_key to enalbe UDP GRO, then a UDP encapsulation socket will enable it too. Not sure if this is intentional.
That said, enabling UDP GRO on a UDP encapsulation socket (ESP in UPD etc.) will fail badly as then encrypted ESP packets might be merged together. So we somehow should make sure that this does not happen.
Anyway, this reminds me that we can support GRO for UDP encapsulation. It just requires separate GRO callbacks for the different encapsulation types.
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 19 lines · Steffen Klassert
Quoted 6 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.Signed-off-by Willem de Bruijn <[email protected]>...
Quoted 8 lines
net/ipv4/udp_offload.c +0
-0@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb);
Quoted 7 lines
net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 409
Quoted 3 lines
net/ipv4/udp_offload.c +1
-1-if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;If you use
udp_encap_needed_keyto enalbe UDP GRO, then a UDP encapsulation socket will enable it too. Not sure if this is intentional.
Yes. That is already the case to a certain point. The function was introduced with tunnels and is enabled by tunnels, but so far only compiles out the encap_rcv() branch in udp_qeueue_rcv_skb.
With patch 7/8 it also toggles the GRO path. Critically, both are enabled as soon as a tunnel is registered.
Quoted 5 lines
That said, enabling UDP GRO on a UDP encapsulation socket (ESP in UPD etc.) will fail badly as then encrypted ESP packets might be merged together. So we somehow should make sure that this does not happen.
Absolutely. This initial implementation probably breaks UDP tunnels badly. That needs to be addressed.
Quoted 4 lines
Anyway, this reminds me that we can support GRO for UDP encapsulation. It just requires separate GRO callbacks for the different encapsulation types.
Paolo Abeni
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
In reply to Willem de Bruijnnet/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 411
Quoted 13 lines · Willem de Bruijn
net/ipv4/udp_offload.c +1
-1@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;/* Don't bother verifying checksum if we're going to flush anyway. */
If I read this correctly, once udp_encap_needed_key is enabled, it will never be turned off, because the tunnel and encap socket shut down does not cope with udp_encap_needed_key.
Perhaps we should take care of that, too.
Cheers,
Paolo
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 14 lines · Paolo Abeni
Quoted 12 lines · Willem de Bruijn
net/ipv4/udp_offload.c +1
-1@@ -405,7 +405,7 @@ static struct sk_buff *udp4_gro_receive(struct list_head *head, {struct udphdr *uh =udp_gro_udphdr (skb); -if (unlikely (!uh)) +if (unlikely (!uh) || !static_branch_unlikely (&udp_encap_needed_key)) goto flush;
Quoted 7 lines
net/ipv4/udp_offload.c static struct sk_buff *udp4_gro_receive(struct list_head *head, · line 411
Quoted 1 line
net/ipv4/udp_offload.c +0
-0/* Don't bother verifying checksum if we're going to flush anyway. */ If I read this correctly, once
udp_encap_needed_keyis enabled, it will never be turned off, because the tunnel and encap socket shut down does not cope withudp_encap_needed_key.Perhaps we should take care of that, too.
Agreed. For now I reused what's already there, but I can extend that with refcounting using static_branch_inc/static_branch_dec.
Steffen Klassert
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
In reply to Willem de BruijnQuoted 4 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.
It would be nice if we could do GRO not just for GRO configured sockets, but also for flows that are going to be IPsec transformed or directly forwarded.
Maybe in case that forwarding is enabled on the receiving device, inet_gro_receive() could do a route lookup and allow GRO if the route lookup returned at forwarding route.
For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting. This could be done similar to what I proposed for the list receive case:
https://www.spinics.net/lists/netdev/msg522706.html
How GRO should be done could be even configured
by replacing the net_offload pointer similar
to what Paolo propsed in his pachset with
the inet_update_offload() function.
Willem de Bruijn
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 10 lines · Steffen Klassert
Quoted 4 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Avoid the socket lookup cost in
udp_gro_receiveif no socket has a gro callback configured.It would be nice if we could do GRO not just for GRO configured sockets, but also for flows that are going to be IPsec transformed or directly forwarded.
I thought about that, as we have GSO. An egregious hack enables GRO for all registered local sockets that support it and for any flow for which no local socket is registered:
@@ -365,11 +369,13 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,
rcu_read_lock();
sk = (*lookup)(skb, uh->source, uh->dest);- if (sk && udp_sk(sk)->gro_receive)
- goto unflush;
- goto out_unlock;
+ if (!sk)
+ gro_receive_cb = udp_gro_receive_cb;
+ else if (!udp_sk(sk)->gro_receive)
+ goto out_unlock;
+ else
+ gro_receive_cb = udp_sk(sk)->gro_receive;
@@ -392,7 +398,7 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,
skb_gro_pull(skb, sizeof(struct udphdr)); /* pull
encapsulating udp header */
skb_gro_postpull_rcsum(skb, uh, sizeof(struct udphdr));
- pp = call_gro_receive_sk(udp_sk(sk)->gro_receive, sk, head, skb);
+ pp = call_gro_receive_sk(gro_receive_cb, sk, head, skb);
But not having a local socket does not imply forwarding path, of course.
Quoted 4 lines
Maybe in case that forwarding is enabled on the receiving device,
inet_gro_receive()could do a route lookup and allow GRO if the route lookup returned at forwarding route.
That's a better solution, if the cost is acceptable. We do have to be careful against increasing per packet cycle cost in this path given that it's a possible vector for DoS attempts.
Quoted 4 lines
For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting.
With software GSO that is faster, as it would have to allocate all the separate segment skbs in skb_segment later. Though there is some complexity if MTUs differ.
With hardware UDP GSO, having a single skb will be cheaper in the forwarding path. Using napi_gro_frags, device drivers really do only end up allocating one skb for the GSO packet.
Quoted 9 lines
This could be done similar to what I proposed for the list receive case:
https://www.spinics.net/lists/netdev/msg522706.html
How GRO should be done could be even configured
by replacing thenet_offloadpointer similar
to what Paolo propsed in his pachset with
theinet_update_offload()function.
Right. The above hack also already has to use two distinct callback assignments.
Steffen Klassert
maintainerRe: [PATCH net-next RFC 7/8] udp: gro behind static key
Quoted 10 lines · Willem de Bruijn
Quoted 4 lines · Steffen Klassert
Maybe in case that forwarding is enabled on the receiving device,
inet_gro_receive()could do a route lookup and allow GRO if the route lookup returned at forwarding route.That's a better solution, if the cost is acceptable. We do have to be careful against increasing per packet cycle cost in this path given that it's a possible vector for DoS attempts.
I played with this already and I have not seen any significant overhead when doing a route lookup in GRO. We have to do a route lookup anyway, so it should not matter too much if we do it here or later in the IP layer.
Quoted 9 lines
Quoted 5 lines
For flows that are likely software segmented after that, it would be worth to build packet chains insted of merging the payload. Packets of the same flow could travel together, but it would save the cost of the packet merging and segmenting.
With software GSO that is faster, as it would have to allocate all the separate segment skbs in
skb_segmentlater. Though there is some complexity if MTUs differ.
This can be handled the same way as it is done for TCP GSO packets. If the size of the original packets is bigger than the outgoing MTU, we either signal ICMP_FRAG_NEEDED to the sender, or split the chain back into single packets and do fragmentation on them.
Quoted 4 lines
With hardware UDP GSO, having a single skb will be cheaper in the forwarding path. Using
napi_gro_frags, device drivers really do only end up allocating one skb for the GSO packet.
Right, this is why it would be nice to have this
configurable.
Willem de Bruijn
maintainerRFC PATCH 8/8 udp: add gro
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Very rough initial version of udp gro, for discussion purpose only at this point.
Among others it
- lacks the cmsg UDP_SEGMENT to return gso_size
- probably breaks udp tunnels
- hard breaks at 40 segments
- does not allow a last segment of unequal size
--- include/uapi/linux/udp.h | 1 + net/ipv4/udp.c | 71 ++++++++++++++++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 11 +++---- 3 files changed, 76 insertions(+), 7 deletions(-)
include/uapi/linux/udp.h +1 -0
@@ -33,6 +33,7 @@ struct udphdr {#define UDP_NO_CHECK6_TX 101/* Disable sending checksum for UDP6X */ #define UDP_NO_CHECK6_RX 102/* Disable accpeting checksum for UDP6 */ #define UDP_SEGMENT 103/* Set GSO segmentation size */ +#define UDP_GRO 104/* Enable GRO */ /* UDP encapsulation types */ #define UDP_ENCAP_ESPINUDP_NON_IKE 1/* draft-ietf-ipsec-nat-t-ike-00/01 */
net/ipv4/udp.c +71 -0
83 lines not on this page · raw message
net/ipv4/udp_offload.c +4 -7
@@ -386,6 +386,8 @@ struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb,NAPI_GRO_CB (p)->same_flow =0 ;continue ; } + +/* TODO: for UDP_GRO: match size */ }skb_gro_pull (skb,sizeof (struct udphdr ));/* pull encapsulating udp header */
@@ -437,11 +439,6 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff, uh->len = newlen; -/* Set encapsulation before calling into inner gro_complete() functions -* to make them set up the inner offsets. -*/ - skb->encapsulation =1 ; -rcu_read_lock (); sk = (*lookup)(skb, uh->source, uh->dest);if (sk &&udp_sk (sk)->gro_complete)
@@ -462,11 +459,11 @@ static int udp4_gro_complete(struct sk_buff *skb, int nhoff)struct udphdr *uh = (struct udphdr *)(skb->data + nhoff);if (uh->check) { -skb_shinfo (skb)->gso_type |=SKB_GSO_UDP_TUNNEL_CSUM ; +skb_shinfo (skb)->gso_type |=SKB_GSO_UDP_L4 ; uh->check = ~udp_v4_check (skb->len - nhoff, iph->saddr, iph->daddr,0 ); }else { -skb_shinfo (skb)->gso_type |=SKB_GSO_UDP_TUNNEL ; +skb_shinfo (skb)->gso_type |=SKB_GSO_UDP_L4 ; }return udp_gro_complete (skb, nhoff, udp4_lib_lookup_skb);
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Paolo Abeni
maintainerHi all,
Quoted 2 lines · Willem de Bruijn
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
I'm wondering how we go on from this ? I'm fine with either approaches.
Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?
Cheers,
Paolo
Willem de Bruijn
maintainerQuoted 8 lines · Paolo Abeni
Hi all,
Quoted 2 lines · Willem de Bruijn
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
I'm wondering how we go on from this ? I'm fine with either approaches.
Let me send the udp gro static_key patch. Then we don't need the enable udp on demand logic (patch 2/4).
Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate UDP_GRO socket option instead of adding this to the existing UDP_SEGMENT.
Sounds good?
Quoted 3 lines
Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?
Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).
Paolo Abeni
maintainerHi,
Thank you for the prompt reply!
Quoted 11 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni
Hi all,
Quoted 2 lines · Willem de Bruijn
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
I'm wondering how we go on from this ? I'm fine with either approaches.
Let me send the udp gro
static_keypatch.
Would love that. We need to care of key decr, too (and possibly don't be fooled by encap_rcv() users).
Quoted 1 line
Then we don't need the enable udp on demand logic (patch 2/4).
ok.
Quoted 6 lines
Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate
UDP_GROsocket option instead of adding this to the existingUDP_SEGMENT.Sounds good?
Indeed!
I need also to add a cmsg to expose to the user the skb gro_size, and some test cases. Locally I'm [ab-]using the GRO functionality introduced recently on veth to test the code in a namespace pair (attaching a dummy XDP program to the RX-side veth). I'm not sure if that could fit a selftest.
Quoted 10 lines
Quoted 3 lines
Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?
Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).
Agreed.
Let's try the first step first ;)
Final but relevant note: I'll try my best to avoid delaying this, but lately I tend to be pre-empted by other tasks, it's difficult for me to assure a deadline here.
Cheers,
Paolo
Willem de Bruijn
maintainerQuoted 5 lines · Paolo Abeni
Hi,
Thank you for the prompt reply!
Not at all. Thanks for moving this forward :)
Quoted 9 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni
Quoted 7 lines
Hi all,
Quoted 2 lines · Willem de Bruijn
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I'm wondering how we go on from this ? I'm fine with either approaches.
Let me send the udp gro
static_keypatch.Would love that. We need to care of key decr, too (and possibly don't be fooled by
encap_rcv()users).
I just sent http://patchwork.ozlabs.org/patch/979525/
Right now all users are those that call setup_udp_tunnel_sock to register encap_rcv.
If accepted, I'll add a separate patch to decrement the key. That's probably in udp_tunnel_sock_release, but I need to take a closer look.
Quoted 9 lines
Quoted 1 line
Then we don't need the enable udp on demand logic (patch 2/4).
ok.
Quoted 5 lines
Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate
UDP_GROsocket option instead of adding this to the existingUDP_SEGMENT.
Quoted 8 lines
Quoted 1 line
Sounds good?
Indeed!
I need also to add a cmsg to expose to the user the skbgro_size, and some test cases. Locally I'm [ab-]using the GRO functionality introduced recently on veth to test the code in a namespace pair (attaching a dummy XDP program to the RX-side veth). I'm not sure if that could fit a selftest.
Very nice. Yes, veth only implements napi in xdp mode.
Quoted 24 lines
Quoted 11 lines
Quoted 4 lines
Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?
Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).
Agreed.
Let's try the first step first ;)
Final but relevant note: I'll try my best to avoid delaying this, but lately I tend to be pre-empted by other tasks, it's difficult for me to assure a deadline here.
Cheers,
Paolo
Paolo Abeni
maintainerQuoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
Would love that. We need to care of key decr, too (and possibly don't be fooled by
encap_rcv()users).I just sent http://patchwork.ozlabs.org/patch/979525/
Right now all users are those that call
setup_udp_tunnel_sockto registerencap_rcv.
plus setsockopt(UDP_ENCAP)
Quoted 3 lines
If accepted, I'll add a separate patch to decrement the key. That's probably in
udp_tunnel_sock_release, but I need to take a closer look.
l2tp calls setup_udp_tunnel_sock() but don't use udp_tunnel_sock_release(). Possibly it would be safer checking for:
up->encap_type || up(sk)->gro_receive
in udp_destroy_sock()
Cheers,
Paolo
Willem de Bruijn
maintainerQuoted 17 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
Would love that. We need to care of key decr, too (and possibly don't be fooled by
encap_rcv()users).I just sent http://patchwork.ozlabs.org/patch/979525/
Right now all users are those that call
setup_udp_tunnel_sockto registerencap_rcv.plus
setsockopt(UDP_ENCAP)Quoted 3 lines
If accepted, I'll add a separate patch to decrement the key. That's probably in
udp_tunnel_sock_release, but I need to take a closer look.
Quoted 6 lines
l2tp calls
setup_udp_tunnel_sock()but don't useudp_tunnel_sock_release(). Possibly it would be safer checking for:
up->encap_type|| up(sk)->gro_receivein
udp_destroy_sock()
Ah indeed. And gtp might be another example.
Thanks!
Steffen Klassert
maintainerQuoted 24 lines · Willem de Bruijn
Quoted 8 lines · Paolo Abeni
Hi all,
Quoted 2 lines · Willem de Bruijn
This is a *very rough* draft. Mainly for discussion while we also look at another partially overlapping approach [1].
I'm wondering how we go on from this ? I'm fine with either approaches.
Let me send the udp gro
static_keypatch. Then we don't need the enable udp on demand logic (patch 2/4).Your implementation of GRO is more fleshed out (patch 3/4) than my quick hack. My only request would be to use a separate
UDP_GROsocket option instead of adding this to the existingUDP_SEGMENT.Sounds good?
Quoted 3 lines
Also, I'm interested in [try to] enable GRO/GSO batching in the forwarding path, as you outlined initially in the GSO series submission. That should cover Steffen use-case, too, right?
Quoted 6 lines
Great. Indeed. Though there is some unresolved discussion on one large gso skb vs frag list. There has been various concerns around the use of frag lists for GSO in the past, and it does not match h/w offload. So I think the answer would be the first unless the second proves considerably faster (in which case it could also be added later as optimization).
I think it depends a bit on the usecase and hardware etc. if the first or the second approach is faster. So it would be good if we can choose which one to use depending on that.
For local socket receiving, building big GSO packets is likely faster than the chaining method.
But on forwarding the chaining method might be faster because we don't have the overhead of creating GSO packets and of segmenting them back to their native form (at least as long as we don't have NICs that support hardware UDP GSO). Same applies to packets that undergo IPsec transformation.
Another thing where the chaining method could be intersting is when we receive already big LRO or HW GRO packets from the NIC. Packets of the same flow could still travel together through the stack with the chaining method. I've never tried this, though. For now it is just an idea.
I have the code for the chaining mehthod here, I'd just need some method to hook it in. Maybe it could be done with some sort of an inet_update_offload() as Paolo already propsed in his pachset, or we could make it configurable per device...