net: gro: encapsulation bug fix and flush checks improvements
Subsystems (6)
v6 cover net: gro: encapsulation bug fix and flush checks improvements
This series fixes a bug in the complete phase of UDP in GRO, in which socket lookup fails due to using network_header when parsing encapsulated packets. The fix is to pass p_off parameter in *_gro_complete.
Next, the fields network_offset and inner_network_offset are added to napi_gro_cb, and are both set during the receive phase of GRO. This is then leveraged in the next commit to remove flush_id state from napi_gro_cb, and stateful code in {ipv6,inet}_gro_receive which may be unnecessarily complicated due to encapsulation support in GRO.
In addition, udpgro_fwd selftest is adjusted to include the socket lookup
case for vxlan. This selftest will test its supposed functionality once
local bind support is merged (https://lore.kernel.org/netdev/[email protected]/).
v5 -> v6:
- Write inner_network_offset in vxlan, geneve and ipsec
- Ignore is_atomic when DF=0
- v5:
https://lore.kernel.org/all/[email protected]/
v4 -> v5:
- Add 1st commit - flush id checks in udp_gro_receive segment which can be backported by itself
- Add TCP measurements for the 5th commit
- Add flush id tests to ensure flush id logic is preserved in GRO
- Simplify gro_inet_flush by removing a branch
- v4:
https://lore.kernel.org/all/[email protected]/
v3 -> v4:
- Fix code comment and commit message typos
- v3:
https://lore.kernel.org/all/[email protected]/
v2 -> v3:
- Use napi_gro_cb instead of skb->{offset}
- v2:
https://lore.kernel.org/all/[email protected]/
v1 -> v2:
- Pass p_off in *_gro_complete to fix UDP bug
- Remove more conditionals and memory fetches from inet_gro_flush
- v1:
https://lore.kernel.org/netdev/[email protected]/
Richard Gobert (6):
net: gro: add flush check in udp_gro_receive_segment
net: gro: add p_off param in *_gro_complete
selftests/net: add local address bind in vxlan selftest
net: gro: add {inner_}network_offset to napi_gro_cb
net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment selftests/net: add flush id selftests
drivers/net/geneve.c | 8 +- drivers/net/vxlan/vxlan_core.c | 12 +- include/linux/etherdevice.h | 2 +- include/linux/netdevice.h | 3 +- include/linux/udp.h | 2 +- include/net/gro.h | 93 ++++++++++++-- include/net/inet_common.h | 2 +- include/net/tcp.h | 6 +- include/net/udp.h | 8 +- include/net/udp_tunnel.h | 2 +- net/8021q/vlan_core.c | 6 +- net/core/gro.c | 7 +- net/ethernet/eth.c | 5 +- net/ipv4/af_inet.c | 54 +------- net/ipv4/fou_core.c | 9 +- net/ipv4/gre_offload.c | 6 +- net/ipv4/tcp_offload.c | 22 +--- net/ipv4/udp.c | 3 +- net/ipv4/udp_offload.c | 31 +++-- net/ipv6/ip6_offload.c | 41 +++--- net/ipv6/tcpv6_offload.c | 7 +- net/ipv6/udp.c | 3 +- net/ipv6/udp_offload.c | 13 +- tools/testing/selftests/net/gro.c | 144 ++++++++++++++++++++++ tools/testing/selftests/net/udpgro_fwd.sh | 10 +- 25 files changed, 338 insertions(+), 161 deletions(-)
Signature · 1 line
2.36.1
PATCH v6 1/6 net: gro: add flush check in udp_gro_receive_segment
GRO-GSO path is supposed to be transparent and as such L3 flush checks are relevant to all flows which call skb_gro_receive. This patch uses the same logic and code from tcp_gro_receive but in the relevant flow path in udp_gro_receive_segment.
--- net/ipv4/udp_offload.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
net/ipv4/udp_offload.c +12 -1
@@ -471,6 +471,7 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head,struct sk_buff *p;unsigned int ulen;int ret =0 ; +int flush;/* requires non zero csum, for symmetry with GSO */ if (!uh->check) {
@@ -528,7 +529,17 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head,skb_gro_postpull_rcsum (skb, uh,sizeof (struct udphdr )); - ret =skb_gro_receive (p, skb); + flush =NAPI_GRO_CB (p)->flush; + +if (NAPI_GRO_CB (p)->flush_id !=1 || +NAPI_GRO_CB (p)->count !=1 || + !NAPI_GRO_CB (p)->is_atomic) + flush |=NAPI_GRO_CB (p)->flush_id; +else +NAPI_GRO_CB (p)->is_atomic = false; + +if (flush ||skb_gro_receive (p, skb)) + ret =1 ; } }
Signature · 1 line
2.36.1
PATCH v6 2/6 net: gro: add p_off param in *_gro_complete
In reply to Richard GobertCommits a602456 ("udp: Add GRO functions to UDP socket") and 57c67ff ("udp:
additional GRO support") introduce incorrect usage of {ip,ipv6}_hdr in the complete phase of gro. The functions always return skb->network_header, which in the case of encapsulated packets at the gro complete phase, is always set to the innermost L3 of the packet. That means that calling {ip,ipv6}_hdr for skbs which completed the GRO receive phase (both in gro_list and *_gro_complete) when parsing an encapsulated packet's _outer_ L3/L4 may return an unexpected value.
This incorrect usage leads to a bug in GRO's UDP socket lookup. udp{4,6}_lib_lookup_skb functions use ip_hdr/ipv6_hdr respectively. These *_hdr functions return network_header which will point to the innermost L3, resulting in the wrong offset being used in __udp{4,6}_lib_lookup with encapsulated packets.
To fix this issue p_off param is used in *_gro_complete to pass off the offset of the previous layer.
Reproduction example:
Endpoint configuration example (fou + local address bind)
# ip fou add port 6666 ipproto 4
# ip link add name tun1 type ipip remote 2.2.2.1 local 2.2.2.2 encap fou encap-dport 5555 encap-sport 6666 mode ipip
# ip link set tun1 up
# ip a add 1.1.1.2/24 dev tun1Netperf TCP_STREAM result on net-next before patch is applied:
net-next main, GRO enabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec131072 16384 16384 5.28 2.37
net-next main, GRO disabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec131072 16384 16384 5.01 2745.06
patch applied, GRO enabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec131072 16384 16384 5.01 2877.38
--- drivers/net/geneve.c | 7 ++++--- drivers/net/vxlan/vxlan_core.c | 11 +++++++---- include/linux/etherdevice.h | 2 +- include/linux/netdevice.h | 3 ++- include/linux/udp.h | 2 +- include/net/gro.h | 11 ++++++----- include/net/inet_common.h | 2 +- include/net/tcp.h | 6 ++++-- include/net/udp.h | 8 ++++---- include/net/udp_tunnel.h | 2 +- net/8021q/vlan_core.c | 4 ++-- net/core/gro.c | 2 +- net/ethernet/eth.c | 4 ++-- net/ipv4/af_inet.c | 8 ++++---- net/ipv4/fou_core.c | 9 +++++---- net/ipv4/gre_offload.c | 5 +++-- net/ipv4/tcp_offload.c | 7 ++++--- net/ipv4/udp.c | 3 ++- net/ipv4/udp_offload.c | 26 ++++++++++++++------------ net/ipv6/ip6_offload.c | 22 ++++++++++++---------- net/ipv6/tcpv6_offload.c | 7 ++++--- net/ipv6/udp.c | 3 ++- net/ipv6/udp_offload.c | 13 +++++++------ 23 files changed, 93 insertions(+), 74 deletions(-)
drivers/net/geneve.c +4 -3
@@ -555,7 +555,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk, }static int geneve_gro_complete (struct sock *sk,struct sk_buff *skb, -int nhoff) +int p_off,int nhoff) {struct genevehdr *gh;struct packet_offload *ptype;
@@ -569,11 +569,12 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb,/* since skb->encapsulation is set, eth_gro_complete() sets the inner mac header */ if (likely (type ==htons (ETH_P_TEB ))) -return eth_gro_complete (skb, nhoff + gh_len); +return eth_gro_complete (skb, p_off, nhoff + gh_len); ptype =gro_find_complete_by_type (type);if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff + gh_len); + err = ptype->callbacks.gro_complete (skb, p_off, + nhoff + gh_len);skb_set_inner_mac_header (skb, nhoff + gh_len);
drivers/net/vxlan/vxlan_core.c +7 -4
@@ -767,15 +767,17 @@ static struct sk_buff *vxlan_gpe_gro_receive(struct sock *sk,return pp; } -static int vxlan_gro_complete (struct sock *sk,struct sk_buff *skb,int nhoff) +static int vxlan_gro_complete (struct sock *sk,struct sk_buff *skb, +int p_off,int nhoff) {/* Sets 'skb->inner_mac_header' since we are always called with * 'skb->encapsulation' set. */ -return eth_gro_complete (skb, nhoff +sizeof (struct vxlanhdr )); +return eth_gro_complete (skb, p_off, nhoff +sizeof (struct vxlanhdr )); } -static int vxlan_gpe_gro_complete (struct sock *sk,struct sk_buff *skb,int nhoff) +static int vxlan_gpe_gro_complete (struct sock *sk,struct sk_buff *skb, +int p_off,int nhoff) {struct vxlanhdr *vh = (struct vxlanhdr *)(skb->data + nhoff);const struct packet_offload *ptype;
@@ -786,7 +788,8 @@ static int vxlan_gpe_gro_complete(struct sock *sk, struct sk_buff *skb, int nhofreturn err; ptype =gro_find_complete_by_type (protocol);if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff +sizeof (struct vxlanhdr )); + err = ptype->callbacks.gro_complete (skb, p_off, nhoff + +sizeof (struct vxlanhdr ));return err; }
include/linux/etherdevice.h +1 -1
@@ -64,7 +64,7 @@ struct net_device *devm_alloc_etherdev_mqs(struct device *dev, int sizeof_priv,#define devm_alloc_etherdev (dev, sizeof_priv) devm_alloc_etherdev_mqs(dev, sizeof_priv, 1, 1)struct sk_buff *eth_gro_receive (struct list_head *head,struct sk_buff *skb); -int eth_gro_complete (struct sk_buff *skb,int nhoff); +int eth_gro_complete (struct sk_buff *skb,int p_off,int nhoff);/* Reserved Ethernet Addresses per IEEE 802.1Q */ static const u8 eth_reserved_addr_base[ETH_ALEN] __aligned(2) =
include/linux/netdevice.h +2 -1
@@ -2710,7 +2710,8 @@ struct offload_callbacks { netdev_features_t features);struct sk_buff *(*gro_receive)(struct list_head *head,struct sk_buff *skb); -int (*gro_complete)(struct sk_buff *skb,int nhoff); +int (*gro_complete)(struct sk_buff *skb,int nhoff, +int thoff); };struct packet_offload {
include/linux/udp.h +1 -1
@@ -82,7 +82,7 @@ struct udp_sock { struct sk_buff *skb);int (*gro_complete)(struct sock *sk,struct sk_buff *skb, -int nhoff); +int nhoff,int thoff);/* udp_recvmsg try to use this before splicing sk_receive_queue */ struct sk_buff_head reader_queue____cacheline_aligned_in_smp ;
include/net/gro.h +6 -5
@@ -383,18 +383,18 @@ static inline void skb_gro_flush_final_remcsum(struct sk_buff *skb, INDIRECT_CALLABLE_DECLARE(struct sk_buff *ipv6_gro_receive(struct list_head *,struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *inet_gro_receive(struct list_head *,struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int inet_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int inet_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *udp4_gro_receive(struct list_head *,struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int udp4_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int udp4_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *udp6_gro_receive(struct list_head *,struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int, int));#define indirect_call_gro_receive_inet (cb, f2, f1, head, skb) \ ({ \
@@ -405,7 +405,8 @@ INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int));struct sk_buff *udp_gro_receive (struct list_head *head,struct sk_buff *skb,struct udphdr *uh,struct sock *sk); -int udp_gro_complete (struct sk_buff *skb,int nhoff,udp_lookup_t lookup); +int udp_gro_complete (struct sk_buff *skb,int nhoff,int thoff, +udp_lookup_t lookup);static inline struct udphdr *udp_gro_udphdr (struct sk_buff *skb) {
include/net/inet_common.h +1 -1
@@ -64,7 +64,7 @@ int inet_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len);struct sk_buff *inet_gro_receive (struct list_head *head,struct sk_buff *skb); -int inet_gro_complete (struct sk_buff *skb,int nhoff); +int inet_gro_complete (struct sk_buff *skb,int nhoff,int thoff);struct sk_buff *inet_gso_segment (struct sk_buff *skb,netdev_features_t features);
include/net/tcp.h +4 -2
@@ -2196,9 +2196,11 @@ void tcp_v4_destroy_sock(struct sock *sk);struct sk_buff *tcp_gso_segment (struct sk_buff *skb,netdev_features_t features);struct sk_buff *tcp_gro_receive (struct list_head *head,struct sk_buff *skb); -INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); +INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); -INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff)); +INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int nhoff, +int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb));#ifdef CONFIG_INET void tcp_gro_complete (struct sk_buff *skb);
include/net/udp.h +4 -4
@@ -166,8 +166,8 @@ static inline void udp_csum_pull_header(struct sk_buff *skb)UDP_SKB_CB (skb)->cscov -=sizeof (struct udphdr ); } -typedef struct sock *(*udp_lookup_t )(const struct sk_buff *skb,__be16 sport, -__be16 dport); +typedef struct sock *(*udp_lookup_t )(const struct sk_buff *skb,int nhoff, +__be16 sport,__be16 dport);void udp_v6_early_demux (struct sk_buff *skb); INDIRECT_CALLABLE_DECLARE(int udpv6_rcv(struct sk_buff *));
@@ -301,7 +301,7 @@ struct sock *udp4_lib_lookup(struct net *net, __be32 saddr, __be16 sport,struct sock *__udp4_lib_lookup (struct net *net,__be32 saddr,__be16 sport,__be32 daddr,__be16 dport,int dif,int sdif,struct udp_table *tbl,struct sk_buff *skb); -struct sock *udp4_lib_lookup_skb (const struct sk_buff *skb, +struct sock *udp4_lib_lookup_skb (const struct sk_buff *skb,int nhoff,__be16 sport,__be16 dport); struct sock *udp6_lib_lookup(struct net *net, const struct in6_addr *saddr, __be16 sport,
@@ -312,7 +312,7 @@ struct sock *__udp6_lib_lookup(struct net *net, const struct in6_addr *daddr, __be16 dport, int dif, int sdif, struct udp_table *tbl, struct sk_buff *skb); -struct sock *udp6_lib_lookup_skb (const struct sk_buff *skb, +struct sock *udp6_lib_lookup_skb (const struct sk_buff *skb,int nhoff,__be16 sport,__be16 dport);int udp_read_skb (struct sock *sk,skb_read_actor_t recv_actor);
include/net/udp_tunnel.h +1 -1
@@ -75,7 +75,7 @@ typedef struct sk_buff *(*udp_tunnel_gro_receive_t)(struct sock *sk,struct list_head *head, struct sk_buff *skb);typedef int (*udp_tunnel_gro_complete_t )(struct sock *sk,struct sk_buff *skb, -int nhoff); +int nhoff,int thoff);struct udp_tunnel_sock_cfg {void *sk_user_data;/* user data used by encap_rcv call back */
net/8021q/vlan_core.c +2 -2
@@ -510,7 +510,7 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head,return pp; } -static int vlan_gro_complete (struct sk_buff *skb,int nhoff) +static int vlan_gro_complete (struct sk_buff *skb,int p_off,int nhoff) {struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);__be16 type = vhdr->h_vlan_encapsulated_proto;
@@ -521,7 +521,7 @@ static int vlan_gro_complete(struct sk_buff *skb, int nhoff)if (ptype) err =INDIRECT_CALL_INET (ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb, nhoff +sizeof (*vhdr)); + skb, p_off, nhoff +sizeof (*vhdr));return err; }
net/core/gro.c +1 -1
@@ -252,7 +252,7 @@ static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) err =INDIRECT_CALL_INET (ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb,0 ); + skb,0 ,0 );break ; }rcu_read_unlock ();
net/ethernet/eth.c +2 -2
@@ -453,7 +453,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) }EXPORT_SYMBOL (eth_gro_receive); -int eth_gro_complete (struct sk_buff *skb,int nhoff) +int eth_gro_complete (struct sk_buff *skb,int p_off,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff);__be16 type = eh->h_proto;
@@ -467,7 +467,7 @@ int eth_gro_complete(struct sk_buff *skb, int nhoff)if (ptype !=NULL ) err =INDIRECT_CALL_INET (ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb, nhoff +sizeof (*eh)); + skb, p_off, nhoff +sizeof (*eh));return err; }
net/ipv4/af_inet.c +4 -4
@@ -1641,7 +1641,7 @@ int inet_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len) }EXPORT_SYMBOL (inet_recv_error); -int inet_gro_complete (struct sk_buff *skb,int nhoff) +int inet_gro_complete (struct sk_buff *skb,int prior_off,int nhoff) {struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);const struct net_offload *ops;
@@ -1667,17 +1667,17 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)*/ err =INDIRECT_CALL_2 (ops->callbacks.gro_complete, tcp4_gro_complete, udp4_gro_complete, - skb, nhoff +sizeof (*iph)); + skb, nhoff, nhoff +sizeof (*iph)); out:return err; } -static int ipip_gro_complete (struct sk_buff *skb,int nhoff) +static int ipip_gro_complete (struct sk_buff *skb,int prior_off,int nhoff) { skb->encapsulation =1 ;skb_shinfo (skb)->gso_type |=SKB_GSO_IPXIP4 ; -return inet_gro_complete (skb, nhoff); +return inet_gro_complete (skb, prior_off, nhoff); } int inet_ctl_sock_create(struct sock **sk, unsigned short family,
net/ipv4/fou_core.c +5 -4
@@ -260,7 +260,7 @@ static struct sk_buff *fou_gro_receive(struct sock *sk, }static int fou_gro_complete (struct sock *sk,struct sk_buff *skb, -int nhoff) +int p_off,int nhoff) {const struct net_offload __rcu **offloads;u8 proto =fou_from_sock (sk)->protocol;
@@ -272,7 +272,7 @@ static int fou_gro_complete(struct sock *sk, struct sk_buff *skb,if (WARN_ON (!ops || !ops->callbacks.gro_complete)) goto out; - err = ops->callbacks.gro_complete (skb, nhoff); + err = ops->callbacks.gro_complete (skb, p_off, nhoff);skb_set_inner_mac_header (skb, nhoff);
@@ -445,7 +445,8 @@ static struct sk_buff *gue_gro_receive(struct sock *sk,return pp; } -static int gue_gro_complete (struct sock *sk,struct sk_buff *skb,int nhoff) +static int gue_gro_complete (struct sock *sk,struct sk_buff *skb, +int p_off,int nhoff) {struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff);const struct net_offload __rcu **offloads;
@@ -480,7 +481,7 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff)if (WARN_ON (!ops || !ops->callbacks.gro_complete)) goto out; - err = ops->callbacks.gro_complete (skb, nhoff + guehlen); + err = ops->callbacks.gro_complete (skb, p_off, nhoff + guehlen);skb_set_inner_mac_header (skb, nhoff + guehlen);
net/ipv4/gre_offload.c +3 -2
@@ -233,7 +233,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,return pp; } -static int gre_gro_complete (struct sk_buff *skb,int nhoff) +static int gre_gro_complete (struct sk_buff *skb,int p_off,int nhoff) {struct gre_base_hdr *greh = (struct gre_base_hdr *)(skb->data + nhoff);struct packet_offload *ptype;
@@ -253,7 +253,8 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff) ptype =gro_find_complete_by_type (type);if (ptype) - err = ptype->callbacks.gro_complete (skb, nhoff + grehlen); + err = ptype->callbacks.gro_complete (skb, p_off, + nhoff + grehlen);skb_set_inner_mac_header (skb, nhoff + grehlen);
net/ipv4/tcp_offload.c +4 -3
@@ -328,10 +328,11 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)return tcp_gro_receive (head, skb); } -INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct iphdr *iph =ip_hdr (skb); -struct tcphdr *th =tcp_hdr (skb); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); +struct tcphdr *th = (struct tcphdr *)(skb->data + thoff); th->check = ~tcp_v4_check (skb->len - thoff, iph->saddr, iph->daddr,0 );
net/ipv4/udp.c +2 -1
@@ -530,9 +530,10 @@ static inline struct sock *__udp4_lib_lookup_skb(struct sk_buff *skb, }struct sock *udp4_lib_lookup_skb (const struct sk_buff *skb, +int nhoff,__be16 sport,__be16 dport) { -const struct iphdr *iph =ip_hdr (skb); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff);struct net *net =dev_net (skb->dev);int iif, sdif;
net/ipv4/udp_offload.c +14 -12
@@ -689,18 +689,19 @@ static int udp_gro_complete_segment(struct sk_buff *skb)return 0 ; } -int udp_gro_complete (struct sk_buff *skb,int nhoff, +int udp_gro_complete (struct sk_buff *skb,int nhoff,int thoff,udp_lookup_t lookup) { -__be16 newlen =htons (skb->len - nhoff); -struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); +struct udphdr *uh = (struct udphdr *)(skb->data + thoff); +__be16 newlen =htons (skb->len - thoff);struct sock *sk;int err; uh->len = newlen; sk =INDIRECT_CALL_INET (lookup, udp6_lib_lookup_skb, - udp4_lib_lookup_skb, skb, uh->source, uh->dest); + udp4_lib_lookup_skb, skb, nhoff, uh->source, + uh->dest);if (sk &&udp_sk (sk)->gro_complete) {skb_shinfo (skb)->gso_type = uh->check ?SKB_GSO_UDP_TUNNEL_CSUM :SKB_GSO_UDP_TUNNEL ;
@@ -714,8 +715,8 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff,* functions to make them set up the inner offsets. */ skb->encapsulation =1 ; - err =udp_sk (sk)->gro_complete (sk, skb, - nhoff +sizeof (struct udphdr )); + err =udp_sk (sk)->gro_complete (sk, skb, nhoff, + thoff +sizeof (struct udphdr )); }else { err =udp_gro_complete_segment (skb); }
@@ -727,14 +728,15 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff, }EXPORT_SYMBOL (udp_gro_complete); -INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct iphdr *iph =ip_hdr (skb); -struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); +struct udphdr *uh = (struct udphdr *)(skb->data + thoff);/* do fraglist only if there is no outer UDP encap (or we already processed it) */ if (NAPI_GRO_CB (skb)->is_flist && !NAPI_GRO_CB (skb)->encap_mark) { - uh->len =htons (skb->len - nhoff); + uh->len =htons (skb->len - thoff);skb_shinfo (skb)->gso_type |= (SKB_GSO_FRAGLIST |SKB_GSO_UDP_L4 );skb_shinfo (skb)->gso_segs =NAPI_GRO_CB (skb)->count;
@@ -745,10 +747,10 @@ INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) }if (uh->check) - uh->check = ~udp_v4_check (skb->len - nhoff, iph->saddr, + uh->check = ~udp_v4_check (skb->len - thoff, iph->saddr, iph->daddr,0 ); -return udp_gro_complete (skb, nhoff, udp4_lib_lookup_skb); +return udp_gro_complete (skb, nhoff, thoff, udp4_lib_lookup_skb); }int __init udpv4_offload_init (void )
net/ipv6/ip6_offload.c +12 -10
@@ -346,12 +346,14 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head,return inet_gro_receive (head, skb); } -INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, +int p_off,int nhoff) {const struct net_offload *ops;struct ipv6hdr *iph;int err = -ENOSYS ;u32 payload_len; +int nhlen;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IPV6 ));
@@ -387,36 +389,36 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); } - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); -if (WARN_ON (!ops || !ops->callbacks.gro_complete)) + nhlen =sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (WARN_ON_ONCE (!ops || !ops->callbacks.gro_complete)) goto out; err =INDIRECT_CALL_L4 (ops->callbacks.gro_complete, tcp6_gro_complete, - udp6_gro_complete, skb, nhoff); + udp6_gro_complete, skb, nhoff, nhoff + nhlen); out:return err; } -static int sit_gro_complete (struct sk_buff *skb,int nhoff) +static int sit_gro_complete (struct sk_buff *skb,int p_off,int nhoff) { skb->encapsulation =1 ;skb_shinfo (skb)->gso_type |=SKB_GSO_IPXIP4 ; -return ipv6_gro_complete (skb, nhoff); +return ipv6_gro_complete (skb, p_off, nhoff); } -static int ip6ip6_gro_complete (struct sk_buff *skb,int nhoff) +static int ip6ip6_gro_complete (struct sk_buff *skb,int p_off,int nhoff) { skb->encapsulation =1 ;skb_shinfo (skb)->gso_type |=SKB_GSO_IPXIP6 ; -return ipv6_gro_complete (skb, nhoff); +return ipv6_gro_complete (skb, p_off, nhoff); } -static int ip4ip6_gro_complete (struct sk_buff *skb,int nhoff) +static int ip4ip6_gro_complete (struct sk_buff *skb,int p_off,int nhoff) { skb->encapsulation =1 ;skb_shinfo (skb)->gso_type |=SKB_GSO_IPXIP6 ; -return inet_gro_complete (skb, nhoff); +return inet_gro_complete (skb, p_off, nhoff); }
net/ipv6/tcpv6_offload.c +4 -3
@@ -27,10 +27,11 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)return tcp_gro_receive (head, skb); } -INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, +int nhoff,int thoff) { -const struct ipv6hdr *iph =ipv6_hdr (skb); -struct tcphdr *th =tcp_hdr (skb); +const struct ipv6hdr *iph = (const struct ipv6hdr *)(skb->data + nhoff); +struct tcphdr *th = (struct tcphdr *)(skb->data + thoff); th->check = ~tcp_v6_check (skb->len - thoff, &iph->saddr, &iph->daddr,0 );
net/ipv6/udp.c +2 -1
@@ -271,9 +271,10 @@ static struct sock *__udp6_lib_lookup_skb(struct sk_buff *skb, }struct sock *udp6_lib_lookup_skb (const struct sk_buff *skb, +int nhoff,__be16 sport,__be16 dport) { -const struct ipv6hdr *iph =ipv6_hdr (skb); +const struct ipv6hdr *iph = (const struct ipv6hdr *)(skb->data + nhoff);struct net *net =dev_net (skb->dev);int iif, sdif;
net/ipv6/udp_offload.c +7 -6
@@ -162,14 +162,15 @@ struct sk_buff *udp6_gro_receive(struct list_head *head, struct sk_buff *skb)return NULL ; } -INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct ipv6hdr *ipv6h =ipv6_hdr (skb); -struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); +const struct ipv6hdr *ipv6h = (const struct ipv6hdr *)(skb->data + nhoff); +struct udphdr *uh = (struct udphdr *)(skb->data + thoff);/* do fraglist only if there is no outer UDP encap (or we already processed it) */ if (NAPI_GRO_CB (skb)->is_flist && !NAPI_GRO_CB (skb)->encap_mark) { - uh->len =htons (skb->len - nhoff); + uh->len =htons (skb->len - thoff);skb_shinfo (skb)->gso_type |= (SKB_GSO_FRAGLIST |SKB_GSO_UDP_L4 );skb_shinfo (skb)->gso_segs =NAPI_GRO_CB (skb)->count;
@@ -180,10 +181,10 @@ INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) }if (uh->check) - uh->check = ~udp_v6_check (skb->len - nhoff, &ipv6h->saddr, + uh->check = ~udp_v6_check (skb->len - thoff, &ipv6h->saddr, &ipv6h->daddr,0 ); -return udp_gro_complete (skb, nhoff, udp6_lib_lookup_skb); +return udp_gro_complete (skb, nhoff, thoff, udp6_lib_lookup_skb); }int __init udpv6_offload_init (void )
Signature · 1 line
2.36.1
Willem de Bruijn
maintainerRe: [PATCH net-next v6 2/6] net: gro: add p_off param in *_gro_complete
Quoted 9 lines · Richard Gobert
Commits a602456 ("udp: Add GRO functions to UDP socket") and 57c67ff ("udp:
additional GRO support") introduce incorrect usage of {ip,ipv6}_hdr in the complete phase of gro. The functions always returnskb->network_header, which in the case of encapsulated packets at the gro complete phase, is always set to the innermost L3 of the packet. That means that calling {ip,ipv6}_hdr for skbs which completed the GRO receive phase (both ingro_listand*_gro_complete) when parsing an encapsulated packet's _outer_ L3/L4 may return an unexpected value.
Quoted 8 lines
This incorrect usage leads to a bug in GRO's UDP socket lookup. udp{4,6}_lib_lookup_skb functions use ip_hdr/ipv6_hdr respectively. These
*_hdrfunctions returnnetwork_headerwhich will point to the innermost L3, resulting in the wrong offset being used in __udp{4,6}_lib_lookup with encapsulated packets.To fix this issue
p_offparam is used in*_gro_completeto pass off the offset of the previous layer.
What exactly does this mean?
This patch changes the definition of gro_complete to add a thoff alongside the existing "nhoff"..
include/linux/netdevice.h struct offload_callbacks { · line 2714
Quoted 3 lines
include/linux/netdevice.h +2
-1-int (*gro_complete)(struct sk_buff *skb,int nhoff); +int (*gro_complete)(struct sk_buff *skb,int nhoff, +int thoff);
.. but also fixes up implementations to interpret the existing argument as a thoff
net/ipv4/tcp_offload.c struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) · line 335
Quoted 8 lines
net/ipv4/tcp_offload.c +4
-3-INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct iphdr *iph =ip_hdr (skb); -struct tcphdr *th =tcp_hdr (skb); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); +struct tcphdr *th = (struct tcphdr *)(skb->data + thoff);
But in some cases the new argument is not nhoff but p_off, e.g.,
Review
Quoted 3 lines
static int geneve_gro_complete(struct sock *sk, struct
sk_buff*skb,(quoted diff) +1
-1- int nhoff) + int p_off, int nhoff)
Really, the argument is the start of the next header, each callback just casts to its expected header (ethhdr, tcphdr, etc.)
The only place where we need to pass an extra argument is in udp, because that needs a pointer to the network header right before the transport header pointed to by nhoff.
And only due to possible IPv4 options or IPv6 extension headers, we cannot just do
+ struct udphdr *iph = (struct iphdr *)(skb->data + nhoff - sizeof(*iph));
struct udphdr *uh = (struct udphdr *)(skb->data + nhoff);I also do not immediately see an a way to avoid all the boilerplate of a new argument in every callback. Aside from a per_cpu var -- but that is excessive.
But it can just be left zero in all callsites, except for inet_gro_complete/ipv6_gro_complete, which pass in nhoff.
Quoted 37 lines
Reproduction example:
Endpoint configuration example (fou + local address bind)
# ip fou add port 6666 ipproto 4 # ip link add name tun1 type ipip remote 2.2.2.1 local 2.2.2.2 encap fou encap-dport 5555 encap-sport 6666 mode ipip # ip link set tun1 up # ip a add 1.1.1.2/24 dev tun1Netperf
TCP_STREAMresult on net-next before patch is applied:net-next main, GRO enabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec131072 16384 16384 5.28 2.37net-next main, GRO disabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec131072 16384 16384 5.01 2745.06patch applied, GRO enabled:
$ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
Quoted 3 lines
131072 16384 16384 5.01 2877.38Fixes 57c67ff4bd92 ("udp: additional GRO support")
This and the previous change should really target net, as they are bug fixes.
Willem de Bruijn
maintainerRe: [PATCH net-next v6 2/6] net: gro: add p_off param in *_gro_complete
Quoted 59 lines · Willem de Bruijn
Quoted 9 lines · Richard Gobert
Commits a602456 ("udp: Add GRO functions to UDP socket") and 57c67ff ("udp:
additional GRO support") introduce incorrect usage of {ip,ipv6}_hdr in the complete phase of gro. The functions always returnskb->network_header, which in the case of encapsulated packets at the gro complete phase, is always set to the innermost L3 of the packet. That means that calling {ip,ipv6}_hdr for skbs which completed the GRO receive phase (both ingro_listand*_gro_complete) when parsing an encapsulated packet's _outer_ L3/L4 may return an unexpected value.Quoted 8 lines
This incorrect usage leads to a bug in GRO's UDP socket lookup. udp{4,6}_lib_lookup_skb functions use ip_hdr/ipv6_hdr respectively. These
*_hdrfunctions returnnetwork_headerwhich will point to the innermost L3, resulting in the wrong offset being used in __udp{4,6}_lib_lookup with encapsulated packets.To fix this issue
p_offparam is used in*_gro_completeto pass off the offset of the previous layer.What exactly does this mean?
This patch changes the definition of
gro_completeto add a thoff alongside the existing "nhoff"..include/linux/netdevice.h struct offload_callbacks { · line 2714
Quoted 3 lines
include/linux/netdevice.h +2
-1-int (*gro_complete)(struct sk_buff *skb,int nhoff); +int (*gro_complete)(struct sk_buff *skb,int nhoff, +int thoff);.. but also fixes up implementations to interpret the existing argument as a thoff
net/ipv4/tcp_offload.c struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) · line 335
Quoted 8 lines
net/ipv4/tcp_offload.c +4
-3-INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct iphdr *iph =ip_hdr (skb); -struct tcphdr *th =tcp_hdr (skb); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); +struct tcphdr *th = (struct tcphdr *)(skb->data + thoff);But in some cases the new argument is not nhoff but
p_off, e.g.,Review
Quoted 3 lines
static int geneve_gro_complete(struct sock *sk, struct
sk_buff*skb,(quoted diff) +1
-1- int nhoff) + int p_off, int nhoff)Really, the argument is the start of the next header, each callback just casts to its expected header (ethhdr, tcphdr, etc.)
The only place where we need to pass an extra argument is in udp, because that needs a pointer to the network header right before the transport header pointed to by nhoff.
And only due to possible IPv4 options or IPv6 extension headers, we cannot just do
+ struct udphdr *iph = (struct iphdr *)(skb->data + nhoff - sizeof(*iph)); struct udphdr *uh = (struct udphdr *)(skb->data + nhoff);
Quoted 6 lines
I also do not immediately see an a way to avoid all the boilerplate of a new argument in every callback. Aside from a
per_cpuvar -- but that is excessive.But it can just be left zero in all callsites, except for inet_gro_complete/ipv6_gro_complete, which pass in nhoff.
Actually, we can avoid the boilerplate changes that add an extra arg.
By changing the contract between network layer callbacks (inet_gro_complete/ipv6_gro_complete) and transport layer callbacks (tcp4_gro_complete et al).
If the first pass their own unmodified offset, nhoff:
err = INDIRECT_CALL_2(ops->callbacks.gro_complete,
tcp4_gro_complete, udp4_gro_complete,- skb, nhoff + sizeof(*iph));
+ skb, nhoff);
And the latter parse the network header for total_len/payload_len, to find their original offset.
It's also a bit of a hack. But a lot smaller patch, probably.
Re: [PATCH net-next v6 2/6] net: gro: add p_off param in *_gro_complete
Quoted 4 lines · Willem de Bruijn
And the latter parse the network header for total_len/payload_len, to find their original offset.
It's also a bit of a hack. But a lot smaller patch, probably.
While producing a smaller change - this is a bit of a hacky code change.
My initial idea was to use network_offset added to the CB in the 4th commit
- making a very small change. (https://lore.kernel.org/netdev/[email protected]/)
Eric suggested it'd make more sense to pass it off as a parameter throughout the complete chain.
Willem de Bruijn
maintainerRe: [PATCH net-next v6 2/6] net: gro: add p_off param in *_gro_complete
In reply to Willem de BruijnQuoted 64 lines · Willem de Bruijn
Quoted 63 lines · Willem de Bruijn
Quoted 9 lines · Richard Gobert
Commits a602456 ("udp: Add GRO functions to UDP socket") and 57c67ff ("udp:
additional GRO support") introduce incorrect usage of {ip,ipv6}_hdr in the complete phase of gro. The functions always returnskb->network_header, which in the case of encapsulated packets at the gro complete phase, is always set to the innermost L3 of the packet. That means that calling {ip,ipv6}_hdr for skbs which completed the GRO receive phase (both ingro_listand*_gro_complete) when parsing an encapsulated packet's _outer_ L3/L4 may return an unexpected value.Quoted 8 lines
This incorrect usage leads to a bug in GRO's UDP socket lookup. udp{4,6}_lib_lookup_skb functions use ip_hdr/ipv6_hdr respectively. These
*_hdrfunctions returnnetwork_headerwhich will point to the innermost L3, resulting in the wrong offset being used in __udp{4,6}_lib_lookup with encapsulated packets.To fix this issue
p_offparam is used in*_gro_completeto pass off the offset of the previous layer.What exactly does this mean?
This patch changes the definition of
gro_completeto add a thoff alongside the existing "nhoff"..include/linux/netdevice.h struct offload_callbacks { · line 2714
Quoted 3 lines
include/linux/netdevice.h +2
-1-int (*gro_complete)(struct sk_buff *skb,int nhoff); +int (*gro_complete)(struct sk_buff *skb,int nhoff, +int thoff);.. but also fixes up implementations to interpret the existing argument as a thoff
net/ipv4/tcp_offload.c struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) · line 335
Quoted 8 lines
net/ipv4/tcp_offload.c +4
-3-INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int nhoff, +int thoff) { -const struct iphdr *iph =ip_hdr (skb); -struct tcphdr *th =tcp_hdr (skb); +const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); +struct tcphdr *th = (struct tcphdr *)(skb->data + thoff);But in some cases the new argument is not nhoff but
p_off, e.g.,Review
Quoted 3 lines
static int geneve_gro_complete(struct sock *sk, struct
sk_buff*skb,(quoted diff) +1
-1- int nhoff) + int p_off, int nhoff)Really, the argument is the start of the next header, each callback just casts to its expected header (ethhdr, tcphdr, etc.)
The only place where we need to pass an extra argument is in udp, because that needs a pointer to the network header right before the transport header pointed to by nhoff.
And only due to possible IPv4 options or IPv6 extension headers, we cannot just do
+ struct udphdr *iph = (struct iphdr *)(skb->data + nhoff - sizeof(*iph)); struct udphdr *uh = (struct udphdr *)(skb->data + nhoff);I also do not immediately see an a way to avoid all the boilerplate of a new argument in every callback. Aside from a
per_cpuvar -- but that is excessive.
Quoted 8 lines
Quoted 2 lines
But it can just be left zero in all callsites, except for inet_gro_complete/ipv6_gro_complete, which pass in nhoff.
Actually, we can avoid the boilerplate changes that add an extra arg.
By changing the contract between network layer callbacks (inet_gro_complete/ipv6_gro_complete) and transport layer callbacks (
tcp4_gro_completeet al).
Actually, only when calling udp4_gro_complete or udp6_gro_complete.
Quoted 1 line
It's also a bit of a hack. But a lot smaller patch, probably.
Feel free to disagree with this approach. Just a suggestion.
PATCH v6 3/6 selftests/net: add local address bind in vxlan selftest
In reply to Richard GobertAdd local address bind support to existing udpgro_fwd.sh vxlan selftest, to ensure UDP socket lookup in GRO is working.
--- tools/testing/selftests/net/udpgro_fwd.sh | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
tools/testing/selftests/net/udpgro_fwd.sh +7 -3
@@ -62,11 +62,13 @@ create_vxlan_endpoint() { local -r bm_rem_addr=$3 local -r vxlan_dev=$4 local -r vxlan_id=$5 + local -r bm_local_addr=$6 local -r vxlan_port=4789 ip -n $netns link set dev $bm_dev up ip -n $netns link add dev $vxlan_dev type vxlan id $vxlan_id \ - dstport $vxlan_port remote $bm_rem_addr + dstport $vxlan_port local $bm_local_addr \ + remote $bm_rem_addr ip -n $netns link set dev $vxlan_dev up }
@@ -77,11 +79,13 @@ create_vxlan_pair() { for ns in $SRC $DST; do # note that 3 - $SRC == $DST and 3 - $DST == $SRC - create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V4$((3 - $ns)) vxlan$ns 4 + create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V4$((3 - $ns)) \ + vxlan$ns 4 $BM_NET_V4$ns ip -n $BASE$ns addr add dev vxlan$ns $OL_NET_V4$ns/24 done for ns in $SRC $DST; do - create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V6$((3 - $ns)) vxlan6$ns 6 + create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V6$((3 - $ns)) \ + vxlan6$ns 6 $BM_NET_V6$ns ip -n $BASE$ns addr add dev vxlan6$ns $OL_NET_V6$ns/24 nodad done
Signature · 1 line
2.36.1
PATCH v6 4/6 net: gro: add {inner_}network_offset to napi_gro_cb
In reply to Richard GobertThis patch adds network_offset and inner_network_offset to napi_gro_cb, and makes sure both are set correctly. In the common path there's only one write (skb_gro_reset_offset).
--- drivers/net/geneve.c | 1 + drivers/net/vxlan/vxlan_core.c | 1 + include/net/gro.h | 18 ++++++++++++++++-- net/8021q/vlan_core.c | 2 ++ net/core/gro.c | 1 + net/ethernet/eth.c | 1 + net/ipv4/af_inet.c | 5 +---- net/ipv4/gre_offload.c | 1 + net/ipv6/ip6_offload.c | 8 ++++---- 9 files changed, 28 insertions(+), 10 deletions(-)
drivers/net/geneve.c +1 -0
@@ -545,6 +545,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk,if (!ptype) goto out; +NAPI_GRO_CB (skb)->inner_network_offset = hlen; pp =call_gro_receive (ptype->callbacks.gro_receive, head, skb); flush =0 ;
drivers/net/vxlan/vxlan_core.c +1 -0
@@ -754,6 +754,7 @@ static struct sk_buff *vxlan_gpe_gro_receive(struct sock *sk, vh =vxlan_gro_prepare_receive (sk, head, skb, &grc);if (vh) { +NAPI_GRO_CB (skb)->inner_network_offset =skb_gro_offset (skb);if (!vxlan_parse_gpe_proto (vh, &protocol)) goto out; ptype =gro_find_receive_by_type (protocol);
include/net/gro.h +16 -2
@@ -87,6 +87,15 @@ struct napi_gro_cb {/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; + +/* L3 offsets */ +union { +struct { +u16 network_offset; +u16 inner_network_offset; + }; +u16 network_offsets[2 ]; + }; };#define NAPI_GRO_CB (skb) ((struct napi_gro_cb *)(skb)->cb)
@@ -172,12 +181,17 @@ static inline void *skb_gro_header(struct sk_buff *skb, unsigned int hlen,return ptr; } +static inline int skb_gro_network_offset (const struct sk_buff *skb) +{ +return NAPI_GRO_CB (skb)->network_offsets[NAPI_GRO_CB (skb)->encap_mark]; +} +static inline void *skb_gro_network_header (const struct sk_buff *skb) {if (skb_gro_may_pull (skb,skb_gro_offset (skb))) -return skb_gro_header_fast (skb,skb_network_offset (skb)); +return skb_gro_header_fast (skb,skb_gro_network_offset (skb)); -return skb_network_header (skb); +return skb->data +skb_gro_network_offset (skb); } static inline __wsum inet_gro_compute_pseudo(const struct sk_buff *skb,
net/8021q/vlan_core.c +2 -0
@@ -478,6 +478,8 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head,if (unlikely (!vhdr)) goto out; +NAPI_GRO_CB (skb)->network_offsets[NAPI_GRO_CB (skb)->encap_mark] = hlen; + type = vhdr->h_vlan_encapsulated_proto; ptype =gro_find_receive_by_type (type);
net/core/gro.c +1 -0
@@ -371,6 +371,7 @@ static inline void skb_gro_reset_offset(struct sk_buff *skb, u32 nhoff)const skb_frag_t *frag0;unsigned int headlen; +NAPI_GRO_CB (skb)->network_offset =0 ;NAPI_GRO_CB (skb)->data_offset =0 ; headlen =skb_headlen (skb);NAPI_GRO_CB (skb)->frag0 = skb->data;
net/ethernet/eth.c +1 -0
@@ -441,6 +441,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb)skb_gro_pull (skb,sizeof (*eh));skb_gro_postpull_rcsum (skb, eh,sizeof (*eh)); +NAPI_GRO_CB (skb)->inner_network_offset = hlen; pp = indirect_call_gro_receive_inet(ptype->callbacks.gro_receive, ipv6_gro_receive, inet_gro_receive,
net/ipv4/af_inet.c +1 -4
@@ -1568,10 +1568,6 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)NAPI_GRO_CB (skb)->is_atomic = !!(iph->frag_off &htons (IP_DF ));NAPI_GRO_CB (skb)->flush |= flush; -skb_set_network_header (skb, off); -/* The above will be needed by the transport layer if there is one -* immediately following this IP hdr. -*/ /* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0
@@ -1597,6 +1593,7 @@ static struct sk_buff *ipip_gro_receive(struct list_head *head, }NAPI_GRO_CB (skb)->encap_mark =1 ; +NAPI_GRO_CB (skb)->inner_network_offset =skb_gro_offset (skb);return inet_gro_receive (head, skb); }
net/ipv4/gre_offload.c +1 -0
@@ -224,6 +224,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head,/* Adjusted NAPI_GRO_CB(skb)->csum after skb_gro_pull()*/ skb_gro_postpull_rcsum (skb, greh, grehlen); +NAPI_GRO_CB (skb)->inner_network_offset = hlen; pp =call_gro_receive (ptype->callbacks.gro_receive, head, skb); flush =0 ;
net/ipv6/ip6_offload.c +4 -4
@@ -67,7 +67,7 @@ static int ipv6_gro_pull_exthdrs(struct sk_buff *skb, int off, int proto) off += len; } -skb_gro_pull (skb, off -skb_network_offset (skb)); +skb_gro_pull (skb, off -skb_gro_network_offset (skb));return proto; }
@@ -236,8 +236,6 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head,if (unlikely (!iph)) goto out; -skb_set_network_header (skb, off); - flush +=ntohs (iph->payload_len) != skb->len - hlen; proto = iph->nexthdr;
@@ -259,7 +257,7 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head,NAPI_GRO_CB (skb)->proto = proto; flush--; - nlen =skb_network_header_len (skb); + nlen =skb_gro_offset (skb) - off;list_for_each_entry (p, head, list) {const struct ipv6hdr *iph2;
@@ -327,6 +325,7 @@ static struct sk_buff *sit_ip6ip6_gro_receive(struct list_head *head, }NAPI_GRO_CB (skb)->encap_mark =1 ; +NAPI_GRO_CB (skb)->inner_network_offset =skb_gro_offset (skb);return ipv6_gro_receive (head, skb); }
@@ -342,6 +341,7 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head, }NAPI_GRO_CB (skb)->encap_mark =1 ; +NAPI_GRO_CB (skb)->inner_network_offset =skb_gro_offset (skb);return inet_gro_receive (head, skb); }
Signature · 1 line
2.36.1
PATCH v6 5/6 net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment
In reply to Richard Gobert{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags, iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO.
These checks need to be done only once in tcp_gro_receive and only against the found p skb, since they only affect flush and not same_flow.
Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in tcp_gro_receive. As a result, NAPI_GRO_CB(p)->flush is not used at all. In addition, flush_id checks are more declarative and contained in inet_gro_flush, thus removing the need for flush_id in napi_gro_cb.
This results in less parsing code for UDP flows and non-loop flush tests for TCP flows.
To make sure results are not within noise range - I've made netfilter drop all TCP packets, and measured CPU performance in GRO (in this case GRO is responsible for about 50% of the CPU utilization). gro_network_flush is compiled inline to tcp_gro_receive.
perf top while replaying 64 parallel IP/TCP streams merging in GRO net-next:
6.94% [kernel] [k] inet_gro_receive
3.02% [kernel] [k] tcp_gro_receivepatch applied:
4.27% [kernel] [k] tcp_gro_receive
4.22% [kernel] [k] inet_gro_receiveperf top while replaying 64 parallel IP/IP/TCP streams merging in GRO (same results for any encapsulation, in this case inet_gro_receive is top offender in net-next)
net-next:
10.09% [kernel] [k] inet_gro_receive
2.08% [kernel] [k] tcp_gro_receivepatch applied:
6.97% [kernel] [k] inet_gro_receive
3.68% [kernel] [k] tcp_gro_receiveperf top -g while running 64 IP/UDP netperf connections NOT merging in GRO (udp_gro_receive is included because of -g, in this case GRO is just overhead)
net-next:
1.26% [kernel] [k] inet_gro_receive
patch applied:
0.85% [kernel] [k] inet_gro_receive
--- include/net/gro.h | 66 ++++++++++++++++++++++++++++++++++++++---- net/core/gro.c | 4 --- net/ipv4/af_inet.c | 41 +------------------------- net/ipv4/tcp_offload.c | 15 ++-------- net/ipv4/udp_offload.c | 16 +++------- net/ipv6/ip6_offload.c | 11 ------- 6 files changed, 67 insertions(+), 86 deletions(-)
include/net/gro.h +60 -6
@@ -36,15 +36,15 @@ struct napi_gro_cb {/* This is non-zero if the packet cannot be merged with the new skb. */ u16 flush; -/* Save the IP ID here and check when we get to the transport layer */ -u16 flush_id; -/* Number of segments aggregated. */ u16 count;/* Used in ipv6_gro_receive() and foo-over-udp and esp-in-udp */ u16 proto; +/* used to support CHECKSUM_COMPLETE for tunneling protocols */ +__wsum csum; +/* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1#define NAPI_GRO_FREE_STOLEN_HEAD 2
@@ -85,9 +85,6 @@ struct napi_gro_cb { u8 is_flist:1; ); -/* used to support CHECKSUM_COMPLETE for tunneling protocols */ -__wsum csum; -/* L3 offsets */ union { struct {
@@ -443,6 +440,63 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb, skb_gro_len(skb), proto, 0)); } +static inline int inet_gro_flush (const struct iphdr *iph,const struct iphdr *iph2, +struct sk_buff *p,bool outer) +{ +const u32 id =ntohl (*(__be32 *)&iph->id); +const u32 id2 =ntohl (*(__be32 *)&iph2->id); +const u16 flush_id = (id >>16 ) - (id2 >>16 ); +const u16 count =NAPI_GRO_CB (p)->count; +const u32 df = id &IP_DF ; +u32 is_atomic; +int flush; + +/* All fields must match except length and checksum. */ + flush = (iph->ttl ^ iph2->ttl) | (iph->tos ^ iph2->tos) | (df ^ (id2 &IP_DF )); + +if (outer && df) +return flush; + +/* When we receive our second frame we can make a decision on if we +* continue this flow as an atomic flow with a fixed ID or if we use +* an incrementing ID. +*/ +NAPI_GRO_CB (p)->is_atomic |= (count ==1 && df && flush_id ==0 ); + is_atomic = (df &&NAPI_GRO_CB (p)->is_atomic) -1 ; + +return flush | (flush_id ^ (count & is_atomic)); +} + +static inline int ipv6_gro_flush (const struct ipv6hdr *iph,const struct ipv6hdr *iph2) +{ +/* <Version:4><Traffic_Class:8><Flow_Label:20> */ +__be32 first_word = *(__be32 *)iph ^ *(__be32 *)iph2; + +/* Flush if Traffic Class fields are different. */ +return !!((first_word &htonl (0x0FF00000 )) | + (__force __be32)(iph->hop_limit ^ iph2->hop_limit)); +} + +static inline int gro_network_flush (const void *th,const void *th2,struct sk_buff *p,int off) +{ +const bool encap_mark =NAPI_GRO_CB (p)->encap_mark; +int flush =0 ; +int i; + +for (i =0 ; i <= encap_mark; i++) { +const u16 diff = off -NAPI_GRO_CB (p)->network_offsets[i]; +const void *nh = th - diff; +const void *nh2 = th2 - diff; + +if (((struct iphdr *)nh)->version ==6 ) + flush |=ipv6_gro_flush (nh, nh2); +else + flush |=inet_gro_flush (nh, nh2, p, i != encap_mark); + } + +return flush; +} +int skb_gro_receive (struct sk_buff *p,struct sk_buff *skb);/* Pass the currently batched GRO_NORMAL SKBs up to the stack. */
net/core/gro.c +0 -4
@@ -89,7 +89,6 @@ void dev_remove_offload(struct packet_offload *po) }EXPORT_SYMBOL (dev_remove_offload); -int skb_gro_receive (struct sk_buff *p,struct sk_buff *skb) {struct skb_shared_info *pinfo, *skbinfo =skb_shinfo (skb);
@@ -330,8 +329,6 @@ static void gro_list_prepare(const struct list_head *head,list_for_each_entry (p, head, list) {unsigned long diffs; -NAPI_GRO_CB (p)->flush =0 ; -if (hash !=skb_get_hash_raw (p)) {NAPI_GRO_CB (p)->same_flow =0 ;continue ;
@@ -471,7 +468,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff sizeof(u32))); /* Avoid slow unaligned acc */ *(u32 *)&NAPI_GRO_CB (skb)->zeroed =0 ;NAPI_GRO_CB (skb)->flush =skb_has_frag_list (skb); -NAPI_GRO_CB (skb)->is_atomic =1 ;NAPI_GRO_CB (skb)->count =1 ;if (unlikely (skb_is_gso (skb))) {NAPI_GRO_CB (skb)->count =skb_shinfo (skb)->gso_segs;
net/ipv4/af_inet.c +1 -40
64 lines not on this page · raw message
net/ipv4/tcp_offload.c +2 -13
@@ -232,9 +232,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) goto out_check_final; found: -/* Include the IP ID check below from the inner most IP hdr */ - flush =NAPI_GRO_CB (p)->flush; - flush |= (__force int )(flags &TCP_FLAG_CWR ); + flush = (__force int )(flags &TCP_FLAG_CWR ); flush |= (__force int )((flags ^tcp_flag_word (th2)) & ~(TCP_FLAG_CWR |TCP_FLAG_FIN |TCP_FLAG_PSH )); flush |= (__force int )(th->ack_seq ^ th2->ack_seq);
@@ -242,16 +240,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) flush |= *(u32 *)((u8 *)th + i) ^ *(u32 *)((u8 *)th2 + i); -/* When we receive our second frame we can made a decision on if we -* continue this flow as an atomic flow with a fixed ID or if we use -* an incrementing ID. -*/ -if (NAPI_GRO_CB (p)->flush_id !=1 || -NAPI_GRO_CB (p)->count !=1 || - !NAPI_GRO_CB (p)->is_atomic) - flush |=NAPI_GRO_CB (p)->flush_id; -else -NAPI_GRO_CB (p)->is_atomic = false; + flush |=gro_network_flush (th, th2, p, off); mss =skb_shinfo (p)->gso_size;
net/ipv4/udp_offload.c +4 -12
@@ -466,12 +466,12 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head, struct sk_buff *skb) {struct udphdr *uh =udp_gro_udphdr (skb); +int off =skb_gro_offset (skb);struct sk_buff *pp =NULL ;struct udphdr *uh2;struct sk_buff *p;unsigned int ulen;int ret =0 ; -int flush;/* requires non zero csum, for symmetry with GSO */ if (!uh->check) {
@@ -529,17 +529,9 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head,skb_gro_postpull_rcsum (skb, uh,sizeof (struct udphdr )); - flush =NAPI_GRO_CB (p)->flush; - -if (NAPI_GRO_CB (p)->flush_id !=1 || -NAPI_GRO_CB (p)->count !=1 || - !NAPI_GRO_CB (p)->is_atomic) - flush |=NAPI_GRO_CB (p)->flush_id; -else -NAPI_GRO_CB (p)->is_atomic = false; - -if (flush ||skb_gro_receive (p, skb)) - ret =1 ; + ret =gro_network_flush (uh, uh2, p, off); +if (!ret) + ret =skb_gro_receive (p, skb); } }
net/ipv6/ip6_offload.c +0 -11
@@ -288,19 +288,8 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, nlen - sizeof(struct ipv6hdr))) goto not_same_flow; } -/* flush if Traffic Class fields are different */ -NAPI_GRO_CB (p)->flush |= !!((first_word &htonl (0x0FF00000 )) | - (__force __be32)(iph->hop_limit ^ iph2->hop_limit)); -NAPI_GRO_CB (p)->flush |= flush; - -/* If the previous IP ID value was based on an atomic -* datagram we can overwrite the value and ignore it. -*/ -if (NAPI_GRO_CB (skb)->is_atomic) -NAPI_GRO_CB (p)->flush_id =0 ; } -NAPI_GRO_CB (skb)->is_atomic = true;NAPI_GRO_CB (skb)->flush |= flush;skb_gro_postpull_rcsum (skb, iph, nlen);
Signature · 1 line
2.36.1
Willem de Bruijn
maintainerRe: [PATCH net-next v6 5/6] net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment
Quoted 6 lines · Richard Gobert
{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags,
iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO.These checks need to be done only once in
tcp_gro_receiveand only against the found p skb, since they only affect flush and notsame_flow.
I don't quite understand where the performance improvements arise. As inet_gro_receive will skip any p that does not match:
if (!NAPI_GRO_CB(p)->same_flow)
continue;
iph2 = (struct iphdr *)(p->data + off);
/* The above works because, with the exception of the top
* (inner most) layer, we only aggregate pkts with the same
* hdr length so all the hdrs we'll need to verify will start
* at the same offset.
*/
if ((iph->protocol ^ iph2->protocol) |
((__force u32)iph->saddr ^ (__force u32)iph2->saddr) |
((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) {
NAPI_GRO_CB(p)->same_flow = 0;
continue;
}So these checks are already only performed against a p that matches.
Quoted 3 lines
Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in
tcp_gro_receive. As a
Comments should be updated to reflect both TCP and L4 UDP. Can generalize to transport callbacks.
Quoted 6 lines
result, NAPI_GRO_CB(p)->flush is not used at all. In addition,
flush_idchecks are more declarative and contained ininet_gro_flush, thus removing the need forflush_idinnapi_gro_cb.This results in less parsing code for UDP flows and non-loop flush tests for TCP flows.
This moves network layer tests out of the network layer callbacks into helpers called from the transport layer callback. And then the helper has to look up the network layer header and demultiplex the protocol again:
+ if (((struct iphdr *)nh)->version == 6)
+ flush |= ipv6_gro_flush(nh, nh2);
+ else
+ flush |= inet_gro_flush(nh, nh2, p, i != encap_mark);That just seems a bit roundabout.
Re: [PATCH net-next v6 5/6] net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment
Quoted 27 lines · Willem de Bruijn
Quoted 6 lines · Richard Gobert
{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags,
iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO.These checks need to be done only once in
tcp_gro_receiveand only against the found p skb, since they only affect flush and notsame_flow.I don't quite understand where the performance improvements arise. As
inet_gro_receivewill skip any p that does not match:if (!NAPI_GRO_CB(p)->same_flow) continue; iph2 = (struct iphdr *)(p->data + off); /* The above works because, with the exception of the top * (inner most) layer, we only aggregate pkts with the same * hdr length so all the hdrs we'll need to verify will start * at the same offset. */ if ((iph->protocol ^ iph2->protocol) | ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) | ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) { NAPI_GRO_CB(p)->same_flow = 0; continue; }
Quoted 2 lines
So these checks are already only performed against a p that matches.
Thanks for the review!
flush/flush_id is calculated for all other p with same_flow = 1 (which is not always determined to be 0 before inet_gro_receive) and same src/dst addr in the bucket. Moving it to udp_gro_receive_segment/tcp_gro_receive will make it run only once when a matching p is found.
In addition, UDP flows where skb_gro_receive_list is called - flush/flush_id is not relevant and does not need to be calculated. In these cases total CPU time in GRO should drop. I could post perf numbers for this flow as well.
Quoted 19 lines
Quoted 3 lines
Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in
tcp_gro_receive. As aComments should be updated to reflect both TCP and L4 UDP. Can generalize to transport callbacks.
Quoted 6 lines
result, NAPI_GRO_CB(p)->flush is not used at all. In addition,
flush_idchecks are more declarative and contained ininet_gro_flush, thus removing the need forflush_idinnapi_gro_cb.This results in less parsing code for UDP flows and non-loop flush tests for TCP flows.
This moves network layer tests out of the network layer callbacks into helpers called from the transport layer callback. And then the helper has to look up the network layer header and demultiplex the protocol again:
Quoted 6 lines
+ if (((struct iphdr *)nh)->version == 6) + flush |= ipv6_gro_flush(nh, nh2); + else + flush |= inet_gro_flush(nh, nh2, p, i != encap_mark);That just seems a bit roundabout.
IMO this commit could be a part of a larger change, where all loops in gro_list_prepare, inet_gro_receive and ipv6_gro_receive can be removed, and the logic for finding a matching p will be moved to L4. This means that when p is found, the rest of the gro_list would not need to be traversed and thus would not even dirty cache lines at all. I can provide a code snippet which would explain it better.
Willem de Bruijn
maintainerRe: [PATCH net-next v6 5/6] net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment
Quoted 32 lines · Richard Gobert
Quoted 27 lines · Willem de Bruijn
Quoted 6 lines · Richard Gobert
{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags,
iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO.These checks need to be done only once in
tcp_gro_receiveand only against the found p skb, since they only affect flush and notsame_flow.I don't quite understand where the performance improvements arise. As
inet_gro_receivewill skip any p that does not match:if (!NAPI_GRO_CB(p)->same_flow) continue; iph2 = (struct iphdr *)(p->data + off); /* The above works because, with the exception of the top * (inner most) layer, we only aggregate pkts with the same * hdr length so all the hdrs we'll need to verify will start * at the same offset. */ if ((iph->protocol ^ iph2->protocol) | ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) | ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) { NAPI_GRO_CB(p)->same_flow = 0; continue; }Quoted 2 lines
So these checks are already only performed against a p that matches.
Quoted 6 lines
Thanks for the review!
flush/flush_id is calculated for all other p with
same_flow= 1 (which is not always determined to be 0 beforeinet_gro_receive) and same src/dst addr in the bucket. Moving it to udp_gro_receive_segment/tcp_gro_receive will make it run only once when a matching p is found.
So this optimization is for flows that are the same up to having the same saddr/daddr. Aside from stress tests, it seems rare to have many concurrent flows between the same pair of machines?
Quoted 3 lines
In addition, UDP flows where
skb_gro_receive_listis called - flush/flush_id is not relevant and does not need to be calculated.
That makes sense
Quoted 29 lines
In these
cases total CPU time in GRO should drop. I could post perf numbers for this flow as well.Quoted 24 lines
Quoted 3 lines
Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in
tcp_gro_receive. As aComments should be updated to reflect both TCP and L4 UDP. Can generalize to transport callbacks.
Quoted 6 lines
result, NAPI_GRO_CB(p)->flush is not used at all. In addition,
flush_idchecks are more declarative and contained ininet_gro_flush, thus removing the need forflush_idinnapi_gro_cb.This results in less parsing code for UDP flows and non-loop flush tests for TCP flows.
This moves network layer tests out of the network layer callbacks into helpers called from the transport layer callback. And then the helper has to look up the network layer header and demultiplex the protocol again:
+ if (((struct iphdr *)nh)->version == 6) + flush |= ipv6_gro_flush(nh, nh2); + else + flush |= inet_gro_flush(nh, nh2, p, i != encap_mark);
Quoted 8 lines
Quoted 1 line
That just seems a bit roundabout.
IMO this commit could be a part of a larger change, where all loops in
gro_list_prepare,inet_gro_receiveandipv6_gro_receivecan be removed, and the logic for finding a matching p will be moved to L4. This means that when p is found, the rest of thegro_listwould not need to be traversed and thus would not even dirty cache lines at all. I can provide a code snippet which would explain it better.
These loops are exactly the mechanism to find a matching p. Though with all the callbacks perhaps not the most efficient model. The hashtable should have solved much of that.
Yes, please share a snippet to understand how you would replace this.
In the meantime, I do suggest sending the first two patches to net, as they have Fixes tags. And then follow up with this for net-next separately.
Re: [PATCH net-next v6 5/6] net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment
Quoted 35 lines · Willem de Bruijn
Quoted 34 lines · Richard Gobert
Quoted 27 lines · Willem de Bruijn
Quoted 6 lines · Richard Gobert
{inet,ipv6}_gro_receive functions perform flush checks (ttl, flags,
iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO.These checks need to be done only once in
tcp_gro_receiveand only against the found p skb, since they only affect flush and notsame_flow.I don't quite understand where the performance improvements arise. As
inet_gro_receivewill skip any p that does not match:if (!NAPI_GRO_CB(p)->same_flow) continue; iph2 = (struct iphdr *)(p->data + off); /* The above works because, with the exception of the top * (inner most) layer, we only aggregate pkts with the same * hdr length so all the hdrs we'll need to verify will start * at the same offset. */ if ((iph->protocol ^ iph2->protocol) | ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) | ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) { NAPI_GRO_CB(p)->same_flow = 0; continue; }Quoted 2 lines
So these checks are already only performed against a p that matches.
Thanks for the review!
Quoted 9 lines
Quoted 4 lines
flush/flush_id is calculated for all other p with
same_flow= 1 (which is not always determined to be 0 beforeinet_gro_receive) and same src/dst addr in the bucket. Moving it to udp_gro_receive_segment/tcp_gro_receive will make it run only once when a matching p is found.So this optimization is for flows that are the same up to having the same saddr/daddr. Aside from stress tests, it seems rare to have many concurrent flows between the same pair of machines?
Yes exactly, sorry if I wasn't clear enough earlier. Multiple connections with the same srcaddr+dstaddr are not necessarily rare (e.g. devices behind a large NAT network all connecting to the same servers, thus sharing the same IP srcaddr).
Quoted 6 lines
Quoted 3 lines
In addition, UDP flows where
skb_gro_receive_listis called - flush/flush_id is not relevant and does not need to be calculated.That makes sense
I ran a UDP forwarding benchmark similar to how I did in the commit message. GRO's flush/flush_id values were not relevant as all packets reached skb_gro_receive_list instead of skb_gro_receive.
These numbers show CPU utilization under the same load (64 concurrent IP/UDP connections):
net-next:
3.03% [kernel] [k] inet_gro_receive
patch applied:
2.78% [kernel] [k] inet_gro_receive
And under encapsulated load, 64 concurrent IP/IP/UDP connections:
net-next:
10.50% [kernel] [k] inet_gro_receive
patch applied:
8.19% [kernel] [k] inet_gro_receive
Total time spent in GRO reduced significantly due to fewer opcodes and branches in inet_gro_receive :)
Quoted 38 lines
Quoted 29 lines
In these
cases total CPU time in GRO should drop. I could post perf numbers for this flow as well.Quoted 24 lines
Quoted 3 lines
Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in
tcp_gro_receive. As aComments should be updated to reflect both TCP and L4 UDP. Can generalize to transport callbacks.
Quoted 6 lines
result, NAPI_GRO_CB(p)->flush is not used at all. In addition,
flush_idchecks are more declarative and contained ininet_gro_flush, thus removing the need forflush_idinnapi_gro_cb.This results in less parsing code for UDP flows and non-loop flush tests for TCP flows.
This moves network layer tests out of the network layer callbacks into helpers called from the transport layer callback. And then the helper has to look up the network layer header and demultiplex the protocol again:
+ if (((struct iphdr *)nh)->version == 6) + flush |= ipv6_gro_flush(nh, nh2); + else + flush |= inet_gro_flush(nh, nh2, p, i != encap_mark);Quoted 8 lines
Quoted 1 line
That just seems a bit roundabout.
IMO this commit could be a part of a larger change, where all loops in
gro_list_prepare,inet_gro_receiveandipv6_gro_receivecan be removed, and the logic for finding a matching p will be moved to L4. This means that when p is found, the rest of thegro_listwould not need to be traversed and thus would not even dirty cache lines at all. I can provide a code snippet which would explain it better.
Quoted 6 lines
These loops are exactly the mechanism to find a matching p. Though with all the callbacks perhaps not the most efficient model. The hashtable should have solved much of that.
Yes, please share a snippet to understand how you would replace this.
This is still a rough idea, and I still think the current patch is significant by itself due to its performance gains and more readable code.
The idea is that an skb from gro_list will be loaded to cache only when GRO is trying to find out if it matches the current skb, avoiding the current multiple list traversals design in GRO for IP-TCP.
For a simple IP-TCP packet, gro_list_prapare will not be called from dev_gro_receive anymore:
@@ -450,8 +489,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
if (netif_elide_gro(skb->dev))
goto normal;- gro_list_prepare(&gro_list->list, skb);
The matching loop from inet_gro_receive is removed as well:
@@ -1507,27 +1506,6 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
NAPI_GRO_CB(skb)->proto = proto;
flush = (u16)((ntohl(*(__be32 *)iph) ^ skb_gro_len(skb)) | (ntohl(*(__be32 *)&iph->id) & ~IP_DF));-
- list_for_each_entry(p, head, list) {
- struct iphdr *iph2;
-
- if (!NAPI_GRO_CB(p)->same_flow)
- continue;
-
- iph2 = (struct iphdr *)(p->data + off);
- /* The above works because, with the exception of the top
- * (inner most) layer, we only aggregate pkts with the same - * hdr length so all the hdrs we'll need to verify will start - * at the same offset. - */
- if ((iph->protocol ^ iph2->protocol) |
- ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) |
- ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) {
- NAPI_GRO_CB(p)->same_flow = 0;
- continue;
- }
- }
-
and change tcp_gro_receive such that p is retrieved from gro_list using a new function:
net/ipv4/tcp_offload.c +4 -16
@@ -215,24 +215,12 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) len =skb_gro_len (skb); flags =tcp_flag_word (th); -list_for_each_entry (p, head, list) { -if (!NAPI_GRO_CB (p)->same_flow) -continue ; - - th2 =tcp_hdr (p); - -if (*(u32 *)&th->source ^ *(u32 *)&th2->source) { -NAPI_GRO_CB (p)->same_flow =0 ; -continue ; - } +if (!(p =gro_list_find (head, skb, &flush))) + goto out_check_final; - goto found; - } - p =NULL ; - goto out_check_final; + th2 =tcp_hdr (p); -found: - flush = (__force int )(flags &TCP_FLAG_CWR ); + flush |= (__force int )(flags &TCP_FLAG_CWR ); flush |= (__force int )((flags ^tcp_flag_word (th2)) & ~(TCP_FLAG_CWR |TCP_FLAG_FIN |TCP_FLAG_PSH )); flush |= (__force int )(th->ack_seq ^ th2->ack_seq);
The first time gro_list is traversed is in tcp_gro_receive. gro_list_find is very similar to gro_list_prepare - but it also matches L3 & L4 (UDP and TCP port offsets are identical), and it returns L3 flush value as well.
The main difference between the current design and what I propose is that once a matched p is found - there's no need to keep traversing the rest of gro_list (as opposed to gro_list_prepare and inet_gro_receive for example), and that's why certain checks are saved especially under load. This way less data is loaded to cache overall for every skb entering dev_gro_receive.
Quoted 3 lines
In the meantime, I do suggest sending the first two patches to net, as they have Fixes tags. And then follow up with this for net-next separately.
Just submitted the first two commits to net.
Thanks.
PATCH v6 6/6 selftests/net: add flush id selftests
In reply to Richard GobertAdded flush id selftests to test different cases where DF flag is set or unset and id value changes in the following packets. All cases where the packets should coalesce or should not coalesce are tested.
--- tools/testing/selftests/net/gro.c | 144 ++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+)
tools/testing/selftests/net/gro.c +144 -0
@@ -617,6 +617,120 @@ static void add_ipv6_exthdr(void *buf, void *optpkt, __u8 exthdr_type, char *ext iph->payload_len =htons (ntohs (iph->payload_len) +MIN_EXTHDR_SIZE ); } +static void fix_ip4_checksum (struct iphdr *iph) +{ + iph->check =0 ; + iph->check =checksum_fold (iph,sizeof (struct iphdr ),0 ); +} + +static void send_flush_id_case (int fd,struct sockaddr_ll *daddr,int tcase) +{ +bool send_three = false; +static char buf1[MAX_HDR_LEN +PAYLOAD_LEN ]; +static char buf2[MAX_HDR_LEN +PAYLOAD_LEN ]; +static char buf3[MAX_HDR_LEN +PAYLOAD_LEN ]; + +create_packet (buf1,0 ,0 ,PAYLOAD_LEN ,0 ); +create_packet (buf2,PAYLOAD_LEN ,0 ,PAYLOAD_LEN ,0 ); +create_packet (buf3,PAYLOAD_LEN *2 ,0 ,PAYLOAD_LEN ,0 ); + +struct iphdr *iph1 = (struct iphdr *)(buf1 +ETH_HLEN ); +struct iphdr *iph2 = (struct iphdr *)(buf2 +ETH_HLEN ); +struct iphdr *iph3 = (struct iphdr *)(buf3 +ETH_HLEN ); + +switch (tcase) { +case 0 :/* DF=1, Incrementing - should coalesce */ + iph1->frag_off |=htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off |=htons (IP_DF ); + iph2->id =htons (9 ); +fix_ip4_checksum (iph2); +break ; + +case 1 :/* DF=1, Fixed - should coalesce */ + iph1->frag_off |=htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off |=htons (IP_DF ); + iph2->id =htons (8 ); +fix_ip4_checksum (iph2); +break ; + +case 2 :/* DF=0, Incrementing - should coalesce */ + iph1->frag_off &= ~htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off &= ~htons (IP_DF ); + iph2->id =htons (9 ); +fix_ip4_checksum (iph2); +break ; + +case 3 :/* DF=0, Fixed - should not coalesce */ + iph1->frag_off &= ~htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off &= ~htons (IP_DF ); + iph2->id =htons (8 ); +fix_ip4_checksum (iph2); +break ; + +case 4 :/* DF=1, two packets incrementing, and one fixed - should +* coalesce only the first two packets +*/ + iph1->frag_off |=htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off |=htons (IP_DF ); + iph2->id =htons (9 ); +fix_ip4_checksum (iph2); + + iph3->frag_off |=htons (IP_DF ); + iph3->id =htons (9 ); +fix_ip4_checksum (iph3); + send_three = true; +break ; + +case 5 :/* DF=1, two packets fixed, and one incrementing - should +* coalesce only the first two packets +*/ + iph1->frag_off |=htons (IP_DF ); + iph1->id =htons (8 ); +fix_ip4_checksum (iph1); + + iph2->frag_off |=htons (IP_DF ); + iph2->id =htons (8 ); +fix_ip4_checksum (iph2); + + iph3->frag_off |=htons (IP_DF ); + iph3->id =htons (9 ); +fix_ip4_checksum (iph3); + send_three = true; +break ; + } + +write_packet (fd, buf1, total_hdr_len +PAYLOAD_LEN , daddr); +write_packet (fd, buf2, total_hdr_len +PAYLOAD_LEN , daddr); + +if (send_three) +write_packet (fd, buf3, total_hdr_len +PAYLOAD_LEN , daddr); +} + +static void test_flush_id (int fd,struct sockaddr_ll *daddr,char *fin_pkt) +{ +for (int i =0 ; i <6 ; i++) { +sleep (1 ); +send_flush_id_case (fd, daddr, i); +sleep (1 ); +write_packet (fd, fin_pkt, total_hdr_len, daddr); + } +} +static void send_ipv6_exthdr (int fd,struct sockaddr_ll *daddr,char *ext_data1,char *ext_data2) {static char buf[MAX_HDR_LEN +PAYLOAD_LEN ];
@@ -935,6 +1049,8 @@ static void gro_sender(void)send_fragment4 (txfd, &daddr);sleep (1 );write_packet (txfd, fin_pkt, total_hdr_len, &daddr); + +test_flush_id (txfd, &daddr, fin_pkt); }else if (proto ==PF_INET6 ) {sleep (1 );send_fragment6 (txfd, &daddr);
@@ -1061,6 +1177,34 @@ static void gro_receiver(void)printf ("fragmented ip4 doesn't coalesce: " );check_recv_pkts (rxfd, correct_payload,2 ); + +/* is_atomic checks */ +printf ("DF=1, Incrementing - should coalesce: " ); + correct_payload[0 ] =PAYLOAD_LEN *2 ; +check_recv_pkts (rxfd, correct_payload,1 ); + +printf ("DF=1, Fixed - should coalesce: " ); + correct_payload[0 ] =PAYLOAD_LEN *2 ; +check_recv_pkts (rxfd, correct_payload,1 ); + +printf ("DF=0, Incrementing - should coalesce: " ); + correct_payload[0 ] =PAYLOAD_LEN *2 ; +check_recv_pkts (rxfd, correct_payload,1 ); + +printf ("DF=0, Fixed - should not coalesce: " ); + correct_payload[0 ] =PAYLOAD_LEN ; + correct_payload[1 ] =PAYLOAD_LEN ; +check_recv_pkts (rxfd, correct_payload,2 ); + +printf ("DF=1, 2 Incrementing and one fixed - should coalesce only first 2 packets: " ); + correct_payload[0 ] =PAYLOAD_LEN *2 ; + correct_payload[1 ] =PAYLOAD_LEN ; +check_recv_pkts (rxfd, correct_payload,2 ); + +printf ("DF=1, 2 Fixed and one incrementing - should coalesce only first 2 packets: " ); + correct_payload[0 ] =PAYLOAD_LEN *2 ; + correct_payload[1 ] =PAYLOAD_LEN ; +check_recv_pkts (rxfd, correct_payload,2 ); }else if (proto ==PF_INET6 ) {/* GRO doesn't check for ipv6 hop limit when flushing. * Hence no corresponding test to the ipv4 case.
Signature · 1 line
2.36.1