gro: optimise redundant parsing of packets
Subsystems (3)
cover gro: optimise redundant parsing of packets
Currently, the IPv6 extension headers are parsed twice: first in ipv6_gro_receive, and then again in ipv6_gro_complete.
The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time in ipv6_gro_complete.
The first commit frees up space in the GRO CB. The second commit reduces the redundant parsing during the complete phase, using the freed CB space.
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
Richard Gobert (2):
gro: decrease size of CB
gro: optimise redundant parsing of packets
include/net/gro.h | 32 +++++++++++++++++++++----------- net/core/gro.c | 18 +++++++++++------- net/ethernet/eth.c | 11 +++++++++-- net/ipv4/af_inet.c | 8 +++++++- net/ipv6/ip6_offload.c | 15 ++++++++++++--- 5 files changed, 60 insertions(+), 24 deletions(-)
Signature · 1 line
2.36.1
PATCH 1/2 gro: decrease size of CB
The GRO control block (NAPI_GRO_CB) is currently at its maximum size. This commit reduces its size by putting two groups of fields that are used only at different times into a union.
Specifically, the fields frag0 and frag0_len are the fields that make up the frag0 optimisation mechanism, which is used during the initial parsing of the SKB.
The fields last and age are used after the initial parsing, while the SKB is stored in the GRO list, waiting for other packets to arrive.
There was one location in dev_gro_receive that modified the frag0 fields after setting last and age. I changed this accordingly without altering the code behaviour.
--- include/net/gro.h | 26 ++++++++++++++++---------- net/core/gro.c | 18 +++++++++++------- 2 files changed, 27 insertions(+), 17 deletions(-)
include/net/gro.h +16 -10
@@ -11,11 +11,23 @@#include <net/udp.h> struct napi_gro_cb { -/* Virtual address of skb_shinfo(skb)->frags[0].page + offset. */ -void *frag0; +union { +struct { +/* Virtual address of skb_shinfo(skb)->frags[0].page + offset. */ +void *frag0; -/* Length of frag0. */ -unsigned int frag0_len; +/* Length of frag0. */ +unsigned int frag0_len; + }; + +struct { +/* used in skb_gro_receive() slow path */ +struct sk_buff *last; + +/* jiffies when first packet was created/queued */ +unsigned long age; + }; + };/* This indicates where we are processing relative to skb->data. */ int data_offset;
@@ -32,9 +44,6 @@ struct napi_gro_cb {/* Used in ipv6_gro_receive() and foo-over-udp */ u16 proto; -/* jiffies when first packet was created/queued */ -unsigned long age; -/* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1#define NAPI_GRO_FREE_STOLEN_HEAD 2
@@ -77,9 +86,6 @@ struct napi_gro_cb {/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; - -/* used in skb_gro_receive() slow path */ -struct sk_buff *last; };#define NAPI_GRO_CB (skb) ((struct napi_gro_cb *)(skb)->cb)
net/core/gro.c +11 -7
@@ -449,6 +449,14 @@ static void gro_pull_from_frag0(struct sk_buff *skb, int grow) } } +static inline void gro_try_pull_from_frag0 (struct sk_buff *skb) +{ +int grow =skb_gro_offset (skb) -skb_headlen (skb); + +if (grow >0 ) +gro_pull_from_frag0 (skb, grow); +} +static void gro_flush_oldest (struct napi_struct *napi,struct list_head *head) {struct sk_buff *oldest;
@@ -478,7 +486,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buffstruct sk_buff *pp =NULL ;enum gro_result ret;int same_flow; -int grow;if (netif_elide_gro (skb->dev)) goto normal;
@@ -553,17 +560,13 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buffelse gro_list->count++; +gro_try_pull_from_frag0 (skb);NAPI_GRO_CB (skb)->age = jiffies;NAPI_GRO_CB (skb)->last = skb;if (!skb_is_gso (skb))skb_shinfo (skb)->gso_size =skb_gro_len (skb);list_add (&skb->list, &gro_list->list); ret =GRO_HELD ; - -pull: - grow =skb_gro_offset (skb) -skb_headlen (skb); -if (grow >0 ) -gro_pull_from_frag0 (skb, grow); ok:if (gro_list->count) {if (!test_bit (bucket, &napi->gro_bitmask))
@@ -576,7 +579,8 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff normal: ret =GRO_NORMAL ; - goto pull; +gro_try_pull_from_frag0 (skb); + goto ok; }struct packet_offload *gro_find_receive_by_type (__be16 type)
Signature · 1 line
2.36.1
PATCH 2/2 gro: optimise redundant parsing of packets
In reply to Richard GobertCurrently, the IPv6 extension headers are parsed twice: first in ipv6_gro_receive, and then again in ipv6_gro_complete.
The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time in ipv6_gro_complete.
The implementation had to handle both inner and outer layers in case of encapsulation (as they can't use the same field).
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
--- include/net/gro.h | 8 ++++++-- net/ethernet/eth.c | 11 +++++++++-- net/ipv4/af_inet.c | 8 +++++++- net/ipv6/ip6_offload.c | 15 ++++++++++++--- 4 files changed, 34 insertions(+), 8 deletions(-)
include/net/gro.h +6 -2
@@ -41,8 +41,8 @@ struct napi_gro_cb {/* Number of segments aggregated. */ u16 count; -/* Used in ipv6_gro_receive() and foo-over-udp */ -u16 proto; +/* Used in eth_gro_receive() */ +__be16 network_proto;/* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1
@@ -86,6 +86,10 @@ struct napi_gro_cb {/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; + +/* Used in inet and ipv6 _gro_receive() */ +u16 network_len; +u8 transport_proto; };#define NAPI_GRO_CB (skb) ((struct napi_gro_cb *)(skb)->cb)
net/ethernet/eth.c +9 -2
@@ -439,6 +439,9 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) goto out; } +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->network_proto = type; +skb_gro_pull (skb,sizeof (*eh));skb_gro_postpull_rcsum (skb, eh,sizeof (*eh));
@@ -456,12 +459,16 @@ EXPORT_SYMBOL(eth_gro_receive);int eth_gro_complete (struct sk_buff *skb,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff); -__be16 type = eh->h_proto; +__be16 type;struct packet_offload *ptype;int err = -ENOSYS ; -if (skb->encapsulation) +if (skb->encapsulation) {skb_set_inner_mac_header (skb, nhoff); + type = eh->h_proto; + }else { + type =NAPI_GRO_CB (skb)->network_proto; + } ptype =gro_find_complete_by_type (type);if (ptype !=NULL )
net/ipv4/af_inet.c +7 -1
@@ -1551,6 +1551,9 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)* immediately following this IP hdr. */ +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->transport_proto = proto; +/* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0 */
@@ -1621,12 +1624,15 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)__be16 newlen =htons (skb->len - nhoff);struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);const struct net_offload *ops; -int proto = iph->protocol; +int proto;int err = -ENOSYS ;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IP ));skb_set_inner_network_header (skb, nhoff); + proto = iph->protocol; + }else { + proto =NAPI_GRO_CB (skb)->transport_proto; }csum_replace2 (&iph->check, iph->tot_len, newlen);
net/ipv6/ip6_offload.c +12 -3
@@ -227,11 +227,14 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, iph =ipv6_hdr (skb); } -NAPI_GRO_CB (skb)->proto = proto; - flush--; nlen =skb_network_header_len (skb); +if (!NAPI_GRO_CB (skb)->encap_mark) { +NAPI_GRO_CB (skb)->transport_proto = proto; +NAPI_GRO_CB (skb)->network_len = nlen; + } +list_for_each_entry (p, head, list) {const struct ipv6hdr *iph2;__be32 first_word;/* <Version:4><Traffic_Class:8><Flow_Label:20> */
@@ -358,7 +361,13 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); } - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (!skb->encapsulation) { + ops =rcu_dereference (inet6_offloads[NAPI_GRO_CB (skb)->transport_proto]); + nhoff +=NAPI_GRO_CB (skb)->network_len; + }else { + nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); + } +if (WARN_ON (!ops || !ops->callbacks.gro_complete)) goto out;
Signature · 1 line
2.36.1
From: Richard Gobert <[email protected]>
Date: Mon, 30 Jan 2023 14:07:55 +0100
Quoted 9 lines
Currently, the IPv6 extension headers are parsed twice: first in
ipv6_gro_receive, and then again inipv6_gro_complete.The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in
ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time inipv6_gro_complete.
Quoted 6 lines
The implementation had to handle both inner and outer layers in case of encapsulation (as they can't use the same field).
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
Would be nice to see some perf numbers. "there was an improvement" doesn't say a lot TBH...
Quoted 8 lines
Signed-off-by Richard Gobert <[email protected]>--- include/net/gro.h | 8 ++++++-- net/ethernet/eth.c | 11 +++++++++-- net/ipv4/af_inet.c | 8 +++++++- net/ipv6/ip6_offload.c | 15 ++++++++++++--- 4 files changed, 34 insertions(+), 8 deletions(-)
[...]
net/ethernet/eth.c EXPORT_SYMBOL(eth_gro_receive); · line 462
Quoted 6 lines
net/ethernet/eth.c +1
-1@@ -456,12 +459,16 @@ EXPORT_SYMBOL(eth_gro_receive);int eth_gro_complete (struct sk_buff *skb,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff); -__be16 type = eh->h_proto; +__be16 type;
Please don't break RCT style when shortening/expanding variable declaration lines.
Quoted 28 lines
net/ethernet/eth.c +5
-1struct packet_offload *ptype;int err = -ENOSYS ; -if (skb->encapsulation) +if (skb->encapsulation) {skb_set_inner_mac_header (skb, nhoff); + type = eh->h_proto; + }else { + type =NAPI_GRO_CB (skb)->network_proto; + } ptype =gro_find_complete_by_type (type);if (ptype !=NULL )net/ipv4/af_inet.c +3
-0@@ -1551,6 +1551,9 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)* immediately following this IP hdr. */ +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->transport_proto = proto; +/* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0 */
net/ipv4/af_inet.c int inet_gro_complete(struct sk_buff *skb, int nhoff) · line 1627
Quoted 5 lines
net/ipv4/af_inet.c +1
-1@@ -1621,12 +1624,15 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)__be16 newlen =htons (skb->len - nhoff);struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);const struct net_offload *ops; -int proto = iph->protocol; +int proto;
(same)
Quoted 11 lines
net/ipv4/af_inet.c +3
-0int err = -ENOSYS ;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IP ));skb_set_inner_network_header (skb, nhoff); + proto = iph->protocol; + }else { + proto =NAPI_GRO_CB (skb)->transport_proto; }csum_replace2 (&iph->check, iph->tot_len, newlen);
[...]
net/ipv4/af_inet.c INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) · line 366
Quoted 8 lines
net/ipv4/af_inet.c +3
-1@@ -358,7 +361,13 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); } - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (!skb->encapsulation) { + ops =rcu_dereference (inet6_offloads[NAPI_GRO_CB (skb)->transport_proto]); + nhoff +=NAPI_GRO_CB (skb)->network_len;
Why not use the same skb_network_header_len() here? Both skb->network_header and skb->transport_header must be set and correct at this point (if not, you can always fix that).
net/ipv4/af_inet.c INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) · line 373
Quoted 7 lines
net/ipv4/af_inet.c +4
-0+ }else { + nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); + } +if (WARN_ON (!ops || !ops->callbacks.gro_complete)) goto out;
Thanks,
Olek
Quoted 12 lines
Quoted 12 lines
Currently, the IPv6 extension headers are parsed twice: first in
ipv6_gro_receive, and then again inipv6_gro_complete.The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in
ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time inipv6_gro_complete.The implementation had to handle both inner and outer layers in case of encapsulation (as they can't use the same field).
Quoted 7 lines
Quoted 3 lines
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
Would be nice to see some perf numbers. "there was an improvement" doesn't say a lot TBH...
I just posted raw performance numbers as a reply to Eric's message. Take a look there.
Quoted 9 lines
net/ethernet/eth.c EXPORT_SYMBOL(eth_gro_receive); · line 462
Quoted 6 lines
net/ethernet/eth.c +1
-1@@ -456,12 +459,16 @@ EXPORT_SYMBOL(eth_gro_receive);int eth_gro_complete (struct sk_buff *skb,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff); -__be16 type = eh->h_proto; +__be16 type;Please don't break RCT style when shortening/expanding variable declaration lines.
Will be fixed in v2.
Quoted 4 lines
Quoted 4 lines
net/ipv6/ip6_offload.c +0
-0@@ -358,7 +361,13 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); }
Quoted 9 lines
net/ipv6/ip6_offload.c INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) · line 366
Quoted 4 lines
net/ipv6/ip6_offload.c +3
-1- nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (!skb->encapsulation) { + ops =rcu_dereference (inet6_offloads[NAPI_GRO_CB (skb)->transport_proto]); + nhoff +=NAPI_GRO_CB (skb)->network_len;Why not use the same
skb_network_header_len()here? Bothskb->network_headerandskb->transport_headermust be set and correct at this point (if not, you can always fix that).
When processing packets with encapsulation the network_header field is overwritten when processing the inner IP header, so skb_network_header_len won't return the correct value.
Quoted 10 lines · Richard Gobert
Currently, the IPv6 extension headers are parsed twice: first in
ipv6_gro_receive, and then again inipv6_gro_complete.The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in
ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time inipv6_gro_complete.
Quoted 6 lines
The implementation had to handle both inner and outer layers in case of encapsulation (as they can't use the same field).
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
It seems your patch adds a lot of conditional checks, which will alternate true/false
for encapsulated protocols.
So please give us raw numbers, ftrace is too heavy weight for such claims.
Quoted 50 lines
Signed-off-by Richard Gobert <[email protected]>--- include/net/gro.h | 8 ++++++-- net/ethernet/eth.c | 11 +++++++++-- net/ipv4/af_inet.c | 8 +++++++- net/ipv6/ip6_offload.c | 15 ++++++++++++--- 4 files changed, 34 insertions(+), 8 deletions(-)include/net/gro.h +6
-2@@ -41,8 +41,8 @@ struct napi_gro_cb {/* Number of segments aggregated. */ u16 count; -/* Used in ipv6_gro_receive() and foo-over-udp */ -u16 proto; +/* Used in eth_gro_receive() */ +__be16 network_proto;/* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1@@ -86,6 +86,10 @@ struct napi_gro_cb {/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; + +/* Used in inet and ipv6 _gro_receive() */ +u16 network_len; +u8 transport_proto; };#define NAPI_GRO_CB (skb) ((struct napi_gro_cb *)(skb)->cb)net/ethernet/eth.c +3
-0@@ -439,6 +439,9 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) goto out; } +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->network_proto = type; +skb_gro_pull (skb,sizeof (*eh));skb_gro_postpull_rcsum (skb, eh,sizeof (*eh));
net/ethernet/eth.c EXPORT_SYMBOL(eth_gro_receive); · line 461
Quoted 3 lines
net/ethernet/eth.c +0
-0@@ -456,12 +459,16 @@ EXPORT_SYMBOL(eth_gro_receive);int eth_gro_complete (struct sk_buff *skb,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff);
Why initializing @eh here is needed ?
Presumably, for !skb->encapsulation, @eh would not be used.
Quoted 33 lines
net/ethernet/eth.c +6
-2-__be16 type = eh->h_proto; +__be16 type;struct packet_offload *ptype;int err = -ENOSYS ; -if (skb->encapsulation) +if (skb->encapsulation) {skb_set_inner_mac_header (skb, nhoff); + type = eh->h_proto; + }else { + type =NAPI_GRO_CB (skb)->network_proto; + } ptype =gro_find_complete_by_type (type);if (ptype !=NULL )net/ipv4/af_inet.c +3
-0@@ -1551,6 +1551,9 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)* immediately following this IP hdr. */ +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->transport_proto = proto; +/* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0 */ @@ -1621,12 +1624,15 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)__be16 newlen =htons (skb->len - nhoff);struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);const struct net_offload *ops;
net/ipv4/af_inet.c int inet_gro_complete(struct sk_buff *skb, int nhoff) · line 1635
Quoted 10 lines
net/ipv4/af_inet.c +4
-1-int proto = iph->protocol; +int proto;int err = -ENOSYS ;if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IP ));skb_set_inner_network_header (skb, nhoff); + proto = iph->protocol; + }else { + proto =NAPI_GRO_CB (skb)->transport_proto;
I really doubt this change is needed.
We need to access iph->fields in the following lines. Adding an else {} branch is adding extra code, and makes your patch longer to review.
net/ipv6/ip6_offload.c INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, · line 230
Quoted 12 lines
net/ipv4/af_inet.c +0
-0}csum_replace2 (&iph->check, iph->tot_len, newlen);net/ipv6/ip6_offload.c +0
-1@@ -227,11 +227,14 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, iph =ipv6_hdr (skb); } -NAPI_GRO_CB (skb)->proto = proto;
I guess you missed BIG TCP ipv4 changes under review... ->proto is now used.
Quoted 13 lines
net/ipv6/ip6_offload.c +5
-1- flush--; nlen =skb_network_header_len (skb); +if (!NAPI_GRO_CB (skb)->encap_mark) { +NAPI_GRO_CB (skb)->transport_proto = proto; +NAPI_GRO_CB (skb)->network_len = nlen; + } +list_for_each_entry (p, head, list) {const struct ipv6hdr *iph2;__be32 first_word;/* <Version:4><Traffic_Class:8><Flow_Label:20> */
net/ipv6/ip6_offload.c INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) · line 368
Quoted 9 lines
net/ipv6/ip6_offload.c +5
-1@@ -358,7 +361,13 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); } - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (!skb->encapsulation) { + ops =rcu_dereference (inet6_offloads[NAPI_GRO_CB (skb)->transport_proto]); + nhoff +=NAPI_GRO_CB (skb)->network_len; + }else { + nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops);
IMO ipv6_exthdrs_len() is quite fast for the typical case where we have no extension headers.
This new conditional check seems expensive to me.
Quoted 8 lines
net/ipv6/ip6_offload.c +2
-1+ } +if (WARN_ON (!ops || !ops->callbacks.gro_complete)) goto out; --2.36.1
Quoted 17 lines
Quoted 10 lines
Currently, the IPv6 extension headers are parsed twice: first in
ipv6_gro_receive, and then again inipv6_gro_complete.The field NAPI_GRO_CB(skb)->proto is used by GRO to hold the layer 4 protocol type that comes after the IPv6 layer. I noticed that it is set in
ipv6_gro_receive, but isn't used anywhere. By using this field, and also storing the size of the network header, we can avoid parsing extension headers a second time inipv6_gro_complete.Quoted 6 lines
The implementation had to handle both inner and outer layers in case of encapsulation (as they can't use the same field).
I've applied this optimisation to all base protocols (IPv6, IPv4, Ethernet). Then, I benchmarked this patch on my machine, using ftrace to measure ipv6_gro_complete's performance, and there was an improvement.
Quoted 6 lines
It seems your patch adds a lot of conditional checks, which will alternate true/false
for encapsulated protocols.So please give us raw numbers, ftrace is too heavy weight for such claims.
For the benchmarks, I used 100Gbit NIC mlx5 single-core (power management off), turboboost off.
Typical IPv6 traffic (zero extension headers):
for i in {1..5}; do netperf -t TCP_STREAM -H 2001:db8:2:2::2 -l 90 | tail -1; done
# before
131072 16384 16384 90.00 16391.20
131072 16384 16384 90.00 16403.50
131072 16384 16384 90.00 16403.30
131072 16384 16384 90.00 16397.84
131072 16384 16384 90.00 16398.00
# after
131072 16384 16384 90.00 16399.85
131072 16384 16384 90.00 16392.37
131072 16384 16384 90.00 16403.06
131072 16384 16384 90.00 16406.97
131072 16384 16384 90.00 16406.09IPv6 over IPv6 traffic:
for i in {1..5}; do netperf -t TCP_STREAM -H 4001:db8:2:2::2 -l 90 | tail -1; done
# before
131072 16384 16384 90.00 14791.61
131072 16384 16384 90.00 14791.66
131072 16384 16384 90.00 14783.47
131072 16384 16384 90.00 14810.17
131072 16384 16384 90.00 14806.15
# after
131072 16384 16384 90.00 14793.49
131072 16384 16384 90.00 14816.10
131072 16384 16384 90.00 14818.41
131072 16384 16384 90.00 14780.35
131072 16384 16384 90.00 14800.48IPv6 traffic with varying extension headers:
for i in {1..5}; do netperf -t TCP_STREAM -H 2001:db8:2:2::2 -l 90 | tail -1; done
# before
131072 16384 16384 90.00 14812.37
131072 16384 16384 90.00 14813.04
131072 16384 16384 90.00 14802.54
131072 16384 16384 90.00 14804.06
131072 16384 16384 90.00 14819.08
# after
131072 16384 16384 90.00 14927.11
131072 16384 16384 90.00 14910.45
131072 16384 16384 90.00 14917.36
131072 16384 16384 90.00 14916.53
131072 16384 16384 90.00 14928.88Quoted 49 lines
Quoted 49 lines
Signed-off-by Richard Gobert <[email protected]>--- include/net/gro.h | 8 ++++++-- net/ethernet/eth.c | 11 +++++++++-- net/ipv4/af_inet.c | 8 +++++++- net/ipv6/ip6_offload.c | 15 ++++++++++++--- 4 files changed, 34 insertions(+), 8 deletions(-)include/net/gro.h +6
-2@@ -41,8 +41,8 @@ struct napi_gro_cb {/* Number of segments aggregated. */ u16 count; -/* Used in ipv6_gro_receive() and foo-over-udp */ -u16 proto; +/* Used in eth_gro_receive() */ +__be16 network_proto;/* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1@@ -86,6 +86,10 @@ struct napi_gro_cb {/* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; + +/* Used in inet and ipv6 _gro_receive() */ +u16 network_len; +u8 transport_proto; };#define NAPI_GRO_CB (skb) ((struct napi_gro_cb *)(skb)->cb)net/ethernet/eth.c +3
-0@@ -439,6 +439,9 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) goto out; } +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->network_proto = type; +skb_gro_pull (skb,sizeof (*eh));skb_gro_postpull_rcsum (skb, eh,sizeof (*eh));
Quoted 8 lines
net/ethernet/eth.c EXPORT_SYMBOL(eth_gro_receive); · line 461
Quoted 4 lines
net/ethernet/eth.c +0
-0@@ -456,12 +459,16 @@ EXPORT_SYMBOL(eth_gro_receive);int eth_gro_complete (struct sk_buff *skb,int nhoff) {struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff);Why initializing @eh here is needed ?
Presumably, for !skb->encapsulation, @eh would not be used.
Fixed in v2, thanks
Quoted 37 lines
Quoted 37 lines
net/ethernet/eth.c +6
-2-__be16 type = eh->h_proto; +__be16 type;struct packet_offload *ptype;int err = -ENOSYS ; -if (skb->encapsulation) +if (skb->encapsulation) {skb_set_inner_mac_header (skb, nhoff); + type = eh->h_proto; + }else { + type =NAPI_GRO_CB (skb)->network_proto; + } ptype =gro_find_complete_by_type (type);if (ptype !=NULL )net/ipv4/af_inet.c +4
-1@@ -1551,6 +1551,9 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)* immediately following this IP hdr. */ +if (!NAPI_GRO_CB (skb)->encap_mark) +NAPI_GRO_CB (skb)->transport_proto = proto; +/* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0 */ @@ -1621,12 +1624,15 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff)__be16 newlen =htons (skb->len - nhoff);struct iphdr *iph = (struct iphdr *)(skb->data + nhoff);const struct net_offload *ops; -int proto = iph->protocol; +int proto;int err = -ENOSYS ;
Quoted 12 lines
net/ipv4/af_inet.c int inet_gro_complete(struct sk_buff *skb, int nhoff) · line 1635
Quoted 6 lines
net/ipv4/af_inet.c +3
-0if (skb->encapsulation) {skb_set_inner_protocol (skb,cpu_to_be16 (ETH_P_IP ));skb_set_inner_network_header (skb, nhoff); + proto = iph->protocol; + }else { + proto =NAPI_GRO_CB (skb)->transport_proto;I really doubt this change is needed.
We need to accessiph->fieldsin the following lines. Adding an else {} branch is adding extra code, and makes your patch longer to review.
Good point, removed in v2.
Quoted 11 lines
Quoted 11 lines
net/ipv4/af_inet.c +0
-0}csum_replace2 (&iph->check, iph->tot_len, newlen);net/ipv6/ip6_offload.c +0
-0@@ -227,11 +227,14 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, iph =ipv6_hdr (skb); }
Quoted 4 lines
net/ipv6/ip6_offload.c INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, · line 230
Quoted 1 line
net/ipv6/ip6_offload.c +0
-1-NAPI_GRO_CB (skb)->proto = proto;I guess you missed BIG TCP ipv4 changes under review... ->proto is now used.
I rebased the patch now that ipv4 BIG TCP is merged, and made proto and transport_proto be separate variables.
Quoted 26 lines
Quoted 13 lines
net/ipv6/ip6_offload.c +5
-1- flush--; nlen =skb_network_header_len (skb); +if (!NAPI_GRO_CB (skb)->encap_mark) { +NAPI_GRO_CB (skb)->transport_proto = proto; +NAPI_GRO_CB (skb)->network_len = nlen; + } +list_for_each_entry (p, head, list) {const struct ipv6hdr *iph2;__be32 first_word;/* <Version:4><Traffic_Class:8><Flow_Label:20> */ net/ipv6/ip6_offload.c INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) · line 368
Quoted 9 lines
net/ipv6/ip6_offload.c +5
-1@@ -358,7 +361,13 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len =htons (payload_len); } - nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops); +if (!skb->encapsulation) { + ops =rcu_dereference (inet6_offloads[NAPI_GRO_CB (skb)->transport_proto]); + nhoff +=NAPI_GRO_CB (skb)->network_len; + }else { + nhoff +=sizeof (*iph) +ipv6_exthdrs_len (iph, &ops);IMO
ipv6_exthdrs_len()is quite fast for the typical case where we have no extension headers.
Quoted 2 lines
This new conditional check seems expensive to me.
In v2 I moved the encapsulation branch at the beginning of the function to this spot, merging the conditions. So for the typical case, instead of another redundant extension header length calculation (which requires at least one branch and one dereference of iph data), it's a simple dereference to CB. Thus, performance for the typical case is unharmed, and possibly even slightly improved. For cases with a varying amount of extension headers in ipv6, there's a performance upgrade (multiple memory dereference to iph data and conditional checks are saved).
Also, after further inspection, I noticed that there is a potential problem in ipv6_gro_complete(), where inner_network_header is initialized to the wrong value. The initialization of the inner_network_header field should be performed after the BIG TCP if block. I fixed this in v2 by combining this initialization with the new conditional check after the BIG TCP so the patch does not add a conditional check anymore.