Add TCP fraglist GRO support
Subsystems (3)
v5 cover Add TCP fraglist GRO support
When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.
When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.
Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.
rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idle
Changes since v4:
- add likely() to prefer the non-fraglist path in check
Changes since v3:
- optimize __tcpv4_gso_segment_csum
- add unlikely()
- reorder dev_net/skb_gro_network_header calls after NETIF_F_GRO_FRAGLIST check
- add support for ipv6 nat
- drop redundant pskb_may_pull check
Changes since v2:
- create tcp_gro_header_pull helper function to pull tcp header only once
- optimize __tcpv4_gso_segment_list_csum, drop obsolete flags check
Changes since v1:
- revert bogus tcp flags overwrite on segmentation
- fix kbuild issue with !CONFIG_IPV6
- only perform socket lookup for the first skb in the GRO train
Changes since RFC:
- split up patches
- handle TCP flags mutations
Felix Fietkau (6):
net: move skb_gro_receive_list from udp to core
net: add support for segmenting TCP fraglist GSO packets
net: add code for TCP fraglist GRO
net: create tcp_gro_lookup helper function
net: create tcp_gro_header_pull helper function
net: add heuristic for enabling TCP fraglist GRO
include/net/gro.h | 1 + include/net/tcp.h | 5 +- net/core/gro.c | 27 +++++ net/ipv4/tcp_offload.c | 214 ++++++++++++++++++++++++++++++++------- net/ipv4/udp_offload.c | 27 ----- net/ipv6/tcpv6_offload.c | 120 +++++++++++++++++++++- 6 files changed, 325 insertions(+), 69 deletions(-)
Signature · 1 line
2.44.0
PATCH v5 1/6 net: move skb_gro_receive_list from udp to core
This helper function will be used for TCP fraglist GRO support
--- include/net/gro.h | 1 + net/core/gro.c | 27 +++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 27 --------------------------- 3 files changed, 28 insertions(+), 27 deletions(-)
include/net/gro.h +1 -0
@@ -429,6 +429,7 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb, }int skb_gro_receive (struct sk_buff *p,struct sk_buff *skb); +int skb_gro_receive_list (struct sk_buff *p,struct sk_buff *skb);/* Pass the currently batched GRO_NORMAL SKBs up to the stack. */ static inline void gro_normal_list (struct napi_struct *napi)
net/core/gro.c +27 -0
@@ -231,6 +231,33 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)return 0 ; } +int skb_gro_receive_list (struct sk_buff *p,struct sk_buff *skb) +{ +if (unlikely (p->len + skb->len >=65536 )) +return -E2BIG ; + +if (NAPI_GRO_CB (p)->last == p) +skb_shinfo (p)->frag_list = skb; +else +NAPI_GRO_CB (p)->last->next = skb; + +skb_pull (skb,skb_gro_offset (skb)); + +NAPI_GRO_CB (p)->last = skb; +NAPI_GRO_CB (p)->count++; + p->data_len += skb->len; + +/* sk ownership - if any - completely transferred to the aggregated packet */ + skb->destructor =NULL ; + skb->sk =NULL ; + p->truesize += skb->truesize; + p->len += skb->len; + +NAPI_GRO_CB (skb)->same_flow =1 ; + +return 0 ; +} +static void napi_gro_complete (struct napi_struct *napi,struct sk_buff *skb) {
net/ipv4/udp_offload.c +0 -27
@@ -433,33 +433,6 @@ static struct sk_buff *udp4_ufo_fragment(struct sk_buff *skb,return segs; } -static int skb_gro_receive_list (struct sk_buff *p,struct sk_buff *skb) -{ -if (unlikely (p->len + skb->len >=65536 )) -return -E2BIG ; - -if (NAPI_GRO_CB (p)->last == p) -skb_shinfo (p)->frag_list = skb; -else -NAPI_GRO_CB (p)->last->next = skb; - -skb_pull (skb,skb_gro_offset (skb)); - -NAPI_GRO_CB (p)->last = skb; -NAPI_GRO_CB (p)->count++; - p->data_len += skb->len; - -/* sk ownership - if any - completely transferred to the aggregated packet */ - skb->destructor =NULL ; - skb->sk =NULL ; - p->truesize += skb->truesize; - p->len += skb->len; - -NAPI_GRO_CB (skb)->same_flow =1 ; - -return 0 ; -} -#define UDP_GRO_CNT_MAX 64 static struct sk_buff *udp_gro_receive_segment(struct list_head *head,
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 1/6] net: move skb_gro_receive_list from udp to core
Quoted 11 lines · Felix Fietkau
This helper function will be used for TCP fraglist GRO support
Acked-by Paolo Abeni <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- include/net/gro.h | 1 + net/core/gro.c | 27 +++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 27 --------------------------- 3 files changed, 28 insertions(+), 27 deletions(-)
PATCH v5 2/6 net: add support for segmenting TCP fraglist GSO packets
In reply to Felix FietkauPreparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.
--- net/ipv4/tcp_offload.c | 67 ++++++++++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 58 ++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+)
net/ipv4/tcp_offload.c +67 -0
@@ -28,6 +28,70 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, } } +static void __tcpv4_gso_segment_csum (struct sk_buff *seg, +__be32 *oldip,__be32 newip, +__be16 *oldport,__be16 newport) +{ +struct tcphdr *th; +struct iphdr *iph; + +if (*oldip == newip && *oldport == newport) +return ; + + th =tcp_hdr (seg); + iph =ip_hdr (seg); + +inet_proto_csum_replace4 (&th->check, seg, *oldip, newip, true); +inet_proto_csum_replace2 (&th->check, seg, *oldport, newport, false); + *oldport = newport; + +csum_replace4 (&iph->check, *oldip, newip); + *oldip = newip; +} + +static struct sk_buff *__tcpv4_gso_segment_list_csum (struct sk_buff *segs) +{ +const struct tcphdr *th; +const struct iphdr *iph; +struct sk_buff *seg; +struct tcphdr *th2; +struct iphdr *iph2; + + seg = segs; + th =tcp_hdr (seg); + iph =ip_hdr (seg); + th2 =tcp_hdr (seg->next); + iph2 =ip_hdr (seg->next); + +if (!(*(const u32 *)&th->source ^ *(const u32 *)&th2->source) && + iph->daddr == iph2->daddr && iph->saddr == iph2->saddr) +return segs; + +while ((seg = seg->next)) { + th2 =tcp_hdr (seg); + iph2 =ip_hdr (seg); + +__tcpv4_gso_segment_csum (seg, + &iph2->saddr, iph->saddr, + &th2->source, th->source); +__tcpv4_gso_segment_csum (seg, + &iph2->daddr, iph->daddr, + &th2->dest, th->dest); + } + +return segs; +} + +static struct sk_buff *__tcp4_gso_segment_list (struct sk_buff *skb, +netdev_features_t features) +{ + skb =skb_segment_list (skb, features,skb_mac_header_len (skb)); +if (IS_ERR (skb)) +return skb; + +return __tcpv4_gso_segment_list_csum (skb); +} +static struct sk_buff *tcp4_gso_segment (struct sk_buff *skb,netdev_features_t features) {
@@ -37,6 +101,9 @@ static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb,if (!pskb_may_pull (skb,sizeof (struct tcphdr )))return ERR_PTR (-EINVAL ); +if (skb_shinfo (skb)->gso_type &SKB_GSO_FRAGLIST ) +return __tcp4_gso_segment_list (skb, features); +if (unlikely (skb->ip_summed !=CHECKSUM_PARTIAL )) {const struct iphdr *iph =ip_hdr (skb);struct tcphdr *th =tcp_hdr (skb);
net/ipv6/tcpv6_offload.c +58 -0
70 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 2/6] net: add support for segmenting TCP fraglist GSO packets
Quoted 4 lines · Felix Fietkau
Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.
Quoted 8 lines
Acked-by Paolo Abeni <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- net/ipv4/tcp_offload.c | 67 ++++++++++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 58 ++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+)
PATCH v5 3/6 net: add code for TCP fraglist GRO
In reply to Felix FietkauThis implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.
--- net/ipv4/tcp_offload.c | 21 +++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 9 +++++++++ 2 files changed, 30 insertions(+)
net/ipv4/tcp_offload.c +21 -0
@@ -334,6 +334,18 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) flush |= (ntohl (th2->seq) +skb_gro_len (p)) ^ntohl (th->seq); flush |=skb_cmp_decrypted (p, skb); +if (unlikely (NAPI_GRO_CB (p)->is_flist)) { + flush |= (__force int )(flags ^tcp_flag_word (th2)); + flush |= skb->ip_summed != p->ip_summed; + flush |= skb->csum_level != p->csum_level; + flush |=NAPI_GRO_CB (p)->count >=64 ; + +if (flush ||skb_gro_receive_list (p, skb)) + mss =1 ; + + goto out_check_final; + } +if (flush ||skb_gro_receive (p, skb)) { mss =1 ; goto out_check_final;
@@ -400,6 +412,15 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff)const struct iphdr *iph =ip_hdr (skb);struct tcphdr *th =tcp_hdr (skb); +if (unlikely (NAPI_GRO_CB (skb)->is_flist)) { +skb_shinfo (skb)->gso_type |=SKB_GSO_FRAGLIST |SKB_GSO_TCPV4 ; +skb_shinfo (skb)->gso_segs =NAPI_GRO_CB (skb)->count; + +__skb_incr_checksum_unnecessary (skb); + +return 0 ; + } + th->check = ~tcp_v4_check (skb->len - thoff, iph->saddr, iph->daddr,0 );
net/ipv6/tcpv6_offload.c +9 -0
@@ -32,6 +32,15 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)const struct ipv6hdr *iph =ipv6_hdr (skb);struct tcphdr *th =tcp_hdr (skb); +if (unlikely (NAPI_GRO_CB (skb)->is_flist)) { +skb_shinfo (skb)->gso_type |=SKB_GSO_FRAGLIST |SKB_GSO_TCPV6 ; +skb_shinfo (skb)->gso_segs =NAPI_GRO_CB (skb)->count; + +__skb_incr_checksum_unnecessary (skb); + +return 0 ; + } + th->check = ~tcp_v6_check (skb->len - thoff, &iph->saddr, &iph->daddr,0 );skb_shinfo (skb)->gso_type |=SKB_GSO_TCPV6 ;
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 3/6] net: add code for TCP fraglist GRO
Quoted 7 lines · Felix Fietkau
This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.
Acked-by Paolo Abeni <[email protected]>Signed-off-by Felix Fietkau <[email protected]>
Re: [PATCH v5 net-next v5 3/6] net: add code for TCP fraglist GRO
In reply to Felix FietkauQuoted 11 lines · Felix Fietkau
This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.
Acked-by Paolo Abeni <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- net/ipv4/tcp_offload.c | 21 +++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 9 +++++++++ 2 files changed, 30 insertions(+)
PATCH v5 4/6 net: create tcp_gro_lookup helper function
In reply to Felix FietkauThis pulls the flow port matching out of tcp_gro_receive, so that it can be reused for the next change, which adds the TCP fraglist GRO heuristic.
--- include/net/tcp.h | 1 + net/ipv4/tcp_offload.c | 41 +++++++++++++++++++++++++---------------- 2 files changed, 26 insertions(+), 16 deletions(-)
include/net/tcp.h +1 -0
@@ -2199,6 +2199,7 @@ void tcp_v4_destroy_sock(struct sock *sk);struct sk_buff *tcp_gso_segment (struct sk_buff *skb,netdev_features_t features); +struct sk_buff *tcp_gro_lookup (struct list_head *head,struct tcphdr *th);struct sk_buff *tcp_gro_receive (struct list_head *head,struct sk_buff *skb); INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb));
net/ipv4/tcp_offload.c +25 -16
55 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 4/6] net: create tcp_gro_lookup helper function
Quoted 11 lines · Felix Fietkau
This pulls the flow port matching out of
tcp_gro_receive, so that it can be reused for the next change, which adds the TCP fraglist GRO heuristic.Acked-by Paolo Abeni <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- include/net/tcp.h | 1 + net/ipv4/tcp_offload.c | 41 +++++++++++++++++++++++++---------------- 2 files changed, 26 insertions(+), 16 deletions(-)
PATCH v5 5/6 net: create tcp_gro_header_pull helper function
In reply to Felix FietkauPull the code out of tcp_gro_receive in order to access the tcp header from tcp4/6_gro_receive.
--- include/net/tcp.h | 4 ++- net/ipv4/tcp_offload.c | 55 +++++++++++++++++++++++++--------------- net/ipv6/tcpv6_offload.c | 18 +++++++++---- 3 files changed, 50 insertions(+), 27 deletions(-)
include/net/tcp.h +3 -1
@@ -2199,8 +2199,10 @@ void tcp_v4_destroy_sock(struct sock *sk);struct sk_buff *tcp_gso_segment (struct sk_buff *skb,netdev_features_t features); +struct tcphdr *tcp_gro_pull_header (struct sk_buff *skb);struct sk_buff *tcp_gro_lookup (struct list_head *head,struct tcphdr *th); -struct sk_buff *tcp_gro_receive (struct list_head *head,struct sk_buff *skb); +struct sk_buff *tcp_gro_receive (struct list_head *head,struct sk_buff *skb, +struct tcphdr *th); INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff));
net/ipv4/tcp_offload.c +34 -21
96 lines not on this page · raw message
net/ipv6/tcpv6_offload.c +13 -5
@@ -16,15 +16,23 @@INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) { +struct tcphdr *th; +/* Don't bother verifying checksum if we're going to flush anyway. */ if (!NAPI_GRO_CB (skb)->flush &&skb_gro_checksum_validate (skb,IPPROTO_TCP , - ip6_gro_compute_pseudo)) { -NAPI_GRO_CB (skb)->flush =1 ; -return NULL ; - } + ip6_gro_compute_pseudo)) + goto flush; + + th =tcp_gro_pull_header (skb); +if (!th) + goto flush; + +return tcp_gro_receive (head, skb, th); -return tcp_gro_receive (head, skb); +flush: +NAPI_GRO_CB (skb)->flush =1 ; +return NULL ; } INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 5/6] net: create tcp_gro_header_pull helper function
Quoted 12 lines · Felix Fietkau
Pull the code out of
tcp_gro_receivein order to access the tcp header from tcp4/6_gro_receive.Acked-by Paolo Abeni <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- include/net/tcp.h | 4 ++- net/ipv4/tcp_offload.c | 55 +++++++++++++++++++++++++--------------- net/ipv6/tcpv6_offload.c | 18 +++++++++---- 3 files changed, 50 insertions(+), 27 deletions(-)
PATCH v5 6/6 net: add heuristic for enabling TCP fraglist GRO
In reply to Felix FietkauWhen forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.
When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.
Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.
rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idle
--- net/ipv4/tcp_offload.c | 32 ++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)
net/ipv4/tcp_offload.c +32 -0
@@ -407,6 +407,36 @@ void tcp_gro_complete(struct sk_buff *skb) }EXPORT_SYMBOL (tcp_gro_complete); +static void tcp4_check_fraglist_gro (struct list_head *head,struct sk_buff *skb, +struct tcphdr *th) +{ +const struct iphdr *iph; +struct sk_buff *p; +struct sock *sk; +struct net *net; +int iif, sdif; + +if (likely (!(skb->dev->features &NETIF_F_GRO_FRAGLIST ))) +return ; + + p =tcp_gro_lookup (head, th); +if (p) { +NAPI_GRO_CB (skb)->is_flist =NAPI_GRO_CB (p)->is_flist; +return ; + } + +inet_get_iif_sdif (skb, &iif, &sdif); + iph =skb_gro_network_header (skb); + net =dev_net (skb->dev); + sk =__inet_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, + iph->saddr, th->source, + iph->daddr,ntohs (th->dest), + iif, sdif); +NAPI_GRO_CB (skb)->is_flist = !sk; +if (sk) +sock_put (sk); +} +INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) {
@@ -422,6 +452,8 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)if (!th) goto flush; +tcp4_check_fraglist_gro (head, skb, th); +return tcp_gro_receive (head, skb, th); flush:
net/ipv6/tcpv6_offload.c +35 -0
@@ -7,12 +7,45 @@*/ #include <linux/indirect_call_wrapper.h> #include <linux/skbuff.h> +#include <net/inet6_hashtables.h> #include <net/gro.h> #include <net/protocol.h> #include <net/tcp.h> #include <net/ip6_checksum.h> #include "ip6_offload.h" +static void tcp6_check_fraglist_gro (struct list_head *head,struct sk_buff *skb, +struct tcphdr *th) +{ +#if IS_ENABLED (CONFIG_IPV6 ) +const struct ipv6hdr *hdr; +struct sk_buff *p; +struct sock *sk; +struct net *net; +int iif, sdif; + +if (likely (!(skb->dev->features &NETIF_F_GRO_FRAGLIST ))) +return ; + + p =tcp_gro_lookup (head, th); +if (p) { +NAPI_GRO_CB (skb)->is_flist =NAPI_GRO_CB (p)->is_flist; +return ; + } + +inet6_get_iif_sdif (skb, &iif, &sdif); + hdr =skb_gro_network_header (skb); + net =dev_net (skb->dev); + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, + &hdr->saddr, th->source, + &hdr->daddr,ntohs (th->dest), + iif, sdif); +NAPI_GRO_CB (skb)->is_flist = !sk; +if (sk) +sock_put (sk); +#endif /* IS_ENABLED(CONFIG_IPV6) */ +} +INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) {
@@ -28,6 +61,8 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)if (!th) goto flush; +tcp6_check_fraglist_gro (head, skb, th); +return tcp_gro_receive (head, skb, th); flush:
Signature · 1 line
2.44.0
Re: [PATCH v5 net-next v5 6/6] net: add heuristic for enabling TCP fraglist GRO
Quoted 19 lines · Felix Fietkau
When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.When
NETIF_F_GRO_FRAGLISTis enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.
rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idle
Quoted 8 lines
Acked-by Paolo Abeni <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Felix Fietkau <[email protected]>--- net/ipv4/tcp_offload.c | 32 ++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)
Quoted 43 lines · Felix Fietkau
When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.When
NETIF_F_GRO_FRAGLISTis enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.
rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idleChanges since v4:
- addlikely()to prefer the non-fraglist path in checkChanges since v3:
- optimize__tcpv4_gso_segment_csum
- addunlikely()
- reorder dev_net/skb_gro_network_header calls afterNETIF_F_GRO_FRAGLISTcheck
- add support for ipv6 nat
- drop redundantpskb_may_pullcheckChanges since v2:
- createtcp_gro_header_pullhelper function to pull tcp header only once
- optimize__tcpv4_gso_segment_list_csum, drop obsolete flags checkChanges since v1:
- revert bogus tcp flags overwrite on segmentation
- fix kbuild issue with !CONFIG_IPV6
- only perform socket lookup for the first skb in the GRO trainChanges since RFC:
- split up patches
- handle TCP flags mutations
Quoted 7 lines
Felix Fietkau (6):
net: moveskb_gro_receive_listfrom udp to core
net: add support for segmenting TCP fraglist GSO packets
net: add code for TCP fraglist GRO
net: createtcp_gro_lookuphelper function
net: createtcp_gro_header_pullhelper function
net: add heuristic for enabling TCP fraglist GRO
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 6 lines · On Thu, 2 May 2024 10:44:41 +0200 you wrote:
When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.
Quoted 6 lines
When
NETIF_F_GRO_FRAGLISTis enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.[...]
Here is the summary with links:
- [v5,net-next,v5,1/6] net: move skb_gro_receive_list from udp to core
https://git.kernel.org/netdev/net-next/c/8928756d53d5 - [v5,net-next,v5,2/6] net: add support for segmenting TCP fraglist GSO packets https://git.kernel.org/netdev/net-next/c/bee88cd5bd83 - [v5,net-next,v5,3/6] net: add code for TCP fraglist GRO https://git.kernel.org/netdev/net-next/c/8d95dc474f85 - [v5,net-next,v5,4/6] net: create tcp_gro_lookup helper function https://git.kernel.org/netdev/net-next/c/80e85fbdf19e - [v5,net-next,v5,5/6] net: create tcp_gro_header_pull helper function https://git.kernel.org/netdev/net-next/c/7516b27c555c - [v5,net-next,v5,6/6] net: add heuristic for enabling TCP fraglist GRO https://git.kernel.org/netdev/net-next/c/c9d1d23e5239
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html