Skip to contentSkip to conversation

Add TCP fraglist GRO support

PATCH Superseded Felix Fietkau 3 participants 22 messages Apr 26, 2024

Subsystems (3)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add TCP fraglist GRO support

When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.

When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.

Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.

rx-gro-list off: 630 Mbit/s, CPU 35% idle
rx-gro-list on:  770 Mbit/s, CPU 40% idle

Changes since v2:
- create tcp_gro_header_pull helper function to pull tcp header only once
- optimize __tcpv4_gso_segment_list_csum, drop obsolete flags check

Changes since v1:
- revert bogus tcp flags overwrite on segmentation
- fix kbuild issue with !CONFIG_IPV6
- only perform socket lookup for the first skb in the GRO train

Changes since RFC:
- split up patches
- handle TCP flags mutations

Felix Fietkau (6):
net: move skb_gro_receive_list from udp to core
net: add support for segmenting TCP fraglist GSO packets
net: add code for TCP fraglist GRO
net: create tcp_gro_lookup helper function
net: create tcp_gro_header_pull helper function
net: add heuristic for enabling TCP fraglist GRO

 include/net/gro.h        |   1 +
 include/net/tcp.h        |   5 +-
 net/core/gro.c           |  27 +++++
 net/ipv4/tcp_offload.c   | 211 ++++++++++++++++++++++++++++++++-------
 net/ipv4/udp_offload.c   |  27 -----
 net/ipv6/tcpv6_offload.c |  63 +++++++++++-
 6 files changed, 265 insertions(+), 69 deletions(-)
Signature · 1 line
2.44.0
#1 lore raw

PATCH v3 1/6 net: move skb_gro_receive_list from udp to core

This helper function will be used for TCP fraglist GRO support

---
 include/net/gro.h      |  1 +
 net/core/gro.c         | 27 +++++++++++++++++++++++++++
 net/ipv4/udp_offload.c | 27 ---------------------------
 3 files changed, 28 insertions(+), 27 deletions(-)
include/net/gro.h +1 -0
index 50f1e403dbbb..ca8e4b3de044 100644
@@ -429,6 +429,7 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb,
 }
 
 int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb);
+int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb);
 
 /* Pass the currently batched GRO_NORMAL SKBs up to the stack. */
 static inline void gro_normal_list(struct napi_struct *napi)
net/core/gro.c +27 -0
index 2459ab697f7f..268c6c826d09 100644
@@ -231,6 +231,33 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)
 	return 0;
 }
 
+int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb)
+{
+	if (unlikely(p->len + skb->len >= 65536))
+		return -E2BIG;
+
+	if (NAPI_GRO_CB(p)->last == p)
+		skb_shinfo(p)->frag_list = skb;
+	else
+		NAPI_GRO_CB(p)->last->next = skb;
+
+	skb_pull(skb, skb_gro_offset(skb));
+
+	NAPI_GRO_CB(p)->last = skb;
+	NAPI_GRO_CB(p)->count++;
+	p->data_len += skb->len;
+
+	/* sk ownership - if any - completely transferred to the aggregated packet */
+	skb->destructor = NULL;
+	skb->sk = NULL;
+	p->truesize += skb->truesize;
+	p->len += skb->len;
+
+	NAPI_GRO_CB(skb)->same_flow = 1;
+
+	return 0;
+}
+
 
 static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb)
 {
net/ipv4/udp_offload.c +0 -27
index 3498dd1d0694..a3cd546a1aea 100644
@@ -433,33 +433,6 @@ static struct sk_buff *udp4_ufo_fragment(struct sk_buff *skb,
 	return segs;
 }
 
-static int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb)
-{
-	if (unlikely(p->len + skb->len >= 65536))
-		return -E2BIG;
-
-	if (NAPI_GRO_CB(p)->last == p)
-		skb_shinfo(p)->frag_list = skb;
-	else
-		NAPI_GRO_CB(p)->last->next = skb;
-
-	skb_pull(skb, skb_gro_offset(skb));
-
-	NAPI_GRO_CB(p)->last = skb;
-	NAPI_GRO_CB(p)->count++;
-	p->data_len += skb->len;
-
-	/* sk ownership - if any - completely transferred to the aggregated packet */
-	skb->destructor = NULL;
-	skb->sk = NULL;
-	p->truesize += skb->truesize;
-	p->len += skb->len;
-
-	NAPI_GRO_CB(skb)->same_flow = 1;
-
-	return 0;
-}
-
 
 #define UDP_GRO_CNT_MAX 64
 static struct sk_buff *udp_gro_receive_segment(struct list_head *head,
Signature · 1 line
2.44.0
#3 lore raw

PATCH v3 2/6 net: add support for segmenting TCP fraglist GSO packets

In reply to Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +65 -0
index fab0973f995b..c493e95e09a5 100644
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
 	}
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+				     __be32 *oldip, __be32 *newip,
+				     __be16 *oldport, __be16 *newport)
+{
+	struct tcphdr *th;
+	struct iphdr *iph;
+
+	if (*oldip == *newip && *oldport == *newport)
+		return;
+
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+
+	inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+	inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+	*oldport = *newport;
+
+	csum_replace4(&iph->check, *oldip, *newip);
+	*oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+	struct sk_buff *seg;
+	struct tcphdr *th, *th2;
+	struct iphdr *iph, *iph2;
+
+	seg = segs;
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+	th2 = tcp_hdr(seg->next);
+	iph2 = ip_hdr(seg->next);
+
+	if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
+	    iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+		return segs;
+
+	while ((seg = seg->next)) {
+		th2 = tcp_hdr(seg);
+		iph2 = ip_hdr(seg);
+
+		__tcpv4_gso_segment_csum(seg,
+					 &iph2->saddr, &iph->saddr,
+					 &th2->source, &th->source);
+		__tcpv4_gso_segment_csum(seg,
+					 &iph2->daddr, &iph->daddr,
+					 &th2->dest, &th->dest);
+	}
+
+	return segs;
+}
+
+static struct sk_buff *__tcp4_gso_segment_list(struct sk_buff *skb,
+					      netdev_features_t features)
+{
+	skb = skb_segment_list(skb, features, skb_mac_header_len(skb));
+	if (IS_ERR(skb))
+		return skb;
+
+	return __tcpv4_gso_segment_list_csum(skb);
+}
+
 static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb,
 					netdev_features_t features)
 {
@@ -37,6 +99,9 @@ static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb,
 	if (!pskb_may_pull(skb, sizeof(struct tcphdr)))
 		return ERR_PTR(-EINVAL);
 
+	if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST)
+		return __tcp4_gso_segment_list(skb, features);
+
 	if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) {
 		const struct iphdr *iph = ip_hdr(skb);
 		struct tcphdr *th = tcp_hdr(skb);
net/ipv6/tcpv6_offload.c +3 -0
index 4b07d1e6c952..b3b8e1f6b92a 100644
@@ -51,6 +51,9 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb,
 	if (!pskb_may_pull(skb, sizeof(*th)))
 		return ERR_PTR(-EINVAL);
 
+	if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST)
+		return skb_segment_list(skb, features, skb_mac_header_len(skb));
+
 	if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) {
 		const struct ipv6hdr *ipv6h = ipv6_hdr(skb);
 		struct tcphdr *th = tcp_hdr(skb);
Signature · 1 line
2.44.0
#4 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

Quoted 16 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index fab0973f995b..c493e95e09a5 100644

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 33

Quoted 6 lines
net/ipv4/tcp_offload.c +3 -0
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
        }
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+                                    __be32 *oldip, __be32 *newip,
+                                    __be16 *oldport, __be16 *newport)

Do we really need pointers for newip and newport ?

Quoted 13 lines
net/ipv4/tcp_offload.c +13 -0
+{
+       struct tcphdr *th;
+       struct iphdr *iph;
+
+       if (*oldip == *newip && *oldport == *newport)
+               return;
+
+       th = tcp_hdr(seg);
+       iph = ip_hdr(seg);
+
+       inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+       inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+       *oldport = *newport;

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 56

Quoted 10 lines
net/ipv4/tcp_offload.c +10 -0
+
+       csum_replace4(&iph->check, *oldip, *newip);
+       *oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+       struct sk_buff *seg;
+       struct tcphdr *th, *th2;
+       struct iphdr *iph, *iph2;

I would probably add a const qualifier to th and iph

Quoted 28 lines
net/ipv4/tcp_offload.c +25 -0
+
+       seg = segs;
+       th = tcp_hdr(seg);
+       iph = ip_hdr(seg);
+       th2 = tcp_hdr(seg->next);
+       iph2 = ip_hdr(seg->next);
+
+       if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
 
 
+           iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+               return segs;
+
+       while ((seg = seg->next)) {
+               th2 = tcp_hdr(seg);
+               iph2 = ip_hdr(seg);
+
+               __tcpv4_gso_segment_csum(seg,
+                                        &iph2->saddr, &iph->saddr,
+                                        &th2->source, &th->source);
+               __tcpv4_gso_segment_csum(seg,
+                                        &iph2->daddr, &iph->daddr,
+                                        &th2->dest, &th->dest);
+       }
+
+       return segs;
+}
 
#5 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

Quoted 44 lines · Eric Dumazet
Quoted 16 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index fab0973f995b..c493e95e09a5 100644

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 33

Quoted 6 lines
net/ipv4/tcp_offload.c +3 -0
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
        }
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+                                    __be32 *oldip, __be32 *newip,
+                                    __be16 *oldport, __be16 *newport)

Do we really need pointers for newip and newport ?

Quoted 17 lines
net/ipv4/tcp_offload.c +17 -0
+{
+       struct tcphdr *th;
+       struct iphdr *iph;
+
+       if (*oldip == *newip && *oldport == *newport)
+               return;
+
+       th = tcp_hdr(seg);
+       iph = ip_hdr(seg);
+
+       inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+       inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+       *oldport = *newport;
+
+       csum_replace4(&iph->check, *oldip, *newip);
+       *oldip = *newip;
+}
Quoted 8 lines

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 56

Quoted 6 lines
net/ipv4/tcp_offload.c +6 -0
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+       struct sk_buff *seg;
+       struct tcphdr *th, *th2;
+       struct iphdr *iph, *iph2;

I would probably add a const qualifier to th and iph

Will do, thanks.

- Felix

maintainer
#6 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

In reply to Felix Fietkau
Quoted 44 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +26 -0
index fab0973f995b..c493e95e09a5 100644
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
 	}
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+				     __be32 *oldip, __be32 *newip,
+				     __be16 *oldport, __be16 *newport)
+{
+	struct tcphdr *th;
+	struct iphdr *iph;
+
+	if (*oldip == *newip && *oldport == *newport)
+		return;
+
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+
+	inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+	inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+	*oldport = *newport;
+
+	csum_replace4(&iph->check, *oldip, *newip);
+	*oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+	struct sk_buff *seg;
+	struct tcphdr *th, *th2;
+	struct iphdr *iph, *iph2;

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 66

Quoted 10 lines
net/ipv4/tcp_offload.c +10 -0
+
+	seg = segs;
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+	th2 = tcp_hdr(seg->next);
+	iph2 = ip_hdr(seg->next);
+
+	if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
+	    iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+		return segs;

As mentioned in previous revisions, I think a problem with this approach is that the stack could make other changes to the TCP header after the GRO stage, that are unnoticed here and could cause csum corruption, if the egress device does not recompute the packet csum.

Cheers,

Paolo

#7 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

Quoted 52 lines · Paolo Abeni
Quoted 51 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +33 -0
index fab0973f995b..c493e95e09a5 100644
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
 	}
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+				     __be32 *oldip, __be32 *newip,
+				     __be16 *oldport, __be16 *newport)
+{
+	struct tcphdr *th;
+	struct iphdr *iph;
+
+	if (*oldip == *newip && *oldport == *newport)
+		return;
+
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+
+	inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+	inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+	*oldport = *newport;
+
+	csum_replace4(&iph->check, *oldip, *newip);
+	*oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+	struct sk_buff *seg;
+	struct tcphdr *th, *th2;
+	struct iphdr *iph, *iph2;
+
+	seg = segs;
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+	th2 = tcp_hdr(seg->next);
+	iph2 = ip_hdr(seg->next);
+
Quoted 8 lines

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 66

Quoted 3 lines
net/ipv4/tcp_offload.c +3 -0
+	if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
+	    iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+		return segs;

As mentioned in previous revisions, I think a problem with this approach is that the stack could make other changes to the TCP header after the GRO stage, that are unnoticed here and could cause csum corruption, if the egress device does not recompute the packet csum.

On segmentation, each packet keeps its original TCP header and csum. If the stack makes changes, they apply to the first packet only. I don't see how we could get csum corruption.

- Felix

maintainer
#8 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

Quoted 62 lines · Felix Fietkau
Quoted 52 lines · Paolo Abeni
Quoted 51 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +33 -0
index fab0973f995b..c493e95e09a5 100644
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
 	}
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+				     __be32 *oldip, __be32 *newip,
+				     __be16 *oldport, __be16 *newport)
+{
+	struct tcphdr *th;
+	struct iphdr *iph;
+
+	if (*oldip == *newip && *oldport == *newport)
+		return;
+
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+
+	inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+	inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+	*oldport = *newport;
+
+	csum_replace4(&iph->check, *oldip, *newip);
+	*oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+	struct sk_buff *seg;
+	struct tcphdr *th, *th2;
+	struct iphdr *iph, *iph2;
+
+	seg = segs;
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+	th2 = tcp_hdr(seg->next);
+	iph2 = ip_hdr(seg->next);
+
Quoted 8 lines

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 66

Quoted 3 lines
net/ipv4/tcp_offload.c +3 -0
+	if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
+	    iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+		return segs;

As mentioned in previous revisions, I think a problem with this approach is that the stack could make other changes to the TCP header after the GRO stage, that are unnoticed here and could cause csum corruption, if the egress device does not recompute the packet csum.

Quoted 3 lines

On segmentation, each packet keeps its original TCP header and csum. If the stack makes changes, they apply to the first packet only. I don't see how we could get csum corruption.

You are right. I did not take in account that such changes (to the first skb) are not reflected to the frag_list at segmentation time. The end result could be different from what the user/admin is expecting, but at least should not impact drops.

Side note: alike UDP, this is not supporting IPv6 NAT...

Thanks,

Paolo

#9 lore raw

Re: [PATCH v3 net-next v3 2/6] net: add support for segmenting TCP fraglist GSO packets

Quoted 63 lines · Paolo Abeni
Quoted 62 lines · Felix Fietkau
Quoted 60 lines · Paolo Abeni
Quoted 44 lines · Felix Fietkau

Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets.
For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO.

---
 net/ipv4/tcp_offload.c   | 65 ++++++++++++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  3 ++
 2 files changed, 68 insertions(+)
net/ipv4/tcp_offload.c +26 -0
index fab0973f995b..c493e95e09a5 100644
@@ -28,6 +28,68 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq,
 	}
 }
 
+static void __tcpv4_gso_segment_csum(struct sk_buff *seg,
+				     __be32 *oldip, __be32 *newip,
+				     __be16 *oldport, __be16 *newport)
+{
+	struct tcphdr *th;
+	struct iphdr *iph;
+
+	if (*oldip == *newip && *oldport == *newport)
+		return;
+
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+
+	inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true);
+	inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false);
+	*oldport = *newport;
+
+	csum_replace4(&iph->check, *oldip, *newip);
+	*oldip = *newip;
+}
+
+static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs)
+{
+	struct sk_buff *seg;
+	struct tcphdr *th, *th2;
+	struct iphdr *iph, *iph2;

net/ipv4/tcp_offload.c static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, · line 66

Quoted 10 lines
net/ipv4/tcp_offload.c +10 -0
+
+	seg = segs;
+	th = tcp_hdr(seg);
+	iph = ip_hdr(seg);
+	th2 = tcp_hdr(seg->next);
+	iph2 = ip_hdr(seg->next);
+
+	if (!(*(u32 *)&th->source ^ *(u32 *)&th2->source) &&
+	    iph->daddr == iph2->daddr && iph->saddr == iph2->saddr)
+		return segs;

As mentioned in previous revisions, I think a problem with this approach is that the stack could make other changes to the TCP header after the GRO stage, that are unnoticed here and could cause csum corruption, if the egress device does not recompute the packet csum.

Quoted 8 lines
Quoted 1 line

On segmentation, each packet keeps its original TCP header and csum. If the stack makes changes, they apply to the first packet only. I don't see how we could get csum corruption.

You are right. I did not take in account that such changes (to the first skb) are not reflected to the frag_list at segmentation time. The end result could be different from what the user/admin is expecting, but at least should not impact drops.

Side note: alike UDP, this is not supporting IPv6 NAT...

I will add that for both in the next version.

Thanks,

- Felix

#10 lore raw

PATCH v3 3/6 net: add code for TCP fraglist GRO

In reply to Felix Fietkau

This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.

---
 net/ipv4/tcp_offload.c   | 22 ++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  9 +++++++++
 2 files changed, 31 insertions(+)
net/ipv4/tcp_offload.c +22 -0
index c493e95e09a5..ffd6b7a4163a 100644
@@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
 	flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq);
 	flush |= skb_cmp_decrypted(p, skb);
 
+	if (NAPI_GRO_CB(p)->is_flist) {
+		flush |= (__force int)(flags ^ tcp_flag_word(th2));
+		flush |= skb->ip_summed != p->ip_summed;
+		flush |= skb->csum_level != p->csum_level;
+		flush |= !pskb_may_pull(skb, skb_gro_offset(skb));
+		flush |= NAPI_GRO_CB(p)->count >= 64;
+
+		if (flush || skb_gro_receive_list(p, skb))
+			mss = 1;
+
+		goto out_check_final;
+	}
+
 	if (flush || skb_gro_receive(p, skb)) {
 		mss = 1;
 		goto out_check_final;
@@ -398,6 +411,15 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff)
 	const struct iphdr *iph = ip_hdr(skb);
 	struct tcphdr *th = tcp_hdr(skb);
 
+	if (NAPI_GRO_CB(skb)->is_flist) {
+		skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV4;
+		skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count;
+
+		__skb_incr_checksum_unnecessary(skb);
+
+		return 0;
+	}
+
 	th->check = ~tcp_v4_check(skb->len - thoff, iph->saddr,
 				  iph->daddr, 0);
 
net/ipv6/tcpv6_offload.c +9 -0
index b3b8e1f6b92a..c97d55cf036f 100644
@@ -32,6 +32,15 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)
 	const struct ipv6hdr *iph = ipv6_hdr(skb);
 	struct tcphdr *th = tcp_hdr(skb);
 
+	if (NAPI_GRO_CB(skb)->is_flist) {
+		skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV6;
+		skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count;
+
+		__skb_incr_checksum_unnecessary(skb);
+
+		return 0;
+	}
+
 	th->check = ~tcp_v6_check(skb->len - thoff, &iph->saddr,
 				  &iph->daddr, 0);
 	skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV6;
Signature · 1 line
2.44.0
#11 lore raw

Re: [PATCH v3 net-next v3 3/6] net: add code for TCP fraglist GRO

Quoted 16 lines · Felix Fietkau

This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.

---
 net/ipv4/tcp_offload.c   | 22 ++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  9 +++++++++
 2 files changed, 31 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index c493e95e09a5..ffd6b7a4163a 100644

net/ipv4/tcp_offload.c struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) · line 335

Quoted 4 lines
net/ipv4/tcp_offload.c +1 -0
@@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
        flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq);
        flush |= skb_cmp_decrypted(p, skb);
 
+       if (NAPI_GRO_CB(p)->is_flist) {

Please add unlikely() for all NAPI_GRO_CB(p)->is_flist checks added in this patch.

Quoted 54 lines
net/ipv4/tcp_offload.c +21 -0
+               flush |= (__force int)(flags ^ tcp_flag_word(th2));
+               flush |= skb->ip_summed != p->ip_summed;
+               flush |= skb->csum_level != p->csum_level;
+               flush |= !pskb_may_pull(skb, skb_gro_offset(skb));
+               flush |= NAPI_GRO_CB(p)->count >= 64;
+
+               if (flush || skb_gro_receive_list(p, skb))
+                       mss = 1;
+
+               goto out_check_final;
+       }
+
        if (flush || skb_gro_receive(p, skb)) {
                mss = 1;
                goto out_check_final;
@@ -398,6 +411,15 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff)
        const struct iphdr *iph = ip_hdr(skb);
        struct tcphdr *th = tcp_hdr(skb);
 
+       if (NAPI_GRO_CB(skb)->is_flist) {
+               skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV4;
+               skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count;
+
+               __skb_incr_checksum_unnecessary(skb);
+
+               return 0;
+       }
+
        th->check = ~tcp_v4_check(skb->len - thoff, iph->saddr,
                                  iph->daddr, 0);
 
net/ipv6/tcpv6_offload.c +9 -1
index b3b8e1f6b92a..c97d55cf036f 100644
@@ -32,6 +32,15 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)
        const struct ipv6hdr *iph = ipv6_hdr(skb);
        struct tcphdr *th = tcp_hdr(skb);
 
+       if (NAPI_GRO_CB(skb)->is_flist) {
+               skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV6;
+               skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count;
+
+               __skb_incr_checksum_unnecessary(skb);
+
+               return 0;
+       }
+
        th->check = ~tcp_v6_check(skb->len - thoff, &iph->saddr,
                                  &iph->daddr, 0);
        skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV6;
--

2.44.0

#12 lore raw

Re: [PATCH v3 net-next v3 3/6] net: add code for TCP fraglist GRO

Quoted 20 lines · Eric Dumazet
Quoted 19 lines · Felix Fietkau

This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.

---
 net/ipv4/tcp_offload.c   | 22 ++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  9 +++++++++
 2 files changed, 31 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index c493e95e09a5..ffd6b7a4163a 100644
@@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
        flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq);
        flush |= skb_cmp_decrypted(p, skb);
 
Quoted 5 lines

net/ipv4/tcp_offload.c struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) · line 335

Quoted 1 line
net/ipv4/tcp_offload.c +1 -0
+       if (NAPI_GRO_CB(p)->is_flist) {

Please add unlikely() for all NAPI_GRO_CB(p)->is_flist checks added in this patch.

Will do, thanks.

- Felix

maintainer
#13 lore raw

Re: [PATCH v3 net-next v3 3/6] net: add code for TCP fraglist GRO

In reply to Felix Fietkau
Quoted 15 lines · Felix Fietkau

This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.

---
 net/ipv4/tcp_offload.c   | 22 ++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  9 +++++++++
 2 files changed, 31 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index c493e95e09a5..ffd6b7a4163a 100644

net/ipv4/tcp_offload.c struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) · line 339

Quoted 8 lines
net/ipv4/tcp_offload.c +5 -0
@@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
 	flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq);
 	flush |= skb_cmp_decrypted(p, skb);
 
+	if (NAPI_GRO_CB(p)->is_flist) {
+		flush |= (__force int)(flags ^ tcp_flag_word(th2));
+		flush |= skb->ip_summed != p->ip_summed;
+		flush |= skb->csum_level != p->csum_level;
+		flush |= !pskb_may_pull(skb, skb_gro_offset(skb));

I'm sorry, I'm lagging behind. I think the TCP flags handling here is correct - preserving the original ones should work.

The question a made WRT 2 above checks being non necessary/redundant:

		flush |= (__force int)(flags ^ tcp_flag_word(th2));
		flush |= !pskb_may_pull(skb, skb_gro_offset(skb));

still stands, I think.

Thanks,

Paolo

#14 lore raw

Re: [PATCH v3 net-next v3 3/6] net: add code for TCP fraglist GRO

Quoted 28 lines · Paolo Abeni
Quoted 15 lines · Felix Fietkau

This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO.

---
 net/ipv4/tcp_offload.c   | 22 ++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c |  9 +++++++++
 2 files changed, 31 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index c493e95e09a5..ffd6b7a4163a 100644

net/ipv4/tcp_offload.c struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) · line 339

Quoted 8 lines
net/ipv4/tcp_offload.c +5 -0
@@ -332,6 +332,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
 	flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq);
 	flush |= skb_cmp_decrypted(p, skb);
 
+	if (NAPI_GRO_CB(p)->is_flist) {
+		flush |= (__force int)(flags ^ tcp_flag_word(th2));
+		flush |= skb->ip_summed != p->ip_summed;
+		flush |= skb->csum_level != p->csum_level;
+		flush |= !pskb_may_pull(skb, skb_gro_offset(skb));

I'm sorry, I'm lagging behind. I think the TCP flags handling here is correct - preserving the original ones should work.

Quoted 3 lines

The question a made WRT 2 above checks being non necessary/redundant:

		flush |= (__force int)(flags ^ tcp_flag_word(th2));

This one is not redundant, because the earlier flags check includes this part: & ~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH))

Quoted 1 line
		flush |= !pskb_may_pull(skb, skb_gro_offset(skb));

This one looks like a redundant leftover, I will remove it in the next version.

Thanks,

- Felix

#15 lore raw

PATCH v3 4/6 net: create tcp_gro_lookup helper function

In reply to Felix Fietkau

This pulls the flow port matching out of tcp_gro_receive, so that it can be reused for the next change, which adds the TCP fraglist GRO heuristic.

---
 include/net/tcp.h      |  1 +
 net/ipv4/tcp_offload.c | 41 +++++++++++++++++++++++++----------------
 2 files changed, 26 insertions(+), 16 deletions(-)
include/net/tcp.h +1 -0
index ffc9371fe9de..a3f09aa44487 100644
@@ -2198,6 +2198,7 @@ void tcp_v4_destroy_sock(struct sock *sk);
 
 struct sk_buff *tcp_gso_segment(struct sk_buff *skb,
 				netdev_features_t features);
+struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th);
 struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb);
 INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff));
 INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb));
net/ipv4/tcp_offload.c +25 -16

55 lines not on this page · raw message

Signature · 1 line
2.44.0
#17 lore raw

PATCH v3 5/6 net: create tcp_gro_header_pull helper function

In reply to Felix Fietkau

Pull the code out of tcp_gro_receive in order to access the tcp header from tcp4/6_gro_receive.

---
 include/net/tcp.h        |  4 ++-
 net/ipv4/tcp_offload.c   | 55 +++++++++++++++++++++++++---------------
 net/ipv6/tcpv6_offload.c | 18 +++++++++----
 3 files changed, 50 insertions(+), 27 deletions(-)
include/net/tcp.h +3 -1
index a3f09aa44487..52371be181a2 100644
@@ -2198,8 +2198,10 @@ void tcp_v4_destroy_sock(struct sock *sk);
 
 struct sk_buff *tcp_gso_segment(struct sk_buff *skb,
 				netdev_features_t features);
+struct tcphdr *tcp_gro_pull_header(struct sk_buff *skb);
 struct sk_buff *tcp_gro_lookup(struct list_head *head, struct tcphdr *th);
-struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb);
+struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb,
+				struct tcphdr *th);
 INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff));
 INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb));
 INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff));
net/ipv4/tcp_offload.c +34 -21

96 lines not on this page · raw message

net/ipv6/tcpv6_offload.c +13 -5
index c97d55cf036f..c01ace2e9ff0 100644
@@ -16,15 +16,23 @@
 INDIRECT_CALLABLE_SCOPE
 struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
+	struct tcphdr *th;
+
 	/* Don't bother verifying checksum if we're going to flush anyway. */
 	if (!NAPI_GRO_CB(skb)->flush &&
 	    skb_gro_checksum_validate(skb, IPPROTO_TCP,
-				      ip6_gro_compute_pseudo)) {
-		NAPI_GRO_CB(skb)->flush = 1;
-		return NULL;
-	}
+				      ip6_gro_compute_pseudo))
+		goto flush;
+
+	th = tcp_gro_pull_header(skb);
+	if (!th)
+		goto flush;
+
+	return tcp_gro_receive(head, skb, th);
 
-	return tcp_gro_receive(head, skb);
+flush:
+	NAPI_GRO_CB(skb)->flush = 1;
+	return NULL;
 }
 
 INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)
Signature · 1 line
2.44.0
#19 lore raw

PATCH v3 6/6 net: add heuristic for enabling TCP fraglist GRO

In reply to Felix Fietkau

When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.

When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.

Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.

rx-gro-list off: 630 Mbit/s, CPU 35% idle
rx-gro-list on:  770 Mbit/s, CPU 40% idle
---
 net/ipv4/tcp_offload.c   | 30 ++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
net/ipv4/tcp_offload.c +30 -0
index ee5403760775..2ae83f4394dc 100644
@@ -406,6 +406,34 @@ void tcp_gro_complete(struct sk_buff *skb)
 }
 EXPORT_SYMBOL(tcp_gro_complete);
 
+static void tcp4_check_fraglist_gro(struct list_head *head, struct sk_buff *skb,
+				    struct tcphdr *th)
+{
+	const struct iphdr *iph = skb_gro_network_header(skb);
+	struct net *net = dev_net(skb->dev);
+	struct sk_buff *p;
+	struct sock *sk;
+	int iif, sdif;
+
+	if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST))
+		return;
+
+	p = tcp_gro_lookup(head, th);
+	if (p) {
+		NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist;
+		return;
+	}
+
+	inet_get_iif_sdif(skb, &iif, &sdif);
+	sk = __inet_lookup_established(net, net->ipv4.tcp_death_row.hashinfo,
+				       iph->saddr, th->source,
+				       iph->daddr, ntohs(th->dest),
+				       iif, sdif);
+	NAPI_GRO_CB(skb)->is_flist = !sk;
+	if (sk)
+		sock_put(sk);
+}
+
 INDIRECT_CALLABLE_SCOPE
 struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
@@ -421,6 +449,8 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)
 	if (!th)
 		goto flush;
 
+	tcp4_check_fraglist_gro(head, skb, th);
+
 	return tcp_gro_receive(head, skb, th);
 
 flush:
net/ipv6/tcpv6_offload.c +33 -0
index c01ace2e9ff0..1ab45cca3936 100644
@@ -7,12 +7,43 @@
  */
 #include <linux/indirect_call_wrapper.h>
 #include <linux/skbuff.h>
+#include <net/inet6_hashtables.h>
 #include <net/gro.h>
 #include <net/protocol.h>
 #include <net/tcp.h>
 #include <net/ip6_checksum.h>
 #include "ip6_offload.h"
 
+static void tcp6_check_fraglist_gro(struct list_head *head, struct sk_buff *skb,
+				    struct tcphdr *th)
+{
+#if IS_ENABLED(CONFIG_IPV6)
+	const struct ipv6hdr *hdr = skb_gro_network_header(skb);
+	struct net *net = dev_net(skb->dev);
+	struct sk_buff *p;
+	struct sock *sk;
+	int iif, sdif;
+
+	if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST))
+		return;
+
+	p = tcp_gro_lookup(head, th);
+	if (p) {
+		NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist;
+		return;
+	}
+
+	inet6_get_iif_sdif(skb, &iif, &sdif);
+	sk = __inet6_lookup_established(net, net->ipv4.tcp_death_row.hashinfo,
+					&hdr->saddr, th->source,
+					&hdr->daddr, ntohs(th->dest),
+					iif, sdif);
+	NAPI_GRO_CB(skb)->is_flist = !sk;
+	if (sk)
+		sock_put(sk);
+#endif /* IS_ENABLED(CONFIG_IPV6) */
+}
+
 INDIRECT_CALLABLE_SCOPE
 struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
@@ -28,6 +59,8 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)
 	if (!th)
 		goto flush;
 
+	tcp6_check_fraglist_gro(head, skb, th);
+
 	return tcp_gro_receive(head, skb, th);
 
 flush:
Signature · 1 line
2.44.0
#20 lore raw

Re: [PATCH v3 net-next v3 6/6] net: add heuristic for enabling TCP fraglist GRO

Quoted 31 lines · Felix Fietkau

When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.

When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.

Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.

rx-gro-list off: 630 Mbit/s, CPU 35% idle
rx-gro-list on:  770 Mbit/s, CPU 40% idle
---
 net/ipv4/tcp_offload.c   | 30 ++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index ee5403760775..2ae83f4394dc 100644

net/ipv4/tcp_offload.c void tcp_gro_complete(struct sk_buff *skb) · line 413

Quoted 8 lines
net/ipv4/tcp_offload.c +5 -0
@@ -406,6 +406,34 @@ void tcp_gro_complete(struct sk_buff *skb)
 }
 EXPORT_SYMBOL(tcp_gro_complete);
 
+static void tcp4_check_fraglist_gro(struct list_head *head, struct sk_buff *skb,
+                                   struct tcphdr *th)
+{
+       const struct iphdr *iph = skb_gro_network_header(skb);
+       struct net *net = dev_net(skb->dev);

Could you defer the initializations of iph and net after the NETIF_F_GRO_FRAGLIST check ?

dev_net() has an implicit READ_ONCE() ...

Quoted 46 lines
net/ipv4/tcp_offload.c +25 -0
+       struct sk_buff *p;
+       struct sock *sk;
+       int iif, sdif;
+
+       if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST))
+               return;
+
+       p = tcp_gro_lookup(head, th);
+       if (p) {
+               NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist;
+               return;
+       }
+
+       inet_get_iif_sdif(skb, &iif, &sdif);
+       sk = __inet_lookup_established(net, net->ipv4.tcp_death_row.hashinfo,
+                                      iph->saddr, th->source,
+                                      iph->daddr, ntohs(th->dest),
+                                      iif, sdif);
+       NAPI_GRO_CB(skb)->is_flist = !sk;
+       if (sk)
+               sock_put(sk);
+}
+
 INDIRECT_CALLABLE_SCOPE
 struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
@@ -421,6 +449,8 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)
        if (!th)
                goto flush;
 
+       tcp4_check_fraglist_gro(head, skb, th);
+
        return tcp_gro_receive(head, skb, th);
 
 flush:
net/ipv6/tcpv6_offload.c +1 -0
index c01ace2e9ff0..1ab45cca3936 100644
@@ -7,12 +7,43 @@
  */
 #include <linux/indirect_call_wrapper.h>
 #include <linux/skbuff.h>
+#include <net/inet6_hashtables.h>
 #include <net/gro.h>
 #include <net/protocol.h>

net/ipv6/tcpv6_offload.c line 22

Quoted 10 lines
net/ipv6/tcpv6_offload.c +6 -0
 #include <net/tcp.h>
 #include <net/ip6_checksum.h>
 #include "ip6_offload.h"
 
+static void tcp6_check_fraglist_gro(struct list_head *head, struct sk_buff *skb,
+                                   struct tcphdr *th)
+{
+#if IS_ENABLED(CONFIG_IPV6)
+       const struct ipv6hdr *hdr = skb_gro_network_header(skb);
+       struct net *net = dev_net(skb->dev);

Same remark here.

Quoted 39 lines
net/ipv6/tcpv6_offload.c +26 -1
+       struct sk_buff *p;
+       struct sock *sk;
+       int iif, sdif;
+
+       if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST))
+               return;
+
+       p = tcp_gro_lookup(head, th);
+       if (p) {
+               NAPI_GRO_CB(skb)->is_flist = NAPI_GRO_CB(p)->is_flist;
+               return;
+       }
+
+       inet6_get_iif_sdif(skb, &iif, &sdif);
+       sk = __inet6_lookup_established(net, net->ipv4.tcp_death_row.hashinfo,
+                                       &hdr->saddr, th->source,
+                                       &hdr->daddr, ntohs(th->dest),
+                                       iif, sdif);
+       NAPI_GRO_CB(skb)->is_flist = !sk;
+       if (sk)
+               sock_put(sk);
+#endif /* IS_ENABLED(CONFIG_IPV6) */
+}
+
 INDIRECT_CALLABLE_SCOPE
 struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)
 {
@@ -28,6 +59,8 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)
        if (!th)
                goto flush;
 
+       tcp6_check_fraglist_gro(head, skb, th);
+
        return tcp_gro_receive(head, skb, th);
 
 flush:
--

2.44.0

#21 lore raw

Re: [PATCH v3 net-next v3 6/6] net: add heuristic for enabling TCP fraglist GRO

Quoted 44 lines · Eric Dumazet
Quoted 31 lines · Felix Fietkau

When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated.
One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP.

When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this.

Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps.

rx-gro-list off: 630 Mbit/s, CPU 35% idle
rx-gro-list on:  770 Mbit/s, CPU 40% idle
---
 net/ipv4/tcp_offload.c   | 30 ++++++++++++++++++++++++++++++
 net/ipv6/tcpv6_offload.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
net/ipv4/tcp_offload.c +0 -0
index ee5403760775..2ae83f4394dc 100644

net/ipv4/tcp_offload.c void tcp_gro_complete(struct sk_buff *skb) · line 413

Quoted 8 lines
net/ipv4/tcp_offload.c +5 -0
@@ -406,6 +406,34 @@ void tcp_gro_complete(struct sk_buff *skb)
 }
 EXPORT_SYMBOL(tcp_gro_complete);
 
+static void tcp4_check_fraglist_gro(struct list_head *head, struct sk_buff *skb,
+                                   struct tcphdr *th)
+{
+       const struct iphdr *iph = skb_gro_network_header(skb);
+       struct net *net = dev_net(skb->dev);

Could you defer the initializations of iph and net after the NETIF_F_GRO_FRAGLIST check ?

Quoted 1 line

dev_net() has an implicit READ_ONCE() ...

Will do, thanks.

- Felix