Skip to contentSkip to conversation

gro: some minor optimization

PATCH In review Paolo Abeni 4 participants 12 messages Feb 3, 2022

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

cover gro: some minor optimization

This series collects a few small optimization for the GRO engine. I measure a 10% performance improvements in micro-benchmarks around dev_gro_receive(), but deltas are within noise range in tput tests.

Still with big TCP coming every cycle saved from the GRO engine will count - I hope ;)

The only change from the RFC is in patch 2, as per Alexander feedback

Paolo Abeni (3):
net: gro: avoid re-computing truesize twice on recycle
net: gro: minor optimization for dev_gro_receive()
net: gro: register gso and gro offload on separate lists

 include/linux/netdevice.h |   3 +-
 include/net/gro.h         |  52 +++++++++---------
 net/core/gro.c            | 109 +++++++++++++++++++++-----------------
 3 files changed, 91 insertions(+), 73 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH 1/3 net: gro: avoid re-computing truesize twice on recycle

After commit 5e10da5385d2 ("skbuff: allow 'slow_gro' for skb carring sock reference") and commit af352460b465 ("net: fix GRO skb truesize update") the truesize of freed skb is properly updated by the GRO engine, we don't need anymore resetting it at recycle time.

---
 net/core/gro.c | 1 -
 1 file changed, 1 deletion(-)
net/core/gro.c +0 -1
index a11b286d1495..d43d42215bdb 100644
@@ -634,7 +634,6 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
 	skb->encapsulation = 0;
 	skb_shinfo(skb)->gso_type = 0;
-	skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
 	if (unlikely(skb->slow_gro)) {
 		skb_orphan(skb);
 		skb_ext_reset(skb);
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH net-next 1/3] net: gro: avoid re-computing truesize twice on recycle

From: Paolo Abeni <[email protected]>
Date: Thu, 3 Feb 2022 16:48:21 +0100

Quoted 3 lines

After commit 5e10da5385d2 ("skbuff: allow 'slow_gro' for skb carring sock reference") and commit af352460b465 ("net: fix GRO skb truesize update") the truesize of freed skb is properly updated

                                        ^^^^^

One nit here, I'd change this to "truesize of skb with stolen head" or so. It took me a bit of time to get why we should update the truesize of skb already freed (: Right, napi_reuse_skb() makes use of stolen-data skbs.

Quoted 21 lines

by the GRO engine, we don't need anymore resetting it at recycle time.

---
 net/core/gro.c | 1 -
 1 file changed, 1 deletion(-)
net/core/gro.c +0 -1
index a11b286d1495..d43d42215bdb 100644
@@ -634,7 +634,6 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
 	skb->encapsulation = 0;
 	skb_shinfo(skb)->gso_type = 0;
-	skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
 	if (unlikely(skb->slow_gro)) {
 		skb_orphan(skb);
 		skb_ext_reset(skb);
Signature · 1 line
2.34.1

Thanks,
Al

author maintainer
#3 lore raw

PATCH 2/3 net: gro: minor optimization for dev_gro_receive()

In reply to Paolo Abeni

While inspecting some perf report, I noticed that the compiler emits suboptimal code for the napi CB initialization, fetching and storing multiple times the memory for flags bitfield. This is with gcc 10.3.1, but I observed the same with older compiler versions.

We can help the compiler to do a nicer work clearing several fields at once using an u32 alias. The generated code is quite smaller, with the same number of conditional.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 0000000000000357 dev_gro_receive

After:
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

RFC -> v1:
- use __struct_group to delimt the zeroed area (Alexander)

---
 include/net/gro.h | 52 +++++++++++++++++++++++++----------------------
 net/core/gro.c    | 18 +++++++---------
 2 files changed, 35 insertions(+), 35 deletions(-)
include/net/gro.h +28 -24
index 8f75802d50fd..fa1bb0f0ad28 100644
@@ -29,46 +29,50 @@ struct napi_gro_cb {
 	/* Number of segments aggregated. */
 	u16	count;
 
-	/* Start offset for remote checksum offload */
-	u16	gro_remcsum_start;
+	/* Used in ipv6_gro_receive() and foo-over-udp */
+	u16	proto;
 
 	/* jiffies when first packet was created/queued */
 	unsigned long age;
 
-	/* Used in ipv6_gro_receive() and foo-over-udp */
-	u16	proto;
+	/* portion of the cb set to zero at every gro iteration */
+	__struct_group(/* no tag */, zeroed, /* no attrs */,
+
+		/* Start offset for remote checksum offload */
+		u16	gro_remcsum_start;
 
-	/* This is non-zero if the packet may be of the same flow. */
-	u8	same_flow:1;
+		/* This is non-zero if the packet may be of the same flow. */
+		u8	same_flow:1;
 
-	/* Used in tunnel GRO receive */
-	u8	encap_mark:1;
+		/* Used in tunnel GRO receive */
+		u8	encap_mark:1;
 
-	/* GRO checksum is valid */
-	u8	csum_valid:1;
+		/* GRO checksum is valid */
+		u8	csum_valid:1;
 
-	/* Number of checksums via CHECKSUM_UNNECESSARY */
-	u8	csum_cnt:3;
+		/* Number of checksums via CHECKSUM_UNNECESSARY */
+		u8	csum_cnt:3;
 
-	/* Free the skb? */
-	u8	free:2;
+		/* Free the skb? */
+		u8	free:2;
 #define NAPI_GRO_FREE		  1
 #define NAPI_GRO_FREE_STOLEN_HEAD 2
 
-	/* Used in foo-over-udp, set in udp[46]_gro_receive */
-	u8	is_ipv6:1;
+		/* Used in foo-over-udp, set in udp[46]_gro_receive */
+		u8	is_ipv6:1;
 
-	/* Used in GRE, set in fou/gue_gro_receive */
-	u8	is_fou:1;
+		/* Used in GRE, set in fou/gue_gro_receive */
+		u8	is_fou:1;
 
-	/* Used to determine if flush_id can be ignored */
-	u8	is_atomic:1;
+		/* Used to determine if flush_id can be ignored */
+		u8	is_atomic:1;
 
-	/* Number of gro_receive callbacks this packet already went through */
-	u8 recursion_counter:4;
+		/* Number of gro_receive callbacks this packet already went through */
+		u8 recursion_counter:4;
 
-	/* GRO is done by frag_list pointer chaining. */
-	u8	is_flist:1;
+		/* GRO is done by frag_list pointer chaining. */
+		u8	is_flist:1;
+	);
 
 	/* used to support CHECKSUM_COMPLETE for tunneling protocols */
 	__wsum	csum;
net/core/gro.c +7 -11
index d43d42215bdb..fc56be9408c7 100644
@@ -435,6 +435,9 @@ static void gro_flush_oldest(struct napi_struct *napi, struct list_head *head)
 	napi_gro_complete(napi, oldest);
 }
 
+#define zeroed_len	(offsetof(struct napi_gro_cb, zeroed_end) - \
+			 offsetof(struct napi_gro_cb, zeroed_start))
+
 static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
 {
 	u32 bucket = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1);
@@ -459,29 +462,22 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 
 		skb_set_network_header(skb, skb_gro_offset(skb));
 		skb_reset_mac_len(skb);
-		NAPI_GRO_CB(skb)->same_flow = 0;
+		BUILD_BUG_ON(sizeof_field(struct napi_gro_cb, zeroed) != sizeof(u32));
+		BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
+					 sizeof(u32))); /* Avoid slow unaligned acc */
+		*(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
 		NAPI_GRO_CB(skb)->flush = skb_is_gso(skb) || skb_has_frag_list(skb);
-		NAPI_GRO_CB(skb)->free = 0;
-		NAPI_GRO_CB(skb)->encap_mark = 0;
-		NAPI_GRO_CB(skb)->recursion_counter = 0;
-		NAPI_GRO_CB(skb)->is_fou = 0;
 		NAPI_GRO_CB(skb)->is_atomic = 1;
-		NAPI_GRO_CB(skb)->gro_remcsum_start = 0;
 
 		/* Setup for GRO checksum validation */
 		switch (skb->ip_summed) {
 		case CHECKSUM_COMPLETE:
 			NAPI_GRO_CB(skb)->csum = skb->csum;
 			NAPI_GRO_CB(skb)->csum_valid = 1;
-			NAPI_GRO_CB(skb)->csum_cnt = 0;
 			break;
 		case CHECKSUM_UNNECESSARY:
 			NAPI_GRO_CB(skb)->csum_cnt = skb->csum_level + 1;
-			NAPI_GRO_CB(skb)->csum_valid = 0;
 			break;
-		default:
-			NAPI_GRO_CB(skb)->csum_cnt = 0;
-			NAPI_GRO_CB(skb)->csum_valid = 0;
 		}
 
 		pp = INDIRECT_CALL_INET(ptype->callbacks.gro_receive,
Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH net-next 2/3] net: gro: minor optimization for dev_gro_receive()

From: Paolo Abeni <[email protected]>
Date: Thu, 3 Feb 2022 16:48:22 +0100

Quoted 14 lines

While inspecting some perf report, I noticed that the compiler emits suboptimal code for the napi CB initialization, fetching and storing multiple times the memory for flags bitfield. This is with gcc 10.3.1, but I observed the same with older compiler versions.

We can help the compiler to do a nicer work clearing several fields at once using an u32 alias. The generated code is quite smaller, with the same number of conditional.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 0000000000000357 dev_gro_receive

Quoted 5 lines

After:
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

RFC -> v1:
- use __struct_group to delimt the zeroed area (Alexander)

"delimit"?

Quoted 17 lines
---
 include/net/gro.h | 52 +++++++++++++++++++++++++----------------------
 net/core/gro.c    | 18 +++++++---------
 2 files changed, 35 insertions(+), 35 deletions(-)
include/net/gro.h +0 -2
index 8f75802d50fd..fa1bb0f0ad28 100644
@@ -29,46 +29,50 @@ struct napi_gro_cb {
 	/* Number of segments aggregated. */
 	u16	count;
 
-	/* Start offset for remote checksum offload */
-	u16	gro_remcsum_start;

include/net/gro.h struct napi_gro_cb { · line 39

Quoted 10 lines
include/net/gro.h +4 -2
+	/* Used in ipv6_gro_receive() and foo-over-udp */
+	u16	proto;
 
 	/* jiffies when first packet was created/queued */
 	unsigned long age;
 
-	/* Used in ipv6_gro_receive() and foo-over-udp */
-	u16	proto;
+	/* portion of the cb set to zero at every gro iteration */
+	__struct_group(/* no tag */, zeroed, /* no attrs */,

Oh, only after sending this in my reply I noticed that there's a shorthand for this

struct_group(name, ...)

means exactly

__struct_group(/* no tag */, name, /* no attrs */, ...)

Sorry for that. I got confused by that Kees used __struct_group() directly in one place although there was a possibility to use short struct_group() instead.
Since there should already be a v2 (see below), probably worth changing.

Quoted 64 lines
include/net/gro.h +24 -20
+
+		/* Start offset for remote checksum offload */
+		u16	gro_remcsum_start;
 
-	/* This is non-zero if the packet may be of the same flow. */
-	u8	same_flow:1;
+		/* This is non-zero if the packet may be of the same flow. */
+		u8	same_flow:1;
 
-	/* Used in tunnel GRO receive */
-	u8	encap_mark:1;
+		/* Used in tunnel GRO receive */
+		u8	encap_mark:1;
 
-	/* GRO checksum is valid */
-	u8	csum_valid:1;
+		/* GRO checksum is valid */
+		u8	csum_valid:1;
 
-	/* Number of checksums via CHECKSUM_UNNECESSARY */
-	u8	csum_cnt:3;
+		/* Number of checksums via CHECKSUM_UNNECESSARY */
+		u8	csum_cnt:3;
 
-	/* Free the skb? */
-	u8	free:2;
+		/* Free the skb? */
+		u8	free:2;
 #define NAPI_GRO_FREE		  1
 #define NAPI_GRO_FREE_STOLEN_HEAD 2
 
-	/* Used in foo-over-udp, set in udp[46]_gro_receive */
-	u8	is_ipv6:1;
+		/* Used in foo-over-udp, set in udp[46]_gro_receive */
+		u8	is_ipv6:1;
 
-	/* Used in GRE, set in fou/gue_gro_receive */
-	u8	is_fou:1;
+		/* Used in GRE, set in fou/gue_gro_receive */
+		u8	is_fou:1;
 
-	/* Used to determine if flush_id can be ignored */
-	u8	is_atomic:1;
+		/* Used to determine if flush_id can be ignored */
+		u8	is_atomic:1;
 
-	/* Number of gro_receive callbacks this packet already went through */
-	u8 recursion_counter:4;
+		/* Number of gro_receive callbacks this packet already went through */
+		u8 recursion_counter:4;
 
-	/* GRO is done by frag_list pointer chaining. */
-	u8	is_flist:1;
+		/* GRO is done by frag_list pointer chaining. */
+		u8	is_flist:1;
+	);
 
 	/* used to support CHECKSUM_COMPLETE for tunneling protocols */
 	__wsum	csum;
net/core/gro.c +0 -0
index d43d42215bdb..fc56be9408c7 100644

net/core/gro.c static void gro_flush_oldest(struct napi_struct *napi, struct list_head *head) · line 440

Quoted 6 lines
net/core/gro.c +3 -0
@@ -435,6 +435,9 @@ static void gro_flush_oldest(struct napi_struct *napi, struct list_head *head)
 	napi_gro_complete(napi, oldest);
 }
 
+#define zeroed_len	(offsetof(struct napi_gro_cb, zeroed_end) - \
+			 offsetof(struct napi_gro_cb, zeroed_start))
+

A leftover from the RFC I guess? It's not used anywhere in the code, so it doesn't break build.

net/core/gro.c static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff · line 468

Quoted 12 lines
net/core/gro.c +4 -1
 static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
 {
 	u32 bucket = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1);
@@ -459,29 +462,22 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 
 		skb_set_network_header(skb, skb_gro_offset(skb));
 		skb_reset_mac_len(skb);
-		NAPI_GRO_CB(skb)->same_flow = 0;
+		BUILD_BUG_ON(sizeof_field(struct napi_gro_cb, zeroed) != sizeof(u32));
+		BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
+					 sizeof(u32))); /* Avoid slow unaligned acc */
+		*(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;

Looks relatively elegant and self-explanatory to me, nice.

Quoted 27 lines
net/core/gro.c +0 -10
 		NAPI_GRO_CB(skb)->flush = skb_is_gso(skb) || skb_has_frag_list(skb);
-		NAPI_GRO_CB(skb)->free = 0;
-		NAPI_GRO_CB(skb)->encap_mark = 0;
-		NAPI_GRO_CB(skb)->recursion_counter = 0;
-		NAPI_GRO_CB(skb)->is_fou = 0;
 		NAPI_GRO_CB(skb)->is_atomic = 1;
-		NAPI_GRO_CB(skb)->gro_remcsum_start = 0;
 
 		/* Setup for GRO checksum validation */
 		switch (skb->ip_summed) {
 		case CHECKSUM_COMPLETE:
 			NAPI_GRO_CB(skb)->csum = skb->csum;
 			NAPI_GRO_CB(skb)->csum_valid = 1;
-			NAPI_GRO_CB(skb)->csum_cnt = 0;
 			break;
 		case CHECKSUM_UNNECESSARY:
 			NAPI_GRO_CB(skb)->csum_cnt = skb->csum_level + 1;
-			NAPI_GRO_CB(skb)->csum_valid = 0;
 			break;
-		default:
-			NAPI_GRO_CB(skb)->csum_cnt = 0;
-			NAPI_GRO_CB(skb)->csum_valid = 0;
 		}
 
 		pp = INDIRECT_CALL_INET(ptype->callbacks.gro_receive,
Signature · 1 line
2.34.1

Al

#5 lore raw

Re: [PATCH net-next 2/3] net: gro: minor optimization for dev_gro_receive()

In reply to Paolo Abeni
Quoted 39 lines · Paolo Abeni

While inspecting some perf report, I noticed that the compiler emits suboptimal code for the napi CB initialization, fetching and storing multiple times the memory for flags bitfield. This is with gcc 10.3.1, but I observed the same with older compiler versions.

We can help the compiler to do a nicer work clearing several fields at once using an u32 alias. The generated code is quite smaller, with the same number of conditional.

Before:
objdump -t net/core/gro.o | grep " F .text" 0000000000000bb0 l     F .text  0000000000000357 dev_gro_receive

After:
0000000000000bb0 l     F .text  000000000000033c dev_gro_receive

RFC -> v1:
 - use __struct_group to delimt the zeroed area (Alexander)

---
 include/net/gro.h | 52 +++++++++++++++++++++++++-------------------- --
 net/core/gro.c    | 18 +++++++---------

 2 files changed, 35 insertions(+), 35 deletions(-)
include/net/gro.h +0 -0
index 8f75802d50fd..fa1bb0f0ad28 100644
@@ -29,46 +29,50 @@ struct napi_gro_cb {
         /* Number of segments aggregated. */
         u16     count;


-       /* Start offset for remote checksum offload */ -       u16     gro_remcsum_start;
+       /* Used in ipv6_gro_receive() and foo-over-udp */ +       u16     proto;

Quoted 8 lines


        /* jiffies when first packet was created/queued */ unsigned long age;

-       /* Used in ipv6_gro_receive() and foo-over-udp */ -       u16     proto;
+       /* portion of the cb set to zero at every gro iteration */ +       __struct_group(/* no tag */, zeroed, /* no attrs */,

Any specific reason for using __struct_group here rather than the struct_group macro instead?

author maintainer
#6 lore raw

Re: [PATCH net-next 2/3] net: gro: minor optimization for dev_gro_receive()

Quoted 43 lines · Alexander H Duyck
Quoted 42 lines · Paolo Abeni

While inspecting some perf report, I noticed that the compiler emits suboptimal code for the napi CB initialization, fetching and storing multiple times the memory for flags bitfield. This is with gcc 10.3.1, but I observed the same with older compiler versions.

We can help the compiler to do a nicer work clearing several fields at once using an u32 alias. The generated code is quite smaller, with the same number of conditional.

Before:
objdump -t net/core/gro.o | grep " F .text" 0000000000000bb0 l     F .text  0000000000000357 dev_gro_receive

After:
0000000000000bb0 l     F .text  000000000000033c dev_gro_receive

RFC -> v1:
 - use __struct_group to delimt the zeroed area (Alexander)

---
 include/net/gro.h | 52 +++++++++++++++++++++++++-------------------- --
 net/core/gro.c    | 18 +++++++---------

 2 files changed, 35 insertions(+), 35 deletions(-)
include/net/gro.h +0 -0
index 8f75802d50fd..fa1bb0f0ad28 100644
@@ -29,46 +29,50 @@ struct napi_gro_cb {
         /* Number of segments aggregated. */
         u16     count;


-       /* Start offset for remote checksum offload */ -       u16     gro_remcsum_start;
+       /* Used in ipv6_gro_receive() and foo-over-udp */ +       u16     proto;

        /* jiffies when first packet was created/queued */ unsigned long age;

Quoted 8 lines
Quoted 5 lines


-       /* Used in ipv6_gro_receive() and foo-over-udp */ -       u16     proto;
+       /* portion of the cb set to zero at every gro iteration */ +       __struct_group(/* no tag */, zeroed, /* no attrs */,

Any specific reason for using __struct_group here rather than the struct_group macro instead?

Just sheer ignorance on my side. I'll fix on the next iteration.

Thanks!

Paolo

author maintainer
#7 lore raw

PATCH 3/3 net: gro: register gso and gro offload on separate lists

In reply to Paolo Abeni

So that we know each element in gro_list has valid gro callbacks (and the same for gso). This allows dropping a bunch of conditional in fastpath.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

After:
0000000000000bb0 l F .text 0000000000000325 dev_gro_receive

---
 include/linux/netdevice.h |  3 +-
 net/core/gro.c            | 90 +++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 37 deletions(-)
include/linux/netdevice.h +2 -1
index 3213c7227b59..406cb457d788 100644
@@ -2564,7 +2564,8 @@ struct packet_offload {
 	__be16			 type;	/* This is really htons(ether_type). */
 	u16			 priority;
 	struct offload_callbacks callbacks;
-	struct list_head	 list;
+	struct list_head	 gro_list;
+	struct list_head	 gso_list;
 };
 
 /* often modified stats are per-CPU, other are shared (netdev->stats) */
net/core/gro.c +54 -36

176 lines not on this page · raw message

Signature · 1 line
2.34.1
#8 lore raw

Re: [PATCH net-next 3/3] net: gro: register gso and gro offload on separate lists

Quoted 23 lines · Paolo Abeni

So that we know each element in gro_list has valid gro callbacks (and the same for gso). This allows dropping a bunch of conditional in fastpath.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

After:
0000000000000bb0 l F .text 0000000000000325 dev_gro_receive

---
 include/linux/netdevice.h |  3 +-
 net/core/gro.c            | 90 +++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 37 deletions(-)
include/linux/netdevice.h +0 -0
index 3213c7227b59..406cb457d788 100644

include/linux/netdevice.h struct packet_offload { · line 2570

Quoted 8 lines
include/linux/netdevice.h +2 -1
@@ -2564,7 +2564,8 @@ struct packet_offload {
        __be16                   type;  /* This is really htons(ether_type). */
        u16                      priority;
        struct offload_callbacks callbacks;
-       struct list_head         list;
+       struct list_head         gro_list;
+       struct list_head         gso_list;
 };
 

On the other hand, this makes this object bigger, increasing the risk of spanning cache lines.

It would be nice to group all struct packet_offload together in the same section to increase data locality.

I played in the past with a similar idea, but splitting struct packet_offload in two structures, one for GRO, one for GSO. (Note that GSO is hardly ever use with modern NIC)

But the gains were really marginal.

#9 lore raw

Re: [PATCH net-next 3/3] net: gro: register gso and gro offload on separate lists

From: Eric Dumazet <[email protected]>
Date: Thu, 3 Feb 2022 08:11:43 -0800

Quoted 19 lines · Paolo Abeni
Quoted 18 lines

So that we know each element in gro_list has valid gro callbacks (and the same for gso). This allows dropping a bunch of conditional in fastpath.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

After:
0000000000000bb0 l F .text 0000000000000325 dev_gro_receive

---
 include/linux/netdevice.h |  3 +-
 net/core/gro.c            | 90 +++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 37 deletions(-)
Quoted 16 lines

include/linux/netdevice.h struct packet_offload { · line 2570

Quoted 13 lines
include/linux/netdevice.h +2 -1
index 3213c7227b59..406cb457d788 100644
@@ -2564,7 +2564,8 @@ struct packet_offload {
        __be16                   type;  /* This is really htons(ether_type). */
        u16                      priority;
        struct offload_callbacks callbacks;
-       struct list_head         list;
+       struct list_head         gro_list;
+       struct list_head         gso_list;
 };
 

On the other hand, this makes this object bigger, increasing the risk of spanning cache lines.

As you said, GSO callbacks are barely used with most modern NICs. Plus GRO and GSO callbacks are used in the completely different operations.
gro_list occupies the same place where the list previously was. Does it make a lot of sense to care about gso_list being placed in a cacheline separate from gro_list?

Quoted 9 lines

It would be nice to group all struct packet_offload together in the same section to increase data locality.

I played in the past with a similar idea, but splitting struct packet_offload in two structures, one for GRO, one for GSO. (Note that GSO is hardly ever use with modern NIC)

But the gains were really marginal.

Thanks,
Al

#10 lore raw

Re: [PATCH net-next 3/3] net: gro: register gso and gro offload on separate lists

Quoted 40 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Thu, 3 Feb 2022 08:11:43 -0800

Quoted 19 lines · Paolo Abeni
Quoted 18 lines

So that we know each element in gro_list has valid gro callbacks (and the same for gso). This allows dropping a bunch of conditional in fastpath.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

After:
0000000000000bb0 l F .text 0000000000000325 dev_gro_receive

---
 include/linux/netdevice.h |  3 +-
 net/core/gro.c            | 90 +++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 37 deletions(-)
Quoted 16 lines

include/linux/netdevice.h struct packet_offload { · line 2570

Quoted 13 lines
include/linux/netdevice.h +2 -1
index 3213c7227b59..406cb457d788 100644
@@ -2564,7 +2564,8 @@ struct packet_offload {
        __be16                   type;  /* This is really htons(ether_type). */
        u16                      priority;
        struct offload_callbacks callbacks;
-       struct list_head         list;
+       struct list_head         gro_list;
+       struct list_head         gso_list;
 };
 

On the other hand, this makes this object bigger, increasing the risk of spanning cache lines.

Quoted 6 lines

As you said, GSO callbacks are barely used with most modern NICs. Plus GRO and GSO callbacks are used in the completely different operations.
gro_list occupies the same place where the list previously was. Does it make a lot of sense to care about gso_list being placed in a cacheline separate from gro_list?

Not sure what you are asking in this last sentence.

Putting together all struct packet_offload and struct net_offload would reduce number of cache lines,
but making these objects bigger is conflicting.

Another approach to avoiding conditional tests would be to provide non NULL values
for all "struct offload_callbacks" members, just in case we missed something.

I think the test over ptype->type == type should be enough, right ?

author maintainer
#11 lore raw

Re: [PATCH net-next 3/3] net: gro: register gso and gro offload on separate lists

Quoted 55 lines · Eric Dumazet
Quoted 40 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Thu, 3 Feb 2022 08:11:43 -0800

Quoted 19 lines · Paolo Abeni
Quoted 18 lines

So that we know each element in gro_list has valid gro callbacks (and the same for gso). This allows dropping a bunch of conditional in fastpath.

Before:
objdump -t net/core/gro.o | grep " F .text"
0000000000000bb0 l F .text 000000000000033c dev_gro_receive

After:
0000000000000bb0 l F .text 0000000000000325 dev_gro_receive

---
 include/linux/netdevice.h |  3 +-
 net/core/gro.c            | 90 +++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 37 deletions(-)
Quoted 16 lines

include/linux/netdevice.h struct packet_offload { · line 2570

Quoted 13 lines
include/linux/netdevice.h +2 -1
index 3213c7227b59..406cb457d788 100644
@@ -2564,7 +2564,8 @@ struct packet_offload {
        __be16                   type;  /* This is really htons(ether_type). */
        u16                      priority;
        struct offload_callbacks callbacks;
-       struct list_head         list;
+       struct list_head         gro_list;
+       struct list_head         gso_list;
 };
 

On the other hand, this makes this object bigger, increasing the risk of spanning cache lines.

Quoted 6 lines

As you said, GSO callbacks are barely used with most modern NICs. Plus GRO and GSO callbacks are used in the completely different operations.
gro_list occupies the same place where the list previously was. Does it make a lot of sense to care about gso_list being placed in a cacheline separate from gro_list?

Not sure what you are asking in this last sentence.

Putting together all struct packet_offload and struct net_offload would reduce number of cache lines,
but making these objects bigger is conflicting.

Quoted 5 lines

Another approach to avoiding conditional tests would be to provide non NULL values
for all "struct offload_callbacks" members, just in case we missed something.

I think the test over ptype->type == type should be enough, right ?

I'll try this later approach, it looks simpler.

Also I'll keep this patch separate, as the others looks less controversial.

Thanks!

Paolo