Skip to contentSkip to conversation

net: intel: start The Great Code Dedup + Page Pool for iavf

PATCH Superseded Alexander Lobakin 6 participants 39 messages Nov 24, 2023 – Dec 12, 2023

Subsystems (5)

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Alexander Lobakin

author maintainer
#0 lore raw

v5 cover net: intel: start The Great Code Dedup + Page Pool for iavf

Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details are in the commit messages; here's a summary:

Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. The first name that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" (-> one word, no "lib I E" pls) and can be expanded as "lib Internet Explorer" :P
The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks:
"can I share this?". The final destination is very ambitious: have only one unified driver for at least i40e, ice, iavf, and idpf with a struct ops for each generation. That's never gonna happen, right? But you still can at least try.
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so that a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.

Alexander Lobakin (14):
page_pool: make sure frag API fields don't span between cachelines
page_pool: don't use driver-set flags field directly
page_pool: avoid calling no-op externals when possible
net: intel: introduce Intel Ethernet common library
iavf: kill "legacy-rx" for good
iavf: drop page splitting and recycling
page_pool: constify some read-only function arguments
page_pool: add DMA-sync-for-CPU inline helpers
libie: add Rx buffer management (via Page Pool)
iavf: pack iavf_ring more efficiently
iavf: switch to Page Pool
libie: add common queue stats
libie: add per-queue Page Pool stats
iavf: switch queue stats to libie

 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/intel/Kconfig            |   6 +
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 -------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  72 +-
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 ---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 -------
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 241 +------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  42 +-
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 626 ++++--------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 174 +----
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  90 ---
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  17 +-
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ---------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +--
 drivers/net/ethernet/intel/libie/Kconfig      |   9 +
 drivers/net/ethernet/intel/libie/Makefile     |   7 +
 drivers/net/ethernet/intel/libie/internal.h   |  20 +
 drivers/net/ethernet/intel/libie/rx.c         | 188 ++++++
 drivers/net/ethernet/intel/libie/stats.c      | 190 ++++++
 include/linux/net/intel/libie/rx.h            | 263 ++++++++
 include/linux/net/intel/libie/stats.h         | 213 ++++++
 include/net/page_pool/helpers.h               | 115 +++-
 include/net/page_pool/types.h                 |  10 +-
 net/core/page_pool.c                          |  50 +-
 30 files changed, 1270 insertions(+), 2069 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Kconfig
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/rx.h
 create mode 100644 include/linux/net/intel/libie/stats.h

---
Directly to net-next, has non-Intel code changes :p

From v4[0]:
* make use of Jakub's &page_pool_params split;
* #01: prevent frag fields from spanning into 2 cachelines after splitting &page_pool_params into fast and slow;
* #02-03: bring back the DMA sync shortcut, now as a per-page flag (me, Yunsheng);
* #04: let libie have its own Kconfig to stop further bloating of poor intel/Kconfig;
* #06: merge page split-reuse-recycle drop into one commit (Alex);
* #07: decouple constifying of several Page Pool function arguments into a separate commit, constify some more;
* #09: stop abusing internal PP fields in the driver code (Yunsheng);
* #09: calculate DMA sync size (::max_len) correctly: within one page, not one buffer (Yunsheng);
* #10: decouple rearranging &iavf_ring into separate commit, optimize it even more;
* #11: let the driver get back to the last descriptor to process after an skb allocation fail, don't drop it (Alex);
* #11: stop touching unrelated stuff like watchdog timeout etc. (Alex);
* fix "Return:" in the kdoc (now W=12 C=1 is clean), misc typos.

From v3[1]:
* base on the latest net-next, update bloat-o-meter and perf stats;
* split generic PP optimizations into a separate series;
* drop "optimize hotpath a bunch" commit: a lot of [controversial] changes in one place, worth own series (Alex);
* 02: pick Rev-by (Alex);
* 03: move in-place recycling removal here from the dropped patch;
* 05: new, add libie Rx buffer API separatelly from IAVF changes;
* 05-06: use new "hybrid" allocation API from[2] to reduce memory usage when a page can fit more than 1 truesize (also asked by David);
* 06: merge with "always use order-0 page" commit to reduce diffs and simplify things (Alex);
* 09: fix page_alloc_fail counter.

From v2[3]:
* 0006: fix page_pool.h include in OcteonTX2 files (Jakub, Patchwork);
* no functional changes.

From v1[4]:
* 0006: new (me, Jakub);
* 0008: give the helpers more intuitive names (Jakub, Ilias);
* -^-: also expand their kdoc a bit for the same reason;
* -^-: fix kdoc copy-paste issue (Patchwork, Jakub);
* 0011: drop inline from C file (Patchwork, Jakub).

[0] https://lore.kernel.org/netdev/[email protected]
[1] https://lore.kernel.org/netdev/[email protected]
[2] https://lore.kernel.org/netdev/[email protected]
[3] https://lore.kernel.org/netdev/[email protected]
[4] https://lore.kernel.org/netdev/[email protected]

Signature · 1 line
2.42.0
author maintainer
#1 lore raw

PATCH v5 1/14 page_pool: make sure frag API fields don't span between cachelines

After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again.
To not bother with this each time fast params get expanded or shrunk, let's just align them to 4 * sizeof(long), the closest upper pow-2 to their actual size (2 longs + 2 ints). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing.

---
 include/net/page_pool/types.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/page_pool/types.h +1 -1
index e1bb92c192de..989d07b831fc 100644
@@ -127,7 +127,7 @@ struct page_pool {
 
 	bool has_init_callback;
 
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));
 	struct page *frag_page;
 	unsigned int frag_offset;
 	u32 pages_state_hold_cnt;
Signature · 1 line
2.42.0
#2 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

Quoted 20 lines · Alexander Lobakin

After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again.
To not bother with this each time fast params get expanded or shrunk, let's just align them to 4 * sizeof(long), the closest upper pow-2 to their actual size (2 longs + 2 ints). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing.

---
 include/net/page_pool/types.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/page_pool/types.h +0 -0
index e1bb92c192de..989d07b831fc 100644

include/net/page_pool/types.h struct page_pool { · line 128

Quoted 2 lines
include/net/page_pool/types.h +0 -0
@@ -127,7 +127,7 @@ struct page_pool {
 
 	bool has_init_callback;

It seems odd to have only a slow field between tow fast field group, isn't it better to move it to the end of page_pool or where is more appropriate?

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 3 lines
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

If we need that, why not just use '____cacheline_aligned_in_smp'?

Quoted 4 lines
include/net/page_pool/types.h +0 -0
 	struct page *frag_page;
 	unsigned int frag_offset;
 	u32 pages_state_hold_cnt;
 
author maintainer
#3 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

From: Yunsheng Lin <[email protected]>
Date: Sat, 25 Nov 2023 20:29:22 +0800

Quoted 22 lines · Alexander Lobakin
Quoted 21 lines

After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again.
To not bother with this each time fast params get expanded or shrunk, let's just align them to 4 * sizeof(long), the closest upper pow-2 to their actual size (2 longs + 2 ints). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing.

---
 include/net/page_pool/types.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/page_pool/types.h +0 -0
index e1bb92c192de..989d07b831fc 100644
@@ -127,7 +127,7 @@ struct page_pool {
 
Quoted 5 lines

include/net/page_pool/types.h struct page_pool { · line 128

Quoted 1 line
include/net/page_pool/types.h +0 -0
 	bool has_init_callback;

It seems odd to have only a slow field between tow fast field group, isn't it better to move it to the end of page_pool or where is more appropriate?

1. There will be more in the subsequent patches.
2. ::has_init_callback happens each new page allocation, it's not slow. Jakub did put it here for purpose.

Quoted 6 lines

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 4 lines
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

If we need that, why not just use '____cacheline_aligned_in_smp'?

It can be an overkill. We don't need a full cacheline, but only these fields to stay within one, no matter whether they are in the beginning of it or at the end.

Quoted 5 lines

include/net/page_pool/types.h struct page_pool {

Quoted 5 lines
include/net/page_pool/types.h +0 -0
 	struct page *frag_page;
 	unsigned int frag_offset;
 	u32 pages_state_hold_cnt;
 

Thanks,
Olek

#4 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

Quoted 37 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Sat, 25 Nov 2023 20:29:22 +0800

Quoted 22 lines · Alexander Lobakin
Quoted 21 lines

After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again.
To not bother with this each time fast params get expanded or shrunk, let's just align them to 4 * sizeof(long), the closest upper pow-2 to their actual size (2 longs + 2 ints). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing.

---
 include/net/page_pool/types.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/page_pool/types.h +0 -0
index e1bb92c192de..989d07b831fc 100644
@@ -127,7 +127,7 @@ struct page_pool {
 
Quoted 5 lines

include/net/page_pool/types.h struct page_pool { · line 128

Quoted 1 line
include/net/page_pool/types.h +0 -0
 	bool has_init_callback;

It seems odd to have only a slow field between tow fast field group, isn't it better to move it to the end of page_pool or where is more appropriate?

1. There will be more in the subsequent patches.
2. ::has_init_callback happens each new page allocation, it's not slow. Jakub did put it here for purpose.

Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 4 lines

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 2 lines
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

If we need that, why not just use '____cacheline_aligned_in_smp'?

It can be an overkill. We don't need a full cacheline, but only these fields to stay within one, no matter whether they are in the beginning of it or at the end.

I am still a little lost here, A comment explaining why using '4' in the above would be really helpful here.

author maintainer
#5 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 10:55:00 +0800

Quoted 43 lines · Alexander Lobakin
Quoted 42 lines

From: Yunsheng Lin <[email protected]>
Date: Sat, 25 Nov 2023 20:29:22 +0800

Quoted 22 lines · Alexander Lobakin
Quoted 21 lines

After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again.
To not bother with this each time fast params get expanded or shrunk, let's just align them to 4 * sizeof(long), the closest upper pow-2 to their actual size (2 longs + 2 ints). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing.

---
 include/net/page_pool/types.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/page_pool/types.h +0 -0
index e1bb92c192de..989d07b831fc 100644
@@ -127,7 +127,7 @@ struct page_pool {
 
Quoted 5 lines

include/net/page_pool/types.h struct page_pool { · line 128

Quoted 1 line
include/net/page_pool/types.h +0 -0
 	bool has_init_callback;

It seems odd to have only a slow field between tow fast field group, isn't it better to move it to the end of page_pool or where is more appropriate?

1. There will be more in the subsequent patches.
2. ::has_init_callback happens each new page allocation, it's not slow. Jakub did put it here for purpose.

Quoted 6 lines

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 4 lines
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

If we need that, why not just use '____cacheline_aligned_in_smp'?

Quoted 6 lines
Quoted 3 lines

It can be an overkill. We don't need a full cacheline, but only these fields to stay within one, no matter whether they are in the beginning of it or at the end.

I am still a little lost here, A comment explaining why using '4' in the above would be really helpful here.

The block is: 2 longs (users, frag pointer) and 2 ints (offset, cnt). On 32-bit architectures, longs == ints, so that this effectively means 4 longs.
On 64-bit architectures, long is 8 bytes and int is 4, so that the value becomes 2 * 8 + 2 * 4 = 24, but the alignment must be a pow-2. The closest pow-2 to 24 is 32, which equals to 4 * 8 = 4 longs. At the end, regardless of the architecture, the desired alignment would end up as 4 * longs. As I wrote earlier, we could do something like

__aligned(roundup_pow_of_2(2 * sizeof(long) + 2 * sizeof(int)))

but doesn't that seem ugly as hell?

As I replied to Jakub, I'll add a comment in the code (so that you wouldn't need refer to the Git history / commit message) in the next version.

Thanks,
Olek

maintainer
#6 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

In reply to Alexander Lobakin

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 2 lines · On Fri, 24 Nov 2023 16:47:19 +0100 Alexander Lobakin wrote:
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

A comment for the somewhat unusual alignment size would be good.

author maintainer
#7 lore raw

Re: [PATCH net-next v5 01/14] page_pool: make sure frag API fields don't span between cachelines

From: Jakub Kicinski <[email protected]>
Date: Sun, 26 Nov 2023 14:54:57 -0800

Quoted 5 lines · On Fri, 24 Nov 2023 16:47:19 +0100 Alexander Lobakin wrote:

include/net/page_pool/types.h struct page_pool { · line 130

Quoted 2 lines
include/net/page_pool/types.h +1 -1
-	long frag_users;
+	long frag_users __aligned(4 * sizeof(long));

A comment for the somewhat unusual alignment size would be good.

Roger that. Will paste a couple words from the commit message.

FYI, I had an idea of doing something like

__aligned(roundup_pow_of_2(2 * sizeof(long) + 2 * sizeof(int)))

but that looks horrible, so I stopped on the current :D There are no functional changes between them either way.

Thanks,
Olek

author maintainer
#8 lore raw

PATCH v5 2/14 page_pool: don't use driver-set flags field directly

In reply to Alexander Lobakin

page_pool::p is driver-defined params, copied directly from the structure passed to page_pool_create(). The structure isn't meant to be modified by the Page Pool core code and this even might look confusing[0][1].
In order to be able to alter some flags, let's define our own, internal fields the same way as the already existing one (::has_init_callback). They are defined as bits in the driver-set params, leave them so here as well, to not waste byte-per-bit or so. Almost 30 bits are still free for future extensions.
We could've defined only new flags here or only the ones we may need to alter, but checking some flags in one place while others in another doesn't sound convenient or intuitive. ::flags passed by the driver can now go to the "slow" PP params.

Link[0]: https://lore.kernel.org/netdev/[email protected]

Link[1]: https://lore.kernel.org/netdev/CAKgT0UfZCGnWgOH96E4GV3ZP6LLbROHM7SHE8NKwq+exX+Gk_Q@mail.gmail.com

---
 include/net/page_pool/types.h |  8 +++++---
 net/core/page_pool.c          | 34 ++++++++++++++++++----------------
 2 files changed, 23 insertions(+), 19 deletions(-)
include/net/page_pool/types.h +5 -3
index 989d07b831fc..0ab6e14eb473 100644
@@ -43,7 +43,6 @@ struct pp_alloc_cache {
 
 /**
  * struct page_pool_params - page pool parameters
- * @flags:	PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV
  * @order:	2^order pages on allocation
  * @pool_size:	size of the ptr_ring
  * @nid:	NUMA node id to allocate from pages from
@@ -52,10 +51,10 @@ struct pp_alloc_cache {
  * @dma_dir:	DMA mapping direction
  * @max_len:	max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV
  * @offset:	DMA sync address offset for PP_FLAG_DMA_SYNC_DEV
+ * @flags:	PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV
  */
 struct page_pool_params {
 	struct_group_tagged(page_pool_params_fast, fast,
-		unsigned int	flags;
 		unsigned int	order;
 		unsigned int	pool_size;
 		int		nid;
@@ -66,6 +65,7 @@ struct page_pool_params {
 		unsigned int	offset;
 	);
 	struct_group_tagged(page_pool_params_slow, slow,
+		unsigned int	flags;
 /* private: used by test code only */
 		void (*init_callback)(struct page *page, void *arg);
 		void *init_arg;
@@ -125,7 +125,9 @@ struct page_pool_stats {
 struct page_pool {
 	struct page_pool_params_fast p;
 
-	bool has_init_callback;
+	bool dma_map:1;				/* Perform DMA mapping */
+	bool dma_sync:1;			/* Perform DMA sync */
+	bool has_init_callback:1;		/* slow.init_callback is set */
 
 	long frag_users __aligned(4 * sizeof(long));
 	struct page *frag_page;
net/core/page_pool.c +18 -16
index df2a06d7da52..a8da3ba3b1e8 100644
@@ -177,7 +177,7 @@ static int page_pool_init(struct page_pool *pool,
 	memcpy(&pool->slow, &params->slow, sizeof(pool->slow));
 
 	/* Validate only known flags were used */
-	if (pool->p.flags & ~(PP_FLAG_ALL))
+	if (pool->slow.flags & ~(PP_FLAG_ALL))
 		return -EINVAL;
 
 	if (pool->p.pool_size)
@@ -191,22 +191,26 @@ static int page_pool_init(struct page_pool *pool,
 	 * DMA_BIDIRECTIONAL is for allowing page used for DMA sending,
 	 * which is the XDP_TX use-case.
 	 */
-	if (pool->p.flags & PP_FLAG_DMA_MAP) {
+	if (pool->slow.flags & PP_FLAG_DMA_MAP) {
 		if ((pool->p.dma_dir != DMA_FROM_DEVICE) &&
 		    (pool->p.dma_dir != DMA_BIDIRECTIONAL))
 			return -EINVAL;
+
+		pool->dma_map = true;
 	}
 
-	if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) {
+	if (pool->slow.flags & PP_FLAG_DMA_SYNC_DEV) {
 		/* In order to request DMA-sync-for-device the page
 		 * needs to be mapped
 		 */
-		if (!(pool->p.flags & PP_FLAG_DMA_MAP))
+		if (!(pool->slow.flags & PP_FLAG_DMA_MAP))
 			return -EINVAL;
 
 		if (!pool->p.max_len)
 			return -EINVAL;
 
+		pool->dma_sync = true;
+
 		/* pool->p.offset has to be set according to the address
 		 * offset used by the DMA engine to start copying rx data
 		 */
@@ -232,7 +236,7 @@ static int page_pool_init(struct page_pool *pool,
 	/* Driver calling page_pool_create() also call page_pool_destroy() */
 	refcount_set(&pool->user_cnt, 1);
 
-	if (pool->p.flags & PP_FLAG_DMA_MAP)
+	if (pool->dma_map)
 		get_device(pool->p.dev);
 
 	return 0;
@@ -365,7 +369,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page)
 	if (page_pool_set_dma_addr(page, dma))
 		goto unmap_failed;
 
-	if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+	if (pool->dma_sync)
 		page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
 
 	return true;
@@ -411,8 +415,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
 	if (unlikely(!page))
 		return NULL;
 
-	if ((pool->p.flags & PP_FLAG_DMA_MAP) &&
-	    unlikely(!page_pool_dma_map(pool, page))) {
+	if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page))) {
 		put_page(page);
 		return NULL;
 	}
@@ -432,8 +435,8 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
 						 gfp_t gfp)
 {
 	const int bulk = PP_ALLOC_CACHE_REFILL;
-	unsigned int pp_flags = pool->p.flags;
 	unsigned int pp_order = pool->p.order;
+	bool dma_map = pool->dma_map;
 	struct page *page;
 	int i, nr_pages;
 
@@ -458,8 +461,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
 	 */
 	for (i = 0; i < nr_pages; i++) {
 		page = pool->alloc.cache[i];
-		if ((pp_flags & PP_FLAG_DMA_MAP) &&
-		    unlikely(!page_pool_dma_map(pool, page))) {
+		if (dma_map && unlikely(!page_pool_dma_map(pool, page))) {
 			put_page(page);
 			continue;
 		}
@@ -531,7 +533,7 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 	dma_addr_t dma;
 	int count;
 
-	if (!(pool->p.flags & PP_FLAG_DMA_MAP))
+	if (!pool->dma_map)
 		/* Always account for inflight pages, even if we didn't
 		 * map them
 		 */
@@ -597,7 +599,7 @@ static bool page_pool_recycle_in_cache(struct page *page,
 }
 
 /* If the page refcnt == 1, this will try to recycle the page.
- * if PP_FLAG_DMA_SYNC_DEV is set, we'll try to sync the DMA area for
+ * If pool->dma_sync is set, we'll try to sync the DMA area for
  * the configured size min(dma_sync_size, pool->max_len).
  * If the page refcnt != 1, then the page will be returned to memory
  * subsystem.
@@ -620,7 +622,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,
 	if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) {
 		/* Read barrier done in page_ref_count / READ_ONCE */
 
-		if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+		if (pool->dma_sync)
 			page_pool_dma_sync_for_device(pool, page,
 						      dma_sync_size);
 
@@ -733,7 +735,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool,
 		return NULL;
 
 	if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) {
-		if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+		if (pool->dma_sync)
 			page_pool_dma_sync_for_device(pool, page, -1);
 
 		return page;
@@ -823,7 +825,7 @@ static void __page_pool_destroy(struct page_pool *pool)
 
 	ptr_ring_cleanup(&pool->ring, NULL);
 
-	if (pool->p.flags & PP_FLAG_DMA_MAP)
+	if (pool->dma_map)
 		put_device(pool->p.dev);
 
 #ifdef CONFIG_PAGE_POOL_STATS
Signature · 1 line
2.42.0
author maintainer
#9 lore raw

PATCH v5 3/14 page_pool: avoid calling no-op externals when possible

In reply to Alexander Lobakin

Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles even on DMA-coherent platforms (like x86) with no active IOMMU or swiotlb, just for the call ladder.
Indeed, it's

page_pool_put_page()
page_pool_put_defragged_page() <- external
__page_pool_put_page()
page_pool_dma_sync_for_device() <- non-inline
dma_sync_single_range_for_device()
dma_sync_single_for_device() <- external
dma_direct_sync_single_for_device()
dev_is_dma_coherent() <- exit

For the inline functions, no guarantees the compiler won't uninline them (they're clearly not one-liners and sometimes the compilers manage to uninline 2 + 2). The first external call is necessary, but the rest 2+ are done for nothing each time, plus a bunch of checks here and there. Since Page Pool mappings are long-term and for one "device + addr" pair dma_need_sync() will always return the same value (basically, whether it belongs to an swiotlb pool), addresses can be tested once right after they're obtained and the result can be reused until the page is unmapped.
Do this test after a successful DMA mapping and reuse the lowest bit of page::dma_addr to store the result, since Page Pool addresses are always page-aligned and the lowest bits are unused. page_pool_{g,s}et_dma_addr() will then take this bit into account and exclude it from the actual address. Call page_pool_dma_sync_for_device() only if the bit is set, shaving off several function calls back and forth. If pool::dma_sync is not set, i.e. the driver didn't ask to perform syncs, don't do this test and never touch the lowest bit. On my x86_64, this gives from 2% to 5% performance benefit with no negative impact for cases when IOMMU is on and the shortcut can't be used.

---
 include/net/page_pool/helpers.h | 60 +++++++++++++++++++++++++++++----
 net/core/page_pool.c            | 10 +++---
 2 files changed, 58 insertions(+), 12 deletions(-)
include/net/page_pool/helpers.h +53 -7
index 4ebd544ae977..28bec368b8e9 100644
@@ -52,6 +52,8 @@
 #ifndef _NET_PAGE_POOL_HELPERS_H
 #define _NET_PAGE_POOL_HELPERS_H
 
+#include <linux/dma-mapping.h>
+
 #include <net/page_pool/types.h>
 
 #ifdef CONFIG_PAGE_POOL_STATS
@@ -354,6 +356,11 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
 	page_pool_put_page(pool, virt_to_head_page(va), -1, allow_direct);
 }
 
+/* Occupies page::dma_addr[0:0] and indicates whether the mapping needs to be
+ * synchronized (either for device, for CPU, or both).
+ */
+#define PAGE_POOL_DMA_ADDR_NEED_SYNC		BIT(0)
+
 /**
  * page_pool_get_dma_addr() - Retrieve the stored DMA address.
  * @page:	page allocated from a page pool
@@ -363,27 +370,66 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
  */
 static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
 {
-	dma_addr_t ret = page->dma_addr;
+	dma_addr_t ret = page->dma_addr & ~PAGE_POOL_DMA_ADDR_NEED_SYNC;
 
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA)
-		ret <<= PAGE_SHIFT;
+		ret <<= PAGE_SHIFT - 1;
 
 	return ret;
 }
 
-static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
+/**
+ * page_pool_set_dma_addr - save the obtained DMA address for @page
+ * @pool: &page_pool this page belongs to and was mapped by
+ * @page: page to save the address for
+ * @addr: DMA address to save
+ *
+ * If the pool was created with the DMA synchronization enabled, it will test
+ * whether the address needs to be synced and store the result as well to
+ * conserve CPU cycles.
+ *
+ * Return: false if the address doesn't fit into the corresponding field and
+ * thus can't be stored, true on success.
+ */
+static inline bool page_pool_set_dma_addr(const struct page_pool *pool,
+					  struct page *page,
+					  dma_addr_t addr)
 {
+	unsigned long val = addr;
+
+	if (unlikely(!addr)) {
+		page->dma_addr = 0;
+		return true;
+	}
+
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) {
-		page->dma_addr = addr >> PAGE_SHIFT;
+		val = addr >> (PAGE_SHIFT - 1);
 
 		/* We assume page alignment to shave off bottom bits,
 		 * if this "compression" doesn't work we need to drop.
 		 */
-		return addr != (dma_addr_t)page->dma_addr << PAGE_SHIFT;
+		if (addr != (dma_addr_t)val << (PAGE_SHIFT - 1))
+			return false;
 	}
 
-	page->dma_addr = addr;
-	return false;
+	if (pool->dma_sync && dma_need_sync(pool->p.dev, addr))
+		val |= PAGE_POOL_DMA_ADDR_NEED_SYNC;
+
+	page->dma_addr = val;
+
+	return true;
+}
+
+/**
+ * page_pool_dma_addr_need_sync - check whether a mapped page needs DMA syncs
+ * @page: page to check
+ *
+ * Return: the result previously saved by page_pool_set_dma_addr() to skip
+ * synchronization when not needed.
+ */
+static inline bool page_pool_dma_addr_need_sync(const struct page *page)
+{
+	return page->dma_addr & PAGE_POOL_DMA_ADDR_NEED_SYNC;
 }
 
 static inline bool page_pool_put(struct page_pool *pool)
net/core/page_pool.c +5 -5
index a8da3ba3b1e8..09bdfdfc4788 100644
@@ -366,10 +366,10 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page)
 	if (dma_mapping_error(pool->p.dev, dma))
 		return false;
 
-	if (page_pool_set_dma_addr(page, dma))
+	if (!page_pool_set_dma_addr(pool, page, dma))
 		goto unmap_failed;
 
-	if (pool->dma_sync)
+	if (page_pool_dma_addr_need_sync(page))
 		page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
 
 	return true;
@@ -545,7 +545,7 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 	dma_unmap_page_attrs(pool->p.dev, dma,
 			     PAGE_SIZE << pool->p.order, pool->p.dma_dir,
 			     DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING);
-	page_pool_set_dma_addr(page, 0);
+	page_pool_set_dma_addr(pool, page, 0);
 skip_dma_unmap:
 	page_pool_clear_pp_info(page);
 
@@ -622,7 +622,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,
 	if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) {
 		/* Read barrier done in page_ref_count / READ_ONCE */
 
-		if (pool->dma_sync)
+		if (page_pool_dma_addr_need_sync(page))
 			page_pool_dma_sync_for_device(pool, page,
 						      dma_sync_size);
 
@@ -735,7 +735,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool,
 		return NULL;
 
 	if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) {
-		if (pool->dma_sync)
+		if (page_pool_dma_addr_need_sync(page))
 			page_pool_dma_sync_for_device(pool, page, -1);
 
 		return page;
Signature · 1 line
2.42.0
#10 lore raw

Re: [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

Quoted 35 lines · Alexander Lobakin

Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles even on DMA-coherent platforms (like x86) with no active IOMMU or swiotlb, just for the call ladder.
Indeed, it's

page_pool_put_page()
page_pool_put_defragged_page() <- external
__page_pool_put_page()
page_pool_dma_sync_for_device() <- non-inline
dma_sync_single_range_for_device()
dma_sync_single_for_device() <- external
dma_direct_sync_single_for_device()
dev_is_dma_coherent() <- exit

For the inline functions, no guarantees the compiler won't uninline them (they're clearly not one-liners and sometimes the compilers manage to uninline 2 + 2). The first external call is necessary, but the rest 2+ are done for nothing each time, plus a bunch of checks here and there. Since Page Pool mappings are long-term and for one "device + addr" pair dma_need_sync() will always return the same value (basically, whether it belongs to an swiotlb pool), addresses can be tested once right after they're obtained and the result can be reused until the page is unmapped.
Do this test after a successful DMA mapping and reuse the lowest bit of page::dma_addr to store the result, since Page Pool addresses are always page-aligned and the lowest bits are unused. page_pool_{g,s}et_dma_addr() will then take this bit into account and exclude it from the actual address. Call page_pool_dma_sync_for_device() only if the bit is set, shaving off several function calls back and forth. If pool::dma_sync is not set, i.e. the driver didn't ask to perform syncs, don't do this test and never touch the lowest bit. On my x86_64, this gives from 2% to 5% performance benefit with no negative impact for cases when IOMMU is on and the shortcut can't be used.

Quoted 1 line

Sorry for not remembering the suggestion:(

If most of cases is that some pages need syncing and some page does not need syncing for the same device, then perhaps per-page flag is more beneficial, if not, maybe per-device flag is enough too?

What i wonder is that is there any reason this kind trick isn't done in the dma layer instead of each subsystem or driver using dma api duplicating this kind of trick?

At least xsk_buff seems to using the similar trick?

You may have explained that, it has been a litte long between v4 and v5, so I may have forgotten about that.

Quoted 20 lines
---
 include/net/page_pool/helpers.h | 60 +++++++++++++++++++++++++++++----
 net/core/page_pool.c            | 10 +++---
 2 files changed, 58 insertions(+), 12 deletions(-)
include/net/page_pool/helpers.h +2 -0
index 4ebd544ae977..28bec368b8e9 100644
@@ -52,6 +52,8 @@
 #ifndef _NET_PAGE_POOL_HELPERS_H
 #define _NET_PAGE_POOL_HELPERS_H
 
+#include <linux/dma-mapping.h>
+
 #include <net/page_pool/types.h>
 
 #ifdef CONFIG_PAGE_POOL_STATS

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 362

Quoted 7 lines
include/net/page_pool/helpers.h +4 -0
@@ -354,6 +356,11 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
 	page_pool_put_page(pool, virt_to_head_page(va), -1, allow_direct);
 }
 
+/* Occupies page::dma_addr[0:0] and indicates whether the mapping needs to be
+ * synchronized (either for device, for CPU, or both).
+ */
+#define PAGE_POOL_DMA_ADDR_NEED_SYNC		BIT(0)

It seems we have less dma space from 32 bit arch with 64 bit dma address:( But from my last searching in google, it seems we still have enough bit for that:

arm: Large Physical Address Extension or LPAE, 40 bits of phys addr.
arc: Physical Address Extension or PAE, 40 bits of phys addr.
mips: eXtended Physical Addressing or PXA, 40 bits of phys addr.
powerpc: does not seems to have a name for the feature, and have 36
bits of phys addr.
riscv: large physical address, 34 bits of phys addr.
x86: Physical Address Extension or PAE, 36 bits of phys addr.

Quoted 32 lines
include/net/page_pool/helpers.h +16 -3
+
 /**
  * page_pool_get_dma_addr() - Retrieve the stored DMA address.
  * @page:	page allocated from a page pool
@@ -363,27 +370,66 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
  */
 static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
 {
-	dma_addr_t ret = page->dma_addr;
+	dma_addr_t ret = page->dma_addr & ~PAGE_POOL_DMA_ADDR_NEED_SYNC;
 
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA)
-		ret <<= PAGE_SHIFT;
+		ret <<= PAGE_SHIFT - 1;
 
 	return ret;
 }
 
-static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
+/**
+ * page_pool_set_dma_addr - save the obtained DMA address for @page
+ * @pool: &page_pool this page belongs to and was mapped by
+ * @page: page to save the address for
+ * @addr: DMA address to save
+ *
+ * If the pool was created with the DMA synchronization enabled, it will test
+ * whether the address needs to be synced and store the result as well to
+ * conserve CPU cycles.
+ *
+ * Return: false if the address doesn't fit into the corresponding field and
+ * thus can't be stored, true on success.
+ */

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 403

Quoted 10 lines
include/net/page_pool/helpers.h +9 -0
+static inline bool page_pool_set_dma_addr(const struct page_pool *pool,
+					  struct page *page,
+					  dma_addr_t addr)
 {
+	unsigned long val = addr;
+
+	if (unlikely(!addr)) {
+		page->dma_addr = 0;
+		return true;
+	}

The above seems unrelated change?

Quoted 23 lines
include/net/page_pool/helpers.h +12 -4
+
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) {
-		page->dma_addr = addr >> PAGE_SHIFT;
+		val = addr >> (PAGE_SHIFT - 1);
 
 		/* We assume page alignment to shave off bottom bits,
 		 * if this "compression" doesn't work we need to drop.
 		 */
-		return addr != (dma_addr_t)page->dma_addr << PAGE_SHIFT;
+		if (addr != (dma_addr_t)val << (PAGE_SHIFT - 1))
+			return false;
 	}
 
-	page->dma_addr = addr;
-	return false;
+	if (pool->dma_sync && dma_need_sync(pool->p.dev, addr))
+		val |= PAGE_POOL_DMA_ADDR_NEED_SYNC;
+
+	page->dma_addr = val;
+
+	return true;
+}
+
author maintainer
#11 lore raw

Re: [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

From: Yunsheng Lin <[email protected]>
Date: Sat, 25 Nov 2023 21:04:28 +0800

Quoted 36 lines · Alexander Lobakin
Quoted 35 lines

Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles even on DMA-coherent platforms (like x86) with no active IOMMU or swiotlb, just for the call ladder.
Indeed, it's

page_pool_put_page()
page_pool_put_defragged_page() <- external
__page_pool_put_page()
page_pool_dma_sync_for_device() <- non-inline
dma_sync_single_range_for_device()
dma_sync_single_for_device() <- external
dma_direct_sync_single_for_device()
dev_is_dma_coherent() <- exit

For the inline functions, no guarantees the compiler won't uninline them (they're clearly not one-liners and sometimes the compilers manage to uninline 2 + 2). The first external call is necessary, but the rest 2+ are done for nothing each time, plus a bunch of checks here and there. Since Page Pool mappings are long-term and for one "device + addr" pair dma_need_sync() will always return the same value (basically, whether it belongs to an swiotlb pool), addresses can be tested once right after they're obtained and the result can be reused until the page is unmapped.
Do this test after a successful DMA mapping and reuse the lowest bit of page::dma_addr to store the result, since Page Pool addresses are always page-aligned and the lowest bits are unused. page_pool_{g,s}et_dma_addr() will then take this bit into account and exclude it from the actual address. Call page_pool_dma_sync_for_device() only if the bit is set, shaving off several function calls back and forth. If pool::dma_sync is not set, i.e. the driver didn't ask to perform syncs, don't do this test and never touch the lowest bit. On my x86_64, this gives from 2% to 5% performance benefit with no negative impact for cases when IOMMU is on and the shortcut can't be used.

Quoted 3 lines
Quoted 1 line

Sorry for not remembering the suggestion:(

In the previous versions of this change I used a global flag per whole page_pool, just like XSk does for the whole XSk buff pool, then you proposed to use the lowest bit of ::dma_addr and store it per page, so that it would be more granular/precise. I tested it and it doesn't perform worse than global, but in some cases may be beneficial.

Quoted 4 lines

If most of cases is that some pages need syncing and some page does not need syncing for the same device, then perhaps per-page flag is more beneficial, if not, maybe per-device flag is enough too?

Per-device is not.
You never know when dma_alloc_* or dma_map_* will return an address needing syncs. I realize that in most cases it will always be the same, but since there is a possibility of more precise optimization, why not use it :>

Quoted 9 lines

What i wonder is that is there any reason this kind trick isn't done in the dma layer instead of each subsystem or driver using dma api duplicating this kind of trick?

At least xsk_buff seems to using the similar trick?

You may have explained that, it has been a litte long between v4 and v5, so I may have forgotten about that.

Yeah, as I wrote previously, there's currently no easy way to combine these two optimizations.

I can't even use this patch in the XSk layer, since it doesn't operate with struct page, rather xsk_buff etc., which may not have a direct pointer to struct page. And in case of DMA sync functions, there are no pointers to xsk_buff passed there, only xsk_buff_pool and dma_addr_t, with no direct association with a particular page.
Also note that xsk_buff_pool still needs to be converted to use dma_buf API. This should be done soon and will involve quite big changes regarding DMA, it makes no sense to touch anything there prior to this.

Field in the struct device may work out, but in some cases such global gate might disable this optimization where it could be still enabled for page_pool or xsk_buff_pool.

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

Quoted 34 lines
Quoted 22 lines
---
 include/net/page_pool/helpers.h | 60 +++++++++++++++++++++++++++++----
 net/core/page_pool.c            | 10 +++---
 2 files changed, 58 insertions(+), 12 deletions(-)
include/net/page_pool/helpers.h +2 -0
index 4ebd544ae977..28bec368b8e9 100644
@@ -52,6 +52,8 @@
 #ifndef _NET_PAGE_POOL_HELPERS_H
 #define _NET_PAGE_POOL_HELPERS_H
 
+#include <linux/dma-mapping.h>
+
 #include <net/page_pool/types.h>
 
 #ifdef CONFIG_PAGE_POOL_STATS

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 362

Quoted 7 lines
include/net/page_pool/helpers.h +4 -0
@@ -354,6 +356,11 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
 	page_pool_put_page(pool, virt_to_head_page(va), -1, allow_direct);
 }
 
+/* Occupies page::dma_addr[0:0] and indicates whether the mapping needs to be
+ * synchronized (either for device, for CPU, or both).
+ */
+#define PAGE_POOL_DMA_ADDR_NEED_SYNC		BIT(0)

It seems we have less dma space from 32 bit arch with 64 bit dma address:( But from my last searching in google, it seems we still have enough bit for that:

Quoted 7 lines

arm: Large Physical Address Extension or LPAE, 40 bits of phys addr.
arc: Physical Address Extension or PAE, 40 bits of phys addr.
mips: eXtended Physical Addressing or PXA, 40 bits of phys addr.
powerpc: does not seems to have a name for the feature, and have 36
bits of phys addr.
riscv: large physical address, 34 bits of phys addr.
x86: Physical Address Extension or PAE, 36 bits of phys addr.

Yeah, 1 bit less, but from I've read, in most cases it will still work. For 4 Kb pages, shift is 12, so that we're able to address 44 bits. Eating one bit for DMA sync flag shrinks it to 43 -> still enough for the cases you described above. Also note that some of these arches may have bigger pages, where the shift is higher -> even more available bits.

Quoted 37 lines
Quoted 37 lines
include/net/page_pool/helpers.h +19 -3
+
 /**
  * page_pool_get_dma_addr() - Retrieve the stored DMA address.
  * @page:	page allocated from a page pool
@@ -363,27 +370,66 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
  */
 static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
 {
-	dma_addr_t ret = page->dma_addr;
+	dma_addr_t ret = page->dma_addr & ~PAGE_POOL_DMA_ADDR_NEED_SYNC;
 
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA)
-		ret <<= PAGE_SHIFT;
+		ret <<= PAGE_SHIFT - 1;
 
 	return ret;
 }
 
-static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
+/**
+ * page_pool_set_dma_addr - save the obtained DMA address for @page
+ * @pool: &page_pool this page belongs to and was mapped by
+ * @page: page to save the address for
+ * @addr: DMA address to save
+ *
+ * If the pool was created with the DMA synchronization enabled, it will test
+ * whether the address needs to be synced and store the result as well to
+ * conserve CPU cycles.
+ *
+ * Return: false if the address doesn't fit into the corresponding field and
+ * thus can't be stored, true on success.
+ */
+static inline bool page_pool_set_dma_addr(const struct page_pool *pool,
+					  struct page *page,
+					  dma_addr_t addr)
 {
Quoted 8 lines

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 403

Quoted 6 lines
include/net/page_pool/helpers.h +6 -0
+	unsigned long val = addr;
+
+	if (unlikely(!addr)) {
+		page->dma_addr = 0;
+		return true;
+	}

The above seems unrelated change?

Related. We use page_put_set_dma_addr() to clear ::dma_addr as well (grep for it in page_pool.c). In this case, we don't want dma_need_sync() to be called as we explicitly pass zero. This check zeroes the field and exits as quickly as possible.
In case with the call mentioned above, zero is a compile-time constant there, so that this little branch will be inlined with the rest dropped.

Quoted 14 lines
Quoted 14 lines
include/net/page_pool/helpers.h +4 -2
+
 	if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) {
-		page->dma_addr = addr >> PAGE_SHIFT;
+		val = addr >> (PAGE_SHIFT - 1);
 
 		/* We assume page alignment to shave off bottom bits,
 		 * if this "compression" doesn't work we need to drop.
 		 */
-		return addr != (dma_addr_t)page->dma_addr << PAGE_SHIFT;
+		if (addr != (dma_addr_t)val << (PAGE_SHIFT - 1))
+			return false;
 	}
 
Quoted 11 lines

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 422

Quoted 10 lines
include/net/page_pool/helpers.h +8 -2
-	page->dma_addr = addr;
-	return false;
+	if (pool->dma_sync && dma_need_sync(pool->p.dev, addr))
+		val |= PAGE_POOL_DMA_ADDR_NEED_SYNC;
+
+	page->dma_addr = val;
+
+	return true;
+}
+

Thanks,
Olek

maintainer
#12 lore raw

Re: [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

Quoted 7 lines · On Mon, 27 Nov 2023 15:32:19 +0100 Alexander Lobakin wrote:
Quoted 1 line

Sorry for not remembering the suggestion:(

In the previous versions of this change I used a global flag per whole page_pool, just like XSk does for the whole XSk buff pool, then you proposed to use the lowest bit of ::dma_addr and store it per page, so that it would be more granular/precise. I tested it and it doesn't perform worse than global, but in some cases may be beneficial.

FWIW I'd vote to stick to per-page pool. You seem to handle the sizeof(dma_addr_t) > sizeof(long) case correctly but the code is growing in complexity, providing no known/measurable benefit. We can always do this later but for now it seems like a premature optimization to me.

author maintainer
#13 lore raw

Re: [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

From: Jakub Kicinski <[email protected]>
Date: Mon, 27 Nov 2023 10:17:20 -0800

Quoted 9 lines · On Mon, 27 Nov 2023 15:32:19 +0100 Alexander Lobakin wrote:
Quoted 7 lines
Quoted 1 line

Sorry for not remembering the suggestion:(

In the previous versions of this change I used a global flag per whole page_pool, just like XSk does for the whole XSk buff pool, then you proposed to use the lowest bit of ::dma_addr and store it per page, so that it would be more granular/precise. I tested it and it doesn't perform worse than global, but in some cases may be beneficial.

Quoted 5 lines

FWIW I'd vote to stick to per-page pool. You seem to handle the sizeof(dma_addr_t) > sizeof(long) case correctly but the code is growing in complexity, providing no known/measurable benefit. We can always do this later but for now it seems like a premature optimization to me.

Yeah, this also seems more logical and optimal to me. Will wait a bit for a possible reply from Chris and then send the next rev.

Thanks,
Olek

#14 lore raw

Re: [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

In reply to Alexander Lobakin
Quoted 2 lines · Alexander Lobakin

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

https://lore.kernel.org/netdev/[email protected]/T/

Quoted 17 lines
Quoted 4 lines
Quoted 12 lines

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 403

Quoted 10 lines
include/net/page_pool/helpers.h +9 -0
+static inline bool page_pool_set_dma_addr(const struct page_pool *pool,
+					  struct page *page,
+					  dma_addr_t addr)
 {
+	unsigned long val = addr;
+
+	if (unlikely(!addr)) {
+		page->dma_addr = 0;
+		return true;
+	}

The above seems unrelated change?

Quoted 4 lines

Related. We use page_put_set_dma_addr() to clear ::dma_addr as well (grep for it in page_pool.c). In this case, we don't want dma_need_sync() to be called as we explicitly pass zero. This check zeroes the field and exits as quickly as possible.

The question seems to be about if we need to ensure the LSB of page->dma_addr is not set when page_pool releases a page back to page allocator?

Quoted 3 lines

In case with the call mentioned above, zero is a compile-time constant there, so that this little branch will be inlined with the rest dropped.

author maintainer
#15 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 11:17:50 +0800

Quoted 7 lines · Alexander Lobakin
Quoted 2 lines

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

https://lore.kernel.org/netdev/[email protected]/T/

It addresses a different problem, meaningless indirect calls, while this one addresses meaningless direct calls :>
When the above gets merged, we could combine these two into one global, but Eric wasn't active with his patch and I remember there were some problems, so I wouldn't count on that it will arrive soon.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 5 lines

include/net/page_pool/helpers.h static inline void page_pool_free_va(struct page_pool *pool, void *va, · line 403

Quoted 10 lines
include/net/page_pool/helpers.h +9 -0
+static inline bool page_pool_set_dma_addr(const struct page_pool *pool,
+					  struct page *page,
+					  dma_addr_t addr)
 {
+	unsigned long val = addr;
+
+	if (unlikely(!addr)) {
+		page->dma_addr = 0;
+		return true;
+	}

The above seems unrelated change?

Quoted 8 lines
Quoted 4 lines

Related. We use page_put_set_dma_addr() to clear ::dma_addr as well (grep for it in page_pool.c). In this case, we don't want dma_need_sync() to be called as we explicitly pass zero. This check zeroes the field and exits as quickly as possible.

The question seems to be about if we need to ensure the LSB of page->dma_addr is not set when page_pool releases a page back to page allocator?

But why do we need to call dma_need_sync(0) when freeing a page wasting CPU cycles on relatively hot path?

Quoted 3 lines
Quoted 3 lines

In case with the call mentioned above, zero is a compile-time constant there, so that this little branch will be inlined with the rest dropped.

Thanks,
Olek

#16 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

Quoted 9 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 11:17:50 +0800

Quoted 6 lines · Alexander Lobakin
Quoted 2 lines

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

Quoted 7 lines
Quoted 1 line

https://lore.kernel.org/netdev/[email protected]/T/

It addresses a different problem, meaningless indirect calls, while this one addresses meaningless direct calls :>
When the above gets merged, we could combine these two into one global, but Eric wasn't active with his patch and I remember there were some problems, so I wouldn't count on that it will arrive soon.

I went through the above patch, It seems to me that there was no fundamental problem that stopping us from implementing it in the dma layer basing on Eric' patch if Eric is no longer interested in working on a newer version?

It is just that if we allow every subsystem and driver using dma API doing their own trick of skipping dma sync, then there is less willing to implement it in the dma layer.

author maintainer
#17 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

From: Yunsheng Lin <[email protected]>
Date: Thu, 30 Nov 2023 16:46:11 +0800

Quoted 18 lines · Alexander Lobakin
Quoted 9 lines

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 11:17:50 +0800

Quoted 6 lines · Alexander Lobakin
Quoted 2 lines

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

Quoted 7 lines
Quoted 1 line

https://lore.kernel.org/netdev/[email protected]/T/

It addresses a different problem, meaningless indirect calls, while this one addresses meaningless direct calls :>
When the above gets merged, we could combine these two into one global, but Eric wasn't active with his patch and I remember there were some problems, so I wouldn't count on that it will arrive soon.

Quoted 4 lines

I went through the above patch, It seems to me that there was no fundamental problem that stopping us from implementing it in the dma layer basing on Eric' patch if Eric is no longer interested in working on a newer version?

I'm somewhat interested in continuing working on Eric's patch, but not now. Have some urgent projects to work on, I could take this in January I guess...
This PP-specific shortcut was done earlier and gives good boosts. It would be trivial to remove it together with the XSk shortcut once a generic both indirect and direct call DMA shortcut lands. Does this sounds good / justified enough? Or you and other reviewers/maintainers would prefer to wait for the generic one without taking this patch?

Quoted 4 lines

It is just that if we allow every subsystem and driver using dma API doing their own trick of skipping dma sync, then there is less willing to implement it in the dma layer.

Thanks,
Olek

#18 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

Quoted 26 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Thu, 30 Nov 2023 16:46:11 +0800

Quoted 18 lines · Alexander Lobakin
Quoted 9 lines

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 11:17:50 +0800

Quoted 6 lines · Alexander Lobakin
Quoted 2 lines

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

Quoted 7 lines
Quoted 1 line

https://lore.kernel.org/netdev/[email protected]/T/

It addresses a different problem, meaningless indirect calls, while this one addresses meaningless direct calls :>
When the above gets merged, we could combine these two into one global, but Eric wasn't active with his patch and I remember there were some problems, so I wouldn't count on that it will arrive soon.

Quoted 4 lines

I went through the above patch, It seems to me that there was no fundamental problem that stopping us from implementing it in the dma layer basing on Eric' patch if Eric is no longer interested in working on a newer version?

Quoted 10 lines

I'm somewhat interested in continuing working on Eric's patch, but not now. Have some urgent projects to work on, I could take this in January I guess...
This PP-specific shortcut was done earlier and gives good boosts. It would be trivial to remove it together with the XSk shortcut once a generic both indirect and direct call DMA shortcut lands. Does this sounds good / justified enough? Or you and other reviewers/maintainers would prefer to wait for the generic one without taking this patch?

I would prefer we could wait for the generic one as there is only about one month between now and january:)

author maintainer
#19 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

From: Yunsheng Lin <[email protected]>
Date: Thu, 30 Nov 2023 20:20:44 +0800

Quoted 38 lines · Alexander Lobakin
Quoted 26 lines

From: Yunsheng Lin <[email protected]>
Date: Thu, 30 Nov 2023 16:46:11 +0800

Quoted 18 lines · Alexander Lobakin
Quoted 9 lines

From: Yunsheng Lin <[email protected]>
Date: Wed, 29 Nov 2023 11:17:50 +0800

Quoted 6 lines · Alexander Lobakin
Quoted 2 lines

Chris, any thoughts on a global flag for skipping DMA syncs ladder?

It seems there was one already in the past:

Quoted 7 lines
Quoted 1 line

https://lore.kernel.org/netdev/[email protected]/T/

It addresses a different problem, meaningless indirect calls, while this one addresses meaningless direct calls :>
When the above gets merged, we could combine these two into one global, but Eric wasn't active with his patch and I remember there were some problems, so I wouldn't count on that it will arrive soon.

Quoted 4 lines

I went through the above patch, It seems to me that there was no fundamental problem that stopping us from implementing it in the dma layer basing on Eric' patch if Eric is no longer interested in working on a newer version?

Quoted 10 lines

I'm somewhat interested in continuing working on Eric's patch, but not now. Have some urgent projects to work on, I could take this in January I guess...
This PP-specific shortcut was done earlier and gives good boosts. It would be trivial to remove it together with the XSk shortcut once a generic both indirect and direct call DMA shortcut lands. Does this sounds good / justified enough? Or you and other reviewers/maintainers would prefer to wait for the generic one without taking this patch?

Quoted 2 lines

I would prefer we could wait for the generic one as there is only about one month between now and january:)

Ok, let's do it this way. I'll try to revive Eric's idea soon and get it taken if he doesn't mind :>

Thanks,
Olek

#20 lore raw

Re: [Intel-wired-lan] [PATCH net-next v5 03/14] page_pool: avoid calling no-op externals when possible

In reply to Yunsheng Lin
Quoted 2 lines · Yunsheng Lin

I would prefer we could wait for the generic one as there is only about one month between now and january:)

Same here. Please fix this properly for everyone instead of adding a pile of hac

author maintainer
#21 lore raw

PATCH v5 4/14 net: intel: introduce Intel Ethernet common library

In reply to Alexander Lobakin

Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, add unlikely() condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as a pkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. But skb_set_hash() only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hidden Kconfig symbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.

---
 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/intel/Kconfig            |   6 +
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 --------------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  72 +---
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 -----
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 --------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |   1 +
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  70 +---
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  88 -----
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ------------------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +---
 drivers/net/ethernet/intel/libie/Kconfig      |   8 +
 drivers/net/ethernet/intel/libie/Makefile     |   6 +
 drivers/net/ethernet/intel/libie/rx.c         | 110 ++++++
 include/linux/net/intel/libie/rx.h            | 128 +++++++
 20 files changed, 315 insertions(+), 1178 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Kconfig
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 include/linux/net/intel/libie/rx.h
MAINTAINERS +2 -1
index 509281e9e169..ab100b712e9b 100644
@@ -10664,7 +10664,8 @@ F:	Documentation/networking/device_drivers/ethernet/intel/
 F:	drivers/net/ethernet/intel/
 F:	drivers/net/ethernet/intel/*/
 F:	include/linux/avf/virtchnl.h
-F:	include/linux/net/intel/iidc.h
+F:	include/linux/net/intel/
+F:	include/linux/net/intel/*/
 
 INTEL ETHERNET PROTOCOL DRIVER FOR RDMA
 M:	Mustafa Ismail <[email protected]>
drivers/net/ethernet/intel/Kconfig +6 -0
index 06ddd7147c7f..dca17b095c0f 100644
@@ -16,6 +16,8 @@ config NET_VENDOR_INTEL
 
 if NET_VENDOR_INTEL
 
+source "drivers/net/ethernet/intel/libie/Kconfig"
+
 config E100
 	tristate "Intel(R) PRO/100+ support"
 	depends on PCI
@@ -225,6 +227,7 @@ config I40E
 	depends on PTP_1588_CLOCK_OPTIONAL
 	depends on PCI
 	select AUXILIARY_BUS
+	select LIBIE
 	select NET_DEVLINK
 	help
 	  This driver supports Intel(R) Ethernet Controller XL710 Family of
@@ -253,6 +256,8 @@ config I40E_DCB
 # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF
 config IAVF
 	tristate
+	select LIBIE
+
 config I40EVF
 	tristate "Intel(R) Ethernet Adaptive Virtual Function support"
 	select IAVF
@@ -283,6 +288,7 @@ config ICE
 	depends on GNSS || GNSS = n
 	select AUXILIARY_BUS
 	select DIMLIB
+	select LIBIE
 	select NET_DEVLINK
 	select PLDMFW
 	select DPLL
drivers/net/ethernet/intel/Makefile +1 -0
index dacb481ee5b1..60ca00a01de4 100644
@@ -16,3 +16,4 @@ obj-$(CONFIG_IAVF) += iavf/
 obj-$(CONFIG_FM10K) += fm10k/
 obj-$(CONFIG_ICE) += ice/
 obj-$(CONFIG_IDPF) += idpf/
+obj-$(CONFIG_LIBIE) += libie/
drivers/net/ethernet/intel/i40e/i40e_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_main.c +1 -0
index 7ded598a68e6..0517d5813a75 100644
@@ -100,6 +100,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Connection XL710 Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static struct workqueue_struct *i40e_wq;
drivers/net/ethernet/intel/i40e/i40e_prototype.h +0 -7
index af4269330581..f3dbf6dd2bc6 100644
@@ -371,13 +371,6 @@ void i40e_set_pci_config_data(struct i40e_hw *hw, u16 link_status);
 
 int i40e_set_mac_type(struct i40e_hw *hw);
 
-extern struct i40e_rx_ptype_decoded i40e_ptype_lookup[];
-
-static inline struct i40e_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return i40e_ptype_lookup[ptype];
-}
-
 /**
  * i40e_virtchnl_link_speed - Convert AdminQ link_speed to virtchnl definition
  * @link_speed: the speed to convert
drivers/net/ethernet/intel/i40e/i40e_txrx.c +17 -55

129 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -0
index c862ebcd2e39..ef9ae406d285 100644
@@ -45,6 +45,7 @@ MODULE_DEVICE_TABLE(pci, iavf_pci_tbl);
 MODULE_ALIAS("i40evf");
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Adaptive Virtual Function Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static const struct net_device_ops iavf_netdev_ops;
drivers/net/ethernet/intel/iavf/iavf_prototype.h +0 -7
index 4a48e6171405..48c3901381b4 100644
@@ -45,13 +45,6 @@ enum iavf_status iavf_aq_set_rss_lut(struct iavf_hw *hw, u16 seid,
 enum iavf_status iavf_aq_set_rss_key(struct iavf_hw *hw, u16 seid,
 				     struct iavf_aqc_get_set_rss_key_data *key);
 
-extern struct iavf_rx_ptype_decoded iavf_ptype_lookup[];
-
-static inline struct iavf_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return iavf_ptype_lookup[ptype];
-}
-
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/net/ethernet/intel/iavf/iavf_txrx.c +16 -54

125 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h +0 -316

326 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_main.c +1 -0
index 43ba3e55b8c1..04f5dcf57ba6 100644
@@ -35,6 +35,7 @@ static const char ice_copyright[] = "Copyright (c) 2018, Intel Corporation.";
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION(DRV_SUMMARY);
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 MODULE_FIRMWARE(ICE_DDP_PKG_FILE);
 
drivers/net/ethernet/intel/ice/ice_txrx_lib.c +18 -56

137 lines not on this page · raw message

drivers/net/ethernet/intel/libie/Kconfig new file +8 -0
new file mode 100644
index 000000000000..1eda4a5faa5a
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright(c) 2023 Intel Corporation.
+
+config LIBIE
+	tristate
+	help
+	  libie (Intel Ethernet library) is a common library containing
+	  routines shared between several Intel Ethernet drivers.
drivers/net/ethernet/intel/libie/Makefile new file +6 -0
new file mode 100644
index 000000000000..95e81d09b474
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright(c) 2023 Intel Corporation.
+
+obj-$(CONFIG_LIBIE)	+= libie.o
+
+libie-objs		+= rx.o
drivers/net/ethernet/intel/libie/rx.c new file +110 -0

110 lines not on this page · raw message

include/linux/net/intel/libie/rx.h new file +128 -0

128 lines not on this page · raw message

Signature · 1 line
2.42.0
author maintainer
#22 lore raw

PATCH v5 5/14 iavf: kill "legacy-rx" for good

In reply to Alexander Lobakin

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible, and thus obsolete.

* It had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* It implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.

Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.

There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft, and so on. Some stats:

Function: add/remove: 0/4 grow/shrink: 0/7 up/down: 0/-721 (-721) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)

---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 141 ------------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  10 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  84 +----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  27 +---
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   3 +-
 6 files changed, 8 insertions(+), 259 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
index e7ab89dc883a..64c94e9a6b2d 100644
@@ -287,7 +287,7 @@ struct iavf_adapter {
 #define IAVF_FLAG_RESET_PENDING		BIT(4)
 #define IAVF_FLAG_RESET_NEEDED		BIT(5)
 #define IAVF_FLAG_WB_ON_ITR_CAPABLE		BIT(6)
-#define IAVF_FLAG_LEGACY_RX			BIT(15)
+/* BIT(15) is free, was IAVF_FLAG_LEGACY_RX */
 #define IAVF_FLAG_REINIT_ITR_NEEDED		BIT(16)
 #define IAVF_FLAG_QUEUES_DISABLED		BIT(17)
 #define IAVF_FLAG_SETUP_NETDEV_FEATURES		BIT(18)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +0 -141

182 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -9
index ef9ae406d285..e99f6bdb4b4a 100644
@@ -740,9 +740,7 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 	struct iavf_hw *hw = &adapter->hw;
 	int i;
 
-	/* Legacy Rx will always default to a 2048 buffer size. */
-#if (PAGE_SIZE < 8192)
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX)) {
+	if (PAGE_SIZE < 8192) {
 		struct net_device *netdev = adapter->netdev;
 
 		/* For jumbo frames on systems with 4K pages we have to use
@@ -759,16 +757,10 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 		    (netdev->mtu <= ETH_DATA_LEN))
 			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 	}
-#endif
 
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
 		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-
-		if (adapter->flags & IAVF_FLAG_LEGACY_RX)
-			clear_ring_build_skb_enabled(&adapter->rx_rings[i]);
-		else
-			set_ring_build_skb_enabled(&adapter->rx_rings[i]);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -81

115 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -25
index 7e6ee32d19b6..4b412f7662e4 100644
@@ -82,20 +82,11 @@ enum iavf_dyn_idx_t {
 	BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP))
 
 /* Supported Rx Buffer Sizes (a multiple of 128) */
-#define IAVF_RXBUFFER_256   256
 #define IAVF_RXBUFFER_1536  1536  /* 128B aligned standard Ethernet frame */
 #define IAVF_RXBUFFER_2048  2048
 #define IAVF_RXBUFFER_3072  3072  /* Used for large frames w/ padding */
 #define IAVF_MAX_RXBUFFER   9728  /* largest size for single descriptor */
 
-/* NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we
- * reserve 2 more, and skb_shared_info adds an additional 384 bytes more,
- * this adds up to 512 bytes of extra data meaning the smallest allocation
- * we could have is 1K.
- * i.e. RXBUFFER_256 --> 960 byte skb (size-1024 slab)
- * i.e. RXBUFFER_512 --> 1216 byte skb (size-2048 slab)
- */
-#define IAVF_RX_HDR_SIZE IAVF_RXBUFFER_256
 #define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))
 #define iavf_rx_desc iavf_32byte_rx_desc
 
@@ -362,7 +353,8 @@ struct iavf_ring {
 
 	u16 flags;
 #define IAVF_TXR_FLAGS_WB_ON_ITR		BIT(0)
-#define IAVF_RXR_FLAGS_BUILD_SKB_ENABLED	BIT(1)
+/* BIT(1) is free, was IAVF_RXR_FLAGS_BUILD_SKB_ENABLED */
+/* BIT(2) is free */
 #define IAVF_TXRX_FLAGS_VLAN_TAG_LOC_L2TAG1	BIT(3)
 #define IAVF_TXR_FLAGS_VLAN_TAG_LOC_L2TAG2	BIT(4)
 #define IAVF_RXR_FLAGS_VLAN_TAG_LOC_L2TAG2_2	BIT(5)
@@ -393,21 +385,6 @@ struct iavf_ring {
 					 */
 } ____cacheline_internodealigned_in_smp;
 
-static inline bool ring_uses_build_skb(struct iavf_ring *ring)
-{
-	return !!(ring->flags & IAVF_RXR_FLAGS_BUILD_SKB_ENABLED);
-}
-
-static inline void set_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags |= IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
-static inline void clear_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags &= ~IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
 #define IAVF_ITR_ADAPTIVE_MIN_INC	0x0002
 #define IAVF_ITR_ADAPTIVE_MIN_USECS	0x0002
 #define IAVF_ITR_ADAPTIVE_MAX_USECS	0x007e
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -2
index 64c4443dbef9..37d0e4313130 100644
@@ -289,8 +289,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		return;
 
 	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX) &&
-	    (adapter->netdev->mtu <= ETH_DATA_LEN))
+	if (adapter->netdev->mtu <= ETH_DATA_LEN)
 		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
Signature · 1 line
2.42.0
author maintainer
#23 lore raw

PATCH v5 6/14 iavf: drop page splitting and recycling

In reply to Alexander Lobakin

As an intermediate step, remove all page splitting/recycling code. Just always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63 &iavf_rx_buf structures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:

Function: add/remove: 0/2 grow/shrink: 0/7 up/down: 0/-744 (-744)

Although the array of &iavf_rx_buf is barely used now and could be replaced with just page pointer array, don't touch it now to not complicate replacing it with libie Rx buffer struct later on. No surprise perf loses up to 30% here, but that regression will go away once PP lands.
Note that iavf_rx_pg_*() definitions are left to reduce diffstat. They will be removed with the conversion to Page Pool.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  24 +--
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 152 +-----------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  65 --------
 drivers/net/ethernet/intel/iavf/iavf_type.h   |   2 -
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   8 +-
 5 files changed, 10 insertions(+), 241 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -23
index e99f6bdb4b4a..37c79d78388e 100644
@@ -736,32 +736,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)
  **/
 static void iavf_configure_rx(struct iavf_adapter *adapter)
 {
-	unsigned int rx_buf_len = IAVF_RXBUFFER_2048;
 	struct iavf_hw *hw = &adapter->hw;
-	int i;
-
-	if (PAGE_SIZE < 8192) {
-		struct net_device *netdev = adapter->netdev;
 
-		/* For jumbo frames on systems with 4K pages we have to use
-		 * an order 1 page, so we might as well increase the size
-		 * of our Rx buffer to make better use of the available space
-		 */
-		rx_buf_len = IAVF_RXBUFFER_3072;
-
-		/* We use a 1536 buffer size for configurations with
-		 * standard Ethernet mtu.  On x86 this gives us enough room
-		 * for shared info and 192 bytes of padding.
-		 */
-		if (!IAVF_2K_TOO_SMALL_WITH_PADDING &&
-		    (netdev->mtu <= ETH_DATA_LEN))
-			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-	}
-
-	for (i = 0; i < adapter->num_active_queues; i++) {
+	for (u32 i = 0; i < adapter->num_active_queues; i++)
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
-		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-	}
 }
 
 /**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +8 -144

258 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -65

109 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -2
index 23ded4fcd94f..f6b09e57abce 100644
@@ -10,8 +10,6 @@
 #include "iavf_adminq.h"
 #include "iavf_devids.h"
 
-#define IAVF_RXQ_CTX_DBUFF_SHIFT 7
-
 /* IAVF_MASK is a macro used on 32 bit registers */
 #define IAVF_MASK(mask, shift) ((u32)(mask) << (shift))
 
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -7
index 37d0e4313130..12da9401c46d 100644
@@ -288,10 +288,6 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 	if (!vqci)
 		return;
 
-	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (adapter->netdev->mtu <= ETH_DATA_LEN)
-		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
 	vqci->num_queue_pairs = pairs;
 	vqpi = vqci->qpair;
@@ -308,9 +304,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		vqpi->rxq.ring_len = adapter->rx_rings[i].count;
 		vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
 		vqpi->rxq.max_pkt_size = max_frame;
-		vqpi->rxq.databuffer_size =
-			ALIGN(adapter->rx_rings[i].rx_buf_len,
-			      BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));
+		vqpi->rxq.databuffer_size = IAVF_RXBUFFER_3072;
 		if (CRC_OFFLOAD_ALLOWED(adapter))
 			vqpi->rxq.crc_disable = !!(adapter->netdev->features &
 						   NETIF_F_RXFCS);
Signature · 1 line
2.42.0
author maintainer
#24 lore raw

PATCH v5 7/14 page_pool: constify some read-only function arguments

In reply to Alexander Lobakin

There are several functions taking pointers to data they don't modify. This includes statistics fetching, page and page_pool parameters, etc. Constify the pointers, so that call sites will be able to pass const pointers as well.
No functional changes, no visible changes in functions sizes.

---
 include/net/page_pool/helpers.h | 12 ++++++------
 net/core/page_pool.c            | 12 ++++++------
 2 files changed, 12 insertions(+), 12 deletions(-)
include/net/page_pool/helpers.h +6 -6
index 28bec368b8e9..528a76c66270 100644
@@ -59,14 +59,14 @@
 #ifdef CONFIG_PAGE_POOL_STATS
 int page_pool_ethtool_stats_get_count(void);
 u8 *page_pool_ethtool_stats_get_strings(u8 *data);
-u64 *page_pool_ethtool_stats_get(u64 *data, void *stats);
+u64 *page_pool_ethtool_stats_get(u64 *data, const void *stats);
 
 /*
  * Drivers that wish to harvest page pool stats and report them to users
  * (perhaps via ethtool, debugfs, or another mechanism) can allocate a
  * struct page_pool_stats call page_pool_get_stats to get stats for the specified pool.
  */
-bool page_pool_get_stats(struct page_pool *pool,
+bool page_pool_get_stats(const struct page_pool *pool,
 			 struct page_pool_stats *stats);
 #else
 static inline int page_pool_ethtool_stats_get_count(void)
@@ -79,7 +79,7 @@ static inline u8 *page_pool_ethtool_stats_get_strings(u8 *data)
 	return data;
 }
 
-static inline u64 *page_pool_ethtool_stats_get(u64 *data, void *stats)
+static inline u64 *page_pool_ethtool_stats_get(u64 *data, const void *stats)
 {
 	return data;
 }
@@ -210,8 +210,8 @@ static inline void *page_pool_dev_alloc_va(struct page_pool *pool,
  * Get the stored dma direction. A driver might decide to store this locally
  * and avoid the extra cache line from page_pool to determine the direction.
  */
-static
-inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)
+static inline enum dma_data_direction
+page_pool_get_dma_dir(const struct page_pool *pool)
 {
 	return pool->p.dma_dir;
 }
@@ -368,7 +368,7 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va,
  * Fetch the DMA address of the page. The page pool to which the page belongs
  * must had been created with PP_FLAG_DMA_MAP.
  */
-static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
+static inline dma_addr_t page_pool_get_dma_addr(const struct page *page)
 {
 	dma_addr_t ret = page->dma_addr & ~PAGE_POOL_DMA_ADDR_NEED_SYNC;
 
net/core/page_pool.c +6 -6
index 09bdfdfc4788..e5ee49e8799c 100644
@@ -69,7 +69,7 @@ static const char pp_stats[][ETH_GSTRING_LEN] = {
  * is passed to this API which is filled in. The caller can then report
  * those stats to the user (perhaps via ethtool, debugfs, etc.).
  */
-bool page_pool_get_stats(struct page_pool *pool,
+bool page_pool_get_stats(const struct page_pool *pool,
 			 struct page_pool_stats *stats)
 {
 	int cpu = 0;
@@ -119,9 +119,9 @@ int page_pool_ethtool_stats_get_count(void)
 }
 EXPORT_SYMBOL(page_pool_ethtool_stats_get_count);
 
-u64 *page_pool_ethtool_stats_get(u64 *data, void *stats)
+u64 *page_pool_ethtool_stats_get(u64 *data, const void *stats)
 {
-	struct page_pool_stats *pool_stats = stats;
+	const struct page_pool_stats *pool_stats = stats;
 
 	*data++ = pool_stats->alloc_stats.fast;
 	*data++ = pool_stats->alloc_stats.slow;
@@ -338,8 +338,8 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)
 	return page;
 }
 
-static void page_pool_dma_sync_for_device(struct page_pool *pool,
-					  struct page *page,
+static void page_pool_dma_sync_for_device(const struct page_pool *pool,
+					  const struct page *page,
 					  unsigned int dma_sync_size)
 {
 	dma_addr_t dma_addr = page_pool_get_dma_addr(page);
@@ -509,7 +509,7 @@ EXPORT_SYMBOL(page_pool_alloc_pages);
  */
 #define _distance(a, b)	(s32)((a) - (b))
 
-static s32 page_pool_inflight(struct page_pool *pool)
+static s32 page_pool_inflight(const struct page_pool *pool)
 {
 	u32 release_cnt = atomic_read(&pool->pages_state_release_cnt);
 	u32 hold_cnt = READ_ONCE(pool->pages_state_hold_cnt);
Signature · 1 line
2.42.0
author maintainer
#25 lore raw

PATCH v5 8/14 page_pool: add DMA-sync-for-CPU inline helpers

In reply to Alexander Lobakin

Each driver is responsible for syncing buffers written by HW for CPU before accessing them. Almost each PP-enabled driver uses the same pattern, which could be shorthanded into a static inline to make driver code a little bit more compact.
Introduce a couple such functions. The first one is sorta internal and performs DMA synchronization unconditionally for the size passed from the driver. The second checks whether the synchronization is needed for this pool first and is the main one to be used by the drivers.

---
 include/net/page_pool/helpers.h | 43 +++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
include/net/page_pool/helpers.h +43 -0
index 528a76c66270..797275e75d38 100644
@@ -432,6 +432,49 @@ static inline bool page_pool_dma_addr_need_sync(const struct page *page)
 	return page->dma_addr & PAGE_POOL_DMA_ADDR_NEED_SYNC;
 }
 
+/**
+ * __page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by HW
+ * @pool: &page_pool the @page belongs to
+ * @page: page to sync
+ * @offset: offset from page start to "hard" start if using frags
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Can be used as a shorthand to sync Rx pages before accessing them in the
+ * driver. Caller must ensure the pool was created with ```PP_FLAG_DMA_MAP```.
+ * Note that this version performs DMA sync unconditionally, even if the
+ * associated PP doesn't perform sync-for-device. Consider the non-underscored
+ * version first if unsure.
+ */
+static inline void __page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+						const struct page *page,
+						u32 offset, u32 dma_sync_size)
+{
+	dma_sync_single_range_for_cpu(pool->p.dev,
+				      page_pool_get_dma_addr(page),
+				      offset + pool->p.offset, dma_sync_size,
+				      page_pool_get_dma_dir(pool));
+}
+
+/**
+ * page_pool_dma_sync_for_cpu - sync Rx page for CPU if needed
+ * @pool: &page_pool the @page belongs to
+ * @page: page to sync
+ * @offset: offset from page start to "hard" start if using frags
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Performs DMA sync for CPU, but *only* when both:
+ * 1) page_pool was created with ```PP_FLAG_DMA_SYNC_DEV``` to manage DMA sync;
+ * 2) sync shortcut is not available (IOMMU, swiotlb, non-coherent DMA, ...)
+ */
+static inline void page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+					      const struct page *page,
+					      u32 offset, u32 dma_sync_size)
+{
+	if (page_pool_dma_addr_need_sync(page))
+		__page_pool_dma_sync_for_cpu(pool, page, offset,
+					     dma_sync_size);
+}
+
 static inline bool page_pool_put(struct page_pool *pool)
 {
 	return refcount_dec_and_test(&pool->user_cnt);
Signature · 1 line
2.42.0
author maintainer
#26 lore raw

PATCH v5 9/14 libie: add Rx buffer management (via Page Pool)

In reply to Alexander Lobakin

Add a couple intuitive helpers to hide Rx buffer implementation details in the library and not multiplicate it between drivers. The settings are optimized for Intel hardware, but nothing really HW-specific here. Use the new page_pool_dev_alloc() to dynamically switch between split-page and full-page modes depending on MTU, page size, required headroom etc. For example, on x86_64 with the default driver settings each page is shared between 2 buffers. Turning on XDP (not in this series) -> increasing headroom requirement pushes truesize out of 2048 boundary, leading to that each buffer starts getting a full page. The "ceiling" limit is %PAGE_SIZE, as only order-0 pages are used to avoid compound overhead. For the above architecture, this means maximum linear frame size of 3712 w/o XDP.
Not that &libie_rx_queue is not a complete queue/ring structure for now, rather a shim, but eventually the libie-enabled drivers will move to it, with iavf being the first one.

---
 drivers/net/ethernet/intel/libie/Kconfig |   1 +
 drivers/net/ethernet/intel/libie/rx.c    |  69 ++++++++++++
 include/linux/net/intel/libie/rx.h       | 132 ++++++++++++++++++++++-
 3 files changed, 201 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/libie/Kconfig +1 -0
index 1eda4a5faa5a..6e0162fb94d2 100644
@@ -3,6 +3,7 @@
 
 config LIBIE
 	tristate
+	select PAGE_POOL
 	help
 	  libie (Intel Ethernet library) is a common library containing
 	  routines shared between several Intel Ethernet drivers.
drivers/net/ethernet/intel/libie/rx.c +69 -0

75 lines not on this page · raw message

include/linux/net/intel/libie/rx.h +131 -1

138 lines not on this page · raw message

Signature · 1 line
2.42.0
author maintainer
#27 lore raw

PATCH v5 10/14 iavf: pack iavf_ring more efficiently

In reply to Alexander Lobakin

Before replacing the Rx buffer management with libie, clean up &iavf_ring a bit.
There are several fields not used anywhere in the code -- simply remove them. Move ::tail up to remove a hole. Replace ::arm_wb boolean with 1-bit flag in ::flags to free 1 more byte. Finally, move ::prev_pkt_ctr out of &iavf_tx_queue_stats -- it doesn't belong there (used for Tx stall detection). Place it next to the stats on the ring itself to fill the 4-byte slot.
The result: no holes and all the hot fields fit into the first 64-byte cacheline.

---
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 12 +++++------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h | 22 +++------------------
 2 files changed, 9 insertions(+), 25 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +6 -6
index 665ee1feb877..62f976d322ab 100644
@@ -184,7 +184,7 @@ void iavf_detect_recover_hung(struct iavf_vsi *vsi)
 			 * pending work.
 			 */
 			packets = tx_ring->stats.packets & INT_MAX;
-			if (tx_ring->tx_stats.prev_pkt_ctr == packets) {
+			if (tx_ring->prev_pkt_ctr == packets) {
 				iavf_force_wb(vsi, tx_ring->q_vector);
 				continue;
 			}
@@ -193,7 +193,7 @@ void iavf_detect_recover_hung(struct iavf_vsi *vsi)
 			 * to iavf_get_tx_pending()
 			 */
 			smp_rmb();
-			tx_ring->tx_stats.prev_pkt_ctr =
+			tx_ring->prev_pkt_ctr =
 			  iavf_get_tx_pending(tx_ring, true) ? packets : -1;
 		}
 	}
@@ -319,7 +319,7 @@ static bool iavf_clean_tx_irq(struct iavf_vsi *vsi,
 		    ((j / WB_STRIDE) == 0) && (j > 0) &&
 		    !test_bit(__IAVF_VSI_DOWN, vsi->state) &&
 		    (IAVF_DESC_UNUSED(tx_ring) != tx_ring->count))
-			tx_ring->arm_wb = true;
+			tx_ring->flags |= IAVF_TXR_FLAGS_ARM_WB;
 	}
 
 	/* notify netdev of completed buffers */
@@ -674,7 +674,7 @@ int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring)
 
 	tx_ring->next_to_use = 0;
 	tx_ring->next_to_clean = 0;
-	tx_ring->tx_stats.prev_pkt_ctr = -1;
+	tx_ring->prev_pkt_ctr = -1;
 	return 0;
 
 err:
@@ -1494,8 +1494,8 @@ int iavf_napi_poll(struct napi_struct *napi, int budget)
 			clean_complete = false;
 			continue;
 		}
-		arm_wb |= ring->arm_wb;
-		ring->arm_wb = false;
+		arm_wb |= !!(ring->flags & IAVF_TXR_FLAGS_ARM_WB);
+		ring->flags &= ~IAVF_TXR_FLAGS_ARM_WB;
 	}
 
 	/* Handle case where we are called by netpoll with a budget of 0 */
drivers/net/ethernet/intel/iavf/iavf_txrx.h +3 -19
index 720bca0e6716..9b8154c5f4fb 100644
@@ -228,7 +228,6 @@ struct iavf_tx_queue_stats {
 	u64 tx_done_old;
 	u64 tx_linearize;
 	u64 tx_force_wb;
-	int prev_pkt_ctr;
 	u64 tx_lost_interrupt;
 };
 
@@ -238,12 +237,6 @@ struct iavf_rx_queue_stats {
 	u64 alloc_buff_failed;
 };
 
-enum iavf_ring_state_t {
-	__IAVF_TX_FDIR_INIT_DONE,
-	__IAVF_TX_XPS_INIT_DONE,
-	__IAVF_RING_STATE_NBITS /* must be last */
-};
-
 /* some useful defines for virtchannel interface, which
  * is the only remaining user of header split
  */
@@ -265,10 +258,8 @@ struct iavf_ring {
 		struct iavf_tx_buffer *tx_bi;
 		struct iavf_rx_buffer *rx_bi;
 	};
-	DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
-	u16 queue_index;		/* Queue number of ring */
-	u8 dcb_tc;			/* Traffic class of ring */
 	u8 __iomem *tail;
+	u16 queue_index;		/* Queue number of ring */
 
 	/* high bit set means dynamic, use accessors routines to read/write.
 	 * hardware only supports 2us resolution for the ITR registers.
@@ -278,22 +269,14 @@ struct iavf_ring {
 	u16 itr_setting;
 
 	u16 count;			/* Number of descriptors */
-	u16 reg_idx;			/* HW register index of the ring */
 
 	/* used in interrupt processing */
 	u16 next_to_use;
 	u16 next_to_clean;
 
-	u8 atr_sample_rate;
-	u8 atr_count;
-
-	bool ring_active;		/* is ring online or not */
-	bool arm_wb;		/* do something to arm write back */
-	u8 packet_stride;
-
 	u16 flags;
 #define IAVF_TXR_FLAGS_WB_ON_ITR		BIT(0)
-/* BIT(1) is free, was IAVF_RXR_FLAGS_BUILD_SKB_ENABLED */
+#define IAVF_TXR_FLAGS_ARM_WB			BIT(1)
 /* BIT(2) is free */
 #define IAVF_TXRX_FLAGS_VLAN_TAG_LOC_L2TAG1	BIT(3)
 #define IAVF_TXR_FLAGS_VLAN_TAG_LOC_L2TAG2	BIT(4)
@@ -307,6 +290,7 @@ struct iavf_ring {
 		struct iavf_rx_queue_stats rx_stats;
 	};
 
+	int prev_pkt_ctr;		/* For Tx stall detection */
 	unsigned int size;		/* length of descriptor ring in bytes */
 	dma_addr_t dma;			/* physical address of ring */
 
Signature · 1 line
2.42.0
author maintainer
#28 lore raw

PATCH v5 11/14 iavf: switch to Page Pool

In reply to Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c   |   7 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 246 ++++++------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  34 +--
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  10 +-
 4 files changed, 92 insertions(+), 205 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +4 -3
index 37c79d78388e..7aa3b072a416 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 /* All iavf tracepoints are defined by the include below, which must
@@ -1608,7 +1610,6 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		rx_ring = &adapter->rx_rings[i];
 		rx_ring->queue_index = i;
 		rx_ring->netdev = adapter->netdev;
-		rx_ring->dev = &adapter->pdev->dev;
 		rx_ring->count = adapter->rx_desc_count;
 		rx_ring->itr_setting = IAVF_ITR_RX_DEF;
 	}
@@ -2642,9 +2643,9 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 	iavf_set_ethtool_ops(netdev);
 	netdev->watchdog_timeo = 5 * HZ;
 
-	/* MTU range: 68 - 9710 */
+	/* MTU range: 68 - 16356 */
 	netdev->min_mtu = ETH_MIN_MTU;
-	netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+	netdev->max_mtu = LIBIE_MAX_MTU;
 
 	if (!is_valid_ether_addr(adapter->hw.mac.addr)) {
 		dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",
drivers/net/ethernet/intel/iavf/iavf_txrx.c +73 -173

406 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +9 -25
index 9b8154c5f4fb..72c6bc64d94e 100644
@@ -81,18 +81,8 @@ enum iavf_dyn_idx_t {
 	BIT_ULL(IAVF_FILTER_PCTYPE_NONF_UNICAST_IPV6_UDP) | \
 	BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP))
 
-/* Supported Rx Buffer Sizes (a multiple of 128) */
-#define IAVF_RXBUFFER_3072  3072  /* Used for large frames w/ padding */
-#define IAVF_MAX_RXBUFFER   9728  /* largest size for single descriptor */
-
-#define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))
 #define iavf_rx_desc iavf_32byte_rx_desc
 
-#define IAVF_RX_DMA_ATTR \
-	(DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
-
-#define IAVF_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
-
 /**
  * iavf_test_staterr - tests bits in Rx descriptor status and error fields
  * @rx_desc: pointer to receive descriptor (in le64 format)
@@ -211,12 +201,6 @@ struct iavf_tx_buffer {
 	u32 tx_flags;
 };
 
-struct iavf_rx_buffer {
-	dma_addr_t dma;
-	struct page *page;
-	__u32 page_offset;
-};
-
 struct iavf_queue_stats {
 	u64 packets;
 	u64 bytes;
@@ -252,13 +236,18 @@ struct iavf_rx_queue_stats {
 struct iavf_ring {
 	struct iavf_ring *next;		/* pointer to next ring in q_vector */
 	void *desc;			/* Descriptor ring memory */
-	struct device *dev;		/* Used for DMA mapping */
+	union {
+		struct page_pool *pp;	/* Used on Rx for buffer management */
+		struct device *dev;	/* Used on Tx for DMA mapping */
+	};
 	struct net_device *netdev;	/* netdev ring maps to */
 	union {
+		struct libie_rx_buffer *rx_bi;
 		struct iavf_tx_buffer *tx_bi;
-		struct iavf_rx_buffer *rx_bi;
 	};
 	u8 __iomem *tail;
+	u32 truesize;
+
 	u16 queue_index;		/* Queue number of ring */
 
 	/* high bit set means dynamic, use accessors routines to read/write.
@@ -306,6 +295,8 @@ struct iavf_ring {
 					 * iavf_clean_rx_ring_irq() is called
 					 * for this ring.
 					 */
+
+	u32 rx_buf_len;
 } ____cacheline_internodealigned_in_smp;
 
 #define IAVF_ITR_ADAPTIVE_MIN_INC	0x0002
@@ -329,13 +320,6 @@ struct iavf_ring_container {
 #define iavf_for_each_ring(pos, head) \
 	for (pos = (head).ring; pos != NULL; pos = pos->next)
 
-static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
-{
-	return 0;
-}
-
-#define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
-
 bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring);
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +6 -4
index 12da9401c46d..58dad471004d 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 
@@ -268,13 +270,13 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)
 void iavf_configure_queues(struct iavf_adapter *adapter)
 {
 	struct virtchnl_vsi_queue_config_info *vqci;
-	int i, max_frame = adapter->vf_res->max_mtu;
 	int pairs = adapter->num_active_queues;
 	struct virtchnl_queue_pair_info *vqpi;
+	u32 i, max_frame;
 	size_t len;
 
-	if (max_frame > IAVF_MAX_RXBUFFER || !max_frame)
-		max_frame = IAVF_MAX_RXBUFFER;
+	max_frame = LIBIE_MAX_RX_FRM_LEN(adapter->rx_rings->pp->p.offset);
+	max_frame = min_not_zero(adapter->vf_res->max_mtu, max_frame);
 
 	if (adapter->current_op != VIRTCHNL_OP_UNKNOWN) {
 		/* bail because we already have a command pending */
@@ -304,7 +306,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		vqpi->rxq.ring_len = adapter->rx_rings[i].count;
 		vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
 		vqpi->rxq.max_pkt_size = max_frame;
-		vqpi->rxq.databuffer_size = IAVF_RXBUFFER_3072;
+		vqpi->rxq.databuffer_size = adapter->rx_rings[i].rx_buf_len;
 		if (CRC_OFFLOAD_ALLOWED(adapter))
 			vqpi->rxq.crc_disable = !!(adapter->netdev->features &
 						   NETIF_F_RXFCS);
Signature · 1 line
2.42.0
author maintainer
#29 lore raw

PATCH v5 12/14 libie: add common queue stats

In reply to Alexander Lobakin

Next stop, per-queue private stats. They have only subtle differences from driver to driver and can easily be resolved.
Define common structures, inline helpers and Ethtool helpers to collect, update and export the statistics. Use u64_stats_t right from the start, as well as the corresponding helpers to ensure tear-free operations. For the NAPI parts of both Rx and Tx, also define small onstack containers to update them in polling loops and then sync the actual containers once a loop ends.
The drivers will be switched to use this API later on a per-driver basis, along with conversion to PP.

---
 drivers/net/ethernet/intel/libie/Makefile |   1 +
 drivers/net/ethernet/intel/libie/stats.c  | 122 +++++++++++++++
 include/linux/net/intel/libie/rx.h        |   5 +
 include/linux/net/intel/libie/stats.h     | 181 ++++++++++++++++++++++
 4 files changed, 309 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/stats.h
drivers/net/ethernet/intel/libie/Makefile +1 -0
index 95e81d09b474..76f32253481b 100644
@@ -4,3 +4,4 @@
 obj-$(CONFIG_LIBIE)	+= libie.o
 
 libie-objs		+= rx.o
+libie-objs		+= stats.o
drivers/net/ethernet/intel/libie/stats.c new file +122 -0

122 lines not on this page · raw message

include/linux/net/intel/libie/rx.h +5 -0
index 06c4f00dad63..8542815c9973 100644
@@ -46,6 +46,8 @@
 
 /* Rx buffer management */
 
+struct libie_rq_stats;
+
 /**
  * struct libie_rx_buffer - structure representing an Rx buffer
  * @page: page holding the buffer
@@ -68,6 +70,7 @@ struct libie_rx_buffer {
  * @rx_bi: array of Rx buffers
  * @truesize: size to allocate per buffer, w/overhead
  * @count: number of descriptors/buffers the queue has
+ * @stats: pointer to the software per-queue stats
  * @rx_buf_len: HW-writeable length per each buffer
  */
 struct libie_rx_queue {
@@ -77,6 +80,8 @@ struct libie_rx_queue {
 	u32			truesize;
 	u32			count;
 
+	struct libie_rq_stats	*stats;
+
 	/* Cold fields */
 	u32			rx_buf_len;
 };
include/linux/net/intel/libie/stats.h new file +181 -0

181 lines not on this page · raw message

Signature · 1 line
2.42.0
author maintainer
#30 lore raw

PATCH v5 13/14 libie: add per-queue Page Pool stats

In reply to Alexander Lobakin

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL_STATS is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.
They are also a bit special in terms of how they are obtained. One &page_pool accumulates statistics until it's destroyed obviously, which happens on ifdown. So, in order to not lose any statistics, get the stats and store them in the queue container before destroying the pool. This container survives ifups/downs, so it basically stores the statistics accumulated since the very first pool was allocated on this queue. When it's needed to export the stats, first get the numbers from this container and then add the "live" numbers -- the ones that the current active pool returns. The result values will always represent the actual device-lifetime stats.
There's a cast from &page_pool_stats to u64 * in a couple functions, but they are guarded with stats asserts to make sure it's safe to do. FWIW it saves a lot of object code.

---
 drivers/net/ethernet/intel/libie/internal.h | 20 ++++++
 drivers/net/ethernet/intel/libie/rx.c       |  9 +++
 drivers/net/ethernet/intel/libie/stats.c    | 68 +++++++++++++++++++++
 include/linux/net/intel/libie/stats.h       | 34 ++++++++++-
 4 files changed, 130 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
drivers/net/ethernet/intel/libie/internal.h new file +20 -0
new file mode 100644
index 000000000000..13bb0a89f59e
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* libie internal declarations not to be used in the drivers.
+ *
+ * Copyright(c) 2023 Intel Corporation.
+ */
+
+#ifndef __LIBIE_INTERNAL_H
+#define __LIBIE_INTERNAL_H
+
+struct libie_rx_queue;
+
+#ifdef CONFIG_PAGE_POOL_STATS
+void libie_rq_stats_sync_pp(const struct libie_rx_queue *rq);
+#else
+static inline void libie_rq_stats_sync_pp(const struct libie_rx_queue *rq)
+{
+}
+#endif
+
+#endif /* __LIBIE_INTERNAL_H */
drivers/net/ethernet/intel/libie/rx.c +9 -0
index 520a269f7d31..fcc5c3c44645 100644
@@ -3,6 +3,8 @@
 
 #include <linux/net/intel/libie/rx.h>
 
+#include "internal.h"
+
 /* Rx buffer management */
 
 /**
@@ -64,9 +66,16 @@ EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);
 /**
  * libie_rx_page_pool_destroy - destroy a &page_pool created by libie
  * @rq: receive queue to process
+ *
+ * As the stats usually has the same lifetime as the device, but PP is usually
+ * created/destroyed on ifup/ifdown, in order to not lose the stats accumulated
+ * during the last ifup, the PP stats need to be added to the driver stats
+ * container. Then the PP gets destroyed.
  */
 void libie_rx_page_pool_destroy(struct libie_rx_queue *rq)
 {
+	libie_rq_stats_sync_pp(rq);
+
 	page_pool_destroy(rq->pp);
 	rq->pp = NULL;
 }
drivers/net/ethernet/intel/libie/stats.c +68 -0

86 lines not on this page · raw message

include/linux/net/intel/libie/stats.h +33 -1
index 4e6dfb8c715f..f913968d7516 100644
@@ -49,6 +49,17 @@
  * fragments: number of processed descriptors carrying only a fragment
  * alloc_page_fail: number of Rx page allocation fails
  * build_skb_fail: number of build_skb() fails
+ * pp_alloc_fast: pages taken from the cache or ring
+ * pp_alloc_slow: actual page allocations
+ * pp_alloc_slow_ho: non-order-0 page allocations
+ * pp_alloc_empty: number of times the pool was empty
+ * pp_alloc_refill: number of cache refills
+ * pp_alloc_waive: NUMA node mismatches during recycling
+ * pp_recycle_cached: direct recyclings into the cache
+ * pp_recycle_cache_full: number of times the cache was full
+ * pp_recycle_ring: recyclings into the ring
+ * pp_recycle_ring_full: number of times the ring was full
+ * pp_recycle_released_ref: pages released due to elevated refcnt
  */
 
 #define DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
@@ -60,9 +71,27 @@
 	act(alloc_page_fail)				\
 	act(build_skb_fail)
 
+#ifdef CONFIG_PAGE_POOL_STATS
+#define DECLARE_LIBIE_RQ_PP_STATS(act)			\
+	act(pp_alloc_fast)				\
+	act(pp_alloc_slow)				\
+	act(pp_alloc_slow_ho)				\
+	act(pp_alloc_empty)				\
+	act(pp_alloc_refill)				\
+	act(pp_alloc_waive)				\
+	act(pp_recycle_cached)				\
+	act(pp_recycle_cache_full)			\
+	act(pp_recycle_ring)				\
+	act(pp_recycle_ring_full)			\
+	act(pp_recycle_released_ref)
+#else
+#define DECLARE_LIBIE_RQ_PP_STATS(act)
+#endif
+
 #define DECLARE_LIBIE_RQ_STATS(act)			\
 	DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
-	DECLARE_LIBIE_RQ_FAIL_STATS(act)
+	DECLARE_LIBIE_RQ_FAIL_STATS(act)		\
+	DECLARE_LIBIE_RQ_PP_STATS(act)
 
 struct libie_rx_queue;
 
@@ -74,6 +103,9 @@ struct libie_rq_stats {
 #define act(s)	u64_stats_t	s;
 			DECLARE_LIBIE_RQ_NAPI_STATS(act);
 			DECLARE_LIBIE_RQ_FAIL_STATS(act);
+			struct_group(pp,
+				DECLARE_LIBIE_RQ_PP_STATS(act);
+			);
 #undef act
 		};
 		DECLARE_FLEX_ARRAY(u64_stats_t, raw);
Signature · 1 line
2.42.0
author maintainer
#31 lore raw

Re: [PATCH net-next v5 13/14] libie: add per-queue Page Pool stats

From: Alexander Lobakin <[email protected]>
Date: Fri, 24 Nov 2023 16:47:31 +0100

Quoted 4 lines

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL_STATS is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.

Jakub,

Do I get it correctly that after Page Pool Netlink introspection was merged, this commit makes no sense and we shouln't add PP stats to the drivers private ones?

Quoted 5 lines

They are also a bit special in terms of how they are obtained. One &page_pool accumulates statistics until it's destroyed obviously, which happens on ifdown. So, in order to not lose any statistics, get the stats and store them in the queue container before destroying the pool. This container survives ifups/downs, so it basically stores the

Quoted 8 lines

statistics accumulated since the very first pool was allocated on this queue. When it's needed to export the stats, first get the numbers from this container and then add the "live" numbers -- the ones that the current active pool returns. The result values will always represent the actual device-lifetime stats.
There's a cast from &page_pool_stats to u64 * in a couple functions, but they are guarded with stats asserts to make sure it's safe to do. FWIW it saves a lot of object code.

Thanks,
Olek

maintainer
#32 lore raw

Re: [PATCH net-next v5 13/14] libie: add per-queue Page Pool stats

Quoted 8 lines · On Wed, 29 Nov 2023 14:40:33 +0100 Alexander Lobakin wrote:
Quoted 4 lines

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL_STATS is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.

Do I get it correctly that after Page Pool Netlink introspection was merged, this commit makes no sense and we shouln't add PP stats to the drivers private ones?

Yes, 100%.

FWIW I am aware that better tooling would be good so non-developers could access to the PP Netlink :( I'm thinking we should clean up YNL lib packaging a little and try to convince iproute2 maintainers to accept simple CLI built on top of it.

author maintainer
#33 lore raw

Re: [PATCH net-next v5 13/14] libie: add per-queue Page Pool stats

From: Jakub Kicinski <[email protected]>
Date: Wed, 29 Nov 2023 06:29:14 -0800

Quoted 11 lines · On Wed, 29 Nov 2023 14:40:33 +0100 Alexander Lobakin wrote:
Quoted 8 lines
Quoted 4 lines

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL_STATS is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.

Do I get it correctly that after Page Pool Netlink introspection was merged, this commit makes no sense and we shouln't add PP stats to the drivers private ones?

Yes, 100%.

Meh, this way the stats won't survive ifdown/ifup cycles as usually page_pools get destroyed on ifdown :z
In that patch, I backup the PP stats to a device-lifetime container when the pool gets destroyed, maybe we could do something similar?

Quoted 5 lines

FWIW I am aware that better tooling would be good so non-developers could access to the PP Netlink :( I'm thinking we should clean up YNL lib packaging a little and try to convince iproute2 maintainers to accept simple CLI built on top of it.

Thanks,
Olek

author maintainer
#34 lore raw

Re: [PATCH net-next v5 13/14] libie: add per-queue Page Pool stats

From: Alexander Lobakin <[email protected]>
Date: Thu, 30 Nov 2023 17:01:23 +0100

Quoted 13 lines

From: Jakub Kicinski <[email protected]>
Date: Wed, 29 Nov 2023 06:29:14 -0800

Quoted 10 lines · On Wed, 29 Nov 2023 14:40:33 +0100 Alexander Lobakin wrote:
Quoted 8 lines
Quoted 4 lines

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL_STATS is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.

Do I get it correctly that after Page Pool Netlink introspection was merged, this commit makes no sense and we shouln't add PP stats to the drivers private ones?

Quoted 6 lines
Quoted 1 line

Yes, 100%.

Meh, this way the stats won't survive ifdown/ifup cycles as usually page_pools get destroyed on ifdown :z
In that patch, I backup the PP stats to a device-lifetime container when the pool gets destroyed, maybe we could do something similar?

I still can pull the PP stats to the driver before destroying it, but there's no way to tell the PP I have some archived stats for it. Maybe we could have page_pool_params_slow::get_stats() or smth like this?

Quoted 6 lines
Quoted 6 lines

FWIW I am aware that better tooling would be good so non-developers could access to the PP Netlink :( I'm thinking we should clean up YNL lib packaging a little and try to convince iproute2 maintainers to accept simple CLI built on top of it.

Thanks,
Olek

maintainer
#35 lore raw

Re: [PATCH net-next v5 13/14] libie: add per-queue Page Pool stats

Quoted 8 lines · On Thu, 30 Nov 2023 17:45:10 +0100 Alexander Lobakin wrote:
Quoted 4 lines

Meh, this way the stats won't survive ifdown/ifup cycles as usually page_pools get destroyed on ifdown :z
In that patch, I backup the PP stats to a device-lifetime container when the pool gets destroyed, maybe we could do something similar?

I still can pull the PP stats to the driver before destroying it, but there's no way to tell the PP I have some archived stats for it. Maybe we could have page_pool_params_slow::get_stats() or smth like this?

Why do you think the historic values matter?
User space monitoring will care about incremental values. It's not like for page pool we need to match the Rx packet count.

author maintainer
#36 lore raw

PATCH v5 14/14 iavf: switch queue stats to libie

In reply to Alexander Lobakin

iavf is pretty much ready for using the generic libie stats, so drop all the custom code and just use generic definitions. The only thing is that it previously lacked the counter of Tx queue stops. It's present in the other drivers, so add it here as well.
The rest is straightforward. Note that it makes the ring structure 1 CL bigger, but no layout changes since the stats structures are 32-byte aligned.

---
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 100 +++++-------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |   2 +
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  86 ++++++++-------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  28 +----
 4 files changed, 80 insertions(+), 136 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +28 -72
index 5fd41f0284a6..d52ad3bc5829 100644
@@ -2,9 +2,10 @@
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
 /* ethtool support for iavf */
-#include "iavf.h"
 
-#include <linux/uaccess.h>
+#include <linux/net/intel/libie/rx.h>
+
+#include "iavf.h"
 
 /* ethtool statistics helpers */
 
@@ -46,16 +47,6 @@ struct iavf_stats {
 	.stat_offset = offsetof(_type, _stat) \
 }
 
-/* Helper macro for defining some statistics related to queues */
-#define IAVF_QUEUE_STAT(_name, _stat) \
-	IAVF_STAT(struct iavf_ring, _name, _stat)
-
-/* Stats associated with a Tx or Rx ring */
-static const struct iavf_stats iavf_gstrings_queue_stats[] = {
-	IAVF_QUEUE_STAT("%s-%u.packets", stats.packets),
-	IAVF_QUEUE_STAT("%s-%u.bytes", stats.bytes),
-};
-
 /**
  * iavf_add_one_ethtool_stat - copy the stat into the supplied buffer
  * @data: location to store the stat value
@@ -141,43 +132,6 @@ __iavf_add_ethtool_stats(u64 **data, void *pointer,
 #define iavf_add_ethtool_stats(data, pointer, stats) \
 	__iavf_add_ethtool_stats(data, pointer, stats, ARRAY_SIZE(stats))
 
-/**
- * iavf_add_queue_stats - copy queue statistics into supplied buffer
- * @data: ethtool stats buffer
- * @ring: the ring to copy
- *
- * Queue statistics must be copied while protected by
- * u64_stats_fetch_begin, so we can't directly use iavf_add_ethtool_stats.
- * Assumes that queue stats are defined in iavf_gstrings_queue_stats. If the
- * ring pointer is null, zero out the queue stat values and update the data
- * pointer. Otherwise safely copy the stats from the ring into the supplied
- * buffer and update the data pointer when finished.
- *
- * This function expects to be called while under rcu_read_lock().
- **/
-static void
-iavf_add_queue_stats(u64 **data, struct iavf_ring *ring)
-{
-	const unsigned int size = ARRAY_SIZE(iavf_gstrings_queue_stats);
-	const struct iavf_stats *stats = iavf_gstrings_queue_stats;
-	unsigned int start;
-	unsigned int i;
-
-	/* To avoid invalid statistics values, ensure that we keep retrying
-	 * the copy until we get a consistent value according to
-	 * u64_stats_fetch_retry. But first, make sure our ring is
-	 * non-null before attempting to access its syncp.
-	 */
-	do {
-		start = !ring ? 0 : u64_stats_fetch_begin(&ring->syncp);
-		for (i = 0; i < size; i++)
-			iavf_add_one_ethtool_stat(&(*data)[i], ring, &stats[i]);
-	} while (ring && u64_stats_fetch_retry(&ring->syncp, start));
-
-	/* Once we successfully copy the stats in, update the data pointer */
-	*data += size;
-}
-
 /**
  * __iavf_add_stat_strings - copy stat strings into ethtool buffer
  * @p: ethtool supplied buffer
@@ -237,8 +191,6 @@ static const struct iavf_stats iavf_gstrings_stats[] = {
 
 #define IAVF_STATS_LEN	ARRAY_SIZE(iavf_gstrings_stats)
 
-#define IAVF_QUEUE_STATS_LEN	ARRAY_SIZE(iavf_gstrings_queue_stats)
-
 /**
  * iavf_get_link_ksettings - Get Link Speed and Duplex settings
  * @netdev: network interface device structure
@@ -308,18 +260,22 @@ static int iavf_get_link_ksettings(struct net_device *netdev,
  **/
 static int iavf_get_sset_count(struct net_device *netdev, int sset)
 {
-	/* Report the maximum number queues, even if not every queue is
-	 * currently configured. Since allocation of queues is in pairs,
-	 * use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set
-	 * at device creation and never changes.
-	 */
+	u32 num;
 
-	if (sset == ETH_SS_STATS)
-		return IAVF_STATS_LEN +
-			(IAVF_QUEUE_STATS_LEN * 2 *
-			 netdev->real_num_tx_queues);
-	else
+	switch (sset) {
+	case ETH_SS_STATS:
+		/* Per-queue */
+		num = libie_rq_stats_get_sset_count();
+		num += libie_sq_stats_get_sset_count();
+		num *= netdev->real_num_tx_queues;
+
+		/* Global */
+		num += IAVF_STATS_LEN;
+
+		return num;
+	default:
 		return -EINVAL;
+	}
 }
 
 /**
@@ -346,15 +302,17 @@ static void iavf_get_ethtool_stats(struct net_device *netdev,
 	 * it to iterate over rings' stats.
 	 */
 	for (i = 0; i < adapter->num_active_queues; i++) {
-		struct iavf_ring *ring;
-
-		/* Tx rings stats */
-		ring = &adapter->tx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		struct iavf_ring *ring = &adapter->rx_rings[i];
+		const struct libie_rx_queue rq = {
+			.pp	= ring->pp,
+			.stats	= &ring->rq_stats,
+		};
 
 		/* Rx rings stats */
-		ring = &adapter->rx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		libie_rq_stats_get_data(&data, &rq);
+
+		/* Tx rings stats */
+		libie_sq_stats_get_data(&data, &adapter->tx_rings[i].sq_stats);
 	}
 	rcu_read_unlock();
 }
@@ -376,10 +334,8 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data)
 	 * real_num_tx_queues for both Tx and Rx queues.
 	 */
 	for (i = 0; i < netdev->real_num_tx_queues; i++) {
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "tx", i);
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "rx", i);
+		libie_rq_stats_get_strings(&data, i);
+		libie_sq_stats_get_strings(&data, i);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_main.c +2 -0
index 7aa3b072a416..ff65a937c766 100644
@@ -1606,12 +1606,14 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		tx_ring->itr_setting = IAVF_ITR_TX_DEF;
 		if (adapter->flags & IAVF_FLAG_WB_ON_ITR_CAPABLE)
 			tx_ring->flags |= IAVF_TXR_FLAGS_WB_ON_ITR;
+		u64_stats_init(&tx_ring->sq_stats.syncp);
 
 		rx_ring = &adapter->rx_rings[i];
 		rx_ring->queue_index = i;
 		rx_ring->netdev = adapter->netdev;
 		rx_ring->count = adapter->rx_desc_count;
 		rx_ring->itr_setting = IAVF_ITR_RX_DEF;
+		u64_stats_init(&rx_ring->rq_stats.syncp);
 	}
 
 	adapter->num_active_queues = num_active_queues;
drivers/net/ethernet/intel/iavf/iavf_txrx.c +46 -40

245 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +4 -24
index 72c6bc64d94e..016c8b6e6160 100644
@@ -4,6 +4,8 @@
 #ifndef _IAVF_TXRX_H_
 #define _IAVF_TXRX_H_
 
+#include <linux/net/intel/libie/stats.h>
+
 /* Interrupt Throttling and Rate Limiting Goodies */
 #define IAVF_DEFAULT_IRQ_WORK      256
 
@@ -201,26 +203,6 @@ struct iavf_tx_buffer {
 	u32 tx_flags;
 };
 
-struct iavf_queue_stats {
-	u64 packets;
-	u64 bytes;
-};
-
-struct iavf_tx_queue_stats {
-	u64 restart_queue;
-	u64 tx_busy;
-	u64 tx_done_old;
-	u64 tx_linearize;
-	u64 tx_force_wb;
-	u64 tx_lost_interrupt;
-};
-
-struct iavf_rx_queue_stats {
-	u64 non_eop_descs;
-	u64 alloc_page_failed;
-	u64 alloc_buff_failed;
-};
-
 /* some useful defines for virtchannel interface, which
  * is the only remaining user of header split
  */
@@ -272,11 +254,9 @@ struct iavf_ring {
 #define IAVF_RXR_FLAGS_VLAN_TAG_LOC_L2TAG2_2	BIT(5)
 
 	/* stats structs */
-	struct iavf_queue_stats	stats;
-	struct u64_stats_sync syncp;
 	union {
-		struct iavf_tx_queue_stats tx_stats;
-		struct iavf_rx_queue_stats rx_stats;
+		struct libie_rq_stats rq_stats;
+		struct libie_sq_stats sq_stats;
 	};
 
 	int prev_pkt_ctr;		/* For Tx stall detection */
Signature · 1 line
2.42.0
#37 lore raw
In reply to Alexander Lobakin
Quoted 17 lines · Fri, Nov 24, 2023 at 04:47:18PM CET, [email protected] wrote:

Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details are in the commit messages; here's a summary:

Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. The first name that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" (-> one word, no "lib I E" pls) and can be expanded as "lib Internet Explorer" :P
The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks:
"can I share this?". The final destination is very ambitious: have only

Quoted 8 lines

one unified driver for at least i40e, ice, iavf, and idpf with a struct ops for each generation. That's never gonna happen, right? But you still can at least try.
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so that a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.

The world would not be the same without intel driver duplicates :/

Out of curiosity, what changed? I always thought this is done for sake of easier out of tree driver development and old device support dropping.

maintainer
#38 lore raw
Quoted 27 lines · Jiri Pirko
Quoted 17 lines · Fri, Nov 24, 2023 at 04:47:18PM CET, [email protected] wrote:

Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details are in the commit messages; here's a summary:

Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. The first name that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" (-> one word, no "lib I E" pls) and can be expanded as "lib Internet Explorer" :P
The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks:
"can I share this?". The final destination is very ambitious: have only

Quoted 8 lines

one unified driver for at least i40e, ice, iavf, and idpf with a struct ops for each generation. That's never gonna happen, right? But you still can at least try.
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so that a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.

Quoted 1 line

The world would not be the same without intel driver duplicates :/

:D

Quoted 5 lines

Out of curiosity, what changed? I always thought this is done for sake of easier out of tree driver development and old device support dropping.

Some of Intel folks, with Olek as one of leading examples, really care about our upstream drivers.

Recently (on HW-company timescale) Intel recognizes high costs of current duplicated-SW model, and I believe we will follow more modern-SW model where you apply such novel methodologies as code reuse via lib. (Disclaimer: it's not an official statement).

OTOH our OOT drivers would benefit from libie too, but that was not a driving force.