Skip to contentSkip to conversation

net: intel: start The Great Code Dedup + Page Pool for iavf

RFC PATCH Superseded Alexander Lobakin 5 participants 33 messages Jul 5, 2023 – Jul 18, 2023

Subsystems (4)

v4 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Alexander Lobakin

author maintainer
#0 lore raw

v4 cover net: intel: start The Great Code Dedup + Page Pool for iavf

Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details in the commit messages; here's summary:

Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers.
The first thing that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" and can be expanded as "lib Internet Explorer" :P I'm open for anything else (but justified). The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks: "can I share this?".
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.

Alexander Lobakin (9):
net: intel: introduce Intel Ethernet common library
iavf: kill "legacy-rx" for good
iavf: drop page splitting and recycling
net: page_pool: add DMA-sync-for-CPU inline helpers
libie: add Rx buffer management (via Page Pool)
iavf: switch to Page Pool
libie: add common queue stats
libie: add per-queue Page Pool stats
iavf: switch queue stats to libie

 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/intel/Kconfig            |  11 +
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 --------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  74 +--
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 ---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 --------
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 226 +------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  44 +-
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 608 ++++--------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 176 +----
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  90 ---
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  20 +-
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ---------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +--
 drivers/net/ethernet/intel/libie/Makefile     |   7 +
 drivers/net/ethernet/intel/libie/internal.h   |  23 +
 drivers/net/ethernet/intel/libie/rx.c         | 183 ++++++
 drivers/net/ethernet/intel/libie/stats.c      | 190 ++++++
 include/linux/net/intel/libie/rx.h            | 241 +++++++
 include/linux/net/intel/libie/stats.h         | 214 ++++++
 include/net/page_pool.h                       |  49 +-
 27 files changed, 1137 insertions(+), 2025 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/rx.h
 create mode 100644 include/linux/net/intel/libie/stats.h

---
Directly to net-next, has non-Intel code change (#4) :p

Based on the PP "hybrid" allocation series[0] and requires it to work.

From v3[1]:
* base on the latest net-next, update bloat-o-meter and perf stats;
* split generic PP optimizations into a separate series;
* drop "optimize hotpath a bunch" commit: a lot of [controversial] changes in one place, worth own series (Alex);
* 02: pick Rev-by (Alex);
* 03: move in-place recycling removal here from the dropped patch;
* 05: new, add libie Rx buffer API separatelly from IAVF changes;
* 05-06: use new "hybrid" allocation API from[0] to reduce memory usage when a page can fit more than 1 truesize (also asked by David);
* 06: merge with "always use order-0 page" commit to reduce diffs and simplify things (Alex);
* 09: fix page_alloc_fail counter.

From v2[2]:
* 0006: fix page_pool.h include in OcteonTX2 files (Jakub, Patchwork);
* no functional changes.

From v1[3]:
* 0006: new (me, Jakub);
* 0008: give the helpers more intuitive names (Jakub, Ilias);
* -^-: also expand their kdoc a bit for the same reason;
* -^-: fix kdoc copy-paste issue (Patchwork, Jakub);
* 0011: drop inline from C file (Patchwork, Jakub).

[0] https://lore.kernel.org/netdev/[email protected]
[1] https://lore.kernel.org/netdev/[email protected]
[2] https://lore.kernel.org/netdev/[email protected]
[3] https://lore.kernel.org/netdev/[email protected]

Signature · 1 line
2.41.0
author maintainer
#1 lore raw

RFC PATCH v4 1/9 net: intel: introduce Intel Ethernet common library

Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, add unlikely() condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as a pkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. But skb_set_hash() only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hidden Kconfig symbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.

---
 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/intel/Kconfig            |  10 +
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 --------------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  74 +---
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 -----
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 --------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |   1 +
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  70 +---
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  88 -----
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ------------------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +---
 drivers/net/ethernet/intel/libie/Makefile     |   6 +
 drivers/net/ethernet/intel/libie/rx.c         | 110 ++++++
 include/linux/net/intel/libie/rx.h            | 128 +++++++
 19 files changed, 312 insertions(+), 1179 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 include/linux/net/intel/libie/rx.h
MAINTAINERS +2 -1
index acbe54087d1c..480fe286e3f9 100644
@@ -10388,7 +10388,8 @@ F:	Documentation/networking/device_drivers/ethernet/intel/
 F:	drivers/net/ethernet/intel/
 F:	drivers/net/ethernet/intel/*/
 F:	include/linux/avf/virtchnl.h
-F:	include/linux/net/intel/iidc.h
+F:	include/linux/net/intel/
+F:	include/linux/net/intel/*/
 
 INTEL ETHERNET PROTOCOL DRIVER FOR RDMA
 M:	Mustafa Ismail <[email protected]>
drivers/net/ethernet/intel/Kconfig +10 -0
index 9bc0a9519899..86ecedeac115 100644
@@ -84,6 +84,12 @@ config E1000E_HWTS
 	 devices. The cross-timestamp is available through the PTP clock
 	 driver precise cross-timestamp ioctl (PTP_SYS_OFFSET_PRECISE).
 
+config LIBIE
+	tristate
+	help
+	  libie (Intel Ethernet library) is a common library containing
+	  routines shared by several Intel Ethernet drivers.
+
 config IGB
 	tristate "Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support"
 	depends on PCI
@@ -225,6 +231,7 @@ config I40E
 	depends on PTP_1588_CLOCK_OPTIONAL
 	depends on PCI
 	select AUXILIARY_BUS
+	select LIBIE
 	help
 	  This driver supports Intel(R) Ethernet Controller XL710 Family of
 	  devices.  For more information on how to identify your adapter, go
@@ -252,6 +259,8 @@ config I40E_DCB
 # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF
 config IAVF
 	tristate
+	select LIBIE
+
 config I40EVF
 	tristate "Intel(R) Ethernet Adaptive Virtual Function support"
 	select IAVF
@@ -282,6 +291,7 @@ config ICE
 	depends on GNSS || GNSS = n
 	select AUXILIARY_BUS
 	select DIMLIB
+	select LIBIE
 	select NET_DEVLINK
 	select PLDMFW
 	help
drivers/net/ethernet/intel/Makefile +1 -0
index d80d04132073..ce622b4d825d 100644
@@ -15,3 +15,4 @@ obj-$(CONFIG_I40E) += i40e/
 obj-$(CONFIG_IAVF) += iavf/
 obj-$(CONFIG_FM10K) += fm10k/
 obj-$(CONFIG_ICE) += ice/
+obj-$(CONFIG_LIBIE) += libie/
drivers/net/ethernet/intel/i40e/i40e_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_main.c +1 -0
index 29ad1797adce..fda6ccc3fc3e 100644
@@ -97,6 +97,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Connection XL710 Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static struct workqueue_struct *i40e_wq;
drivers/net/ethernet/intel/i40e/i40e_prototype.h +0 -7
index fe845987d99a..5287d0ef32d5 100644
@@ -380,13 +380,6 @@ void i40e_set_pci_config_data(struct i40e_hw *hw, u16 link_status);
 
 int i40e_set_mac_type(struct i40e_hw *hw);
 
-extern struct i40e_rx_ptype_decoded i40e_ptype_lookup[];
-
-static inline struct i40e_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return i40e_ptype_lookup[ptype];
-}
-
 /**
  * i40e_virtchnl_link_speed - Convert AdminQ link_speed to virtchnl definition
  * @link_speed: the speed to convert
drivers/net/ethernet/intel/i40e/i40e_txrx.c +18 -56

132 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -0
index a483eb185c99..23fbd45dd986 100644
@@ -46,6 +46,7 @@ MODULE_DEVICE_TABLE(pci, iavf_pci_tbl);
 MODULE_ALIAS("i40evf");
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Adaptive Virtual Function Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static const struct net_device_ops iavf_netdev_ops;
drivers/net/ethernet/intel/iavf/iavf_prototype.h +0 -7
index 940cb4203fbe..c2028beccde7 100644
@@ -47,13 +47,6 @@ enum iavf_status iavf_aq_set_rss_key(struct iavf_hw *hw, u16 seid,
 
 enum iavf_status iavf_set_mac_type(struct iavf_hw *hw);
 
-extern struct iavf_rx_ptype_decoded iavf_ptype_lookup[];
-
-static inline struct iavf_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return iavf_ptype_lookup[ptype];
-}
-
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/net/ethernet/intel/iavf/iavf_txrx.c +16 -54

125 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h +0 -316

326 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_main.c +1 -0
index 93979ab18bc1..eb116c8dc6ad 100644
@@ -34,6 +34,7 @@ static const char ice_copyright[] = "Copyright (c) 2018, Intel Corporation.";
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION(DRV_SUMMARY);
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 MODULE_FIRMWARE(ICE_DDP_PKG_FILE);
 
drivers/net/ethernet/intel/ice/ice_txrx_lib.c +18 -56

137 lines not on this page · raw message

drivers/net/ethernet/intel/libie/Makefile new file +6 -0
new file mode 100644
index 000000000000..95e81d09b474
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright(c) 2023 Intel Corporation.
+
+obj-$(CONFIG_LIBIE)	+= libie.o
+
+libie-objs		+= rx.o
drivers/net/ethernet/intel/libie/rx.c new file +110 -0

110 lines not on this page · raw message

include/linux/net/intel/libie/rx.h new file +128 -0

128 lines not on this page · raw message

Signature · 1 line
2.41.0
maintainer Reviewed-by on v4 1/9
#2 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 1/9] net: intel: introduce Intel Ethernet common library

Quoted 45 lines · Alexander Lobakin

Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, add unlikely() condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as a pkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. But skb_set_hash() only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hidden Kconfig symbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.

---
  MAINTAINERS                                   |   3 +-
  drivers/net/ethernet/intel/Kconfig            |  10 +
  drivers/net/ethernet/intel/Makefile           |   1 +
  drivers/net/ethernet/intel/i40e/i40e_common.c | 253 --------------
  drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
  .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
  drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  74 +---
  drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 -----
  drivers/net/ethernet/intel/iavf/iavf_common.c | 253 --------------
  drivers/net/ethernet/intel/iavf/iavf_main.c   |   1 +
  .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
  drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  70 +---
  drivers/net/ethernet/intel/iavf/iavf_type.h   |  88 -----
  .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ------------------
  drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
  drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +---
  drivers/net/ethernet/intel/libie/Makefile     |   6 +
  drivers/net/ethernet/intel/libie/rx.c         | 110 ++++++
  include/linux/net/intel/libie/rx.h            | 128 +++++++
  19 files changed, 312 insertions(+), 1179 deletions(-)
Quoted 3 lines

create mode 100644 drivers/net/ethernet/intel/libie/Makefile create mode 100644 drivers/net/ethernet/intel/libie/rx.c create mode 100644 include/linux/net/intel/libie/rx.h

Thanks a lot for whole effort on this series, it's refreshing to see a move in that direction.

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#3 lore raw

RFC PATCH v4 2/9 iavf: kill "legacy-rx" for good

In reply to Alexander Lobakin

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.

* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.

Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.

There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft, and so on. Some stats:

Function: add/remove: 0/4 grow/shrink: 0/7 up/down: 0/-757 (-757) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)

---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 140 ------------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  10 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  84 +----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  27 +---
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   3 +-
 6 files changed, 8 insertions(+), 258 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
index f80f2735e688..71c5d9b18692 100644
@@ -298,7 +298,7 @@ struct iavf_adapter {
 #define IAVF_FLAG_CLIENT_NEEDS_L2_PARAMS	BIT(12)
 #define IAVF_FLAG_PROMISC_ON			BIT(13)
 #define IAVF_FLAG_ALLMULTI_ON			BIT(14)
-#define IAVF_FLAG_LEGACY_RX			BIT(15)
+/* BIT(15) is free, was IAVF_FLAG_LEGACY_RX */
 #define IAVF_FLAG_REINIT_ITR_NEEDED		BIT(16)
 #define IAVF_FLAG_QUEUES_DISABLED		BIT(17)
 #define IAVF_FLAG_SETUP_NETDEV_FEATURES		BIT(18)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +0 -140

181 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -9
index 23fbd45dd986..db1ed13f11bb 100644
@@ -702,9 +702,7 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 	struct iavf_hw *hw = &adapter->hw;
 	int i;
 
-	/* Legacy Rx will always default to a 2048 buffer size. */
-#if (PAGE_SIZE < 8192)
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX)) {
+	if (PAGE_SIZE < 8192) {
 		struct net_device *netdev = adapter->netdev;
 
 		/* For jumbo frames on systems with 4K pages we have to use
@@ -721,16 +719,10 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 		    (netdev->mtu <= ETH_DATA_LEN))
 			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 	}
-#endif
 
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
 		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-
-		if (adapter->flags & IAVF_FLAG_LEGACY_RX)
-			clear_ring_build_skb_enabled(&adapter->rx_rings[i]);
-		else
-			set_ring_build_skb_enabled(&adapter->rx_rings[i]);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -81

115 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -25
index 7e6ee32d19b6..4b412f7662e4 100644
@@ -82,20 +82,11 @@ enum iavf_dyn_idx_t {
 	BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP))
 
 /* Supported Rx Buffer Sizes (a multiple of 128) */
-#define IAVF_RXBUFFER_256   256
 #define IAVF_RXBUFFER_1536  1536  /* 128B aligned standard Ethernet frame */
 #define IAVF_RXBUFFER_2048  2048
 #define IAVF_RXBUFFER_3072  3072  /* Used for large frames w/ padding */
 #define IAVF_MAX_RXBUFFER   9728  /* largest size for single descriptor */
 
-/* NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we
- * reserve 2 more, and skb_shared_info adds an additional 384 bytes more,
- * this adds up to 512 bytes of extra data meaning the smallest allocation
- * we could have is 1K.
- * i.e. RXBUFFER_256 --> 960 byte skb (size-1024 slab)
- * i.e. RXBUFFER_512 --> 1216 byte skb (size-2048 slab)
- */
-#define IAVF_RX_HDR_SIZE IAVF_RXBUFFER_256
 #define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))
 #define iavf_rx_desc iavf_32byte_rx_desc
 
@@ -362,7 +353,8 @@ struct iavf_ring {
 
 	u16 flags;
 #define IAVF_TXR_FLAGS_WB_ON_ITR		BIT(0)
-#define IAVF_RXR_FLAGS_BUILD_SKB_ENABLED	BIT(1)
+/* BIT(1) is free, was IAVF_RXR_FLAGS_BUILD_SKB_ENABLED */
+/* BIT(2) is free */
 #define IAVF_TXRX_FLAGS_VLAN_TAG_LOC_L2TAG1	BIT(3)
 #define IAVF_TXR_FLAGS_VLAN_TAG_LOC_L2TAG2	BIT(4)
 #define IAVF_RXR_FLAGS_VLAN_TAG_LOC_L2TAG2_2	BIT(5)
@@ -393,21 +385,6 @@ struct iavf_ring {
 					 */
 } ____cacheline_internodealigned_in_smp;
 
-static inline bool ring_uses_build_skb(struct iavf_ring *ring)
-{
-	return !!(ring->flags & IAVF_RXR_FLAGS_BUILD_SKB_ENABLED);
-}
-
-static inline void set_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags |= IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
-static inline void clear_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags &= ~IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
 #define IAVF_ITR_ADAPTIVE_MIN_INC	0x0002
 #define IAVF_ITR_ADAPTIVE_MIN_USECS	0x0002
 #define IAVF_ITR_ADAPTIVE_MAX_USECS	0x007e
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -2
index 7c0578b5457b..fdddc3588487 100644
@@ -290,8 +290,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		return;
 
 	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX) &&
-	    (adapter->netdev->mtu <= ETH_DATA_LEN))
+	if (adapter->netdev->mtu <= ETH_DATA_LEN)
 		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
Signature · 1 line
2.41.0
maintainer Reviewed-by on v4 2/9
#4 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 2/9] iavf: kill "legacy-rx" for good

Quoted 35 lines · Alexander Lobakin

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.

* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.

Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.

There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft, and so on. Some stats:

Function: add/remove: 0/4 grow/shrink: 0/7 up/down: 0/-757 (-757) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)

Quoted 8 lines
---
  drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
  .../net/ethernet/intel/iavf/iavf_ethtool.c    | 140 ------------------
  drivers/net/ethernet/intel/iavf/iavf_main.c   |  10 +-
  drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  84 +----------
  drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  27 +---
  .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   3 +-
  6 files changed, 8 insertions(+), 258 deletions(-)

Good one, there were some random questions in my mind during the read, but all are resolved by subsequent patches.
(It's a pity I have not found yet time to fully read them though)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#5 lore raw

RFC PATCH v4 3/9 iavf: drop page splitting and recycling

In reply to Alexander Lobakin

As an intermediate step, remove all page splitting/recyclig code. Just always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63 &iavf_rx_buf structures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:

Function: add/remove: 0/2 grow/shrink: 1/3 up/down: 3/-494 (-491)

Although the array of &iavf_rx_buf is barely used now and could be replaced with just page pointer array, don't touch it now to not complicate replacing it with libie Rx buffer struct later on. No surprise perf loses up to 30% here, but that regression will go away once PP lands.

---
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 151 ++------------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   8 --
 2 files changed, 13 insertions(+), 146 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +13 -138
index a85b270fc769..789b10815d7f 100644
@@ -723,7 +723,7 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
 				     DMA_FROM_DEVICE,
 				     IAVF_RX_DMA_ATTR);
 
-		__page_frag_cache_drain(rx_bi->page, rx_bi->pagecnt_bias);
+		__free_pages(rx_bi->page, iavf_rx_pg_order(rx_ring));
 
 		rx_bi->page = NULL;
 		rx_bi->page_offset = 0;
@@ -735,7 +735,6 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
 	/* Zero out the descriptor ring */
 	memset(rx_ring->desc, 0, rx_ring->size);
 
-	rx_ring->next_to_alloc = 0;
 	rx_ring->next_to_clean = 0;
 	rx_ring->next_to_use = 0;
 }
@@ -791,7 +790,6 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 		goto err;
 	}
 
-	rx_ring->next_to_alloc = 0;
 	rx_ring->next_to_clean = 0;
 	rx_ring->next_to_use = 0;
 
@@ -811,9 +809,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)
 {
 	rx_ring->next_to_use = val;
 
-	/* update next to alloc since we have filled the ring */
-	rx_ring->next_to_alloc = val;
-
 	/* Force memory writes to complete before letting h/w
 	 * know there are new descriptors to fetch.  (Only
 	 * applicable for weak-ordered memory model archs,
@@ -837,12 +832,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
 	struct page *page = bi->page;
 	dma_addr_t dma;
 
-	/* since we are recycling buffers we should seldom need to alloc */
-	if (likely(page)) {
-		rx_ring->rx_stats.page_reuse_count++;
-		return true;
-	}
-
 	/* alloc new page for storage */
 	page = dev_alloc_pages(iavf_rx_pg_order(rx_ring));
 	if (unlikely(!page)) {
@@ -869,9 +858,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
 	bi->page = page;
 	bi->page_offset = IAVF_SKB_PAD;
 
-	/* initialize pagecnt_bias to 1 representing we fully own page */
-	bi->pagecnt_bias = 1;
-
 	return true;
 }
 
@@ -1103,91 +1089,6 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
 	return false;
 }
 
-/**
- * iavf_reuse_rx_page - page flip buffer and store it back on the ring
- * @rx_ring: rx descriptor ring to store buffers on
- * @old_buff: donor buffer to have page reused
- *
- * Synchronizes page for reuse by the adapter
- **/
-static void iavf_reuse_rx_page(struct iavf_ring *rx_ring,
-			       struct iavf_rx_buffer *old_buff)
-{
-	struct iavf_rx_buffer *new_buff;
-	u16 nta = rx_ring->next_to_alloc;
-
-	new_buff = &rx_ring->rx_bi[nta];
-
-	/* update, and store next to alloc */
-	nta++;
-	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
-
-	/* transfer page from old buffer to new buffer */
-	new_buff->dma		= old_buff->dma;
-	new_buff->page		= old_buff->page;
-	new_buff->page_offset	= old_buff->page_offset;
-	new_buff->pagecnt_bias	= old_buff->pagecnt_bias;
-}
-
-/**
- * iavf_can_reuse_rx_page - Determine if this page can be reused by
- * the adapter for another receive
- *
- * @rx_buffer: buffer containing the page
- *
- * If page is reusable, rx_buffer->page_offset is adjusted to point to
- * an unused region in the page.
- *
- * For small pages, @truesize will be a constant value, half the size
- * of the memory at page.  We'll attempt to alternate between high and
- * low halves of the page, with one half ready for use by the hardware
- * and the other half being consumed by the stack.  We use the page
- * ref count to determine whether the stack has finished consuming the
- * portion of this page that was passed up with a previous packet.  If
- * the page ref count is >1, we'll assume the "other" half page is
- * still busy, and this page cannot be reused.
- *
- * For larger pages, @truesize will be the actual space used by the
- * received packet (adjusted upward to an even multiple of the cache
- * line size).  This will advance through the page by the amount
- * actually consumed by the received packets while there is still
- * space for a buffer.  Each region of larger pages will be used at
- * most once, after which the page will not be reused.
- *
- * In either case, if the page is reusable its refcount is increased.
- **/
-static bool iavf_can_reuse_rx_page(struct iavf_rx_buffer *rx_buffer)
-{
-	unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
-	struct page *page = rx_buffer->page;
-
-	/* Is any reuse possible? */
-	if (!dev_page_is_reusable(page))
-		return false;
-
-#if (PAGE_SIZE < 8192)
-	/* if we are only owner of page we can reuse it */
-	if (unlikely((page_count(page) - pagecnt_bias) > 1))
-		return false;
-#else
-#define IAVF_LAST_OFFSET \
-	(SKB_WITH_OVERHEAD(PAGE_SIZE) - IAVF_RXBUFFER_2048)
-	if (rx_buffer->page_offset > IAVF_LAST_OFFSET)
-		return false;
-#endif
-
-	/* If we have drained the page fragment pool we need to update
-	 * the pagecnt_bias and page count so that we fully restock the
-	 * number of references the driver holds.
-	 */
-	if (unlikely(!pagecnt_bias)) {
-		page_ref_add(page, USHRT_MAX);
-		rx_buffer->pagecnt_bias = USHRT_MAX;
-	}
-
-	return true;
-}
-
 /**
  * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
  * @rx_ring: rx descriptor ring to transact packets on
@@ -1216,13 +1117,6 @@ static void iavf_add_rx_frag(struct iavf_ring *rx_ring,
 
 	skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
 			rx_buffer->page_offset, size, truesize);
-
-	/* page is being used so we must update the page offset */
-#if (PAGE_SIZE < 8192)
-	rx_buffer->page_offset ^= truesize;
-#else
-	rx_buffer->page_offset += truesize;
-#endif
 }
 
 /**
@@ -1250,9 +1144,6 @@ static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring,
 				      size,
 				      DMA_FROM_DEVICE);
 
-	/* We have pulled a buffer for use, so decrement pagecnt_bias */
-	rx_buffer->pagecnt_bias--;
-
 	return rx_buffer;
 }
 
@@ -1293,23 +1184,15 @@ static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
 	skb_reserve(skb, IAVF_SKB_PAD);
 	__skb_put(skb, size);
 
-	/* buffer is used by skb, update page_offset */
-#if (PAGE_SIZE < 8192)
-	rx_buffer->page_offset ^= truesize;
-#else
-	rx_buffer->page_offset += truesize;
-#endif
-
 	return skb;
 }
 
 /**
- * iavf_put_rx_buffer - Clean up used buffer and either recycle or free
+ * iavf_put_rx_buffer - Unmap used buffer
  * @rx_ring: rx descriptor ring to transact packets on
  * @rx_buffer: rx buffer to pull data from
  *
- * This function will clean up the contents of the rx_buffer.  It will
- * either recycle the buffer or unmap it and free the associated resources.
+ * This function will unmap the buffer after it's written by HW.
  */
 static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
 			       struct iavf_rx_buffer *rx_buffer)
@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
 	if (!rx_buffer)
 		return;
 
-	if (iavf_can_reuse_rx_page(rx_buffer)) {
-		/* hand second half of page back to the ring */
-		iavf_reuse_rx_page(rx_ring, rx_buffer);
-		rx_ring->rx_stats.page_reuse_count++;
-	} else {
-		/* we are not reusing the buffer so unmap it */
-		dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
-				     iavf_rx_pg_size(rx_ring),
-				     DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
-		__page_frag_cache_drain(rx_buffer->page,
-					rx_buffer->pagecnt_bias);
-	}
-
-	/* clear contents of buffer_info */
-	rx_buffer->page = NULL;
+	/* we are not reusing the buffer so unmap it */
+	dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+			     iavf_rx_pg_size(rx_ring),
+			     DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
 }
 
 /**
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 		else
 			skb = iavf_build_skb(rx_ring, rx_buffer, size);
 
+		iavf_put_rx_buffer(rx_ring, rx_buffer);
+
 		/* exit if we failed to retrieve a buffer */
 		if (!skb) {
 			rx_ring->rx_stats.alloc_buff_failed++;
-			if (rx_buffer && size)
-				rx_buffer->pagecnt_bias++;
+			__free_pages(rx_buffer->page,
+				     iavf_rx_pg_order(rx_ring));
+			rx_buffer->page = NULL;
 			break;
 		}
 
-		iavf_put_rx_buffer(rx_ring, rx_buffer);
+		rx_buffer->page = NULL;
 		cleaned_count++;
 
 		if (iavf_is_non_eop(rx_ring, rx_desc, skb))
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -8
index 4b412f7662e4..2170a77f8c8d 100644
@@ -266,12 +266,7 @@ struct iavf_tx_buffer {
 struct iavf_rx_buffer {
 	dma_addr_t dma;
 	struct page *page;
-#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
 	__u32 page_offset;
-#else
-	__u16 page_offset;
-#endif
-	__u16 pagecnt_bias;
 };
 
 struct iavf_queue_stats {
@@ -293,8 +288,6 @@ struct iavf_rx_queue_stats {
 	u64 non_eop_descs;
 	u64 alloc_page_failed;
 	u64 alloc_buff_failed;
-	u64 page_reuse_count;
-	u64 realloc_count;
 };
 
 enum iavf_ring_state_t {
@@ -374,7 +367,6 @@ struct iavf_ring {
 	struct iavf_q_vector *q_vector;	/* Backreference to associated vector */
 
 	struct rcu_head rcu;		/* to avoid race on free */
-	u16 next_to_alloc;
 	struct sk_buff *skb;		/* When iavf_clean_rx_ring_irq() must
 					 * return before it sees the EOP for
 					 * the current packet, we save that skb
Signature · 1 line
2.41.0
#6 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling

Quoted 2 lines · Alexander Lobakin

As an intermediate step, remove all page splitting/recyclig code. Just

Spelling issue: "recycling"

Quoted 239 lines

always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63 &iavf_rx_buf structures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:

Function: add/remove: 0/2 grow/shrink: 1/3 up/down: 3/-494 (-491)

Although the array of &iavf_rx_buf is barely used now and could be replaced with just page pointer array, don't touch it now to not complicate replacing it with libie Rx buffer struct later on. No surprise perf loses up to 30% here, but that regression will go away once PP lands.

---
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 151 ++------------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   8 --
 2 files changed, 13 insertions(+), 146 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -129
index a85b270fc769..789b10815d7f 100644
@@ -723,7 +723,7 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
                                     DMA_FROM_DEVICE,
                                     IAVF_RX_DMA_ATTR);
 
-               __page_frag_cache_drain(rx_bi->page, rx_bi->pagecnt_bias);
+               __free_pages(rx_bi->page, iavf_rx_pg_order(rx_ring));
 
                rx_bi->page = NULL;
                rx_bi->page_offset = 0;
@@ -735,7 +735,6 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
        /* Zero out the descriptor ring */
        memset(rx_ring->desc, 0, rx_ring->size);
 
-       rx_ring->next_to_alloc = 0;
        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;
 }
@@ -791,7 +790,6 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
                goto err;
        }
 
-       rx_ring->next_to_alloc = 0;
        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;
 
@@ -811,9 +809,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)
 {
        rx_ring->next_to_use = val;
 
-       /* update next to alloc since we have filled the ring */
-       rx_ring->next_to_alloc = val;
-
        /* Force memory writes to complete before letting h/w
         * know there are new descriptors to fetch.  (Only
         * applicable for weak-ordered memory model archs,
@@ -837,12 +832,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
        struct page *page = bi->page;
        dma_addr_t dma;
 
-       /* since we are recycling buffers we should seldom need to alloc */
-       if (likely(page)) {
-               rx_ring->rx_stats.page_reuse_count++;
-               return true;
-       }
-
        /* alloc new page for storage */
        page = dev_alloc_pages(iavf_rx_pg_order(rx_ring));
        if (unlikely(!page)) {
@@ -869,9 +858,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
        bi->page = page;
        bi->page_offset = IAVF_SKB_PAD;
 
-       /* initialize pagecnt_bias to 1 representing we fully own page */
-       bi->pagecnt_bias = 1;
-
        return true;
 }
 
@@ -1103,91 +1089,6 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
        return false;
 }
 
-/**
- * iavf_reuse_rx_page - page flip buffer and store it back on the ring
- * @rx_ring: rx descriptor ring to store buffers on
- * @old_buff: donor buffer to have page reused
- *
- * Synchronizes page for reuse by the adapter
- **/
-static void iavf_reuse_rx_page(struct iavf_ring *rx_ring,
-                              struct iavf_rx_buffer *old_buff)
-{
-       struct iavf_rx_buffer *new_buff;
-       u16 nta = rx_ring->next_to_alloc;
-
-       new_buff = &rx_ring->rx_bi[nta];
-
-       /* update, and store next to alloc */
-       nta++;
-       rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
-
-       /* transfer page from old buffer to new buffer */
-       new_buff->dma           = old_buff->dma;
-       new_buff->page          = old_buff->page;
-       new_buff->page_offset   = old_buff->page_offset;
-       new_buff->pagecnt_bias  = old_buff->pagecnt_bias;
-}
-
-/**
- * iavf_can_reuse_rx_page - Determine if this page can be reused by
- * the adapter for another receive
- *
- * @rx_buffer: buffer containing the page
- *
- * If page is reusable, rx_buffer->page_offset is adjusted to point to
- * an unused region in the page.
- *
- * For small pages, @truesize will be a constant value, half the size
- * of the memory at page.  We'll attempt to alternate between high and
- * low halves of the page, with one half ready for use by the hardware
- * and the other half being consumed by the stack.  We use the page
- * ref count to determine whether the stack has finished consuming the
- * portion of this page that was passed up with a previous packet.  If
- * the page ref count is >1, we'll assume the "other" half page is
- * still busy, and this page cannot be reused.
- *
- * For larger pages, @truesize will be the actual space used by the
- * received packet (adjusted upward to an even multiple of the cache
- * line size).  This will advance through the page by the amount
- * actually consumed by the received packets while there is still
- * space for a buffer.  Each region of larger pages will be used at
- * most once, after which the page will not be reused.
- *
- * In either case, if the page is reusable its refcount is increased.
- **/
-static bool iavf_can_reuse_rx_page(struct iavf_rx_buffer *rx_buffer)
-{
-       unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
-       struct page *page = rx_buffer->page;
-
-       /* Is any reuse possible? */
-       if (!dev_page_is_reusable(page))
-               return false;
-
-#if (PAGE_SIZE < 8192)
-       /* if we are only owner of page we can reuse it */
-       if (unlikely((page_count(page) - pagecnt_bias) > 1))
-               return false;
-#else
-#define IAVF_LAST_OFFSET \
-       (SKB_WITH_OVERHEAD(PAGE_SIZE) - IAVF_RXBUFFER_2048)
-       if (rx_buffer->page_offset > IAVF_LAST_OFFSET)
-               return false;
-#endif
-
-       /* If we have drained the page fragment pool we need to update
-        * the pagecnt_bias and page count so that we fully restock the
-        * number of references the driver holds.
-        */
-       if (unlikely(!pagecnt_bias)) {
-               page_ref_add(page, USHRT_MAX);
-               rx_buffer->pagecnt_bias = USHRT_MAX;
-       }
-
-       return true;
-}
-
 /**
  * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
  * @rx_ring: rx descriptor ring to transact packets on
@@ -1216,13 +1117,6 @@ static void iavf_add_rx_frag(struct iavf_ring *rx_ring,
 
        skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
                        rx_buffer->page_offset, size, truesize);
-
-       /* page is being used so we must update the page offset */
-#if (PAGE_SIZE < 8192)
-       rx_buffer->page_offset ^= truesize;
-#else
-       rx_buffer->page_offset += truesize;
-#endif
 }
 
 /**
@@ -1250,9 +1144,6 @@ static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring,
                                      size,
                                      DMA_FROM_DEVICE);
 
-       /* We have pulled a buffer for use, so decrement pagecnt_bias */
-       rx_buffer->pagecnt_bias--;
-
        return rx_buffer;
 }
 
@@ -1293,23 +1184,15 @@ static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
        skb_reserve(skb, IAVF_SKB_PAD);
        __skb_put(skb, size);
 
-       /* buffer is used by skb, update page_offset */
-#if (PAGE_SIZE < 8192)
-       rx_buffer->page_offset ^= truesize;
-#else
-       rx_buffer->page_offset += truesize;
-#endif
-
        return skb;
 }
 
 /**
- * iavf_put_rx_buffer - Clean up used buffer and either recycle or free
+ * iavf_put_rx_buffer - Unmap used buffer
  * @rx_ring: rx descriptor ring to transact packets on
  * @rx_buffer: rx buffer to pull data from
  *
- * This function will clean up the contents of the rx_buffer.  It will
- * either recycle the buffer or unmap it and free the associated resources.
+ * This function will unmap the buffer after it's written by HW.
  */
 static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
                               struct iavf_rx_buffer *rx_buffer)
@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
        if (!rx_buffer)
                return;
 
-       if (iavf_can_reuse_rx_page(rx_buffer)) {
-               /* hand second half of page back to the ring */
-               iavf_reuse_rx_page(rx_ring, rx_buffer);
-               rx_ring->rx_stats.page_reuse_count++;
-       } else {
-               /* we are not reusing the buffer so unmap it */
-               dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
-                                    iavf_rx_pg_size(rx_ring),
-                                    DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_put_rx_buffer(struct iavf_ring *rx_ring, · line 1206

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -6
-               __page_frag_cache_drain(rx_buffer->page,
-                                       rx_buffer->pagecnt_bias);
-       }
-
-       /* clear contents of buffer_info */
-       rx_buffer->page = NULL;
+       /* we are not reusing the buffer so unmap it */
+       dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+                            iavf_rx_pg_size(rx_ring),
+                            DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);

Rather than reorder all this I would just do the dma_unmap_page_attrs and then leave the assignment of NULL to rx_buffer->page. It should make this a bit easier to clean up the code below.

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307

Quoted 9 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -0
 }
 
 /**
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
                else
                        skb = iavf_build_skb(rx_ring, rx_buffer, size);
 
+               iavf_put_rx_buffer(rx_ring, rx_buffer);
+

This should stay below where it was.

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1315

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
                /* exit if we failed to retrieve a buffer */
                if (!skb) {
                        rx_ring->rx_stats.alloc_buff_failed++;
-                       if (rx_buffer && size)
-                               rx_buffer->pagecnt_bias++;
+                       __free_pages(rx_buffer->page,
+                                    iavf_rx_pg_order(rx_ring));
+                       rx_buffer->page = NULL;
                        break;
                }

This code was undoing the iavf_get_rx_buffer decrement of pagecnt_bias and then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment of pagecnt_bias.

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -1
-               iavf_put_rx_buffer(rx_ring, rx_buffer);
+               rx_buffer->page = NULL;
                cleaned_count++;
 
                if (iavf_is_non_eop(rx_ring, rx_desc, skb))

If iavf_put_rx_buffer just does the unmap and assignment of NULL then it could just be left here as is.

Quoted 37 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -9
index 4b412f7662e4..2170a77f8c8d 100644
@@ -266,12 +266,7 @@ struct iavf_tx_buffer {
 struct iavf_rx_buffer {
        dma_addr_t dma;
        struct page *page;
-#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
        __u32 page_offset;
-#else
-       __u16 page_offset;
-#endif
-       __u16 pagecnt_bias;
 };
 
 struct iavf_queue_stats {
@@ -293,8 +288,6 @@ struct iavf_rx_queue_stats {
        u64 non_eop_descs;
        u64 alloc_page_failed;
        u64 alloc_buff_failed;
-       u64 page_reuse_count;
-       u64 realloc_count;
 };
 
 enum iavf_ring_state_t {
@@ -374,7 +367,6 @@ struct iavf_ring {
        struct iavf_q_vector *q_vector; /* Backreference to associated vector */
 
        struct rcu_head rcu;            /* to avoid race on free */
-       u16 next_to_alloc;
        struct sk_buff *skb;            /* When iavf_clean_rx_ring_irq() must
                                         * return before it sees the EOP for
                                         * the current packet, we save that skb
--

2.41.0

Quoted 4 lines

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#7 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700

Quoted 6 lines
Quoted 2 lines · Alexander Lobakin

As an intermediate step, remove all page splitting/recyclig code. Just

Spelling issue: "recycling"

checkpatch w/codespell didn't catch this one =\

Quoted 8 lines
Quoted 8 lines

always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63 &iavf_rx_buf structures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:

[...]

Quoted 35 lines
Quoted 13 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -9
@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
        if (!rx_buffer)
                return;
 
-       if (iavf_can_reuse_rx_page(rx_buffer)) {
-               /* hand second half of page back to the ring */
-               iavf_reuse_rx_page(rx_ring, rx_buffer);
-               rx_ring->rx_stats.page_reuse_count++;
-       } else {
-               /* we are not reusing the buffer so unmap it */
-               dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
-                                    iavf_rx_pg_size(rx_ring),
-                                    DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_put_rx_buffer(struct iavf_ring *rx_ring, · line 1206

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -6
-               __page_frag_cache_drain(rx_buffer->page,
-                                       rx_buffer->pagecnt_bias);
-       }
-
-       /* clear contents of buffer_info */
-       rx_buffer->page = NULL;
+       /* we are not reusing the buffer so unmap it */
+       dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+                            iavf_rx_pg_size(rx_ring),
+                            DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);

Rather than reorder all this I would just do the dma_unmap_page_attrs and then leave the assignment of NULL to rx_buffer->page. It should make this a bit easier to clean up the code below.

Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -0
 }
 
 /**
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
                else
                        skb = iavf_build_skb(rx_ring, rx_buffer, size);
 
Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307

Quoted 2 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -0
+               iavf_put_rx_buffer(rx_ring, rx_buffer);
+

This should stay below where it was.

Wait-wait-wait.

if (!skb) break breaks the loop. put_rx_buffer() unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.

Quoted 18 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1315

Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
                /* exit if we failed to retrieve a buffer */
                if (!skb) {
                        rx_ring->rx_stats.alloc_buff_failed++;
-                       if (rx_buffer && size)
-                               rx_buffer->pagecnt_bias++;
+                       __free_pages(rx_buffer->page,
+                                    iavf_rx_pg_order(rx_ring));
+                       rx_buffer->page = NULL;
                        break;
                }

This code was undoing the iavf_get_rx_buffer decrement of pagecnt_bias and then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment of pagecnt_bias.

Quoted 1 line
Quoted 8 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320

Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -1
-               iavf_put_rx_buffer(rx_ring, rx_buffer);
+               rx_buffer->page = NULL;
                cleaned_count++;
 
                if (iavf_is_non_eop(rx_ring, rx_desc, skb))

If iavf_put_rx_buffer just does the unmap and assignment of NULL then it could just be left here as is.

I guess those two are tied with the one above.

[...]

Thanks,
Olek

#8 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling

Quoted 6 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700

Quoted 2 lines

On Wed, Jul 5, 2023 at 8:57 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 8 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -0
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
                else
                        skb = iavf_build_skb(rx_ring, rx_buffer, size);
 
+               iavf_put_rx_buffer(rx_ring, rx_buffer);
+

This should stay below where it was.

Quoted 6 lines

Wait-wait-wait.

if (!skb) break breaks the loop. put_rx_buffer() unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.

Specifically you don't want to be unmapping and freeing this page until after the !skb check. The problem is if skb is NULL the skb allocation failed and so processing of Rx is meant to stop in place without removing the page. It is where we will resume on the next pass assuming memory has been freed that can then be used. The problem is the skb allocation, not the page. We used to do the skb allocation before we would acquire the buffer, but with XDP there are cases where we aren't supposed to allocate it so it got moved to after which causes this confusion.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
                /* exit if we failed to retrieve a buffer */
                if (!skb) {
                        rx_ring->rx_stats.alloc_buff_failed++;
-                       if (rx_buffer && size)
-                               rx_buffer->pagecnt_bias++;
+                       __free_pages(rx_buffer->page,
+                                    iavf_rx_pg_order(rx_ring));
+                       rx_buffer->page = NULL;
                        break;
                }
Quoted 4 lines
Quoted 4 lines

This code was undoing the iavf_get_rx_buffer decrement of pagecnt_bias and then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment of pagecnt_bias.

The key bit here is the allocation failure is the reason why we halted processing. So the page contains valid data and should not be freed. We just need to leave it in place and wait for an allocation to succeed and then we can resume processing.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -1
-               iavf_put_rx_buffer(rx_ring, rx_buffer);
+               rx_buffer->page = NULL;
                cleaned_count++;
 
Quoted 6 lines
Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320

Quoted 1 line
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -0
                if (iavf_is_non_eop(rx_ring, rx_desc, skb))

If iavf_put_rx_buffer just does the unmap and assignment of NULL then it could just be left here as is.

I guess those two are tied with the one above.

Yeah, the iavf_put_rx_buffer should be left down here.

author maintainer
#9 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 10:06:29 -0700

Quoted 54 lines
Quoted 6 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700

Quoted 2 lines

On Wed, Jul 5, 2023 at 8:57 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 8 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -0
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
                else
                        skb = iavf_build_skb(rx_ring, rx_buffer, size);
 
+               iavf_put_rx_buffer(rx_ring, rx_buffer);
+

This should stay below where it was.

Quoted 6 lines

Wait-wait-wait.

if (!skb) break breaks the loop. put_rx_buffer() unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.

Specifically you don't want to be unmapping and freeing this page until after the !skb check. The problem is if skb is NULL the skb allocation failed and so processing of Rx is meant to stop in place without removing the page. It is where we will resume on the next pass assuming memory has been freed that can then be used. The problem is the skb allocation, not the page. We used to do the skb allocation before we would acquire the buffer, but with XDP there are cases where we aren't supposed to allocate it so it got moved to after which causes this confusion.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
                /* exit if we failed to retrieve a buffer */
                if (!skb) {
                        rx_ring->rx_stats.alloc_buff_failed++;
-                       if (rx_buffer && size)
-                               rx_buffer->pagecnt_bias++;
+                       __free_pages(rx_buffer->page,
+                                    iavf_rx_pg_order(rx_ring));
+                       rx_buffer->page = NULL;
                        break;
                }
Quoted 4 lines
Quoted 4 lines

This code was undoing the iavf_get_rx_buffer decrement of pagecnt_bias and then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment of pagecnt_bias.

Quoted 4 lines

The key bit here is the allocation failure is the reason why we halted processing. So the page contains valid data and should not be freed. We just need to leave it in place and wait for an allocation to succeed and then we can resume processing.

Aaah, okay, you want to try once again with the same buffer next time. I see. Makes sense!

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -1
-               iavf_put_rx_buffer(rx_ring, rx_buffer);
+               rx_buffer->page = NULL;
                cleaned_count++;
 
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320

Quoted 1 line
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -0
                if (iavf_is_non_eop(rx_ring, rx_desc, skb))

If iavf_put_rx_buffer just does the unmap and assignment of NULL then it could just be left here as is.

I guess those two are tied with the one above.

Yeah, the iavf_put_rx_buffer should be left down here.

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#10 lore raw

RFC PATCH v4 4/9 net: page_pool: add DMA-sync-for-CPU inline helpers

In reply to Alexander Lobakin

Each driver is responsible for syncing buffers written by HW for CPU before accessing them. Almost each PP-enabled driver uses the same pattern, which could be shorthanded into a static inline to make driver code a little bit more compact.
Introduce a couple such functions. The first one takes the actual size of the data written by HW and is the main one to be used on Rx. The second does the same, but only if the PP performs DMA synchronizations at all.
Also constify pointer arguments of page_pool_get_dma_dir() and page_pool_get_dma_addr() to give a bit more room for optimization, as both of them are read-only.

---
 include/net/page_pool.h | 49 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 46 insertions(+), 3 deletions(-)
include/net/page_pool.h +46 -3
index ff3772fab707..89acf9e33639 100644
@@ -304,8 +304,8 @@ static inline void *page_pool_dev_cache_alloc(struct page_pool *pool,
 /* get the stored dma direction. A driver might decide to treat this locally and
  * avoid the extra cache line from page_pool to determine the direction
  */
-static
-inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)
+static inline enum dma_data_direction
+page_pool_get_dma_dir(const struct page_pool *pool)
 {
 	return pool->p.dma_dir;
 }
@@ -460,7 +460,7 @@ static inline void page_pool_cache_free(struct page_pool *pool, void *data,
 	page_pool_put_page(pool, virt_to_head_page(data), -1, allow_direct);
 }
 
-static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
+static inline dma_addr_t page_pool_get_dma_addr(const struct page *page)
 {
 	dma_addr_t ret = page->dma_addr;
 
@@ -477,6 +477,49 @@ static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
 		page->dma_addr_upper = upper_32_bits(addr);
 }
 
+/**
+ * __page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by HW
+ * @pool: &page_pool the page belongs to
+ * @page: page to sync
+ * @offset: offset from page start to "hard" start if using frags
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Can be used as a shorthand to sync Rx pages before accessing them in the
+ * driver. Caller must ensure the pool was created with %PP_FLAG_DMA_MAP.
+ * Note that this version performs DMA sync unconditionally, even if the
+ * associated PP doesn't perform sync-for-device. Consider the non-underscored
+ * version first if unsure.
+ */
+static inline void __page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+						const struct page *page,
+						u32 offset, u32 dma_sync_size)
+{
+	dma_sync_single_range_for_cpu(pool->p.dev,
+				      page_pool_get_dma_addr(page),
+				      offset + pool->p.offset, dma_sync_size,
+				      page_pool_get_dma_dir(pool));
+}
+
+/**
+ * page_pool_dma_sync_for_cpu - sync Rx page for CPU if needed
+ * @pool: &page_pool the page belongs to
+ * @page: page to sync
+ * @offset: offset from page start to "hard" start if using frags
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Performs DMA sync for CPU, but *only* when:
+ * 1) page_pool was created with %PP_FLAG_DMA_SYNC_DEV to manage DMA syncs;
+ * 2) AND sync shortcut is not available (IOMMU, swiotlb, non-coherent DMA, ...)
+ */
+static inline void page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+					      const struct page *page,
+					      u32 offset, u32 dma_sync_size)
+{
+	if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+		__page_pool_dma_sync_for_cpu(pool, page, offset,
+					     dma_sync_size);
+}
+
 static inline bool is_page_pool_compiled_in(void)
 {
 #ifdef CONFIG_PAGE_POOL
Signature · 1 line
2.41.0
author maintainer
#11 lore raw

RFC PATCH v4 5/9 libie: add Rx buffer management (via Page Pool)

In reply to Alexander Lobakin

Add a couple intuitive helpers to hide Rx buffer implementation details in the library and not multiplicate it between drivers. The settings are optimized for Intel hardware, but nothing really HW-specific here. Use the new page_pool_dev_alloc() to dynamically switch between split-page and full-page modes depending on MTU, page size, required headroom etc. For example, on x86_64 with the default driver settings each page is shared between 2 buffers. Turning on XDP (not in this series) -> increasing headroom requirement pushes truesize out of 2048 boundary, leading to that each buffer starts getting a full page. The "ceiling" limit is %PAGE_SIZE, as only order-0 pages are used to avoid compound overhead. For the above architecture, this means maximum linear frame size of 3712 w/o XDP.

---
 drivers/net/ethernet/intel/Kconfig    |   1 +
 drivers/net/ethernet/intel/libie/rx.c |  54 +++++++++++++
 include/linux/net/intel/libie/rx.h    | 111 +++++++++++++++++++++++++-
 3 files changed, 165 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/Kconfig +1 -0
index 86ecedeac115..e187942b5a5c 100644
@@ -86,6 +86,7 @@ config E1000E_HWTS
 
 config LIBIE
 	tristate
+	select PAGE_POOL
 	help
 	  libie (Intel Ethernet library) is a common library containing
 	  routines shared by several Intel Ethernet drivers.
drivers/net/ethernet/intel/libie/rx.c +54 -0

60 lines not on this page · raw message

include/linux/net/intel/libie/rx.h +110 -1

117 lines not on this page · raw message

Signature · 6 lines
2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#12 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

Quoted 9 lines · Alexander Lobakin
drivers/net/ethernet/intel/libie/rx.c +9 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 10 lines
drivers/net/ethernet/intel/libie/rx.c +10 -0
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

drivers/net/ethernet/intel/libie/rx.c line 54

Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c +8 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);
+
+	/* "Wanted" truesize, passed to page_pool_dev_alloc() */
+	truesize = roundup_pow_of_two(SKB_HEAD_ALIGN(pp.offset + pp.max_len));
+	pp.init_arg = (void *)truesize;

I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with init_callback. And if we want to change the implemetation of init_callback, we may stuck with it as the driver is using it very differently here.

Is it possible to pass the 'wanted true size' by adding a parameter for libie_rx_alloc()?

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	return page_pool_create(&pp);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);
author maintainer
#13 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

Quoted 12 lines

drivers/net/ethernet/intel/libie/rx.c line 54

Quoted 9 lines
drivers/net/ethernet/intel/libie/rx.c +8 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);
+
+	/* "Wanted" truesize, passed to page_pool_dev_alloc() */
+	truesize = roundup_pow_of_two(SKB_HEAD_ALIGN(pp.offset + pp.max_len));
+	pp.init_arg = (void *)truesize;

I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with init_callback. And if we want to change the implemetation

I know. I abused it to save 1 function argument :p It's safe since I don't use init_callback (not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).

Quoted 5 lines

of init_callback, we may stuck with it as the driver is using it very differently here.

Is it possible to pass the 'wanted true size' by adding a parameter for libie_rx_alloc()?

Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D

Quoted 5 lines

drivers/net/ethernet/intel/libie/rx.c line 58

Quoted 5 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	return page_pool_create(&pp);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#14 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

Quoted 29 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Quoted 3 lines

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.

When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.

Quoted 6 lines
Quoted 6 lines

drivers/net/ethernet/intel/libie/rx.c line 50

Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);

As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.

Quoted 17 lines
Quoted 7 lines

drivers/net/ethernet/intel/libie/rx.c line 54

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	/* "Wanted" truesize, passed to page_pool_dev_alloc() */
+	truesize = roundup_pow_of_two(SKB_HEAD_ALIGN(pp.offset + pp.max_len));
+	pp.init_arg = (void *)truesize;

I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with init_callback. And if we want to change the implemetation

I know. I abused it to save 1 function argument :p It's safe since I don't use init_callback (not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).

Quoted 3 lines

of init_callback, we may stuck with it as the driver is using it very differently here.

Quoted 6 lines
Quoted 2 lines

Is it possible to pass the 'wanted true size' by adding a parameter for libie_rx_alloc()?

Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D

It seems most hw driver know the size of memory it needs when creating the ring/queue, setting the frag size and deciding how many is a page split into before allocation seems like a possible future optimization.

For now, it would be better to add helper to acess pp.init_arg at least instead of acess pp.init_arg directly to make it more obvious and make the future optimization more easier.

Quoted 9 lines
Quoted 6 lines

drivers/net/ethernet/intel/libie/rx.c line 58

Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	return page_pool_create(&pp);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);

Thanks,
Olek

author maintainer
#15 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800

Quoted 34 lines · Alexander Lobakin
Quoted 29 lines

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Quoted 3 lines

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

Quoted 6 lines

So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.

Not sure I'm following :s

Quoted 14 lines

When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.

Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

drivers/net/ethernet/intel/libie/rx.c line 50

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);

As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.

                                               ^^^^^^^^^^^^

You mean the whole page or...?
I think it's not the driver's duty to track all this. We always set .offset to data - data_hard_start and .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.

Quoted 25 lines
Quoted 18 lines
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 54

Quoted 5 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	/* "Wanted" truesize, passed to page_pool_dev_alloc() */
+	truesize = roundup_pow_of_two(SKB_HEAD_ALIGN(pp.offset + pp.max_len));
+	pp.init_arg = (void *)truesize;

I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with init_callback. And if we want to change the implemetation

I know. I abused it to save 1 function argument :p It's safe since I don't use init_callback (not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).

Quoted 3 lines

of init_callback, we may stuck with it as the driver is using it very differently here.

Quoted 6 lines
Quoted 2 lines

Is it possible to pass the 'wanted true size' by adding a parameter for libie_rx_alloc()?

Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D

Quoted 7 lines

It seems most hw driver know the size of memory it needs when creating the ring/queue, setting the frag size and deciding how many is a page split into before allocation seems like a possible future optimization.

For now, it would be better to add helper to acess pp.init_arg at least instead of acess pp.init_arg directly to make it more obvious and make the future optimization more easier.

Makes senses.

Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

drivers/net/ethernet/intel/libie/rx.c line 58

Quoted 7 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+
+	return page_pool_create(&pp);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);

Thanks,
Olek

Thanks,
Olek

#16 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

Quoted 37 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800

Quoted 34 lines · Alexander Lobakin
Quoted 32 lines

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

Quoted 8 lines
Quoted 6 lines

So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.

Not sure I'm following :s

It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.

If it is common enough that most drivers are using it the same way, adding a helper for that would be great.

Quoted 23 lines
Quoted 15 lines

When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.

Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

drivers/net/ethernet/intel/libie/rx.c line 50

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);

As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.

Quoted 3 lines
                                               ^^^^^^^^^^^^

You mean the whole page or...?

If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.

Quoted 6 lines

I think it's not the driver's duty to track all this. We always set .offset to data - data_hard_start and .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.

There problem is that when page_pool_put_page() is called with a split page, the page pool does not know which frag is freeing too.

setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:

https://elixir.free-electrons.com/linux/v6.4-rc6/source/net/core/page_pool.c#L325

author maintainer
#17 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:39:28 +0800

Quoted 45 lines · Alexander Lobakin
Quoted 44 lines

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800

Quoted 40 lines · Alexander Lobakin
Quoted 29 lines

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

drivers/net/ethernet/intel/libie/rx.c line 46

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Quoted 3 lines

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.

Quoted 8 lines
Quoted 1 line

Not sure I'm following :s

It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.

If it is common enough that most drivers are using it the same way, adding a helper for that would be great.

How comes page_pool_params is internal if it's defined purely by the driver and then exists read-only :D I even got warned in the adjacent thread that the Page Pool core code shouldn't change it anyhow.

Quoted 39 lines
Quoted 24 lines
Quoted 16 lines

When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.

Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

drivers/net/ethernet/intel/libie/rx.c line 50

Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);

As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.

Quoted 3 lines
                                               ^^^^^^^^^^^^

You mean the whole page or...?

If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.

Quoted 6 lines

I think it's not the driver's duty to track all this. We always set .offset to data - data_hard_start and .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.

Quoted 5 lines

There problem is that when page_pool_put_page() is called with a split page, the page pool does not know which frag is freeing too.

setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:

Maybe Page Pool should synchronize DMA even when !last_frag then? Setting .max_len to anything bigger than the maximum frame size you're planning to receive is counter-intuitive.
All three xdp_buff, xdp_frame and skb always have all info needed to determine which piece of the page we're recycling, it should be possible to do with no complications. Hypothetical forcing drivers to do DMA syncs on their own when they use frags is counter-intuitive as well, Page Pool should be able to handle this itself.

Alternatively, Page Pool may do as follows:

1. !last_frag -- do nothing, same as today.
2. last_frag -- sync, but not [offset, offset + max_len), but [offset, PAGE_SIZE).

This would also cover non-HW-writeable pieces like 2th-nth frame's headroom and each frame's skb_shared_info, but it's the only alternative to syncing each frag separately.
Yes, it's almost the same as to set .max_len to %PAGE_SIZE, but as I said, it feels weird to set .max_len to 4k when you allocate 2k frags. You don't know anyway how much of a page will be used.

For example, when I turn on driver-side XDP, increased headroom makes truesize cross the 2k border with 1500 MTU, so that 2 frag per page converts into 1 frag per page. In fact, instead of using the whole 4k, I use ~2200 and don't need to sync the entire 4k.
Setting .max_len to 4k gives you way heavier overhead to non DMA-coherent systems that per-frag-syncing would do.

Quoted 3 lines

https://elixir.free-electrons.com/linux/v6.4-rc6/source/net/core/page_pool.c#L325

Thanks,
Olek

#18 lore raw

Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)

Quoted 54 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:39:28 +0800

Quoted 51 lines · Alexander Lobakin
Quoted 37 lines

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800

Quoted 34 lines · Alexander Lobakin
Quoted 32 lines

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800

Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
(quoted diff) +15 -0
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.offset		= LIBIE_SKB_HEADROOM,

I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.

Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".

Quoted 8 lines
Quoted 6 lines

So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.

Not sure I'm following :s

It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.

Quoted 6 lines
Quoted 2 lines

If it is common enough that most drivers are using it the same way, adding a helper for that would be great.

How comes page_pool_params is internal if it's defined purely by the driver and then exists read-only :D I even got warned in the adjacent thread that the Page Pool core code shouldn't change it anyhow.

Personally I am not one hundred percent convinced that page_pool_params will not be changed considering the discussion about improving/replacing the page pool to support P2P case.

Quoted 43 lines
Quoted 43 lines
Quoted 25 lines
Quoted 17 lines

When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.

Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	};
+	size_t truesize;
+
+	pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);

As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.

Quoted 3 lines
                                               ^^^^^^^^^^^^

You mean the whole page or...?

If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.

Quoted 6 lines

I think it's not the driver's duty to track all this. We always set .offset to data - data_hard_start and .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.

There problem is that when page_pool_put_page() is called with a split page, the page pool does not know which frag is freeing too.

Quoted 6 lines
Quoted 2 lines

setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:

Maybe Page Pool should synchronize DMA even when !last_frag then? Setting .max_len to anything bigger than the maximum frame size you're planning to receive is counter-intuitive.

This is simplest way to support dma sync for page frag case, the question is if the batching of the dma sync for all frag of a page can even out the additional cost of dma sync for dma untouched range of a page.

Quoted 11 lines

All three xdp_buff, xdp_frame and skb always have all info needed to determine which piece of the page we're recycling, it should be possible to do with no complications. Hypothetical forcing drivers to do DMA syncs on their own when they use frags is counter-intuitive as well, Page Pool should be able to handle this itself.

Alternatively, Page Pool may do as follows:

1. !last_frag -- do nothing, same as today.
2. last_frag -- sync, but not [offset, offset + max_len), but [offset, PAGE_SIZE).

When a frag is free, we don't know if it is the last frag or not for now yet.

Quoted 7 lines

This would also cover non-HW-writeable pieces like 2th-nth frame's headroom and each frame's skb_shared_info, but it's the only alternative to syncing each frag separately.
Yes, it's almost the same as to set .max_len to %PAGE_SIZE, but as I said, it feels weird to set .max_len to 4k when you allocate 2k frags. You don't know anyway how much of a page will be used.

In that that case, we may need to make it more generic for the case when a page is spilt into more than two frags, especially for system with 64K page size.

author maintainer
#19 lore raw

RFC PATCH v4 6/9 iavf: switch to Page Pool

In reply to Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  33 +--
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 247 +++++-------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  91 +------
 drivers/net/ethernet/intel/iavf/iavf_type.h   |   2 -
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  19 +-
 5 files changed, 82 insertions(+), 310 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +4 -29
index db1ed13f11bb..39c6d83e80a1 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -698,32 +700,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)
  **/
 static void iavf_configure_rx(struct iavf_adapter *adapter)
 {
-	unsigned int rx_buf_len = IAVF_RXBUFFER_2048;
 	struct iavf_hw *hw = &adapter->hw;
-	int i;
-
-	if (PAGE_SIZE < 8192) {
-		struct net_device *netdev = adapter->netdev;
 
-		/* For jumbo frames on systems with 4K pages we have to use
-		 * an order 1 page, so we might as well increase the size
-		 * of our Rx buffer to make better use of the available space
-		 */
-		rx_buf_len = IAVF_RXBUFFER_3072;
-
-		/* We use a 1536 buffer size for configurations with
-		 * standard Ethernet mtu.  On x86 this gives us enough room
-		 * for shared info and 192 bytes of padding.
-		 */
-		if (!IAVF_2K_TOO_SMALL_WITH_PADDING &&
-		    (netdev->mtu <= ETH_DATA_LEN))
-			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-	}
-
-	for (i = 0; i < adapter->num_active_queues; i++) {
+	for (u32 i = 0; i < adapter->num_active_queues; i++)
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
-		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-	}
 }
 
 /**
@@ -1590,7 +1570,6 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		rx_ring = &adapter->rx_rings[i];
 		rx_ring->queue_index = i;
 		rx_ring->netdev = adapter->netdev;
-		rx_ring->dev = &adapter->pdev->dev;
 		rx_ring->count = adapter->rx_desc_count;
 		rx_ring->itr_setting = IAVF_ITR_RX_DEF;
 	}
@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 
 	netdev->netdev_ops = &iavf_netdev_ops;
 	iavf_set_ethtool_ops(netdev);
-	netdev->watchdog_timeo = 5 * HZ;
-
-	/* MTU range: 68 - 9710 */
-	netdev->min_mtu = ETH_MIN_MTU;
-	netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+	netdev->max_mtu = LIBIE_MAX_MTU;
 
 	if (!is_valid_ether_addr(adapter->hw.mac.addr)) {
 		dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",
drivers/net/ethernet/intel/iavf/iavf_txrx.c +62 -185

400 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +7 -84

131 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -2
index 3030ba330326..bb90d8f3ad7e 100644
@@ -10,8 +10,6 @@
 #include "iavf_adminq.h"
 #include "iavf_devids.h"
 
-#define IAVF_RXQ_CTX_DBUFF_SHIFT 7
-
 /* IAVF_MASK is a macro used on 32 bit registers */
 #define IAVF_MASK(mask, shift) ((u32)(mask) << (shift))
 
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +9 -10
index fdddc3588487..5341508b429f 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -269,14 +271,13 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)
 void iavf_configure_queues(struct iavf_adapter *adapter)
 {
 	struct virtchnl_vsi_queue_config_info *vqci;
-	int i, max_frame = adapter->vf_res->max_mtu;
+	u32 i, max_frame = adapter->vf_res->max_mtu;
+	struct iavf_ring *rxq = adapter->rx_rings;
 	int pairs = adapter->num_active_queues;
 	struct virtchnl_queue_pair_info *vqpi;
+	u32 hr, max_len;
 	size_t len;
 
-	if (max_frame > IAVF_MAX_RXBUFFER || !max_frame)
-		max_frame = IAVF_MAX_RXBUFFER;
-
 	if (adapter->current_op != VIRTCHNL_OP_UNKNOWN) {
 		/* bail because we already have a command pending */
 		dev_err(&adapter->pdev->dev, "Cannot configure queues, command %d pending\n",
@@ -289,9 +290,9 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 	if (!vqci)
 		return;
 
-	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (adapter->netdev->mtu <= ETH_DATA_LEN)
-		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
+	hr = rxq->pp->p.offset;
+	max_len = rxq->pp->p.max_len;
+	max_frame = min_not_zero(max_frame, LIBIE_MAX_RX_FRM_LEN(hr));
 
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
 	vqci->num_queue_pairs = pairs;
@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		vqpi->rxq.ring_len = adapter->rx_rings[i].count;
 		vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
 		vqpi->rxq.max_pkt_size = max_frame;
-		vqpi->rxq.databuffer_size =
-			ALIGN(adapter->rx_rings[i].rx_buf_len,
-			      BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));
+		vqpi->rxq.databuffer_size = max_len;
 		vqpi++;
 	}
 
Signature · 6 lines
2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#20 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

Quoted 14 lines · Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

---

...

drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2565

Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0 -1
@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 
 	netdev->netdev_ops = &iavf_netdev_ops;
 	iavf_set_ethtool_ops(netdev);
-	netdev->watchdog_timeo = 5 * HZ;

This seems like a unrelated change here?

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -4
-
-	/* MTU range: 68 - 9710 */
-	netdev->min_mtu = ETH_MIN_MTU;
-	netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+	netdev->max_mtu = LIBIE_MAX_MTU;
 

...

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 11 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

Quoted 14 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +6 -2
+	if (IS_ERR(pool))
+		return PTR_ERR(pool);
+
+	rx_ring->pp = pool;
 
 	/* warn if we are about to overwrite the pointer */
 	WARN_ON(rx_ring->rx_bi);
-	bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-	rx_ring->rx_bi = kzalloc(bi_size, GFP_KERNEL);
+	rx_ring->rx_bi = kcalloc(rx_ring->count, sizeof(*rx_ring->rx_bi),
+				 GFP_KERNEL);
 	if (!rx_ring->rx_bi)
 		goto err;
 

...

Quoted 31 lines

/**
* iavf_build_skb - Build skb around an existing buffer
- * @rx_ring: Rx descriptor ring to transact packets on
* @rx_buffer: Rx buffer to pull data from
* @size: size of buffer to add to skb
*
* This function builds an skb around an existing Rx buffer, taking care
* to set up the skb correctly and avoid any memcpy overhead. */

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -12
-static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
-				      struct iavf_rx_buffer *rx_buffer,
+static struct sk_buff *iavf_build_skb(const struct libie_rx_buffer *rx_buffer,
 				      unsigned int size)
 {
-	void *va;
-#if (PAGE_SIZE < 8192)
-	unsigned int truesize = iavf_rx_pg_size(rx_ring) / 2;
-#else
-	unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
-				SKB_DATA_ALIGN(IAVF_SKB_PAD + size);
-#endif
+	struct page *page = rx_buffer->page;
+	u32 hr = page->pp->p.offset;
 	struct sk_buff *skb;
+	void *va;
 
-	if (!rx_buffer || !size)
-		return NULL;
 	/* prefetch first cache line of first page */
-	va = page_address(rx_buffer->page) + rx_buffer->page_offset;

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +5 -3
-	net_prefetch(va);
+	va = page_address(page) + rx_buffer->offset;
+	net_prefetch(va + hr);
 
 	/* build an skb around the page buffer */
-	skb = napi_build_skb(va - IAVF_SKB_PAD, truesize);
-	if (unlikely(!skb))
+	skb = napi_build_skb(va, rx_buffer->truesize);
+	if (unlikely(!skb)) {
+		page_pool_put_page(page->pp, page, size, true);

Isn't it more correct to call page_pool_put_full_page() here? as we do not know which frag is used for the rx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that in Documentation/networking/page_pool.rst.

Quoted 2 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -0
 		return NULL;
+	}

...

Quoted 10 lines

struct iavf_queue_stats {

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -1
 	u64 packets;
 	u64 bytes;
@@ -311,16 +243,19 @@ enum iavf_ring_state_t {
 struct iavf_ring {
 	struct iavf_ring *next;		/* pointer to next ring in q_vector */
 	void *desc;			/* Descriptor ring memory */
-	struct device *dev;		/* Used for DMA mapping */
+	union {
+		struct page_pool *pp;	/* Used on Rx for buffer management */

drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 256

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +4 -1
+		struct device *dev;	/* Used on Tx for DMA mapping */
+	};
 	struct net_device *netdev;	/* netdev ring maps to */
 	union {
+		struct libie_rx_buffer *rx_bi;
 		struct iavf_tx_buffer *tx_bi;
-		struct iavf_rx_buffer *rx_bi;
 	};
 	DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+	u8 __iomem *tail;

Is there a reason to move it here?

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -1
 	u16 queue_index;		/* Queue number of ring */
 	u8 dcb_tc;			/* Traffic class of ring */
-	u8 __iomem *tail;
 
author maintainer
#21 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:22 +0800

Quoted 18 lines · Alexander Lobakin
Quoted 14 lines

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

---

...

Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2565

Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0 -1
@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 
 	netdev->netdev_ops = &iavf_netdev_ops;
 	iavf_set_ethtool_ops(netdev);
-	netdev->watchdog_timeo = 5 * HZ;

This seems like a unrelated change here?

Sorta. Default timeout is 5 seconds already, so I removed this redundance. But I expected someone to spot this, so I'm perfectly fine with not doing this [and stop doing such things in general].

Quoted 10 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -4
-
-	/* MTU range: 68 - 9710 */
-	netdev->min_mtu = ETH_MIN_MTU;
-	netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+	netdev->max_mtu = LIBIE_MAX_MTU;
 

...

Quoted 14 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 11 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -0
+	if (IS_ERR(pool))
+		return PTR_ERR(pool);
+
+	rx_ring->pp = pool;

[...]

Quoted 11 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
 	/* build an skb around the page buffer */
-	skb = napi_build_skb(va - IAVF_SKB_PAD, truesize);
-	if (unlikely(!skb))
+	skb = napi_build_skb(va, rx_buffer->truesize);
+	if (unlikely(!skb)) {
+		page_pool_put_page(page->pp, page, size, true);

Isn't it more correct to call page_pool_put_full_page() here? as we do not know which frag is used for the rx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that in Documentation/networking/page_pool.rst.

Ooof, maybe. My first try with PP frags. So when we use frags, we always must use _full_page() / p.max_len instead of the actual frame size?

Quoted 20 lines
Quoted 3 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -0
 		return NULL;
+	}

...

Quoted 14 lines

struct iavf_queue_stats {

drivers/net/ethernet/intel/iavf/iavf_txrx.h +4 -1
 	u64 packets;
 	u64 bytes;
@@ -311,16 +243,19 @@ enum iavf_ring_state_t {
 struct iavf_ring {
 	struct iavf_ring *next;		/* pointer to next ring in q_vector */
 	void *desc;			/* Descriptor ring memory */
-	struct device *dev;		/* Used for DMA mapping */
+	union {
+		struct page_pool *pp;	/* Used on Rx for buffer management */
+		struct device *dev;	/* Used on Tx for DMA mapping */
+	};
 	struct net_device *netdev;	/* netdev ring maps to */
 	union {
Quoted 8 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 256

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -1
+		struct libie_rx_buffer *rx_bi;
 		struct iavf_tx_buffer *tx_bi;
-		struct iavf_rx_buffer *rx_bi;
 	};
 	DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+	u8 __iomem *tail;

Is there a reason to move it here?

Oops, seems like it's a leftover. There is reason: removing hole in the structure, but it needs to be done right when I change its layout. In this commit I just alter unions with no actual layout changes. Will fix.

Quoted 6 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 259

Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -1
 	u16 queue_index;		/* Queue number of ring */
 	u8 dcb_tc;			/* Traffic class of ring */
-	u8 __iomem *tail;
 

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#22 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

On 2023/7/7 0:38, Alexander Lobakin wrote:

...

Quoted 16 lines
Quoted 15 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 12 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

Quoted 5 lines

But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.

It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?

Quoted 3 lines

BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.

The problem is that we may use more memory than before for certain case if we don't limit the size of ptr_ring, unless we can ensure all of recycling is done directly through cache, not ptr_ring.

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -0
+	if (IS_ERR(pool))
+		return PTR_ERR(pool);
+
+	rx_ring->pp = pool;

[...]

Quoted 11 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -2
 	/* build an skb around the page buffer */
-	skb = napi_build_skb(va - IAVF_SKB_PAD, truesize);
-	if (unlikely(!skb))
+	skb = napi_build_skb(va, rx_buffer->truesize);
+	if (unlikely(!skb)) {
+		page_pool_put_page(page->pp, page, size, true);

Isn't it more correct to call page_pool_put_full_page() here? as we do not know which frag is used for the rx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that in Documentation/networking/page_pool.rst.

Quoted 2 lines

Ooof, maybe. My first try with PP frags. So when we use frags, we always must use _full_page() / p.max_len instead of the actual frame size?

Currently, _full_page() / p.max_len must be used to ensure the correct dma sync operation.
But as mentioned in the previous patch, it might be about what is the semantics of dma sync operation for page split case.

Quoted 1 line
author maintainer
#23 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800

Quoted 26 lines · Alexander Lobakin

...

Quoted 16 lines
Quoted 15 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 12 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

Quoted 5 lines

But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.

Quoted 3 lines

It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?

But libie can't say for sure how PP will split pages for it, right?

Quoted 8 lines
Quoted 4 lines

BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.

The problem is that we may use more memory than before for certain case if we don't limit the size of ptr_ring, unless we can ensure all of recycling is done directly through cache, not ptr_ring.

Also not sure I'm following =\

[...]

Thanks,
Olek

#24 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

Quoted 36 lines · Alexander Lobakin

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800

Quoted 26 lines · Alexander Lobakin

...

Quoted 16 lines
Quoted 15 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 12 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

Quoted 5 lines

But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.

Quoted 3 lines

It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?

But libie can't say for sure how PP will split pages for it, right?

Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.

The problem is that we may use more memory than before for certain case if we don't limit the size of ptr_ring, unless we can ensure all of recycling is done directly through cache, not ptr_ring.

Also not sure I'm following =\

Before adding page pool support, the max memory used in the driver is as below:
rx_ring->count * PAGE_SIZE;

After adding page pool support, the max memory used in the driver is as below:

ptr_ring->size * PAGE_SIZE +
PP_ALLOC_CACHE_SIZE * PAGE_SIZE +
rx_ring->count * PAGE_SIZE / pp.init_arg

Quoted 8 lines

[...]

Thanks,
Olek

.

author maintainer
#25 lore raw

Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:47:14 +0800

Quoted 50 lines · Alexander Lobakin
Quoted 36 lines

From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800

Quoted 26 lines · Alexander Lobakin

...

Quoted 16 lines
Quoted 15 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748

Quoted 12 lines

/**

drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -2
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-	struct device *dev = rx_ring->dev;
-	int bi_size;
+	struct page_pool *pool;
+
+	pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+					 rx_ring->count);

If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.

Quoted 5 lines

But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.

Quoted 3 lines

It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?

But libie can't say for sure how PP will split pages for it, right?

Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.

The problem is that we may use more memory than before for certain case if we don't limit the size of ptr_ring, unless we can ensure all of recycling is done directly through cache, not ptr_ring.

Also not sure I'm following =\

Before adding page pool support, the max memory used in the driver is as below:
rx_ring->count * PAGE_SIZE;

Quoted 6 lines

After adding page pool support, the max memory used in the driver is as below:

ptr_ring->size * PAGE_SIZE +
PP_ALLOC_CACHE_SIZE * PAGE_SIZE +
rx_ring->count * PAGE_SIZE / pp.init_arg

I know. I was wondering how your message connects with what I said regarding that using direct recycling more aggressively allows us to reduce ptr_ring sizes, as e.g. on my setup ptr_ring is not used at all (it still will be in other cases, but a lot less often than before).

Quoted 9 lines
Quoted 9 lines

[...]

Thanks,
Olek

.

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#26 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

In reply to Alexander Lobakin
Quoted 13 lines · Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

Quoted 1 line

One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.

Quoted 64 lines
---
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  33 +--
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 247 +++++-------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  91 +------
 drivers/net/ethernet/intel/iavf/iavf_type.h   |   2 -
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  19 +-
 5 files changed, 82 insertions(+), 310 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +3 -24
index db1ed13f11bb..39c6d83e80a1 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -698,32 +700,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)
  **/
 static void iavf_configure_rx(struct iavf_adapter *adapter)
 {
-       unsigned int rx_buf_len = IAVF_RXBUFFER_2048;
        struct iavf_hw *hw = &adapter->hw;
-       int i;
-
-       if (PAGE_SIZE < 8192) {
-               struct net_device *netdev = adapter->netdev;
 
-               /* For jumbo frames on systems with 4K pages we have to use
-                * an order 1 page, so we might as well increase the size
-                * of our Rx buffer to make better use of the available space
-                */
-               rx_buf_len = IAVF_RXBUFFER_3072;
-
-               /* We use a 1536 buffer size for configurations with
-                * standard Ethernet mtu.  On x86 this gives us enough room
-                * for shared info and 192 bytes of padding.
-                */
-               if (!IAVF_2K_TOO_SMALL_WITH_PADDING &&
-                   (netdev->mtu <= ETH_DATA_LEN))
-                       rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-       }
-
-       for (i = 0; i < adapter->num_active_queues; i++) {
+       for (u32 i = 0; i < adapter->num_active_queues; i++)
                adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
-               adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-       }
 }
 
 /**
@@ -1590,7 +1570,6 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
                rx_ring = &adapter->rx_rings[i];
                rx_ring->queue_index = i;
                rx_ring->netdev = adapter->netdev;
-               rx_ring->dev = &adapter->pdev->dev;
                rx_ring->count = adapter->rx_desc_count;
                rx_ring->itr_setting = IAVF_ITR_RX_DEF;
        }

drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2566

Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0 -2
@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 
        netdev->netdev_ops = &iavf_netdev_ops;
        iavf_set_ethtool_ops(netdev);
-       netdev->watchdog_timeo = 5 * HZ;
-

These removals should be pulled out into a seperate patch where we can call out that they are not needed due to redundancies w/ other code on the core network code.

drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2544

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -3
-       /* MTU range: 68 - 9710 */
-       netdev->min_mtu = ETH_MIN_MTU;
-       netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+       netdev->max_mtu = LIBIE_MAX_MTU;

Same here for min_mtu.

Quoted 51 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0 -0
        if (!is_valid_ether_addr(adapter->hw.mac.addr)) {
                dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4 -27
index 789b10815d7f..d1491b481eac 100644
@@ -689,9 +689,6 @@ int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring)
  **/
 static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
 {
-       unsigned long bi_size;
-       u16 i;
-
        /* ring already cleared, nothing to do */
        if (!rx_ring->rx_bi)
                return;
@@ -701,40 +698,16 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
                rx_ring->skb = NULL;
        }
 
-       /* Free all the Rx ring sk_buffs */
-       for (i = 0; i < rx_ring->count; i++) {
-               struct iavf_rx_buffer *rx_bi = &rx_ring->rx_bi[i];
+       /* Free all the Rx ring buffers */
+       for (u32 i = rx_ring->next_to_clean; i != rx_ring->next_to_use; ) {
+               const struct libie_rx_buffer *rx_bi = &rx_ring->rx_bi[i];
 
-               if (!rx_bi->page)
-                       continue;
+               page_pool_put_full_page(rx_ring->pp, rx_bi->page, false);
 
-               /* Invalidate cache lines that may have been written to by
-                * device so that we avoid corrupting memory.
-                */
-               dma_sync_single_range_for_cpu(rx_ring->dev,
-                                             rx_bi->dma,
-                                             rx_bi->page_offset,
-                                             rx_ring->rx_buf_len,
-                                             DMA_FROM_DEVICE);
-
-               /* free resources associated with mapping */
-               dma_unmap_page_attrs(rx_ring->dev, rx_bi->dma,
-                                    iavf_rx_pg_size(rx_ring),
-                                    DMA_FROM_DEVICE,
-                                    IAVF_RX_DMA_ATTR);
-
-               __free_pages(rx_bi->page, iavf_rx_pg_order(rx_ring));
-
-               rx_bi->page = NULL;
-               rx_bi->page_offset = 0;

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -6
+               if (unlikely(++i == rx_ring->count))
+                       i = 0;
        }
 
-       bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-       memset(rx_ring->rx_bi, 0, bi_size);
-
-       /* Zero out the descriptor ring */
-       memset(rx_ring->desc, 0, rx_ring->size);
-

I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things in setup_rx_descriptors.

Quoted 462 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +56 -152
        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;
 }
@@ -752,10 +725,13 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
        rx_ring->rx_bi = NULL;
 
        if (rx_ring->desc) {
-               dma_free_coherent(rx_ring->dev, rx_ring->size,
+               dma_free_coherent(rx_ring->pp->p.dev, rx_ring->size,
                                  rx_ring->desc, rx_ring->dma);
                rx_ring->desc = NULL;
        }
+
+       page_pool_destroy(rx_ring->pp);
+       rx_ring->pp = NULL;
 }
 
 /**
@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)
  **/
 int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 {
-       struct device *dev = rx_ring->dev;
-       int bi_size;
+       struct page_pool *pool;
+
+       pool = libie_rx_page_pool_create(&rx_ring->q_vector->napi,
+                                        rx_ring->count);
+       if (IS_ERR(pool))
+               return PTR_ERR(pool);
+
+       rx_ring->pp = pool;
 
        /* warn if we are about to overwrite the pointer */
        WARN_ON(rx_ring->rx_bi);
-       bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-       rx_ring->rx_bi = kzalloc(bi_size, GFP_KERNEL);
+       rx_ring->rx_bi = kcalloc(rx_ring->count, sizeof(*rx_ring->rx_bi),
+                                GFP_KERNEL);
        if (!rx_ring->rx_bi)
                goto err;
 
@@ -781,22 +763,27 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
        /* Round up to nearest 4K */
        rx_ring->size = rx_ring->count * sizeof(union iavf_32byte_rx_desc);
        rx_ring->size = ALIGN(rx_ring->size, 4096);
-       rx_ring->desc = dma_alloc_coherent(dev, rx_ring->size,
+       rx_ring->desc = dma_alloc_coherent(pool->p.dev, rx_ring->size,
                                           &rx_ring->dma, GFP_KERNEL);
 
        if (!rx_ring->desc) {
-               dev_info(dev, "Unable to allocate memory for the Rx descriptor ring, size=%d\n",
+               dev_info(pool->p.dev, "Unable to allocate memory for the Rx descriptor ring, size=%d\n",
                         rx_ring->size);
-               goto err;
+               goto err_free_buf;
        }
 
        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;
 
        return 0;
-err:
+
+err_free_buf:
        kfree(rx_ring->rx_bi);
        rx_ring->rx_bi = NULL;
+
+err:
+       page_pool_destroy(rx_ring->pp);
+
        return -ENOMEM;
 }
 
@@ -818,49 +805,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)
        writel(val, rx_ring->tail);
 }
 
-/**
- * iavf_alloc_mapped_page - recycle or make a new page
- * @rx_ring: ring to use
- * @bi: rx_buffer struct to modify
- *
- * Returns true if the page was successfully allocated or
- * reused.
- **/
-static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
-                                  struct iavf_rx_buffer *bi)
-{
-       struct page *page = bi->page;
-       dma_addr_t dma;
-
-       /* alloc new page for storage */
-       page = dev_alloc_pages(iavf_rx_pg_order(rx_ring));
-       if (unlikely(!page)) {
-               rx_ring->rx_stats.alloc_page_failed++;
-               return false;
-       }
-
-       /* map page for use */
-       dma = dma_map_page_attrs(rx_ring->dev, page, 0,
-                                iavf_rx_pg_size(rx_ring),
-                                DMA_FROM_DEVICE,
-                                IAVF_RX_DMA_ATTR);
-
-       /* if mapping failed free memory back to system since
-        * there isn't much point in holding memory we can't use
-        */
-       if (dma_mapping_error(rx_ring->dev, dma)) {
-               __free_pages(page, iavf_rx_pg_order(rx_ring));
-               rx_ring->rx_stats.alloc_page_failed++;
-               return false;
-       }
-
-       bi->dma = dma;
-       bi->page = page;
-       bi->page_offset = IAVF_SKB_PAD;
-
-       return true;
-}
-
 /**
  * iavf_receive_skb - Send a completed packet up the stack
  * @rx_ring:  rx ring in play
@@ -893,36 +837,29 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
 {
        u16 ntu = rx_ring->next_to_use;
        union iavf_rx_desc *rx_desc;
-       struct iavf_rx_buffer *bi;
 
        /* do nothing if no valid netdev defined */
        if (!rx_ring->netdev || !cleaned_count)
                return false;
 
        rx_desc = IAVF_RX_DESC(rx_ring, ntu);
-       bi = &rx_ring->rx_bi[ntu];
 
        do {
-               if (!iavf_alloc_mapped_page(rx_ring, bi))
-                       goto no_buffers;
+               dma_addr_t addr;
 
-               /* sync the buffer for use by the device */
-               dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
-                                                bi->page_offset,
-                                                rx_ring->rx_buf_len,
-                                                DMA_FROM_DEVICE);
+               addr = libie_rx_alloc(rx_ring->pp, &rx_ring->rx_bi[ntu]);
+               if (unlikely(addr == DMA_MAPPING_ERROR))
+                       goto no_buffers;
 
                /* Refresh the desc even if buffer_addrs didn't change
                 * because each write-back erases this info.
                 */
-               rx_desc->read.pkt_addr = cpu_to_le64(bi->dma + bi->page_offset);
+               rx_desc->read.pkt_addr = cpu_to_le64(addr);
 
                rx_desc++;
-               bi++;
                ntu++;
                if (unlikely(ntu == rx_ring->count)) {
                        rx_desc = IAVF_RX_DESC(rx_ring, 0);
-                       bi = rx_ring->rx_bi;
                        ntu = 0;
                }
 
@@ -941,6 +878,8 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
        if (rx_ring->next_to_use != ntu)
                iavf_release_rx_desc(rx_ring, ntu);
 
+       rx_ring->rx_stats.alloc_page_failed++;
+
        /* make sure to come back via polling to try again after
         * allocation failure
         */
@@ -1091,9 +1030,8 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
 
 /**
  * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
- * @rx_ring: rx descriptor ring to transact packets on
- * @rx_buffer: buffer containing page to add
  * @skb: sk_buff to place the data into
+ * @rx_buffer: buffer containing page to add
  * @size: packet length from rx_desc
  *
  * This function will add the data contained in rx_buffer->page to the skb.
@@ -1101,111 +1039,52 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
  *
  * The function will then update the page offset.
  **/
-static void iavf_add_rx_frag(struct iavf_ring *rx_ring,
-                            struct iavf_rx_buffer *rx_buffer,
-                            struct sk_buff *skb,
+static void iavf_add_rx_frag(struct sk_buff *skb,
+                            const struct libie_rx_buffer *rx_buffer,
                             unsigned int size)
 {
-#if (PAGE_SIZE < 8192)
-       unsigned int truesize = iavf_rx_pg_size(rx_ring) / 2;
-#else
-       unsigned int truesize = SKB_DATA_ALIGN(size + IAVF_SKB_PAD);
-#endif
-
-       if (!size)
-               return;
+       u32 hr = rx_buffer->page->pp->p.offset;
 
        skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
-                       rx_buffer->page_offset, size, truesize);
-}
-
-/**
- * iavf_get_rx_buffer - Fetch Rx buffer and synchronize data for use
- * @rx_ring: rx descriptor ring to transact packets on
- * @size: size of buffer to add to skb
- *
- * This function will pull an Rx buffer from the ring and synchronize it
- * for use by the CPU.
- */
-static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring,
-                                                const unsigned int size)
-{
-       struct iavf_rx_buffer *rx_buffer;
-
-       rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean];
-       prefetchw(rx_buffer->page);
-       if (!size)
-               return rx_buffer;
-
-       /* we are reusing so sync this buffer for CPU use */
-       dma_sync_single_range_for_cpu(rx_ring->dev,
-                                     rx_buffer->dma,
-                                     rx_buffer->page_offset,
-                                     size,
-                                     DMA_FROM_DEVICE);
-
-       return rx_buffer;
+                       rx_buffer->offset + hr, size, rx_buffer->truesize);
 }
 
 /**
  * iavf_build_skb - Build skb around an existing buffer
- * @rx_ring: Rx descriptor ring to transact packets on
  * @rx_buffer: Rx buffer to pull data from
  * @size: size of buffer to add to skb
  *
  * This function builds an skb around an existing Rx buffer, taking care
  * to set up the skb correctly and avoid any memcpy overhead.
  */
-static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
-                                     struct iavf_rx_buffer *rx_buffer,
+static struct sk_buff *iavf_build_skb(const struct libie_rx_buffer *rx_buffer,
                                      unsigned int size)
 {
-       void *va;
-#if (PAGE_SIZE < 8192)
-       unsigned int truesize = iavf_rx_pg_size(rx_ring) / 2;
-#else
-       unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
-                               SKB_DATA_ALIGN(IAVF_SKB_PAD + size);
-#endif
+       struct page *page = rx_buffer->page;
+       u32 hr = page->pp->p.offset;
        struct sk_buff *skb;
+       void *va;
 
-       if (!rx_buffer || !size)
-               return NULL;
        /* prefetch first cache line of first page */
-       va = page_address(rx_buffer->page) + rx_buffer->page_offset;
-       net_prefetch(va);
+       va = page_address(page) + rx_buffer->offset;
+       net_prefetch(va + hr);
 
        /* build an skb around the page buffer */
-       skb = napi_build_skb(va - IAVF_SKB_PAD, truesize);
-       if (unlikely(!skb))
+       skb = napi_build_skb(va, rx_buffer->truesize);
+       if (unlikely(!skb)) {
+               page_pool_put_page(page->pp, page, size, true);
                return NULL;
+       }
+
+       skb_mark_for_recycle(skb);
 
        /* update pointers within the skb to store the data */
-       skb_reserve(skb, IAVF_SKB_PAD);
+       skb_reserve(skb, hr);
        __skb_put(skb, size);
 
        return skb;
 }
 
-/**
- * iavf_put_rx_buffer - Unmap used buffer
- * @rx_ring: rx descriptor ring to transact packets on
- * @rx_buffer: rx buffer to pull data from
- *
- * This function will unmap the buffer after it's written by HW.
- */
-static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
-                              struct iavf_rx_buffer *rx_buffer)
-{
-       if (!rx_buffer)
-               return;
-
-       /* we are not reusing the buffer so unmap it */
-       dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
-                            iavf_rx_pg_size(rx_ring),
-                            DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
-}
-
 /**
  * iavf_is_non_eop - process handling of non-EOP buffers
  * @rx_ring: Rx ring being processed
@@ -1259,7 +1138,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
        bool failure = false;
 
        while (likely(total_rx_packets < (unsigned int)budget)) {
-               struct iavf_rx_buffer *rx_buffer;
+               struct libie_rx_buffer *rx_buffer;
                union iavf_rx_desc *rx_desc;
                unsigned int size;
                u16 vlan_tag = 0;
@@ -1295,26 +1174,24 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
                       IAVF_RXD_QW1_LENGTH_PBUF_SHIFT;
 
                iavf_trace(clean_rx_irq, rx_ring, rx_desc, skb);
-               rx_buffer = iavf_get_rx_buffer(rx_ring, size);
+
+               rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean];
+               if (!libie_rx_sync_for_cpu(rx_buffer, size))
+                       goto skip_data;
 
                /* retrieve a buffer from the ring */
                if (skb)
-                       iavf_add_rx_frag(rx_ring, rx_buffer, skb, size);
+                       iavf_add_rx_frag(skb, rx_buffer, size);
                else
-                       skb = iavf_build_skb(rx_ring, rx_buffer, size);
-
-               iavf_put_rx_buffer(rx_ring, rx_buffer);
+                       skb = iavf_build_skb(rx_buffer, size);
 
                /* exit if we failed to retrieve a buffer */
                if (!skb) {
                        rx_ring->rx_stats.alloc_buff_failed++;
-                       __free_pages(rx_buffer->page,
-                                    iavf_rx_pg_order(rx_ring));
-                       rx_buffer->page = NULL;
                        break;
                }
 
-               rx_buffer->page = NULL;
+skip_data:
                cleaned_count++;
 
                if (iavf_is_non_eop(rx_ring, rx_desc, skb))
drivers/net/ethernet/intel/iavf/iavf_txrx.h +5 -69
index 2170a77f8c8d..b13d878c74c6 100644
@@ -81,70 +81,8 @@ enum iavf_dyn_idx_t {
        BIT_ULL(IAVF_FILTER_PCTYPE_NONF_UNICAST_IPV6_UDP) | \
        BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP))
 
-/* Supported Rx Buffer Sizes (a multiple of 128) */
-#define IAVF_RXBUFFER_1536  1536  /* 128B aligned standard Ethernet frame */
-#define IAVF_RXBUFFER_2048  2048
-#define IAVF_RXBUFFER_3072  3072  /* Used for large frames w/ padding */
-#define IAVF_MAX_RXBUFFER   9728  /* largest size for single descriptor */
-
-#define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))
 #define iavf_rx_desc iavf_32byte_rx_desc
 
-#define IAVF_RX_DMA_ATTR \
-       (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
-
-/* Attempt to maximize the headroom available for incoming frames.  We
- * use a 2K buffer for receives and need 1536/1534 to store the data for
- * the frame.  This leaves us with 512 bytes of room.  From that we need
- * to deduct the space needed for the shared info and the padding needed
- * to IP align the frame.
- *
- * Note: For cache line sizes 256 or larger this value is going to end
- *      up negative.  In these cases we should fall back to the legacy
- *      receive path.
- */
-#if (PAGE_SIZE < 8192)
-#define IAVF_2K_TOO_SMALL_WITH_PADDING \
-((NET_SKB_PAD + IAVF_RXBUFFER_1536) > SKB_WITH_OVERHEAD(IAVF_RXBUFFER_2048))
-
-static inline int iavf_compute_pad(int rx_buf_len)
-{
-       int page_size, pad_size;
-
-       page_size = ALIGN(rx_buf_len, PAGE_SIZE / 2);
-       pad_size = SKB_WITH_OVERHEAD(page_size) - rx_buf_len;
-
-       return pad_size;
-}
-
-static inline int iavf_skb_pad(void)
-{
-       int rx_buf_len;
-
-       /* If a 2K buffer cannot handle a standard Ethernet frame then
-        * optimize padding for a 3K buffer instead of a 1.5K buffer.
-        *
-        * For a 3K buffer we need to add enough padding to allow for
-        * tailroom due to NET_IP_ALIGN possibly shifting us out of
-        * cache-line alignment.
-        */
-       if (IAVF_2K_TOO_SMALL_WITH_PADDING)
-               rx_buf_len = IAVF_RXBUFFER_3072 + SKB_DATA_ALIGN(NET_IP_ALIGN);
-       else
-               rx_buf_len = IAVF_RXBUFFER_1536;
-
-       /* if needed make room for NET_IP_ALIGN */
-       rx_buf_len -= NET_IP_ALIGN;
-
-       return iavf_compute_pad(rx_buf_len);
-}
-
-#define IAVF_SKB_PAD iavf_skb_pad()
-#else
-#define IAVF_2K_TOO_SMALL_WITH_PADDING false
-#define IAVF_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
-#endif
-
 /**
  * iavf_test_staterr - tests bits in Rx descriptor status and error fields
  * @rx_desc: pointer to receive descriptor (in le64 format)
@@ -263,12 +201,6 @@ struct iavf_tx_buffer {
        u32 tx_flags;
 };
 
-struct iavf_rx_buffer {
-       dma_addr_t dma;
-       struct page *page;
-       __u32 page_offset;
-};
-
 struct iavf_queue_stats {
        u64 packets;
        u64 bytes;
@@ -311,16 +243,19 @@ enum iavf_ring_state_t {
 struct iavf_ring {
        struct iavf_ring *next;         /* pointer to next ring in q_vector */
        void *desc;                     /* Descriptor ring memory */
-       struct device *dev;             /* Used for DMA mapping */
+       union {
+               struct page_pool *pp;   /* Used on Rx for buffer management */
+               struct device *dev;     /* Used on Tx for DMA mapping */
+       };
        struct net_device *netdev;      /* netdev ring maps to */
        union {
+               struct libie_rx_buffer *rx_bi;
                struct iavf_tx_buffer *tx_bi;

drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 261

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
-               struct iavf_rx_buffer *rx_bi;
        };
        DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+       u8 __iomem *tail;
        u16 queue_index;                /* Queue number of ring */
        u8 dcb_tc;                      /* Traffic class of ring */
-       u8 __iomem *tail;
 
        /* high bit set means dynamic, use accessors routines to read/write.
         * hardware only supports 2us resolution for the ITR registers.

I'm assuming "tail" was moved here since it is a pointer and fills a hole?

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268

Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
@@ -329,9 +264,8 @@ struct iavf_ring {
         */
        u16 itr_setting;
 
-       u16 count;                      /* Number of descriptors */
        u16 reg_idx;                    /* HW register index of the ring */
-       u16 rx_buf_len;
+       u16 count;                      /* Number of descriptors */

Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.

Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -1
        /* used in interrupt processing */
        u16 next_to_use;
@@ -398,17 +332,6 @@ struct iavf_ring_container {
 #define iavf_for_each_ring(pos, head) \
        for (pos = (head).ring; pos != NULL; pos = pos->next)
 
-static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -10
-{
-#if (PAGE_SIZE < 8192)
-       if (ring->rx_buf_len > (PAGE_SIZE / 2))
-               return 1;
-#endif
-       return 0;
-}
-
-#define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
-

All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.

Quoted 60 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
 bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring);
drivers/net/ethernet/intel/iavf/iavf_type.h +0 -2
index 3030ba330326..bb90d8f3ad7e 100644
@@ -10,8 +10,6 @@
 #include "iavf_adminq.h"
 #include "iavf_devids.h"
 
-#define IAVF_RXQ_CTX_DBUFF_SHIFT 7
-
 /* IAVF_MASK is a macro used on 32 bit registers */
 #define IAVF_MASK(mask, shift) ((u32)(mask) << (shift))
 
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +8 -7
index fdddc3588487..5341508b429f 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -269,14 +271,13 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)
 void iavf_configure_queues(struct iavf_adapter *adapter)
 {
        struct virtchnl_vsi_queue_config_info *vqci;
-       int i, max_frame = adapter->vf_res->max_mtu;
+       u32 i, max_frame = adapter->vf_res->max_mtu;
+       struct iavf_ring *rxq = adapter->rx_rings;
        int pairs = adapter->num_active_queues;
        struct virtchnl_queue_pair_info *vqpi;
+       u32 hr, max_len;
        size_t len;
 
-       if (max_frame > IAVF_MAX_RXBUFFER || !max_frame)
-               max_frame = IAVF_MAX_RXBUFFER;
-
        if (adapter->current_op != VIRTCHNL_OP_UNKNOWN) {
                /* bail because we already have a command pending */
                dev_err(&adapter->pdev->dev, "Cannot configure queues, command %d pending\n",
@@ -289,9 +290,9 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
        if (!vqci)
                return;
 
-       /* Limit maximum frame size when jumbo frames is not enabled */
-       if (adapter->netdev->mtu <= ETH_DATA_LEN)
-               max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
+       hr = rxq->pp->p.offset;
+       max_len = rxq->pp->p.max_len;
+       max_frame = min_not_zero(max_frame, LIBIE_MAX_RX_FRM_LEN(hr));
        vqci->vsi_id = adapter->vsi_res->vsi_id;
        vqci->num_queue_pairs = pairs;

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0 -3
@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
                vqpi->rxq.ring_len = adapter->rx_rings[i].count;
                vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
                vqpi->rxq.max_pkt_size = max_frame;
-               vqpi->rxq.databuffer_size =
-                       ALIGN(adapter->rx_rings[i].rx_buf_len,
-                             BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));

Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.

Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -1
+               vqpi->rxq.databuffer_size = max_len;
                vqpi++;
        }
 
--

2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#27 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700

Quoted 15 lines
Quoted 13 lines · Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

Quoted 7 lines
Quoted 1 line

One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.

min_mtu and watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s

[...]

Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -6
-       bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-       memset(rx_ring->rx_bi, 0, bi_size);
-
-       /* Zero out the descriptor ring */
-       memset(rx_ring->desc, 0, rx_ring->size);
-
Quoted 7 lines

I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things in setup_rx_descriptors.

Both arrays will be freed couple instructions below, why zero them?

Quoted 5 lines
Quoted 5 lines

        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;

}

[...]

Quoted 12 lines
Quoted 12 lines

struct net_device *netdev; /* netdev ring maps to */
union {

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -2
+               struct libie_rx_buffer *rx_bi;
                struct iavf_tx_buffer *tx_bi;
-               struct iavf_rx_buffer *rx_bi;
        };
        DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+       u8 __iomem *tail;
        u16 queue_index;                /* Queue number of ring */
        u8 dcb_tc;                      /* Traffic class of ring */
-       u8 __iomem *tail;
Quoted 4 lines
Quoted 2 lines

/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.

I'm assuming "tail" was moved here since it is a pointer and fills a hole?

(see above)

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
@@ -329,9 +264,8 @@ struct iavf_ring {
         */
        u16 itr_setting;
 
Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
-       u16 count;                      /* Number of descriptors */
        u16 reg_idx;                    /* HW register index of the ring */
-       u16 rx_buf_len;
+       u16 count;                      /* Number of descriptors */

Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.

With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.

Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
        /* used in interrupt processing */
        u16 next_to_use;
@@ -398,17 +332,6 @@ struct iavf_ring_container {
 #define iavf_for_each_ring(pos, head) \
        for (pos = (head).ring; pos != NULL; pos = pos->next)
 
Quoted 16 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411

Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -11
-static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
-{
-#if (PAGE_SIZE < 8192)
-       if (ring->rx_buf_len > (PAGE_SIZE / 2))
-               return 1;
-#endif
-       return 0;
-}
-
-#define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
-

All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.

This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.

Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
 bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring);

[...]

Quoted 10 lines

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314

Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0 -3
@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
                vqpi->rxq.ring_len = adapter->rx_rings[i].count;
                vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
                vqpi->rxq.max_pkt_size = max_frame;
-               vqpi->rxq.databuffer_size =
-                       ALIGN(adapter->rx_rings[i].rx_buf_len,
-                             BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));

Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.

See the previous commit, the place where I calculate max_len for the PP params. 128 byte is Intel-wide HW req, so it lives there now.

Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -0
+               vqpi->rxq.databuffer_size = max_len;
                vqpi++;

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#28 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

Quoted 48 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700

Quoted 15 lines
Quoted 13 lines · Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

Quoted 7 lines
Quoted 1 line

One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.

min_mtu and watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s

[...]

Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -6
-       bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-       memset(rx_ring->rx_bi, 0, bi_size);
-
-       /* Zero out the descriptor ring */
-       memset(rx_ring->desc, 0, rx_ring->size);
-
Quoted 7 lines

I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things in setup_rx_descriptors.

Quoted 1 line

Both arrays will be freed couple instructions below, why zero them?

Ugh. You are right, but not for a good reason. So the other Intel drivers in the past would be doing the clean_rx_ring calls on the _down() with the freeing of resources on _close(). Specifically it allowed reducing the overhead for things like resets or setting changes since it didn't require reallocating the descriptor rings and buffer info structures.

I guess you are good to remove these since this code doesn't do that.

Quoted 40 lines
Quoted 5 lines
Quoted 5 lines

        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;

}

[...]

Quoted 12 lines
Quoted 12 lines

struct net_device *netdev; /* netdev ring maps to */
union {

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -2
+               struct libie_rx_buffer *rx_bi;
                struct iavf_tx_buffer *tx_bi;
-               struct iavf_rx_buffer *rx_bi;
        };
        DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+       u8 __iomem *tail;
        u16 queue_index;                /* Queue number of ring */
        u8 dcb_tc;                      /* Traffic class of ring */
-       u8 __iomem *tail;
Quoted 4 lines
Quoted 2 lines

/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.

I'm assuming "tail" was moved here since it is a pointer and fills a hole?

(see above)

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
@@ -329,9 +264,8 @@ struct iavf_ring {
         */
        u16 itr_setting;
 
Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
-       u16 count;                      /* Number of descriptors */
        u16 reg_idx;                    /* HW register index of the ring */
-       u16 rx_buf_len;
+       u16 count;                      /* Number of descriptors */

Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.

Quoted 4 lines

With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.

The performance impact should be minimal. Odds are the placement was the way it was since it was probably just copying the original code that has been there since igb/ixgbe. The general idea is just keep the read-mostly items grouped at the top and try to order them somewhat by frequency of being read so that wherever the cache line ends up you won't take much of a penalty as hopefully you will just have the infrequently read items end up getting pulled into the active cache line.

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
        /* used in interrupt processing */
        u16 next_to_use;
@@ -398,17 +332,6 @@ struct iavf_ring_container {
 #define iavf_for_each_ring(pos, head) \
        for (pos = (head).ring; pos != NULL; pos = pos->next)
 
Quoted 16 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411

Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -11
-static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
-{
-#if (PAGE_SIZE < 8192)
-       if (ring->rx_buf_len > (PAGE_SIZE / 2))
-               return 1;
-#endif
-       return 0;
-}
-
-#define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
-

All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.

Quoted 4 lines

This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.

It all depends on how much is having to be added to accommodate this. In my mind when we did away with the page splitting/recycling we also did away with the need for the higher order pages. That is why I was thinking it might make more sense there as it would just be more removals with very few if any additions needed to support it.

Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
 bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring);

[...]

Quoted 10 lines

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314

Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0 -3
@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
                vqpi->rxq.ring_len = adapter->rx_rings[i].count;
                vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
                vqpi->rxq.max_pkt_size = max_frame;
-               vqpi->rxq.databuffer_size =
-                       ALIGN(adapter->rx_rings[i].rx_buf_len,
-                             BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));

Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.

Quoted 2 lines

See the previous commit, the place where I calculate max_len for the PP params. 128 byte is Intel-wide HW req, so it lives there now.

Okay, that is the piece I missed. It was converted from a BIT_ULL(7) to just a 128. Thanks.

Quoted 6 lines
Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -0
+               vqpi->rxq.databuffer_size = max_len;
                vqpi++;

Thanks,
Olek

author maintainer
#29 lore raw

Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 10:28:06 -0700

Quoted 52 lines
Quoted 48 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700

Quoted 15 lines
Quoted 13 lines · Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.

Quoted 7 lines
Quoted 1 line

One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.

min_mtu and watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s

[...]

Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737

Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -6
-       bi_size = sizeof(struct iavf_rx_buffer) * rx_ring->count;
-       memset(rx_ring->rx_bi, 0, bi_size);
-
-       /* Zero out the descriptor ring */
-       memset(rx_ring->desc, 0, rx_ring->size);
-
Quoted 7 lines

I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things in setup_rx_descriptors.

Quoted 1 line

Both arrays will be freed couple instructions below, why zero them?

Quoted 8 lines

Ugh. You are right, but not for a good reason. So the other Intel drivers in the past would be doing the clean_rx_ring calls on the _down() with the freeing of resources on _close(). Specifically it allowed reducing the overhead for things like resets or setting changes since it didn't require reallocating the descriptor rings and buffer info structures.

I guess you are good to remove these since this code doesn't do that.

We might go back to this to not always do a full circle when not needed, but currently this is redundant.

Quoted 46 lines
Quoted 41 lines
Quoted 6 lines
Quoted 6 lines


        rx_ring->next_to_clean = 0;
        rx_ring->next_to_use = 0;

}

[...]

Quoted 12 lines
Quoted 12 lines

struct net_device *netdev; /* netdev ring maps to */
union {

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -2
+               struct libie_rx_buffer *rx_bi;
                struct iavf_tx_buffer *tx_bi;
-               struct iavf_rx_buffer *rx_bi;
        };
        DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
+       u8 __iomem *tail;
        u16 queue_index;                /* Queue number of ring */
        u8 dcb_tc;                      /* Traffic class of ring */
-       u8 __iomem *tail;
Quoted 4 lines
Quoted 2 lines

/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.

I'm assuming "tail" was moved here since it is a pointer and fills a hole?

(see above)

Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
@@ -329,9 +264,8 @@ struct iavf_ring {
         */
        u16 itr_setting;
 
Quoted 7 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268

Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
-       u16 count;                      /* Number of descriptors */
        u16 reg_idx;                    /* HW register index of the ring */
-       u16 rx_buf_len;
+       u16 count;                      /* Number of descriptors */

Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.

Quoted 4 lines

With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.

Quoted 8 lines

The performance impact should be minimal. Odds are the placement was the way it was since it was probably just copying the original code that has been there since igb/ixgbe. The general idea is just keep the read-mostly items grouped at the top and try to order them somewhat by frequency of being read so that wherever the cache line ends up you won't take much of a penalty as hopefully you will just have the infrequently read items end up getting pulled into the active cache line.

+

Quoted 30 lines
Quoted 25 lines
Quoted 8 lines
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -0
        /* used in interrupt processing */
        u16 next_to_use;
@@ -398,17 +332,6 @@ struct iavf_ring_container {
 #define iavf_for_each_ring(pos, head) \
        for (pos = (head).ring; pos != NULL; pos = pos->next)
 
Quoted 16 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411

Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -11
-static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
-{
-#if (PAGE_SIZE < 8192)
-       if (ring->rx_buf_len > (PAGE_SIZE / 2))
-               return 1;
-#endif
-       return 0;
-}
-
-#define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
-

All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.

Quoted 4 lines

This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.

Quoted 5 lines

It all depends on how much is having to be added to accommodate this. In my mind when we did away with the page splitting/recycling we also did away with the need for the higher order pages. That is why I was thinking it might make more sense there as it would just be more removals with very few if any additions needed to support it.

Yeah, I'll try and see whether any pieces can be grouped differently for better reading/logics.

[...]

Thanks!
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#30 lore raw

RFC PATCH v4 7/9 libie: add common queue stats

In reply to Alexander Lobakin

Next stop, per-queue private stats. They have only subtle differences from driver to driver and can easily be resolved.
Define common structures, inline helpers and Ethtool helpers to collect, update and export the statistics. Use u64_stats_t right from the start, as well as the corresponding helpers to ensure tear-free operations. For the NAPI parts of both Rx and Tx, also define small onstack containers to update them in polling loops and then sync the actual containers once a loop ends.
The drivers will be switched to use this API later on a per-driver basis, along with conversion to PP.

---
 drivers/net/ethernet/intel/libie/Makefile |   1 +
 drivers/net/ethernet/intel/libie/stats.c  | 119 ++++++++++++++
 include/linux/net/intel/libie/stats.h     | 179 ++++++++++++++++++++++
 3 files changed, 299 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/stats.h
drivers/net/ethernet/intel/libie/Makefile +1 -0
index 95e81d09b474..76f32253481b 100644
@@ -4,3 +4,4 @@
 obj-$(CONFIG_LIBIE)	+= libie.o
 
 libie-objs		+= rx.o
+libie-objs		+= stats.o
drivers/net/ethernet/intel/libie/stats.c new file +119 -0

119 lines not on this page · raw message

include/linux/net/intel/libie/stats.h new file +179 -0

179 lines not on this page · raw message

Signature · 6 lines
2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#31 lore raw

RFC PATCH v4 8/9 libie: add per-queue Page Pool stats

In reply to Alexander Lobakin

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.
They are also a bit special in terms of how they are obtained. One &page_pool accumulates statistics until it's destroyed obviously, which happens on ifdown. So, in order to not lose any statistics, get the stats and store in the queue container before destroying a pool. This container survives ifups/downs, so it basically stores the statistics accumulated since the very first pool was allocated on this queue. When it's needed to export the stats, first get the numbers from this container and then add the "live" numbers -- the ones that the current active pool returns. The result values will always represent the actual device-lifetime* stats.
There's a cast from &page_pool_stats to u64 * in a couple functions, but they are guarded with stats asserts to make sure it's safe to do. FWIW it saves a lot of object code.

---
 drivers/net/ethernet/intel/libie/internal.h | 23 +++++++
 drivers/net/ethernet/intel/libie/rx.c       | 19 ++++++
 drivers/net/ethernet/intel/libie/stats.c    | 73 ++++++++++++++++++++-
 include/linux/net/intel/libie/rx.h          |  4 ++
 include/linux/net/intel/libie/stats.h       | 39 ++++++++++-
 5 files changed, 155 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
drivers/net/ethernet/intel/libie/internal.h new file +23 -0
new file mode 100644
index 000000000000..083398dc37c6
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* libie internal declarations not to be used in drivers.
+ *
+ * Copyright(c) 2023 Intel Corporation.
+ */
+
+#ifndef __LIBIE_INTERNAL_H
+#define __LIBIE_INTERNAL_H
+
+struct libie_rq_stats;
+struct page_pool;
+
+#ifdef CONFIG_PAGE_POOL_STATS
+void libie_rq_stats_sync_pp(struct libie_rq_stats *stats,
+			    struct page_pool *pool);
+#else
+static inline void libie_rq_stats_sync_pp(struct libie_rq_stats *stats,
+					  struct page_pool *pool)
+{
+}
+#endif
+
+#endif /* __LIBIE_INTERNAL_H */
drivers/net/ethernet/intel/libie/rx.c +19 -0
index c60d7b20ed20..0c26bd066587 100644
@@ -2,6 +2,7 @@
 /* Copyright(c) 2023 Intel Corporation. */
 
 #include <linux/net/intel/libie/rx.h>
+#include "internal.h"
 
 /* Rx buffer management */
 
@@ -57,6 +58,24 @@ struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
 }
 EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);
 
+/**
+ * libie_rx_page_pool_destroy - destroy a &page_pool created by libie
+ * @pool: pool to destroy
+ * @stats: RQ stats from the ring (or %NULL to skip updating PP stats)
+ *
+ * As the stats usually has the same lifetime as the device, but PP is usually
+ * created/destroyed on ifup/ifdown, in order to not lose the stats accumulated
+ * during the last ifup, the PP stats need to be added to the driver stats
+ * container. Then the PP gets destroyed.
+ */
+void libie_rx_page_pool_destroy(struct page_pool *pool,
+				struct libie_rq_stats *stats)
+{
+	libie_rq_stats_sync_pp(stats, pool);
+	page_pool_destroy(pool);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_destroy, LIBIE);
+
 /* O(1) converting i40e/ice/iavf's 8/10-bit hardware packet type to a parsed
  * bitfield struct.
  */
drivers/net/ethernet/intel/libie/stats.c +72 -1

98 lines not on this page · raw message

include/linux/net/intel/libie/rx.h +4 -0
index 8c0ccdff9a37..c6c85f956f95 100644
@@ -62,8 +62,12 @@ struct libie_rx_buffer {
 	u32		truesize;
 };
 
+struct libie_rq_stats;
+
 struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
 					    u32 size);
+void libie_rx_page_pool_destroy(struct page_pool *pool,
+				struct libie_rq_stats *stats);
 
 /**
  * libie_rx_alloc - allocate a new Rx buffer
include/linux/net/intel/libie/stats.h +37 -2
index dbbc98bbd3a7..23ca0079a905 100644
@@ -49,6 +49,17 @@
  * fragments: number of processed descriptors carrying only a fragment
  * alloc_page_fail: number of Rx page allocation fails
  * build_skb_fail: number of build_skb() fails
+ * pp_alloc_fast: pages taken from the cache or ring
+ * pp_alloc_slow: actual page allocations
+ * pp_alloc_slow_ho: non-order-0 page allocations
+ * pp_alloc_empty: number of times the pool was empty
+ * pp_alloc_refill: number of cache refills
+ * pp_alloc_waive: NUMA node mismatches during recycling
+ * pp_recycle_cached: direct recyclings into the cache
+ * pp_recycle_cache_full: number of times the cache was full
+ * pp_recycle_ring: recyclings into the ring
+ * pp_recycle_ring_full: number of times the ring was full
+ * pp_recycle_released_ref: pages released due to elevated refcnt
  */
 
 #define DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
@@ -60,9 +71,29 @@
 	act(alloc_page_fail)				\
 	act(build_skb_fail)
 
+#ifdef CONFIG_PAGE_POOL_STATS
+#define DECLARE_LIBIE_RQ_PP_STATS(act)			\
+	act(pp_alloc_fast)				\
+	act(pp_alloc_slow)				\
+	act(pp_alloc_slow_ho)				\
+	act(pp_alloc_empty)				\
+	act(pp_alloc_refill)				\
+	act(pp_alloc_waive)				\
+	act(pp_recycle_cached)				\
+	act(pp_recycle_cache_full)			\
+	act(pp_recycle_ring)				\
+	act(pp_recycle_ring_full)			\
+	act(pp_recycle_released_ref)
+#else
+#define DECLARE_LIBIE_RQ_PP_STATS(act)
+#endif
+
 #define DECLARE_LIBIE_RQ_STATS(act)			\
 	DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
-	DECLARE_LIBIE_RQ_FAIL_STATS(act)
+	DECLARE_LIBIE_RQ_FAIL_STATS(act)		\
+	DECLARE_LIBIE_RQ_PP_STATS(act)
+
+struct page_pool;
 
 struct libie_rq_stats {
 	struct u64_stats_sync	syncp;
@@ -72,6 +103,9 @@ struct libie_rq_stats {
 #define act(s)	u64_stats_t	s;
 			DECLARE_LIBIE_RQ_NAPI_STATS(act);
 			DECLARE_LIBIE_RQ_FAIL_STATS(act);
+			struct_group(pp,
+				DECLARE_LIBIE_RQ_PP_STATS(act);
+			);
 #undef act
 		};
 		DECLARE_FLEX_ARRAY(u64_stats_t, raw);
@@ -110,7 +144,8 @@ libie_rq_napi_stats_add(struct libie_rq_stats *qs,
 
 u32 libie_rq_stats_get_sset_count(void);
 void libie_rq_stats_get_strings(u8 **data, u32 qid);
-void libie_rq_stats_get_data(u64 **data, const struct libie_rq_stats *stats);
+void libie_rq_stats_get_data(u64 **data, const struct libie_rq_stats *stats,
+			     struct page_pool *pool);
 
 /* Tx per-queue stats:
  * packets: packets sent from this queue
Signature · 6 lines
2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#32 lore raw

RFC PATCH v4 9/9 iavf: switch queue stats to libie

In reply to Alexander Lobakin

iavf is pretty much ready for using the generic libie stats, so drop all the custom code and just use generic definitions. The only thing is that it previously lacked the counter of Tx queue stops. It's present in the other drivers, so add it here as well.
The rest is straightforward. There were two fields in the Tx stats struct, which didn't belong there. The first one has never been used, wipe it; and move the other to the queue structure. Plus move around a couple fields in &iavf_ring to account stats structs' alignment.

---
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 86 ++++-------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  2 +
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 92 ++++++++++---------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 54 +++--------
 4 files changed, 83 insertions(+), 151 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +19 -67
index de3050c02b6f..cdf09c5412a0 100644
@@ -46,16 +46,6 @@ struct iavf_stats {
 	.stat_offset = offsetof(_type, _stat) \
 }
 
-/* Helper macro for defining some statistics related to queues */
-#define IAVF_QUEUE_STAT(_name, _stat) \
-	IAVF_STAT(struct iavf_ring, _name, _stat)
-
-/* Stats associated with a Tx or Rx ring */
-static const struct iavf_stats iavf_gstrings_queue_stats[] = {
-	IAVF_QUEUE_STAT("%s-%u.packets", stats.packets),
-	IAVF_QUEUE_STAT("%s-%u.bytes", stats.bytes),
-};
-
 /**
  * iavf_add_one_ethtool_stat - copy the stat into the supplied buffer
  * @data: location to store the stat value
@@ -141,43 +131,6 @@ __iavf_add_ethtool_stats(u64 **data, void *pointer,
 #define iavf_add_ethtool_stats(data, pointer, stats) \
 	__iavf_add_ethtool_stats(data, pointer, stats, ARRAY_SIZE(stats))
 
-/**
- * iavf_add_queue_stats - copy queue statistics into supplied buffer
- * @data: ethtool stats buffer
- * @ring: the ring to copy
- *
- * Queue statistics must be copied while protected by
- * u64_stats_fetch_begin, so we can't directly use iavf_add_ethtool_stats.
- * Assumes that queue stats are defined in iavf_gstrings_queue_stats. If the
- * ring pointer is null, zero out the queue stat values and update the data
- * pointer. Otherwise safely copy the stats from the ring into the supplied
- * buffer and update the data pointer when finished.
- *
- * This function expects to be called while under rcu_read_lock().
- **/
-static void
-iavf_add_queue_stats(u64 **data, struct iavf_ring *ring)
-{
-	const unsigned int size = ARRAY_SIZE(iavf_gstrings_queue_stats);
-	const struct iavf_stats *stats = iavf_gstrings_queue_stats;
-	unsigned int start;
-	unsigned int i;
-
-	/* To avoid invalid statistics values, ensure that we keep retrying
-	 * the copy until we get a consistent value according to
-	 * u64_stats_fetch_retry. But first, make sure our ring is
-	 * non-null before attempting to access its syncp.
-	 */
-	do {
-		start = !ring ? 0 : u64_stats_fetch_begin(&ring->syncp);
-		for (i = 0; i < size; i++)
-			iavf_add_one_ethtool_stat(&(*data)[i], ring, &stats[i]);
-	} while (ring && u64_stats_fetch_retry(&ring->syncp, start));
-
-	/* Once we successfully copy the stats in, update the data pointer */
-	*data += size;
-}
-
 /**
  * __iavf_add_stat_strings - copy stat strings into ethtool buffer
  * @p: ethtool supplied buffer
@@ -237,8 +190,6 @@ static const struct iavf_stats iavf_gstrings_stats[] = {
 
 #define IAVF_STATS_LEN	ARRAY_SIZE(iavf_gstrings_stats)
 
-#define IAVF_QUEUE_STATS_LEN	ARRAY_SIZE(iavf_gstrings_queue_stats)
-
 /**
  * iavf_get_link_ksettings - Get Link Speed and Duplex settings
  * @netdev: network interface device structure
@@ -308,18 +259,22 @@ static int iavf_get_link_ksettings(struct net_device *netdev,
  **/
 static int iavf_get_sset_count(struct net_device *netdev, int sset)
 {
-	/* Report the maximum number queues, even if not every queue is
-	 * currently configured. Since allocation of queues is in pairs,
-	 * use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set
-	 * at device creation and never changes.
-	 */
+	u32 num;
 
-	if (sset == ETH_SS_STATS)
-		return IAVF_STATS_LEN +
-			(IAVF_QUEUE_STATS_LEN * 2 *
-			 netdev->real_num_tx_queues);
-	else
+	switch (sset) {
+	case ETH_SS_STATS:
+		/* Per-queue */
+		num = libie_rq_stats_get_sset_count();
+		num += libie_sq_stats_get_sset_count();
+		num *= netdev->real_num_tx_queues;
+
+		/* Global */
+		num += IAVF_STATS_LEN;
+
+		return num;
+	default:
 		return -EINVAL;
+	}
 }
 
 /**
@@ -346,15 +301,14 @@ static void iavf_get_ethtool_stats(struct net_device *netdev,
 	 * it to iterate over rings' stats.
 	 */
 	for (i = 0; i < adapter->num_active_queues; i++) {
-		struct iavf_ring *ring;
+		const struct iavf_ring *ring;
 
 		/* Tx rings stats */
-		ring = &adapter->tx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		libie_sq_stats_get_data(&data, &adapter->tx_rings[i].sq_stats);
 
 		/* Rx rings stats */
 		ring = &adapter->rx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		libie_rq_stats_get_data(&data, &ring->rq_stats, ring->pp);
 	}
 	rcu_read_unlock();
 }
@@ -376,10 +330,8 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data)
 	 * real_num_tx_queues for both Tx and Rx queues.
 	 */
 	for (i = 0; i < netdev->real_num_tx_queues; i++) {
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "tx", i);
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "rx", i);
+		libie_sq_stats_get_strings(&data, i);
+		libie_rq_stats_get_strings(&data, i);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_main.c +2 -0
index 39c6d83e80a1..659bdfc33e0b 100644
@@ -1566,12 +1566,14 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		tx_ring->itr_setting = IAVF_ITR_TX_DEF;
 		if (adapter->flags & IAVF_FLAG_WB_ON_ITR_CAPABLE)
 			tx_ring->flags |= IAVF_TXR_FLAGS_WB_ON_ITR;
+		u64_stats_init(&tx_ring->sq_stats.syncp);
 
 		rx_ring = &adapter->rx_rings[i];
 		rx_ring->queue_index = i;
 		rx_ring->netdev = adapter->netdev;
 		rx_ring->count = adapter->rx_desc_count;
 		rx_ring->itr_setting = IAVF_ITR_RX_DEF;
+		u64_stats_init(&rx_ring->rq_stats.syncp);
 	}
 
 	adapter->num_active_queues = num_active_queues;
drivers/net/ethernet/intel/iavf/iavf_txrx.c +48 -44

257 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +14 -40

87 lines not on this page · raw message

Signature · 6 lines
2.41.0

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan