Skip to contentSkip to conversation

net: intel: start The Great Code Dedup + Page Pool for iavf

PATCH Superseded Alexander Lobakin 5 participants 30 messages May 25, 2023 – Jun 6, 2023

Subsystems (10)

v2 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Alexander Lobakin

author maintainer
#0 lore raw

v2 cover net: intel: start The Great Code Dedup + Page Pool for iavf

Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details in the commit messages; here's summary:

Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers.
The first thing that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" and can be expanded as "lib Internet Explorer" :P I'm open for anything else (but justified). The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks: "can I share this?".
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.

Alexander Lobakin (12):
net: intel: introduce Intel Ethernet common library
iavf: kill "legacy-rx" for good
iavf: optimize Rx buffer allocation a bunch
iavf: remove page splitting/recycling
iavf: always use a full order-0 page
net: skbuff: don't include <net/page_pool.h> into <linux/skbuff.h>
net: page_pool: avoid calling no-op externals when possible
net: page_pool: add DMA-sync-for-CPU inline helpers
iavf: switch to Page Pool
libie: add common queue stats
libie: add per-queue Page Pool stats
iavf: switch queue stats to libie

 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/engleder/tsnep_main.c    |   1 +
 drivers/net/ethernet/freescale/fec_main.c     |   1 +
 drivers/net/ethernet/intel/Kconfig            |  12 +-
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 -------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  74 +-
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 ---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 -------
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 227 +-----
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  45 +-
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 715 +++++-------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 185 +----
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  90 ---
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  16 +-
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 --------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +-
 drivers/net/ethernet/intel/libie/Makefile     |   7 +
 drivers/net/ethernet/intel/libie/internal.h   |  23 +
 drivers/net/ethernet/intel/libie/rx.c         | 158 ++++
 drivers/net/ethernet/intel/libie/stats.c      | 190 +++++
 .../ethernet/mellanox/mlx5/core/en/params.c   |   1 +
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  |   1 +
 drivers/net/wireless/mediatek/mt76/mt76.h     |   1 +
 include/linux/net/intel/libie/rx.h            | 170 +++++
 include/linux/net/intel/libie/stats.h         | 214 ++++++
 include/linux/skbuff.h                        |   4 +-
 include/net/page_pool.h                       |  69 +-
 net/core/page_pool.c                          |  10 +
 34 files changed, 1139 insertions(+), 2081 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/rx.h
 create mode 100644 include/linux/net/intel/libie/stats.h

---
Directly to net-next, has non-Intel code changes (0006-0008).

From v1[0]:
* 0006: new (me, Jakub);
* 0008: give the helpers more intuitive names (Jakub, Ilias);
* -^-: also expand their kdoc a bit for the same reason;
* -^-: fix kdoc copy-paste issue (Patchwork, Jakub);
* 0011: drop inline from C file (Patchwork, Jakub).

[0] https://lore.kernel.org/netdev/[email protected]

Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#1 lore raw

PATCH v2 1/12 net: intel: introduce Intel Ethernet common library

Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, add unlikely() condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as a pkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. But skb_set_hash() only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hidden Kconfig symbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.

---
 MAINTAINERS                                   |   3 +-
 drivers/net/ethernet/intel/Kconfig            |  11 +-
 drivers/net/ethernet/intel/Makefile           |   1 +
 drivers/net/ethernet/intel/i40e/i40e_common.c | 253 --------------
 drivers/net/ethernet/intel/i40e/i40e_main.c   |   1 +
 .../net/ethernet/intel/i40e/i40e_prototype.h  |   7 -
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   |  74 +---
 drivers/net/ethernet/intel/i40e/i40e_type.h   |  88 -----
 drivers/net/ethernet/intel/iavf/iavf_common.c | 253 --------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |   1 +
 .../net/ethernet/intel/iavf/iavf_prototype.h  |   7 -
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  70 +---
 drivers/net/ethernet/intel/iavf/iavf_type.h   |  88 -----
 .../net/ethernet/intel/ice/ice_lan_tx_rx.h    | 316 ------------------
 drivers/net/ethernet/intel/ice/ice_main.c     |   1 +
 drivers/net/ethernet/intel/ice/ice_txrx_lib.c |  74 +---
 drivers/net/ethernet/intel/libie/Makefile     |   6 +
 drivers/net/ethernet/intel/libie/rx.c         | 110 ++++++
 include/linux/net/intel/libie/rx.h            | 128 +++++++
 19 files changed, 312 insertions(+), 1180 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/Makefile
 create mode 100644 drivers/net/ethernet/intel/libie/rx.c
 create mode 100644 include/linux/net/intel/libie/rx.h
MAINTAINERS +2 -1
index c25172d6471a..709399ba39a3 100644
@@ -10345,7 +10345,8 @@ F:	Documentation/networking/device_drivers/ethernet/intel/
 F:	drivers/net/ethernet/intel/
 F:	drivers/net/ethernet/intel/*/
 F:	include/linux/avf/virtchnl.h
-F:	include/linux/net/intel/iidc.h
+F:	include/linux/net/intel/
+F:	include/linux/net/intel/*/
 
 INTEL ETHERNET PROTOCOL DRIVER FOR RDMA
 M:	Mustafa Ismail <[email protected]>
drivers/net/ethernet/intel/Kconfig +10 -1
index 9bc0a9519899..cec4a938fbd0 100644
@@ -84,6 +84,12 @@ config E1000E_HWTS
 	 devices. The cross-timestamp is available through the PTP clock
 	 driver precise cross-timestamp ioctl (PTP_SYS_OFFSET_PRECISE).
 
+config LIBIE
+	tristate
+	help
+	  libie (Intel Ethernet library) is a common library containing
+	  routines shared by several Intel Ethernet drivers.
+
 config IGB
 	tristate "Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support"
 	depends on PCI
@@ -225,6 +231,7 @@ config I40E
 	depends on PTP_1588_CLOCK_OPTIONAL
 	depends on PCI
 	select AUXILIARY_BUS
+	select LIBIE
 	help
 	  This driver supports Intel(R) Ethernet Controller XL710 Family of
 	  devices.  For more information on how to identify your adapter, go
@@ -254,8 +261,9 @@ config IAVF
 	tristate
 config I40EVF
 	tristate "Intel(R) Ethernet Adaptive Virtual Function support"
-	select IAVF
 	depends on PCI_MSI
+	select IAVF
+	select LIBIE
 	help
 	  This driver supports virtual functions for Intel XL710,
 	  X710, X722, XXV710, and all devices advertising support for
@@ -282,6 +290,7 @@ config ICE
 	depends on GNSS || GNSS = n
 	select AUXILIARY_BUS
 	select DIMLIB
+	select LIBIE
 	select NET_DEVLINK
 	select PLDMFW
 	help
drivers/net/ethernet/intel/Makefile +1 -0
index d80d04132073..ce622b4d825d 100644
@@ -15,3 +15,4 @@ obj-$(CONFIG_I40E) += i40e/
 obj-$(CONFIG_IAVF) += iavf/
 obj-$(CONFIG_FM10K) += fm10k/
 obj-$(CONFIG_ICE) += ice/
+obj-$(CONFIG_LIBIE) += libie/
drivers/net/ethernet/intel/i40e/i40e_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_main.c +1 -0
index b847bd105b16..f3bca6e9bd18 100644
@@ -97,6 +97,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Connection XL710 Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static struct workqueue_struct *i40e_wq;
drivers/net/ethernet/intel/i40e/i40e_prototype.h +0 -7
index fe845987d99a..5287d0ef32d5 100644
@@ -380,13 +380,6 @@ void i40e_set_pci_config_data(struct i40e_hw *hw, u16 link_status);
 
 int i40e_set_mac_type(struct i40e_hw *hw);
 
-extern struct i40e_rx_ptype_decoded i40e_ptype_lookup[];
-
-static inline struct i40e_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return i40e_ptype_lookup[ptype];
-}
-
 /**
  * i40e_virtchnl_link_speed - Convert AdminQ link_speed to virtchnl definition
  * @link_speed: the speed to convert
drivers/net/ethernet/intel/i40e/i40e_txrx.c +18 -56

132 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_common.c +0 -253

259 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -0
index 2de4baff4c20..c17e909d3ff0 100644
@@ -46,6 +46,7 @@ MODULE_DEVICE_TABLE(pci, iavf_pci_tbl);
 MODULE_ALIAS("i40evf");
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION("Intel(R) Ethernet Adaptive Virtual Function Network Driver");
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 
 static const struct net_device_ops iavf_netdev_ops;
drivers/net/ethernet/intel/iavf/iavf_prototype.h +0 -7
index edebfbbcffdc..c2e5dbc0a75a 100644
@@ -51,13 +51,6 @@ enum iavf_status iavf_aq_set_rss_key(struct iavf_hw *hw, u16 seid,
 
 enum iavf_status iavf_set_mac_type(struct iavf_hw *hw);
 
-extern struct iavf_rx_ptype_decoded iavf_ptype_lookup[];
-
-static inline struct iavf_rx_ptype_decoded decode_rx_desc_ptype(u8 ptype)
-{
-	return iavf_ptype_lookup[ptype];
-}
-
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
 enum iavf_status iavf_vf_reset(struct iavf_hw *hw);
drivers/net/ethernet/intel/iavf/iavf_txrx.c +16 -54

125 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -88

94 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h +0 -316

326 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_main.c +1 -0
index 62e91512aeab..2f039f48d7a0 100644
@@ -34,6 +34,7 @@ static const char ice_copyright[] = "Copyright (c) 2018, Intel Corporation.";
 
 MODULE_AUTHOR("Intel Corporation, <[email protected]>");
 MODULE_DESCRIPTION(DRV_SUMMARY);
+MODULE_IMPORT_NS(LIBIE);
 MODULE_LICENSE("GPL v2");
 MODULE_FIRMWARE(ICE_DDP_PKG_FILE);
 
drivers/net/ethernet/intel/ice/ice_txrx_lib.c +18 -56

137 lines not on this page · raw message

drivers/net/ethernet/intel/libie/Makefile new file +6 -0
new file mode 100644
index 000000000000..95e81d09b474
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright(c) 2023 Intel Corporation.
+
+obj-$(CONFIG_LIBIE)	+= libie.o
+
+libie-objs		+= rx.o
drivers/net/ethernet/intel/libie/rx.c new file +110 -0

110 lines not on this page · raw message

include/linux/net/intel/libie/rx.h new file +128 -0

128 lines not on this page · raw message

Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#2 lore raw

PATCH v2 2/12 iavf: kill "legacy-rx" for good

In reply to Alexander Lobakin

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.

* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.

Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.

There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft and so on. Some stats:

Function: add/remove: 0/2 grow/shrink: 0/7 up/down: 0/-774 (-774) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)

---
 drivers/net/ethernet/intel/iavf/iavf.h        |   2 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 140 ------------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  10 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  84 +----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   |  18 +--
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   3 +-
 6 files changed, 8 insertions(+), 249 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
index 9abaff1f2aff..a780e7aa1c2f 100644
@@ -298,7 +298,7 @@ struct iavf_adapter {
 #define IAVF_FLAG_CLIENT_NEEDS_L2_PARAMS	BIT(12)
 #define IAVF_FLAG_PROMISC_ON			BIT(13)
 #define IAVF_FLAG_ALLMULTI_ON			BIT(14)
-#define IAVF_FLAG_LEGACY_RX			BIT(15)
+/* BIT(15) is free, was IAVF_FLAG_LEGACY_RX */
 #define IAVF_FLAG_REINIT_ITR_NEEDED		BIT(16)
 #define IAVF_FLAG_QUEUES_DISABLED		BIT(17)
 #define IAVF_FLAG_SETUP_NETDEV_FEATURES		BIT(18)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +0 -140

181 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_main.c +1 -9
index c17e909d3ff0..a5a6c9861a93 100644
@@ -713,9 +713,7 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 	struct iavf_hw *hw = &adapter->hw;
 	int i;
 
-	/* Legacy Rx will always default to a 2048 buffer size. */
-#if (PAGE_SIZE < 8192)
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX)) {
+	if (PAGE_SIZE < 8192) {
 		struct net_device *netdev = adapter->netdev;
 
 		/* For jumbo frames on systems with 4K pages we have to use
@@ -732,16 +730,10 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)
 		    (netdev->mtu <= ETH_DATA_LEN))
 			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 	}
-#endif
 
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
 		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-
-		if (adapter->flags & IAVF_FLAG_LEGACY_RX)
-			clear_ring_build_skb_enabled(&adapter->rx_rings[i]);
-		else
-			set_ring_build_skb_enabled(&adapter->rx_rings[i]);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -81

115 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -16
index 2624bf6d009e..234e189c1987 100644
@@ -362,7 +362,8 @@ struct iavf_ring {
 
 	u16 flags;
 #define IAVF_TXR_FLAGS_WB_ON_ITR		BIT(0)
-#define IAVF_RXR_FLAGS_BUILD_SKB_ENABLED	BIT(1)
+/* BIT(1) is free, was IAVF_RXR_FLAGS_BUILD_SKB_ENABLED */
+/* BIT(2) is free */
 #define IAVF_TXRX_FLAGS_VLAN_TAG_LOC_L2TAG1	BIT(3)
 #define IAVF_TXR_FLAGS_VLAN_TAG_LOC_L2TAG2	BIT(4)
 #define IAVF_RXR_FLAGS_VLAN_TAG_LOC_L2TAG2_2	BIT(5)
@@ -393,21 +394,6 @@ struct iavf_ring {
 					 */
 } ____cacheline_internodealigned_in_smp;
 
-static inline bool ring_uses_build_skb(struct iavf_ring *ring)
-{
-	return !!(ring->flags & IAVF_RXR_FLAGS_BUILD_SKB_ENABLED);
-}
-
-static inline void set_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags |= IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
-static inline void clear_ring_build_skb_enabled(struct iavf_ring *ring)
-{
-	ring->flags &= ~IAVF_RXR_FLAGS_BUILD_SKB_ENABLED;
-}
-
 #define IAVF_ITR_ADAPTIVE_MIN_INC	0x0002
 #define IAVF_ITR_ADAPTIVE_MIN_USECS	0x0002
 #define IAVF_ITR_ADAPTIVE_MAX_USECS	0x007e
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -2
index 7c0578b5457b..fdddc3588487 100644
@@ -290,8 +290,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		return;
 
 	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (!(adapter->flags & IAVF_FLAG_LEGACY_RX) &&
-	    (adapter->netdev->mtu <= ETH_DATA_LEN))
+	if (adapter->netdev->mtu <= ETH_DATA_LEN)
 		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
 
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Reviewed-by Alexander Duyck on v2 2/12
#3 lore raw

Re: [PATCH net-next v2 02/12] iavf: kill "legacy-rx" for good

Quoted 14 lines · Alexander Lobakin

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.

* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.

Quoted 7 lines

Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.

The "legacy-rx" was never about performance. It was mostly about providing a fall back in the event of an unexpected behavior. Keep in mind that in order to enable this we are leaving the page mapped and syncing it multiple times. In order to enable support for this we had to add several new items that I had deemed to be a bit risky such as support for DMA pages that were synced by the driver instead of on map/unmap and the use of the build_skb logic.

My main concern was that if we ever ran into header corruption we could switch this on and then the pages would only be writable by the device.

Quoted 9 lines

There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft and so on. Some stats:

Quoted 4 lines

Function: add/remove: 0/2 grow/shrink: 0/7 up/down: 0/-774 (-774) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)

I support this 100%. The legacy-rx flag was meant as more of a fall- back in the event that the build_skb code wasn't present or was broken in some way. It wasn't really meant to be carried forward into drivers as the last one I had added this to was i40e over 6 years ago.

Since it has been about 6 years without any issues I would say we are safe to remove it.

author maintainer
#4 lore raw

Re: [PATCH net-next v2 02/12] iavf: kill "legacy-rx" for good

From: Alexander H Duyck <[email protected]>
Date: Tue, 30 May 2023 08:29:10 -0700

Quoted 4 lines · Alexander Lobakin
Quoted 3 lines

Ever since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.

[...]

Quoted 7 lines

The "legacy-rx" was never about performance. It was mostly about providing a fall back in the event of an unexpected behavior. Keep in mind that in order to enable this we are leaving the page mapped and syncing it multiple times. In order to enable support for this we had to add several new items that I had deemed to be a bit risky such as support for DMA pages that were synced by the driver instead of on map/unmap and the use of the build_skb logic.

And now pretty much every driver doesn't do map/unmap each time and keeps its mapping for a very long time. And now Chris noticed that and says it's not supposed to be used like that :D

Quoted 4 lines

My main concern was that if we ever ran into header corruption we could switch this on and then the pages would only be writable by the device.

[...]

Quoted 4 lines

Since it has been about 6 years without any issues I would say we are safe to remove it.

Thanks!
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#5 lore raw

PATCH v2 3/12 iavf: optimize Rx buffer allocation a bunch

In reply to Alexander Lobakin

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.
2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.
3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.
4. 1 budget unit == 1 descriptor, not skb.
There could be underflow when receiving a lot of fragmented frames. If each of them would consist of 2 frags, it means that we'd process 64 descriptors at the point where we pass the 32th skb to the stack. But the driver would count that only as a half, which could make NAPI re-enable interrupts prematurely and create unnecessary CPU load.
5. Shortcut !size case.
It's super rare, but possible -- for example, if the last buffer of the fragmented frame contained only FCS, which was then stripped by the HW. Instead of checking for size several times when processing, quickly reuse the buffer and jump to the skb fields part.
6. Refill the ring after finishing the polling loop. Previously, the loop wasn't starting a new iteration after the 64th desc, meaning that we were always leaving 16 buffers non-refilled until the next NAPI poll. It's better to refill them while they're still hot, so do that right after exiting the loop as well. For a full cycle of 64 descs, there will be 4 refills of 16 descs from now on.

Function: add/remove: 4/2 grow/shrink: 0/5 up/down: 473/-647 (-174)

+ up to 2% performance.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 259 +++++++++-----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   3 +-
 3 files changed, 114 insertions(+), 150 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -1
index a5a6c9861a93..ade32aa1ed78 100644
@@ -1236,7 +1236,7 @@ static void iavf_configure(struct iavf_adapter *adapter)
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		struct iavf_ring *ring = &adapter->rx_rings[i];
 
-		iavf_alloc_rx_buffers(ring, IAVF_DESC_UNUSED(ring));
+		iavf_alloc_rx_buffers(ring);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +112 -147

464 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -2
index 234e189c1987..9c6661a6edf2 100644
@@ -383,7 +383,6 @@ struct iavf_ring {
 	struct iavf_q_vector *q_vector;	/* Backreference to associated vector */
 
 	struct rcu_head rcu;		/* to avoid race on free */
-	u16 next_to_alloc;
 	struct sk_buff *skb;		/* When iavf_clean_rx_ring_irq() must
 					 * return before it sees the EOP for
 					 * the current packet, we save that skb
@@ -426,7 +425,7 @@ static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
 
 #define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
 
-bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
+void iavf_alloc_rx_buffers(struct iavf_ring *rxr);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 void iavf_clean_tx_ring(struct iavf_ring *tx_ring);
 void iavf_clean_rx_ring(struct iavf_ring *rx_ring);
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#6 lore raw

Re: [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

Quoted 10 lines · Alexander Lobakin

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.

The advantage of this is going to vary based on the attribute. One of the reasons why I left most of this on the ring is because the section of the ring most of these variables were meant to be read-mostly and shouldn't have resulted in any additional overhead versus accessing them from the stack.

Quoted 10 lines

2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.

The next_to_alloc logic was put in place to deal with systems that are experiencing memory issues. Specifically what can end up happening is that the ring can stall due to failing memory allocations and the memory can get stuck on the ring. For that reason we were essentially defragmenting the buffers when we started suffering memory pressure so that they could be reusued and/or freed following immediate use.

Basically what you are trading off is some exception handling for performance by removing it.

Quoted 3 lines

3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Quoted 6 lines

4. 1 budget unit == 1 descriptor, not skb.
There could be underflow when receiving a lot of fragmented frames. If each of them would consist of 2 frags, it means that we'd process 64 descriptors at the point where we pass the 32th skb to the stack. But the driver would count that only as a half, which could make NAPI re-enable interrupts prematurely and create unnecessary CPU load.

Not sure I agree with this. The problem is the overhead for an skb going up the stack versus a fragment are pretty signficant. Keep in mind that most of the overhead for a single buffer occurs w/ napi_gro_receive and is not actually at the driver itself. The whole point of the budget is to meter out units of work, not to keep you in the busy loop. This starts looking like the old code where the Intel drivers were returning either budget or 0 instead of supporting the middle ground.

Quoted 13 lines

5. Shortcut !size case.
It's super rare, but possible -- for example, if the last buffer of the fragmented frame contained only FCS, which was then stripped by the HW. Instead of checking for size several times when processing, quickly reuse the buffer and jump to the skb fields part.
6. Refill the ring after finishing the polling loop. Previously, the loop wasn't starting a new iteration after the 64th desc, meaning that we were always leaving 16 buffers non-refilled until the next NAPI poll. It's better to refill them while they're still hot, so do that right after exiting the loop as well. For a full cycle of 64 descs, there will be 4 refills of 16 descs from now on.

Quoted 4 lines

Function: add/remove: 4/2 grow/shrink: 0/5 up/down: 473/-647 (-174)

+ up to 2% performance.

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

Quoted 1 line

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

Quoted 146 lines
---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 259 +++++++++-----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   3 +-
 3 files changed, 114 insertions(+), 150 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -1
index a5a6c9861a93..ade32aa1ed78 100644
@@ -1236,7 +1236,7 @@ static void iavf_configure(struct iavf_adapter *adapter)
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		struct iavf_ring *ring = &adapter->rx_rings[i];
 
-		iavf_alloc_rx_buffers(ring, IAVF_DESC_UNUSED(ring));
+		iavf_alloc_rx_buffers(ring);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +26 -24
index a7121dc5c32b..fd08ce67380e 100644
@@ -736,7 +736,6 @@ void iavf_clean_rx_ring(struct iavf_ring *rx_ring)
 	/* Zero out the descriptor ring */
 	memset(rx_ring->desc, 0, rx_ring->size);
 
-	rx_ring->next_to_alloc = 0;
 	rx_ring->next_to_clean = 0;
 	rx_ring->next_to_use = 0;
 }
@@ -792,7 +791,6 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)
 		goto err;
 	}
 
-	rx_ring->next_to_alloc = 0;
 	rx_ring->next_to_clean = 0;
 	rx_ring->next_to_use = 0;
 
@@ -812,9 +810,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)
 {
 	rx_ring->next_to_use = val;
 
-	/* update next to alloc since we have filled the ring */
-	rx_ring->next_to_alloc = val;
-
 	/* Force memory writes to complete before letting h/w
 	 * know there are new descriptors to fetch.  (Only
 	 * applicable for weak-ordered memory model archs,
@@ -828,12 +823,17 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)
  * iavf_alloc_mapped_page - recycle or make a new page
  * @rx_ring: ring to use
  * @bi: rx_buffer struct to modify
+ * @dev: device used for DMA mapping
+ * @order: page order to allocate
+ * @gfp: GFP mask to allocate page
  *
  * Returns true if the page was successfully allocated or
  * reused.
  **/
 static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
-				   struct iavf_rx_buffer *bi)
+				   struct iavf_rx_buffer *bi,
+				   struct device *dev, u32 order,
+				   gfp_t gfp)
 {
 	struct page *page = bi->page;
 	dma_addr_t dma;
@@ -845,23 +845,21 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,
 	}
 
 	/* alloc new page for storage */
-	page = dev_alloc_pages(iavf_rx_pg_order(rx_ring));
+	page = __dev_alloc_pages(gfp, order);
 	if (unlikely(!page)) {
 		rx_ring->rx_stats.alloc_page_failed++;
 		return false;
 	}
 
 	/* map page for use */
-	dma = dma_map_page_attrs(rx_ring->dev, page, 0,
-				 iavf_rx_pg_size(rx_ring),
-				 DMA_FROM_DEVICE,
-				 IAVF_RX_DMA_ATTR);
+	dma = dma_map_page_attrs(dev, page, 0, PAGE_SIZE << order,
+				 DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
 
 	/* if mapping failed free memory back to system since
 	 * there isn't much point in holding memory we can't use
 	 */
-	if (dma_mapping_error(rx_ring->dev, dma)) {
-		__free_pages(page, iavf_rx_pg_order(rx_ring));
+	if (dma_mapping_error(dev, dma)) {
+		__free_pages(page, order);
 		rx_ring->rx_stats.alloc_page_failed++;
 		return false;
 	}
@@ -898,32 +896,36 @@ static void iavf_receive_skb(struct iavf_ring *rx_ring,
 }
 
 /**
- * iavf_alloc_rx_buffers - Replace used receive buffers
+ * __iavf_alloc_rx_buffers - Replace used receive buffers
  * @rx_ring: ring to place buffers on
- * @cleaned_count: number of buffers to replace
+ * @to_refill: number of buffers to replace
+ * @gfp: GFP mask to allocate pages
  *
- * Returns false if all allocations were successful, true if any fail
+ * Returns 0 if all allocations were successful or the number of buffers left
+ * to refill in case of an allocation failure.
  **/
-bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
+static u32 __iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u32 to_refill,
+				   gfp_t gfp)
 {
-	u16 ntu = rx_ring->next_to_use;
+	u32 order = iavf_rx_pg_order(rx_ring);
+	struct device *dev = rx_ring->dev;
+	u32 ntu = rx_ring->next_to_use;
 	union iavf_rx_desc *rx_desc;
 	struct iavf_rx_buffer *bi;
 
 	/* do nothing if no valid netdev defined */
-	if (!rx_ring->netdev || !cleaned_count)
-		return false;
+	if (unlikely(!rx_ring->netdev || !to_refill))
+		return 0;
 
 	rx_desc = IAVF_RX_DESC(rx_ring, ntu);
 	bi = &rx_ring->rx_bi[ntu];
 
 	do {
-		if (!iavf_alloc_mapped_page(rx_ring, bi))
-			goto no_buffers;
+		if (!iavf_alloc_mapped_page(rx_ring, bi, dev, order, gfp))
+			break;
 
 		/* sync the buffer for use by the device */
-		dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
-						 bi->page_offset,
+		dma_sync_single_range_for_device(dev, bi->dma, bi->page_offset,
 						 rx_ring->rx_buf_len,
 						 DMA_FROM_DEVICE);
 

drivers/net/ethernet/intel/iavf/iavf_txrx.c bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count) · line 948

Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -3
@@ -943,23 +945,17 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
 
 		/* clear the status bits for the next_to_use descriptor */
 		rx_desc->wb.qword1.status_error_len = 0;
-
-		cleaned_count--;
-	} while (cleaned_count);
+	} while (--to_refill);

Just a nit. You might want to break this up into two statements like I had before. I know some people within Intel weren't a huge fan of when I used to do that kind of thing all the time in loops where I would do the decrement and test in one line.. :)

Quoted 229 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +42 -96
 	if (rx_ring->next_to_use != ntu)
 		iavf_release_rx_desc(rx_ring, ntu);
 
-	return false;
-
-no_buffers:
-	if (rx_ring->next_to_use != ntu)
-		iavf_release_rx_desc(rx_ring, ntu);
+	return to_refill;
+}
 
-	/* make sure to come back via polling to try again after
-	 * allocation failure
-	 */
-	return true;
+void iavf_alloc_rx_buffers(struct iavf_ring *rxr)
+{
+	__iavf_alloc_rx_buffers(rxr, IAVF_DESC_UNUSED(rxr), GFP_KERNEL);
 }
 
 /**
@@ -1104,32 +1100,6 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)
 	return false;
 }
 
-/**
- * iavf_reuse_rx_page - page flip buffer and store it back on the ring
- * @rx_ring: rx descriptor ring to store buffers on
- * @old_buff: donor buffer to have page reused
- *
- * Synchronizes page for reuse by the adapter
- **/
-static void iavf_reuse_rx_page(struct iavf_ring *rx_ring,
-			       struct iavf_rx_buffer *old_buff)
-{
-	struct iavf_rx_buffer *new_buff;
-	u16 nta = rx_ring->next_to_alloc;
-
-	new_buff = &rx_ring->rx_bi[nta];
-
-	/* update, and store next to alloc */
-	nta++;
-	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
-
-	/* transfer page from old buffer to new buffer */
-	new_buff->dma		= old_buff->dma;
-	new_buff->page		= old_buff->page;
-	new_buff->page_offset	= old_buff->page_offset;
-	new_buff->pagecnt_bias	= old_buff->pagecnt_bias;
-}
-
 /**
  * iavf_can_reuse_rx_page - Determine if this page can be reused by
  * the adapter for another receive
@@ -1191,30 +1161,26 @@ static bool iavf_can_reuse_rx_page(struct iavf_rx_buffer *rx_buffer)
 
 /**
  * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff
- * @rx_ring: rx descriptor ring to transact packets on
- * @rx_buffer: buffer containing page to add
  * @skb: sk_buff to place the data into
+ * @rx_buffer: buffer containing page to add
  * @size: packet length from rx_desc
+ * @pg_size: Rx buffer page size
  *
  * This function will add the data contained in rx_buffer->page to the skb.
  * It will just attach the page as a frag to the skb.
  *
  * The function will then update the page offset.
  **/
-static void iavf_add_rx_frag(struct iavf_ring *rx_ring,
+static void iavf_add_rx_frag(struct sk_buff *skb,
 			     struct iavf_rx_buffer *rx_buffer,
-			     struct sk_buff *skb,
-			     unsigned int size)
+			     u32 size, u32 pg_size)
 {
 #if (PAGE_SIZE < 8192)
-	unsigned int truesize = iavf_rx_pg_size(rx_ring) / 2;
+	unsigned int truesize = pg_size / 2;
 #else
 	unsigned int truesize = SKB_DATA_ALIGN(size + IAVF_SKB_PAD);
 #endif
 
-	if (!size)
-		return;
-
 	skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
 			rx_buffer->page_offset, size, truesize);
 
@@ -1224,63 +1190,47 @@ static void iavf_add_rx_frag(struct iavf_ring *rx_ring,
 #else
 	rx_buffer->page_offset += truesize;
 #endif
+
+	/* We have pulled a buffer for use, so decrement pagecnt_bias */
+	rx_buffer->pagecnt_bias--;
 }
 
 /**
- * iavf_get_rx_buffer - Fetch Rx buffer and synchronize data for use
- * @rx_ring: rx descriptor ring to transact packets on
- * @size: size of buffer to add to skb
+ * iavf_sync_rx_buffer - Synchronize received data for use
+ * @dev: device used for DMA mapping
+ * @buf: Rx buffer containing the data
+ * @size: size of the received data
  *
- * This function will pull an Rx buffer from the ring and synchronize it
- * for use by the CPU.
+ * This function will synchronize the Rx buffer for use by the CPU.
  */
-static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring,
-						 const unsigned int size)
+static void iavf_sync_rx_buffer(struct device *dev, struct iavf_rx_buffer *buf,
+				u32 size)
 {
-	struct iavf_rx_buffer *rx_buffer;
-
-	rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean];
-	prefetchw(rx_buffer->page);
-	if (!size)
-		return rx_buffer;
-
-	/* we are reusing so sync this buffer for CPU use */
-	dma_sync_single_range_for_cpu(rx_ring->dev,
-				      rx_buffer->dma,
-				      rx_buffer->page_offset,
-				      size,
+	dma_sync_single_range_for_cpu(dev, buf->dma, buf->page_offset, size,
 				      DMA_FROM_DEVICE);
-
-	/* We have pulled a buffer for use, so decrement pagecnt_bias */
-	rx_buffer->pagecnt_bias--;
-
-	return rx_buffer;
 }
 
 /**
  * iavf_build_skb - Build skb around an existing buffer
- * @rx_ring: Rx descriptor ring to transact packets on
- * @rx_buffer: Rx buffer to pull data from
- * @size: size of buffer to add to skb
+ * @rx_buffer: Rx buffer with the data
+ * @size: size of the data
+ * @pg_size: size of the Rx page
  *
  * This function builds an skb around an existing Rx buffer, taking care
  * to set up the skb correctly and avoid any memcpy overhead.
  */
-static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
-				      struct iavf_rx_buffer *rx_buffer,
-				      unsigned int size)
+static struct sk_buff *iavf_build_skb(struct iavf_rx_buffer *rx_buffer,
+				      u32 size, u32 pg_size)
 {
 	void *va;
 #if (PAGE_SIZE < 8192)
-	unsigned int truesize = iavf_rx_pg_size(rx_ring) / 2;
+	unsigned int truesize = pg_size / 2;
 #else
 	unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
 				SKB_DATA_ALIGN(IAVF_SKB_PAD + size);
 #endif
 	struct sk_buff *skb;
 
-	if (!rx_buffer || !size)
-		return NULL;
 	/* prefetch first cache line of first page */
 	va = page_address(rx_buffer->page) + rx_buffer->page_offset;
 	net_prefetch(va);
@@ -1301,36 +1251,33 @@ static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,
 	rx_buffer->page_offset += truesize;
 #endif
 
+	rx_buffer->pagecnt_bias--;
+
 	return skb;
 }
 
 /**
- * iavf_put_rx_buffer - Clean up used buffer and either recycle or free
+ * iavf_put_rx_buffer - Recycle or free used buffer
  * @rx_ring: rx descriptor ring to transact packets on
- * @rx_buffer: rx buffer to pull data from
+ * @dev: device used for DMA mapping
+ * @rx_buffer: Rx buffer to handle
+ * @pg_size: Rx page size
  *
- * This function will clean up the contents of the rx_buffer.  It will
- * either recycle the buffer or unmap it and free the associated resources.
+ * Either recycle the buffer if possible or unmap and free the page.
  */
-static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,
-			       struct iavf_rx_buffer *rx_buffer)
+static void iavf_put_rx_buffer(struct iavf_ring *rx_ring, struct device *dev,
+			       struct iavf_rx_buffer *rx_buffer, u32 pg_size)
 {
-	if (!rx_buffer)
-		return;
-
 	if (iavf_can_reuse_rx_page(rx_buffer)) {
-		/* hand second half of page back to the ring */
-		iavf_reuse_rx_page(rx_ring, rx_buffer);
 		rx_ring->rx_stats.page_reuse_count++;
-	} else {
-		/* we are not reusing the buffer so unmap it */
-		dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
-				     iavf_rx_pg_size(rx_ring),
-				     DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
-		__page_frag_cache_drain(rx_buffer->page,
-					rx_buffer->pagecnt_bias);
+		return;
 	}
 
+	/* we are not reusing the buffer so unmap it */
+	dma_unmap_page_attrs(dev, rx_buffer->dma, pg_size,
+			     DMA_FROM_DEVICE, IAVF_RX_DMA_ATTR);
+	__page_frag_cache_drain(rx_buffer->page, rx_buffer->pagecnt_bias);
+
 	/* clear contents of buffer_info */
 	rx_buffer->page = NULL;
 }
@@ -1350,14 +1297,6 @@ static bool iavf_is_non_eop(struct iavf_ring *rx_ring,
 			    union iavf_rx_desc *rx_desc,
 			    struct sk_buff *skb)
 {
-	u32 ntc = rx_ring->next_to_clean + 1;

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_is_non_eop(struct iavf_ring *rx_ring, · line 1302

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -7
-
-	/* fetch, update, and store next to clean */
-	ntc = (ntc < rx_ring->count) ? ntc : 0;
-	rx_ring->next_to_clean = ntc;
-
-	prefetch(IAVF_RX_DESC(rx_ring, ntc));
-
 	/* if we are the last buffer then there is nothing else to do */
 #define IAVF_RXD_EOF BIT(IAVF_RX_DESC_STATUS_EOF_SHIFT)
 	if (likely(iavf_test_staterr(rx_desc, IAVF_RXD_EOF)))

You may want to see if you can get rid of this function entirely, perhaps you do in a later patch. This function was added for ixgbe back in the day to allow us to place the skb back in the ring for the RSC based workloads where we had to deal with interleaved frames in the Rx path.

For example, one question here would be why are we passing skb? It isn't used as far as I can tell.

Quoted 125 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +43 -17
@@ -1383,11 +1322,16 @@ static bool iavf_is_non_eop(struct iavf_ring *rx_ring,
 static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 {
 	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+	const gfp_t gfp = GFP_ATOMIC | __GFP_NOWARN;
+	u32 to_refill = IAVF_DESC_UNUSED(rx_ring);
+	u32 pg_size = iavf_rx_pg_size(rx_ring);
 	struct sk_buff *skb = rx_ring->skb;
-	u16 cleaned_count = IAVF_DESC_UNUSED(rx_ring);
-	bool failure = false;
+	struct device *dev = rx_ring->dev;
+	u32 ntc = rx_ring->next_to_clean;
+	u32 ring_size = rx_ring->count;
+	u32 cleaned_count = 0;
 
-	while (likely(total_rx_packets < (unsigned int)budget)) {
+	while (likely(cleaned_count < budget)) {
 		struct iavf_rx_buffer *rx_buffer;
 		union iavf_rx_desc *rx_desc;
 		unsigned int size;
@@ -1396,13 +1340,11 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 		u64 qword;
 
 		/* return some buffers to hardware, one at a time is too slow */
-		if (cleaned_count >= IAVF_RX_BUFFER_WRITE) {
-			failure = failure ||
-				  iavf_alloc_rx_buffers(rx_ring, cleaned_count);
-			cleaned_count = 0;
-		}
+		if (to_refill >= IAVF_RX_BUFFER_WRITE)
+			to_refill = __iavf_alloc_rx_buffers(rx_ring, to_refill,
+							    gfp);
 
-		rx_desc = IAVF_RX_DESC(rx_ring, rx_ring->next_to_clean);
+		rx_desc = IAVF_RX_DESC(rx_ring, ntc);
 
 		/* status_error_len will always be zero for unused descriptors
 		 * because it's cleared in cleanup, and overlaps with hdr_addr
@@ -1424,24 +1366,38 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 		       IAVF_RXD_QW1_LENGTH_PBUF_SHIFT;
 
 		iavf_trace(clean_rx_irq, rx_ring, rx_desc, skb);
-		rx_buffer = iavf_get_rx_buffer(rx_ring, size);
+		rx_buffer = &rx_ring->rx_bi[ntc];
+
+		/* Very rare, but possible case. The most common reason:
+		 * the last fragment contained FCS only, which was then
+		 * stripped by the HW.
+		 */
+		if (unlikely(!size))
+			goto skip_data;
+
+		iavf_sync_rx_buffer(dev, rx_buffer, size);
 
 		/* retrieve a buffer from the ring */
 		if (skb)
-			iavf_add_rx_frag(rx_ring, rx_buffer, skb, size);
+			iavf_add_rx_frag(skb, rx_buffer, size, pg_size);
 		else
-			skb = iavf_build_skb(rx_ring, rx_buffer, size);
+			skb = iavf_build_skb(rx_buffer, size, pg_size);
 
 		/* exit if we failed to retrieve a buffer */
 		if (!skb) {
 			rx_ring->rx_stats.alloc_buff_failed++;
-			if (rx_buffer && size)
-				rx_buffer->pagecnt_bias++;
 			break;
 		}
 
-		iavf_put_rx_buffer(rx_ring, rx_buffer);
+skip_data:
+		iavf_put_rx_buffer(rx_ring, dev, rx_buffer, pg_size);
+
 		cleaned_count++;
+		to_refill++;
+		if (unlikely(++ntc == ring_size))
+			ntc = 0;
+
+		prefetch(IAVF_RX_DESC(rx_ring, ntc));
 
 		if (iavf_is_non_eop(rx_ring, rx_desc, skb))
 			continue;
@@ -1488,8 +1444,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 		total_rx_packets++;
 	}
 
+	rx_ring->next_to_clean = ntc;
 	rx_ring->skb = skb;
 
+	if (to_refill >= IAVF_RX_BUFFER_WRITE) {
+		to_refill = __iavf_alloc_rx_buffers(rx_ring, to_refill, gfp);
+		/* guarantee a trip back through this routine if there was
+		 * a failure
+		 */
+		if (unlikely(to_refill))
+			cleaned_count = budget;
+	}
+
 	u64_stats_update_begin(&rx_ring->syncp);
 	rx_ring->stats.packets += total_rx_packets;
 	rx_ring->stats.bytes += total_rx_bytes;
@@ -1497,8 +1463,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)
 	rx_ring->q_vector->rx.total_packets += total_rx_packets;
 	rx_ring->q_vector->rx.total_bytes += total_rx_bytes;
 
-	/* guarantee a trip back through this routine if there was a failure */
-	return failure ? budget : (int)total_rx_packets;
+	return cleaned_count;
 }
 
 static inline u32 iavf_buildreg_itr(const int type, u16 itr)
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -1
index 234e189c1987..9c6661a6edf2 100644
@@ -383,7 +383,6 @@ struct iavf_ring {
 	struct iavf_q_vector *q_vector;	/* Backreference to associated vector */
 
 	struct rcu_head rcu;		/* to avoid race on free */
-	u16 next_to_alloc;
 	struct sk_buff *skb;		/* When iavf_clean_rx_ring_irq() must
 					 * return before it sees the EOP for
 					 * the current packet, we save that skb

drivers/net/ethernet/intel/iavf/iavf_txrx.h static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring) · line 431

Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1 -1
@@ -426,7 +425,7 @@ static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
 
 #define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
 
-bool iavf_alloc_rx_buffers(struct iavf_ring *rxr, u16 cleaned_count);
+void iavf_alloc_rx_buffers(struct iavf_ring *rxr);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 void iavf_clean_tx_ring(struct iavf_ring *tx_ring);
 void iavf_clean_rx_ring(struct iavf_ring *rx_ring);

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#7 lore raw

Re: [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

On Tue, May 30, 2023 at 09:18:40AM -0700, Alexander H Duyck wrote:

FWIW I agree with what Alex is saying over here.

Quoted 12 lines · Alexander Lobakin
Quoted 10 lines

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.

Quoted 5 lines

The advantage of this is going to vary based on the attribute. One of the reasons why I left most of this on the ring is because the section of the ring most of these variables were meant to be read-mostly and shouldn't have resulted in any additional overhead versus accessing them from the stack.

I believe it depends on ring struct layout which vary across our drivers, no? On ice using making more usage of stack as described above improved perf.

Quoted 19 lines
Quoted 11 lines

2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.

The next_to_alloc logic was put in place to deal with systems that are experiencing memory issues. Specifically what can end up happening is that the ring can stall due to failing memory allocations and the memory can get stuck on the ring. For that reason we were essentially defragmenting the buffers when we started suffering memory pressure so that they could be reusued and/or freed following immediate use.

Quoted 2 lines

Basically what you are trading off is some exception handling for performance by removing it.

With all of the mix of the changes this patch carries, I find it hard to follow from description which parts of diff I should be looking at.

Quoted 38 lines
Quoted 4 lines

3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Quoted 6 lines

4. 1 budget unit == 1 descriptor, not skb.
There could be underflow when receiving a lot of fragmented frames. If each of them would consist of 2 frags, it means that we'd process 64 descriptors at the point where we pass the 32th skb to the stack. But the driver would count that only as a half, which could make NAPI re-enable interrupts prematurely and create unnecessary CPU load.

Not sure I agree with this. The problem is the overhead for an skb going up the stack versus a fragment are pretty signficant. Keep in mind that most of the overhead for a single buffer occurs w/ napi_gro_receive and is not actually at the driver itself. The whole point of the budget is to meter out units of work, not to keep you in the busy loop. This starts looking like the old code where the Intel drivers were returning either budget or 0 instead of supporting the middle ground.

Quoted 13 lines

5. Shortcut !size case.
It's super rare, but possible -- for example, if the last buffer of the fragmented frame contained only FCS, which was then stripped by the HW. Instead of checking for size several times when processing, quickly reuse the buffer and jump to the skb fields part.
6. Refill the ring after finishing the polling loop. Previously, the loop wasn't starting a new iteration after the 64th desc, meaning that we were always leaving 16 buffers non-refilled until the next NAPI poll. It's better to refill them while they're still hot, so do that right after exiting the loop as well. For a full cycle of 64 descs, there will be 4 refills of 16 descs from now on.

Quoted 7 lines
Quoted 4 lines

Function: add/remove: 4/2 grow/shrink: 0/5 up/down: 473/-647 (-174)

+ up to 2% performance.

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

+1, can you say more about that measurement?

Quoted 7 lines
Quoted 2 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

+1

Quoted 7 lines
Quoted 7 lines
---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 259 +++++++++-----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   3 +-
 3 files changed, 114 insertions(+), 150 deletions(-)

(...)

Quoted 23 lines
Quoted 6 lines

}

drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -1
@@ -1350,14 +1297,6 @@ static bool iavf_is_non_eop(struct iavf_ring *rx_ring,
 			    union iavf_rx_desc *rx_desc,
 			    struct sk_buff *skb)
 {
-	u32 ntc = rx_ring->next_to_clean + 1;

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_is_non_eop(struct iavf_ring *rx_ring, · line 1302

Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -7
-
-	/* fetch, update, and store next to clean */
-	ntc = (ntc < rx_ring->count) ? ntc : 0;
-	rx_ring->next_to_clean = ntc;
-
-	prefetch(IAVF_RX_DESC(rx_ring, ntc));
-
 	/* if we are the last buffer then there is nothing else to do */
 #define IAVF_RXD_EOF BIT(IAVF_RX_DESC_STATUS_EOF_SHIFT)
 	if (likely(iavf_test_staterr(rx_desc, IAVF_RXD_EOF)))

You may want to see if you can get rid of this function entirely, perhaps you do in a later patch. This function was added for ixgbe back in the day to allow us to place the skb back in the ring for the RSC based workloads where we had to deal with interleaved frames in the Rx path.

Quoted 3 lines

For example, one question here would be why are we passing skb? It isn't used as far as I can tell.

this was used back when skb was stored within the Rx buffer and now we just store skb on Rx ring struct, so good catch, this arg is redundant.

I'll go and take a look at code on v3.
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#8 lore raw

Re: [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

From: Maciej Fijalkowski <[email protected]>
Date: Wed, 31 May 2023 13:14:12 +0200

Quoted 3 lines · Alexander H Duyck

FWIW I agree with what Alex is saying over here.

There are 2 Alexes, "choose wisely" :P

Quoted 19 lines
Quoted 13 lines · Alexander Lobakin
Quoted 10 lines

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.

Quoted 5 lines

The advantage of this is going to vary based on the attribute. One of the reasons why I left most of this on the ring is because the section of the ring most of these variables were meant to be read-mostly and shouldn't have resulted in any additional overhead versus accessing them from the stack.

Quoted 3 lines

I believe it depends on ring struct layout which vary across our drivers, no? On ice using making more usage of stack as described above improved perf.

It's +/- the same, most layout changes usually come with us moving stuff around to optimize paddings and cachelines lol. Here's the same as with ice, I don't think it's driver specific to get some positive results from shortcutting more hotties. The sole time I was surprised is when you were getting worse results storing xdp_buff on the stack vs on the ring :D

Quoted 20 lines
Quoted 20 lines
Quoted 12 lines

2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.

The next_to_alloc logic was put in place to deal with systems that are experiencing memory issues. Specifically what can end up happening is that the ring can stall due to failing memory allocations and the memory can get stuck on the ring. For that reason we were essentially defragmenting the buffers when we started suffering memory pressure so that they could be reusued and/or freed following immediate use.

Quoted 5 lines
Quoted 2 lines

Basically what you are trading off is some exception handling for performance by removing it.

With all of the mix of the changes this patch carries, I find it hard to follow from description which parts of diff I should be looking at.

Huge piece removed before add_rx_frag_blah.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.

[...]

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

+ up to 2% performance.

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

+1, can you say more about that measurement?

My prev reply to Alex.

Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

+1

+1 :D

[...]

Quoted 10 lines
Quoted 10 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_is_non_eop(struct iavf_ring *rx_ring, · line 1302

Quoted 3 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -0
 	/* if we are the last buffer then there is nothing else to do */
 #define IAVF_RXD_EOF BIT(IAVF_RX_DESC_STATUS_EOF_SHIFT)
 	if (likely(iavf_test_staterr(rx_desc, IAVF_RXD_EOF)))

You may want to see if you can get rid of this function entirely, perhaps you do in a later patch. This function was added for ixgbe back in the day to allow us to place the skb back in the ring for the RSC based workloads where we had to deal with interleaved frames in the Rx path.

Quoted 5 lines
Quoted 3 lines

For example, one question here would be why are we passing skb? It isn't used as far as I can tell.

this was used back when skb was stored within the Rx buffer and now we just store skb on Rx ring struct, so good catch, this arg is redundant.

Also prev reply. I'm removing it later in the series hehe.

Quoted 2 lines

I'll go and take a look at code on v3.

No changes apart fixing OcteonTX2 compilation =\

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#9 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

In reply to Alexander H Duyck

From: Alexander H Duyck <[email protected]>
Date: Tue, 30 May 2023 09:18:40 -0700

Quoted 12 lines · Alexander Lobakin
Quoted 10 lines

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.

Quoted 5 lines

The advantage of this is going to vary based on the attribute. One of the reasons why I left most of this on the ring is because the section of the ring most of these variables were meant to be read-mostly and shouldn't have resulted in any additional overhead versus accessing them from the stack.

But not all of these variables are read-only. E.g. NTC is often modified. Page size was calculated per descriptor, but could be once a poll cycle starts, and so on.

Quoted 19 lines
Quoted 11 lines

2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.

The next_to_alloc logic was put in place to deal with systems that are experiencing memory issues. Specifically what can end up happening is that the ring can stall due to failing memory allocations and the memory can get stuck on the ring. For that reason we were essentially defragmenting the buffers when we started suffering memory pressure so that they could be reusued and/or freed following immediate use.

Quoted 2 lines

Basically what you are trading off is some exception handling for performance by removing it.

I'm not sure this helps a lot, but OTOH this really slows down things, esp. given that this code is run all the time, not only when a memory allocation fail happens.

Quoted 8 lines
Quoted 4 lines

3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Yes, I'm no MM expert, but I've seen plenty of times messages from the MM folks that ATOMIC shouldn't be used in non-atomic contexts. Atomic allocation is able to grab memory from some sort of critical reservs and all that, and the less we touch them, the better. Outside of atomic contexts they should not be touched.

Quoted 8 lines
Quoted 7 lines

4. 1 budget unit == 1 descriptor, not skb.
There could be underflow when receiving a lot of fragmented frames. If each of them would consist of 2 frags, it means that we'd process 64 descriptors at the point where we pass the 32th skb to the stack. But the driver would count that only as a half, which could make NAPI re-enable interrupts prematurely and create unnecessary CPU load.

Quoted 8 lines

Not sure I agree with this. The problem is the overhead for an skb going up the stack versus a fragment are pretty signficant. Keep in mind that most of the overhead for a single buffer occurs w/ napi_gro_receive and is not actually at the driver itself. The whole point of the budget is to meter out units of work, not to keep you in the busy loop. This starts looking like the old code where the Intel drivers were returning either budget or 0 instead of supporting the middle ground.

No, certainly not this :D

The point of budget is to limit the amount of time drivers can spend on cleaning their rings. Making skb the unit makes the unit very logical and flexible, but I'd say it should always be solid. Imagine you get a frame which got spanned across 5 buffers. You spend x5 time (roughly) to build an skb and pass it up the stack vs when you get a linear frame in one buffer, but according to your logics both of these cases count as 1 unit, while the amount of time spent differs significantly. I can't say that's fair enough.

Quoted 14 lines
Quoted 14 lines

5. Shortcut !size case.
It's super rare, but possible -- for example, if the last buffer of the fragmented frame contained only FCS, which was then stripped by the HW. Instead of checking for size several times when processing, quickly reuse the buffer and jump to the skb fields part.
6. Refill the ring after finishing the polling loop. Previously, the loop wasn't starting a new iteration after the 64th desc, meaning that we were always leaving 16 buffers non-refilled until the next NAPI poll. It's better to refill them while they're still hot, so do that right after exiting the loop as well. For a full cycle of 64 descs, there will be 4 refills of 16 descs from now on.

Quoted 7 lines
Quoted 4 lines

Function: add/remove: 4/2 grow/shrink: 0/5 up/down: 473/-647 (-174)

+ up to 2% performance.

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

Not XDP, it's not present in this driver at this point :D Stack test, but without usercopy overhead. Trafgen bombs the NIC, the driver builds skbs and passes it up the stack, the stack does GRO etc, and then the frames get dropped on IP input because there's no socket.

Quoted 7 lines
Quoted 2 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

Agree BTW, I'm not a fan of this patch either. I wasn't sure what to do with it, as splitting it into 6 explodes the series into a monster, but proceeding without it increases diffstat and complicates things later on. I'll try the latter, but will see. 17 patches is not the End of Days after all.

Quoted 6 lines
Quoted 6 lines
---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 259 +++++++++-----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   3 +-
 3 files changed, 114 insertions(+), 150 deletions(-)

[...]

Quoted 13 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count) · line 948

Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -3
@@ -943,23 +945,17 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
 
 		/* clear the status bits for the next_to_use descriptor */
 		rx_desc->wb.qword1.status_error_len = 0;
-
-		cleaned_count--;
-	} while (cleaned_count);
+	} while (--to_refill);

Just a nit. You might want to break this up into two statements like I had before. I know some people within Intel weren't a huge fan of when I used to do that kind of thing all the time in loops where I would do the decrement and test in one line.. :)

Should I please them or do it as I want to? :D I realize from the compiler's PoV it's most likely the same, but dunno, why not.

Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -1
 	if (rx_ring->next_to_use != ntu)
 		iavf_release_rx_desc(rx_ring, ntu);
 
-	return false;

[...]

Quoted 4 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_is_non_eop(struct iavf_ring *rx_ring, · line 1302

Quoted 3 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -0
 	/* if we are the last buffer then there is nothing else to do */
 #define IAVF_RXD_EOF BIT(IAVF_RX_DESC_STATUS_EOF_SHIFT)
 	if (likely(iavf_test_staterr(rx_desc, IAVF_RXD_EOF)))
Quoted 8 lines

You may want to see if you can get rid of this function entirely, perhaps you do in a later patch. This function was added for ixgbe back in the day to allow us to place the skb back in the ring for the RSC based workloads where we had to deal with interleaved frames in the Rx path.

For example, one question here would be why are we passing skb? It isn't used as far as I can tell.

Yes, I'm optimizing all this out later in the series. I was surprised just as much as you when I saw skb getting passed to do nothing ._.

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

Olek

#10 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

Quoted 22 lines · Alexander Lobakin

From: Alexander H Duyck <[email protected]>
Date: Tue, 30 May 2023 09:18:40 -0700

Quoted 12 lines · Alexander Lobakin
Quoted 10 lines

The Rx hotpath code of IAVF is not well-optimized TBH. Before doing any further buffer model changes, shake it up a bit. Notably:

1. Cache more variables on the stack.
DMA device, Rx page size, NTC -- these are the most common things used all throughout the hotpath, often in loops on each iteration. Instead of fetching (or even calculating, as with the page size) them from the ring all the time, cache them on the stack at the beginning of the NAPI polling callback. NTC will be written back at the end, the rest are used read-only, so no sync needed.

Quoted 5 lines

The advantage of this is going to vary based on the attribute. One of the reasons why I left most of this on the ring is because the section of the ring most of these variables were meant to be read-mostly and shouldn't have resulted in any additional overhead versus accessing them from the stack.

Quoted 3 lines

But not all of these variables are read-only. E.g. NTC is often modified. Page size was calculated per descriptor, but could be once a poll cycle starts, and so on.

Yeah, the ntc should be carried in the stack. The only reason for using the ring variable was because in the case of ixgbe we had to do some tricks with it to deal with RSC as we were either accessing ntc or the buffer pointed to by the descriptor. I think most of that code has been removed for i40e though.

Quoted 19 lines
Quoted 19 lines
Quoted 11 lines

2. Don't move the recycled buffers around the ring.
The idea of passing the page of the right-now-recycled-buffer to a different buffer, in this case, the first one that needs to be allocated, moreover, on each new frame, is fundamentally wrong. It involves a few o' fetches, branches and then writes (and one Rx buffer struct is at least 32 bytes) where they're completely unneeded, but gives no good -- the result is the same as if we'd recycle it inplace, at the same position where it was used. So drop this and let the main refilling function take care of all the buffers, which were processed and now need to be recycled/refilled.

The next_to_alloc logic was put in place to deal with systems that are experiencing memory issues. Specifically what can end up happening is that the ring can stall due to failing memory allocations and the memory can get stuck on the ring. For that reason we were essentially defragmenting the buffers when we started suffering memory pressure so that they could be reusued and/or freed following immediate use.

Quoted 6 lines
Quoted 2 lines

Basically what you are trading off is some exception handling for performance by removing it.

I'm not sure this helps a lot, but OTOH this really slows down things, esp. given that this code is run all the time, not only when a memory allocation fail happens.

The thing is when things break they have to be fixed. For our purposes since you are getting rid of the recycling anyway it is going to be ripped out one way or the other. Just wanted to provide a bit of history as to why it is being done that way.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

3. Don't allocate with %GPF_ATOMIC on ifup.
This involved introducing the @gfp parameter to a couple functions. Doesn't change anything for Rx -> softirq.

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Quoted 5 lines

Yes, I'm no MM expert, but I've seen plenty of times messages from the MM folks that ATOMIC shouldn't be used in non-atomic contexts. Atomic allocation is able to grab memory from some sort of critical reservs and all that, and the less we touch them, the better. Outside of atomic contexts they should not be touched.

For our purposes though the Rx path is more-or-less always in interrupt context. That is why it had defaulted to just always using GFP_ATOMIC. For your purposes you could probably leave it that way since you are going to be pulling out most of this code anyway.

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines

4. 1 budget unit == 1 descriptor, not skb.
There could be underflow when receiving a lot of fragmented frames. If each of them would consist of 2 frags, it means that we'd process 64 descriptors at the point where we pass the 32th skb to the stack. But the driver would count that only as a half, which could make NAPI re-enable interrupts prematurely and create unnecessary CPU load.

Quoted 8 lines

Not sure I agree with this. The problem is the overhead for an skb going up the stack versus a fragment are pretty signficant. Keep in mind that most of the overhead for a single buffer occurs w/ napi_gro_receive and is not actually at the driver itself. The whole point of the budget is to meter out units of work, not to keep you in the busy loop. This starts looking like the old code where the Intel drivers were returning either budget or 0 instead of supporting the middle ground.

No, certainly not this :D

Quoted 8 lines

The point of budget is to limit the amount of time drivers can spend on cleaning their rings. Making skb the unit makes the unit very logical and flexible, but I'd say it should always be solid. Imagine you get a frame which got spanned across 5 buffers. You spend x5 time (roughly) to build an skb and pass it up the stack vs when you get a linear frame in one buffer, but according to your logics both of these cases count as 1 unit, while the amount of time spent differs significantly. I can't say that's fair enough.

I would say it is. Like I said most of the overhead is the stack, not the driver. So if we are cleaning 5 descriptors but only processing one skb then I would say it is only one unit in terms of budget. This is one of the reasons why we don't charge Tx to the NAPI budget. Tx clean up is extremely lightweight as it is only freeing memory, and in cases of Tx and Rx being mixed can essentially be folded in as Tx buffers could be reused for Rx.

If we are wanting to increase the work being done per poll it would make more sense to stick to interrupts and force it to backlog more packets per interrupt so that it is processing 64 skbs per call.

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

5. Shortcut !size case.
It's super rare, but possible -- for example, if the last buffer of the fragmented frame contained only FCS, which was then stripped by the HW. Instead of checking for size several times when processing, quickly reuse the buffer and jump to the skb fields part.
6. Refill the ring after finishing the polling loop. Previously, the loop wasn't starting a new iteration after the 64th desc, meaning that we were always leaving 16 buffers non-refilled until the next NAPI poll. It's better to refill them while they're still hot, so do that right after exiting the loop as well. For a full cycle of 64 descs, there will be 4 refills of 16 descs from now on.

Function: add/remove: 4/2 grow/shrink: 0/5 up/down: 473/-647 (-174)

+ up to 2% performance.

Quoted 7 lines
Quoted 2 lines

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

Not XDP, it's not present in this driver at this point :D Stack test, but without usercopy overhead. Trafgen bombs the NIC, the driver builds skbs and passes it up the stack, the stack does GRO etc, and then the frames get dropped on IP input because there's no socket.

So one thing you might want to look at would be a full stack test w/ something such as netperf versus optimizing for a drop only test. Otherwise that can lead to optimizations that will actually hurt driver performance in the long run.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

Quoted 5 lines

Agree BTW, I'm not a fan of this patch either. I wasn't sure what to do with it, as splitting it into 6 explodes the series into a monster, but proceeding without it increases diffstat and complicates things later on. I'll try the latter, but will see. 17 patches is not the End of Days after all.

One thing you may want to consider to condense some of these patches would be to look at possibly combining patches 4 and 5 which disable recycling and use a full 4K page. It seems like of those patches one ends up redoing the other since so many of the dma_sync calls are updated in both.

Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 259 +++++++++-----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   3 +-
 3 files changed, 114 insertions(+), 150 deletions(-)

[...]

Quoted 13 lines

drivers/net/ethernet/intel/iavf/iavf_txrx.c bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count) · line 948

Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1 -3
@@ -943,23 +945,17 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)
 
             /* clear the status bits for the next_to_use descriptor */
             rx_desc->wb.qword1.status_error_len = 0;
-
-            cleaned_count--;
-    } while (cleaned_count);
+    } while (--to_refill);

Just a nit. You might want to break this up into two statements like I had before. I know some people within Intel weren't a huge fan of when I used to do that kind of thing all the time in loops where I would do the decrement and test in one line.. :)

Quoted 2 lines

Should I please them or do it as I want to? :D I realize from the compiler's PoV it's most likely the same, but dunno, why not.

If nobody internally is bugging you about it then I am fine with it. I just know back during my era people would complain about that from a maintainability perspective. I guess I got trained to catch those kind of things as a result.

Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0 -1
     if (rx_ring->next_to_use != ntu)
             iavf_release_rx_desc(rx_ring, ntu);
 
-    return false;

[...]

Quoted 10 lines
Quoted 3 lines
     /* if we are the last buffer then there is nothing else to do */

#define IAVF_RXD_EOF BIT(IAVF_RX_DESC_STATUS_EOF_SHIFT)
if (likely(iavf_test_staterr(rx_desc, IAVF_RXD_EOF)))

You may want to see if you can get rid of this function entirely, perhaps you do in a later patch. This function was added for ixgbe back in the day to allow us to place the skb back in the ring for the RSC based workloads where we had to deal with interleaved frames in the Rx path.

Quoted 5 lines
Quoted 2 lines

For example, one question here would be why are we passing skb? It isn't used as far as I can tell.

Yes, I'm optimizing all this out later in the series. I was surprised just as much as you when I saw skb getting passed to do nothing ._.

The funny part for me is that it is like reviewing code written via a game of telephone. I recognize the code but have to think about it since there are all the bits of changes and such from the original ixgbe.

Quoted 3 lines

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

No problem, it was the least I could do since I am responsible for so much of this code in the earlier drivers anyway. If nothing else I figured I could provide a bit of history on why some of this was the way it was.

author maintainer
#11 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

From: Alexander Duyck <[email protected]>
Date: Wed, 31 May 2023 10:22:18 -0700

Quoted 2 lines

On Wed, May 31, 2023 at 8:14 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 3 lines

But not all of these variables are read-only. E.g. NTC is often modified. Page size was calculated per descriptor, but could be once a poll cycle starts, and so on.

Yeah, the ntc should be carried in the stack. The only reason for using the ring variable was because in the case of ixgbe we had to do some tricks with it to deal with RSC as we were either accessing ntc or the buffer pointed to by the descriptor. I think most of that code has been removed for i40e though.

IAVF was forked off ixgbe as per Jesse's statement :D

[...]

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Yes, I'm no MM expert, but I've seen plenty of times messages from the MM folks that ATOMIC shouldn't be used in non-atomic contexts. Atomic allocation is able to grab memory from some sort of critical reservs and all that, and the less we touch them, the better. Outside of atomic contexts they should not be touched.

Quoted 4 lines

For our purposes though the Rx path is more-or-less always in interrupt context. That is why it had defaulted to just always using GFP_ATOMIC. For your purposes you could probably leave it that way since you are going to be pulling out most of this code anyway.

That's for Rx path, but don't forget that the initial allocation on ifup is done in the process context. That's what the maintainers and reviewers usually warn about: to not allocate with %GFP_ATOMIC on ifups.

[...]

Quoted 17 lines
Quoted 8 lines

The point of budget is to limit the amount of time drivers can spend on cleaning their rings. Making skb the unit makes the unit very logical and flexible, but I'd say it should always be solid. Imagine you get a frame which got spanned across 5 buffers. You spend x5 time (roughly) to build an skb and pass it up the stack vs when you get a linear frame in one buffer, but according to your logics both of these cases count as 1 unit, while the amount of time spent differs significantly. I can't say that's fair enough.

I would say it is. Like I said most of the overhead is the stack, not the driver. So if we are cleaning 5 descriptors but only processing one skb then I would say it is only one unit in terms of budget. This is one of the reasons why we don't charge Tx to the NAPI budget. Tx clean up is extremely lightweight as it is only freeing memory, and in cases of Tx and Rx being mixed can essentially be folded in as Tx buffers could be reused for Rx.

Quoted 3 lines

If we are wanting to increase the work being done per poll it would make more sense to stick to interrupts and force it to backlog more packets per interrupt so that it is processing 64 skbs per call.

Oh, I feel like I'm starting to agree :D OK, then the following doesn't really get out of my head: why do we store skb pointer on the ring then, if we count 1 skb as 1 unit, so that we won't leave the loop until the EOP? Only to handle allocation failures? But skb is already allocated at this point... <confused>

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

Not XDP, it's not present in this driver at this point :D Stack test, but without usercopy overhead. Trafgen bombs the NIC, the driver builds skbs and passes it up the stack, the stack does GRO etc, and then the frames get dropped on IP input because there's no socket.

Quoted 4 lines

So one thing you might want to look at would be a full stack test w/ something such as netperf versus optimizing for a drop only test. Otherwise that can lead to optimizations that will actually hurt driver performance in the long run.

I was doing some netperf (or that Microsoft's tool, don't remember the name) tests, but the problem is that usercopy is such a bottleneck, so that you don't notice any optimizations or regressions most of time. Also, userspace tools usually just pass huge payload chunks and then the drivers GSO them into MTU-sized frames, so you always get line rate and that's it. Short frames or interleave/imix (randomly-mix-sized) are the most stressful from my experience and are able to show actual outcome.

Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

Quoted 5 lines

Agree BTW, I'm not a fan of this patch either. I wasn't sure what to do with it, as splitting it into 6 explodes the series into a monster, but proceeding without it increases diffstat and complicates things later on. I'll try the latter, but will see. 17 patches is not the End of Days after all.

Quoted 5 lines

One thing you may want to consider to condense some of these patches would be to look at possibly combining patches 4 and 5 which disable recycling and use a full 4K page. It seems like of those patches one ends up redoing the other since so many of the dma_sync calls are updated in both.

Or maybe I'll move this one into the subsequent series, since it's only pt. 1 of Rx optimizations. There's also the second commit, but it's probably as messy as this one and these two could be just converted into a series.

[...]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Just a nit. You might want to break this up into two statements like I had before. I know some people within Intel weren't a huge fan of when I used to do that kind of thing all the time in loops where I would do the decrement and test in one line.. :)

Quoted 7 lines
Quoted 2 lines

Should I please them or do it as I want to? :D I realize from the compiler's PoV it's most likely the same, but dunno, why not.

If nobody internally is bugging you about it then I am fine with it. I just know back during my era people would complain about that from a maintainability perspective. I guess I got trained to catch those kind of things as a result.

Haha understand. I usually say: "please some good arguments or I didn't hear this", maybe that's why nobody complained on --var yet :D

[...]

Quoted 7 lines
Quoted 2 lines

Yes, I'm optimizing all this out later in the series. I was surprised just as much as you when I saw skb getting passed to do nothing ._.

The funny part for me is that it is like reviewing code written via a game of telephone. I recognize the code but have to think about it since there are all the bits of changes and such from the original ixgbe.

Lots of things are still recognizable even in IDPF. That's how this series was born... :D

Quoted 9 lines
Quoted 4 lines

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

No problem, it was the least I could do since I am responsible for so much of this code in the earlier drivers anyway. If nothing else I figured I could provide a bit of history on why some of this was the way it was.

These history bits are nice and interesting to read actually! And also useful since they give some context and understanding of what is obsolete and can be removed/changed.

Thanks,
Olek

#12 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

Quoted 13 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Wed, 31 May 2023 10:22:18 -0700

Quoted 2 lines

On Wed, May 31, 2023 at 8:14 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 4 lines
Quoted 3 lines

But not all of these variables are read-only. E.g. NTC is often modified. Page size was calculated per descriptor, but could be once a poll cycle starts, and so on.

Quoted 7 lines
Quoted 5 lines

Yeah, the ntc should be carried in the stack. The only reason for using the ring variable was because in the case of ixgbe we had to do some tricks with it to deal with RSC as we were either accessing ntc or the buffer pointed to by the descriptor. I think most of that code has been removed for i40e though.

IAVF was forked off ixgbe as per Jesse's statement :D

Yes, but point is they are forked off the same driver and this code has fallen a bit behind i40e. Really both should probably have been updated at the same time.

The fact is everything since igb is more or less based on the same design. I just kept tweaking it as I moved from one driver to the next. So in terms of refactoring to use a common library you could probably go back that far without too much trouble. The only exceptions to all that are fm10k and igbvf which while being similar also have some significant design differences that might make it a bit more difficult.

Quoted 12 lines

[...]

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

Any specific reason for this? Just wondering if this is meant to address some sort of memory pressure issue since it basically just means the allocation can go out and try to free other memory.

Yes, I'm no MM expert, but I've seen plenty of times messages from the MM folks that ATOMIC shouldn't be used in non-atomic contexts. Atomic allocation is able to grab memory from some sort of critical reservs and all that, and the less we touch them, the better. Outside of atomic contexts they should not be touched.

Quoted 8 lines
Quoted 4 lines

For our purposes though the Rx path is more-or-less always in interrupt context. That is why it had defaulted to just always using GFP_ATOMIC. For your purposes you could probably leave it that way since you are going to be pulling out most of this code anyway.

That's for Rx path, but don't forget that the initial allocation on ifup is done in the process context. That's what the maintainers and reviewers usually warn about: to not allocate with %GFP_ATOMIC on ifups.

I can see that for the static values like the queue vectors and rings, however for the buffers themselves, but I don't see the point in doing that for the regular buffer allocations. Basically it is adding overhead for something that should have minimal impact as it usually happens early on during boot when the memory should be free anyway so GFP_ATOMIC vs GFP_KERNEL wouldn't have much impact in either case

Quoted 23 lines

[...]

Quoted 17 lines
Quoted 8 lines

The point of budget is to limit the amount of time drivers can spend on cleaning their rings. Making skb the unit makes the unit very logical and flexible, but I'd say it should always be solid. Imagine you get a frame which got spanned across 5 buffers. You spend x5 time (roughly) to build an skb and pass it up the stack vs when you get a linear frame in one buffer, but according to your logics both of these cases count as 1 unit, while the amount of time spent differs significantly. I can't say that's fair enough.

I would say it is. Like I said most of the overhead is the stack, not the driver. So if we are cleaning 5 descriptors but only processing one skb then I would say it is only one unit in terms of budget. This is one of the reasons why we don't charge Tx to the NAPI budget. Tx clean up is extremely lightweight as it is only freeing memory, and in cases of Tx and Rx being mixed can essentially be folded in as Tx buffers could be reused for Rx.

Quoted 3 lines

If we are wanting to increase the work being done per poll it would make more sense to stick to interrupts and force it to backlog more packets per interrupt so that it is processing 64 skbs per call.

Quoted 5 lines

Oh, I feel like I'm starting to agree :D OK, then the following doesn't really get out of my head: why do we store skb pointer on the ring then, if we count 1 skb as 1 unit, so that we won't leave the loop until the EOP? Only to handle allocation failures? But skb is already allocated at this point... <confused>

The skb is there to essentially hold the frags. Keep in mind that when ixgbe was coded up XDP didn't exist yet.

I think there are drivers that are already getting away from this, such as mvneta, by storing an xdp_buff instead of an skb. In theory we could do away with most of this and just use a shared_info structure, but since that exists in the first frag we still need a pointer to the first frag as well.

Also multi-frag frames are typically not that likely on a normal network as most of the frames are less than 1514B in length. In addition as I mentioned before a jumbo frame workload will be less demanding since the frame rates are so much lower. So when I coded this up I had optimized for the non-fragged case with the fragmented case being more of an afterthought needed mostly as exception handling.

Quoted 15 lines

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

What is the test you saw the 2% performance improvement in? Is it something XDP related or a full stack test?

Not XDP, it's not present in this driver at this point :D Stack test, but without usercopy overhead. Trafgen bombs the NIC, the driver builds skbs and passes it up the stack, the stack does GRO etc, and then the frames get dropped on IP input because there's no socket.

Quoted 4 lines

So one thing you might want to look at would be a full stack test w/ something such as netperf versus optimizing for a drop only test. Otherwise that can lead to optimizations that will actually hurt driver performance in the long run.

Quoted 7 lines

I was doing some netperf (or that Microsoft's tool, don't remember the name) tests, but the problem is that usercopy is such a bottleneck, so that you don't notice any optimizations or regressions most of time. Also, userspace tools usually just pass huge payload chunks and then the drivers GSO them into MTU-sized frames, so you always get line rate and that's it. Short frames or interleave/imix (randomly-mix-sized) are the most stressful from my experience and are able to show actual outcome.

That is kind of what I figured. So one thing to watch out for is stating performance improvements without providing context on what exactly it is you are doing to see that gain. So essentially what we have is a microbenchmark that is seeing the gain.

Admittedly my goto used to be IPv4 routing since that exercised both the Tx and Rx path for much the same reason. However one thing you need to keep in mind is that if you cannot see a gain in the full-stack test odds are most users may not notice much of an impact.

Quoted 36 lines
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Also one thing I am not a huge fan of is a patch that is really a patchset onto itself. With all 6 items called out here I would have preferred to see this as 6 patches as it would have been easier to review.

Quoted 5 lines

Agree BTW, I'm not a fan of this patch either. I wasn't sure what to do with it, as splitting it into 6 explodes the series into a monster, but proceeding without it increases diffstat and complicates things later on. I'll try the latter, but will see. 17 patches is not the End of Days after all.

Quoted 5 lines

One thing you may want to consider to condense some of these patches would be to look at possibly combining patches 4 and 5 which disable recycling and use a full 4K page. It seems like of those patches one ends up redoing the other since so many of the dma_sync calls are updated in both.

Or maybe I'll move this one into the subsequent series, since it's only pt. 1 of Rx optimizations. There's also the second commit, but it's probably as messy as this one and these two could be just converted into a series.

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Just a nit. You might want to break this up into two statements like I had before. I know some people within Intel weren't a huge fan of when I used to do that kind of thing all the time in loops where I would do the decrement and test in one line.. :)

Should I please them or do it as I want to? :D I realize from the compiler's PoV it's most likely the same, but dunno, why not.

Quoted 7 lines
Quoted 4 lines

If nobody internally is bugging you about it then I am fine with it. I just know back during my era people would complain about that from a maintainability perspective. I guess I got trained to catch those kind of things as a result.

Haha understand. I usually say: "please some good arguments or I didn't hear this", maybe that's why nobody complained on --var yet :D

Either that or they were already worn down by the time you started adding this type of stuff.. :)

The one I used to do that would really drive people nuts was:

    for (i = loop_count; i--;)

It is more efficient since I don't have to do the comparison to the loop counter, but it is definitely counterintuitive to run loops backwards like that. I tried to break myself of the habit of using those sort of loops anywhere that wasn't performance critical such as driver init.

Quoted 5 lines

[...]

Quoted 3 lines
Quoted 2 lines

Yes, I'm optimizing all this out later in the series. I was surprised just as much as you when I saw skb getting passed to do nothing ._.

Quoted 7 lines
Quoted 4 lines

The funny part for me is that it is like reviewing code written via a game of telephone. I recognize the code but have to think about it since there are all the bits of changes and such from the original ixgbe.

Lots of things are still recognizable even in IDPF. That's how this series was born... :D

Yep, now the question is how many drivers can be pulled into using this library. The issue is going to be all the extra features and workarounds outside of your basic Tx/Rx will complicate the code since all the drivers implement them a bit differently. One of the reasons for not consolidating them was to allow for performance optimizing for each driver. By combining them you are going to likely need to add a number of new conditional paths to the fast path.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

Quoted 7 lines
Quoted 4 lines

No problem, it was the least I could do since I am responsible for so much of this code in the earlier drivers anyway. If nothing else I figured I could provide a bit of history on why some of this was the way it was.

These history bits are nice and interesting to read actually! And also useful since they give some context and understanding of what is obsolete and can be removed/changed.

Yeah, it is easiest to do these sort of refactors when you have somebody to answer the "why" of most of this. I recall going through this when I was refactoring the igb/ixgbe drivers back in the day and having to purge the dead e1000 code throughout. Of course, after this refactor it will be all yours right?.. :D
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#13 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

From: Alexander Duyck <[email protected]>
Date: Fri, 2 Jun 2023 08:04:53 -0700

Quoted 2 lines

On Fri, Jun 2, 2023 at 7:00 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 3 lines

That's for Rx path, but don't forget that the initial allocation on ifup is done in the process context. That's what the maintainers and reviewers usually warn about: to not allocate with %GFP_ATOMIC on ifups.

I can see that for the static values like the queue vectors and rings, however for the buffers themselves, but I don't see the point in doing that for the regular buffer allocations. Basically it is adding overhead for something that should have minimal impact as it usually happens early on during boot when the memory should be free anyway so GFP_ATOMIC vs GFP_KERNEL wouldn't have much impact in either case

Queue vectors and rings get allocated earlier than buffers, on device probing :D ifup happens later and it depends on the networking scripts etc. -- now every init system enables all the interfaces when booting up like systemd does. Plus, ifdowns-ifups can occur during the normal system functioning -- resets, XDP setup/remove, Ethtool configuration, and so on. I wouldn't say Rx buffer allocation happens only on early boot.

[...]

Quoted 8 lines
Quoted 5 lines

Oh, I feel like I'm starting to agree :D OK, then the following doesn't really get out of my head: why do we store skb pointer on the ring then, if we count 1 skb as 1 unit, so that we won't leave the loop until the EOP? Only to handle allocation failures? But skb is already allocated at this point... <confused>

The skb is there to essentially hold the frags. Keep in mind that when ixgbe was coded up XDP didn't exist yet.

Ok, maybe I phrased it badly.
If we don't stop the loop until skb is passed up the stack, how we can go out of the loop with an unfinished skb? Previously, I thought lots of drivers do that, as you may exhaust your budget prior to reaching the last fragment, so you'll get back to the skb on the next poll. But if we count 1 skb as budget unit, not descriptor, how we can end up breaking the loop prior to finishing the skb? I can imagine only one
situation: HW gave us some buffers, but still processes the EOP buffer, so we don't have any more descriptors to process, but the skb is still unfinished. But sounds weird TBH, I thought HW processes frames "atomically", i.e. it doesn't give you buffers until they hold the whole frame :D

Quoted 6 lines

I think there are drivers that are already getting away from this, such as mvneta, by storing an xdp_buff instead of an skb. In theory we could do away with most of this and just use a shared_info structure, but since that exists in the first frag we still need a pointer to the first frag as well.

ice has xdp_buff on the ring for XDP multi-buffer. It's more lightweight than skb, but also carries the frags, since frags is a part of shinfo, not skb.
It's totally fine and we'll end up doing the same here, my question was as I explained below.

Quoted 8 lines

Also multi-frag frames are typically not that likely on a normal network as most of the frames are less than 1514B in length. In addition as I mentioned before a jumbo frame workload will be less demanding since the frame rates are so much lower. So when I coded this up I had optimized for the non-fragged case with the fragmented case being more of an afterthought needed mostly as exception handling.

[...]

Quoted 9 lines

That is kind of what I figured. So one thing to watch out for is stating performance improvements without providing context on what exactly it is you are doing to see that gain. So essentially what we have is a microbenchmark that is seeing the gain.

Admittedly my goto used to be IPv4 routing since that exercised both the Tx and Rx path for much the same reason. However one thing you need to keep in mind is that if you cannot see a gain in the full-stack test odds are most users may not notice much of an impact.

Yeah sure. I think more than a half of optimizations in such drivers nowadays is unnoticeable to end users :D

[...]

Quoted 5 lines

Either that or they were already worn down by the time you started adding this type of stuff.. :)

The one I used to do that would really drive people nuts was:

    for (i = loop_count; i--;)

Oh, nice one! Never thought of something like that hehe.

Quoted 6 lines

It is more efficient since I don't have to do the comparison to the loop counter, but it is definitely counterintuitive to run loops backwards like that. I tried to break myself of the habit of using those sort of loops anywhere that wasn't performance critical such as driver init.

[...]

Quoted 7 lines

Yep, now the question is how many drivers can be pulled into using this library. The issue is going to be all the extra features and workarounds outside of your basic Tx/Rx will complicate the code since all the drivers implement them a bit differently. One of the reasons for not consolidating them was to allow for performance optimizing for each driver. By combining them you are going to likely need to add a number of new conditional paths to the fast path.

When I was counting the number of spots in the Rx polling function that need to have switch-cases/ifs in order to be able to merge the code (e.g. parsing the descriptors), it was something around 4-5 (per packet). So it can only be figured out during the testing whether adding new branches actually hurts there.
XDP is relatively easy to unify in one place, most of code is software-only. Ring structures are also easy to merge, wrapping a couple driver-specific pointers into static inline accessors. Hotpath in general is the easiest part, that's why I started from it.

But anyway, I'd say if one day I'd have to choice whether to remove 400 locs per driver with having -1% in synthetic tests or not do that at all, I'd go for the former. As discussed above, it's very unlikely for such changes to hurt real workloads, esp. with usercopy.

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

Quoted 7 lines
Quoted 4 lines

No problem, it was the least I could do since I am responsible for so much of this code in the earlier drivers anyway. If nothing else I figured I could provide a bit of history on why some of this was the way it was.

These history bits are nice and interesting to read actually! And also useful since they give some context and understanding of what is obsolete and can be removed/changed.

Quoted 5 lines

Yeah, it is easiest to do these sort of refactors when you have somebody to answer the "why" of most of this. I recall going through this when I was refactoring the igb/ixgbe drivers back in the day and having to purge the dead e1000 code throughout. Of course, after this refactor it will be all yours right?.. :D

Nope, maybe from the git-blame PoV only :p

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#14 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

Quoted 20 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Fri, 2 Jun 2023 08:04:53 -0700

Quoted 2 lines

On Fri, Jun 2, 2023 at 7:00 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 3 lines

That's for Rx path, but don't forget that the initial allocation on ifup is done in the process context. That's what the maintainers and reviewers usually warn about: to not allocate with %GFP_ATOMIC on ifups.

I can see that for the static values like the queue vectors and rings, however for the buffers themselves, but I don't see the point in doing that for the regular buffer allocations. Basically it is adding overhead for something that should have minimal impact as it usually happens early on during boot when the memory should be free anyway so GFP_ATOMIC vs GFP_KERNEL wouldn't have much impact in either case

Quoted 6 lines

Queue vectors and rings get allocated earlier than buffers, on device probing :D ifup happens later and it depends on the networking scripts etc. -- now every init system enables all the interfaces when booting up like systemd does. Plus, ifdowns-ifups can occur during the normal system functioning -- resets, XDP setup/remove, Ethtool configuration, and so on. I wouldn't say Rx buffer allocation happens only on early boot.

I agree it isn't only on early boot, but that is the most common case. The rings and such tend to get allocated early and unless there is a need for some unforeseen change the rings typically are not modified.

I just don't see the point of special casing the allocations since if they fail we will be turning around and just immediately calling the GFP_ATOMIC version within 2 seconds anyway to try and fill out the empty rings.

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Oh, I feel like I'm starting to agree :D OK, then the following doesn't really get out of my head: why do we store skb pointer on the ring then, if we count 1 skb as 1 unit, so that we won't leave the loop until the EOP? Only to handle allocation failures? But skb is already allocated at this point... <confused>

The skb is there to essentially hold the frags. Keep in mind that when ixgbe was coded up XDP didn't exist yet.

Quoted 12 lines

Ok, maybe I phrased it badly.
If we don't stop the loop until skb is passed up the stack, how we can go out of the loop with an unfinished skb? Previously, I thought lots of drivers do that, as you may exhaust your budget prior to reaching the last fragment, so you'll get back to the skb on the next poll. But if we count 1 skb as budget unit, not descriptor, how we can end up breaking the loop prior to finishing the skb? I can imagine only one
situation: HW gave us some buffers, but still processes the EOP buffer, so we don't have any more descriptors to process, but the skb is still unfinished. But sounds weird TBH, I thought HW processes frames "atomically", i.e. it doesn't give you buffers until they hold the whole frame :D

The problem is the frames aren't necessarily written back atomically. One big issue is descriptor write back. The hardware will try to cache line optimize things in order to improve performance. It is possible for a single frame to straddle either side of a cache line. As a result the first half may be written back, the driver then processes that cache line, and finds the next one isn't populated while the hardware is collecting enough descriptors to write back the next one.

It is also one of the reasons why I went to so much effort to prevent us from writing to the descriptor ring in the cleanup paths. You never know when you might be processing an earlier frame and accidently wander into a section that is in the process of being written. I think that is addressed now mostly through the use of completion queues instead of the single ring that used to process both work and completions.

Quoted 7 lines
Quoted 6 lines

I think there are drivers that are already getting away from this, such as mvneta, by storing an xdp_buff instead of an skb. In theory we could do away with most of this and just use a shared_info structure, but since that exists in the first frag we still need a pointer to the first frag as well.

Quoted 5 lines

ice has xdp_buff on the ring for XDP multi-buffer. It's more lightweight than skb, but also carries the frags, since frags is a part of shinfo, not skb.
It's totally fine and we'll end up doing the same here, my question was as I explained below.

Okay. I haven't looked at ice that closely so I wasn't aware of that.

Quoted 51 lines
Quoted 8 lines

Also multi-frag frames are typically not that likely on a normal network as most of the frames are less than 1514B in length. In addition as I mentioned before a jumbo frame workload will be less demanding since the frame rates are so much lower. So when I coded this up I had optimized for the non-fragged case with the fragmented case being more of an afterthought needed mostly as exception handling.

[...]

Quoted 9 lines

That is kind of what I figured. So one thing to watch out for is stating performance improvements without providing context on what exactly it is you are doing to see that gain. So essentially what we have is a microbenchmark that is seeing the gain.

Admittedly my goto used to be IPv4 routing since that exercised both the Tx and Rx path for much the same reason. However one thing you need to keep in mind is that if you cannot see a gain in the full-stack test odds are most users may not notice much of an impact.

Yeah sure. I think more than a half of optimizations in such drivers nowadays is unnoticeable to end users :D

[...]

Quoted 5 lines

Either that or they were already worn down by the time you started adding this type of stuff.. :)

The one I used to do that would really drive people nuts was:

    for (i = loop_count; i--;)

Oh, nice one! Never thought of something like that hehe.

Quoted 6 lines

It is more efficient since I don't have to do the comparison to the loop counter, but it is definitely counterintuitive to run loops backwards like that. I tried to break myself of the habit of using those sort of loops anywhere that wasn't performance critical such as driver init.

[...]

Quoted 7 lines

Yep, now the question is how many drivers can be pulled into using this library. The issue is going to be all the extra features and workarounds outside of your basic Tx/Rx will complicate the code since all the drivers implement them a bit differently. One of the reasons for not consolidating them was to allow for performance optimizing for each driver. By combining them you are going to likely need to add a number of new conditional paths to the fast path.

Quoted 5 lines

When I was counting the number of spots in the Rx polling function that need to have switch-cases/ifs in order to be able to merge the code (e.g. parsing the descriptors), it was something around 4-5 (per packet). So it can only be figured out during the testing whether adding new branches actually hurts there.

The other thing is you may want to double check CPU(s) you are expected to support as last I knew switch statements were still expensive due to all the old spectre/meltdown workarounds.

Quoted 9 lines

XDP is relatively easy to unify in one place, most of code is software-only. Ring structures are also easy to merge, wrapping a couple driver-specific pointers into static inline accessors. Hotpath in general is the easiest part, that's why I started from it.

But anyway, I'd say if one day I'd have to choice whether to remove 400 locs per driver with having -1% in synthetic tests or not do that at all, I'd go for the former. As discussed above, it's very unlikely for such changes to hurt real workloads, esp. with usercopy.

+1 assuming no serious regressions.

Quoted 25 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

[...]

Thanks for the detailed reviews, stuff that Intel often lacks :s :D

Quoted 7 lines
Quoted 4 lines

No problem, it was the least I could do since I am responsible for so much of this code in the earlier drivers anyway. If nothing else I figured I could provide a bit of history on why some of this was the way it was.

These history bits are nice and interesting to read actually! And also useful since they give some context and understanding of what is obsolete and can be removed/changed.

Quoted 5 lines

Yeah, it is easiest to do these sort of refactors when you have somebody to answer the "why" of most of this. I recall going through this when I was refactoring the igb/ixgbe drivers back in the day and having to purge the dead e1000 code throughout. Of course, after this refactor it will be all yours right?.. :D

Nope, maybe from the git-blame PoV only :p

Thanks,
Olek

author maintainer
#15 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

From: Alexander Duyck <[email protected]>
Date: Fri, 2 Jun 2023 10:50:02 -0700

Sorry for the silence, had sorta long weekend :p

Quoted 2 lines

On Fri, Jun 2, 2023 at 9:16 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 13 lines
Quoted 4 lines

Ok, maybe I phrased it badly.
If we don't stop the loop until skb is passed up the stack, how we can go out of the loop with an unfinished skb? Previously, I thought lots of drivers do that, as you may exhaust your budget prior to reaching the

Quoted 8 lines

last fragment, so you'll get back to the skb on the next poll. But if we count 1 skb as budget unit, not descriptor, how we can end up breaking the loop prior to finishing the skb? I can imagine only one
situation: HW gave us some buffers, but still processes the EOP buffer, so we don't have any more descriptors to process, but the skb is still unfinished. But sounds weird TBH, I thought HW processes frames "atomically", i.e. it doesn't give you buffers until they hold the whole frame :D

Quoted 7 lines

The problem is the frames aren't necessarily written back atomically. One big issue is descriptor write back. The hardware will try to cache line optimize things in order to improve performance. It is possible for a single frame to straddle either side of a cache line. As a result the first half may be written back, the driver then processes that cache line, and finds the next one isn't populated while the hardware is collecting enough descriptors to write back the next one.

Ah okay, that's was I was suspecting. So it's not atomic and skb/xdp_buff is stored on the ring to handle such cases, not budget exhausting.
Thanks for the detailed explanation. I feel 1 skb = 1 unit more logical optimal to me now :D

Quoted 8 lines

It is also one of the reasons why I went to so much effort to prevent us from writing to the descriptor ring in the cleanup paths. You never know when you might be processing an earlier frame and accidently wander into a section that is in the process of being written. I think that is addressed now mostly through the use of completion queues instead of the single ring that used to process both work and completions.

Completion rings are neat, you totally avoid writing anything to HW on Rx polling and vice versa, no descriptor read on refilling. My preference is to not refill anything on NAPI and do a separate workqueue for that, esp. given that most NICs nowadays have "refill me please" interrupt.
Please don't look at the idpf code, IIRC from what I've been told they do it the "old" way and touch both receive and refill queues on Rx polling :s :D

Quoted 7 lines
Quoted 5 lines

ice has xdp_buff on the ring for XDP multi-buffer. It's more lightweight than skb, but also carries the frags, since frags is a part of shinfo, not skb.
It's totally fine and we'll end up doing the same here, my question was as I explained below.

Okay. I haven't looked at ice that closely so I wasn't aware of that.

No prob, just FYI. This moves us one step closer to passing something more lightweight than skb up the stack in non-extreme cases, so that the stack will take care of it when GROing :)

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

Yep, now the question is how many drivers can be pulled into using this library. The issue is going to be all the extra features and workarounds outside of your basic Tx/Rx will complicate the code since all the drivers implement them a bit differently. One of the reasons for not consolidating them was to allow for performance optimizing for each driver. By combining them you are going to likely need to add a number of new conditional paths to the fast path.

Quoted 5 lines

When I was counting the number of spots in the Rx polling function that need to have switch-cases/ifs in order to be able to merge the code (e.g. parsing the descriptors), it was something around 4-5 (per packet). So it can only be figured out during the testing whether adding new branches actually hurts there.

Quoted 3 lines

The other thing is you may want to double check CPU(s) you are expected to support as last I knew switch statements were still expensive due to all the old spectre/meltdown workarounds.

Wait, are switch-cases also affected? I wasn't aware of that. For sure I didn't even consider using ops/indirect calls, but switch-cases... I saw lots o'times people replacing indirections with switch-cases, what's the point otherwise :D

Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#16 lore raw

Re: [Intel-wired-lan] [PATCH net-next v2 03/12] iavf: optimize Rx buffer allocation a bunch

Quoted 9 lines · Alexander Lobakin

From: Alexander Duyck <[email protected]>
Date: Fri, 2 Jun 2023 10:50:02 -0700

Sorry for the silence, had sorta long weekend :p

Quoted 2 lines

On Fri, Jun 2, 2023 at 9:16 AM Alexander Lobakin
<[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 3 lines

The other thing is you may want to double check CPU(s) you are expected to support as last I knew switch statements were still expensive due to all the old spectre/meltdown workarounds.

Wait, are switch-cases also affected? I wasn't aware of that. For sure I didn't even consider using ops/indirect calls, but switch-cases... I saw lots o'times people replacing indirections with switch-cases, what's the point otherwise :D

Thanks,
Olek

I think it all depends on how the compiler implements them. I know switch cases used to be done as indirections in some cases. It is possible they did away with that in the compilers some time ago or came up with a fix to work around them. I just remember back when that came up people were going through and replacing switch cases with if/else blocks in the case of performance sensitive workloads. Odds are there have been workarounds added to address that, but it is just something to be aware of.

Thanks,

- Alex
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#17 lore raw

PATCH v2 4/12 iavf: remove page splitting/recycling

In reply to Alexander Lobakin

As an intermediate step, remove all page splitting/recyclig code. Just always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
The change allows to greatly simplify certain parts of the code:

Function: add/remove: 2/3 grow/shrink: 0/5 up/down: 543/-963 (-420)

&iavf_rx_buf can even now retire in favor of just storing an array of pages used for Rx. Their DMA addresses can be stored in page::dma_addr -- use Page Pool's function for that.
No surprise perf loses up to 30% here, but that regression will go away once PP lands.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c |   2 +-
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 280 ++++++--------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |  17 +-
 3 files changed, 88 insertions(+), 211 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -1
index ade32aa1ed78..2d00be69fcde 100644
@@ -1236,7 +1236,7 @@ static void iavf_configure(struct iavf_adapter *adapter)
 	for (i = 0; i < adapter->num_active_queues; i++) {
 		struct iavf_ring *ring = &adapter->rx_rings[i];
 
-		iavf_alloc_rx_buffers(ring);
+		iavf_alloc_rx_pages(ring);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_txrx.c +85 -195

516 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -15
index 9c6661a6edf2..c09ac580fe84 100644
@@ -272,17 +272,6 @@ struct iavf_tx_buffer {
 	u32 tx_flags;
 };
 
-struct iavf_rx_buffer {
-	dma_addr_t dma;
-	struct page *page;
-#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
-	__u32 page_offset;
-#else
-	__u16 page_offset;
-#endif
-	__u16 pagecnt_bias;
-};
-
 struct iavf_queue_stats {
 	u64 packets;
 	u64 bytes;
@@ -302,8 +291,6 @@ struct iavf_rx_queue_stats {
 	u64 non_eop_descs;
 	u64 alloc_page_failed;
 	u64 alloc_buff_failed;
-	u64 page_reuse_count;
-	u64 realloc_count;
 };
 
 enum iavf_ring_state_t {
@@ -331,7 +318,7 @@ struct iavf_ring {
 	struct net_device *netdev;	/* netdev ring maps to */
 	union {
 		struct iavf_tx_buffer *tx_bi;
-		struct iavf_rx_buffer *rx_bi;
+		struct page **rx_pages;
 	};
 	DECLARE_BITMAP(state, __IAVF_RING_STATE_NBITS);
 	u16 queue_index;		/* Queue number of ring */
@@ -425,7 +412,7 @@ static inline unsigned int iavf_rx_pg_order(struct iavf_ring *ring)
 
 #define iavf_rx_pg_size(_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring))
 
-void iavf_alloc_rx_buffers(struct iavf_ring *rxr);
+void iavf_alloc_rx_pages(struct iavf_ring *rxr);
 netdev_tx_t iavf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
 void iavf_clean_tx_ring(struct iavf_ring *tx_ring);
 void iavf_clean_rx_ring(struct iavf_ring *rx_ring);
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#18 lore raw

PATCH v2 5/12 iavf: always use a full order-0 page

In reply to Alexander Lobakin

The current scheme with trying to pick the smallest buffer possible for the current MTU in order to flip/split pages is not very optimal. For example, on default MTU of 1500 it gives only 192 bytes of headroom, while XDP may require up to 258. But this also involves unnecessary code complication, which sometimes is even hard to follow.
As page split is no more, always allocate order-0 pages. This optimizes performance a bit and drops some bytes off the object code. Next, always pick the maximum buffer length available for this %PAGE_SIZE to set it up in the hardware. This means it now becomes a constant value, which also has its positive impact.
On x64 this means (without XDP):

4096 page
64 head, 320 tail
3712 HW buffer size
3686 max MTU w/o frags

Previously, the maximum MTU w/o splitting a frame into several buffers was 3046.
Increased buffer size allows us to reach the maximum frame size w/ frags supported by HW: 16382 bytes (MTU 16356). Reflect it in the netdev config as well. Relying on max single buffer size when calculating MTU was not correct.
Move around a couple of fields in &iavf_ring after ::rx_buf_len removal to reduce holes and improve cache locality.
Instead of providing the Rx definitions, which can and will be reused in rest of the drivers, exclusively for IAVF, do that in the libie header. Non-PP drivers could still use at least some of them and lose a couple copied lines.

Function: add/remove: 0/0 grow/shrink: 3/9 up/down: 18/-265 (-247)

+ even reclaims a half percent of performance, nice.

---
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  32 +-----
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   |  96 +++++++----------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 100 +-----------------
 drivers/net/ethernet/intel/iavf/iavf_type.h   |   2 -
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  15 +--
 include/linux/net/intel/libie/rx.h            |  35 ++++++
 6 files changed, 85 insertions(+), 195 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +4 -28
index 2d00be69fcde..120bb6a09ceb 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -709,32 +711,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)
  **/
 static void iavf_configure_rx(struct iavf_adapter *adapter)
 {
-	unsigned int rx_buf_len = IAVF_RXBUFFER_2048;
 	struct iavf_hw *hw = &adapter->hw;
-	int i;
-
-	if (PAGE_SIZE < 8192) {
-		struct net_device *netdev = adapter->netdev;
 
-		/* For jumbo frames on systems with 4K pages we have to use
-		 * an order 1 page, so we might as well increase the size
-		 * of our Rx buffer to make better use of the available space
-		 */
-		rx_buf_len = IAVF_RXBUFFER_3072;
-
-		/* We use a 1536 buffer size for configurations with
-		 * standard Ethernet mtu.  On x86 this gives us enough room
-		 * for shared info and 192 bytes of padding.
-		 */
-		if (!IAVF_2K_TOO_SMALL_WITH_PADDING &&
-		    (netdev->mtu <= ETH_DATA_LEN))
-			rx_buf_len = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-	}
-
-	for (i = 0; i < adapter->num_active_queues; i++) {
+	for (u32 i = 0; i < adapter->num_active_queues; i++)
 		adapter->rx_rings[i].tail = hw->hw_addr + IAVF_QRX_TAIL1(i);
-		adapter->rx_rings[i].rx_buf_len = rx_buf_len;
-	}
 }
 
 /**
@@ -2577,11 +2557,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter)
 
 	netdev->netdev_ops = &iavf_netdev_ops;
 	iavf_set_ethtool_ops(netdev);
-	netdev->watchdog_timeo = 5 * HZ;
-
-	/* MTU range: 68 - 9710 */
-	netdev->min_mtu = ETH_MIN_MTU;
-	netdev->max_mtu = IAVF_MAX_RXBUFFER - IAVF_PACKET_HDR_PAD;
+	netdev->max_mtu = LIBIE_MAX_MTU;
 
 	if (!is_valid_ether_addr(adapter->hw.mac.addr)) {
 		dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",
drivers/net/ethernet/intel/iavf/iavf_txrx.c +38 -58

236 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +3 -97

145 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_type.h +0 -2
index 3030ba330326..bb90d8f3ad7e 100644
@@ -10,8 +10,6 @@
 #include "iavf_adminq.h"
 #include "iavf_devids.h"
 
-#define IAVF_RXQ_CTX_DBUFF_SHIFT 7
-
 /* IAVF_MASK is a macro used on 32 bit registers */
 #define IAVF_MASK(mask, shift) ((u32)(mask) << (shift))
 
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +5 -10
index fdddc3588487..c726d0c91cd8 100644
@@ -1,6 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright(c) 2013 - 2018 Intel Corporation. */
 
+#include <linux/net/intel/libie/rx.h>
+
 #include "iavf.h"
 #include "iavf_prototype.h"
 #include "iavf_client.h"
@@ -269,13 +271,12 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)
 void iavf_configure_queues(struct iavf_adapter *adapter)
 {
 	struct virtchnl_vsi_queue_config_info *vqci;
-	int i, max_frame = adapter->vf_res->max_mtu;
+	u32 i, max_frame = adapter->vf_res->max_mtu;
 	int pairs = adapter->num_active_queues;
 	struct virtchnl_queue_pair_info *vqpi;
 	size_t len;
 
-	if (max_frame > IAVF_MAX_RXBUFFER || !max_frame)
-		max_frame = IAVF_MAX_RXBUFFER;
+	max_frame = min_not_zero(max_frame, LIBIE_MAX_RX_FRM_LEN);
 
 	if (adapter->current_op != VIRTCHNL_OP_UNKNOWN) {
 		/* bail because we already have a command pending */
@@ -289,10 +290,6 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 	if (!vqci)
 		return;
 
-	/* Limit maximum frame size when jumbo frames is not enabled */
-	if (adapter->netdev->mtu <= ETH_DATA_LEN)
-		max_frame = IAVF_RXBUFFER_1536 - NET_IP_ALIGN;
-
 	vqci->vsi_id = adapter->vsi_res->vsi_id;
 	vqci->num_queue_pairs = pairs;
 	vqpi = vqci->qpair;
@@ -309,9 +306,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)
 		vqpi->rxq.ring_len = adapter->rx_rings[i].count;
 		vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
 		vqpi->rxq.max_pkt_size = max_frame;
-		vqpi->rxq.databuffer_size =
-			ALIGN(adapter->rx_rings[i].rx_buf_len,
-			      BIT_ULL(IAVF_RXQ_CTX_DBUFF_SHIFT));
+		vqpi->rxq.databuffer_size = LIBIE_RX_BUF_LEN;
 		vqpi++;
 	}
 
include/linux/net/intel/libie/rx.h +35 -0
index 58bd0f35d025..3e8d0d5206e1 100644
@@ -4,6 +4,7 @@
 #ifndef __LIBIE_RX_H
 #define __LIBIE_RX_H
 
+#include <linux/if_vlan.h>
 #include <linux/netdevice.h>
 
 /* O(1) converting i40e/ice/iavf's 8/10-bit hardware packet type to a parsed
@@ -125,4 +126,38 @@ static inline void libie_skb_set_hash(struct sk_buff *skb, u32 hash,
 	skb_set_hash(skb, hash, parsed.payload_layer);
 }
 
+/* Rx MTU/buffer/truesize helpers. Mostly pure software-side; HW-defined values
+ * are valid for all Intel HW.
+ */
+
+/* Space reserved in front of each frame */
+#define LIBIE_SKB_HEADROOM	(NET_SKB_PAD + NET_IP_ALIGN)
+/* Link layer / L2 overhead: Ethernet, 2 VLAN tags (C + S), FCS */
+#define LIBIE_RX_LL_LEN		(ETH_HLEN + 2 * VLAN_HLEN + ETH_FCS_LEN)
+
+/* Truesize: total space wasted on each frame. Always use order-0 pages */
+#define LIBIE_RX_PAGE_ORDER	0
+#define LIBIE_RX_TRUESIZE	(PAGE_SIZE << LIBIE_RX_PAGE_ORDER)
+/* Rx buffer size config is a multiple of 128 */
+#define LIBIE_RX_BUF_LEN_ALIGN	128
+/* HW-writeable space in one buffer: truesize - headroom/tailroom,
+ * HW-aligned
+ */
+#define __LIBIE_RX_BUF_LEN						    \
+	ALIGN_DOWN(SKB_MAX_ORDER(LIBIE_SKB_HEADROOM, LIBIE_RX_PAGE_ORDER),  \
+		   LIBIE_RX_BUF_LEN_ALIGN)
+/* The largest size for a single descriptor as per HW */
+#define LIBIE_MAX_RX_BUF_LEN	9728U
+/* "True" HW-writeable space: minimum from SW and HW values */
+#define LIBIE_RX_BUF_LEN	min_t(u32, __LIBIE_RX_BUF_LEN,		    \
+				      LIBIE_MAX_RX_BUF_LEN)
+
+/* The maximum frame size as per HW (S/G) */
+#define __LIBIE_MAX_RX_FRM_LEN	16382U
+/* ATST, HW can chain up to 5 Rx descriptors */
+#define LIBIE_MAX_RX_FRM_LEN	min_t(u32, __LIBIE_MAX_RX_FRM_LEN,	    \
+				      LIBIE_RX_BUF_LEN * 5)
+/* Maximum frame size minus LL overhead */
+#define LIBIE_MAX_MTU		(LIBIE_MAX_RX_FRM_LEN - LIBIE_RX_LL_LEN)
+
 #endif /* __LIBIE_RX_H */
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#19 lore raw

RE: [PATCH net-next v2 05/12] iavf: always use a full order-0 page

From: Alexander Lobakin

Quoted 14 lines

Sent: 25 May 2023 13:58

The current scheme with trying to pick the smallest buffer possible for the current MTU in order to flip/split pages is not very optimal. For example, on default MTU of 1500 it gives only 192 bytes of headroom, while XDP may require up to 258. But this also involves unnecessary code complication, which sometimes is even hard to follow.
As page split is no more, always allocate order-0 pages. This optimizes performance a bit and drops some bytes off the object code. Next, always pick the maximum buffer length available for this %PAGE_SIZE to set it up in the hardware. This means it now becomes a constant value, which also has its positive impact.
On x64 this means (without XDP):

Quoted 4 lines

4096 page
64 head, 320 tail
3712 HW buffer size
3686 max MTU w/o frags

I'd have thought it was important to pack multiple buffers for MTU 1500 into a single page.
512 bytes split between head and tail room really ought to be enough for most cases.

Is much tailroom ever used for received packets?
It is used to append data to packets being sent - but that isn't really relevant here.

While the unused memory is moderate for 4k pages, it is horrid for anything with large pages - think 64k and above. IIRC large pages are common on big PPC and maybe some arm cpus.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

author maintainer
#20 lore raw

Re: [PATCH net-next v2 05/12] iavf: always use a full order-0 page

From: David Laight <[email protected]>
Date: Fri, 26 May 2023 08:57:31 +0000

Quoted 2 lines

From: Alexander Lobakin

Quoted 1 line

Sent: 25 May 2023 13:58

[...]

Quoted 11 lines
Quoted 4 lines

4096 page
64 head, 320 tail
3712 HW buffer size
3686 max MTU w/o frags

I'd have thought it was important to pack multiple buffers for MTU 1500 into a single page.
512 bytes split between head and tail room really ought to be enough for most cases.

Is much tailroom ever used for received packets?

You don't have any tailroom at all when you split 4k page into two halves on x86_64. From those 512 bytes, 320+ are used for skb_shared_info. And then you're left with 192 bytes or even less (with increased MAX_SKB_FRAGS, which becomes a new trend thanks to Eric and Big TCP). XDP requires 256 bytes of headroom -- and you already don't have them even with the default number of frags.

Quoted 5 lines

It is used to append data to packets being sent - but that isn't really relevant here.

While the unused memory is moderate for 4k pages, it is horrid for anything with large pages - think 64k and above.

But hey, there's always unused space and it's arbitrary whether to treat it "horrid". For example, imagine a machine mostly handling 64-byte traffic. From that point of view, even splitting pages into 2 halves still is "horrid" -- 2048 bytes of truesize only to receive 64 bytes :>

Quoted 1 line

IIRC large pages are common on big PPC and maybe some arm cpus.

Even in that case, the percentage of 4k-page machines running those NICs are much higher than > 4k. I thought we usually optimize things for the most common usecases. Adding a couple hundred lines, branches on hotpaths, limitations etc. only to serve a particular architecture... Dunno. I have 16k pages on my private development machines and I have no issues with using 1 page per frame in their NIC drivers :s

Quoted 7 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

Thanks,
Olek

author maintainer
#21 lore raw

PATCH v2 6/12 net: skbuff: don't include <net/page_pool.h> into <linux/skbuff.h>

In reply to Alexander Lobakin

Currently, touching <net/page_pool.h> triggers a rebuild of more than a half of the kernel. That's because it's included in <linux/skbuff.h>.

In 6a5bcd84e886 ("page_pool: Allow drivers to hint on SKB recycling"), Matteo included it to be able to call a couple functions defined there. Then, in 57f05bc2ab24 ("page_pool: keep pp info as long as page pool owns the page") one of the calls was removed, so only one left. It's call to page_pool_return_skb_page() in napi_frag_unref(). The function is external and doesn't have any dependencies. Having include of very niche page_pool.h only for that looks like an overkill. Instead, move the declaration of that function to skbuff.h itself, with a small comment that it's a special guest and should not be touched. Now, after a few include fixes in the drivers, touching page_pool.h only triggers rebuilding of the drivers using it and a couple core networking files.

---
 drivers/net/ethernet/engleder/tsnep_main.c          | 1 +
 drivers/net/ethernet/freescale/fec_main.c           | 1 +
 drivers/net/ethernet/mellanox/mlx5/core/en/params.c | 1 +
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c    | 1 +
 drivers/net/wireless/mediatek/mt76/mt76.h           | 1 +
 include/linux/skbuff.h                              | 4 +++-
 include/net/page_pool.h                             | 2 --
 7 files changed, 8 insertions(+), 3 deletions(-)
drivers/net/ethernet/engleder/tsnep_main.c +1 -0
index 84751bb303a6..6222aaa5157f 100644
@@ -28,6 +28,7 @@
 #include <linux/iopoll.h>
 #include <linux/bpf.h>
 #include <linux/bpf_trace.h>
+#include <net/page_pool.h>
 #include <net/xdp_sock_drv.h>
 
 #define TSNEP_RX_OFFSET (max(NET_SKB_PAD, XDP_PACKET_HEADROOM) + NET_IP_ALIGN)
drivers/net/ethernet/freescale/fec_main.c +1 -0
index 3ecf20ee5851..6ef162f8ed33 100644
@@ -38,6 +38,7 @@
 #include <linux/in.h>
 #include <linux/ip.h>
 #include <net/ip.h>
+#include <net/page_pool.h>
 #include <net/selftests.h>
 #include <net/tso.h>
 #include <linux/tcp.h>
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +1 -0
index 9c94807097cb..3235a3a4ed08 100644
@@ -6,6 +6,7 @@
 #include "en/port.h"
 #include "en_accel/en_accel.h"
 #include "en_accel/ipsec.h"
+#include <net/page_pool.h>
 #include <net/xdp_sock_drv.h>
 
 static u8 mlx5e_mpwrq_min_page_shift(struct mlx5_core_dev *mdev)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +1 -0
index f0e6095809fa..1bd91bc09eb8 100644
@@ -35,6 +35,7 @@
 #include "en/xdp.h"
 #include "en/params.h"
 #include <linux/bitfield.h>
+#include <net/page_pool.h>
 
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk)
 {
drivers/net/wireless/mediatek/mt76/mt76.h +1 -0
index 6b07b8fafec2..95c16f11d156 100644
@@ -15,6 +15,7 @@
 #include <linux/average.h>
 #include <linux/soc/mediatek/mtk_wed.h>
 #include <net/mac80211.h>
+#include <net/page_pool.h>
 #include "util.h"
 #include "testmode.h"
 
include/linux/skbuff.h +3 -1
index 8cff3d817131..163d3b2d00cb 100644
@@ -32,7 +32,6 @@
 #include <linux/if_packet.h>
 #include <linux/llist.h>
 #include <net/flow.h>
-#include <net/page_pool.h>
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 #include <linux/netfilter/nf_conntrack_common.h>
 #endif
@@ -3412,6 +3411,9 @@ static inline void skb_frag_ref(struct sk_buff *skb, int f)
 	__skb_frag_ref(&skb_shinfo(skb)->frags[f]);
 }
 
+/* Internal from net/core/page_pool.c, do not use in drivers directly */
+bool page_pool_return_skb_page(struct page *page, bool napi_safe);
+
 static inline void
 napi_frag_unref(skb_frag_t *frag, bool recycle, bool napi_safe)
 {
include/net/page_pool.h +0 -2
index c8ec2f34722b..821c75bba125 100644
@@ -240,8 +240,6 @@ inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)
 	return pool->p.dma_dir;
 }
 
-bool page_pool_return_skb_page(struct page *page, bool napi_safe);
-
 struct page_pool *page_pool_create(const struct page_pool_params *params);
 
 struct xdp_mem_info;
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
maintainer
#22 lore raw

Re: [PATCH net-next v2 06/12] net: skbuff: don't include <net/page_pool.h> into <linux/skbuff.h>

Quoted 15 lines · On Thu, 25 May 2023 14:57:40 +0200 Alexander Lobakin wrote:

Currently, touching <net/page_pool.h> triggers a rebuild of more than a half of the kernel. That's because it's included in <linux/skbuff.h>.

In 6a5bcd84e886 ("page_pool: Allow drivers to hint on SKB recycling"), Matteo included it to be able to call a couple functions defined there. Then, in 57f05bc2ab24 ("page_pool: keep pp info as long as page pool owns the page") one of the calls was removed, so only one left. It's call to page_pool_return_skb_page() in napi_frag_unref(). The function is external and doesn't have any dependencies. Having include of very niche page_pool.h only for that looks like an overkill. Instead, move the declaration of that function to skbuff.h itself, with a small comment that it's a special guest and should not be touched. Now, after a few include fixes in the drivers, touching page_pool.h only triggers rebuilding of the drivers using it and a couple core networking files.

drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c

came in the meantime, and did not bother including page_pool.h.

Signature · 1 line
pw-bot: cr
author maintainer
#23 lore raw

Re: [PATCH net-next v2 06/12] net: skbuff: don't include <net/page_pool.h> into <linux/skbuff.h>

From: Jakub Kicinski <[email protected]>
Date: Fri, 26 May 2023 20:54:10 -0700

Quoted 17 lines · On Thu, 25 May 2023 14:57:40 +0200 Alexander Lobakin wrote:
Quoted 15 lines

Currently, touching <net/page_pool.h> triggers a rebuild of more than a half of the kernel. That's because it's included in <linux/skbuff.h>.

In 6a5bcd84e886 ("page_pool: Allow drivers to hint on SKB recycling"), Matteo included it to be able to call a couple functions defined there. Then, in 57f05bc2ab24 ("page_pool: keep pp info as long as page pool owns the page") one of the calls was removed, so only one left. It's call to page_pool_return_skb_page() in napi_frag_unref(). The function is external and doesn't have any dependencies. Having include of very niche page_pool.h only for that looks like an overkill. Instead, move the declaration of that function to skbuff.h itself, with a small comment that it's a special guest and should not be touched. Now, after a few include fixes in the drivers, touching page_pool.h only triggers rebuilding of the drivers using it and a couple core networking files.

Quoted 3 lines

drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c

came in the meantime, and did not bother including page_pool.h.

Hah, I saw it coming, but thought they did include it :D Will fix in a bit, thanks for noticing!

Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

author maintainer
#24 lore raw

PATCH v2 7/12 net: page_pool: avoid calling no-op externals when possible

In reply to Alexander Lobakin

Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles even when on DMA-coherent platforms (like x86) with no active IOMMU or swiotlb, just for the call ladder.
Indeed, it's

page_pool_put_page()
page_pool_put_defragged_page() <- external
__page_pool_put_page()
page_pool_dma_sync_for_device() <- non-inline
dma_sync_single_range_for_device()
dma_sync_single_for_device() <- external
dma_direct_sync_single_for_device()
dev_is_dma_coherent() <- exit

For the inline functions, no guarantees the compiler won't uninline them (they're clearly not one-liners and sometimes compilers uninline even 2 + 2). The first external call is necessary, but the rest 2+ are done for nothing each time, plus a bunch of checks here and there. Since Page Pool mappings are long-term and for one "device + addr" pair dma_need_sync() will always return the same value (basically, whether it belongs to an swiotlb pool), addresses can be tested once right after they're obtained and the result can be reused until the page is unmapped. Define new PP flag, which will mean "do DMA syncs for device, but only when needed" and turn it on by default when the driver asks to sync pages. When a page is mapped, check whether it needs syncs and if so, replace that "sync when needed" back to "always do syncs" globally for the whole pool (better safe than sorry). As long as a pool has no pages requiring DMA syncs, this cuts off a good piece of calls and checks. On my x86_64, this gives from 2% to 5% performance benefit with no negative impact for cases when IOMMU is on and the shortcut can't be used.

---
 include/net/page_pool.h |  3 +++
 net/core/page_pool.c    | 10 ++++++++++
 2 files changed, 13 insertions(+)
include/net/page_pool.h +3 -0
index 821c75bba125..08e9571d2545 100644
@@ -46,6 +46,9 @@
 					* device driver responsibility
 					*/
 #define PP_FLAG_PAGE_FRAG	BIT(2) /* for page frag feature */
+#define PP_FLAG_DMA_MAYBE_SYNC	BIT(3) /* Internal, should not be used in
+					* drivers
+					*/
 #define PP_FLAG_ALL		(PP_FLAG_DMA_MAP |\
 				 PP_FLAG_DMA_SYNC_DEV |\
 				 PP_FLAG_PAGE_FRAG)
net/core/page_pool.c +10 -0
index e212e9d7edcb..57f323dee6c4 100644
@@ -175,6 +175,10 @@ static int page_pool_init(struct page_pool *pool,
 		/* pool->p.offset has to be set according to the address
 		 * offset used by the DMA engine to start copying rx data
 		 */
+
+		/* Try to avoid calling no-op syncs */
+		pool->p.flags |= PP_FLAG_DMA_MAYBE_SYNC;
+		pool->p.flags &= ~PP_FLAG_DMA_SYNC_DEV;
 	}
 
 	if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT &&
@@ -323,6 +327,12 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page)
 
 	page_pool_set_dma_addr(page, dma);
 
+	if ((pool->p.flags & PP_FLAG_DMA_MAYBE_SYNC) &&
+	    dma_need_sync(pool->p.dev, dma)) {
+		pool->p.flags |= PP_FLAG_DMA_SYNC_DEV;
+		pool->p.flags &= ~PP_FLAG_DMA_MAYBE_SYNC;
+	}
+
 	if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
 		page_pool_dma_sync_for_device(pool, page, pool->p.max_len);
 
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#25 lore raw

PATCH v2 8/12 net: page_pool: add DMA-sync-for-CPU inline helpers

In reply to Alexander Lobakin

Each driver is responsible for syncing buffers written by HW for CPU before accessing them. Almost each PP-enabled driver uses the same pattern, which could be shorthanded into a static inline to make driver code a little bit more compact.
Introduce a couple such functions. The first one takes the actual size of the data written by HW and is the main one to be used on Rx. The second does the same, but only if the PP performs DMA synchronizations at all. The last one picks max_len from the PP params and is designed for more extreme cases when the size is unknown, but the buffer still needs to be synced.
Also constify pointer arguments of page_pool_get_dma_dir() and page_pool_get_dma_addr() to give a bit more room for optimization, as both of them are read-only.

---
 include/net/page_pool.h | 64 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 60 insertions(+), 4 deletions(-)
include/net/page_pool.h +60 -4
index 08e9571d2545..080778e8cf84 100644
@@ -32,7 +32,7 @@
 
 #include <linux/mm.h> /* Needed by ptr_ring */
 #include <linux/ptr_ring.h>
-#include <linux/dma-direction.h>
+#include <linux/dma-mapping.h>
 
 #define PP_FLAG_DMA_MAP		BIT(0) /* Should page_pool do the DMA
 					* map/unmap
@@ -237,8 +237,8 @@ static inline struct page *page_pool_dev_alloc_frag(struct page_pool *pool,
 /* get the stored dma direction. A driver might decide to treat this locally and
  * avoid the extra cache line from page_pool to determine the direction
  */
-static
-inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)
+static inline enum dma_data_direction
+page_pool_get_dma_dir(const struct page_pool *pool)
 {
 	return pool->p.dma_dir;
 }
@@ -361,7 +361,7 @@ static inline void page_pool_recycle_direct(struct page_pool *pool,
 #define PAGE_POOL_DMA_USE_PP_FRAG_COUNT	\
 		(sizeof(dma_addr_t) > sizeof(unsigned long))
 
-static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
+static inline dma_addr_t page_pool_get_dma_addr(const struct page *page)
 {
 	dma_addr_t ret = page->dma_addr;
 
@@ -378,6 +378,62 @@ static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
 		page->dma_addr_upper = upper_32_bits(addr);
 }
 
+/**
+ * __page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by HW
+ * @pool: page_pool which this page belongs to
+ * @page: page to sync
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Can be used as a shorthand to sync Rx pages before accessing them in the
+ * driver. Caller must ensure the pool was created with %PP_FLAG_DMA_MAP.
+ * Note that this version performs DMA sync unconditionally, even if the
+ * associated PP doesn't perform sync-for-device. Consider the non-underscored
+ * version first if unsure.
+ */
+static inline void __page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+						const struct page *page,
+						u32 dma_sync_size)
+{
+	dma_sync_single_range_for_cpu(pool->p.dev,
+				      page_pool_get_dma_addr(page),
+				      pool->p.offset, dma_sync_size,
+				      page_pool_get_dma_dir(pool));
+}
+
+/**
+ * page_pool_dma_sync_for_cpu - sync Rx page for CPU if needed
+ * @pool: page_pool which this page belongs to
+ * @page: page to sync
+ * @dma_sync_size: size of the data written to the page
+ *
+ * Performs DMA sync for CPU, but *only* when:
+ * 1) page_pool was created with %PP_FLAG_DMA_SYNC_DEV to manage DMA syncs;
+ * 2) AND sync shortcut is not available (IOMMU, swiotlb, non-coherent DMA, ...)
+ */
+static inline void page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+					      const struct page *page,
+					      u32 dma_sync_size)
+{
+	if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+		__page_pool_dma_sync_for_cpu(pool, page, dma_sync_size);
+}
+
+/**
+ * page_pool_dma_sync_full_for_cpu - sync full Rx page for CPU (if needed)
+ * @pool: page_pool which this page belongs to
+ * @page: page to sync
+ *
+ * Performs sync for the entire length exposed to hardware. Can be used on
+ * DMA errors or before freeing the page, when it's unknown whether the HW
+ * touched the buffer.
+ */
+static inline void
+page_pool_dma_sync_full_for_cpu(const struct page_pool *pool,
+				const struct page *page)
+{
+	page_pool_dma_sync_for_cpu(pool, page, pool->p.max_len);
+}
+
 static inline bool is_page_pool_compiled_in(void)
 {
 #ifdef CONFIG_PAGE_POOL
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#26 lore raw

PATCH v2 9/12 iavf: switch to Page Pool

In reply to Alexander Lobakin

Now that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics and one whole page per frame, it can easily be switched to using Page Pool API instead.
Introduce libie_rx_page_pool_create(), a wrapper for creating a PP with the default libie settings applicable to all Intel hardware, and replace the alloc/free calls with the corresponding PP functions, including the newly added sync-for-CPU helpers. Use skb_mark_for_recycle() to bring back the recycling and restore the initial performance.

From the important object code changes, worth mentioning that __iavf_alloc_rx_pages() is now inlined due to the greatly reduced size. The resulting driver is on par with the pre-series code and 1-2% slower than the "optimized" version right before the recycling removal. But the number of locs and object code bytes slaughtered is much more important here after all, not speaking of that there's still a vast space for optimization and improvements.

---
 drivers/net/ethernet/intel/Kconfig          |   1 +
 drivers/net/ethernet/intel/iavf/iavf_txrx.c | 126 +++++---------------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h |   8 +-
 drivers/net/ethernet/intel/libie/rx.c       |  28 +++++
 include/linux/net/intel/libie/rx.h          |   5 +-
 5 files changed, 69 insertions(+), 99 deletions(-)
drivers/net/ethernet/intel/Kconfig +1 -0
index cec4a938fbd0..a368afc42b8d 100644
@@ -86,6 +86,7 @@ config E1000E_HWTS
 
 config LIBIE
 	tristate
+	select PAGE_POOL
 	help
 	  libie (Intel Ethernet library) is a common library containing
 	  routines shared by several Intel Ethernet drivers.
drivers/net/ethernet/intel/iavf/iavf_txrx.c +32 -94

252 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +4 -4
index 1421e90c7c4e..8fbe549ce6a5 100644
@@ -83,9 +83,6 @@ enum iavf_dyn_idx_t {
 
 #define iavf_rx_desc iavf_32byte_rx_desc
 
-#define IAVF_RX_DMA_ATTR \
-	(DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
-
 /**
  * iavf_test_staterr - tests bits in Rx descriptor status and error fields
  * @rx_desc: pointer to receive descriptor (in le64 format)
@@ -240,7 +237,10 @@ struct iavf_rx_queue_stats {
 struct iavf_ring {
 	struct iavf_ring *next;		/* pointer to next ring in q_vector */
 	void *desc;			/* Descriptor ring memory */
-	struct device *dev;		/* Used for DMA mapping */
+	union {
+		struct page_pool *pool;	/* Used for Rx page management */
+		struct device *dev;	/* Used for DMA mapping on Tx */
+	};
 	struct net_device *netdev;	/* netdev ring maps to */
 	union {
 		struct iavf_tx_buffer *tx_bi;
drivers/net/ethernet/intel/libie/rx.c +28 -0
index f503476d8eef..d68eab76593c 100644
@@ -105,6 +105,34 @@ const struct libie_rx_ptype_parsed libie_rx_ptype_lut[LIBIE_RX_PTYPE_NUM] = {
 };
 EXPORT_SYMBOL_NS_GPL(libie_rx_ptype_lut, LIBIE);
 
+/* Page Pool */
+
+/**
+ * libie_rx_page_pool_create - create a PP with the default libie settings
+ * @napi: &napi_struct covering this PP (no usage outside its poll loops)
+ * @size: size of the PP, usually simply Rx queue len
+ *
+ * Returns &page_pool on success, casted -errno on failure.
+ */
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size)
+{
+	const struct page_pool_params pp = {
+		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
+		.order		= LIBIE_RX_PAGE_ORDER,
+		.pool_size	= size,
+		.nid		= NUMA_NO_NODE,
+		.dev		= napi->dev->dev.parent,
+		.napi		= napi,
+		.dma_dir	= DMA_FROM_DEVICE,
+		.max_len	= LIBIE_RX_BUF_LEN,
+		.offset		= LIBIE_SKB_HEADROOM,
+	};
+
+	return page_pool_create(&pp);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);
+
 MODULE_AUTHOR("Intel Corporation");
 MODULE_DESCRIPTION("Intel(R) Ethernet common library");
 MODULE_LICENSE("GPL");
include/linux/net/intel/libie/rx.h +4 -1
index 3e8d0d5206e1..b86cadd281f1 100644
@@ -5,7 +5,7 @@
 #define __LIBIE_RX_H
 
 #include <linux/if_vlan.h>
-#include <linux/netdevice.h>
+#include <net/page_pool.h>
 
 /* O(1) converting i40e/ice/iavf's 8/10-bit hardware packet type to a parsed
  * bitfield struct.
@@ -160,4 +160,7 @@ static inline void libie_skb_set_hash(struct sk_buff *skb, u32 hash,
 /* Maximum frame size minus LL overhead */
 #define LIBIE_MAX_MTU		(LIBIE_MAX_RX_FRM_LEN - LIBIE_RX_LL_LEN)
 
+struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
+					    u32 size);
+
 #endif /* __LIBIE_RX_H */
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#27 lore raw

PATCH v2 10/12 libie: add common queue stats

In reply to Alexander Lobakin

Next stop, per-queue private stats. They have only subtle differences from driver to driver and can easily be resolved.
Define common structures, inline helpers and Ethtool helpers to collect, update and export the statistics. Use u64_stats_t right from the start, as well as the corresponding helpers to ensure tear-free operations. For the NAPI parts of both Rx and Tx, also define small onstack containers to update them in polling loops and then sync the actual containers once a loop ends.
The drivers will be switched to use this API later on a per-driver basis, along with conversion to PP.

---
 drivers/net/ethernet/intel/libie/Makefile |   1 +
 drivers/net/ethernet/intel/libie/stats.c  | 119 ++++++++++++++
 include/linux/net/intel/libie/stats.h     | 179 ++++++++++++++++++++++
 3 files changed, 299 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/libie/stats.c
 create mode 100644 include/linux/net/intel/libie/stats.h
drivers/net/ethernet/intel/libie/Makefile +1 -0
index 95e81d09b474..76f32253481b 100644
@@ -4,3 +4,4 @@
 obj-$(CONFIG_LIBIE)	+= libie.o
 
 libie-objs		+= rx.o
+libie-objs		+= stats.o
drivers/net/ethernet/intel/libie/stats.c new file +119 -0

119 lines not on this page · raw message

include/linux/net/intel/libie/stats.h new file +179 -0

179 lines not on this page · raw message

Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#28 lore raw

PATCH v2 11/12 libie: add per-queue Page Pool stats

In reply to Alexander Lobakin

Expand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.
They are also a bit special in terms of how they are obtained. One &page_pool accumulates statistics until it's destroyed obviously, which happens on ifdown. So, in order to not lose any statistics, get the stats and store in the queue container before destroying a pool. This container survives ifups/downs, so it basically stores the statistics accumulated since the very first pool was allocated on this queue. When it's needed to export the stats, first get the numbers from this container and then add the "live" numbers -- the ones that the current active pool returns. The result values will always represent the actual device-lifetime* stats.
There's a cast from &page_pool_stats to u64 * in a couple functions, but they are guarded with stats asserts to make sure it's safe to do. FWIW it saves a lot of object code.

---
 drivers/net/ethernet/intel/libie/internal.h | 23 +++++++
 drivers/net/ethernet/intel/libie/rx.c       | 20 ++++++
 drivers/net/ethernet/intel/libie/stats.c    | 73 ++++++++++++++++++++-
 include/linux/net/intel/libie/rx.h          |  4 ++
 include/linux/net/intel/libie/stats.h       | 39 ++++++++++-
 5 files changed, 156 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/libie/internal.h
drivers/net/ethernet/intel/libie/internal.h new file +23 -0
new file mode 100644
index 000000000000..083398dc37c6
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* libie internal declarations not to be used in drivers.
+ *
+ * Copyright(c) 2023 Intel Corporation.
+ */
+
+#ifndef __LIBIE_INTERNAL_H
+#define __LIBIE_INTERNAL_H
+
+struct libie_rq_stats;
+struct page_pool;
+
+#ifdef CONFIG_PAGE_POOL_STATS
+void libie_rq_stats_sync_pp(struct libie_rq_stats *stats,
+			    struct page_pool *pool);
+#else
+static inline void libie_rq_stats_sync_pp(struct libie_rq_stats *stats,
+					  struct page_pool *pool)
+{
+}
+#endif
+
+#endif /* __LIBIE_INTERNAL_H */
drivers/net/ethernet/intel/libie/rx.c +20 -0
index d68eab76593c..128f134aca6d 100644
@@ -3,6 +3,8 @@
 
 #include <linux/net/intel/libie/rx.h>
 
+#include "internal.h"
+
 /* O(1) converting i40e/ice/iavf's 8/10-bit hardware packet type to a parsed
  * bitfield struct.
  */
@@ -133,6 +135,24 @@ struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
 }
 EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_create, LIBIE);
 
+/**
+ * libie_rx_page_pool_destroy - destroy a &page_pool created by libie
+ * @pool: pool to destroy
+ * @stats: RQ stats from the ring (or %NULL to skip updating PP stats)
+ *
+ * As the stats usually has the same lifetime as the device, but PP is usually
+ * created/destroyed on ifup/ifdown, in order to not lose the stats accumulated
+ * during the last ifup, the PP stats need to be added to the driver stats
+ * container. Then the PP gets destroyed.
+ */
+void libie_rx_page_pool_destroy(struct page_pool *pool,
+				struct libie_rq_stats *stats)
+{
+	libie_rq_stats_sync_pp(stats, pool);
+	page_pool_destroy(pool);
+}
+EXPORT_SYMBOL_NS_GPL(libie_rx_page_pool_destroy, LIBIE);
+
 MODULE_AUTHOR("Intel Corporation");
 MODULE_DESCRIPTION("Intel(R) Ethernet common library");
 MODULE_LICENSE("GPL");
drivers/net/ethernet/intel/libie/stats.c +72 -1

98 lines not on this page · raw message

include/linux/net/intel/libie/rx.h +4 -0
index b86cadd281f1..d401feb17928 100644
@@ -160,7 +160,11 @@ static inline void libie_skb_set_hash(struct sk_buff *skb, u32 hash,
 /* Maximum frame size minus LL overhead */
 #define LIBIE_MAX_MTU		(LIBIE_MAX_RX_FRM_LEN - LIBIE_RX_LL_LEN)
 
+struct libie_rq_stats;
+
 struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi,
 					    u32 size);
+void libie_rx_page_pool_destroy(struct page_pool *pool,
+				struct libie_rq_stats *stats);
 
 #endif /* __LIBIE_RX_H */
include/linux/net/intel/libie/stats.h +37 -2
index dbbc98bbd3a7..23ca0079a905 100644
@@ -49,6 +49,17 @@
  * fragments: number of processed descriptors carrying only a fragment
  * alloc_page_fail: number of Rx page allocation fails
  * build_skb_fail: number of build_skb() fails
+ * pp_alloc_fast: pages taken from the cache or ring
+ * pp_alloc_slow: actual page allocations
+ * pp_alloc_slow_ho: non-order-0 page allocations
+ * pp_alloc_empty: number of times the pool was empty
+ * pp_alloc_refill: number of cache refills
+ * pp_alloc_waive: NUMA node mismatches during recycling
+ * pp_recycle_cached: direct recyclings into the cache
+ * pp_recycle_cache_full: number of times the cache was full
+ * pp_recycle_ring: recyclings into the ring
+ * pp_recycle_ring_full: number of times the ring was full
+ * pp_recycle_released_ref: pages released due to elevated refcnt
  */
 
 #define DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
@@ -60,9 +71,29 @@
 	act(alloc_page_fail)				\
 	act(build_skb_fail)
 
+#ifdef CONFIG_PAGE_POOL_STATS
+#define DECLARE_LIBIE_RQ_PP_STATS(act)			\
+	act(pp_alloc_fast)				\
+	act(pp_alloc_slow)				\
+	act(pp_alloc_slow_ho)				\
+	act(pp_alloc_empty)				\
+	act(pp_alloc_refill)				\
+	act(pp_alloc_waive)				\
+	act(pp_recycle_cached)				\
+	act(pp_recycle_cache_full)			\
+	act(pp_recycle_ring)				\
+	act(pp_recycle_ring_full)			\
+	act(pp_recycle_released_ref)
+#else
+#define DECLARE_LIBIE_RQ_PP_STATS(act)
+#endif
+
 #define DECLARE_LIBIE_RQ_STATS(act)			\
 	DECLARE_LIBIE_RQ_NAPI_STATS(act)		\
-	DECLARE_LIBIE_RQ_FAIL_STATS(act)
+	DECLARE_LIBIE_RQ_FAIL_STATS(act)		\
+	DECLARE_LIBIE_RQ_PP_STATS(act)
+
+struct page_pool;
 
 struct libie_rq_stats {
 	struct u64_stats_sync	syncp;
@@ -72,6 +103,9 @@ struct libie_rq_stats {
 #define act(s)	u64_stats_t	s;
 			DECLARE_LIBIE_RQ_NAPI_STATS(act);
 			DECLARE_LIBIE_RQ_FAIL_STATS(act);
+			struct_group(pp,
+				DECLARE_LIBIE_RQ_PP_STATS(act);
+			);
 #undef act
 		};
 		DECLARE_FLEX_ARRAY(u64_stats_t, raw);
@@ -110,7 +144,8 @@ libie_rq_napi_stats_add(struct libie_rq_stats *qs,
 
 u32 libie_rq_stats_get_sset_count(void);
 void libie_rq_stats_get_strings(u8 **data, u32 qid);
-void libie_rq_stats_get_data(u64 **data, const struct libie_rq_stats *stats);
+void libie_rq_stats_get_data(u64 **data, const struct libie_rq_stats *stats,
+			     struct page_pool *pool);
 
 /* Tx per-queue stats:
  * packets: packets sent from this queue
Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
author maintainer
#29 lore raw

PATCH v2 12/12 iavf: switch queue stats to libie

In reply to Alexander Lobakin

iavf is pretty much ready for using the generic libie stats, so drop all the custom code and just use generic definitions. The only thing is that it previously lacked the counter of Tx queue stops. It's present in the other drivers, so add it here as well.
The rest is straightforward. There were two fields in the Tx stats struct, which didn't belong there. The first one has never been used, wipe it; and move the other to the queue structure. Plus move around a couple fields in &iavf_ring to account stats structs' alignment.

---
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 87 ++++------------
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  2 +
 drivers/net/ethernet/intel/iavf/iavf_txrx.c   | 98 ++++++++++---------
 drivers/net/ethernet/intel/iavf/iavf_txrx.h   | 47 +++------
 4 files changed, 87 insertions(+), 147 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +20 -67
index de3050c02b6f..0dcf50d75f86 100644
@@ -46,16 +46,6 @@ struct iavf_stats {
 	.stat_offset = offsetof(_type, _stat) \
 }
 
-/* Helper macro for defining some statistics related to queues */
-#define IAVF_QUEUE_STAT(_name, _stat) \
-	IAVF_STAT(struct iavf_ring, _name, _stat)
-
-/* Stats associated with a Tx or Rx ring */
-static const struct iavf_stats iavf_gstrings_queue_stats[] = {
-	IAVF_QUEUE_STAT("%s-%u.packets", stats.packets),
-	IAVF_QUEUE_STAT("%s-%u.bytes", stats.bytes),
-};
-
 /**
  * iavf_add_one_ethtool_stat - copy the stat into the supplied buffer
  * @data: location to store the stat value
@@ -141,43 +131,6 @@ __iavf_add_ethtool_stats(u64 **data, void *pointer,
 #define iavf_add_ethtool_stats(data, pointer, stats) \
 	__iavf_add_ethtool_stats(data, pointer, stats, ARRAY_SIZE(stats))
 
-/**
- * iavf_add_queue_stats - copy queue statistics into supplied buffer
- * @data: ethtool stats buffer
- * @ring: the ring to copy
- *
- * Queue statistics must be copied while protected by
- * u64_stats_fetch_begin, so we can't directly use iavf_add_ethtool_stats.
- * Assumes that queue stats are defined in iavf_gstrings_queue_stats. If the
- * ring pointer is null, zero out the queue stat values and update the data
- * pointer. Otherwise safely copy the stats from the ring into the supplied
- * buffer and update the data pointer when finished.
- *
- * This function expects to be called while under rcu_read_lock().
- **/
-static void
-iavf_add_queue_stats(u64 **data, struct iavf_ring *ring)
-{
-	const unsigned int size = ARRAY_SIZE(iavf_gstrings_queue_stats);
-	const struct iavf_stats *stats = iavf_gstrings_queue_stats;
-	unsigned int start;
-	unsigned int i;
-
-	/* To avoid invalid statistics values, ensure that we keep retrying
-	 * the copy until we get a consistent value according to
-	 * u64_stats_fetch_retry. But first, make sure our ring is
-	 * non-null before attempting to access its syncp.
-	 */
-	do {
-		start = !ring ? 0 : u64_stats_fetch_begin(&ring->syncp);
-		for (i = 0; i < size; i++)
-			iavf_add_one_ethtool_stat(&(*data)[i], ring, &stats[i]);
-	} while (ring && u64_stats_fetch_retry(&ring->syncp, start));
-
-	/* Once we successfully copy the stats in, update the data pointer */
-	*data += size;
-}
-
 /**
  * __iavf_add_stat_strings - copy stat strings into ethtool buffer
  * @p: ethtool supplied buffer
@@ -237,8 +190,6 @@ static const struct iavf_stats iavf_gstrings_stats[] = {
 
 #define IAVF_STATS_LEN	ARRAY_SIZE(iavf_gstrings_stats)
 
-#define IAVF_QUEUE_STATS_LEN	ARRAY_SIZE(iavf_gstrings_queue_stats)
-
 /**
  * iavf_get_link_ksettings - Get Link Speed and Duplex settings
  * @netdev: network interface device structure
@@ -308,18 +259,22 @@ static int iavf_get_link_ksettings(struct net_device *netdev,
  **/
 static int iavf_get_sset_count(struct net_device *netdev, int sset)
 {
-	/* Report the maximum number queues, even if not every queue is
-	 * currently configured. Since allocation of queues is in pairs,
-	 * use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set
-	 * at device creation and never changes.
-	 */
+	u32 num;
 
-	if (sset == ETH_SS_STATS)
-		return IAVF_STATS_LEN +
-			(IAVF_QUEUE_STATS_LEN * 2 *
-			 netdev->real_num_tx_queues);
-	else
+	switch (sset) {
+	case ETH_SS_STATS:
+		/* Per-queue */
+		num = libie_rq_stats_get_sset_count();
+		num += libie_sq_stats_get_sset_count();
+		num *= netdev->real_num_tx_queues;
+
+		/* Global */
+		num += IAVF_STATS_LEN;
+
+		return num;
+	default:
 		return -EINVAL;
+	}
 }
 
 /**
@@ -346,15 +301,15 @@ static void iavf_get_ethtool_stats(struct net_device *netdev,
 	 * it to iterate over rings' stats.
 	 */
 	for (i = 0; i < adapter->num_active_queues; i++) {
-		struct iavf_ring *ring;
+		const struct iavf_ring *ring;
 
 		/* Tx rings stats */
-		ring = &adapter->tx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		libie_sq_stats_get_data(&data, &adapter->tx_rings[i].sq_stats);
 
 		/* Rx rings stats */
 		ring = &adapter->rx_rings[i];
-		iavf_add_queue_stats(&data, ring);
+		libie_rq_stats_get_data(&data, &ring->rq_stats,
+					ring->rx_pages ? ring->pool : NULL);
 	}
 	rcu_read_unlock();
 }
@@ -376,10 +331,8 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data)
 	 * real_num_tx_queues for both Tx and Rx queues.
 	 */
 	for (i = 0; i < netdev->real_num_tx_queues; i++) {
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "tx", i);
-		iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
-				      "rx", i);
+		libie_sq_stats_get_strings(&data, i);
+		libie_rq_stats_get_strings(&data, i);
 	}
 }
 
drivers/net/ethernet/intel/iavf/iavf_main.c +2 -0
index 120bb6a09ceb..4a702795ddb3 100644
@@ -1581,6 +1581,7 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		tx_ring->itr_setting = IAVF_ITR_TX_DEF;
 		if (adapter->flags & IAVF_FLAG_WB_ON_ITR_CAPABLE)
 			tx_ring->flags |= IAVF_TXR_FLAGS_WB_ON_ITR;
+		u64_stats_init(&tx_ring->sq_stats.syncp);
 
 		rx_ring = &adapter->rx_rings[i];
 		rx_ring->queue_index = i;
@@ -1588,6 +1589,7 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter)
 		rx_ring->dev = &adapter->pdev->dev;
 		rx_ring->count = adapter->rx_desc_count;
 		rx_ring->itr_setting = IAVF_ITR_RX_DEF;
+		u64_stats_init(&rx_ring->rq_stats.syncp);
 	}
 
 	adapter->num_active_queues = num_active_queues;
drivers/net/ethernet/intel/iavf/iavf_txrx.c +51 -47

274 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_txrx.h +14 -33

74 lines not on this page · raw message

Signature · 6 lines
2.40.1

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan