net: intel: start The Great Code Dedup + Page Pool for iavf
Alexander Lobakin
maintainerv4 cover net: intel: start The Great Code Dedup + Page Pool for iavf
Here's a two-shot: introduce Intel Ethernet common library (libie) and switch iavf to Page Pool. Details in the commit messages; here's summary:
Not a secret there's a ton of code duplication between two and more Intel ethernet modules. Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers.
The first thing that came to my mind was "libie" -- "Intel Ethernet common library". Also this sounds like "lovelie" and can be expanded as "lib Internet Explorer" :P I'm open for anything else (but justified). The series is only the beginning. From now on, adding every new feature or doing any good driver refactoring will remove much more lines than add for quite some time. There's a basic roadmap with some deduplications planned already, not speaking of that touching every line now asks: "can I share this?".
PP conversion for iavf lands within the same series as these two are tied closely. libie will support Page Pool model only, so a driver can't use much of the lib until it's converted. iavf is only the example, the rest will eventually be converted soon on a per-driver basis. That is when it gets really interesting. Stay tech.
Alexander Lobakin (9):
net: intel: introduce Intel Ethernet common library
iavf: kill "legacy-rx" for good
iavf: drop page splitting and recycling
net: page_pool: add DMA-sync-for-CPU inline helpers
libie: add Rx buffer management (via Page Pool)
iavf: switch to Page Pool
libie: add common queue stats
libie: add per-queue Page Pool stats
iavf: switch queue stats to libie
MAINTAINERS | 3 +- drivers/net/ethernet/intel/Kconfig | 11 + drivers/net/ethernet/intel/Makefile | 1 + drivers/net/ethernet/intel/i40e/i40e_common.c | 253 -------- drivers/net/ethernet/intel/i40e/i40e_main.c | 1 + .../net/ethernet/intel/i40e/i40e_prototype.h | 7 - drivers/net/ethernet/intel/i40e/i40e_txrx.c | 74 +-- drivers/net/ethernet/intel/i40e/i40e_type.h | 88 --- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- drivers/net/ethernet/intel/iavf/iavf_common.c | 253 -------- .../net/ethernet/intel/iavf/iavf_ethtool.c | 226 +------ drivers/net/ethernet/intel/iavf/iavf_main.c | 44 +- .../net/ethernet/intel/iavf/iavf_prototype.h | 7 - drivers/net/ethernet/intel/iavf/iavf_txrx.c | 608 ++++-------------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 176 +---- drivers/net/ethernet/intel/iavf/iavf_type.h | 90 --- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 20 +- .../net/ethernet/intel/ice/ice_lan_tx_rx.h | 316 --------- drivers/net/ethernet/intel/ice/ice_main.c | 1 + drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 74 +-- drivers/net/ethernet/intel/libie/Makefile | 7 + drivers/net/ethernet/intel/libie/internal.h | 23 + drivers/net/ethernet/intel/libie/rx.c | 183 ++++++ drivers/net/ethernet/intel/libie/stats.c | 190 ++++++ include/linux/net/intel/libie/rx.h | 241 +++++++ include/linux/net/intel/libie/stats.h | 214 ++++++ include/net/page_pool.h | 49 +- 27 files changed, 1137 insertions(+), 2025 deletions(-) create mode 100644 drivers/net/ethernet/intel/libie/Makefile create mode 100644 drivers/net/ethernet/intel/libie/internal.h create mode 100644 drivers/net/ethernet/intel/libie/rx.c create mode 100644 drivers/net/ethernet/intel/libie/stats.c create mode 100644 include/linux/net/intel/libie/rx.h create mode 100644 include/linux/net/intel/libie/stats.h
---
Directly to net-next, has non-Intel code change (#4) :p
Based on the PP "hybrid" allocation series[0] and requires it to work.
From v3[1]:
* base on the latest net-next, update bloat-o-meter and perf stats;
* split generic PP optimizations into a separate series;
* drop "optimize hotpath a bunch" commit: a lot of [controversial] changes in one place, worth own series (Alex);
* 02: pick Rev-by (Alex);
* 03: move in-place recycling removal here from the dropped patch;
* 05: new, add libie Rx buffer API separatelly from IAVF changes;
* 05-06: use new "hybrid" allocation API from[0] to reduce memory usage when a page can fit more than 1 truesize (also asked by David);
* 06: merge with "always use order-0 page" commit to reduce diffs and simplify things (Alex);
* 09: fix page_alloc_fail counter.
From v2[2]:
* 0006: fix page_pool.h include in OcteonTX2 files (Jakub, Patchwork);
* no functional changes.
From v1[3]:
* 0006: new (me, Jakub);
* 0008: give the helpers more intuitive names (Jakub, Ilias);
* -^-: also expand their kdoc a bit for the same reason;
* -^-: fix kdoc copy-paste issue (Patchwork, Jakub);
* 0011: drop inline from C file (Patchwork, Jakub).
[0] https://lore.kernel.org/netdev/[email protected]
[1] https://lore.kernel.org/netdev/[email protected]
[2] https://lore.kernel.org/netdev/[email protected]
[3] https://lore.kernel.org/netdev/[email protected]
Signature · 1 line
2.41.0
Alexander Lobakin
maintainerRFC PATCH v4 1/9 net: intel: introduce Intel Ethernet common library
Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, add unlikely() condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as a pkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. But skb_set_hash() only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hidden Kconfig symbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.
--- MAINTAINERS | 3 +- drivers/net/ethernet/intel/Kconfig | 10 + drivers/net/ethernet/intel/Makefile | 1 + drivers/net/ethernet/intel/i40e/i40e_common.c | 253 -------------- drivers/net/ethernet/intel/i40e/i40e_main.c | 1 + .../net/ethernet/intel/i40e/i40e_prototype.h | 7 - drivers/net/ethernet/intel/i40e/i40e_txrx.c | 74 +--- drivers/net/ethernet/intel/i40e/i40e_type.h | 88 ----- drivers/net/ethernet/intel/iavf/iavf_common.c | 253 -------------- drivers/net/ethernet/intel/iavf/iavf_main.c | 1 + .../net/ethernet/intel/iavf/iavf_prototype.h | 7 - drivers/net/ethernet/intel/iavf/iavf_txrx.c | 70 +--- drivers/net/ethernet/intel/iavf/iavf_type.h | 88 ----- .../net/ethernet/intel/ice/ice_lan_tx_rx.h | 316 ------------------ drivers/net/ethernet/intel/ice/ice_main.c | 1 + drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 74 +--- drivers/net/ethernet/intel/libie/Makefile | 6 + drivers/net/ethernet/intel/libie/rx.c | 110 ++++++ include/linux/net/intel/libie/rx.h | 128 +++++++ 19 files changed, 312 insertions(+), 1179 deletions(-) create mode 100644 drivers/net/ethernet/intel/libie/Makefile create mode 100644 drivers/net/ethernet/intel/libie/rx.c create mode 100644 include/linux/net/intel/libie/rx.h
MAINTAINERS +2 -1
@@ -10388,7 +10388,8 @@ F: Documentation/networking/device_drivers/ethernet/intel/ F: drivers/net/ethernet/intel/ F: drivers/net/ethernet/intel/*/ F: include/linux/avf/virtchnl.h -F: include/linux/net/intel/iidc.h +F: include/linux/net/intel/ +F: include/linux/net/intel/*/ INTEL ETHERNET PROTOCOL DRIVER FOR RDMA M: Mustafa Ismail <[email protected]>
drivers/net/ethernet/intel/Kconfig +10 -0
@@ -84,6 +84,12 @@ config E1000E_HWTS devices. The cross-timestamp is available through the PTP clock driver precise cross-timestamp ioctl (PTP_SYS_OFFSET_PRECISE). +config LIBIE + tristate + help + libie (Intel Ethernet library) is a common library containing + routines shared by several Intel Ethernet drivers. + config IGB tristate "Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support" depends on PCI
@@ -225,6 +231,7 @@ config I40E depends on PTP_1588_CLOCK_OPTIONAL depends on PCI select AUXILIARY_BUS + select LIBIE help This driver supports Intel(R) Ethernet Controller XL710 Family of devices. For more information on how to identify your adapter, go
@@ -252,6 +259,8 @@ config I40E_DCB # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF config IAVF tristate + select LIBIE + config I40EVF tristate "Intel(R) Ethernet Adaptive Virtual Function support" select IAVF
@@ -282,6 +291,7 @@ config ICE depends on GNSS || GNSS = n select AUXILIARY_BUS select DIMLIB + select LIBIE select NET_DEVLINK select PLDMFW help
drivers/net/ethernet/intel/Makefile +1 -0
@@ -15,3 +15,4 @@ obj-$(CONFIG_I40E) += i40e/ obj-$(CONFIG_IAVF) += iavf/ obj-$(CONFIG_FM10K) += fm10k/ obj-$(CONFIG_ICE) += ice/ +obj-$(CONFIG_LIBIE) += libie/
drivers/net/ethernet/intel/i40e/i40e_common.c +0 -253
259 lines not on this page · raw message
drivers/net/ethernet/intel/i40e/i40e_main.c +1 -0
@@ -97,6 +97,7 @@ MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXXMODULE_AUTHOR ("Intel Corporation, <[email protected]>" );MODULE_DESCRIPTION ("Intel(R) Ethernet Connection XL710 Network Driver" ); +MODULE_IMPORT_NS (LIBIE );MODULE_LICENSE ("GPL v2" );static struct workqueue_struct *i40e_wq;
drivers/net/ethernet/intel/i40e/i40e_prototype.h +0 -7
@@ -380,13 +380,6 @@ void i40e_set_pci_config_data(struct i40e_hw *hw, u16 link_status);int i40e_set_mac_type (struct i40e_hw *hw); -extern struct i40e_rx_ptype_decoded i40e_ptype_lookup[]; - -static inline struct i40e_rx_ptype_decoded decode_rx_desc_ptype (u8 ptype) -{ -return i40e_ptype_lookup[ptype]; -} -/** * i40e_virtchnl_link_speed - Convert AdminQ link_speed to virtchnl definition * @link_speed: the speed to convert
drivers/net/ethernet/intel/i40e/i40e_txrx.c +18 -56
132 lines not on this page · raw message
drivers/net/ethernet/intel/i40e/i40e_type.h +0 -88
94 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_common.c +0 -253
259 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -0
@@ -46,6 +46,7 @@ MODULE_DEVICE_TABLE(pci, iavf_pci_tbl);MODULE_ALIAS ("i40evf" );MODULE_AUTHOR ("Intel Corporation, <[email protected]>" );MODULE_DESCRIPTION ("Intel(R) Ethernet Adaptive Virtual Function Network Driver" ); +MODULE_IMPORT_NS (LIBIE );MODULE_LICENSE ("GPL v2" );static const struct net_device_ops iavf_netdev_ops;
drivers/net/ethernet/intel/iavf/iavf_prototype.h +0 -7
@@ -47,13 +47,6 @@ enum iavf_status iavf_aq_set_rss_key(struct iavf_hw *hw, u16 seid,enum iavf_status iavf_set_mac_type (struct iavf_hw *hw); -extern struct iavf_rx_ptype_decoded iavf_ptype_lookup[]; - -static inline struct iavf_rx_ptype_decoded decode_rx_desc_ptype (u8 ptype) -{ -return iavf_ptype_lookup[ptype]; -} -void iavf_vf_parse_hw_config (struct iavf_hw *hw,struct virtchnl_vf_resource *msg); enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/net/ethernet/intel/iavf/iavf_txrx.c +16 -54
125 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_type.h +0 -88
94 lines not on this page · raw message
drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h +0 -316
326 lines not on this page · raw message
drivers/net/ethernet/intel/ice/ice_main.c +1 -0
@@ -34,6 +34,7 @@ static const char ice_copyright[] = "Copyright (c) 2018, Intel Corporation.";MODULE_AUTHOR ("Intel Corporation, <[email protected]>" );MODULE_DESCRIPTION (DRV_SUMMARY ); +MODULE_IMPORT_NS (LIBIE );MODULE_LICENSE ("GPL v2" );MODULE_FIRMWARE (ICE_DDP_PKG_FILE );
drivers/net/ethernet/intel/ice/ice_txrx_lib.c +18 -56
137 lines not on this page · raw message
drivers/net/ethernet/intel/libie/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright(c) 2023 Intel Corporation. + +obj-$(CONFIG_LIBIE) += libie.o + +libie-objs += rx.o
drivers/net/ethernet/intel/libie/rx.c new file +110 -0
110 lines not on this page · raw message
include/linux/net/intel/libie/rx.h new file +128 -0
128 lines not on this page · raw message
Signature · 1 line
2.41.0
Re: [Intel-wired-lan] [PATCH RFC net-next v4 1/9] net: intel: introduce Intel Ethernet common library
Quoted 45 lines · Alexander Lobakin
Not a secret there's a ton of code duplication between two and more Intel ethernet modules.
Before introducing new changes, which would need to be copied over again, start decoupling the already existing duplicate functionality into a new module, which will be shared between several Intel Ethernet drivers. Add the lookup table which converts 8/10-bit hardware packet type into a parsed bitfield structure for easy checking packet format parameters, such as payload level, IP version, etc. This is currently used by i40e, ice and iavf and it's all the same in all three drivers.
The only difference introduced in this implementation is that instead of defining a 256 (or 1024 in case of ice) element array, addunlikely()condition to limit the input to 154 (current maximum non-reserved packet type). There's no reason to waste 600 (or even 3600) bytes only to not hurt very unlikely exception packets.
The hash computation function now takes payload level directly as apkt_hash_type. There's a couple cases when non-IP ptypes are marked as L3 payload and in the previous versions their hash level would be 2, not
3. Butskb_set_hash()only sees difference between L4 and non-L4, thus this won't change anything at all.
The module is behind the hiddenKconfigsymbol, which the drivers will select when needed. The exports are behind 'LIBIE' namespace to limit the scope of the functions.Signed-off-by Alexander Lobakin <[email protected]>--- MAINTAINERS | 3 +- drivers/net/ethernet/intel/Kconfig | 10 + drivers/net/ethernet/intel/Makefile | 1 + drivers/net/ethernet/intel/i40e/i40e_common.c | 253 -------------- drivers/net/ethernet/intel/i40e/i40e_main.c | 1 + .../net/ethernet/intel/i40e/i40e_prototype.h | 7 - drivers/net/ethernet/intel/i40e/i40e_txrx.c | 74 +--- drivers/net/ethernet/intel/i40e/i40e_type.h | 88 ----- drivers/net/ethernet/intel/iavf/iavf_common.c | 253 -------------- drivers/net/ethernet/intel/iavf/iavf_main.c | 1 + .../net/ethernet/intel/iavf/iavf_prototype.h | 7 - drivers/net/ethernet/intel/iavf/iavf_txrx.c | 70 +--- drivers/net/ethernet/intel/iavf/iavf_type.h | 88 ----- .../net/ethernet/intel/ice/ice_lan_tx_rx.h | 316 ------------------ drivers/net/ethernet/intel/ice/ice_main.c | 1 + drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 74 +--- drivers/net/ethernet/intel/libie/Makefile | 6 + drivers/net/ethernet/intel/libie/rx.c | 110 ++++++ include/linux/net/intel/libie/rx.h | 128 +++++++ 19 files changed, 312 insertions(+), 1179 deletions(-)
Quoted 3 lines
create mode 100644
drivers/net/ethernet/intel/libie/Makefilecreate mode 100644drivers/net/ethernet/intel/libie/rx.ccreate mode 100644include/linux/net/intel/libie/rx.h
Thanks a lot for whole effort on this series, it's refreshing to see a move in that direction.
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 2/9 iavf: kill "legacy-rx" for good
In reply to Alexander LobakinEver since build_skb() became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.
* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it implies memcpy() of packet headers (40+ bytes per each frame);
* the actual header length was calculated via eth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.
Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.
There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft, and so on. Some stats:
Function: add/remove: 0/4 grow/shrink: 0/7 up/down: 0/-757 (-757) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)
--- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 140 ------------------ drivers/net/ethernet/intel/iavf/iavf_main.c | 10 +- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 84 +---------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 27 +--- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 3 +- 6 files changed, 8 insertions(+), 258 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
@@ -298,7 +298,7 @@ struct iavf_adapter {#define IAVF_FLAG_CLIENT_NEEDS_L2_PARAMS BIT(12)#define IAVF_FLAG_PROMISC_ON BIT(13)#define IAVF_FLAG_ALLMULTI_ON BIT(14) -#define IAVF_FLAG_LEGACY_RX BIT(15) +/* BIT(15) is free, was IAVF_FLAG_LEGACY_RX */ #define IAVF_FLAG_REINIT_ITR_NEEDED BIT(16)#define IAVF_FLAG_QUEUES_DISABLED BIT(17)#define IAVF_FLAG_SETUP_NETDEV_FEATURES BIT(18)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +0 -140
181 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_main.c +1 -9
@@ -702,9 +702,7 @@ static void iavf_configure_rx(struct iavf_adapter *adapter)struct iavf_hw *hw = &adapter->hw;int i; -/* Legacy Rx will always default to a 2048 buffer size. */ -#if (PAGE_SIZE <8192 ) -if (!(adapter->flags &IAVF_FLAG_LEGACY_RX )) { +if (PAGE_SIZE <8192 ) {struct net_device *netdev = adapter->netdev;/* For jumbo frames on systems with 4K pages we have to use
@@ -721,16 +719,10 @@ static void iavf_configure_rx(struct iavf_adapter *adapter) (netdev->mtu <= ETH_DATA_LEN)) rx_buf_len =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; } -#endif for (i =0 ; i < adapter->num_active_queues; i++) { adapter->rx_rings[i].tail = hw->hw_addr +IAVF_QRX_TAIL1 (i); adapter->rx_rings[i].rx_buf_len = rx_buf_len; - -if (adapter->flags &IAVF_FLAG_LEGACY_RX ) -clear_ring_build_skb_enabled (&adapter->rx_rings[i]); -else -set_ring_build_skb_enabled (&adapter->rx_rings[i]); } }
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3 -81
115 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_txrx.h +2 -25
@@ -82,20 +82,11 @@ enum iavf_dyn_idx_t { BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP))/* Supported Rx Buffer Sizes (a multiple of 128) */ -#define IAVF_RXBUFFER_256 256#define IAVF_RXBUFFER_1536 1536/* 128B aligned standard Ethernet frame */ #define IAVF_RXBUFFER_2048 2048#define IAVF_RXBUFFER_3072 3072/* Used for large frames w/ padding */ #define IAVF_MAX_RXBUFFER 9728/* largest size for single descriptor */ -/* NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we -* reserve 2 more, and skb_shared_info adds an additional 384 bytes more, -* this adds up to 512 bytes of extra data meaning the smallest allocation -* we could have is 1K. -* i.e. RXBUFFER_256 --> 960 byte skb (size-1024 slab) -* i.e. RXBUFFER_512 --> 1216 byte skb (size-2048 slab) -*/ -#define IAVF_RX_HDR_SIZE IAVF_RXBUFFER_256#define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))#define iavf_rx_desc iavf_32byte_rx_desc
@@ -362,7 +353,8 @@ struct iavf_ring {u16 flags;#define IAVF_TXR_FLAGS_WB_ON_ITR BIT(0) -#define IAVF_RXR_FLAGS_BUILD_SKB_ENABLED BIT(1) +/* BIT(1) is free, was IAVF_RXR_FLAGS_BUILD_SKB_ENABLED */ +/* BIT(2) is free */ #define IAVF_TXRX_FLAGS_VLAN_TAG_LOC_L2TAG1 BIT(3)#define IAVF_TXR_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(4)#define IAVF_RXR_FLAGS_VLAN_TAG_LOC_L2TAG2_2 BIT(5)
@@ -393,21 +385,6 @@ struct iavf_ring {*/ }____cacheline_internodealigned_in_smp ; -static inline bool ring_uses_build_skb (struct iavf_ring *ring) -{ -return !!(ring->flags &IAVF_RXR_FLAGS_BUILD_SKB_ENABLED ); -} - -static inline void set_ring_build_skb_enabled (struct iavf_ring *ring) -{ - ring->flags |=IAVF_RXR_FLAGS_BUILD_SKB_ENABLED ; -} - -static inline void clear_ring_build_skb_enabled (struct iavf_ring *ring) -{ - ring->flags &= ~IAVF_RXR_FLAGS_BUILD_SKB_ENABLED ; -} -#define IAVF_ITR_ADAPTIVE_MIN_INC 0x0002#define IAVF_ITR_ADAPTIVE_MIN_USECS 0x0002#define IAVF_ITR_ADAPTIVE_MAX_USECS 0x007e
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -2
@@ -290,8 +290,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter)return ;/* Limit maximum frame size when jumbo frames is not enabled */ -if (!(adapter->flags &IAVF_FLAG_LEGACY_RX ) && - (adapter->netdev->mtu <=ETH_DATA_LEN )) +if (adapter->netdev->mtu <=ETH_DATA_LEN ) max_frame =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; vqci->vsi_id = adapter->vsi_res->vsi_id;
Signature · 1 line
2.41.0
Re: [Intel-wired-lan] [PATCH RFC net-next v4 2/9] iavf: kill "legacy-rx" for good
Quoted 35 lines · Alexander Lobakin
Ever since
build_skb()became stable, the old way with allocating an skb for storing the headers separately, which will be then copied manually, was slower, less flexible and thus obsolete.* it had higher pressure on MM since it actually allocates new pages, which then get split and refcount-biased (NAPI page cache);
* it impliesmemcpy()of packet headers (40+ bytes per each frame);
* the actual header length was calculated viaeth_get_headlen(), which invokes Flow Dissector and thus wastes a bunch of CPU cycles;
* XDP makes it even more weird since it requires headroom for long and also tailroom for some time (since mbuf landed). Take a look at the ice driver, which is built around work-arounds to make XDP work with it.Even on some quite low-end hardware (not a common case for 100G NICs) it was performing worse.
The only advantage "legacy-rx" had is that it didn't require any reserved headroom and tailroom. But iavf didn't use this, as it always splits pages into two halves of 2k, while that save would only be useful when striding. And again, XDP effectively removes that sole pro.There's a train of features to land in IAVF soon: Page Pool, XDP, XSk, multi-buffer etc. Each new would require adding more and more Danse Macabre for absolutely no reason, besides making hotpath less and less effective.
Remove the "feature" with all the related code. This includes at least one very hot branch (typically hit on each new frame), which was either always-true or always-false at least for a complete NAPI bulk of 64 frames, the whole private flags cruft, and so on. Some stats:Function: add/remove: 0/4 grow/shrink: 0/7 up/down: 0/-757 (-757) RO Data: add/remove: 0/1 grow/shrink: 0/0 up/down: 0/-40 (-40)
Reviewed-by Alexander Duyck <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>
Quoted 8 lines
--- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 140 ------------------ drivers/net/ethernet/intel/iavf/iavf_main.c | 10 +- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 84 +---------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 27 +--- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 3 +- 6 files changed, 8 insertions(+), 258 deletions(-)
Good one, there were some random questions in my mind during the read, but all are resolved by subsequent patches.
(It's a pity I have not found yet time to fully read them though)
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 3/9 iavf: drop page splitting and recycling
In reply to Alexander LobakinAs an intermediate step, remove all page splitting/recyclig code. Just always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63 &iavf_rx_buf structures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:
Function: add/remove: 0/2 grow/shrink: 1/3 up/down: 3/-494 (-491)
Although the array of &iavf_rx_buf is barely used now and could be replaced with just page pointer array, don't touch it now to not complicate replacing it with libie Rx buffer struct later on. No surprise perf loses up to 30% here, but that regression will go away once PP lands.
--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 151 ++------------------ drivers/net/ethernet/intel/iavf/iavf_txrx.h | 8 -- 2 files changed, 13 insertions(+), 146 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +13 -138
@@ -723,7 +723,7 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)DMA_FROM_DEVICE , IAVF_RX_DMA_ATTR); -__page_frag_cache_drain (rx_bi->page, rx_bi->pagecnt_bias); +__free_pages (rx_bi->page,iavf_rx_pg_order (rx_ring)); rx_bi->page =NULL ; rx_bi->page_offset =0 ;
@@ -735,7 +735,6 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)/* Zero out the descriptor ring */ memset (rx_ring->desc,0 , rx_ring->size); - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ; }
@@ -791,7 +790,6 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring) goto err; } - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;
@@ -811,9 +809,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val) { rx_ring->next_to_use = val; -/* update next to alloc since we have filled the ring */ - rx_ring->next_to_alloc = val; -/* Force memory writes to complete before letting h/w * know there are new descriptors to fetch. (Only * applicable for weak-ordered memory model archs,
@@ -837,12 +832,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,struct page *page = bi->page;dma_addr_t dma; -/* since we are recycling buffers we should seldom need to alloc */ -if (likely (page)) { - rx_ring->rx_stats.page_reuse_count++; -return true; - } -/* alloc new page for storage */ page =dev_alloc_pages (iavf_rx_pg_order (rx_ring));if (unlikely (!page)) {
@@ -869,9 +858,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring, bi->page = page; bi->page_offset =IAVF_SKB_PAD ; -/* initialize pagecnt_bias to 1 representing we fully own page */ - bi->pagecnt_bias =1 ; -return true; }
@@ -1103,91 +1089,6 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)return false; } -/** -* iavf_reuse_rx_page - page flip buffer and store it back on the ring -* @rx_ring: rx descriptor ring to store buffers on -* @old_buff: donor buffer to have page reused -* -* Synchronizes page for reuse by the adapter -**/ -static void iavf_reuse_rx_page (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *old_buff) -{ -struct iavf_rx_buffer *new_buff; -u16 nta = rx_ring->next_to_alloc; - - new_buff = &rx_ring->rx_bi[nta]; - -/* update, and store next to alloc */ - nta++; - rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; - -/* transfer page from old buffer to new buffer */ - new_buff->dma = old_buff->dma; - new_buff->page = old_buff->page; - new_buff->page_offset = old_buff->page_offset; - new_buff->pagecnt_bias = old_buff->pagecnt_bias; -} - -/** -* iavf_can_reuse_rx_page - Determine if this page can be reused by -* the adapter for another receive -* -* @rx_buffer: buffer containing the page -* -* If page is reusable, rx_buffer->page_offset is adjusted to point to -* an unused region in the page. -* -* For small pages, @truesize will be a constant value, half the size -* of the memory at page. We'll attempt to alternate between high and -* low halves of the page, with one half ready for use by the hardware -* and the other half being consumed by the stack. We use the page -* ref count to determine whether the stack has finished consuming the -* portion of this page that was passed up with a previous packet. If -* the page ref count is >1, we'll assume the "other" half page is -* still busy, and this page cannot be reused. -* -* For larger pages, @truesize will be the actual space used by the -* received packet (adjusted upward to an even multiple of the cache -* line size). This will advance through the page by the amount -* actually consumed by the received packets while there is still -* space for a buffer. Each region of larger pages will be used at -* most once, after which the page will not be reused. -* -* In either case, if the page is reusable its refcount is increased. -**/ -static bool iavf_can_reuse_rx_page (struct iavf_rx_buffer *rx_buffer) -{ -unsigned int pagecnt_bias = rx_buffer->pagecnt_bias; -struct page *page = rx_buffer->page; - -/* Is any reuse possible? */ -if (!dev_page_is_reusable (page)) -return false; - -#if (PAGE_SIZE <8192 ) -/* if we are only owner of page we can reuse it */ -if (unlikely ((page_count (page) - pagecnt_bias) >1 )) -return false; -#else -#define IAVF_LAST_OFFSET \ - (SKB_WITH_OVERHEAD(PAGE_SIZE) - IAVF_RXBUFFER_2048) -if (rx_buffer->page_offset >IAVF_LAST_OFFSET ) -return false; -#endif - -/* If we have drained the page fragment pool we need to update -* the pagecnt_bias and page count so that we fully restock the -* number of references the driver holds. -*/ -if (unlikely (!pagecnt_bias)) { -page_ref_add (page,USHRT_MAX ); - rx_buffer->pagecnt_bias =USHRT_MAX ; - } - -return true; -} -/** * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff * @rx_ring: rx descriptor ring to transact packets on
@@ -1216,13 +1117,6 @@ static void iavf_add_rx_frag(struct iavf_ring *rx_ring,skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, rx_buffer->page, rx_buffer->page_offset, size, truesize); - -/* page is being used so we must update the page offset */ -#if (PAGE_SIZE <8192 ) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif }/**
@@ -1250,9 +1144,6 @@ static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring, size, DMA_FROM_DEVICE); -/* We have pulled a buffer for use, so decrement pagecnt_bias */ - rx_buffer->pagecnt_bias--; -return rx_buffer; }
@@ -1293,23 +1184,15 @@ static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,skb_reserve (skb,IAVF_SKB_PAD );__skb_put (skb, size); -/* buffer is used by skb, update page_offset */ -#if (PAGE_SIZE <8192 ) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif -return skb; }/** -* iavf_put_rx_buffer - Clean up used buffer and either recycle or free +* iavf_put_rx_buffer - Unmap used buffer * @rx_ring: rx descriptor ring to transact packets on * @rx_buffer: rx buffer to pull data from * -* This function will clean up the contents of the rx_buffer. It will -* either recycle the buffer or unmap it and free the associated resources. +* This function will unmap the buffer after it's written by HW. */ static void iavf_put_rx_buffer (struct iavf_ring *rx_ring,struct iavf_rx_buffer *rx_buffer)
@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,if (!rx_buffer)return ; -if (iavf_can_reuse_rx_page (rx_buffer)) { -/* hand second half of page back to the ring */ -iavf_reuse_rx_page (rx_ring, rx_buffer); - rx_ring->rx_stats.page_reuse_count++; - }else { -/* we are not reusing the buffer so unmap it */ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR ); -__page_frag_cache_drain (rx_buffer->page, - rx_buffer->pagecnt_bias); - } - -/* clear contents of buffer_info */ - rx_buffer->page =NULL ; +/* we are not reusing the buffer so unmap it */ +dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, +iavf_rx_pg_size (rx_ring), +DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR ); }/**
@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)else skb =iavf_build_skb (rx_ring, rx_buffer, size); +iavf_put_rx_buffer (rx_ring, rx_buffer); +/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -if (rx_buffer && size) - rx_buffer->pagecnt_bias++; +__free_pages (rx_buffer->page, +iavf_rx_pg_order (rx_ring)); + rx_buffer->page =NULL ;break ; } -iavf_put_rx_buffer (rx_ring, rx_buffer); + rx_buffer->page =NULL ; cleaned_count++;if (iavf_is_non_eop (rx_ring, rx_desc, skb))
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0 -8
@@ -266,12 +266,7 @@ struct iavf_tx_buffer {struct iavf_rx_buffer {dma_addr_t dma;struct page *page; -#if (BITS_PER_LONG >32 ) || (PAGE_SIZE >=65536 )__u32 page_offset; -#else -__u16 page_offset; -#endif -__u16 pagecnt_bias; };struct iavf_queue_stats {
@@ -293,8 +288,6 @@ struct iavf_rx_queue_stats {u64 non_eop_descs;u64 alloc_page_failed;u64 alloc_buff_failed; -u64 page_reuse_count; -u64 realloc_count; };enum iavf_ring_state_t {
@@ -374,7 +367,6 @@ struct iavf_ring {struct iavf_q_vector *q_vector;/* Backreference to associated vector */ struct rcu_head rcu;/* to avoid race on free */ -u16 next_to_alloc;struct sk_buff *skb;/* When iavf_clean_rx_ring_irq() must * return before it sees the EOP for * the current packet, we save that skb
Signature · 1 line
2.41.0
Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling
Quoted 2 lines · Alexander Lobakin
As an intermediate step, remove all page splitting/recyclig code. Just
Spelling issue: "recycling"
Quoted 239 lines
always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63&iavf_rx_bufstructures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:Function: add/remove: 0/2 grow/shrink: 1/3 up/down: 3/-494 (-491)
Although the array of
&iavf_rx_bufis barely used now and could be replaced with just page pointer array, don't touch it now to not complicate replacing it with libie Rx buffer struct later on. No surprise perf loses up to 30% here, but that regression will go away once PP lands.Signed-off-by Alexander Lobakin <[email protected]>--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 151 ++------------------ drivers/net/ethernet/intel/iavf/iavf_txrx.h | 8 -- 2 files changed, 13 insertions(+), 146 deletions(-)drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-129@@ -723,7 +723,7 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)DMA_FROM_DEVICE , IAVF_RX_DMA_ATTR); -__page_frag_cache_drain (rx_bi->page, rx_bi->pagecnt_bias); +__free_pages (rx_bi->page,iavf_rx_pg_order (rx_ring)); rx_bi->page =NULL ; rx_bi->page_offset =0 ;@@ -735,7 +735,6 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring)/* Zero out the descriptor ring */ memset (rx_ring->desc,0 , rx_ring->size); - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ; }@@ -791,7 +790,6 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring) goto err; } - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;@@ -811,9 +809,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val) { rx_ring->next_to_use = val; -/* update next to alloc since we have filled the ring */ - rx_ring->next_to_alloc = val; -/* Force memory writes to complete before letting h/w * know there are new descriptors to fetch. (Only * applicable for weak-ordered memory model archs, @@ -837,12 +832,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring,struct page *page = bi->page;dma_addr_t dma; -/* since we are recycling buffers we should seldom need to alloc */ -if (likely (page)) { - rx_ring->rx_stats.page_reuse_count++; -return true; - } -/* alloc new page for storage */ page =dev_alloc_pages (iavf_rx_pg_order (rx_ring));if (unlikely (!page)) {@@ -869,9 +858,6 @@ static bool iavf_alloc_mapped_page(struct iavf_ring *rx_ring, bi->page = page; bi->page_offset =IAVF_SKB_PAD ; -/* initialize pagecnt_bias to 1 representing we fully own page */ - bi->pagecnt_bias =1 ; -return true; }@@ -1103,91 +1089,6 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)return false; } -/** -* iavf_reuse_rx_page - page flip buffer and store it back on the ring -* @rx_ring: rx descriptor ring to store buffers on -* @old_buff: donor buffer to have page reused -* -* Synchronizes page for reuse by the adapter -**/ -static void iavf_reuse_rx_page (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *old_buff) -{ -struct iavf_rx_buffer *new_buff; -u16 nta = rx_ring->next_to_alloc; - - new_buff = &rx_ring->rx_bi[nta]; - -/* update, and store next to alloc */ - nta++; - rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; - -/* transfer page from old buffer to new buffer */ - new_buff->dma = old_buff->dma; - new_buff->page = old_buff->page; - new_buff->page_offset = old_buff->page_offset; - new_buff->pagecnt_bias = old_buff->pagecnt_bias; -} - -/** -* iavf_can_reuse_rx_page - Determine if this page can be reused by -* the adapter for another receive -* -* @rx_buffer: buffer containing the page -* -* If page is reusable, rx_buffer->page_offset is adjusted to point to -* an unused region in the page. -* -* For small pages, @truesize will be a constant value, half the size -* of the memory at page. We'll attempt to alternate between high and -* low halves of the page, with one half ready for use by the hardware -* and the other half being consumed by the stack. We use the page -* ref count to determine whether the stack has finished consuming the -* portion of this page that was passed up with a previous packet. If -* the page ref count is >1, we'll assume the "other" half page is -* still busy, and this page cannot be reused. -* -* For larger pages, @truesize will be the actual space used by the -* received packet (adjusted upward to an even multiple of the cache -* line size). This will advance through the page by the amount -* actually consumed by the received packets while there is still -* space for a buffer. Each region of larger pages will be used at -* most once, after which the page will not be reused. -* -* In either case, if the page is reusable its refcount is increased. -**/ -static bool iavf_can_reuse_rx_page (struct iavf_rx_buffer *rx_buffer) -{ -unsigned int pagecnt_bias = rx_buffer->pagecnt_bias; -struct page *page = rx_buffer->page; - -/* Is any reuse possible? */ -if (!dev_page_is_reusable (page)) -return false; - -#if (PAGE_SIZE <8192 ) -/* if we are only owner of page we can reuse it */ -if (unlikely ((page_count (page) - pagecnt_bias) >1 )) -return false; -#else -#define IAVF_LAST_OFFSET \ - (SKB_WITH_OVERHEAD(PAGE_SIZE) - IAVF_RXBUFFER_2048) -if (rx_buffer->page_offset >IAVF_LAST_OFFSET ) -return false; -#endif - -/* If we have drained the page fragment pool we need to update -* the pagecnt_bias and page count so that we fully restock the -* number of references the driver holds. -*/ -if (unlikely (!pagecnt_bias)) { -page_ref_add (page,USHRT_MAX ); - rx_buffer->pagecnt_bias =USHRT_MAX ; - } - -return true; -} -/** * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff * @rx_ring: rx descriptor ring to transact packets on @@ -1216,13 +1117,6 @@ static void iavf_add_rx_frag(struct iavf_ring *rx_ring,skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, rx_buffer->page, rx_buffer->page_offset, size, truesize); - -/* page is being used so we must update the page offset */ -#if (PAGE_SIZE <8192 ) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif }/** @@ -1250,9 +1144,6 @@ static struct iavf_rx_buffer *iavf_get_rx_buffer(struct iavf_ring *rx_ring, size, DMA_FROM_DEVICE); -/* We have pulled a buffer for use, so decrement pagecnt_bias */ - rx_buffer->pagecnt_bias--; -return rx_buffer; }@@ -1293,23 +1184,15 @@ static struct sk_buff *iavf_build_skb(struct iavf_ring *rx_ring,skb_reserve (skb,IAVF_SKB_PAD );__skb_put (skb, size); -/* buffer is used by skb, update page_offset */ -#if (PAGE_SIZE <8192 ) - rx_buffer->page_offset ^= truesize; -#else - rx_buffer->page_offset += truesize; -#endif -return skb; }/** -* iavf_put_rx_buffer - Clean up used buffer and either recycle or free +* iavf_put_rx_buffer - Unmap used buffer * @rx_ring: rx descriptor ring to transact packets on * @rx_buffer: rx buffer to pull data from * -* This function will clean up the contents of the rx_buffer. It will -* either recycle the buffer or unmap it and free the associated resources. +* This function will unmap the buffer after it's written by HW. */ static void iavf_put_rx_buffer (struct iavf_ring *rx_ring,struct iavf_rx_buffer *rx_buffer)@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,if (!rx_buffer)return ; -if (iavf_can_reuse_rx_page (rx_buffer)) { -/* hand second half of page back to the ring */ -iavf_reuse_rx_page (rx_ring, rx_buffer); - rx_ring->rx_stats.page_reuse_count++; - }else { -/* we are not reusing the buffer so unmap it */ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR );
drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_put_rx_buffer(struct iavf_ring *rx_ring, · line 1206
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-6-__page_frag_cache_drain (rx_buffer->page, - rx_buffer->pagecnt_bias); - } - -/* clear contents of buffer_info */ - rx_buffer->page =NULL ; +/* we are not reusing the buffer so unmap it */ +dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, +iavf_rx_pg_size (rx_ring), +DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR );
Rather than reorder all this I would just do the dma_unmap_page_attrs and then leave the assignment of NULL to rx_buffer->page. It should make this a bit easier to clean up the code below.
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307
Quoted 9 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-0}/** @@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)else skb =iavf_build_skb (rx_ring, rx_buffer, size); +iavf_put_rx_buffer (rx_ring, rx_buffer); +
This should stay below where it was.
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1315
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -if (rx_buffer && size) - rx_buffer->pagecnt_bias++; +__free_pages (rx_buffer->page, +iavf_rx_pg_order (rx_ring)); + rx_buffer->page =NULL ;break ; }
This code was undoing the iavf_get_rx_buffer decrement of pagecnt_bias and then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment of pagecnt_bias.
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-1-iavf_put_rx_buffer (rx_ring, rx_buffer); + rx_buffer->page =NULL ; cleaned_count++;if (iavf_is_non_eop (rx_ring, rx_desc, skb))
If iavf_put_rx_buffer just does the unmap and assignment of NULL then it could just be left here as is.
Quoted 37 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-9@@ -266,12 +266,7 @@ struct iavf_tx_buffer {struct iavf_rx_buffer {dma_addr_t dma;struct page *page; -#if (BITS_PER_LONG >32 ) || (PAGE_SIZE >=65536 )__u32 page_offset; -#else -__u16 page_offset; -#endif -__u16 pagecnt_bias; };struct iavf_queue_stats {@@ -293,8 +288,6 @@ struct iavf_rx_queue_stats {u64 non_eop_descs;u64 alloc_page_failed;u64 alloc_buff_failed; -u64 page_reuse_count; -u64 realloc_count; };enum iavf_ring_state_t {@@ -374,7 +367,6 @@ struct iavf_ring {struct iavf_q_vector *q_vector;/* Backreference to associated vector */ struct rcu_head rcu;/* to avoid race on free */ -u16 next_to_alloc;struct sk_buff *skb;/* When iavf_clean_rx_ring_irq() must * return before it sees the EOP for * the current packet, we save that skb -- 2.41.0
Quoted 4 lines
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRe: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700
Quoted 6 lines
Quoted 2 lines · Alexander Lobakin
As an intermediate step, remove all page splitting/recyclig code. Just
Spelling issue: "recycling"
checkpatch w/codespell didn't catch this one =\
Quoted 8 lines
Quoted 8 lines
always allocate a new page and don't touch its refcount, so that it gets freed by the core stack later.
Same for the "in-place" recycling, i.e. when an unused buffer gets assigned to a first needs-refilling descriptor. In some cases, this was leading to moving up to 63&iavf_rx_bufstructures around the ring on a per-field basis -- not something wanted on hotpath.
The change allows to greatly simplify certain parts of the code:
[...]
Quoted 35 lines
Quoted 13 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-9@@ -1317,21 +1200,10 @@ static void iavf_put_rx_buffer(struct iavf_ring *rx_ring,if (!rx_buffer)return ; -if (iavf_can_reuse_rx_page (rx_buffer)) { -/* hand second half of page back to the ring */ -iavf_reuse_rx_page (rx_ring, rx_buffer); - rx_ring->rx_stats.page_reuse_count++; - }else { -/* we are not reusing the buffer so unmap it */ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR );drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_put_rx_buffer(struct iavf_ring *rx_ring, · line 1206
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-6-__page_frag_cache_drain (rx_buffer->page, - rx_buffer->pagecnt_bias); - } - -/* clear contents of buffer_info */ - rx_buffer->page =NULL ; +/* we are not reusing the buffer so unmap it */ +dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, +iavf_rx_pg_size (rx_ring), +DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR );Rather than reorder all this I would just do the
dma_unmap_page_attrsand then leave the assignment of NULL torx_buffer->page. It should make this a bit easier to clean up the code below.Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-0}/** @@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)else skb =iavf_build_skb (rx_ring, rx_buffer, size);
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307
Quoted 2 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-0+iavf_put_rx_buffer (rx_ring, rx_buffer); +This should stay below where it was.
Wait-wait-wait.
if (!skb) break breaks the loop. put_rx_buffer() unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.
Quoted 18 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1315
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -if (rx_buffer && size) - rx_buffer->pagecnt_bias++; +__free_pages (rx_buffer->page, +iavf_rx_pg_order (rx_ring)); + rx_buffer->page =NULL ;break ; }This code was undoing the
iavf_get_rx_bufferdecrement ofpagecnt_biasand then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment ofpagecnt_bias.Quoted 1 line
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-1-iavf_put_rx_buffer (rx_ring, rx_buffer); + rx_buffer->page =NULL ; cleaned_count++;if (iavf_is_non_eop (rx_ring, rx_desc, skb))If
iavf_put_rx_bufferjust does the unmap and assignment of NULL then it could just be left here as is.
I guess those two are tied with the one above.
[...]
Thanks,
Olek
Re: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling
Quoted 6 lines · Alexander Lobakin
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700Quoted 2 lines
On Wed, Jul 5, 2023 at 8:57 AM Alexander Lobakin
<[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-0@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)else skb =iavf_build_skb (rx_ring, rx_buffer, size); +iavf_put_rx_buffer (rx_ring, rx_buffer); +This should stay below where it was.
Quoted 6 lines
Wait-wait-wait.
if (!skb) break breaks the loop.
put_rx_buffer()unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.
Specifically you don't want to be unmapping and freeing this page until after the !skb check. The problem is if skb is NULL the skb allocation failed and so processing of Rx is meant to stop in place without removing the page. It is where we will resume on the next pass assuming memory has been freed that can then be used. The problem is the skb allocation, not the page. We used to do the skb allocation before we would acquire the buffer, but with XDP there are cases where we aren't supposed to allocate it so it got moved to after which causes this confusion.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -if (rx_buffer && size) - rx_buffer->pagecnt_bias++; +__free_pages (rx_buffer->page, +iavf_rx_pg_order (rx_ring)); + rx_buffer->page =NULL ;break ; }
Quoted 4 lines
Quoted 4 lines
This code was undoing the
iavf_get_rx_bufferdecrement ofpagecnt_biasand then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment ofpagecnt_bias.
The key bit here is the allocation failure is the reason why we halted processing. So the page contains valid data and should not be freed. We just need to leave it in place and wait for an allocation to succeed and then we can resume processing.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-1-iavf_put_rx_buffer (rx_ring, rx_buffer); + rx_buffer->page =NULL ; cleaned_count++;
Quoted 6 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320
Quoted 1 line
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-0if (iavf_is_non_eop (rx_ring, rx_desc, skb))If
iavf_put_rx_bufferjust does the unmap and assignment of NULL then it could just be left here as is.I guess those two are tied with the one above.
Yeah, the iavf_put_rx_buffer should be left down here.
Alexander Lobakin
maintainerRe: [Intel-wired-lan] [PATCH RFC net-next v4 3/9] iavf: drop page splitting and recycling
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 10:06:29 -0700
Quoted 54 lines
Quoted 6 lines · Alexander Lobakin
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 07:47:03 -0700Quoted 2 lines
On Wed, Jul 5, 2023 at 8:57 AM Alexander Lobakin
<[email protected]> wrote:[...]
Quoted 9 lines
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1307
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-0@@ -1431,15 +1303,18 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)else skb =iavf_build_skb (rx_ring, rx_buffer, size); +iavf_put_rx_buffer (rx_ring, rx_buffer); +This should stay below where it was.
Quoted 6 lines
Wait-wait-wait.
if (!skb) break breaks the loop.
put_rx_buffer()unmaps the page. So in order to do the first, you need to do the second to avoid leaks. Or you meant "why unmapping and freeing if we fail, just leave it in place"? To make it easier to switch to Page Pool.Specifically you don't want to be unmapping and freeing this page until after the !skb check. The problem is if skb is NULL the skb allocation failed and so processing of Rx is meant to stop in place without removing the page. It is where we will resume on the next pass assuming memory has been freed that can then be used. The problem is the skb allocation, not the page. We used to do the skb allocation before we would acquire the buffer, but with XDP there are cases where we aren't supposed to allocate it so it got moved to after which causes this confusion.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -if (rx_buffer && size) - rx_buffer->pagecnt_bias++; +__free_pages (rx_buffer->page, +iavf_rx_pg_order (rx_ring)); + rx_buffer->page =NULL ;break ; }Quoted 4 lines
Quoted 4 lines
This code was undoing the
iavf_get_rx_bufferdecrement ofpagecnt_biasand then bailing since we have halted forward progress due to an skb allocation failure. As such we should just be removing the if statement and the increment ofpagecnt_bias.
Quoted 4 lines
The key bit here is the allocation failure is the reason why we halted processing. So the page contains valid data and should not be freed. We just need to leave it in place and wait for an allocation to succeed and then we can resume processing.
Aaah, okay, you want to try once again with the same buffer next time. I see. Makes sense!
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-1-iavf_put_rx_buffer (rx_ring, rx_buffer); + rx_buffer->page =NULL ; cleaned_count++;
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) · line 1320
Quoted 1 line
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-0if (iavf_is_non_eop (rx_ring, rx_desc, skb))If
iavf_put_rx_bufferjust does the unmap and assignment of NULL then it could just be left here as is.I guess those two are tied with the one above.
Yeah, the
iavf_put_rx_buffershould be left down here.
Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 4/9 net: page_pool: add DMA-sync-for-CPU inline helpers
In reply to Alexander LobakinEach driver is responsible for syncing buffers written by HW for CPU before accessing them. Almost each PP-enabled driver uses the same pattern, which could be shorthanded into a static inline to make driver code a little bit more compact.
Introduce a couple such functions. The first one takes the actual size of the data written by HW and is the main one to be used on Rx. The second does the same, but only if the PP performs DMA synchronizations at all.
Also constify pointer arguments of page_pool_get_dma_dir() and page_pool_get_dma_addr() to give a bit more room for optimization, as both of them are read-only.
--- include/net/page_pool.h | 49 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-)
include/net/page_pool.h +46 -3
@@ -304,8 +304,8 @@ static inline void *page_pool_dev_cache_alloc(struct page_pool *pool,/* get the stored dma direction. A driver might decide to treat this locally and * avoid the extra cache line from page_pool to determine the direction */ -static -inline enum dma_data_direction page_pool_get_dma_dir (struct page_pool *pool) +static inline enum dma_data_direction +page_pool_get_dma_dir (const struct page_pool *pool) {return pool->p.dma_dir; }
@@ -460,7 +460,7 @@ static inline void page_pool_cache_free(struct page_pool *pool, void *data,page_pool_put_page (pool,virt_to_head_page (data),-1 , allow_direct); } -static inline dma_addr_t page_pool_get_dma_addr (struct page *page) +static inline dma_addr_t page_pool_get_dma_addr (const struct page *page) {dma_addr_t ret = page->dma_addr;
@@ -477,6 +477,49 @@ static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr) page->dma_addr_upper =upper_32_bits (addr); } +/** +* __page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by HW +* @pool: &page_pool the page belongs to +* @page: page to sync +* @offset: offset from page start to "hard" start if using frags +* @dma_sync_size: size of the data written to the page +* +* Can be used as a shorthand to sync Rx pages before accessing them in the +* driver. Caller must ensure the pool was created with %PP_FLAG_DMA_MAP. +* Note that this version performs DMA sync unconditionally, even if the +* associated PP doesn't perform sync-for-device. Consider the non-underscored +* version first if unsure. +*/ +static inline void __page_pool_dma_sync_for_cpu (const struct page_pool *pool, +const struct page *page, +u32 offset,u32 dma_sync_size) +{ +dma_sync_single_range_for_cpu (pool->p.dev, +page_pool_get_dma_addr (page), + offset + pool->p.offset, dma_sync_size, +page_pool_get_dma_dir (pool)); +} + +/** +* page_pool_dma_sync_for_cpu - sync Rx page for CPU if needed +* @pool: &page_pool the page belongs to +* @page: page to sync +* @offset: offset from page start to "hard" start if using frags +* @dma_sync_size: size of the data written to the page +* +* Performs DMA sync for CPU, but *only* when: +* 1) page_pool was created with %PP_FLAG_DMA_SYNC_DEV to manage DMA syncs; +* 2) AND sync shortcut is not available (IOMMU, swiotlb, non-coherent DMA, ...) +*/ +static inline void page_pool_dma_sync_for_cpu (const struct page_pool *pool, +const struct page *page, +u32 offset,u32 dma_sync_size) +{ +if (pool->p.flags &PP_FLAG_DMA_SYNC_DEV ) +__page_pool_dma_sync_for_cpu (pool, page, offset, + dma_sync_size); +} +static inline bool is_page_pool_compiled_in (void ) {#ifdef CONFIG_PAGE_POOL
Signature · 1 line
2.41.0
Alexander Lobakin
maintainerRFC PATCH v4 5/9 libie: add Rx buffer management (via Page Pool)
In reply to Alexander LobakinAdd a couple intuitive helpers to hide Rx buffer implementation details in the library and not multiplicate it between drivers. The settings are optimized for Intel hardware, but nothing really HW-specific here. Use the new page_pool_dev_alloc() to dynamically switch between split-page and full-page modes depending on MTU, page size, required headroom etc. For example, on x86_64 with the default driver settings each page is shared between 2 buffers. Turning on XDP (not in this series) -> increasing headroom requirement pushes truesize out of 2048 boundary, leading to that each buffer starts getting a full page. The "ceiling" limit is %PAGE_SIZE, as only order-0 pages are used to avoid compound overhead. For the above architecture, this means maximum linear frame size of 3712 w/o XDP.
--- drivers/net/ethernet/intel/Kconfig | 1 + drivers/net/ethernet/intel/libie/rx.c | 54 +++++++++++++ include/linux/net/intel/libie/rx.h | 111 +++++++++++++++++++++++++- 3 files changed, 165 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/Kconfig +1 -0
@@ -86,6 +86,7 @@ config E1000E_HWTS config LIBIE tristate + select PAGE_POOL help libie (Intel Ethernet library) is a common library containing routines shared by several Intel Ethernet drivers.
drivers/net/ethernet/intel/libie/rx.c +54 -0
60 lines not on this page · raw message
include/linux/net/intel/libie/rx.h +110 -1
117 lines not on this page · raw message
Signature · 6 lines
2.41.0 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
Quoted 9 lines · Alexander Lobakin
drivers/net/ethernet/intel/libie/rx.c +9
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size)
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 10 lines
drivers/net/ethernet/intel/libie/rx.c +10
-0+{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE , + .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir =DMA_FROM_DEVICE , + .offset =LIBIE_SKB_HEADROOM ,
I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
drivers/net/ethernet/intel/libie/rx.c line 54
Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c +8
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset); + +/* "Wanted" truesize, passed to page_pool_dev_alloc() */ + truesize =roundup_pow_of_two (SKB_HEAD_ALIGN (pp.offset + pp.max_len)); + pp.init_arg = (void *)truesize;
I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with init_callback. And if we want to change the implemetation of init_callback, we may stuck with it as the driver is using it very differently here.
Is it possible to pass the 'wanted true size' by adding a parameter for libie_rx_alloc()?
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +return page_pool_create (&pp); +} +EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_create,LIBIE );
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800
Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size) +{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE ,
Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just xdp->data - xdp->data_hard_start".
Quoted 12 lines
drivers/net/ethernet/intel/libie/rx.c line 54
Quoted 9 lines
drivers/net/ethernet/intel/libie/rx.c +8
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset); + +/* "Wanted" truesize, passed to page_pool_dev_alloc() */ + truesize =roundup_pow_of_two (SKB_HEAD_ALIGN (pp.offset + pp.max_len)); + pp.init_arg = (void *)truesize;I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with
init_callback. And if we want to change the implemetation
I know. I abused it to save 1 function argument :p It's safe since I don't use init_callback (not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).
Quoted 5 lines
of
init_callback, we may stuck with it as the driver is using it very differently here.Is it possible to pass the 'wanted true size' by adding a parameter for
libie_rx_alloc()?
Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D
Quoted 5 lines
drivers/net/ethernet/intel/libie/rx.c line 58
Quoted 5 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +return page_pool_create (&pp); +} +EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_create,LIBIE );
Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
Quoted 29 lines · Alexander Lobakin
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size) +{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE ,Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Quoted 3 lines
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just
xdp->data-xdp->data_hard_start".
So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.
When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c line 50
Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset);
As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.
Quoted 17 lines
Quoted 7 lines
drivers/net/ethernet/intel/libie/rx.c line 54
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +/* "Wanted" truesize, passed to page_pool_dev_alloc() */ + truesize =roundup_pow_of_two (SKB_HEAD_ALIGN (pp.offset + pp.max_len)); + pp.init_arg = (void *)truesize;I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with
init_callback. And if we want to change the implemetationI know. I abused it to save 1 function argument :p It's safe since I don't use
init_callback(not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).Quoted 3 lines
of
init_callback, we may stuck with it as the driver is using it very differently here.
Quoted 6 lines
Quoted 2 lines
Is it possible to pass the 'wanted true size' by adding a parameter for
libie_rx_alloc()?Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D
It seems most hw driver know the size of memory it needs when creating the ring/queue, setting the frag size and deciding how many is a page split into before allocation seems like a possible future optimization.
For now, it would be better to add helper to acess pp.init_arg at least instead of acess pp.init_arg directly to make it more obvious and make the future optimization more easier.
Quoted 9 lines
Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c line 58
Quoted 6 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +return page_pool_create (&pp); +} +EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_create,LIBIE );Thanks,
Olek
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800
Quoted 34 lines · Alexander Lobakin
Quoted 29 lines
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size) +{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE ,Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Quoted 3 lines
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just
xdp->data-xdp->data_hard_start".
Quoted 6 lines
So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.
Not sure I'm following :s
Quoted 14 lines
When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c line 50
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset);As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.
^^^^^^^^^^^^
You mean the whole page or...?
I think it's not the driver's duty to track all this. We always set .offset to data - data_hard_start and .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.
Quoted 25 lines
Quoted 18 lines
Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 54
Quoted 5 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +/* "Wanted" truesize, passed to page_pool_dev_alloc() */ + truesize =roundup_pow_of_two (SKB_HEAD_ALIGN (pp.offset + pp.max_len)); + pp.init_arg = (void *)truesize;I am not sure if it is correct to use pp.init_arg here, as it is supposed to be used along with
init_callback. And if we want to change the implemetationI know. I abused it to save 1 function argument :p It's safe since I don't use
init_callback(not an argument).
I was thinking also of having a union in PP params or even a new field like "wanted true size", so that your function could even take values from there in certain cases (e.g. if I pass 0 as parameter).Quoted 3 lines
of
init_callback, we may stuck with it as the driver is using it very differently here.Quoted 6 lines
Quoted 2 lines
Is it possible to pass the 'wanted true size' by adding a parameter for
libie_rx_alloc()?Yes, or I could store it somewhere on the ring, but looks uglier =\ This one does as well to some degree, but at least hidden in the library and doesn't show up in the drivers :D
Quoted 7 lines
It seems most hw driver know the size of memory it needs when creating the ring/queue, setting the frag size and deciding how many is a page split into before allocation seems like a possible future optimization.
For now, it would be better to add helper to acess pp.init_arg at least instead of acess pp.init_arg directly to make it more obvious and make the future optimization more easier.
Makes senses.
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
drivers/net/ethernet/intel/libie/rx.c line 58
Quoted 7 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ +return page_pool_create (&pp); +} +EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_create,LIBIE );Thanks,
Olek
Thanks,
Olek
Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
Quoted 37 lines · Alexander Lobakin
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800Quoted 34 lines · Alexander Lobakin
Quoted 32 lines
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size) +{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE ,Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just
xdp->data-xdp->data_hard_start".
Quoted 8 lines
Quoted 6 lines
So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.
Not sure I'm following :s
It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.
If it is common enough that most drivers are using it the same way, adding a helper for that would be great.
Quoted 23 lines
Quoted 15 lines
When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c line 50
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset);As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.
Quoted 3 lines
^^^^^^^^^^^^You mean the whole page or...?
If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.
Quoted 6 lines
I think it's not the driver's duty to track all this. We always set .offset to
data - data_hard_startand .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.
There problem is that when page_pool_put_page() is called with a split page, the page pool does not know which frag is freeing too.
setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:
https://elixir.free-electrons.com/linux/v6.4-rc6/source/net/core/page_pool.c#L325
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:39:28 +0800
Quoted 45 lines · Alexander Lobakin
Quoted 44 lines
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800Quoted 40 lines · Alexander Lobakin
Quoted 29 lines
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
drivers/net/ethernet/intel/libie/rx.c +15
-0+/** +* libie_rx_page_pool_create - create a PP with the default libie settings +* @napi: &napi_struct covering this PP (no usage outside its poll loops) +* @size: size of the PP, usually simply Rx queue len +* +* Returns &page_pool on success, casted -errno on failure. +*/ +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi, +u32 size) +{ +struct page_pool_params pp = { + .flags =PP_FLAG_DMA_MAP |PP_FLAG_DMA_SYNC_DEV , + .order =LIBIE_RX_PAGE_ORDER , + .pool_size = size, + .nid =NUMA_NO_NODE ,Quoted 8 lines
drivers/net/ethernet/intel/libie/rx.c line 46
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Quoted 3 lines
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just
xdp->data-xdp->data_hard_start".So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.
Quoted 8 lines
Quoted 1 line
Not sure I'm following :s
It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.
If it is common enough that most drivers are using it the same way, adding a helper for that would be great.
How comes page_pool_params is internal if it's defined purely by the driver and then exists read-only :D I even got warned in the adjacent thread that the Page Pool core code shouldn't change it anyhow.
Quoted 39 lines
Quoted 24 lines
Quoted 16 lines
When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c line 50
Quoted 4 lines
drivers/net/ethernet/intel/libie/rx.c +4
-0+ }; +size_t truesize; + + pp.max_len =libie_rx_sync_len (napi->dev, pp.offset);As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.
Quoted 3 lines
^^^^^^^^^^^^You mean the whole page or...?
If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.
Quoted 6 lines
I think it's not the driver's duty to track all this. We always set .offset to
data - data_hard_startand .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.
Quoted 5 lines
There problem is that when
page_pool_put_page()is called with a split page, the page pool does not know which frag is freeing too.setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:
Maybe Page Pool should synchronize DMA even when !last_frag then? Setting .max_len to anything bigger than the maximum frame size you're planning to receive is counter-intuitive.
All three xdp_buff, xdp_frame and skb always have all info needed to determine which piece of the page we're recycling, it should be possible to do with no complications. Hypothetical forcing drivers to do DMA syncs on their own when they use frags is counter-intuitive as well, Page Pool should be able to handle this itself.
Alternatively, Page Pool may do as follows:
1. !last_frag -- do nothing, same as today.
2. last_frag -- sync, but not [offset, offset + max_len), but [offset, PAGE_SIZE).
This would also cover non-HW-writeable pieces like 2th-nth frame's headroom and each frame's skb_shared_info, but it's the only alternative to syncing each frag separately.
Yes, it's almost the same as to set .max_len to %PAGE_SIZE, but as I said, it feels weird to set .max_len to 4k when you allocate 2k frags. You don't know anyway how much of a page will be used.
For example, when I turn on driver-side XDP, increased headroom makes truesize cross the 2k border with 1500 MTU, so that 2 frag per page converts into 1 frag per page. In fact, instead of using the whole 4k, I use ~2200 and don't need to sync the entire 4k.
Setting .max_len to 4k gives you way heavier overhead to non DMA-coherent systems that per-frag-syncing would do.
Thanks,
Olek
Re: [PATCH RFC net-next v4 5/9] libie: add Rx buffer management (via Page Pool)
Quoted 54 lines · Alexander Lobakin
From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:39:28 +0800Quoted 51 lines · Alexander Lobakin
Quoted 37 lines
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:33 +0800Quoted 34 lines · Alexander Lobakin
Quoted 32 lines
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:28 +0800Quoted 17 lines · Alexander Lobakin
Quoted 16 lines
(quoted diff) +15
-0+/** + * libie_rx_page_pool_create - create a PP with the default libie settings + * @napi: &napi_struct covering this PP (no usage outside its poll loops) + * @size: size of the PP, usually simply Rx queue len + * + * Returns &page_pool on success, casted -errno on failure. + */ +struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi, + u32 size) +{ + struct page_pool_params pp = { + .flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV, + .order = LIBIE_RX_PAGE_ORDER, + .pool_size = size, + .nid = NUMA_NO_NODE,Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ .dev = napi->dev->dev.parent, + .napi = napi, + .dma_dir = DMA_FROM_DEVICE, + .offset = LIBIE_SKB_HEADROOM,I think it worth mentioning that the '.offset' is not really accurate when the page is split, as we do not really know what is the offset of the frag of a page except for the first frag.
Yeah, this is read as "offset from the start of the page or frag to the actual frame start, i.e. its Ethernet header" or "this is just
xdp->data-xdp->data_hard_start".Quoted 8 lines
Quoted 6 lines
So the problem seems to be if most of drivers have a similar reading as libie does here, as .offset seems to have a clear semantics which is used to skip dma sync operation for buffer range that is not touched by the dma operation. Even if it happens to have the same value of "offset from the start of the page or frag to the actual frame start", I am not sure it is future-proofing to reuse it.
Not sure I'm following :s
It would be better to avoid accessing the internal data of the page pool directly as much as possible, as that may be changed to different meaning or removed if the implememtation is changed.
Quoted 6 lines
Quoted 2 lines
If it is common enough that most drivers are using it the same way, adding a helper for that would be great.
How comes
page_pool_paramsis internal if it's defined purely by the driver and then exists read-only :D I even got warned in the adjacent thread that the Page Pool core code shouldn't change it anyhow.
Personally I am not one hundred percent convinced that page_pool_params will not be changed considering the discussion about improving/replacing the page pool to support P2P case.
Quoted 43 lines
Quoted 43 lines
Quoted 25 lines
Quoted 17 lines
When page frag is added, I didn't really give much thought about that as we use it in a cache coherent system.
It seems we might need to extend or update that semantics if we really want to skip dma sync operation for all the buffer ranges that are not touched by the dma operation for page split case.
Or Skipping dma sync operation for all untouched ranges might not be worth the effort, because it might need a per frag dma sync operation, which is more costly than a batched per page dma sync operation. If it is true, page pool already support that currently as my understanding, because the dma sync operation is only done when the last frag is released/freed.Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ }; + size_t truesize; + + pp.max_len = libie_rx_sync_len(napi->dev, pp.offset);As mentioned above, if we depend on the last released/freed frag to do the dma sync, the pp.max_len might need to cover all the frag.
Quoted 3 lines
^^^^^^^^^^^^You mean the whole page or...?
If we don't care about the accurate dma syncing, "cover all the frag" means the whole page here, as page pool doesn't have enough info to do accurate dma sync for now.
Quoted 6 lines
I think it's not the driver's duty to track all this. We always set .offset to
data - data_hard_startand .max_len to the maximum HW-writeable length for one frame. We don't know whether PP will give us a whole page or just a piece. DMA sync for device is performed in the PP core code as well. Driver just creates a PP and don't care about the internals.There problem is that when
page_pool_put_page()is called with a split page, the page pool does not know which frag is freeing too.
Quoted 6 lines
Quoted 2 lines
setting 'maximum HW-writeable length for one frame' only sync the first frag of a page as below:
Maybe Page Pool should synchronize DMA even when !last_frag then? Setting .max_len to anything bigger than the maximum frame size you're planning to receive is counter-intuitive.
This is simplest way to support dma sync for page frag case, the question is if the batching of the dma sync for all frag of a page can even out the additional cost of dma sync for dma untouched range of a page.
Quoted 11 lines
All three
xdp_buff,xdp_frameand skb always have all info needed to determine which piece of the page we're recycling, it should be possible to do with no complications. Hypothetical forcing drivers to do DMA syncs on their own when they use frags is counter-intuitive as well, Page Pool should be able to handle this itself.Alternatively, Page Pool may do as follows:
1. !last_frag -- do nothing, same as today.
2.last_frag-- sync, but not [offset, offset +max_len), but [offset,PAGE_SIZE).
When a frag is free, we don't know if it is the last frag or not for now yet.
Quoted 7 lines
This would also cover non-HW-writeable pieces like 2th-nth frame's headroom and each frame's
skb_shared_info, but it's the only alternative to syncing each frag separately.
Yes, it's almost the same as to set .max_len to %PAGE_SIZE, but as I said, it feels weird to set .max_len to 4k when you allocate 2k frags. You don't know anyway how much of a page will be used.
In that that case, we may need to make it more generic for the case when a page is spilt into more than two frags, especially for system with 64K page size.
Alexander Lobakin
maintainerRFC PATCH v4 6/9 iavf: switch to Page Pool
In reply to Alexander LobakinNow that the IAVF driver simply uses dev_alloc_page() + free_page() with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Use skb_mark_for_recycle() to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.
--- drivers/net/ethernet/intel/iavf/iavf_main.c | 33 +-- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 247 +++++------------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 91 +------ drivers/net/ethernet/intel/iavf/iavf_type.h | 2 - .../net/ethernet/intel/iavf/iavf_virtchnl.c | 19 +- 5 files changed, 82 insertions(+), 310 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +4 -29
@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2013 - 2018 Intel Corporation. */ +#include <linux/net/intel/libie/rx.h> +#include "iavf.h" #include "iavf_prototype.h" #include "iavf_client.h"
@@ -698,32 +700,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)**/ static void iavf_configure_rx (struct iavf_adapter *adapter) { -unsigned int rx_buf_len =IAVF_RXBUFFER_2048 ;struct iavf_hw *hw = &adapter->hw; -int i; - -if (PAGE_SIZE <8192 ) { -struct net_device *netdev = adapter->netdev; -/* For jumbo frames on systems with 4K pages we have to use -* an order 1 page, so we might as well increase the size -* of our Rx buffer to make better use of the available space -*/ - rx_buf_len =IAVF_RXBUFFER_3072 ; - -/* We use a 1536 buffer size for configurations with -* standard Ethernet mtu. On x86 this gives us enough room -* for shared info and 192 bytes of padding. -*/ -if (!IAVF_2K_TOO_SMALL_WITH_PADDING && - (netdev->mtu <=ETH_DATA_LEN )) - rx_buf_len =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; - } - -for (i =0 ; i < adapter->num_active_queues; i++) { +for (u32 i =0 ; i < adapter->num_active_queues; i++) adapter->rx_rings[i].tail = hw->hw_addr +IAVF_QRX_TAIL1 (i); - adapter->rx_rings[i].rx_buf_len = rx_buf_len; - } }/**
@@ -1590,7 +1570,6 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter) rx_ring = &adapter->rx_rings[i]; rx_ring->queue_index = i; rx_ring->netdev = adapter->netdev; - rx_ring->dev = &adapter->pdev->dev; rx_ring->count = adapter->rx_desc_count; rx_ring->itr_setting =IAVF_ITR_RX_DEF ; }
@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter) netdev->netdev_ops = &iavf_netdev_ops;iavf_set_ethtool_ops (netdev); - netdev->watchdog_timeo =5 *HZ ; - -/* MTU range: 68 - 9710 */ - netdev->min_mtu =ETH_MIN_MTU ; - netdev->max_mtu =IAVF_MAX_RXBUFFER -IAVF_PACKET_HDR_PAD ; + netdev->max_mtu =LIBIE_MAX_MTU ;if (!is_valid_ether_addr (adapter->hw.mac.addr)) { dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",
drivers/net/ethernet/intel/iavf/iavf_txrx.c +62 -185
400 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_txrx.h +7 -84
131 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_type.h +0 -2
@@ -10,8 +10,6 @@#include "iavf_adminq.h" #include "iavf_devids.h" -#define IAVF_RXQ_CTX_DBUFF_SHIFT 7 -/* IAVF_MASK is a macro used on 32 bit registers */ #define IAVF_MASK (mask, shift) ((u32)(mask) << (shift))
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +9 -10
@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2013 - 2018 Intel Corporation. */ +#include <linux/net/intel/libie/rx.h> +#include "iavf.h" #include "iavf_prototype.h" #include "iavf_client.h"
@@ -269,14 +271,13 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)void iavf_configure_queues (struct iavf_adapter *adapter) {struct virtchnl_vsi_queue_config_info *vqci; -int i, max_frame = adapter->vf_res->max_mtu; +u32 i, max_frame = adapter->vf_res->max_mtu; +struct iavf_ring *rxq = adapter->rx_rings;int pairs = adapter->num_active_queues;struct virtchnl_queue_pair_info *vqpi; +u32 hr, max_len;size_t len; -if (max_frame >IAVF_MAX_RXBUFFER || !max_frame) - max_frame =IAVF_MAX_RXBUFFER ; -if (adapter->current_op !=VIRTCHNL_OP_UNKNOWN ) {/* bail because we already have a command pending */ dev_err(&adapter->pdev->dev, "Cannot configure queues, command %d pending\n",
@@ -289,9 +290,9 @@ void iavf_configure_queues(struct iavf_adapter *adapter)if (!vqci)return ; -/* Limit maximum frame size when jumbo frames is not enabled */ -if (adapter->netdev->mtu <=ETH_DATA_LEN ) - max_frame =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; + hr = rxq->pp->p.offset; + max_len = rxq->pp->p.max_len; + max_frame =min_not_zero (max_frame,LIBIE_MAX_RX_FRM_LEN (hr)); vqci->vsi_id = adapter->vsi_res->vsi_id; vqci->num_queue_pairs = pairs;
@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter) vqpi->rxq.ring_len = adapter->rx_rings[i].count; vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma; vqpi->rxq.max_pkt_size = max_frame; - vqpi->rxq.databuffer_size = -ALIGN (adapter->rx_rings[i].rx_buf_len, -BIT_ULL (IAVF_RXQ_CTX_DBUFF_SHIFT )); + vqpi->rxq.databuffer_size = max_len; vqpi++; }
Signature · 6 lines
2.41.0 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
Quoted 14 lines · Alexander Lobakin
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.Signed-off-by Alexander Lobakin <[email protected]>---
...
drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2565
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0
-1@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter) netdev->netdev_ops = &iavf_netdev_ops;iavf_set_ethtool_ops (netdev); - netdev->watchdog_timeo =5 *HZ ;
This seems like a unrelated change here?
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1
-4- -/* MTU range: 68 - 9710 */ - netdev->min_mtu =ETH_MIN_MTU ; - netdev->max_mtu =IAVF_MAX_RXBUFFER -IAVF_PACKET_HDR_PAD ; + netdev->max_mtu =LIBIE_MAX_MTU ;
...
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 11 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);
If a page is able to be spilt between more than one desc, perhaps the prt_ring size does not need to be as big as rx_ring->count.
Quoted 14 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +6
-2+if (IS_ERR (pool)) +return PTR_ERR (pool); + + rx_ring->pp = pool;/* warn if we are about to overwrite the pointer */ WARN_ON (rx_ring->rx_bi); - bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; - rx_ring->rx_bi =kzalloc (bi_size,GFP_KERNEL ); + rx_ring->rx_bi =kcalloc (rx_ring->count,sizeof (*rx_ring->rx_bi), +GFP_KERNEL );if (!rx_ring->rx_bi) goto err;
...
Quoted 31 lines
/**
*iavf_build_skb- Build skb around an existing buffer
- * @rx_ring: Rx descriptor ring to transact packets on
* @rx_buffer: Rx buffer to pull data from
* @size: size of buffer to add to skb
*
* This function builds an skb around an existing Rx buffer, taking care
* to set up the skb correctly and avoid any memcpy overhead. */drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-12-static struct sk_buff *iavf_build_skb (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *rx_buffer, +static struct sk_buff *iavf_build_skb (const struct libie_rx_buffer *rx_buffer,unsigned int size) { -void *va; -#if (PAGE_SIZE <8192 ) -unsigned int truesize =iavf_rx_pg_size (rx_ring) /2 ; -#else -unsigned int truesize =SKB_DATA_ALIGN (sizeof (struct skb_shared_info )) + -SKB_DATA_ALIGN (IAVF_SKB_PAD + size); -#endif +struct page *page = rx_buffer->page; +u32 hr = page->pp->p.offset;struct sk_buff *skb; +void *va; -if (!rx_buffer || !size) -return NULL ;/* prefetch first cache line of first page */ - va =page_address (rx_buffer->page) + rx_buffer->page_offset;
drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +5
-3-net_prefetch (va); + va =page_address (page) + rx_buffer->offset; +net_prefetch (va + hr);/* build an skb around the page buffer */ - skb =napi_build_skb (va -IAVF_SKB_PAD , truesize); -if (unlikely (!skb)) + skb =napi_build_skb (va, rx_buffer->truesize); +if (unlikely (!skb)) { +page_pool_put_page (page->pp, page, size, true);
Isn't it more correct to call page_pool_put_full_page() here? as we do not know which frag is used for the rx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that in Documentation/networking/page_pool.rst.
Quoted 2 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-0return NULL ; + }
...
Quoted 10 lines
struct
iavf_queue_stats{drivers/net/ethernet/intel/iavf/iavf_txrx.h +2
-1u64 packets;u64 bytes;@@ -311,16 +243,19 @@ enum iavf_ring_state_t {struct iavf_ring {struct iavf_ring *next;/* pointer to next ring in q_vector */ void *desc;/* Descriptor ring memory */ -struct device *dev;/* Used for DMA mapping */ +union { +struct page_pool *pp;/* Used on Rx for buffer management */
drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 256
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +4
-1+struct device *dev;/* Used on Tx for DMA mapping */ + };struct net_device *netdev;/* netdev ring maps to */ union { +struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi; -struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;
Is there a reason to move it here?
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-1u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
From: Yunsheng Lin <[email protected]>
Date: Thu, 6 Jul 2023 20:47:22 +0800
Quoted 18 lines · Alexander Lobakin
Quoted 14 lines
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.Signed-off-by Alexander Lobakin <[email protected]>---
...
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2565
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0
-1@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter) netdev->netdev_ops = &iavf_netdev_ops;iavf_set_ethtool_ops (netdev); - netdev->watchdog_timeo =5 *HZ ;This seems like a unrelated change here?
Sorta. Default timeout is 5 seconds already, so I removed this redundance. But I expected someone to spot this, so I'm perfectly fine with not doing this [and stop doing such things in general].
Quoted 10 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1
-4- -/* MTU range: 68 - 9710 */ - netdev->min_mtu =ETH_MIN_MTU ; - netdev->max_mtu =IAVF_MAX_RXBUFFER -IAVF_PACKET_HDR_PAD ; + netdev->max_mtu =LIBIE_MAX_MTU ;...
Quoted 14 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 11 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);If a page is able to be spilt between more than one desc, perhaps the
prt_ringsize does not need to be as big asrx_ring->count.
But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
BTW, with recent recycling optimization, most of recycling is done directly through cache, not ptr_ring. So I'd even say it's safe to start creating smaller ptr_rings in the drivers.
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-0+if (IS_ERR (pool)) +return PTR_ERR (pool); + + rx_ring->pp = pool;
[...]
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* build an skb around the page buffer */ - skb =napi_build_skb (va -IAVF_SKB_PAD , truesize); -if (unlikely (!skb)) + skb =napi_build_skb (va, rx_buffer->truesize); +if (unlikely (!skb)) { +page_pool_put_page (page->pp, page, size, true);Isn't it more correct to call
page_pool_put_full_page()here? as we do not know which frag is used for therx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that inDocumentation/networking/page_pool.rst.
Ooof, maybe. My first try with PP frags. So when we use frags, we always must use _full_page() / p.max_len instead of the actual frame size?
Quoted 20 lines
Quoted 3 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-0return NULL ; + }...
Quoted 14 lines
struct
iavf_queue_stats{drivers/net/ethernet/intel/iavf/iavf_txrx.h +4
-1u64 packets;u64 bytes;@@ -311,16 +243,19 @@ enum iavf_ring_state_t {struct iavf_ring {struct iavf_ring *next;/* pointer to next ring in q_vector */ void *desc;/* Descriptor ring memory */ -struct device *dev;/* Used for DMA mapping */ +union { +struct page_pool *pp;/* Used on Rx for buffer management */ +struct device *dev;/* Used on Tx for DMA mapping */ + };struct net_device *netdev;/* netdev ring maps to */ union {
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 256
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +2
-1+struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi; -struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;Is there a reason to move it here?
Oops, seems like it's a leftover. There is reason: removing hole in the structure, but it needs to be done right when I change its layout. In this commit I just alter unions with no actual layout changes. Will fix.
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 259
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-1u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;
Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
On 2023/7/7 0:38, Alexander Lobakin wrote:
...
Quoted 16 lines
Quoted 15 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 12 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);If a page is able to be spilt between more than one desc, perhaps the
prt_ringsize does not need to be as big asrx_ring->count.
Quoted 5 lines
But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?
Quoted 3 lines
BTW, with recent recycling optimization, most of recycling is done directly through cache, not
ptr_ring. So I'd even say it's safe to start creating smallerptr_ringsin the drivers.
The problem is that we may use more memory than before for certain case if we don't limit the size of ptr_ring, unless we can ensure all of recycling is done directly through cache, not ptr_ring.
Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-0+if (IS_ERR (pool)) +return PTR_ERR (pool); + + rx_ring->pp = pool;[...]
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb) · line 1075
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-2/* build an skb around the page buffer */ - skb =napi_build_skb (va -IAVF_SKB_PAD , truesize); -if (unlikely (!skb)) + skb =napi_build_skb (va, rx_buffer->truesize); +if (unlikely (!skb)) { +page_pool_put_page (page->pp, page, size, true);Isn't it more correct to call
page_pool_put_full_page()here? as we do not know which frag is used for therx_buffer, and depend on the last released frag to do the syncing, maybe I should mention that inDocumentation/networking/page_pool.rst.
Quoted 2 lines
Ooof, maybe. My first try with PP frags. So when we use frags, we always must use
_full_page()/ p.max_len instead of the actual frame size?
Currently, _full_page() / p.max_len must be used to ensure the correct dma sync operation.
But as mentioned in the previous patch, it might be about what is the semantics of dma sync operation for page split case.
Quoted 1 line
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800
Quoted 26 lines · Alexander Lobakin
...
Quoted 16 lines
Quoted 15 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 12 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);If a page is able to be spilt between more than one desc, perhaps the
prt_ringsize does not need to be as big asrx_ring->count.Quoted 5 lines
But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
Quoted 3 lines
It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?
But libie can't say for sure how PP will split pages for it, right?
Quoted 8 lines
Quoted 4 lines
BTW, with recent recycling optimization, most of recycling is done directly through cache, not
ptr_ring. So I'd even say it's safe to start creating smallerptr_ringsin the drivers.The problem is that we may use more memory than before for certain case if we don't limit the size of
ptr_ring, unless we can ensure all of recycling is done directly through cache, notptr_ring.
Also not sure I'm following =\
[...]
Thanks,
Olek
Re: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
Quoted 36 lines · Alexander Lobakin
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800Quoted 26 lines · Alexander Lobakin
...
Quoted 16 lines
Quoted 15 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 12 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);If a page is able to be spilt between more than one desc, perhaps the
prt_ringsize does not need to be as big asrx_ring->count.Quoted 5 lines
But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
Quoted 3 lines
It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?
But libie can't say for sure how PP will split pages for it, right?
Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
BTW, with recent recycling optimization, most of recycling is done directly through cache, not
ptr_ring. So I'd even say it's safe to start creating smallerptr_ringsin the drivers.The problem is that we may use more memory than before for certain case if we don't limit the size of
ptr_ring, unless we can ensure all of recycling is done directly through cache, notptr_ring.Also not sure I'm following =\
Before adding page pool support, the max memory used in the driver is as below:rx_ring->count * PAGE_SIZE;
After adding page pool support, the max memory used in the driver is as below:
ptr_ring->size * PAGE_SIZE +PP_ALLOC_CACHE_SIZE * PAGE_SIZE +rx_ring->count * PAGE_SIZE / pp.init_arg
Quoted 8 lines
[...]
Thanks,
Olek.
Alexander Lobakin
maintainerRe: [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
From: Yunsheng Lin <[email protected]>
Date: Tue, 11 Jul 2023 19:47:14 +0800
Quoted 50 lines · Alexander Lobakin
Quoted 36 lines
From: Yunsheng Lin <[email protected]>
Date: Sun, 9 Jul 2023 13:16:39 +0800Quoted 26 lines · Alexander Lobakin
...
Quoted 16 lines
Quoted 15 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c void iavf_free_rx_resources(struct iavf_ring *rx_ring) · line 748
Quoted 12 lines
/**
drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-2@@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count);If a page is able to be spilt between more than one desc, perhaps the
prt_ringsize does not need to be as big asrx_ring->count.Quoted 5 lines
But we doesn't know in advance, right? Esp. given that it's hidden in the lib. But anyway, you can only assume that in regular cases if you always allocate frags of the same size, PP will split pages when 2+ frags can fit there or return the whole page otherwise, but who knows what might happen.
Quoted 3 lines
It seems intel driver is able to know the size of memory it needs when creating the ring/queue/napi/pp, maybe the driver only tell the libie how many descs does it use for queue, and libie can adjust it accordingly?
But libie can't say for sure how PP will split pages for it, right?
Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
BTW, with recent recycling optimization, most of recycling is done directly through cache, not
ptr_ring. So I'd even say it's safe to start creating smallerptr_ringsin the drivers.The problem is that we may use more memory than before for certain case if we don't limit the size of
ptr_ring, unless we can ensure all of recycling is done directly through cache, notptr_ring.Also not sure I'm following =\
Before adding page pool support, the max memory used in the driver is as below:
rx_ring->count*PAGE_SIZE;
Quoted 6 lines
After adding page pool support, the max memory used in the driver is as below:
ptr_ring->size*PAGE_SIZE+PP_ALLOC_CACHE_SIZE*PAGE_SIZE+rx_ring->count*PAGE_SIZE/ pp.init_arg
I know. I was wondering how your message connects with what I said regarding that using direct recycling more aggressively allows us to reduce ptr_ring sizes, as e.g. on my setup ptr_ring is not used at all (it still will be in other cases, but a lot less often than before).
Quoted 9 lines
Quoted 9 lines
[...]
Thanks,
Olek.
Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
In reply to Alexander LobakinQuoted 13 lines · Alexander Lobakin
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.
Quoted 1 line
Signed-off-by Alexander Lobakin <[email protected]>
One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.
Quoted 64 lines
--- drivers/net/ethernet/intel/iavf/iavf_main.c | 33 +-- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 247 +++++------------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 91 +------ drivers/net/ethernet/intel/iavf/iavf_type.h | 2 - .../net/ethernet/intel/iavf/iavf_virtchnl.c | 19 +- 5 files changed, 82 insertions(+), 310 deletions(-)drivers/net/ethernet/intel/iavf/iavf_main.c +3
-24@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2013 - 2018 Intel Corporation. */ +#include <linux/net/intel/libie/rx.h> +#include "iavf.h" #include "iavf_prototype.h" #include "iavf_client.h" @@ -698,32 +700,10 @@ static void iavf_configure_tx(struct iavf_adapter *adapter)**/ static void iavf_configure_rx (struct iavf_adapter *adapter) { -unsigned int rx_buf_len =IAVF_RXBUFFER_2048 ;struct iavf_hw *hw = &adapter->hw; -int i; - -if (PAGE_SIZE <8192 ) { -struct net_device *netdev = adapter->netdev; -/* For jumbo frames on systems with 4K pages we have to use -* an order 1 page, so we might as well increase the size -* of our Rx buffer to make better use of the available space -*/ - rx_buf_len =IAVF_RXBUFFER_3072 ; - -/* We use a 1536 buffer size for configurations with -* standard Ethernet mtu. On x86 this gives us enough room -* for shared info and 192 bytes of padding. -*/ -if (!IAVF_2K_TOO_SMALL_WITH_PADDING && - (netdev->mtu <=ETH_DATA_LEN )) - rx_buf_len =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; - } - -for (i =0 ; i < adapter->num_active_queues; i++) { +for (u32 i =0 ; i < adapter->num_active_queues; i++) adapter->rx_rings[i].tail = hw->hw_addr +IAVF_QRX_TAIL1 (i); - adapter->rx_rings[i].rx_buf_len = rx_buf_len; - } }/** @@ -1590,7 +1570,6 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter) rx_ring = &adapter->rx_rings[i]; rx_ring->queue_index = i; rx_ring->netdev = adapter->netdev; - rx_ring->dev = &adapter->pdev->dev; rx_ring->count = adapter->rx_desc_count; rx_ring->itr_setting =IAVF_ITR_RX_DEF ; }
drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2566
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0
-2@@ -2562,11 +2541,7 @@ static void iavf_init_config_adapter(struct iavf_adapter *adapter) netdev->netdev_ops = &iavf_netdev_ops;iavf_set_ethtool_ops (netdev); - netdev->watchdog_timeo =5 *HZ ; -
These removals should be pulled out into a seperate patch where we can call out that they are not needed due to redundancies w/ other code on the core network code.
drivers/net/ethernet/intel/iavf/iavf_main.c static void iavf_init_config_adapter(struct iavf_adapter *adapter) · line 2544
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +1
-3-/* MTU range: 68 - 9710 */ - netdev->min_mtu =ETH_MIN_MTU ; - netdev->max_mtu =IAVF_MAX_RXBUFFER -IAVF_PACKET_HDR_PAD ; + netdev->max_mtu =LIBIE_MAX_MTU ;
Same here for min_mtu.
Quoted 51 lines
drivers/net/ethernet/intel/iavf/iavf_main.c +0
-0if (!is_valid_ether_addr (adapter->hw.mac.addr)) { dev_info(&pdev->dev, "Invalid MAC address %pM, using random\n",drivers/net/ethernet/intel/iavf/iavf_txrx.c +4
-27@@ -689,9 +689,6 @@ int iavf_setup_tx_descriptors(struct iavf_ring *tx_ring)**/ static void iavf_clean_rx_ring (struct iavf_ring *rx_ring) { -unsigned long bi_size; -u16 i; -/* ring already cleared, nothing to do */ if (!rx_ring->rx_bi)return ;@@ -701,40 +698,16 @@ static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) rx_ring->skb =NULL ; } -/* Free all the Rx ring sk_buffs */ -for (i =0 ; i < rx_ring->count; i++) { -struct iavf_rx_buffer *rx_bi = &rx_ring->rx_bi[i]; +/* Free all the Rx ring buffers */ +for (u32 i = rx_ring->next_to_clean; i != rx_ring->next_to_use; ) { +const struct libie_rx_buffer *rx_bi = &rx_ring->rx_bi[i]; -if (!rx_bi->page) -continue ; +page_pool_put_full_page (rx_ring->pp, rx_bi->page, false); -/* Invalidate cache lines that may have been written to by -* device so that we avoid corrupting memory. -*/ -dma_sync_single_range_for_cpu (rx_ring->dev, - rx_bi->dma, - rx_bi->page_offset, - rx_ring->rx_buf_len, -DMA_FROM_DEVICE ); - -/* free resources associated with mapping */ -dma_unmap_page_attrs (rx_ring->dev, rx_bi->dma, -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IAVF_RX_DMA_ATTR ); - -__free_pages (rx_bi->page,iavf_rx_pg_order (rx_ring)); - - rx_bi->page =NULL ; - rx_bi->page_offset =0 ;
drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-6+if (unlikely (++i == rx_ring->count)) + i =0 ; } - bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; -memset (rx_ring->rx_bi,0 , bi_size); - -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); -
I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things in setup_rx_descriptors.
Quoted 462 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +56
-152rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ; }@@ -752,10 +725,13 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring) rx_ring->rx_bi =NULL ;if (rx_ring->desc) { -dma_free_coherent (rx_ring->dev, rx_ring->size, +dma_free_coherent (rx_ring->pp->p.dev, rx_ring->size, rx_ring->desc, rx_ring->dma); rx_ring->desc =NULL ; } + +page_pool_destroy (rx_ring->pp); + rx_ring->pp =NULL ; }/** @@ -766,13 +742,19 @@ void iavf_free_rx_resources(struct iavf_ring *rx_ring)**/ int iavf_setup_rx_descriptors (struct iavf_ring *rx_ring) { -struct device *dev = rx_ring->dev; -int bi_size; +struct page_pool *pool; + + pool =libie_rx_page_pool_create (&rx_ring->q_vector->napi, + rx_ring->count); +if (IS_ERR (pool)) +return PTR_ERR (pool); + + rx_ring->pp = pool;/* warn if we are about to overwrite the pointer */ WARN_ON (rx_ring->rx_bi); - bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; - rx_ring->rx_bi =kzalloc (bi_size,GFP_KERNEL ); + rx_ring->rx_bi =kcalloc (rx_ring->count,sizeof (*rx_ring->rx_bi), +GFP_KERNEL );if (!rx_ring->rx_bi) goto err;@@ -781,22 +763,27 @@ int iavf_setup_rx_descriptors(struct iavf_ring *rx_ring)/* Round up to nearest 4K */ rx_ring->size = rx_ring->count *sizeof (union iavf_32byte_rx_desc ); rx_ring->size =ALIGN (rx_ring->size,4096 ); - rx_ring->desc =dma_alloc_coherent (dev, rx_ring->size, + rx_ring->desc =dma_alloc_coherent (pool->p.dev, rx_ring->size, &rx_ring->dma,GFP_KERNEL );if (!rx_ring->desc) { -dev_info (dev,"Unable to allocate memory for the Rx descriptor ring, size=%d\n" , +dev_info (pool->p.dev,"Unable to allocate memory for the Rx descriptor ring, size=%d\n" , rx_ring->size); - goto err; + goto err_free_buf; } rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;return 0 ; -err: + +err_free_buf:kfree (rx_ring->rx_bi); rx_ring->rx_bi =NULL ; + +err: +page_pool_destroy (rx_ring->pp); +return -ENOMEM ; }@@ -818,49 +805,6 @@ static inline void iavf_release_rx_desc(struct iavf_ring *rx_ring, u32 val)writel (val, rx_ring->tail); } -/** -* iavf_alloc_mapped_page - recycle or make a new page -* @rx_ring: ring to use -* @bi: rx_buffer struct to modify -* -* Returns true if the page was successfully allocated or -* reused. -**/ -static bool iavf_alloc_mapped_page (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *bi) -{ -struct page *page = bi->page; -dma_addr_t dma; - -/* alloc new page for storage */ - page =dev_alloc_pages (iavf_rx_pg_order (rx_ring)); -if (unlikely (!page)) { - rx_ring->rx_stats.alloc_page_failed++; -return false; - } - -/* map page for use */ - dma =dma_map_page_attrs (rx_ring->dev, page,0 , -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IAVF_RX_DMA_ATTR ); - -/* if mapping failed free memory back to system since -* there isn't much point in holding memory we can't use -*/ -if (dma_mapping_error (rx_ring->dev, dma)) { -__free_pages (page,iavf_rx_pg_order (rx_ring)); - rx_ring->rx_stats.alloc_page_failed++; -return false; - } - - bi->dma = dma; - bi->page = page; - bi->page_offset =IAVF_SKB_PAD ; - -return true; -} -/** * iavf_receive_skb - Send a completed packet up the stack * @rx_ring: rx ring in play @@ -893,36 +837,29 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count) {u16 ntu = rx_ring->next_to_use;union iavf_rx_desc *rx_desc; -struct iavf_rx_buffer *bi;/* do nothing if no valid netdev defined */ if (!rx_ring->netdev || !cleaned_count)return false; rx_desc =IAVF_RX_DESC (rx_ring, ntu); - bi = &rx_ring->rx_bi[ntu]; do { - if (!iavf_alloc_mapped_page(rx_ring, bi)) - goto no_buffers; + dma_addr_t addr; - /* sync the buffer for use by the device */ - dma_sync_single_range_for_device(rx_ring->dev, bi->dma, - bi->page_offset, - rx_ring->rx_buf_len, - DMA_FROM_DEVICE); + addr = libie_rx_alloc(rx_ring->pp, &rx_ring->rx_bi[ntu]); + if (unlikely(addr == DMA_MAPPING_ERROR)) + goto no_buffers; /* Refresh the desc even if buffer_addrs didn't change * because each write-back erases this info. */ - rx_desc->read.pkt_addr = cpu_to_le64(bi->dma + bi->page_offset); + rx_desc->read.pkt_addr = cpu_to_le64(addr); rx_desc++; - bi++; ntu++; if (unlikely(ntu == rx_ring->count)) { rx_desc = IAVF_RX_DESC(rx_ring, 0); - bi = rx_ring->rx_bi; ntu = 0; }@@ -941,6 +878,8 @@ bool iavf_alloc_rx_buffers(struct iavf_ring *rx_ring, u16 cleaned_count)if (rx_ring->next_to_use != ntu)iavf_release_rx_desc (rx_ring, ntu); + rx_ring->rx_stats.alloc_page_failed++; +/* make sure to come back via polling to try again after * allocation failure */ @@ -1091,9 +1030,8 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)/** * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff -* @rx_ring: rx descriptor ring to transact packets on -* @rx_buffer: buffer containing page to add * @skb: sk_buff to place the data into +* @rx_buffer: buffer containing page to add * @size: packet length from rx_desc * * This function will add the data contained in rx_buffer->page to the skb. @@ -1101,111 +1039,52 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)* * The function will then update the page offset. **/ -static void iavf_add_rx_frag (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *rx_buffer, -struct sk_buff *skb, +static void iavf_add_rx_frag (struct sk_buff *skb, +const struct libie_rx_buffer *rx_buffer,unsigned int size) { -#if (PAGE_SIZE <8192 ) -unsigned int truesize =iavf_rx_pg_size (rx_ring) /2 ; -#else -unsigned int truesize =SKB_DATA_ALIGN (size +IAVF_SKB_PAD ); -#endif - -if (!size) -return ; +u32 hr = rx_buffer->page->pp->p.offset;skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, rx_buffer->page, - rx_buffer->page_offset, size, truesize); -} - -/** -* iavf_get_rx_buffer - Fetch Rx buffer and synchronize data for use -* @rx_ring: rx descriptor ring to transact packets on -* @size: size of buffer to add to skb -* -* This function will pull an Rx buffer from the ring and synchronize it -* for use by the CPU. -*/ -static struct iavf_rx_buffer *iavf_get_rx_buffer (struct iavf_ring *rx_ring, -const unsigned int size) -{ -struct iavf_rx_buffer *rx_buffer; - - rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean]; -prefetchw (rx_buffer->page); -if (!size) -return rx_buffer; - -/* we are reusing so sync this buffer for CPU use */ -dma_sync_single_range_for_cpu (rx_ring->dev, - rx_buffer->dma, - rx_buffer->page_offset, - size, -DMA_FROM_DEVICE ); - -return rx_buffer; + rx_buffer->offset + hr, size, rx_buffer->truesize); }/** * iavf_build_skb - Build skb around an existing buffer -* @rx_ring: Rx descriptor ring to transact packets on * @rx_buffer: Rx buffer to pull data from * @size: size of buffer to add to skb * * This function builds an skb around an existing Rx buffer, taking care * to set up the skb correctly and avoid any memcpy overhead. */ -static struct sk_buff *iavf_build_skb (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *rx_buffer, +static struct sk_buff *iavf_build_skb (const struct libie_rx_buffer *rx_buffer,unsigned int size) { -void *va; -#if (PAGE_SIZE <8192 ) -unsigned int truesize =iavf_rx_pg_size (rx_ring) /2 ; -#else -unsigned int truesize =SKB_DATA_ALIGN (sizeof (struct skb_shared_info )) + -SKB_DATA_ALIGN (IAVF_SKB_PAD + size); -#endif +struct page *page = rx_buffer->page; +u32 hr = page->pp->p.offset;struct sk_buff *skb; +void *va; -if (!rx_buffer || !size) -return NULL ;/* prefetch first cache line of first page */ - va =page_address (rx_buffer->page) + rx_buffer->page_offset; -net_prefetch (va); + va =page_address (page) + rx_buffer->offset; +net_prefetch (va + hr);/* build an skb around the page buffer */ - skb =napi_build_skb (va -IAVF_SKB_PAD , truesize); -if (unlikely (!skb)) + skb =napi_build_skb (va, rx_buffer->truesize); +if (unlikely (!skb)) { +page_pool_put_page (page->pp, page, size, true);return NULL ; + } + +skb_mark_for_recycle (skb);/* update pointers within the skb to store the data */ -skb_reserve (skb,IAVF_SKB_PAD ); +skb_reserve (skb, hr);__skb_put (skb, size);return skb; } -/** -* iavf_put_rx_buffer - Unmap used buffer -* @rx_ring: rx descriptor ring to transact packets on -* @rx_buffer: rx buffer to pull data from -* -* This function will unmap the buffer after it's written by HW. -*/ -static void iavf_put_rx_buffer (struct iavf_ring *rx_ring, -struct iavf_rx_buffer *rx_buffer) -{ -if (!rx_buffer) -return ; - -/* we are not reusing the buffer so unmap it */ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -iavf_rx_pg_size (rx_ring), -DMA_FROM_DEVICE ,IAVF_RX_DMA_ATTR ); -} -/** * iavf_is_non_eop - process handling of non-EOP buffers * @rx_ring: Rx ring being processed @@ -1259,7 +1138,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)bool failure = false;while (likely (total_rx_packets < (unsigned int )budget)) { -struct iavf_rx_buffer *rx_buffer; +struct libie_rx_buffer *rx_buffer;union iavf_rx_desc *rx_desc;unsigned int size;u16 vlan_tag =0 ;@@ -1295,26 +1174,24 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)IAVF_RXD_QW1_LENGTH_PBUF_SHIFT ;iavf_trace (clean_rx_irq, rx_ring, rx_desc, skb); - rx_buffer =iavf_get_rx_buffer (rx_ring, size); + + rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean]; +if (!libie_rx_sync_for_cpu (rx_buffer, size)) + goto skip_data;/* retrieve a buffer from the ring */ if (skb) -iavf_add_rx_frag (rx_ring, rx_buffer, skb, size); +iavf_add_rx_frag (skb, rx_buffer, size);else - skb =iavf_build_skb (rx_ring, rx_buffer, size); - -iavf_put_rx_buffer (rx_ring, rx_buffer); + skb =iavf_build_skb (rx_buffer, size);/* exit if we failed to retrieve a buffer */ if (!skb) { rx_ring->rx_stats.alloc_buff_failed++; -__free_pages (rx_buffer->page, -iavf_rx_pg_order (rx_ring)); - rx_buffer->page =NULL ;break ; } - rx_buffer->page =NULL ; +skip_data: cleaned_count++;if (iavf_is_non_eop (rx_ring, rx_desc, skb))drivers/net/ethernet/intel/iavf/iavf_txrx.h +5
-69@@ -81,70 +81,8 @@ enum iavf_dyn_idx_t { BIT_ULL(IAVF_FILTER_PCTYPE_NONF_UNICAST_IPV6_UDP) | \ BIT_ULL(IAVF_FILTER_PCTYPE_NONF_MULTICAST_IPV6_UDP)) -/* Supported Rx Buffer Sizes (a multiple of 128) */ -#define IAVF_RXBUFFER_1536 1536/* 128B aligned standard Ethernet frame */ -#define IAVF_RXBUFFER_2048 2048 -#define IAVF_RXBUFFER_3072 3072/* Used for large frames w/ padding */ -#define IAVF_MAX_RXBUFFER 9728/* largest size for single descriptor */ - -#define IAVF_PACKET_HDR_PAD (ETH_HLEN + ETH_FCS_LEN + (VLAN_HLEN * 2))#define iavf_rx_desc iavf_32byte_rx_desc -#define IAVF_RX_DMA_ATTR \ - (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING) - -/* Attempt to maximize the headroom available for incoming frames. We -* use a 2K buffer for receives and need 1536/1534 to store the data for -* the frame. This leaves us with 512 bytes of room. From that we need -* to deduct the space needed for the shared info and the padding needed -* to IP align the frame. -* -* Note: For cache line sizes 256 or larger this value is going to end -* up negative. In these cases we should fall back to the legacy -* receive path. -*/ -#if (PAGE_SIZE <8192 ) -#define IAVF_2K_TOO_SMALL_WITH_PADDING \ -((NET_SKB_PAD + IAVF_RXBUFFER_1536) > SKB_WITH_OVERHEAD(IAVF_RXBUFFER_2048)) - -static inline int iavf_compute_pad (int rx_buf_len) -{ -int page_size, pad_size; - - page_size =ALIGN (rx_buf_len,PAGE_SIZE /2 ); - pad_size =SKB_WITH_OVERHEAD (page_size) - rx_buf_len; - -return pad_size; -} - -static inline int iavf_skb_pad (void ) -{ -int rx_buf_len; - -/* If a 2K buffer cannot handle a standard Ethernet frame then -* optimize padding for a 3K buffer instead of a 1.5K buffer. -* -* For a 3K buffer we need to add enough padding to allow for -* tailroom due to NET_IP_ALIGN possibly shifting us out of -* cache-line alignment. -*/ -if (IAVF_2K_TOO_SMALL_WITH_PADDING ) - rx_buf_len =IAVF_RXBUFFER_3072 +SKB_DATA_ALIGN (NET_IP_ALIGN ); -else - rx_buf_len =IAVF_RXBUFFER_1536 ; - -/* if needed make room for NET_IP_ALIGN */ - rx_buf_len -=NET_IP_ALIGN ; - -return iavf_compute_pad (rx_buf_len); -} - -#define IAVF_SKB_PAD iavf_skb_pad() -#else -#define IAVF_2K_TOO_SMALL_WITH_PADDING false -#define IAVF_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN) -#endif -/** * iavf_test_staterr - tests bits in Rx descriptor status and error fields * @rx_desc: pointer to receive descriptor (in le64 format) @@ -263,12 +201,6 @@ struct iavf_tx_buffer {u32 tx_flags; }; -struct iavf_rx_buffer { -dma_addr_t dma; -struct page *page; -__u32 page_offset; -}; -struct iavf_queue_stats {u64 packets;u64 bytes;@@ -311,16 +243,19 @@ enum iavf_ring_state_t {struct iavf_ring {struct iavf_ring *next;/* pointer to next ring in q_vector */ void *desc;/* Descriptor ring memory */ -struct device *dev;/* Used for DMA mapping */ +union { +struct page_pool *pp;/* Used on Rx for buffer management */ +struct device *dev;/* Used on Tx for DMA mapping */ + };struct net_device *netdev;/* netdev ring maps to */ union { +struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi;
drivers/net/ethernet/intel/iavf/iavf_txrx.h enum iavf_ring_state_t { · line 261
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1
-2-struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;/* high bit set means dynamic, use accessors routines to read/write. * hardware only supports 2us resolution for the ITR registers.
I'm assuming "tail" was moved here since it is a pointer and fills a hole?
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1
-2@@ -329,9 +264,8 @@ struct iavf_ring {*/ u16 itr_setting; -u16 count;/* Number of descriptors */ u16 reg_idx;/* HW register index of the ring */ -u16 rx_buf_len; +u16 count;/* Number of descriptors */
Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-1/* used in interrupt processing */ u16 next_to_use;@@ -398,17 +332,6 @@ struct iavf_ring_container {#define iavf_for_each_ring (pos, head) \ for (pos = (head).ring; pos != NULL; pos = pos->next) -static inline unsigned int iavf_rx_pg_order (struct iavf_ring *ring)
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-10-{ -#if (PAGE_SIZE <8192 ) -if (ring->rx_buf_len > (PAGE_SIZE /2 )) -return 1 ; -#endif -return 0 ; -} - -#define iavf_rx_pg_size (_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring)) -
All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.
Quoted 60 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0bool iavf_alloc_rx_buffers (struct iavf_ring *rxr,u16 cleaned_count);netdev_tx_t iavf_xmit_frame (struct sk_buff *skb,struct net_device *netdev);int iavf_setup_tx_descriptors (struct iavf_ring *tx_ring);drivers/net/ethernet/intel/iavf/iavf_type.h +0
-2@@ -10,8 +10,6 @@#include "iavf_adminq.h" #include "iavf_devids.h" -#define IAVF_RXQ_CTX_DBUFF_SHIFT 7 -/* IAVF_MASK is a macro used on 32 bit registers */ #define IAVF_MASK (mask, shift) ((u32)(mask) << (shift))drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +8
-7@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2013 - 2018 Intel Corporation. */ +#include <linux/net/intel/libie/rx.h> +#include "iavf.h" #include "iavf_prototype.h" #include "iavf_client.h" @@ -269,14 +271,13 @@ int iavf_get_vf_vlan_v2_caps(struct iavf_adapter *adapter)void iavf_configure_queues (struct iavf_adapter *adapter) {struct virtchnl_vsi_queue_config_info *vqci; -int i, max_frame = adapter->vf_res->max_mtu; +u32 i, max_frame = adapter->vf_res->max_mtu; +struct iavf_ring *rxq = adapter->rx_rings;int pairs = adapter->num_active_queues;struct virtchnl_queue_pair_info *vqpi; +u32 hr, max_len;size_t len; -if (max_frame >IAVF_MAX_RXBUFFER || !max_frame) - max_frame =IAVF_MAX_RXBUFFER ; -if (adapter->current_op !=VIRTCHNL_OP_UNKNOWN ) {/* bail because we already have a command pending */ dev_err(&adapter->pdev->dev, "Cannot configure queues, command %d pending\n",@@ -289,9 +290,9 @@ void iavf_configure_queues(struct iavf_adapter *adapter)if (!vqci)return ; -/* Limit maximum frame size when jumbo frames is not enabled */ -if (adapter->netdev->mtu <=ETH_DATA_LEN ) - max_frame =IAVF_RXBUFFER_1536 -NET_IP_ALIGN ; + hr = rxq->pp->p.offset; + max_len = rxq->pp->p.max_len; + max_frame =min_not_zero (max_frame,LIBIE_MAX_RX_FRM_LEN (hr)); vqci->vsi_id = adapter->vsi_res->vsi_id; vqci->num_queue_pairs = pairs;
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0
-3@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter) vqpi->rxq.ring_len = adapter->rx_rings[i].count; vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma; vqpi->rxq.max_pkt_size = max_frame; - vqpi->rxq.databuffer_size = -ALIGN (adapter->rx_rings[i].rx_buf_len, -BIT_ULL (IAVF_RXQ_CTX_DBUFF_SHIFT ));
Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1
-1+ vqpi->rxq.databuffer_size = max_len; vqpi++; } --2.41.0
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRe: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700
Quoted 15 lines
Quoted 13 lines · Alexander Lobakin
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.
Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Lobakin <[email protected]>One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.
min_mtu and watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s
[...]
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-6- bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; -memset (rx_ring->rx_bi,0 , bi_size); - -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); -
Quoted 7 lines
I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after
setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things insetup_rx_descriptors.
Both arrays will be freed couple instructions below, why zero them?
Quoted 5 lines
Quoted 5 lines
rx_ring->next_to_clean = 0; rx_ring->next_to_use = 0;}
[...]
Quoted 12 lines
Quoted 12 lines
struct
net_device*netdev; /* netdev ring maps to */
union {drivers/net/ethernet/intel/iavf/iavf_txrx.h +2
-2+struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi; -struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;
Quoted 4 lines
Quoted 2 lines
/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.I'm assuming "tail" was moved here since it is a pointer and fills a hole?
(see above)
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0@@ -329,9 +264,8 @@ struct iavf_ring {*/ u16 itr_setting;
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1
-2-u16 count;/* Number of descriptors */ u16 reg_idx;/* HW register index of the ring */ -u16 rx_buf_len; +u16 count;/* Number of descriptors */ Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.
With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0/* used in interrupt processing */ u16 next_to_use;@@ -398,17 +332,6 @@ struct iavf_ring_container {#define iavf_for_each_ring (pos, head) \ for (pos = (head).ring; pos != NULL; pos = pos->next)
Quoted 16 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-11-static inline unsigned int iavf_rx_pg_order (struct iavf_ring *ring) -{ -#if (PAGE_SIZE <8192 ) -if (ring->rx_buf_len > (PAGE_SIZE /2 )) -return 1 ; -#endif -return 0 ; -} - -#define iavf_rx_pg_size (_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring)) -All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.
This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0bool iavf_alloc_rx_buffers (struct iavf_ring *rxr,u16 cleaned_count);netdev_tx_t iavf_xmit_frame (struct sk_buff *skb,struct net_device *netdev);int iavf_setup_tx_descriptors (struct iavf_ring *tx_ring);
[...]
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0
-3@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter) vqpi->rxq.ring_len = adapter->rx_rings[i].count; vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma; vqpi->rxq.max_pkt_size = max_frame; - vqpi->rxq.databuffer_size = -ALIGN (adapter->rx_rings[i].rx_buf_len, -BIT_ULL (IAVF_RXQ_CTX_DBUFF_SHIFT ));Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.
See the previous commit, the place where I calculate max_len for the PP params. 128 byte is Intel-wide HW req, so it lives there now.
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1
-0+ vqpi->rxq.databuffer_size = max_len; vqpi++;
Thanks,
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Re: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
Quoted 48 lines · Alexander Lobakin
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700Quoted 15 lines
Quoted 13 lines · Alexander Lobakin
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Lobakin <[email protected]>One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.
min_mtuand watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s[...]
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-6- bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; -memset (rx_ring->rx_bi,0 , bi_size); - -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); -Quoted 7 lines
I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after
setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things insetup_rx_descriptors.
Quoted 1 line
Both arrays will be freed couple instructions below, why zero them?
Ugh. You are right, but not for a good reason. So the other Intel drivers in the past would be doing the clean_rx_ring calls on the _down() with the freeing of resources on _close(). Specifically it allowed reducing the overhead for things like resets or setting changes since it didn't require reallocating the descriptor rings and buffer info structures.
I guess you are good to remove these since this code doesn't do that.
Quoted 40 lines
Quoted 5 lines
Quoted 5 lines
rx_ring->next_to_clean = 0; rx_ring->next_to_use = 0;}
[...]
Quoted 12 lines
Quoted 12 lines
struct
net_device*netdev; /* netdev ring maps to */
union {drivers/net/ethernet/intel/iavf/iavf_txrx.h +2
-2+struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi; -struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;Quoted 4 lines
Quoted 2 lines
/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.I'm assuming "tail" was moved here since it is a pointer and fills a hole?
(see above)
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0@@ -329,9 +264,8 @@ struct iavf_ring {*/ u16 itr_setting;Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1
-2-u16 count;/* Number of descriptors */ u16 reg_idx;/* HW register index of the ring */ -u16 rx_buf_len; +u16 count;/* Number of descriptors */ Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.
Quoted 4 lines
With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.
The performance impact should be minimal. Odds are the placement was the way it was since it was probably just copying the original code that has been there since igb/ixgbe. The general idea is just keep the read-mostly items grouped at the top and try to order them somewhat by frequency of being read so that wherever the cache line ends up you won't take much of a penalty as hopefully you will just have the infrequently read items end up getting pulled into the active cache line.
Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0/* used in interrupt processing */ u16 next_to_use;@@ -398,17 +332,6 @@ struct iavf_ring_container {#define iavf_for_each_ring (pos, head) \ for (pos = (head).ring; pos != NULL; pos = pos->next)Quoted 16 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-11-static inline unsigned int iavf_rx_pg_order (struct iavf_ring *ring) -{ -#if (PAGE_SIZE <8192 ) -if (ring->rx_buf_len > (PAGE_SIZE /2 )) -return 1 ; -#endif -return 0 ; -} - -#define iavf_rx_pg_size (_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring)) -All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.
Quoted 4 lines
This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.
It all depends on how much is having to be added to accommodate this. In my mind when we did away with the page splitting/recycling we also did away with the need for the higher order pages. That is why I was thinking it might make more sense there as it would just be more removals with very few if any additions needed to support it.
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0bool iavf_alloc_rx_buffers (struct iavf_ring *rxr,u16 cleaned_count);netdev_tx_t iavf_xmit_frame (struct sk_buff *skb,struct net_device *netdev);int iavf_setup_tx_descriptors (struct iavf_ring *tx_ring);[...]
Quoted 10 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +0
-3@@ -309,9 +310,7 @@ void iavf_configure_queues(struct iavf_adapter *adapter) vqpi->rxq.ring_len = adapter->rx_rings[i].count; vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma; vqpi->rxq.max_pkt_size = max_frame; - vqpi->rxq.databuffer_size = -ALIGN (adapter->rx_rings[i].rx_buf_len, -BIT_ULL (IAVF_RXQ_CTX_DBUFF_SHIFT ));Is this rendered redundant by something? Seems like you should be guaranteeing somewhere that you are still aligned to this.
Quoted 2 lines
See the previous commit, the place where I calculate
max_lenfor the PP params. 128 byte is Intel-wide HW req, so it lives there now.
Okay, that is the piece I missed. It was converted from a BIT_ULL(7) to just a 128. Thanks.
Quoted 6 lines
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c void iavf_configure_queues(struct iavf_adapter *adapter) · line 314
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1
-0+ vqpi->rxq.databuffer_size = max_len; vqpi++;Thanks,
Olek
Alexander Lobakin
maintainerRe: [Intel-wired-lan] [PATCH RFC net-next v4 6/9] iavf: switch to Page Pool
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 10:28:06 -0700
Quoted 52 lines
Quoted 48 lines · Alexander Lobakin
From: Alexander Duyck <[email protected]>
Date: Thu, 6 Jul 2023 08:26:00 -0700Quoted 15 lines
Quoted 13 lines · Alexander Lobakin
Now that the IAVF driver simply uses
dev_alloc_page()+free_page()with no custom recycling logics, it can easily be switched to using Page Pool / libie API instead.
This allows to removing the whole dancing around headroom, HW buffer size, and page order. All DMA-for-device is now done in the PP core, for-CPU -- in the libie helper.
Useskb_mark_for_recycle()to bring back the recycling and restore the performance. Speaking of performance: on par with the baseline and faster with the PP optimization series applied. But the memory usage for 1500b MTU is now almost 2x lower (x86_64) thanks to allocating a page every second descriptor.Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Lobakin <[email protected]>One thing I am noticing is that there seems to be a bunch of cleanup changes in here as well. Things like moving around values within structures which I am assuming are to fill holes. You may want to look at breaking some of those out as it makes it a bit harder to review this since they seem like unrelated changes.
min_mtuand watchdog are unrelated, I'll drop those.
Moving tail pointer around was supposed to land in a different commit, not this one, as I wrote 10 minutes ago already :s[...]
Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c static void iavf_clean_rx_ring(struct iavf_ring *rx_ring) · line 737
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.c +0
-6- bi_size =sizeof (struct iavf_rx_buffer ) * rx_ring->count; -memset (rx_ring->rx_bi,0 , bi_size); - -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); -Quoted 7 lines
I have some misgivings about not clearing these. We may want to double check to verify the code paths are resilient enough that it won't cause any issues w/ repeated up/down testing on the interface. The general idea is to keep things consistent w/ the state after
setup_rx_descriptors. If we don't need this when we don't need to be calling the zalloc or calloc version of things insetup_rx_descriptors.Quoted 1 line
Both arrays will be freed couple instructions below, why zero them?
Quoted 8 lines
Ugh. You are right, but not for a good reason. So the other Intel drivers in the past would be doing the
clean_rx_ringcalls on the_down()with the freeing of resources on_close(). Specifically it allowed reducing the overhead for things like resets or setting changes since it didn't require reallocating the descriptor rings and buffer info structures.I guess you are good to remove these since this code doesn't do that.
We might go back to this to not always do a full circle when not needed, but currently this is redundant.
Quoted 46 lines
Quoted 41 lines
Quoted 6 lines
Quoted 6 lines
rx_ring->next_to_clean = 0; rx_ring->next_to_use = 0;}
[...]
Quoted 12 lines
Quoted 12 lines
struct
net_device*netdev; /* netdev ring maps to */
union {drivers/net/ethernet/intel/iavf/iavf_txrx.h +2
-2+struct libie_rx_buffer *rx_bi;struct iavf_tx_buffer *tx_bi; -struct iavf_rx_buffer *rx_bi; };DECLARE_BITMAP (state, __IAVF_RING_STATE_NBITS); +u8 __iomem *tail;u16 queue_index;/* Queue number of ring */ u8 dcb_tc;/* Traffic class of ring */ -u8 __iomem *tail;Quoted 4 lines
Quoted 2 lines
/* high bit set means dynamic, use accessors routines to read/write.
* hardware only supports 2us resolution for the ITR registers.I'm assuming "tail" was moved here since it is a pointer and fills a hole?
(see above)
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0@@ -329,9 +264,8 @@ struct iavf_ring {*/ u16 itr_setting;Quoted 7 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring { · line 268
Quoted 4 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +1
-2-u16 count;/* Number of descriptors */ u16 reg_idx;/* HW register index of the ring */ -u16 rx_buf_len; +u16 count;/* Number of descriptors */ Why move count down here? It is moving the constant value that is read-mostly into an area that will be updated more often.
Quoted 4 lines
With the ::tail put in a different slot, ::count was landing in a different cacheline. I wanted to avoid this. But now I feel like I was just lazy and must've tested both variants to see if this move affects performance. I'll play with this one in the next rev.
Quoted 8 lines
The performance impact should be minimal. Odds are the placement was the way it was since it was probably just copying the original code that has been there since igb/ixgbe. The general idea is just keep the read-mostly items grouped at the top and try to order them somewhat by frequency of being read so that wherever the cache line ends up you won't take much of a penalty as hopefully you will just have the infrequently read items end up getting pulled into the active cache line.
+
Quoted 30 lines
Quoted 25 lines
Quoted 8 lines
Quoted 8 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-0/* used in interrupt processing */ u16 next_to_use;@@ -398,17 +332,6 @@ struct iavf_ring_container {#define iavf_for_each_ring (pos, head) \ for (pos = (head).ring; pos != NULL; pos = pos->next)Quoted 16 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h struct iavf_ring_container { · line 411
Quoted 11 lines
drivers/net/ethernet/intel/iavf/iavf_txrx.h +0
-11-static inline unsigned int iavf_rx_pg_order (struct iavf_ring *ring) -{ -#if (PAGE_SIZE <8192 ) -if (ring->rx_buf_len > (PAGE_SIZE /2 )) -return 1 ; -#endif -return 0 ; -} - -#define iavf_rx_pg_size (_ring) (PAGE_SIZE << iavf_rx_pg_order(_ring)) -All this code probably could have been removed in an earlier patch since I don't think we need the higher order pages once we did away with the recycling. Odds are we can probably move this into the recycling code removal.
Quoted 4 lines
This went here as I merged "always use order 0" commit with "switch to Page Pool". In general, IIRC having removals of all the stuff at once in one commit (#2) was less readable than the current version, but I'll double-check.
Quoted 5 lines
It all depends on how much is having to be added to accommodate this. In my mind when we did away with the page splitting/recycling we also did away with the need for the higher order pages. That is why I was thinking it might make more sense there as it would just be more removals with very few if any additions needed to support it.
Yeah, I'll try and see whether any pieces can be grouped differently for better reading/logics.
[...]
Thanks!
Olek
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 7/9 libie: add common queue stats
In reply to Alexander LobakinNext stop, per-queue private stats. They have only subtle differences from driver to driver and can easily be resolved.
Define common structures, inline helpers and Ethtool helpers to collect, update and export the statistics. Use u64_stats_t right from the start, as well as the corresponding helpers to ensure tear-free operations. For the NAPI parts of both Rx and Tx, also define small onstack containers to update them in polling loops and then sync the actual containers once a loop ends.
The drivers will be switched to use this API later on a per-driver basis, along with conversion to PP.
--- drivers/net/ethernet/intel/libie/Makefile | 1 + drivers/net/ethernet/intel/libie/stats.c | 119 ++++++++++++++ include/linux/net/intel/libie/stats.h | 179 ++++++++++++++++++++++ 3 files changed, 299 insertions(+) create mode 100644 drivers/net/ethernet/intel/libie/stats.c create mode 100644 include/linux/net/intel/libie/stats.h
drivers/net/ethernet/intel/libie/Makefile +1 -0
@@ -4,3 +4,4 @@ obj-$(CONFIG_LIBIE) += libie.o libie-objs += rx.o +libie-objs += stats.o
drivers/net/ethernet/intel/libie/stats.c new file +119 -0
119 lines not on this page · raw message
include/linux/net/intel/libie/stats.h new file +179 -0
179 lines not on this page · raw message
Signature · 6 lines
2.41.0 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 8/9 libie: add per-queue Page Pool stats
In reply to Alexander LobakinExpand the libie generic per-queue stats with the generic Page Pool stats provided by the API itself, when CONFIG_PAGE_POOL is enabled. When it's not, there'll be no such fields in the stats structure, so no space wasted.
They are also a bit special in terms of how they are obtained. One &page_pool accumulates statistics until it's destroyed obviously, which happens on ifdown. So, in order to not lose any statistics, get the stats and store in the queue container before destroying a pool. This container survives ifups/downs, so it basically stores the statistics accumulated since the very first pool was allocated on this queue. When it's needed to export the stats, first get the numbers from this container and then add the "live" numbers -- the ones that the current active pool returns. The result values will always represent the actual device-lifetime* stats.
There's a cast from &page_pool_stats to u64 * in a couple functions, but they are guarded with stats asserts to make sure it's safe to do. FWIW it saves a lot of object code.
--- drivers/net/ethernet/intel/libie/internal.h | 23 +++++++ drivers/net/ethernet/intel/libie/rx.c | 19 ++++++ drivers/net/ethernet/intel/libie/stats.c | 73 ++++++++++++++++++++- include/linux/net/intel/libie/rx.h | 4 ++ include/linux/net/intel/libie/stats.h | 39 ++++++++++- 5 files changed, 155 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/intel/libie/internal.h
drivers/net/ethernet/intel/libie/internal.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* libie internal declarations not to be used in drivers. +* +* Copyright(c) 2023 Intel Corporation. +*/ + +#ifndef __LIBIE_INTERNAL_H +#define __LIBIE_INTERNAL_H + +struct libie_rq_stats ; +struct page_pool ; + +#ifdef CONFIG_PAGE_POOL_STATS +void libie_rq_stats_sync_pp (struct libie_rq_stats *stats, +struct page_pool *pool); +#else +static inline void libie_rq_stats_sync_pp (struct libie_rq_stats *stats, +struct page_pool *pool) +{ +} +#endif + +#endif /* __LIBIE_INTERNAL_H */
drivers/net/ethernet/intel/libie/rx.c +19 -0
@@ -2,6 +2,7 @@/* Copyright(c) 2023 Intel Corporation. */ #include <linux/net/intel/libie/rx.h> +#include "internal.h" /* Rx buffer management */
@@ -57,6 +58,24 @@ struct page_pool *libie_rx_page_pool_create(struct napi_struct *napi, }EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_create,LIBIE ); +/** +* libie_rx_page_pool_destroy - destroy a &page_pool created by libie +* @pool: pool to destroy +* @stats: RQ stats from the ring (or %NULL to skip updating PP stats) +* +* As the stats usually has the same lifetime as the device, but PP is usually +* created/destroyed on ifup/ifdown, in order to not lose the stats accumulated +* during the last ifup, the PP stats need to be added to the driver stats +* container. Then the PP gets destroyed. +*/ +void libie_rx_page_pool_destroy (struct page_pool *pool, +struct libie_rq_stats *stats) +{ +libie_rq_stats_sync_pp (stats, pool); +page_pool_destroy (pool); +} +EXPORT_SYMBOL_NS_GPL (libie_rx_page_pool_destroy,LIBIE ); +/* O(1) converting i40e/ice/iavf's 8/10-bit hardware packet type to a parsed * bitfield struct. */
drivers/net/ethernet/intel/libie/stats.c +72 -1
98 lines not on this page · raw message
include/linux/net/intel/libie/rx.h +4 -0
@@ -62,8 +62,12 @@ struct libie_rx_buffer {u32 truesize; }; +struct libie_rq_stats ; +struct page_pool *libie_rx_page_pool_create (struct napi_struct *napi,u32 size); +void libie_rx_page_pool_destroy (struct page_pool *pool, +struct libie_rq_stats *stats);/** * libie_rx_alloc - allocate a new Rx buffer
include/linux/net/intel/libie/stats.h +37 -2
@@ -49,6 +49,17 @@* fragments: number of processed descriptors carrying only a fragment * alloc_page_fail: number of Rx page allocation fails * build_skb_fail: number of build_skb() fails +* pp_alloc_fast: pages taken from the cache or ring +* pp_alloc_slow: actual page allocations +* pp_alloc_slow_ho: non-order-0 page allocations +* pp_alloc_empty: number of times the pool was empty +* pp_alloc_refill: number of cache refills +* pp_alloc_waive: NUMA node mismatches during recycling +* pp_recycle_cached: direct recyclings into the cache +* pp_recycle_cache_full: number of times the cache was full +* pp_recycle_ring: recyclings into the ring +* pp_recycle_ring_full: number of times the ring was full +* pp_recycle_released_ref: pages released due to elevated refcnt */ #define DECLARE_LIBIE_RQ_NAPI_STATS (act) \
@@ -60,9 +71,29 @@ act(alloc_page_fail) \ act(build_skb_fail) +#ifdef CONFIG_PAGE_POOL_STATS +#define DECLARE_LIBIE_RQ_PP_STATS (act) \ + act(pp_alloc_fast) \ + act(pp_alloc_slow) \ + act(pp_alloc_slow_ho) \ + act(pp_alloc_empty) \ + act(pp_alloc_refill) \ + act(pp_alloc_waive) \ + act(pp_recycle_cached) \ + act(pp_recycle_cache_full) \ + act(pp_recycle_ring) \ + act(pp_recycle_ring_full) \ + act(pp_recycle_released_ref) +#else +#define DECLARE_LIBIE_RQ_PP_STATS (act) +#endif +#define DECLARE_LIBIE_RQ_STATS (act) \ DECLARE_LIBIE_RQ_NAPI_STATS(act) \ - DECLARE_LIBIE_RQ_FAIL_STATS(act) + DECLARE_LIBIE_RQ_FAIL_STATS(act) \ + DECLARE_LIBIE_RQ_PP_STATS(act) + +struct page_pool ;struct libie_rq_stats {struct u64_stats_sync syncp;
@@ -72,6 +103,9 @@ struct libie_rq_stats {#define act (s) u64_stats_t s;DECLARE_LIBIE_RQ_NAPI_STATS (act);DECLARE_LIBIE_RQ_FAIL_STATS (act); +struct_group (pp, + DECLARE_LIBIE_RQ_PP_STATS(act); + );#undef act };DECLARE_FLEX_ARRAY (u64_stats_t, raw);
@@ -110,7 +144,8 @@ libie_rq_napi_stats_add(struct libie_rq_stats *qs,u32 libie_rq_stats_get_sset_count (void );void libie_rq_stats_get_strings (u8 **data,u32 qid); -void libie_rq_stats_get_data (u64 **data,const struct libie_rq_stats *stats); +void libie_rq_stats_get_data (u64 **data,const struct libie_rq_stats *stats, +struct page_pool *pool);/* Tx per-queue stats: * packets: packets sent from this queue
Signature · 6 lines
2.41.0 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Alexander Lobakin
maintainerRFC PATCH v4 9/9 iavf: switch queue stats to libie
In reply to Alexander Lobakiniavf is pretty much ready for using the generic libie stats, so drop all the custom code and just use generic definitions. The only thing is that it previously lacked the counter of Tx queue stops. It's present in the other drivers, so add it here as well.
The rest is straightforward. There were two fields in the Tx stats struct, which didn't belong there. The first one has never been used, wipe it; and move the other to the queue structure. Plus move around a couple fields in &iavf_ring to account stats structs' alignment.
--- .../net/ethernet/intel/iavf/iavf_ethtool.c | 86 ++++------------- drivers/net/ethernet/intel/iavf/iavf_main.c | 2 + drivers/net/ethernet/intel/iavf/iavf_txrx.c | 92 ++++++++++--------- drivers/net/ethernet/intel/iavf/iavf_txrx.h | 54 +++-------- 4 files changed, 83 insertions(+), 151 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +19 -67
@@ -46,16 +46,6 @@ struct iavf_stats { .stat_offset = offsetof(_type, _stat) \ } -/* Helper macro for defining some statistics related to queues */ -#define IAVF_QUEUE_STAT (_name, _stat) \ - IAVF_STAT(struct iavf_ring, _name, _stat) - -/* Stats associated with a Tx or Rx ring */ -static const struct iavf_stats iavf_gstrings_queue_stats[] = { -IAVF_QUEUE_STAT ("%s-%u.packets" , stats.packets), -IAVF_QUEUE_STAT ("%s-%u.bytes" , stats.bytes), -}; -/** * iavf_add_one_ethtool_stat - copy the stat into the supplied buffer * @data: location to store the stat value
@@ -141,43 +131,6 @@ __iavf_add_ethtool_stats(u64 **data, void *pointer,#define iavf_add_ethtool_stats (data, pointer, stats) \ __iavf_add_ethtool_stats(data, pointer, stats, ARRAY_SIZE(stats)) -/** -* iavf_add_queue_stats - copy queue statistics into supplied buffer -* @data: ethtool stats buffer -* @ring: the ring to copy -* -* Queue statistics must be copied while protected by -* u64_stats_fetch_begin, so we can't directly use iavf_add_ethtool_stats. -* Assumes that queue stats are defined in iavf_gstrings_queue_stats. If the -* ring pointer is null, zero out the queue stat values and update the data -* pointer. Otherwise safely copy the stats from the ring into the supplied -* buffer and update the data pointer when finished. -* -* This function expects to be called while under rcu_read_lock(). -**/ -static void -iavf_add_queue_stats (u64 **data,struct iavf_ring *ring) -{ -const unsigned int size =ARRAY_SIZE (iavf_gstrings_queue_stats); -const struct iavf_stats *stats = iavf_gstrings_queue_stats; -unsigned int start; -unsigned int i; - -/* To avoid invalid statistics values, ensure that we keep retrying -* the copy until we get a consistent value according to -* u64_stats_fetch_retry. But first, make sure our ring is -* non-null before attempting to access its syncp. -*/ -do { - start = !ring ?0 :u64_stats_fetch_begin (&ring->syncp); -for (i =0 ; i < size; i++) -iavf_add_one_ethtool_stat (&(*data)[i], ring, &stats[i]); - }while (ring &&u64_stats_fetch_retry (&ring->syncp, start)); - -/* Once we successfully copy the stats in, update the data pointer */ - *data += size; -} -/** * __iavf_add_stat_strings - copy stat strings into ethtool buffer * @p: ethtool supplied buffer
@@ -237,8 +190,6 @@ static const struct iavf_stats iavf_gstrings_stats[] = {#define IAVF_STATS_LEN ARRAY_SIZE(iavf_gstrings_stats) -#define IAVF_QUEUE_STATS_LEN ARRAY_SIZE(iavf_gstrings_queue_stats) -/** * iavf_get_link_ksettings - Get Link Speed and Duplex settings * @netdev: network interface device structure
@@ -308,18 +259,22 @@ static int iavf_get_link_ksettings(struct net_device *netdev,**/ static int iavf_get_sset_count (struct net_device *netdev,int sset) { -/* Report the maximum number queues, even if not every queue is -* currently configured. Since allocation of queues is in pairs, -* use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set -* at device creation and never changes. -*/ +u32 num; -if (sset ==ETH_SS_STATS ) -return IAVF_STATS_LEN + - (IAVF_QUEUE_STATS_LEN *2 * - netdev->real_num_tx_queues); -else +switch (sset) { +case ETH_SS_STATS : +/* Per-queue */ + num =libie_rq_stats_get_sset_count (); + num +=libie_sq_stats_get_sset_count (); + num *= netdev->real_num_tx_queues; + +/* Global */ + num +=IAVF_STATS_LEN ; + +return num; +default :return -EINVAL ; + } }/**
@@ -346,15 +301,14 @@ static void iavf_get_ethtool_stats(struct net_device *netdev,* it to iterate over rings' stats. */ for (i =0 ; i < adapter->num_active_queues; i++) { -struct iavf_ring *ring; +const struct iavf_ring *ring;/* Tx rings stats */ - ring = &adapter->tx_rings[i]; -iavf_add_queue_stats (&data, ring); +libie_sq_stats_get_data (&data, &adapter->tx_rings[i].sq_stats);/* Rx rings stats */ ring = &adapter->rx_rings[i]; -iavf_add_queue_stats (&data, ring); +libie_rq_stats_get_data (&data, &ring->rq_stats, ring->pp); }rcu_read_unlock (); }
@@ -376,10 +330,8 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data)* real_num_tx_queues for both Tx and Rx queues. */ for (i =0 ; i < netdev->real_num_tx_queues; i++) { -iavf_add_stat_strings (&data, iavf_gstrings_queue_stats, -"tx" , i); -iavf_add_stat_strings (&data, iavf_gstrings_queue_stats, -"rx" , i); +libie_sq_stats_get_strings (&data, i); +libie_rq_stats_get_strings (&data, i); } }
drivers/net/ethernet/intel/iavf/iavf_main.c +2 -0
@@ -1566,12 +1566,14 @@ static int iavf_alloc_queues(struct iavf_adapter *adapter) tx_ring->itr_setting =IAVF_ITR_TX_DEF ;if (adapter->flags &IAVF_FLAG_WB_ON_ITR_CAPABLE ) tx_ring->flags |=IAVF_TXR_FLAGS_WB_ON_ITR ; +u64_stats_init (&tx_ring->sq_stats.syncp); rx_ring = &adapter->rx_rings[i]; rx_ring->queue_index = i; rx_ring->netdev = adapter->netdev; rx_ring->count = adapter->rx_desc_count; rx_ring->itr_setting =IAVF_ITR_RX_DEF ; +u64_stats_init (&rx_ring->rq_stats.syncp); } adapter->num_active_queues = num_active_queues;
drivers/net/ethernet/intel/iavf/iavf_txrx.c +48 -44
257 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_txrx.h +14 -40
87 lines not on this page · raw message
Signature · 6 lines
2.41.0 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan