net/mlx5e: Add support for devmem and io_uring TCP zero-copy
Subsystems (7)
Mark Bloch
maintainerv6 cover net/mlx5e: Add support for devmem and io_uring TCP zero-copy
This series adds support for zerocopy rx TCP with devmem and io_uring for ConnectX7 NICs and above. For performance reasons and simplicity HW-GRO will also be turned on when header-data split mode is on.
Performance
===========
Test setup:
* CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz (single NUMA)
* NIC: ConnectX7
* Benchmarking tool: kperf [0]
* Single TCP flow
* Test duration: 60s
With application thread and interrupts pinned to the *same* core:
|------+-----------+----------| | MTU | epoll | io_uring | |------+-----------+----------| | 1500 | 61.6 Gbps | 114 Gbps | | 4096 | 69.3 Gbps | 151 Gbps | | 9000 | 67.8 Gbps | 187 Gbps | |------+-----------+----------|
The CPU usage for io_uring is 95%.
Reproduction steps for io_uring:
server --no-daemon -a 2001:db8::1 --no-memcmp --iou --iou_sendzc \ --iou_zcrx --iou_dev_name eth2 --iou_zcrx_queue_id 2
server --no-daemon -a 2001:db8::2 --no-memcmp --iou --iou_sendzc
client --src 2001:db8::2 --dst 2001:db8::1 \ --msg-zerocopy -t 60 --cpu-min=2 --cpu-max=2
Patch overview:
================
First, a netmem API for skb_can_coalesce is added to the core to be able to do skb fragment coalescing on netmems.
The next patches introduce some cleanups in the internal SHAMPO code and improvements to hw gro capability checks in FW.
A separate page_pool is introduced for headers, to be used only when the rxq has a memory provider.
Then the driver is converted to use the netmem API and to allow support for unreadable netmem page pool.
The queue management ops are implemented.
Finally, the tcp-data-split ring parameter is exposed.
Changelog
=========
Changes from v5 [5]:
- Added a TODO in mlx5e_queue_stop regarding queue restart.
- Added Reviewed-by tag.
Changes from v4 [4]:
- Addressed silly return before goto.
- Removed extraneous '\n' and used NL_SET_ERR_MSG_MOD.
- Removed unnecessary netmem_is_net_iov() check.
- Added comment for non HDS packets being dropped when unreadable memory is used.
- Added page_pool_dev_alloc_netmems() helper.
- Added Reviewed-by tags.
Changes from v3 [3]:
- Dropped ethtool stats for hd_page_pool.
Changes from v2 [2]:
- Added support for netmem TX.
- Changed skb_can_coalesce_netmem() based on Mina's suggestion.
- Reworked tcp_data_split setting to no longer change HW-GRO in wanted_features.
- Switched to a single page pool when rxq has no memory providers.
Changes from v1 [1]:
- Added support for skb_can_coalesce_netmem().
- Avoid netmem_to_page() casts in the driver.
- Fixed code to abide 80 char limit with some exceptions to avoid code churn.
References
==========
[0] kperf: git://git.kernel.dk/kperf.git
[1] v1: https://lore.kernel.org/all/[email protected]/
[2] v2: https://lore.kernel.org/all/[email protected]/
[3] v3: https://lore.kernel.org/netdev/[email protected]/
[4] v4: https://lore.kernel.org/all/[email protected]/
[5] v5: https://lore.kernel.org/netdev/[email protected]/
Dragos Tatulea (4):
net: Allow const args for of page_to_netmem()
net: Add skb_can_coalesce for netmem
page_pool: Add page_pool_dev_alloc_netmems helper net/mlx5e: Add TX support for netmems
Saeed Mahameed (8):
net/mlx5e: SHAMPO: Reorganize mlx5_rq_shampo_alloc net/mlx5e: SHAMPO: Remove redundant params net/mlx5e: SHAMPO: Improve hw gro capability checking net/mlx5e: SHAMPO: Separate pool for headers net/mlx5e: Convert over to netmem
net/mlx5e: Add support for UNREADABLE netmem page pools net/mlx5e: Implement queue mgmt ops and single channel swap net/mlx5e: Support ethtool tcp-data-split settings
drivers/net/ethernet/mellanox/mlx5/core/en.h | 11 +- .../ethernet/mellanox/mlx5/core/en/params.c | 36 ++- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 +- .../ethernet/mellanox/mlx5/core/en_ethtool.c | 33 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 306 +++++++++++++----- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 138 ++++---- include/linux/skbuff.h | 12 +- include/net/netmem.h | 2 +- include/net/page_pool/helpers.h | 7 + 9 files changed, 381 insertions(+), 167 deletions(-)
base-commit: 8909f5f4ecd551c2299b28e05254b77424c8c7dc
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 1/12 net: Allow const args for of page_to_netmem()
From: Dragos Tatulea <[email protected]>
This allows calling page_to_netmem() with a const page * argument.
--- include/net/netmem.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/netmem.h +1 -1
@@ -143,7 +143,7 @@ static inline netmem_ref net_iov_to_netmem(struct net_iov *niov)return (__force netmem_ref)((unsigned long )niov |NET_IOV ); } -static inline netmem_ref page_to_netmem (struct page *page) +static inline netmem_ref page_to_netmem (const struct page *page) {return (__force netmem_ref )page; }
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 2/12 net: Add skb_can_coalesce for netmem
In reply to Mark BlochFrom: Dragos Tatulea <[email protected]>
Allow drivers that have moved over to netmem to do fragment coalescing.
--- include/linux/skbuff.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
include/linux/skbuff.h +9 -3
@@ -3873,20 +3873,26 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int lbool csum_and_copy_from_iter_full (void *addr,size_t bytes,__wsum *csum,struct iov_iter *i)__must_check ; -static inline bool skb_can_coalesce (struct sk_buff *skb,int i, -const struct page *page,int off) +static inline bool skb_can_coalesce_netmem (struct sk_buff *skb,int i, +netmem_ref netmem,int off) {if (skb_zcopy (skb))return false;if (i) {const skb_frag_t *frag = &skb_shinfo (skb)->frags[i -1 ]; -return page ==skb_frag_page (frag) && +return netmem ==skb_frag_netmem (frag) && off ==skb_frag_off (frag) +skb_frag_size (frag); }return false; } +static inline bool skb_can_coalesce (struct sk_buff *skb,int i, +const struct page *page,int off) +{ +return skb_can_coalesce_netmem (skb, i,page_to_netmem (page), off); +} +static inline int __skb_linearize (struct sk_buff *skb) {return __pskb_pull_tail (skb, skb->data_len) ?0 : -ENOMEM ;
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 3/12 page_pool: Add page_pool_dev_alloc_netmems helper
In reply to Mark BlochFrom: Dragos Tatulea <[email protected]>
This is the netmem counterpart of page_pool_dev_alloc_pages() which uses the default GFP flags for RX.
--- include/net/page_pool/helpers.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/net/page_pool/helpers.h +7 -0
@@ -153,6 +153,13 @@ static inline netmem_ref page_pool_dev_alloc_netmem(struct page_pool *pool,return page_pool_alloc_netmem (pool, offset, size, gfp); } +static inline netmem_ref page_pool_dev_alloc_netmems (struct page_pool *pool) +{ +gfp_t gfp =GFP_ATOMIC | __GFP_NOWARN; + +return page_pool_alloc_netmems (pool, gfp); +} +static inline struct page *page_pool_alloc (struct page_pool *pool,unsigned int *offset,unsigned int *size,gfp_t gfp)
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 4/12 net/mlx5e: SHAMPO: Reorganize mlx5_rq_shampo_alloc
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
Drop redundant SHAMPO structure alloc/free functions.
Gather together function calls pertaining to header split info, pass header per WQE (hd_per_wqe) as parameter to those function to avoid use before initialization future mistakes.
Allocate HW GRO related info outside of the header related info scope.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 - .../net/ethernet/mellanox/mlx5/core/en_main.c | 135 +++++++++--------- 2 files changed, 66 insertions(+), 70 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -1
@@ -638,7 +638,6 @@ struct mlx5e_shampo_hd {struct mlx5e_frag_page *pages;u32 hd_per_wq;u16 hd_per_wqe; -u16 pages_per_wq;unsigned long *bitmap;u16 pi;u16 ci;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +66 -69
187 lines not on this page · raw message
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 5/12 net/mlx5e: SHAMPO: Remove redundant params
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
Two SHAMPO params are static and always the same, remove them from the global mlx5e_params struct.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 --- .../ethernet/mellanox/mlx5/core/en/params.c | 36 ++++++++++--------- .../net/ethernet/mellanox/mlx5/core/en_main.c | 4 --- 3 files changed, 20 insertions(+), 24 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -4
@@ -278,10 +278,6 @@ enum packet_merge {struct mlx5e_packet_merge_param {enum packet_merge type;u32 timeout; -struct { -u8 match_criteria_type; -u8 alignment_granularity; - } shampo; };struct mlx5e_params {
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +20 -16
@@ -901,6 +901,7 @@ int mlx5e_build_rq_param(struct mlx5_core_dev *mdev, {void *rqc = param->rqc;void *wq =MLX5_ADDR_OF (rqc, rqc, wq); +u32 lro_timeout;int ndsegs =1 ;int err;
@@ -926,22 +927,25 @@ int mlx5e_build_rq_param(struct mlx5_core_dev *mdev,MLX5_SET (wq, wq, log_wqe_stride_size, log_wqe_stride_size -MLX5_MPWQE_LOG_STRIDE_SZ_BASE );MLX5_SET (wq, wq, log_wq_sz,mlx5e_mpwqe_get_log_rq_size (mdev, params, xsk)); -if (params->packet_merge.type ==MLX5E_PACKET_MERGE_SHAMPO ) { -MLX5_SET (wq, wq, shampo_enable, true); -MLX5_SET (wq, wq, log_reservation_size, -mlx5e_shampo_get_log_rsrv_size (mdev, params)); -MLX5_SET (wq, wq, - log_max_num_of_packets_per_reservation, -mlx5e_shampo_get_log_pkt_per_rsrv (mdev, params)); -MLX5_SET (wq, wq, log_headers_entry_size, -mlx5e_shampo_get_log_hd_entry_size (mdev, params)); -MLX5_SET (rqc, rqc, reservation_timeout, -mlx5e_choose_lro_timeout (mdev,MLX5E_DEFAULT_SHAMPO_TIMEOUT )); -MLX5_SET (rqc, rqc, shampo_match_criteria_type, - params->packet_merge.shampo.match_criteria_type); -MLX5_SET (rqc, rqc, shampo_no_match_alignment_granularity, - params->packet_merge.shampo.alignment_granularity); - } +if (params->packet_merge.type !=MLX5E_PACKET_MERGE_SHAMPO ) +break ; + +MLX5_SET (wq, wq, shampo_enable, true); +MLX5_SET (wq, wq, log_reservation_size, +mlx5e_shampo_get_log_rsrv_size (mdev, params)); +MLX5_SET (wq, wq, + log_max_num_of_packets_per_reservation, +mlx5e_shampo_get_log_pkt_per_rsrv (mdev, params)); +MLX5_SET (wq, wq, log_headers_entry_size, +mlx5e_shampo_get_log_hd_entry_size (mdev, params)); + lro_timeout = +mlx5e_choose_lro_timeout (mdev, +MLX5E_DEFAULT_SHAMPO_TIMEOUT ); +MLX5_SET (rqc, rqc, reservation_timeout, lro_timeout); +MLX5_SET (rqc, rqc, shampo_match_criteria_type, +MLX5_RQC_SHAMPO_MATCH_CRITERIA_TYPE_EXTENDED ); +MLX5_SET (rqc, rqc, shampo_no_match_alignment_granularity, +MLX5_RQC_SHAMPO_NO_MATCH_ALIGNMENT_GRANULARITY_STRIDE );break ; }default :/* MLX5_WQ_TYPE_CYCLIC */
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0 -4
@@ -4040,10 +4040,6 @@ static int set_feature_hw_gro(struct net_device *netdev, bool enable)if (enable) { new_params.packet_merge.type =MLX5E_PACKET_MERGE_SHAMPO ; - new_params.packet_merge.shampo.match_criteria_type = -MLX5_RQC_SHAMPO_MATCH_CRITERIA_TYPE_EXTENDED ; - new_params.packet_merge.shampo.alignment_granularity = -MLX5_RQC_SHAMPO_NO_MATCH_ALIGNMENT_GRANULARITY_STRIDE ; }else if (new_params.packet_merge.type ==MLX5E_PACKET_MERGE_SHAMPO ) { new_params.packet_merge.type =MLX5E_PACKET_MERGE_NONE ; }else {
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 6/12 net/mlx5e: SHAMPO: Improve hw gro capability checking
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
Add missing HW capabilities, declare the feature in netdev->vlan_features, similar to other features in mlx5e_build_nic_netdev. No functional change here as all by default disabled features are explicitly disabled at the bottom of the function.
--- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +7 -6
@@ -78,7 +78,8 @@static bool mlx5e_hw_gro_supported (struct mlx5_core_dev *mdev) { -if (!MLX5_CAP_GEN (mdev, shampo)) +if (!MLX5_CAP_GEN (mdev, shampo) || + !MLX5_CAP_SHAMPO (mdev, shampo_header_split_data_merge))return false;/* Our HW-GRO implementation relies on "KSM Mkey" for
@@ -5499,17 +5500,17 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) MLX5E_MPWRQ_UMR_MODE_ALIGNED)) netdev->vlan_features |= NETIF_F_LRO; +if (mlx5e_hw_gro_supported (mdev) && +mlx5e_check_fragmented_striding_rq_cap (mdev,PAGE_SHIFT , +MLX5E_MPWRQ_UMR_MODE_ALIGNED )) + netdev->vlan_features |=NETIF_F_GRO_HW ; + netdev->hw_features = netdev->vlan_features; netdev->hw_features |=NETIF_F_HW_VLAN_CTAG_TX ; netdev->hw_features |=NETIF_F_HW_VLAN_CTAG_RX ; netdev->hw_features |=NETIF_F_HW_VLAN_CTAG_FILTER ; netdev->hw_features |=NETIF_F_HW_VLAN_STAG_TX ; -if (mlx5e_hw_gro_supported (mdev) && -mlx5e_check_fragmented_striding_rq_cap (mdev,PAGE_SHIFT , -MLX5E_MPWRQ_UMR_MODE_ALIGNED )) - netdev->hw_features |=NETIF_F_GRO_HW ; -if (mlx5e_tunnel_any_tx_proto_supported (mdev)) { netdev->hw_enc_features |=NETIF_F_HW_CSUM ; netdev->hw_enc_features |=NETIF_F_TSO ;
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 7/12 net/mlx5e: SHAMPO: Separate pool for headers
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
Allow allocating a separate page pool for headers when SHAMPO is on. This will be useful for adding support to zc page pool, which has to be different from the headers page pool.
For now, the pools are the same.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 ++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 43 ++++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 41 ++++++++++-------- 3 files changed, 69 insertions(+), 19 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +4 -0
@@ -716,7 +716,11 @@ struct mlx5e_rq {struct bpf_prog __rcu *xdp_prog;struct mlx5e_xdpsq *xdpsq;DECLARE_BITMAP (flags,8 ); + +/* page pools */ struct page_pool *page_pool; +struct page_pool *hd_page_pool; +struct mlx5e_xdp_buff mxbuf;/* AF_XDP zero-copy */
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +41 -2
85 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +24 -17
143 lines not on this page · raw message
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 8/12 net/mlx5e: Convert over to netmem
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
mlx5e_page_frag holds the physical page itself, to naturally support zc page pools, remove physical page reference from mlx5 and replace it with netmem_ref, to avoid internal handling in mlx5 for net_iov backed pages.
SHAMPO can issue packets that are not split into header and data. These packets will be dropped if the data part resides in a net_iov as the driver can't read into this area.
No performance degradation observed.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 105 +++++++++++------- 2 files changed, 63 insertions(+), 44 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -1
@@ -553,7 +553,7 @@ struct mlx5e_icosq { }____cacheline_aligned_in_smp ;struct mlx5e_frag_page { -struct page *page; +netmem_ref netmem;u16 frags; };
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +62 -43
255 lines not on this page · raw message
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 9/12 net/mlx5e: Add support for UNREADABLE netmem page pools
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
On netdev_rx_queue_restart, a special type of page pool maybe expected.
In this patch declare support for UNREADABLE netmem iov pages in the pool params only when header data split shampo RQ mode is enabled, also set the queue index in the page pool params struct.
Shampo mode requirement: Without header split rx needs to peek at the data, we can't do UNREADABLE_NETMEM.
The patch also enables the use of a separate page pool for headers when a memory provider is installed for the queue, otherwise the same common page pool continues to be used.
--- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +8 -1
@@ -749,7 +749,9 @@ static void mlx5e_rq_shampo_hd_info_free(struct mlx5e_rq *rq)static bool mlx5_rq_needs_separate_hd_pool (struct mlx5e_rq *rq) { -return false; +struct netdev_rx_queue *rxq =__netif_get_rx_queue (rq->netdev, rq->ix); + +return !!rxq->mp_params.mp_ops; } static int mlx5_rq_shampo_alloc(struct mlx5_core_dev *mdev,
@@ -964,6 +966,11 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params, pp_params.netdev = rq->netdev; pp_params.dma_dir = rq->buff.map_dir; pp_params.max_len =PAGE_SIZE ; + pp_params.queue_idx = rq->ix; + +/* Shampo header data split allow for unreadable netmem */ +if (test_bit (MLX5E_RQ_STATE_SHAMPO , &rq->state)) + pp_params.flags |=PP_FLAG_ALLOW_UNREADABLE_NETMEM ;/* page_pool can be used even when there is no rq->xdp_prog, * given page_pool does not handle DMA mapping there is no
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 10/12 net/mlx5e: Implement queue mgmt ops and single channel swap
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
The bulk of the work is done in mlx5e_queue_mem_alloc, where we allocate and create the new channel resources, similar to mlx5e_safe_switch_params, but here we do it for a single channel using existing params, sort of a clone channel.
To swap the old channel with the new one, we deactivate and close the old channel then replace it with the new one, since the swap procedure doesn't fail in mlx5, we do it all in one place (mlx5e_queue_start).
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +98 -0
@@ -5494,6 +5494,103 @@ static const struct netdev_stat_ops mlx5e_stat_ops = { .get_base_stats = mlx5e_get_base_stats, }; +struct mlx5_qmgmt_data { +struct mlx5e_channel *c; +struct mlx5e_channel_param cparam; +}; + +static int mlx5e_queue_mem_alloc (struct net_device *dev,void *newq, +int queue_index) +{ +struct mlx5_qmgmt_data *new = (struct mlx5_qmgmt_data *)newq; +struct mlx5e_priv *priv =netdev_priv (dev); +struct mlx5e_channels *chs = &priv->channels; +struct mlx5e_params params = chs->params; +struct mlx5_core_dev *mdev; +int err; + +mutex_lock (&priv->state_lock); +if (!test_bit (MLX5E_STATE_OPENED , &priv->state)) { + err = -ENODEV ; + goto unlock; + } + +if (queue_index >= chs->num) { + err = -ERANGE ; + goto unlock; + } + +if (MLX5E_GET_PFLAG (&chs->params,MLX5E_PFLAG_TX_PORT_TS ) || + chs->params.ptp_rx || + chs->params.xdp_prog || + priv->htb) { +netdev_err (priv->netdev, +"Cloning channels with Port/rx PTP, XDP or HTB is not supported\n" ); + err = -EOPNOTSUPP ; + goto unlock; + } + + mdev =mlx5_sd_ch_ix_get_dev (priv->mdev, queue_index); + err =mlx5e_build_channel_param (mdev, ¶ms, &new->cparam); +if (err) + goto unlock; + + err =mlx5e_open_channel (priv, queue_index, ¶ms,NULL , &new->c); +unlock: +mutex_unlock (&priv->state_lock); +return err; +} + +static void mlx5e_queue_mem_free (struct net_device *dev,void *mem) +{ +struct mlx5_qmgmt_data *data = (struct mlx5_qmgmt_data *)mem; + +/* not supposed to happen since mlx5e_queue_start never fails +* but this is how this should be implemented just in case +*/ +if (data->c) +mlx5e_close_channel (data->c); +} + +static int mlx5e_queue_stop (struct net_device *dev,void *oldq,int queue_index) +{ +/* In mlx5 a txq cannot be simply stopped in isolation, only restarted. +* mlx5e_queue_start does not fail, we stop the old queue there. +* TODO: Improve this. +*/ +return 0 ; +} + +static int mlx5e_queue_start (struct net_device *dev,void *newq, +int queue_index) +{ +struct mlx5_qmgmt_data *new = (struct mlx5_qmgmt_data *)newq; +struct mlx5e_priv *priv =netdev_priv (dev); +struct mlx5e_channel *old; + +mutex_lock (&priv->state_lock); + +/* stop and close the old */ + old = priv->channels.c[queue_index]; +mlx5e_deactivate_priv_channels (priv); +/* close old before activating new, to avoid napi conflict */ +mlx5e_close_channel (old); + +/* start the new */ + priv->channels.c[queue_index] = new->c; +mlx5e_activate_priv_channels (priv); +mutex_unlock (&priv->state_lock); +return 0 ; +} + +static const struct netdev_queue_mgmt_ops mlx5e_queue_mgmt_ops = { + .ndo_queue_mem_size =sizeof (struct mlx5_qmgmt_data ), + .ndo_queue_mem_alloc = mlx5e_queue_mem_alloc, + .ndo_queue_mem_free = mlx5e_queue_mem_free, + .ndo_queue_start = mlx5e_queue_start, + .ndo_queue_stop = mlx5e_queue_stop, +}; +static void mlx5e_build_nic_netdev (struct net_device *netdev) {struct mlx5e_priv *priv =netdev_priv (netdev);
@@ -5504,6 +5601,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)SET_NETDEV_DEV (netdev, mdev->device); netdev->netdev_ops = &mlx5e_netdev_ops; + netdev->queue_mgmt_ops = &mlx5e_queue_mgmt_ops; netdev->xdp_metadata_ops = &mlx5e_xdp_metadata_ops; netdev->xsk_tx_metadata_ops = &mlx5e_xsk_tx_metadata_ops; netdev->request_ops_lock = true;
Signature · 1 line
2.34.1
Re: [PATCH net-next v6 10/12] net/mlx5e: Implement queue mgmt ops and single channel swap
Quoted 11 lines · Mark Bloch
From: Saeed Mahameed <[email protected]>
The bulk of the work is done in
mlx5e_queue_mem_alloc, where we allocate and create the new channel resources, similar tomlx5e_safe_switch_params, but here we do it for a single channel using existing params, sort of a clone channel.
To swap the old channel with the new one, we deactivate and close the old channel then replace it with the new one, since the swap procedure doesn't fail in mlx5, we do it all in one place (mlx5e_queue_start).
Quoted 4 lines
Signed-off-by Saeed Mahameed <[email protected]>Reviewed-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Signed-off-by Mark Bloch <[email protected]>
Quoted 35 lines
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/en_main.c +23
-0@@ -5494,6 +5494,103 @@ static const struct netdev_stat_ops mlx5e_stat_ops = { .get_base_stats = mlx5e_get_base_stats, }; +struct mlx5_qmgmt_data { +struct mlx5e_channel *c; +struct mlx5e_channel_param cparam; +}; + +static int mlx5e_queue_mem_alloc (struct net_device *dev,void *newq, +int queue_index) +{ +struct mlx5_qmgmt_data *new = (struct mlx5_qmgmt_data *)newq; +struct mlx5e_priv *priv =netdev_priv (dev); +struct mlx5e_channels *chs = &priv->channels; +struct mlx5e_params params = chs->params; +struct mlx5_core_dev *mdev; +int err; + +mutex_lock (&priv->state_lock); +if (!test_bit (MLX5E_STATE_OPENED , &priv->state)) { + err = -ENODEV ; + goto unlock; + } + +if (queue_index >= chs->num) { + err = -ERANGE ;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c static const struct netdev_stat_ops mlx5e_stat_ops = { · line 5529
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10
-0+ goto unlock; + } + +if (MLX5E_GET_PFLAG (&chs->params,MLX5E_PFLAG_TX_PORT_TS ) || + chs->params.ptp_rx || + chs->params.xdp_prog || + priv->htb) { +netdev_err (priv->netdev, +"Cloning channels with Port/rx PTP, XDP or HTB is not supported\n" ); + err = -EOPNOTSUPP ;
I would have used a different error code here as EOFNOTSUPP usually means the driver doesn't support queue API at all.
Signature · 2 lines
Thanks, Mina
Re: [PATCH net-next v6 10/12] net/mlx5e: Implement queue mgmt ops and single channel swap
In reply to Mark BlochQuoted 28 lines · Mark Bloch
From: Saeed Mahameed <[email protected]>
The bulk of the work is done in
mlx5e_queue_mem_alloc, where we allocate and create the new channel resources, similar tomlx5e_safe_switch_params, but here we do it for a single channel using existing params, sort of a clone channel.
To swap the old channel with the new one, we deactivate and close the old channel then replace it with the new one, since the swap procedure doesn't fail in mlx5, we do it all in one place (mlx5e_queue_start).Signed-off-by Saeed Mahameed <[email protected]>Reviewed-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Signed-off-by Mark Bloch <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2
-0@@ -5494,6 +5494,103 @@ static const struct netdev_stat_ops mlx5e_stat_ops = { .get_base_stats = mlx5e_get_base_stats, }; +struct mlx5_qmgmt_data { +struct mlx5e_channel *c;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c static const struct netdev_stat_ops mlx5e_stat_ops = { · line 5508
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10
-0+struct mlx5e_channel_param cparam; +}; + +static int mlx5e_queue_mem_alloc (struct net_device *dev,void *newq, +int queue_index) +{ +struct mlx5_qmgmt_data *new = (struct mlx5_qmgmt_data *)newq; +struct mlx5e_priv *priv =netdev_priv (dev); +struct mlx5e_channels *chs = &priv->channels; +struct mlx5e_params params = chs->params;
RCT (Reverse Christmas Tree) ?
Yanjun.Zhu
Quoted 96 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +86
-0+struct mlx5_core_dev *mdev; +int err; + +mutex_lock (&priv->state_lock); +if (!test_bit (MLX5E_STATE_OPENED , &priv->state)) { + err = -ENODEV ; + goto unlock; + } + +if (queue_index >= chs->num) { + err = -ERANGE ; + goto unlock; + } + +if (MLX5E_GET_PFLAG (&chs->params,MLX5E_PFLAG_TX_PORT_TS ) || + chs->params.ptp_rx || + chs->params.xdp_prog || + priv->htb) { +netdev_err (priv->netdev, +"Cloning channels with Port/rx PTP, XDP or HTB is not supported\n" ); + err = -EOPNOTSUPP ; + goto unlock; + } + + mdev =mlx5_sd_ch_ix_get_dev (priv->mdev, queue_index); + err =mlx5e_build_channel_param (mdev, ¶ms, &new->cparam); +if (err) + goto unlock; + + err =mlx5e_open_channel (priv, queue_index, ¶ms,NULL , &new->c); +unlock: +mutex_unlock (&priv->state_lock); +return err; +} + +static void mlx5e_queue_mem_free (struct net_device *dev,void *mem) +{ +struct mlx5_qmgmt_data *data = (struct mlx5_qmgmt_data *)mem; + +/* not supposed to happen since mlx5e_queue_start never fails +* but this is how this should be implemented just in case +*/ +if (data->c) +mlx5e_close_channel (data->c); +} + +static int mlx5e_queue_stop (struct net_device *dev,void *oldq,int queue_index) +{ +/* In mlx5 a txq cannot be simply stopped in isolation, only restarted. +* mlx5e_queue_start does not fail, we stop the old queue there. +* TODO: Improve this. +*/ +return 0 ; +} + +static int mlx5e_queue_start (struct net_device *dev,void *newq, +int queue_index) +{ +struct mlx5_qmgmt_data *new = (struct mlx5_qmgmt_data *)newq; +struct mlx5e_priv *priv =netdev_priv (dev); +struct mlx5e_channel *old; + +mutex_lock (&priv->state_lock); + +/* stop and close the old */ + old = priv->channels.c[queue_index]; +mlx5e_deactivate_priv_channels (priv); +/* close old before activating new, to avoid napi conflict */ +mlx5e_close_channel (old); + +/* start the new */ + priv->channels.c[queue_index] = new->c; +mlx5e_activate_priv_channels (priv); +mutex_unlock (&priv->state_lock); +return 0 ; +} + +static const struct netdev_queue_mgmt_ops mlx5e_queue_mgmt_ops = { + .ndo_queue_mem_size =sizeof (struct mlx5_qmgmt_data ), + .ndo_queue_mem_alloc = mlx5e_queue_mem_alloc, + .ndo_queue_mem_free = mlx5e_queue_mem_free, + .ndo_queue_start = mlx5e_queue_start, + .ndo_queue_stop = mlx5e_queue_stop, +}; +static void mlx5e_build_nic_netdev (struct net_device *netdev) {struct mlx5e_priv *priv =netdev_priv (netdev);@@ -5504,6 +5601,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)SET_NETDEV_DEV (netdev, mdev->device); netdev->netdev_ops = &mlx5e_netdev_ops; + netdev->queue_mgmt_ops = &mlx5e_queue_mgmt_ops; netdev->xdp_metadata_ops = &mlx5e_xdp_metadata_ops; netdev->xsk_tx_metadata_ops = &mlx5e_xsk_tx_metadata_ops; netdev->request_ops_lock = true;
Mark Bloch
maintainerPATCH v6 11/12 net/mlx5e: Support ethtool tcp-data-split settings
In reply to Mark BlochFrom: Saeed Mahameed <[email protected]>
In mlx5, tcp header-data split requires HW GRO to be on.
Enabling it fails when HW GRO is off.mlx5e_fix_features now keeps HW GRO on when tcp data split is enabled. Finally, when tcp data split is disabled, features are updated to maybe remove the forced HW GRO.
--- .../ethernet/mellanox/mlx5/core/en_ethtool.c | 33 ++++++++++++++++--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 8 +++++ 2 files changed, 36 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +28 -5
@@ -32,6 +32,7 @@#include <linux/dim.h> #include <linux/ethtool_netlink.h> +#include <net/netdev_queues.h> #include "en.h" #include "en/channels.h"
@@ -365,11 +366,6 @@ void mlx5e_ethtool_get_ringparam(struct mlx5e_priv *priv, param->tx_max_pending =1 <<MLX5E_PARAMS_MAXIMUM_LOG_SQ_SIZE ; param->rx_pending =1 << priv->channels.params.log_rq_mtu_frames; param->tx_pending =1 << priv->channels.params.log_sq_size; - - kernel_param->tcp_data_split = - (priv->channels.params.packet_merge.type ==MLX5E_PACKET_MERGE_SHAMPO ) ? -ETHTOOL_TCP_DATA_SPLIT_ENABLED : -ETHTOOL_TCP_DATA_SPLIT_DISABLED ; } static void mlx5e_get_ringparam(struct net_device *dev,
@@ -382,6 +378,27 @@ static void mlx5e_get_ringparam(struct net_device *dev,mlx5e_ethtool_get_ringparam (priv, param, kernel_param); } +static bool mlx5e_ethtool_set_tcp_data_split (struct mlx5e_priv *priv, +u8 tcp_data_split, +struct netlink_ext_ack *extack) +{ +struct net_device *dev = priv->netdev; + +if (tcp_data_split ==ETHTOOL_TCP_DATA_SPLIT_ENABLED && + !(dev->features &NETIF_F_GRO_HW )) { +NL_SET_ERR_MSG_MOD (extack, +"TCP-data-split is not supported when GRO HW is disabled" ); +return false; + } + +/* Might need to disable HW-GRO if it was kept on due to hds. */ +if (tcp_data_split ==ETHTOOL_TCP_DATA_SPLIT_DISABLED && + dev->cfg->hds_config ==ETHTOOL_TCP_DATA_SPLIT_ENABLED ) +netdev_update_features (priv->netdev); + +return true; +} +int mlx5e_ethtool_set_ringparam (struct mlx5e_priv *priv,struct ethtool_ringparam *param,struct netlink_ext_ack *extack)
@@ -440,6 +457,11 @@ static int mlx5e_set_ringparam(struct net_device *dev, {struct mlx5e_priv *priv =netdev_priv (dev); +if (!mlx5e_ethtool_set_tcp_data_split (priv, + kernel_param->tcp_data_split, + extack)) +return -EINVAL ; +return mlx5e_ethtool_set_ringparam (priv, param, extack); }
@@ -2623,6 +2645,7 @@ const struct ethtool_ops mlx5e_ethtool_ops = {ETHTOOL_COALESCE_USE_ADAPTIVE | ETHTOOL_COALESCE_USE_CQE, .supported_input_xfrm = RXH_XFRM_SYM_OR_XOR, + .supported_ring_params = ETHTOOL_RING_USE_TCP_DATA_SPLIT, .get_drvinfo = mlx5e_get_drvinfo, .get_link = ethtool_op_get_link, .get_link_ext_state = mlx5e_get_link_ext_state,
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +8 -0
@@ -4413,6 +4413,7 @@ static netdev_features_t mlx5e_fix_uplink_rep_features(struct net_device *netdevstatic netdev_features_t mlx5e_fix_features (struct net_device *netdev,netdev_features_t features) { +struct netdev_config *cfg = netdev->cfg_pending;struct mlx5e_priv *priv =netdev_priv (netdev);struct mlx5e_vlan_table *vlan;struct mlx5e_params *params;
@@ -4479,6 +4480,13 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev, } } +/* The header-data split ring param requires HW GRO to stay enabled. */ +if (cfg && cfg->hds_config ==ETHTOOL_TCP_DATA_SPLIT_ENABLED && + !(features &NETIF_F_GRO_HW )) { +netdev_warn (netdev,"Keeping HW-GRO enabled, TCP header-data split depends on it\n" ); + features |=NETIF_F_GRO_HW ; + } +if (mlx5e_is_uplink_rep (priv)) { features =mlx5e_fix_uplink_rep_features (netdev, features); netdev->netns_immutable = true;
Signature · 1 line
2.34.1
Mark Bloch
maintainerPATCH v6 12/12 net/mlx5e: Add TX support for netmems
In reply to Mark BlochFrom: Dragos Tatulea <[email protected]>
Declare netmem TX support in netdev.
As required, use the netmem aware dma unmapping APIs for unmapping netmems in tx completion path.
--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2 -1
@@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)dma_unmap_single (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ;case MLX5E_DMA_MAP_PAGE : -dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE ); +netmem_dma_unmap_page_attrs (pdev, dma->addr, dma->size, +DMA_TO_DEVICE ,0 );break ;default :WARN_ONCE (true,"mlx5e_tx_dma_unmap unknown DMA type!\n" );
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -0
@@ -5741,6 +5741,8 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) netdev->priv_flags |=IFF_UNICAST_FLT ; + netdev->netmem_tx = true; +netif_set_tso_max_size (netdev,GSO_MAX_SIZE );mlx5e_set_xdp_feature (netdev);mlx5e_set_netdev_dev_addr (netdev);
Signature · 1 line
2.34.1
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
Quoted 21 lines · Mark Bloch
From: Dragos Tatulea <[email protected]>
Declare netmem TX support in netdev.
As required, use the netmem aware dma unmapping APIs for unmapping netmems in tx completion path.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Reviewed-by Mina Almasry <[email protected]>Signed-off-by Mark Bloch <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +0
-0
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma) · line 366
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2
-1@@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)dma_unmap_single (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ;case MLX5E_DMA_MAP_PAGE : -dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE ); +netmem_dma_unmap_page_attrs (pdev, dma->addr, dma->size, +DMA_TO_DEVICE ,0 );
For this to work, the dma->addr needs to be 0, so the callers of the dma_map() need to be adjusted as well, or am I missing something? There is netmem_dma_unmap_addr_set to handle that, but I don't see anybody calling it. Do we need to add the following (untested)?
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3 -1
@@ -210,7 +210,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,if (unlikely (dma_mapping_error (sq->pdev, dma_addr))) goto dma_unmap_wqe_err; - dseg->addr =cpu_to_be64 (dma_addr); + dseg->addr =0 ; +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dseg->addr =cpu_to_be64 (dma_addr); dseg->lkey = sq->mkey_be; dseg->byte_count =cpu_to_be32 (fsz);
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
Quoted 17 lines · Stanislav Fomichev
Quoted 16 lines · Mark Bloch
From: Dragos Tatulea <[email protected]>
Declare netmem TX support in netdev.
As required, use the netmem aware dma unmapping APIs for unmapping netmems in tx completion path.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Reviewed-by Mina Almasry <[email protected]>Signed-off-by Mark Bloch <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)
Quoted 17 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma) · line 366
Quoted 11 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2
-1@@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)dma_unmap_single (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ;case MLX5E_DMA_MAP_PAGE : -dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE ); +netmem_dma_unmap_page_attrs (pdev, dma->addr, dma->size, +DMA_TO_DEVICE ,0 );For this to work, the
dma->addrneeds to be 0, so the callers of thedma_map()need to be adjusted as well, or am I missing something? There isnetmem_dma_unmap_addr_setto handle that, but I don't see anybody calling it. Do we need to add the following (untested)?
Hmmmm... yes. I figured that skb_frag_dma_map() would do the work but I was wrong, it is not enough.
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, · line 215
Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3
-1@@ -210,7 +210,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,if (unlikely (dma_mapping_error (sq->pdev, dma_addr))) goto dma_unmap_wqe_err; - dseg->addr =cpu_to_be64 (dma_addr); + dseg->addr =0 ; +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dseg->addr =cpu_to_be64 (dma_addr);
AFAIU we still want to pass the computed dma_address to the data segment to the HW. We only need to make sure in mlx5e_dma_push() to set dma_addr to 0, to avoid calling netmem_dma_unmap_page_attrs() with dma->addr 0. Like in the snippet below. Do you agree?
We will send a fix patch once the above question is answered. Also, is there a way to test this with more confidence? The ncdevmem tx test passed just fine.
Thanks,
Dragos
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3 -0
@@ -214,6 +214,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, dseg->lkey = sq->mkey_be; dseg->byte_count =cpu_to_be32 (fsz); +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dma_addr =0 ; +mlx5e_dma_push (sq, dma_addr, fsz,MLX5E_DMA_MAP_PAGE ); num_dma++; dseg++;
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
Quoted 47 lines · Dragos Tatulea
Quoted 17 lines · Stanislav Fomichev
Quoted 16 lines · Mark Bloch
From: Dragos Tatulea <[email protected]>
Declare netmem TX support in netdev.
As required, use the netmem aware dma unmapping APIs for unmapping netmems in tx completion path.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Reviewed-by Mina Almasry <[email protected]>Signed-off-by Mark Bloch <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)Quoted 17 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma) · line 366
Quoted 11 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2
-1@@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)dma_unmap_single (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ;case MLX5E_DMA_MAP_PAGE : -dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE ); +netmem_dma_unmap_page_attrs (pdev, dma->addr, dma->size, +DMA_TO_DEVICE ,0 );For this to work, the
dma->addrneeds to be 0, so the callers of thedma_map()need to be adjusted as well, or am I missing something? There isnetmem_dma_unmap_addr_setto handle that, but I don't see anybody calling it. Do we need to add the following (untested)?Hmmmm... yes. I figured that
skb_frag_dma_map()would do the work but I was wrong, it is not enough.Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +0
-0@@ -210,7 +210,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,if (unlikely (dma_mapping_error (sq->pdev, dma_addr))) goto dma_unmap_wqe_err;
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, · line 215
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3
-1- dseg->addr =cpu_to_be64 (dma_addr); + dseg->addr =0 ; +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dseg->addr =cpu_to_be64 (dma_addr);AFAIU we still want to pass the computed
dma_addressto the data segment to the HW. We only need to make sure inmlx5e_dma_push()to setdma_addrto 0,
yes
Quoted 3 lines
to avoid calling
netmem_dma_unmap_page_attrs()withdma->addr0. Like in the snippet below. Do you agree?
the opposite. You want netmem_dma_unmap_page_attrs() to be called with dma->addr == 0, so that is will skip the dma unmapping.
Quoted 4 lines
We will send a fix patch once the above question is answered. Also, is there a way to test this with more confidence? The ncdevmem tx test passed just fine.
You have to test ncdevmem tx on a platform with iommu enabled. Only in this case the netmem_dma_unmap_page_attrs() may cause a problem, and even then it's not a sure thing. It depends on the type of iommu and type of dmabuf i think.
Quoted 8 lines
Thanks,
Dragosdrivers/net/ethernet/mellanox/mlx5/core/en_tx.c +0
-0
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, · line 221
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3
-0@@ -214,6 +214,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, dseg->lkey = sq->mkey_be; dseg->byte_count =cpu_to_be32 (fsz); +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dma_addr =0 ; +mlx5e_dma_push (sq, dma_addr, fsz,MLX5E_DMA_MAP_PAGE ); num_dma++;
If you can find a way to do this via netmem_dma_unmap_addr_set, I think that would be better, so you're not relying on a manual netmem_is_net_iov check.
The way you'd do that is you'd pass skb_frag_netmem(frag) to mlx5e_dma_push, and then replace the dma->addr = addr with netmem_dma_unmap_addr_set. But up to you.
If you decide to do a net_iov check and dma_addr = 0, add a comment please.
Signature · 2 lines
Thanks, Mina
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
Quoted 56 lines · Mina Almasry
Quoted 47 lines · Dragos Tatulea
Quoted 17 lines · Stanislav Fomichev
Quoted 16 lines · Mark Bloch
From: Dragos Tatulea <[email protected]>
Declare netmem TX support in netdev.
As required, use the netmem aware dma unmapping APIs for unmapping netmems in tx completion path.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Reviewed-by Mina Almasry <[email protected]>Signed-off-by Mark Bloch <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)Quoted 17 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma) · line 366
Quoted 11 lines
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2
-1@@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)dma_unmap_single (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ;case MLX5E_DMA_MAP_PAGE : -dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE ); +netmem_dma_unmap_page_attrs (pdev, dma->addr, dma->size, +DMA_TO_DEVICE ,0 );For this to work, the
dma->addrneeds to be 0, so the callers of thedma_map()need to be adjusted as well, or am I missing something? There isnetmem_dma_unmap_addr_setto handle that, but I don't see anybody calling it. Do we need to add the following (untested)?Hmmmm... yes. I figured that
skb_frag_dma_map()would do the work but I was wrong, it is not enough.Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +0
-0@@ -210,7 +210,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,if (unlikely (dma_mapping_error (sq->pdev, dma_addr))) goto dma_unmap_wqe_err;Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, · line 215
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3
-1- dseg->addr =cpu_to_be64 (dma_addr); + dseg->addr =0 ; +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dseg->addr =cpu_to_be64 (dma_addr);AFAIU we still want to pass the computed
dma_addressto the data segment to the HW. We only need to make sure inmlx5e_dma_push()to setdma_addrto 0,
Quoted 9 lines
yes
Quoted 3 lines
to avoid calling
netmem_dma_unmap_page_attrs()withdma->addr0. Like in the snippet below. Do you agree?the opposite. You want
netmem_dma_unmap_page_attrs()to be called withdma->addr== 0, so that is will skip the dma unmapping.
Yes sorry, that's what I meant to say.
Quoted 10 lines
Quoted 4 lines
We will send a fix patch once the above question is answered. Also, is there a way to test this with more confidence? The ncdevmem tx test passed just fine.
You have to test ncdevmem tx on a platform with iommu enabled. Only in this case the
netmem_dma_unmap_page_attrs()may cause a problem, and even then it's not a sure thing. It depends on the type of iommu and type of dmabuf i think.
Is it worth adding a WARN_ON_ONCE(netmem_is_net_iov())
in netmem_dma_unmap_page_attrs() after addr check to catch these kinds of misuse?
Quoted 19 lines
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +0
-0drivers/net/ethernet/mellanox/mlx5/core/en_tx.c mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, · line 221
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +3
-0@@ -214,6 +214,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, dseg->lkey = sq->mkey_be; dseg->byte_count =cpu_to_be32 (fsz); +if (!netmem_is_net_iov (skb_frag_netmem (frag))) + dma_addr =0 ; +mlx5e_dma_push (sq, dma_addr, fsz,MLX5E_DMA_MAP_PAGE ); num_dma++;If you can find a way to do this via
netmem_dma_unmap_addr_set, I think that would be better, so you're not relying on a manualnetmem_is_net_iovcheck.
Quoted 4 lines
The way you'd do that is you'd pass
skb_frag_netmem(frag)tomlx5e_dma_push, and then replace thedma->addr = addrwithnetmem_dma_unmap_addr_set. But up to you.
Thanks for the suggestion. This would require some additional refactoring. I need to play with this to see if it requires a lot of rewiring or not.
Quoted 2 lines
If you decide to do a
net_iovcheck anddma_addr= 0, add a comment please.
Ack.
Thanks,
Dragos
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
Quoted 9 lines · Dragos Tatulea
Quoted 5 lines
You have to test ncdevmem tx on a platform with iommu enabled. Only in this case the
netmem_dma_unmap_page_attrs()may cause a problem, and even then it's not a sure thing. It depends on the type of iommu and type of dmabuf i think.Is it worth adding a WARN_ON_ONCE(netmem_is_net_iov())
innetmem_dma_unmap_page_attrs()after addr check to catch these kinds of misuse?
I would say it's worth it, but it's the same challenge you point to in your reply: netmem_dma_unmap_page_attrs currently doesn't take in a netmem, and it may be a big refactor not worth it if it's callers also don't have a reference to the netmem readily available to pass it.
Signature · 2 lines
Thanks, Mina
Re: [PATCH net-next v6 12/12] net/mlx5e: Add TX support for netmems
In reply to Dragos TatuleaQuoted 4 lines
Quoted 4 lines
If you can find a way to do this via
netmem_dma_unmap_addr_set, I think that would be better, so you're not relying on a manualnetmem_is_net_iovcheck.
Quoted 8 lines
Quoted 4 lines
The way you'd do that is you'd pass
skb_frag_netmem(frag)tomlx5e_dma_push, and then replace thedma->addr = addrwithnetmem_dma_unmap_addr_set. But up to you.Thanks for the suggestion. This would require some additional refactoring. I need to play with this to see if it requires a lot of rewiring or not.
Got around to this. Found a way to use netmem_dma_unmap_addr_set() with a small refactoring that makes sense. We'll send a patch soon.
Thanks,
Dragos
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Mon, 16 Jun 2025 17:14:29 +0300 you wrote:
This series adds support for zerocopy rx TCP with devmem and
io_uringfor ConnectX7 NICs and above. For performance reasons and simplicity HW-GRO will also be turned on when header-data split mode is on.Performance
===========[...]
Here is the summary with links:
- [net-next,v6,01/12] net: Allow const args for of page_to_netmem()
https://git.kernel.org/netdev/net-next/c/c9e1225352d4 - [net-next,v6,02/12] net: Add skb_can_coalesce for netmem https://git.kernel.org/netdev/net-next/c/1cbb49f85b40 - [net-next,v6,03/12] page_pool: Add page_pool_dev_alloc_netmems helper https://git.kernel.org/netdev/net-next/c/a202f24b0858 - [net-next,v6,04/12] net/mlx5e: SHAMPO: Reorganize mlx5_rq_shampo_alloc https://git.kernel.org/netdev/net-next/c/af4312c4c9c1 - [net-next,v6,05/12] net/mlx5e: SHAMPO: Remove redundant params https://git.kernel.org/netdev/net-next/c/16142defd304 - [net-next,v6,06/12] net/mlx5e: SHAMPO: Improve hw gro capability checking https://git.kernel.org/netdev/net-next/c/d2760abdedde - [net-next,v6,07/12] net/mlx5e: SHAMPO: Separate pool for headers https://git.kernel.org/netdev/net-next/c/e225d9bd93ed - [net-next,v6,08/12] net/mlx5e: Convert over to netmem https://git.kernel.org/netdev/net-next/c/d1668f119943 - [net-next,v6,09/12] net/mlx5e: Add support for UNREADABLE netmem page pools https://git.kernel.org/netdev/net-next/c/db3010bb5a01 - [net-next,v6,10/12] net/mlx5e: Implement queue mgmt ops and single channel swap https://git.kernel.org/netdev/net-next/c/b2588ea40ec9 - [net-next,v6,11/12] net/mlx5e: Support ethtool tcp-data-split settings https://git.kernel.org/netdev/net-next/c/46bcce5dfd33 - [net-next,v6,12/12] net/mlx5e: Add TX support for netmems https://git.kernel.org/netdev/net-next/c/5a842c288cfa
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html