mlx5 updates 2022-03-18
Subsystems (5)
From: Saeed Mahameed <[email protected]>
Hi Dave, Hi Jakub,
This series adds XDP multi-buffer support to mlx5. For more information please see tag log below.
Please pull and let me know if there is any problem.
Thanks,
Saeed.
The following changes since commit e89600ebeeb14d18c0b062837a84196f72542830:
af_vsock: SOCK_SEQPACKET broken buffer test (2022-03-18 15:13:19 +0000)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2022-03-18
for you to fetch changes up to 5dc2b581cd2cf518d28d0c703478a0c1fd54436c:
net/mlx5e: HTB, remove unused function declaration (2022-03-18 13:51:16 -0700)
----------------------------------------------------------------
mlx5-updates-2022-03-18
1) XDP multi buffer support
This series enables XDP on non-linear legacy RQ in multi buffer mode.
When XDP is enabled, fragmentation scheme on non-linear legacy RQ is adjusted to comply to limitations of XDP multi buffer (fragments of the same size). DMA addresses of fragments are stored in struct page for the completion handler to be able to unmap them. XDP_TX is supported.
XDP_REDIRECT is not yet supported, the XDP core blocks it for multi buffer packets at the moment.
2) Trivial cleanups
----------------------------------------------------------------
Maxim Mikityanskiy (13):
net/mlx5e: Prepare non-linear legacy RQ for XDP multi buffer support net/mlx5e: Use fragments of the same size in non-linear legacy RQ with XDP net/mlx5e: Use page-sized fragments with XDP multi buffer net/mlx5e: Add XDP multi buffer support to the non-linear legacy RQ net/mlx5e: Store DMA address inside struct page
net/mlx5e: Move mlx5e_xdpi_fifo_push out of xmit_xdp_frame net/mlx5e: Remove assignment of inline_hdr.sz on XDP TX net/mlx5e: Don't prefill WQEs in XDP SQ in the multi buffer mode net/mlx5e: Implement sending multi buffer XDP frames net/mlx5e: Unindent the else-block in mlx5e_xmit_xdp_buff net/mlx5e: Support multi buffer XDP_TX
net/mlx5e: Permit XDP with non-linear legacy RQ
net/mlx5e: Remove MLX5E_XDP_TX_DS_COUNT
Saeed Mahameed (1):
net/mlx5e: HTB, remove unused function declaration
Tariq Toukan (1):
net/mlx5e: Statify function mlx5_cmd_trigger_completions
drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 +- .../net/ethernet/mellanox/mlx5/core/en/params.c | 34 ++-- .../net/ethernet/mellanox/mlx5/core/en/params.h | 2 + drivers/net/ethernet/mellanox/mlx5/core/en/qos.h | 1 - drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 +- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 198 ++++++++++++++++----- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h | 7 +- .../net/ethernet/mellanox/mlx5/core/en/xsk/setup.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en/xsk/tx.c | 5 +- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 53 ++++-- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 126 +++++++++---- .../net/ethernet/mellanox/mlx5/core/mlx5_core.h | 1 - 13 files changed, 329 insertions(+), 113 deletions(-)
PATCH 1/15 net/mlx5e: Prepare non-linear legacy RQ for XDP multi buffer support
From: Maxim Mikityanskiy <[email protected]>
mlx5e_skb_from_cqe_nonlinear creates an xdp_buff first, putting the first fragment as the linear part, and the rest of fragments as fragments to struct skb_shared_info in the tailroom. Then it creates an SKB in place, based on the xdp_buff. The XDP program is not called in this commit yet.
This commit contains no functional change, except the SKB is built over the whole frag_stride of the first fragment, instead of the minimal size required (headroom, data and skb_shared_info).
--- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 75 +++++++++++++++---- 1 file changed, 61 insertions(+), 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +61 -14
@@ -1567,45 +1567,92 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, struct mlx5e_wqe_frag_info *wi, u32 cqe_bcnt) {struct mlx5e_rq_frag_info *frag_info = &rq->wqe.info.arr[0 ]; +struct mlx5e_wqe_frag_info *head_wi = wi;u16 rx_headroom = rq->buff.headroom;struct mlx5e_dma_info *di = wi->di; +struct skb_shared_info *sinfo;u32 frag_consumed_bytes; -u32 first_frag_size; +struct xdp_buff xdp;struct sk_buff *skb; +u32 truesize;void *va; va =page_address (di->page) + wi->offset; frag_consumed_bytes =min_t (u32, frag_info->frag_size, cqe_bcnt); - first_frag_size =MLX5_SKB_FRAG_SZ (rx_headroom + frag_consumed_bytes);dma_sync_single_range_for_cpu (rq->pdev, di->addr, wi->offset, - first_frag_size,DMA_FROM_DEVICE ); + rq->buff.frame0_sz,DMA_FROM_DEVICE );net_prefetch (va + rx_headroom); -/* XDP is not supported in this configuration, as incoming packets -* might spread among multiple pages. -*/ - skb =mlx5e_build_linear_skb (rq, va, first_frag_size, rx_headroom, - frag_consumed_bytes,0 ); -if (unlikely (!skb)) -return NULL ; - -page_ref_inc (di->page); +mlx5e_fill_xdp_buff (rq, va, rx_headroom, frag_consumed_bytes, &xdp); + sinfo =xdp_get_shared_info_from_buff (&xdp); + truesize =0 ; cqe_bcnt -= frag_consumed_bytes; frag_info++; wi++;while (cqe_bcnt) { +skb_frag_t *frag; + + di = wi->di; + frag_consumed_bytes =min_t (u32, frag_info->frag_size, cqe_bcnt); -mlx5e_add_skb_frag (rq, skb, wi->di, wi->offset, - frag_consumed_bytes, frag_info->frag_stride); +dma_sync_single_for_cpu (rq->pdev, di->addr + wi->offset, + frag_consumed_bytes,DMA_FROM_DEVICE ); + +if (!xdp_buff_has_frags (&xdp)) { +/* Init on the first fragment to avoid cold cache access +* when possible. +*/ + sinfo->nr_frags =0 ; + sinfo->xdp_frags_size =0 ; +xdp_buff_set_frags_flag (&xdp); + } + + frag = &sinfo->frags[sinfo->nr_frags++]; +__skb_frag_set_page (frag, di->page); +skb_frag_off_set (frag, wi->offset); +skb_frag_size_set (frag, frag_consumed_bytes); + +if (page_is_pfmemalloc (di->page)) +xdp_buff_set_frag_pfmemalloc (&xdp); + + sinfo->xdp_frags_size += frag_consumed_bytes; + truesize += frag_info->frag_stride; + cqe_bcnt -= frag_consumed_bytes; frag_info++; wi++; } + di = head_wi->di; + + skb =mlx5e_build_linear_skb (rq, xdp.data_hard_start, rq->buff.frame0_sz, + xdp.data - xdp.data_hard_start, + xdp.data_end - xdp.data, + xdp.data - xdp.data_meta); +if (unlikely (!skb)) +return NULL ; + +page_ref_inc (di->page); + +if (unlikely (xdp_buff_has_frags (&xdp))) { +int i; + +/* sinfo->nr_frags is reset by build_skb, calculate again. */ +xdp_update_skb_shared_info (skb, wi - head_wi -1 , + sinfo->xdp_frags_size, truesize, +xdp_buff_is_frag_pfmemalloc (&xdp)); + +for (i =0 ; i < sinfo->nr_frags; i++) { +skb_frag_t *frag = &sinfo->frags[i]; + +page_ref_inc (skb_frag_page (frag)); + } + } +return skb; }
Signature · 1 line
2.35.1
Re: [net-next 01/15] net/mlx5e: Prepare non-linear legacy RQ for XDP multi buffer support
This series was applied to netdev/net-next.git (master) by Saeed Mahameed <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by Saeed Mahameed <[email protected]>:
Quoted 9 lines · On Fri, 18 Mar 2022 13:52:34 -0700 you wrote:
From: Maxim Mikityanskiy <[email protected]>
mlx5e_skb_from_cqe_nonlinearcreates anxdp_bufffirst, putting the first fragment as the linear part, and the rest of fragments as fragments to structskb_shared_infoin the tailroom. Then it creates an SKB in place, based on thexdp_buff. The XDP program is not called in this commit yet.[...]
Here is the summary with links:
- [net-next,01/15] net/mlx5e: Prepare non-linear legacy RQ for XDP multi buffer support
https://git.kernel.org/netdev/net-next/c/4e8231f1c22d - [net-next,02/15] net/mlx5e: Use fragments of the same size in non-linear legacy RQ with XDP https://git.kernel.org/netdev/net-next/c/9cb9482ef10e - [net-next,03/15] net/mlx5e: Use page-sized fragments with XDP multi buffer https://git.kernel.org/netdev/net-next/c/d51f4a4cca6f - [net-next,04/15] net/mlx5e: Add XDP multi buffer support to the non-linear legacy RQ https://git.kernel.org/netdev/net-next/c/ea5d49bdae8b - [net-next,05/15] net/mlx5e: Store DMA address inside struct page https://git.kernel.org/netdev/net-next/c/ddc87e7d4775 - [net-next,06/15] net/mlx5e: Move mlx5e_xdpi_fifo_push out of xmit_xdp_frame https://git.kernel.org/netdev/net-next/c/49529a172685 - [net-next,07/15] net/mlx5e: Remove assignment of inline_hdr.sz on XDP TX https://git.kernel.org/netdev/net-next/c/c090451633f8 - [net-next,08/15] net/mlx5e: Don't prefill WQEs in XDP SQ in the multi buffer mode https://git.kernel.org/netdev/net-next/c/9ded70fa1d81 - [net-next,09/15] net/mlx5e: Implement sending multi buffer XDP frames https://git.kernel.org/netdev/net-next/c/39a1665d16a2 - [net-next,10/15] net/mlx5e: Unindent the else-block in mlx5e_xmit_xdp_buff https://git.kernel.org/netdev/net-next/c/fbeed25bcc45 - [net-next,11/15] net/mlx5e: Support multi buffer XDP_TX https://git.kernel.org/netdev/net-next/c/a48ad58cec18 - [net-next,12/15] net/mlx5e: Permit XDP with non-linear legacy RQ https://git.kernel.org/netdev/net-next/c/1b8a10bbfe77 - [net-next,13/15] net/mlx5e: Remove MLX5E_XDP_TX_DS_COUNT https://git.kernel.org/netdev/net-next/c/08c34e95422b - [net-next,14/15] net/mlx5e: Statify function mlx5_cmd_trigger_completions https://git.kernel.org/netdev/net-next/c/60796198b44f - [net-next,15/15] net/mlx5e: HTB, remove unused function declaration https://git.kernel.org/netdev/net-next/c/5dc2b581cd2c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
PATCH 2/15 net/mlx5e: Use fragments of the same size in non-linear legacy RQ with XDP
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
XDP multi buffer implementation in the kernel assumes that all fragments have the same size. bpf_xdp_frags_increase_tail uses this assumption to get the size of the last fragment, and __xdp_build_skb_from_frame uses it to calculate truesize as nr_frags * xdpf->frame_sz.
The current implementation of mlx5e uses fragments of different size in non-linear legacy RQ. Specifically, the last fragment can be larger than the others. It's an optimization for packets smaller than MTU.
This commit adapts mlx5e to the kernel limitations and makes it use fragments of the same size, in order to add support for XDP multi buffer. The change is applied only if XDP is active, otherwise the old optimization still applies.
--- .../ethernet/mellanox/mlx5/core/en/params.c | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +19 -9
@@ -398,8 +398,12 @@ void mlx5e_build_create_cq_param(struct mlx5e_create_cq_param *ccp, struct mlx5e }; } -static int mlx5e_max_nonlinear_mtu (int first_frag_size,int frag_size) +static int mlx5e_max_nonlinear_mtu (int first_frag_size,int frag_size,bool xdp) { +if (xdp) +/* XDP requires all fragments to be of the same size. */ +return first_frag_size + (MLX5E_MAX_RX_FRAGS -1 ) * frag_size; +/* Optimization for small packets: the last fragment is bigger than the others. */ return first_frag_size + (MLX5E_MAX_RX_FRAGS -2 ) * frag_size +PAGE_SIZE ; }
@@ -438,12 +442,14 @@ static int mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, headroom =mlx5e_get_linear_rq_headroom (params, xsk); first_frag_size_max =SKB_WITH_OVERHEAD (frag_size_max - headroom); - max_mtu =mlx5e_max_nonlinear_mtu (first_frag_size_max, frag_size_max); + max_mtu =mlx5e_max_nonlinear_mtu (first_frag_size_max, frag_size_max, + params->xdp_prog);if (byte_count > max_mtu) { frag_size_max =PAGE_SIZE ; first_frag_size_max =SKB_WITH_OVERHEAD (frag_size_max - headroom); - max_mtu =mlx5e_max_nonlinear_mtu (first_frag_size_max, frag_size_max); + max_mtu =mlx5e_max_nonlinear_mtu (first_frag_size_max, frag_size_max, + params->xdp_prog);if (byte_count > max_mtu) {mlx5_core_err (mdev,"MTU %u is too big for non-linear legacy RQ (max %d)\n" , params->sw_mtu, max_mtu);
@@ -463,14 +469,18 @@ static int mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, info->arr[i].frag_size = frag_size; buf_size += frag_size; -if (i ==0 ) { -/* Ensure that headroom and tailroom are included. */ - frag_size += headroom; - frag_size +=SKB_DATA_ALIGN (sizeof (struct skb_shared_info )); +if (params->xdp_prog) { +/* XDP multi buffer expects fragments of the same size. */ + info->arr[i].frag_stride = frag_size_max; + }else { +if (i ==0 ) { +/* Ensure that headroom and tailroom are included. */ + frag_size += headroom; + frag_size +=SKB_DATA_ALIGN (sizeof (struct skb_shared_info )); + } + info->arr[i].frag_stride =roundup_pow_of_two (frag_size); } - info->arr[i].frag_stride =roundup_pow_of_two (frag_size); - i++; } info->num_frags = i;
Signature · 1 line
2.35.1
PATCH 3/15 net/mlx5e: Use page-sized fragments with XDP multi buffer
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
The implementation of XDP in mlx5e assumes that the frame size is equal to the page size. Force this limitation in the non-linear mode for XDP multi buffer.
--- drivers/net/ethernet/mellanox/mlx5/core/en/params.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +1 -1
@@ -444,7 +444,7 @@ static int mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, max_mtu =mlx5e_max_nonlinear_mtu (first_frag_size_max, frag_size_max, params->xdp_prog); -if (byte_count > max_mtu) { +if (byte_count > max_mtu || params->xdp_prog) { frag_size_max =PAGE_SIZE ; first_frag_size_max =SKB_WITH_OVERHEAD (frag_size_max - headroom);
Signature · 1 line
2.35.1
PATCH 4/15 net/mlx5e: Add XDP multi buffer support to the non-linear legacy RQ
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
This commit adds XDP multi buffer support to the RX path in the non-linear legacy RQ mode. mlx5e_xdp_handle is called from mlx5e_skb_from_cqe_nonlinear.
XDP_TX action for fragmented XDP frames is not yet supported and blocked.
--- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 5 +++++ drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 13 +++++++++++++ 2 files changed, 18 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +5 -0
@@ -68,6 +68,11 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,if (unlikely (!xdpf))return false; +if (unlikely (xdp_frame_has_frags (xdpf))) { +xdp_return_frame (xdpf); +return false; + } + xdptxd.data = xdpf->data; xdptxd.len = xdpf->len;
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +13 -0
@@ -1572,6 +1572,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,struct mlx5e_dma_info *di = wi->di;struct skb_shared_info *sinfo;u32 frag_consumed_bytes; +struct bpf_prog *prog;struct xdp_buff xdp;struct sk_buff *skb;u32 truesize;
@@ -1582,6 +1583,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,dma_sync_single_range_for_cpu (rq->pdev, di->addr, wi->offset, rq->buff.frame0_sz,DMA_FROM_DEVICE ); +net_prefetchw (va);/* xdp_frame data area */ net_prefetch (va + rx_headroom);mlx5e_fill_xdp_buff (rq, va, rx_headroom, frag_consumed_bytes, &xdp);
@@ -1629,6 +1631,17 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, di = head_wi->di; + prog =rcu_dereference (rq->xdp_prog); +if (prog &&mlx5e_xdp_handle (rq, di, prog, &xdp)) { +if (test_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags)) { +int i; + +for (i = wi - head_wi; i < rq->wqe.info.num_frags; i++) +mlx5e_put_rx_frag (rq, &head_wi[i], true); + } +return NULL ;/* page/packet was consumed by XDP */ + } + skb = mlx5e_build_linear_skb(rq, xdp.data_hard_start, rq->buff.frame0_sz, xdp.data - xdp.data_hard_start, xdp.data_end - xdp.data,
Signature · 1 line
2.35.1
PATCH 5/15 net/mlx5e: Store DMA address inside struct page
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
Use page_pool_set_dma_addr() to store the DMA address of a page inside struct page, in order to avoid passing struct mlx5e_dma_info to XDP handlers. Previously, struct mlx5e_dma_info was used to pass both the DMA address and the page, and it worked well for the single-fragment case.
When XDP multi buffer is in use, and a fragmented xdp_frame has to be transmitted, the driver needs to know the DMA addresses of fragments, however, the array of fragments in struct skb_shared_info doesn't contain them. In order to pass the DMA addresses, the driver puts them into struct page itself, which is accessible from the array of fragments in struct skb_shared_info. The existing XDP handlers are modified to remove the dependency on struct mlx5e_dma_info.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 +-- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 14 +++---- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 40 ++++++++++--------- 6 files changed, 33 insertions(+), 33 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -1
@@ -515,7 +515,7 @@ struct mlx5e_xdp_info { } frame;struct {struct mlx5e_rq *rq; -struct mlx5e_dma_info di; +struct page *page; } page; }; };
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +2 -4
@@ -44,10 +44,8 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget);int mlx5e_poll_ico_cq (struct mlx5e_cq *cq);/* RX */ -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info); -void mlx5e_page_release_dynamic (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info, -bool recycle); +void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct page *page); +void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct page *page,bool recycle); INDIRECT_CALLABLE_DECLARE(bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq)); INDIRECT_CALLABLE_DECLARE(bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq));int mlx5e_poll_rx_cq (struct mlx5e_cq *cq,int budget);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +7 -7
@@ -57,7 +57,7 @@ int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk)static inline bool mlx5e_xmit_xdp_buff (struct mlx5e_xdpsq *sq,struct mlx5e_rq *rq, -struct mlx5e_dma_info *di,struct xdp_buff *xdp) +struct page *page,struct xdp_buff *xdp) {struct mlx5e_xmit_data xdptxd;struct mlx5e_xdp_info xdpi;
@@ -110,13 +110,13 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, xdpi.mode =MLX5E_XDP_XMIT_MODE_PAGE ; - dma_addr = di->addr + (xdpf->data - (void *)xdpf); + dma_addr =page_pool_get_dma_addr (page) + (xdpf->data - (void *)xdpf);dma_sync_single_for_device (sq->pdev, dma_addr, xdptxd.len,DMA_TO_DEVICE ); xdptxd.dma_addr = dma_addr; xdpi.page.rq = rq; - xdpi.page.di = *di; + xdpi.page.page = page; } return INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,
@@ -124,7 +124,7 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, }/* returns true if packet was consumed by xdp */ -bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct mlx5e_dma_info *di, +bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct page *page,struct bpf_prog *prog,struct xdp_buff *xdp) {u32 act;
@@ -135,7 +135,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,case XDP_PASS :return false;case XDP_TX : -if (unlikely (!mlx5e_xmit_xdp_buff (rq->xdpsq, rq, di, xdp))) +if (unlikely (!mlx5e_xmit_xdp_buff (rq->xdpsq, rq, page, xdp))) goto xdp_abort;__set_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags);/* non-atomic */ return true;
@@ -147,7 +147,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,__set_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags);__set_bit (MLX5E_RQ_FLAG_XDP_REDIRECT , rq->flags);if (xdp->rxq->mem.type !=MEM_TYPE_XSK_BUFF_POOL ) -mlx5e_page_dma_unmap (rq, di); +mlx5e_page_dma_unmap (rq, page); rq->stats->xdp_redirect++;return true;default :
@@ -384,7 +384,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,break ;case MLX5E_XDP_XMIT_MODE_PAGE :/* XDP_TX from the regular RQ */ -mlx5e_page_release_dynamic (xdpi.page.rq, &xdpi.page.di, recycle); +mlx5e_page_release_dynamic (xdpi.page.rq, xdpi.page.page, recycle);break ;case MLX5E_XDP_XMIT_MODE_XSK :/* AF_XDP send */
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +1 -1
@@ -47,7 +47,7 @@struct mlx5e_xsk_param ;int mlx5e_xdp_max_mtu (struct mlx5e_params *params,struct mlx5e_xsk_param *xsk); -bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct mlx5e_dma_info *di, +bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct page *page,struct bpf_prog *prog,struct xdp_buff *xdp);void mlx5e_xdp_mpwqe_complete (struct mlx5e_xdpsq *sq);bool mlx5e_poll_xdpsq_cq (struct mlx5e_cq *cq);
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -1
@@ -780,7 +780,7 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)* entered, and it's safe to call mlx5e_page_release_dynamic * directly. */ -mlx5e_page_release_dynamic (rq, dma_info, false); +mlx5e_page_release_dynamic (rq, dma_info->page, false); }xdp_rxq_info_unreg (&rq->xdp_rxq);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +21 -19
@@ -222,8 +222,7 @@ static inline u32 mlx5e_decompress_cqes_start(struct mlx5e_rq *rq,return mlx5e_decompress_cqes_cont (rq, wq,1 , budget_rem) -1 ; } -static inline bool mlx5e_rx_cache_put (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info) +static inline bool mlx5e_rx_cache_put (struct mlx5e_rq *rq,struct page *page) {struct mlx5e_page_cache *cache = &rq->page_cache;u32 tail_next = (cache->tail +1 ) & (MLX5E_CACHE_SIZE -1 );
@@ -234,12 +233,13 @@ static inline bool mlx5e_rx_cache_put(struct mlx5e_rq *rq,return false; } -if (!dev_page_is_reusable (dma_info->page)) { +if (!dev_page_is_reusable (page)) { stats->cache_waive++;return false; } - cache->page_cache[cache->tail] = *dma_info; + cache->page_cache[cache->tail].page = page; + cache->page_cache[cache->tail].addr =page_pool_get_dma_addr (page); cache->tail = tail_next;return true; }
@@ -287,6 +287,7 @@ static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq, dma_info->page =NULL ;return -ENOMEM ; } +page_pool_set_dma_addr (dma_info->page, dma_info->addr);return 0 ; }
@@ -300,26 +301,27 @@ static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,return mlx5e_page_alloc_pool (rq, dma_info); } -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) +void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct page *page) { -dma_unmap_page_attrs (rq->pdev, dma_info->addr,PAGE_SIZE , rq->buff.map_dir, +dma_addr_t dma_addr =page_pool_get_dma_addr (page); + +dma_unmap_page_attrs (rq->pdev, dma_addr,PAGE_SIZE , rq->buff.map_dir,DMA_ATTR_SKIP_CPU_SYNC ); +page_pool_set_dma_addr (page,0 ); } -void mlx5e_page_release_dynamic (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info, -bool recycle) +void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct page *page,bool recycle) {if (likely (recycle)) { -if (mlx5e_rx_cache_put (rq, dma_info)) +if (mlx5e_rx_cache_put (rq, page))return ; -mlx5e_page_dma_unmap (rq, dma_info); -page_pool_recycle_direct (rq->page_pool, dma_info->page); +mlx5e_page_dma_unmap (rq, page); +page_pool_recycle_direct (rq->page_pool, page); }else { -mlx5e_page_dma_unmap (rq, dma_info); -page_pool_release_page (rq->page_pool, dma_info->page); -put_page (dma_info->page); +mlx5e_page_dma_unmap (rq, page); +page_pool_release_page (rq->page_pool, page); +put_page (page); } }
@@ -334,7 +336,7 @@ static inline void mlx5e_page_release(struct mlx5e_rq *rq,*/ xsk_buff_free (dma_info->xsk); else -mlx5e_page_release_dynamic (rq ,dma_info ,recycle ); + mlx5e_page_release_dynamic(rq, dma_info->page, recycle); } static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq,
@@ -1544,7 +1546,7 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,net_prefetchw (va);/* xdp_frame data area */ mlx5e_fill_xdp_buff (rq, va, rx_headroom, cqe_bcnt, &xdp); -if (mlx5e_xdp_handle (rq, di, prog, &xdp)) +if (mlx5e_xdp_handle (rq, di->page, prog, &xdp))return NULL ;/* page/packet was consumed by XDP */ rx_headroom = xdp.data - xdp.data_hard_start;
@@ -1632,7 +1634,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, di = head_wi->di; prog =rcu_dereference (rq->xdp_prog); -if (prog &&mlx5e_xdp_handle (rq, di, prog, &xdp)) { +if (prog &&mlx5e_xdp_handle (rq, di->page, prog, &xdp)) {if (test_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags)) {int i;
@@ -1934,7 +1936,7 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,net_prefetchw (va);/* xdp_frame data area */ mlx5e_fill_xdp_buff (rq, va, rx_headroom, cqe_bcnt, &xdp); -if (mlx5e_xdp_handle (rq, di, prog, &xdp)) { +if (mlx5e_xdp_handle (rq, di->page, prog, &xdp)) {if (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags))__set_bit (page_idx, wi->xdp_xmit_bitmap);/* non-atomic */ return NULL ;/* page/packet was consumed by XDP */
Signature · 1 line
2.35.1
PATCH 6/15 net/mlx5e: Move mlx5e_xdpi_fifo_push out of xmit_xdp_frame
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
The implementations of xmit_xdp_frame get the xdpi parameter of type struct mlx5e_xdp_info for the sole purpose of calling mlx5e_xdpi_fifo_push() on success.
This commit moves this call outside of xmit_xdp_frame, shifting this responsibility to the caller. It will allow more fine-grained handling of XDP info for cases when an xdp_frame is fragmented.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 - .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 17 ++++++++++------- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 -- .../net/ethernet/mellanox/mlx5/core/en/xsk/tx.c | 4 +++- 4 files changed, 13 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -1
@@ -537,7 +537,6 @@ struct mlx5e_xdpsq;typedef int (*mlx5e_fp_xmit_xdp_frame_check )(struct mlx5e_xdpsq *);typedef bool (*mlx5e_fp_xmit_xdp_frame )(struct mlx5e_xdpsq *,struct mlx5e_xmit_data *, -struct mlx5e_xdp_info *,int );struct mlx5e_xdpsq {
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10 -7
@@ -119,8 +119,12 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, xdpi.page.page = page; } -return INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi,0 ); +if (unlikely (!INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, + mlx5e_xmit_xdp_frame, sq, &xdptxd,0 ))) +return false; + +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); +return true; }/* returns true if packet was consumed by xdp */
@@ -261,7 +265,7 @@ INDIRECT_CALLABLE_SCOPE int mlx5e_xmit_xdp_frame_check_mpwqe(struct mlx5e_xdpsq INDIRECT_CALLABLE_SCOPE boolmlx5e_xmit_xdp_frame_mpwqe (struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct mlx5e_xdp_info *xdpi,int check_result) +int check_result) {struct mlx5e_tx_mpwqe *session = &sq->mpwqe;struct mlx5e_xdpsq_stats *stats = sq->stats;
@@ -289,7 +293,6 @@ mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxif (unlikely (mlx5e_xdp_mpqwe_is_full (session, sq->max_sq_mpw_wqebbs)))mlx5e_xdp_mpwqe_complete (sq); -mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, xdpi); stats->xmit++;return true; }
@@ -308,7 +311,7 @@ INDIRECT_CALLABLE_SCOPE int mlx5e_xmit_xdp_frame_check(struct mlx5e_xdpsq *sq) INDIRECT_CALLABLE_SCOPE boolmlx5e_xmit_xdp_frame (struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct mlx5e_xdp_info *xdpi,int check_result) +int check_result) {struct mlx5_wq_cyc *wq = &sq->wq;u16 pi =mlx5_wq_cyc_ctr2ix (wq, sq->pc);
@@ -358,7 +361,6 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, sq->doorbell_cseg = cseg; -mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, xdpi); stats->xmit++;return true; }
@@ -537,12 +539,13 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, xdpi.frame.dma_addr = xdptxd.dma_addr; ret =INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi,0 ); + mlx5e_xmit_xdp_frame, sq, &xdptxd,0 );if (unlikely (!ret)) {dma_unmap_single (sq->pdev, xdptxd.dma_addr, xdptxd.len,DMA_TO_DEVICE );break ; } +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); nxmit++; }
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +0 -2
@@ -59,11 +59,9 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct mlx5e_xdp_info *xdpi,int check_result)); INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct mlx5e_xdp_info *xdpi,int check_result)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check_mpwqe(struct mlx5e_xdpsq *sq)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check(struct mlx5e_xdpsq *sq));
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +3 -1
@@ -103,12 +103,14 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)xsk_buff_raw_dma_sync_for_device (pool, xdptxd.dma_addr, xdptxd.len); ret =INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd, &xdpi, check_result); + mlx5e_xmit_xdp_frame, sq, &xdptxd, check_result);if (unlikely (!ret)) {if (sq->mpwqe.wqe)mlx5e_xdp_mpwqe_complete (sq);mlx5e_xsk_tx_post_err (sq, &xdpi); + }else { +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); } flush = true;
Signature · 1 line
2.35.1
PATCH 7/15 net/mlx5e: Remove assignment of inline_hdr.sz on XDP TX
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
When MPWQE is disabled, mlx5e_open_xdpsq prefills the common fields of WQEs in the XDP SQ to save time when sending packets. One of such fields is eseg->inline_hdr.sz, which can be either 0 or MLX5E_XDP_MIN_INLINE, depending on the inline mode of the SQ.
The inline mode can't change during the lifetime of the SQ, so setting this field again in mlx5e_xmit_xdp_frame is redundant. Moreover, the xmit function only sets it to MLX5E_XDP_MIN_INLINE, but not to 0 in the other case.
This commit removes the redundant assignment in mlx5e_xmit_xdp_frame.
--- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 1 - 1 file changed, 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +0 -1
@@ -343,7 +343,6 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,/* copy the inline part if required */ if (sq->min_inline_mode !=MLX5_INLINE_MODE_NONE ) {memcpy (eseg->inline_hdr.start, xdptxd->data,sizeof (eseg->inline_hdr.start)); - eseg->inline_hdr.sz =cpu_to_be16 (MLX5E_XDP_MIN_INLINE );memcpy (dseg, xdptxd->data +sizeof (eseg->inline_hdr.start),MLX5E_XDP_MIN_INLINE -sizeof (eseg->inline_hdr.start)); dma_len -=MLX5E_XDP_MIN_INLINE ;
Signature · 1 line
2.35.1
PATCH 8/15 net/mlx5e: Don't prefill WQEs in XDP SQ in the multi buffer mode
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
When MPWQE is disabled, mlx5e_open_xdpsq() prefills the common fields of WQEs in the XDP SQ to save time when sending packets. mlx5e_xmit_xdp_frame() runs on the prefilled fields, however, sending multi buffer XDP frames would require changing some of these fields on a per-packet basis. Besides that, mlx5e_xmit_xdp_frame() will be used as a fallback to send multi buffer XDP frames when MPWQE is enabled (MPWQE can only handle linear packets).
In order to prepare for XDP multi buffer support, this commit introduces a mode for mlx5e_xmit_xdp_frame() that fills all the fields itself.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../ethernet/mellanox/mlx5/core/en/params.c | 4 ++- .../ethernet/mellanox/mlx5/core/en/params.h | 2 ++ .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 32 +++++++++++++++++-- .../mellanox/mlx5/core/en/xsk/setup.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 10 +++++- 6 files changed, 46 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -405,6 +405,7 @@ enum {MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE ,MLX5E_SQ_STATE_PENDING_XSK_TX ,MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC , + MLX5E_SQ_STATE_XDP_MULTIBUF, }; struct mlx5e_tx_mpwqe {
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +3 -1
@@ -843,6 +843,7 @@ static void mlx5e_build_async_icosq_param(struct mlx5_core_dev *mdev,void mlx5e_build_xdpsq_param (struct mlx5_core_dev *mdev,struct mlx5e_params *params, +struct mlx5e_xsk_param *xsk,struct mlx5e_sq_param *param) {void *sqc = param->sqc;
@@ -851,6 +852,7 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev,mlx5e_build_sq_param_common (mdev, param);MLX5_SET (wq, wq, log_wq_sz, params->log_sq_size); param->is_mpw =MLX5E_GET_PFLAG (params,MLX5E_PFLAG_XDP_TX_MPWQE ); + param->is_xdp_mb = !mlx5e_rx_is_linear_skb (params, xsk);mlx5e_build_tx_cq_param (mdev, params, ¶m->cqp); }
@@ -870,7 +872,7 @@ int mlx5e_build_channel_param(struct mlx5_core_dev *mdev, async_icosq_log_wq_sz =mlx5e_build_async_icosq_log_wq_sz (mdev);mlx5e_build_sq_param (mdev, params, &cparam->txq_sq); -mlx5e_build_xdpsq_param (mdev, params, &cparam->xdp_sq); +mlx5e_build_xdpsq_param (mdev, params,NULL , &cparam->xdp_sq);mlx5e_build_icosq_param (mdev, icosq_log_wq_sz, &cparam->icosq);mlx5e_build_async_icosq_param (mdev, async_icosq_log_wq_sz, &cparam->async_icosq);
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +2 -0
@@ -31,6 +31,7 @@ struct mlx5e_sq_param {struct mlx5_wq_param wq;bool is_mpw;bool is_tls; +bool is_xdp_mb;u16 stop_room; };
@@ -155,6 +156,7 @@ void mlx5e_build_tx_cq_param(struct mlx5_core_dev *mdev, struct mlx5e_cq_param *param);void mlx5e_build_xdpsq_param (struct mlx5_core_dev *mdev,struct mlx5e_params *params, +struct mlx5e_xsk_param *xsk,struct mlx5e_sq_param *param); int mlx5e_build_channel_param(struct mlx5_core_dev *mdev, struct mlx5e_params *params,
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +30 -2
@@ -323,6 +323,7 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,dma_addr_t dma_addr = xdptxd->dma_addr;u32 dma_len = xdptxd->len; +u16 ds_cnt, inline_hdr_sz;struct mlx5e_xdpsq_stats *stats = sq->stats;
@@ -338,7 +339,8 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,if (unlikely (check_result <0 ))return false; - cseg->fm_ce_se =0 ; + ds_cnt =MLX5E_TX_WQE_EMPTY_DS_COUNT +1 ; + inline_hdr_sz =0 ;/* copy the inline part if required */ if (sq->min_inline_mode !=MLX5_INLINE_MODE_NONE ) {
@@ -347,7 +349,9 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, MLX5E_XDP_MIN_INLINE - sizeof(eseg->inline_hdr.start)); dma_len -=MLX5E_XDP_MIN_INLINE ; dma_addr +=MLX5E_XDP_MIN_INLINE ; + inline_hdr_sz =MLX5E_XDP_MIN_INLINE ; dseg++; + ds_cnt++; }/* write the dma part */
@@ -356,7 +360,31 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, cseg->opmod_idx_opcode =cpu_to_be32 ((sq->pc <<8 ) |MLX5_OPCODE_SEND ); - sq->pc++; +if (unlikely (test_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state))) { +u8 num_pkts =1 ; +u8 num_wqebbs; + +memset (&cseg->signature,0 ,sizeof (*cseg) - +sizeof (cseg->opmod_idx_opcode) -sizeof (cseg->qpn_ds)); +memset (eseg,0 ,sizeof (*eseg) -sizeof (eseg->trailer)); + + eseg->inline_hdr.sz =cpu_to_be16 (inline_hdr_sz); + dseg->lkey = sq->mkey_be; + + cseg->qpn_ds =cpu_to_be32 ((sq->sqn <<8 ) | ds_cnt); + + num_wqebbs =DIV_ROUND_UP (ds_cnt,MLX5_SEND_WQEBB_NUM_DS ); + sq->db.wqe_info[pi] = (struct mlx5e_xdp_wqe_info ) { + .num_wqebbs = num_wqebbs, + .num_pkts = num_pkts, + }; + + sq->pc += num_wqebbs; + }else { + cseg->fm_ce_se =0 ; + + sq->pc++; + } sq->doorbell_cseg = cseg;
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c +1 -1
@@ -43,7 +43,7 @@ static void mlx5e_build_xsk_cparam(struct mlx5_core_dev *mdev, struct mlx5e_channel_param *cparam) {mlx5e_build_rq_param (mdev, params, xsk, q_counter, &cparam->rq); -mlx5e_build_xdpsq_param (mdev, params, &cparam->xdp_sq); +mlx5e_build_xdpsq_param (mdev, params, xsk, &cparam->xdp_sq); } static int mlx5e_init_xsk_rq(struct mlx5e_channel *c,
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +9 -1
@@ -1666,13 +1666,21 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params, csp.wq_ctrl = &sq->wq_ctrl; csp.min_inline_mode = sq->min_inline_mode;set_bit (MLX5E_SQ_STATE_ENABLED , &sq->state); + +/* Don't enable multi buffer on XDP_REDIRECT SQ, as it's not yet +* supported by upstream, and there is no defined trigger to allow +* transmitting redirected multi-buffer frames. +*/ +if (param->is_xdp_mb && !is_redirect) +set_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state); + err =mlx5e_create_sq_rdy (c->mdev, param, &csp,0 , &sq->sqn);if (err) goto err_free_xdpsq;mlx5e_set_xmit_fp (sq, param->is_mpw); -if (!param->is_mpw) { +if (!param->is_mpw && !test_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state)) {unsigned int ds_cnt =MLX5E_XDP_TX_DS_COUNT ;unsigned int inline_hdr_sz =0 ;int i;
Signature · 1 line
2.35.1
PATCH 9/15 net/mlx5e: Implement sending multi buffer XDP frames
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
xmit_xdp_frame is extended to support sending fragmented XDP frames. The next commit will start using this functionality.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 96 +++++++++++++++---- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 + .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 3 +- 4 files changed, 80 insertions(+), 22 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -538,6 +538,7 @@ struct mlx5e_xdpsq;typedef int (*mlx5e_fp_xmit_xdp_frame_check )(struct mlx5e_xdpsq *);typedef bool (*mlx5e_fp_xmit_xdp_frame )(struct mlx5e_xdpsq *,struct mlx5e_xmit_data *, +struct skb_shared_info *,int );struct mlx5e_xdpsq {
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +75 -21
172 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +2 -0
@@ -59,9 +59,11 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, +struct skb_shared_info *sinfo,int check_result)); INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, +struct skb_shared_info *sinfo,int check_result)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check_mpwqe(struct mlx5e_xdpsq *sq)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check(struct mlx5e_xdpsq *sq));
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +2 -1
@@ -103,7 +103,8 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)xsk_buff_raw_dma_sync_for_device (pool, xdptxd.dma_addr, xdptxd.len); ret =INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd, check_result); + mlx5e_xmit_xdp_frame, sq, &xdptxd,NULL , + check_result);if (unlikely (!ret)) {if (sq->mpwqe.wqe)mlx5e_xdp_mpwqe_complete (sq);
Signature · 1 line
2.35.1
PATCH 10/15 net/mlx5e: Unindent the else-block in mlx5e_xmit_xdp_buff
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
The next commit will add more indentation levels to mlx5e_xmit_xdp_buff. To keep indentation minimal, unindent the else-block of the if-statement by doing an early return.
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 34 +++++++++++-------- 1 file changed, 20 insertions(+), 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +20 -14
@@ -101,24 +101,30 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, xdptxd.dma_addr = dma_addr; xdpi.frame.xdpf = xdpf; xdpi.frame.dma_addr = dma_addr; - }else { -/* Driver assumes that xdp_convert_buff_to_frame returns -* an xdp_frame that points to the same memory region as -* the original xdp_buff. It allows to map the memory only -* once and to use the DMA_BIDIRECTIONAL mode. -*/ - - xdpi.mode =MLX5E_XDP_XMIT_MODE_PAGE ; - dma_addr =page_pool_get_dma_addr (page) + (xdpf->data - (void *)xdpf); -dma_sync_single_for_device (sq->pdev, dma_addr, xdptxd.len, -DMA_TO_DEVICE ); +if (unlikely (!INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, + mlx5e_xmit_xdp_frame, sq, &xdptxd,NULL ,0 ))) +return false; - xdptxd.dma_addr = dma_addr; - xdpi.page.rq = rq; - xdpi.page.page = page; +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); +return true; } +/* Driver assumes that xdp_convert_buff_to_frame returns an xdp_frame +* that points to the same memory region as the original xdp_buff. It +* allows to map the memory only once and to use the DMA_BIDIRECTIONAL +* mode. +*/ + + xdpi.mode =MLX5E_XDP_XMIT_MODE_PAGE ; + + dma_addr =page_pool_get_dma_addr (page) + (xdpf->data - (void *)xdpf); +dma_sync_single_for_device (sq->pdev, dma_addr, xdptxd.len,DMA_TO_DEVICE ); + + xdptxd.dma_addr = dma_addr; + xdpi.page.rq = rq; + xdpi.page.page = page; +if (unlikely (!INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, mlx5e_xmit_xdp_frame, sq, &xdptxd,NULL ,0 )))return false;
Signature · 1 line
2.35.1
PATCH 11/15 net/mlx5e: Support multi buffer XDP_TX
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
This commit enables passing multi buffer XDP frames to the TX handlers on XDP_TX. Fragments are DMA synchronized to the device and queued to the xdpi_fifo for a subsequent unmapping.
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 39 +++++++++++++++---- 1 file changed, 31 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +31 -8
@@ -59,20 +59,17 @@ static inline bool mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, struct page *page, struct xdp_buff *xdp) { + struct skb_shared_info *sinfo = NULL;struct mlx5e_xmit_data xdptxd;struct mlx5e_xdp_info xdpi;struct xdp_frame *xdpf;dma_addr_t dma_addr; +int i; xdpf =xdp_convert_buff_to_frame (xdp);if (unlikely (!xdpf))return false; -if (unlikely (xdp_frame_has_frags (xdpf))) { -xdp_return_frame (xdpf); -return false; - } - xdptxd.data = xdpf->data; xdptxd.len = xdpf->len;
@@ -117,19 +114,45 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,*/ xdpi.mode =MLX5E_XDP_XMIT_MODE_PAGE ; + xdpi.page.rq = rq; dma_addr =page_pool_get_dma_addr (page) + (xdpf->data - (void *)xdpf);dma_sync_single_for_device (sq->pdev, dma_addr, xdptxd.len,DMA_TO_DEVICE ); +if (unlikely (xdp_frame_has_frags (xdpf))) { + sinfo =xdp_get_shared_info_from_frame (xdpf); + +for (i =0 ; i < sinfo->nr_frags; i++) { +skb_frag_t *frag = &sinfo->frags[i]; +dma_addr_t addr; +u32 len; + + addr =page_pool_get_dma_addr (skb_frag_page (frag)) + +skb_frag_off (frag); + len =skb_frag_size (frag); +dma_sync_single_for_device (sq->pdev, addr, len, +DMA_TO_DEVICE ); + } + } + xdptxd.dma_addr = dma_addr; - xdpi.page.rq = rq; - xdpi.page.page = page;if (unlikely (!INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd,NULL ,0 ))) + mlx5e_xmit_xdp_frame, sq, &xdptxd, sinfo,0 )))return false; + xdpi.page.page = page;mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); + +if (unlikely (xdp_frame_has_frags (xdpf))) { +for (i =0 ; i < sinfo->nr_frags; i++) { +skb_frag_t *frag = &sinfo->frags[i]; + + xdpi.page.page =skb_frag_page (frag); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); + } + } +return true; }
Signature · 1 line
2.35.1
PATCH 12/15 net/mlx5e: Permit XDP with non-linear legacy RQ
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
Now that legacy RQ implements XDP in the non-linear mode, stop blocking this configuration. Allow non-linear mode only for programs aware of multi buffer.
XDP performance with linear mode RQ hasn't changed.
Baseline (MTU 1500, TX MPWQE, legacy RQ, single core):
60-byte packets, XDP_DROP: 11.25 Mpps 60-byte packets, XDP_TX: 9.0 Mpps
60-byte packets, XDP_PASS: 668 kpps
Multi buffer (MTU 9000, TX MPWQE, legacy RQ, single core):
60-byte packets, XDP_DROP: 10.1 Mpps
60-byte packets, XDP_TX: 6.6 Mpps
60-byte packets, XDP_PASS: 658 kpps
8900-byte packets, XDP_DROP: 769 kpps (100% of sent packets) 8900-byte packets, XDP_TX: 674 kpps (100% of sent packets) 8900-byte packets, XDP_PASS: 637 kpps
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 39 +++++++++++++------ 1 file changed, 27 insertions(+), 12 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +27 -12
@@ -3953,6 +3953,31 @@ static bool mlx5e_xsk_validate_mtu(struct net_device *netdev,return true; } +static bool mlx5e_params_validate_xdp (struct net_device *netdev,struct mlx5e_params *params) +{ +bool is_linear; + +/* No XSK params: AF_XDP can't be enabled yet at the point of setting +* the XDP program. +*/ + is_linear =mlx5e_rx_is_linear_skb (params,NULL ); + +if (!is_linear && params->rq_wq_type !=MLX5_WQ_TYPE_CYCLIC ) { +netdev_warn (netdev,"XDP is not allowed with striding RQ and MTU(%d) > %d\n" , + params->sw_mtu, +mlx5e_xdp_max_mtu (params,NULL )); +return false; + } +if (!is_linear && !params->xdp_prog->aux->xdp_has_frags) { +netdev_warn (netdev,"MTU(%d) > %d, too big for an XDP program not aware of multi buffer\n" , + params->sw_mtu, +mlx5e_xdp_max_mtu (params,NULL )); +return false; + } + +return true; +} +int mlx5e_change_mtu (struct net_device *netdev,int new_mtu,mlx5e_fp_preactivate preactivate) {
@@ -3972,10 +3997,7 @@ int mlx5e_change_mtu(struct net_device *netdev, int new_mtu,if (err) goto out; -if (params->xdp_prog && - !mlx5e_rx_is_linear_skb (&new_params,NULL )) { -netdev_err (netdev,"MTU(%d) > %d is not allowed while XDP enabled\n" , - new_mtu,mlx5e_xdp_max_mtu (params,NULL )); +if (new_params.xdp_prog && !mlx5e_params_validate_xdp (netdev, &new_params)) { err = -EINVAL ; goto out; }
@@ -4458,15 +4480,8 @@ static int mlx5e_xdp_allowed(struct mlx5e_priv *priv, struct bpf_prog *prog) new_params = priv->channels.params; new_params.xdp_prog = prog; -/* No XSK params: AF_XDP can't be enabled yet at the point of setting -* the XDP program. -*/ -if (!mlx5e_rx_is_linear_skb (&new_params,NULL )) { -netdev_warn (netdev,"XDP is not allowed with MTU(%d) > %d\n" , - new_params.sw_mtu, -mlx5e_xdp_max_mtu (&new_params,NULL )); +if (!mlx5e_params_validate_xdp (netdev, &new_params))return -EINVAL ; - }return 0 ; }
Signature · 1 line
2.35.1
PATCH 13/15 net/mlx5e: Remove MLX5E_XDP_TX_DS_COUNT
In reply to Saeed MahameedFrom: Maxim Mikityanskiy <[email protected]>
After introducing multi-buffer XDP_TX, the MLX5E_XDP_TX_DS_COUNT define became misleading. It's no longer the DS count of an XDP_TX WQE, this WQE can be longer because of fragments.
As this define is only used at one place in mlx5e_open_xdpsq(), it's also not very useful anymore. This commit removes the define and puts the calculation of ds_count for prefilled single-fragment WQEs inline.
--- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h | 1 - drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +0 -1
@@ -38,7 +38,6 @@#include "en/txrx.h" #define MLX5E_XDP_MIN_INLINE (ETH_HLEN + VLAN_HLEN) -#define MLX5E_XDP_TX_DS_COUNT (MLX5E_TX_WQE_EMPTY_DS_COUNT + 1 /* SG DS */)#define MLX5E_XDP_INLINE_WQE_MAX_DS_CNT 16#define MLX5E_XDP_INLINE_WQE_SZ_THRSD \
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -1
@@ -1681,7 +1681,7 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,mlx5e_set_xmit_fp (sq, param->is_mpw);if (!param->is_mpw && !test_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state)) { -unsigned int ds_cnt =MLX5E_XDP_TX_DS_COUNT ; +unsigned int ds_cnt =MLX5E_TX_WQE_EMPTY_DS_COUNT +1 ;unsigned int inline_hdr_sz =0 ;int i;
Signature · 1 line
2.35.1
PATCH 14/15 net/mlx5e: Statify function mlx5_cmd_trigger_completions
In reply to Saeed MahameedFrom: Tariq Toukan <[email protected]>
Starting from commit4cab346bcf74 ("net/mlx5: No command allowed when command interface is not ready"), no calls to mlx5_cmd_trigger_completions() are external to cmd.c anymore. Make it a static function.
--- drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/cmd.c +1 -1
@@ -1720,7 +1720,7 @@ static void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool force } } -void mlx5_cmd_trigger_completions (struct mlx5_core_dev *dev) +static void mlx5_cmd_trigger_completions (struct mlx5_core_dev *dev) {struct mlx5_cmd *cmd = &dev->cmd;unsigned long bitmask;
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +0 -1
@@ -176,7 +176,6 @@ int mlx5_destroy_scheduling_element_cmd(struct mlx5_core_dev *dev, u8 hierarchy, u32 element_id);int mlx5_wait_for_pages (struct mlx5_core_dev *dev,int *pages); -void mlx5_cmd_trigger_completions (struct mlx5_core_dev *dev);void mlx5_cmd_flush (struct mlx5_core_dev *dev);void mlx5_cq_debugfs_init (struct mlx5_core_dev *dev);void mlx5_cq_debugfs_cleanup (struct mlx5_core_dev *dev);
Signature · 1 line
2.35.1
PATCH 15/15 net/mlx5e: HTB, remove unused function declaration
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
There is no function mlx5e_get_sq(), remove the declaration.
--- drivers/net/ethernet/mellanox/mlx5/core/en/qos.h | 1 - 1 file changed, 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en/qos.h +0 -1
@@ -18,7 +18,6 @@ int mlx5e_qos_cur_leaf_nodes(struct mlx5e_priv *priv);/* TX datapath API */ int mlx5e_get_txq_by_classid (struct mlx5e_priv *priv,u16 classid); -struct mlx5e_txqsq *mlx5e_get_sq (struct mlx5e_priv *priv,int qid);/* SQ lifecycle */ int mlx5e_qos_open_queues (struct mlx5e_priv *priv,struct mlx5e_channels *chs);
Signature · 1 line
2.35.1