net/mlx5e: Extend XDP multi-buffer capabilities
Subsystems (5)
Tariq Toukan
maintainercover net/mlx5e: Extend XDP multi-buffer capabilities
Hi,
This series extends the XDP multi-buffer support in the mlx5e driver.
Patchset breakdown:
- Infrastructural changes and preparations.
- Add XDP multi-buffer support for XDP redirect-in.
- Use TX MPWQE (multi-packet WQE) HW feature for non-linear single-segmented XDP frames.
- Add XDP multi-buffer support for striding RQ.
In Striding RQ, we overcome the lack of headroom and tailroom between the RQ strides by allocating a side page per packet and using it for the xdp_buff descriptor. We structure the xdp_buff so that it contains nothing in the linear part, and the whole packet resides in the fragments.
Performance highlight:
Packet rate test, 64 bytes, 32 channels, MTU 9000 bytes.
CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz.
NIC: ConnectX-6 Dx, at 100 Gbps.
+----------+-------------+-------------+---------+ | Test | Legacy RQ | Striding RQ | Speedup | +----------+-------------+-------------+---------+ | XDP_DROP | 101,615,544 | 117,191,020 | +15% | +----------+-------------+-------------+---------+ | XDP_TX | 95,608,169 | 117,043,422 | +22% | +----------+-------------+-------------+---------+
Series generated against net commit:e61caf04b9f8 Merge branch 'page_pool-allow-caching-from-safely-localized-napi'
I'm submitting this directly as Saeed is traveling.
Regards,
Tariq
Tariq Toukan (15):
net/mlx5e: Move XDP struct and enum to XDP header net/mlx5e: Move struct mlx5e_xmit_data to datapath header net/mlx5e: Introduce extended version for mlx5e_xmit_data net/mlx5e: XDP, Remove doubtful unlikely calls
net/mlx5e: XDP, Use multiple single-entry objects in xdpi_fifo net/mlx5e: XDP, Add support for multi-buffer XDP redirect-in net/mlx5e: XDP, Improve Striding RQ check with XDP net/mlx5e: XDP, Let XDP checker function get the params as input net/mlx5e: XDP, Consider large muti-buffer packets in Striding RQ
params calculations
net/mlx5e: XDP, Remove un-established assumptions on XDP buffer net/mlx5e: XDP, Allow non-linear single-segment frames in XDP TX MPWQE net/mlx5e: RX, Take shared info fragment addition into a function net/mlx5e: RX, Generalize mlx5e_fill_mxbuf()
net/mlx5e: RX, Prepare non-linear striding RQ for XDP multi-buffer
support
net/mlx5e: RX, Add XDP multi-buffer support in Striding RQ
drivers/net/ethernet/mellanox/mlx5/core/en.h | 46 +-- .../ethernet/mellanox/mlx5/core/en/params.c | 32 +- .../ethernet/mellanox/mlx5/core/en/params.h | 3 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 13 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 305 +++++++++++++----- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 55 +++- .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 12 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 92 +++--- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 215 +++++++++--- 9 files changed, 519 insertions(+), 254 deletions(-)
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 1/15 net/mlx5e: Move XDP struct and enum to XDP header
Move struct mlx5e_xdp_info and enum mlx5e_xdp_xmit_mode from the generic en.h to the XDP header, where they belong.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 35 ------------------- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 35 +++++++++++++++++++ 2 files changed, 35 insertions(+), 35 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -35
@@ -475,41 +475,6 @@ struct mlx5e_txqsq {cqe_ts_to_ns ptp_cyc2time; }____cacheline_aligned_in_smp ; -/* XDP packets can be transmitted in different ways. On completion, we need to -* distinguish between them to clean up things in a proper way. -*/ -enum mlx5e_xdp_xmit_mode { -/* An xdp_frame was transmitted due to either XDP_REDIRECT from another -* device or XDP_TX from an XSK RQ. The frame has to be unmapped and -* returned. -*/ -MLX5E_XDP_XMIT_MODE_FRAME , - -/* The xdp_frame was created in place as a result of XDP_TX from a -* regular RQ. No DMA remapping happened, and the page belongs to us. -*/ -MLX5E_XDP_XMIT_MODE_PAGE , - -/* No xdp_frame was created at all, the transmit happened from a UMEM -* page. The UMEM Completion Ring producer pointer has to be increased. -*/ -MLX5E_XDP_XMIT_MODE_XSK , -}; - -struct mlx5e_xdp_info { -enum mlx5e_xdp_xmit_mode mode; -union { -struct { -struct xdp_frame *xdpf; -dma_addr_t dma_addr; - } frame; -struct { -struct mlx5e_rq *rq; -struct page *page; - } page; - }; -}; -struct mlx5e_xmit_data {dma_addr_t dma_addr;void *data;
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +35 -0
@@ -50,6 +50,41 @@ struct mlx5e_xdp_buff {struct mlx5e_rq *rq; }; +/* XDP packets can be transmitted in different ways. On completion, we need to +* distinguish between them to clean up things in a proper way. +*/ +enum mlx5e_xdp_xmit_mode { +/* An xdp_frame was transmitted due to either XDP_REDIRECT from another +* device or XDP_TX from an XSK RQ. The frame has to be unmapped and +* returned. +*/ +MLX5E_XDP_XMIT_MODE_FRAME , + +/* The xdp_frame was created in place as a result of XDP_TX from a +* regular RQ. No DMA remapping happened, and the page belongs to us. +*/ +MLX5E_XDP_XMIT_MODE_PAGE , + +/* No xdp_frame was created at all, the transmit happened from a UMEM +* page. The UMEM Completion Ring producer pointer has to be increased. +*/ +MLX5E_XDP_XMIT_MODE_XSK , +}; + +struct mlx5e_xdp_info { +enum mlx5e_xdp_xmit_mode mode; +union { +struct { +struct xdp_frame *xdpf; +dma_addr_t dma_addr; + } frame; +struct { +struct mlx5e_rq *rq; +struct page *page; + } page; + }; +}; +struct mlx5e_xsk_param ;int mlx5e_xdp_max_mtu (struct mlx5e_params *params,struct mlx5e_xsk_param *xsk); bool mlx5e_xdp_handle(struct mlx5e_rq *rq,
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 2/15 net/mlx5e: Move struct mlx5e_xmit_data to datapath header
In reply to Tariq ToukanMove TX datapath struct from the generic en.h to the datapath txrx.h header, where it belongs.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 7 +------ drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 ++++++ 2 files changed, 7 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -6
@@ -475,12 +475,6 @@ struct mlx5e_txqsq {cqe_ts_to_ns ptp_cyc2time; }____cacheline_aligned_in_smp ; -struct mlx5e_xmit_data { -dma_addr_t dma_addr; -void *data; -u32 len; -}; -struct mlx5e_xdp_info_fifo {struct mlx5e_xdp_info *xi;u32 *cc;
@@ -489,6 +483,7 @@ struct mlx5e_xdp_info_fifo { };struct mlx5e_xdpsq ; +struct mlx5e_xmit_data ;typedef int (*mlx5e_fp_xmit_xdp_frame_check )(struct mlx5e_xdpsq *); typedef bool (*mlx5e_fp_xmit_xdp_frame)(struct mlx5e_xdpsq *, struct mlx5e_xmit_data *,
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +6 -0
@@ -77,6 +77,12 @@ static inline bool mlx5e_rx_hw_stamp(struct hwtstamp_config *config) }/* TX */ +struct mlx5e_xmit_data { +dma_addr_t dma_addr; +void *data; +u32 len; +}; +netdev_tx_t mlx5e_xmit (struct sk_buff *skb,struct net_device *dev);bool mlx5e_poll_tx_cq (struct mlx5e_cq *cq,int napi_budget);void mlx5e_free_txqsq_descs (struct mlx5e_txqsq *sq);
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 3/15 net/mlx5e: Introduce extended version for mlx5e_xmit_data
In reply to Tariq ToukanIntroduce struct mlx5e_xmit_data_frags to be used for non-linear xmit buffers. Let it include sinfo pointer.
Take one bit from the len field to indicate if the descriptor has fragments and can be casted-up into the extended version.
Zero-init to make sure has_frags, and potentially future fields, are zero when not explicitly assigned.
Another field will be added in a downstream patch to indicate and point to dma addresses of the different frags, for redirect-in requests.
This simplifies the mlx5e_xmit_xdp_frame/mlx5e_xmit_xdp_frame_mpwqe functions params.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 - .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 8 ++- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 63 ++++++++++--------- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 - .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 4 +- 5 files changed, 44 insertions(+), 34 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -1
@@ -487,7 +487,6 @@ struct mlx5e_xmit_data;typedef int (*mlx5e_fp_xmit_xdp_frame_check )(struct mlx5e_xdpsq *);typedef bool (*mlx5e_fp_xmit_xdp_frame )(struct mlx5e_xdpsq *,struct mlx5e_xmit_data *, -struct skb_shared_info *,int );struct mlx5e_xdpsq {
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +7 -1
@@ -80,7 +80,13 @@ static inline bool mlx5e_rx_hw_stamp(struct hwtstamp_config *config)struct mlx5e_xmit_data {dma_addr_t dma_addr;void *data; -u32 len; +u32 len :31 ; +u32 has_frags :1 ; +}; + +struct mlx5e_xmit_data_frags { +struct mlx5e_xmit_data xd; +struct skb_shared_info *sinfo; };netdev_tx_t mlx5e_xmit (struct sk_buff *skb,struct net_device *dev);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +35 -28
162 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +0 -2
@@ -101,11 +101,9 @@ extern const struct xdp_metadata_ops mlx5e_xdp_metadata_ops; INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct skb_shared_info *sinfo,int check_result)); INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,struct mlx5e_xmit_data *xdptxd, -struct skb_shared_info *sinfo,int check_result)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check_mpwqe(struct mlx5e_xdpsq *sq)); INDIRECT_CALLABLE_DECLARE(int mlx5e_xmit_xdp_frame_check(struct mlx5e_xdpsq *sq));
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +2 -2
@@ -61,7 +61,6 @@ static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq,bool mlx5e_xsk_tx (struct mlx5e_xdpsq *sq,unsigned int budget) {struct xsk_buff_pool *pool = sq->xsk_pool; -struct mlx5e_xmit_data xdptxd;struct mlx5e_xdp_info xdpi;bool work_done = true;bool flush = false;
@@ -73,6 +72,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) mlx5e_xmit_xdp_frame_check_mpwqe, mlx5e_xmit_xdp_frame_check, sq); +struct mlx5e_xmit_data xdptxd = {};struct xdp_desc desc;bool ret;
@@ -97,7 +97,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)xsk_buff_raw_dma_sync_for_device (pool, xdptxd.dma_addr, xdptxd.len); ret =INDIRECT_CALL_2 (sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, &xdptxd,NULL , + mlx5e_xmit_xdp_frame, sq, &xdptxd, check_result);if (unlikely (!ret)) {if (sq->mpwqe.wqe)
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 4/15 net/mlx5e: XDP, Remove doubtful unlikely calls
In reply to Tariq ToukanIt is not likely nor unlikely that the xdp buff has fragments, it depends on the program loaded and size of the packet received.
--- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +5 -5
@@ -126,7 +126,7 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, dma_addr =page_pool_get_dma_addr (page) + (xdpf->data - (void *)xdpf);dma_sync_single_for_device (sq->pdev, dma_addr, xdptxd->len,DMA_BIDIRECTIONAL ); -if (unlikely (xdptxd->has_frags)) { +if (xdptxd->has_frags) { xdptxdf.sinfo =xdp_get_shared_info_from_frame (xdpf);for (i =0 ; i < xdptxdf.sinfo->nr_frags; i++) {
@@ -151,7 +151,7 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, xdpi.page.page = page;mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); -if (unlikely (xdptxd->has_frags)) { +if (xdptxd->has_frags) {for (i =0 ; i < xdptxdf.sinfo->nr_frags; i++) {skb_frag_t *frag = &xdptxdf.sinfo->frags[i];
@@ -395,7 +395,7 @@ mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxstruct mlx5e_tx_mpwqe *session = &sq->mpwqe;struct mlx5e_xdpsq_stats *stats = sq->stats; -if (unlikely (xdptxd->has_frags)) { +if (xdptxd->has_frags) {/* MPWQE is enabled, but a multi-buffer packet is queued for * transmission. MPWQE can't send fragmented packets, so close * the current session and fall back to a regular WQE.
@@ -483,7 +483,7 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,if (!check_result) {int stop_room =1 ; -if (unlikely (xdptxd->has_frags)) { +if (xdptxd->has_frags) { ds_cnt += xdptxdf->sinfo->nr_frags; num_frags = xdptxdf->sinfo->nr_frags; num_wqebbs =DIV_ROUND_UP (ds_cnt,MLX5_SEND_WQEBB_NUM_DS );
@@ -525,7 +525,7 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, cseg->opmod_idx_opcode =cpu_to_be32 ((sq->pc <<8 ) |MLX5_OPCODE_SEND ); -if (unlikely (test_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state))) { +if (test_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state)) {u8 num_pkts =1 + num_frags;int i;
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 5/15 net/mlx5e: XDP, Use multiple single-entry objects in xdpi_fifo
In reply to Tariq ToukanHere we fix the current wi->num_pkts abuse, as it was used to indicate multiple xdpi entries in the xdpi_fifo.
Instead, reduce mlx5e_xdp_info to the size of a single field, making it a union of unions. Per packet, use as many instances as needed to provide the information needed at the time of completion.
The sequence of xdpi instances pushed is well defined, derived by the xmit_mode.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 95 +++++++++++++------ .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 38 +++++--- .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 8 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 8 +- 5 files changed, 101 insertions(+), 50 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -1
@@ -476,7 +476,7 @@ struct mlx5e_txqsq { }____cacheline_aligned_in_smp ;struct mlx5e_xdp_info_fifo { -struct mlx5e_xdp_info *xi; +union mlx5e_xdp_info *xi;u32 *cc;u32 *pc;u32 mask;
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +66 -29
178 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +25 -13
@@ -71,18 +71,30 @@ enum mlx5e_xdp_xmit_mode { MLX5E_XDP_XMIT_MODE_XSK, }; -struct mlx5e_xdp_info { +/* xmit_mode entry is pushed to the fifo per packet, followed by multiple +* entries, as follows: +* +* MLX5E_XDP_XMIT_MODE_FRAME: +* xdpf, dma_addr_1, dma_addr_2, ... , dma_addr_num. +* 'num' is derived from xdpf. +* +* MLX5E_XDP_XMIT_MODE_PAGE: +* num, page_1, page_2, ... , page_num. +* +* MLX5E_XDP_XMIT_MODE_XSK: +* none. +*/ +union mlx5e_xdp_info {enum mlx5e_xdp_xmit_mode mode;union { -struct { -struct xdp_frame *xdpf; -dma_addr_t dma_addr; - } frame; -struct { -struct mlx5e_rq *rq; -struct page *page; - } page; - }; +struct xdp_frame *xdpf; +dma_addr_t dma_addr; + } frame; +union { +struct mlx5e_rq *rq; +u8 num; +struct page *page; + } page; };struct mlx5e_xsk_param ;
@@ -212,14 +224,14 @@ mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq,static inline void mlx5e_xdpi_fifo_push (struct mlx5e_xdp_info_fifo *fifo, -struct mlx5e_xdp_info *xi) +union mlx5e_xdp_info xi) {u32 i = (*fifo->pc)++ & fifo->mask; - fifo->xi[i] = *xi; + fifo->xi[i] = xi; } -static inline struct mlx5e_xdp_info +static inline union mlx5e_xdp_info mlx5e_xdpi_fifo_pop (struct mlx5e_xdp_info_fifo *fifo) {return fifo->xi[(*fifo->cc)++ & fifo->mask];
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +4 -4
@@ -44,7 +44,7 @@ int mlx5e_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags)* same. */ static void mlx5e_xsk_tx_post_err (struct mlx5e_xdpsq *sq, -struct mlx5e_xdp_info *xdpi) +union mlx5e_xdp_info *xdpi) {u16 pi =mlx5_wq_cyc_ctr2ix (&sq->wq, sq->pc);struct mlx5e_xdp_wqe_info *wi = &sq->db.wqe_info[pi];
@@ -54,14 +54,14 @@ static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq, wi->num_pkts =1 ; nopwqe =mlx5e_post_nop (&sq->wq, sq->sqn, &sq->pc); -mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, xdpi); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, *xdpi); sq->doorbell_cseg = &nopwqe->ctrl; }bool mlx5e_xsk_tx (struct mlx5e_xdpsq *sq,unsigned int budget) {struct xsk_buff_pool *pool = sq->xsk_pool; -struct mlx5e_xdp_info xdpi; +union mlx5e_xdp_info xdpi;bool work_done = true;bool flush = false;
@@ -105,7 +105,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)mlx5e_xsk_tx_post_err (sq, &xdpi); }else { -mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, &xdpi); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, xdpi); } flush = true;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -3
@@ -1300,17 +1300,19 @@ static int mlx5e_alloc_xdpsq_fifo(struct mlx5e_xdpsq *sq, int numa) {struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo;int wq_sz =mlx5_wq_cyc_get_size (&sq->wq); -int dsegs_per_wq = wq_sz *MLX5_SEND_WQEBB_NUM_DS ; +int entries = wq_sz *MLX5_SEND_WQEBB_NUM_DS *2 ;/* upper bound for maximum num of +* entries of all xmit_modes. +*/ size_t size; - size =array_size (sizeof (*xdpi_fifo->xi), dsegs_per_wq); + size =array_size (sizeof (*xdpi_fifo->xi), entries); xdpi_fifo->xi =kvzalloc_node (size,GFP_KERNEL , numa);if (!xdpi_fifo->xi)return -ENOMEM ; xdpi_fifo->pc = &sq->xdpi_fifo_pc; xdpi_fifo->cc = &sq->xdpi_fifo_cc; - xdpi_fifo->mask = dsegs_per_wq -1 ; + xdpi_fifo->mask = entries -1 ;return 0 ; }
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 6/15 net/mlx5e: XDP, Add support for multi-buffer XDP redirect-in
In reply to Tariq ToukanHandle multi-buffer XDP redirect-in requests coming through mlx5e_xdp_xmit.
Extend struct mlx5e_xmit_data_frags with an additional dma_arr field, to point to the fragments dma mapping, as they cannot be retrieved via the page_pool_get_dma_addr() function.
Push a dma_addr xdpi instance per each fragment, and use them in the completion flow to dma_unmap the frags.
Finally, remove the restriction in mlx5e_open_xdpsq, and set the flag in xdp_features.
--- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 1 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 82 ++++++++++++++++--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 9 +- 3 files changed, 75 insertions(+), 17 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +1 -0
@@ -87,6 +87,7 @@ struct mlx5e_xmit_data {struct mlx5e_xmit_data_frags {struct mlx5e_xmit_data xd;struct skb_shared_info *sinfo; +dma_addr_t *dma_arr; };netdev_tx_t mlx5e_xmit (struct sk_buff *skb,struct net_device *dev);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +71 -11
123 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -6
@@ -1862,11 +1862,7 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params, csp.min_inline_mode = sq->min_inline_mode;set_bit (MLX5E_SQ_STATE_ENABLED , &sq->state); -/* Don't enable multi buffer on XDP_REDIRECT SQ, as it's not yet -* supported by upstream, and there is no defined trigger to allow -* transmitting redirected multi-buffer frames. -*/ -if (param->is_xdp_mb && !is_redirect) +if (param->is_xdp_mb)set_bit (MLX5E_SQ_STATE_XDP_MULTIBUF , &sq->state); err =mlx5e_create_sq_rdy (c->mdev, param, &csp,0 , &sq->sqn);
@@ -4068,7 +4064,8 @@ void mlx5e_set_xdp_feature(struct net_device *netdev) val =NETDEV_XDP_ACT_BASIC |NETDEV_XDP_ACT_REDIRECT |NETDEV_XDP_ACT_XSK_ZEROCOPY | -NETDEV_XDP_ACT_NDO_XMIT ; +NETDEV_XDP_ACT_NDO_XMIT | +NETDEV_XDP_ACT_NDO_XMIT_SG ;if (params->rq_wq_type ==MLX5_WQ_TYPE_CYCLIC ) val |=NETDEV_XDP_ACT_RX_SG ;xdp_set_features_flag (netdev, val);
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 7/15 net/mlx5e: XDP, Improve Striding RQ check with XDP
In reply to Tariq ToukanNon-linear mem scheme of Striding RQ does not yet support XDP at this point. Take the check where it belongs, inside the params validation function mlx5e_params_validate_xdp().
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 23 ++++++++----------- 1 file changed, 9 insertions(+), 14 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +9 -14
@@ -4261,8 +4261,16 @@ static bool mlx5e_params_validate_xdp(struct net_device *netdev,/* No XSK params: AF_XDP can't be enabled yet at the point of setting * the XDP program. */ - is_linear =mlx5e_rx_is_linear_skb (mdev, params,NULL ); + is_linear = params->rq_wq_type ==MLX5_WQ_TYPE_CYCLIC ? +mlx5e_rx_is_linear_skb (mdev, params,NULL ) : +mlx5e_rx_mpwqe_is_linear_skb (mdev, params,NULL ); +/* XDP affects striding RQ parameters. Block XDP if striding RQ won't be +* supported with the new parameters: if PAGE_SIZE is bigger than +* MLX5_MPWQE_LOG_STRIDE_SZ_MAX, striding RQ can't be used, even though +* the MTU is small enough for the linear mode, because XDP uses strides +* of PAGE_SIZE on regular RQs. +*/ if (!is_linear && params->rq_wq_type !=MLX5_WQ_TYPE_CYCLIC ) { netdev_warn(netdev, "XDP is not allowed with striding RQ and MTU(%d) > %d\n", params->sw_mtu,
@@ -4817,19 +4825,6 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog) new_params = priv->channels.params; new_params.xdp_prog = prog; -/* XDP affects striding RQ parameters. Block XDP if striding RQ won't be -* supported with the new parameters: if PAGE_SIZE is bigger than -* MLX5_MPWQE_LOG_STRIDE_SZ_MAX, striding RQ can't be used, even though -* the MTU is small enough for the linear mode, because XDP uses strides -* of PAGE_SIZE on regular RQs. -*/ -if (reset &&MLX5E_GET_PFLAG (&new_params,MLX5E_PFLAG_RX_STRIDING_RQ )) { -/* Checking for regular RQs here; XSK RQs were checked on XSK bind. */ - err =mlx5e_mpwrq_validate_regular (priv->mdev, &new_params); -if (err) - goto unlock; - } - old_prog = priv->channels.params.xdp_prog; err =mlx5e_safe_switch_params (priv, &new_params,NULL ,NULL , reset);
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 8/15 net/mlx5e: XDP, Let XDP checker function get the params as input
In reply to Tariq ToukanChange mlx5e_xdp_allowed() so it gets the params structure with the xdp_prog applied, rather than creating a local copy based on the current params in priv.
This reduces the amount of memory on the stack, and acts on the exact params instance that's about to be applied.
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 21 +++++++------------ 1 file changed, 8 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +8 -13
@@ -4773,20 +4773,15 @@ static void mlx5e_tx_timeout(struct net_device *dev, unsigned int txqueue)queue_work (priv->wq, &priv->tx_timeout_work); } -static int mlx5e_xdp_allowed (struct mlx5e_priv *priv,struct bpf_prog *prog) +static int mlx5e_xdp_allowed (struct net_device *netdev,struct mlx5_core_dev *mdev, +struct mlx5e_params *params) { -struct net_device *netdev = priv->netdev; -struct mlx5e_params new_params; - -if (priv->channels.params.packet_merge.type !=MLX5E_PACKET_MERGE_NONE ) { +if (params->packet_merge.type !=MLX5E_PACKET_MERGE_NONE ) {netdev_warn (netdev,"can't set XDP while HW-GRO/LRO is on, disable them first\n" );return -EINVAL ; } - new_params = priv->channels.params; - new_params.xdp_prog = prog; - -if (!mlx5e_params_validate_xdp (netdev, priv->mdev, &new_params)) +if (!mlx5e_params_validate_xdp (netdev, mdev, params))return -EINVAL ;return 0 ;
@@ -4813,8 +4808,11 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog)mutex_lock (&priv->state_lock); + new_params = priv->channels.params; + new_params.xdp_prog = prog; +if (prog) { - err =mlx5e_xdp_allowed (priv, prog); + err =mlx5e_xdp_allowed (netdev, priv->mdev, &new_params);if (err) goto unlock; }
@@ -4822,9 +4820,6 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog)/* no need for full reset when exchanging programs */ reset = (!priv->channels.params.xdp_prog || !prog); - new_params = priv->channels.params; - new_params.xdp_prog = prog; - old_prog = priv->channels.params.xdp_prog; err =mlx5e_safe_switch_params (priv, &new_params,NULL ,NULL , reset);
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 9/15 net/mlx5e: XDP, Consider large muti-buffer packets in Striding RQ params calculations
In reply to Tariq ToukanFunction mlx5e_rx_get_linear_stride_sz() returns PAGE_SIZE immediately in case an XDP program is attached. The more accurate formula is ALIGN(sz, PAGE_SIZE), to prevent two packets from residing on the same page.
The assumption behind the current code is that sz <= PAGE_SIZE holds for all cases with XDP program set.
This is true because it is being called from:
- 3 times from Striding RQ flows, in which XDP is not supported for such large packets.
- 1 time from Legacy RQ flow, under the condition mlx5e_rx_is_linear_skb().
No functional change here, just removing the implied assumption in preparation for supporting XDP multi-buffer in Striding RQ.
--- drivers/net/ethernet/mellanox/mlx5/core/en/params.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +7 -4
@@ -253,17 +253,20 @@ static u32 mlx5e_rx_get_linear_stride_sz(struct mlx5_core_dev *mdev, struct mlx5e_xsk_param *xsk, bool mpwqe) { +u32 sz; +/* XSK frames are mapped as individual pages, because frames may come in * an arbitrary order from random locations in the UMEM. */ if (xsk)return mpwqe ?1 <<mlx5e_mpwrq_page_shift (mdev, xsk) :PAGE_SIZE ; -/* XDP in mlx5e doesn't support multiple packets per page. */ -if (params->xdp_prog) -return PAGE_SIZE ; + sz =roundup_pow_of_two (mlx5e_rx_get_linear_sz_skb (params, false)); -return roundup_pow_of_two (mlx5e_rx_get_linear_sz_skb (params, false)); +/* XDP in mlx5e doesn't support multiple packets per page. +* Do not assume sz <= PAGE_SIZE if params->xdp_prog is set. +*/ +return params->xdp_prog && sz <PAGE_SIZE ?PAGE_SIZE : sz; } static u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5_core_dev *mdev,
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 10/15 net/mlx5e: XDP, Remove un-established assumptions on XDP buffer
In reply to Tariq ToukanRemove the assumption of non-zero linear length in the XDP xmit function, used to serve both internal XDP_TX operations as well as redirected-in requests.
Do not apply the MLX5E_XDP_MIN_INLINE check unless necessary.
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 36 +++++++++++-------- .../net/ethernet/mellanox/mlx5/core/en_main.c | 4 --- 2 files changed, 22 insertions(+), 18 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +22 -14
@@ -477,18 +477,26 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,u16 ds_cnt, inline_hdr_sz;u8 num_wqebbs =1 ;int num_frags =0 ; +bool inline_ok; +bool linear;u16 pi;struct mlx5e_xdpsq_stats *stats = sq->stats; -if (unlikely (dma_len <MLX5E_XDP_MIN_INLINE || sq->hw_mtu < dma_len)) { + inline_ok = sq->min_inline_mode ==MLX5_INLINE_MODE_NONE || + dma_len >=MLX5E_XDP_MIN_INLINE ; + +if (unlikely (!inline_ok || sq->hw_mtu < dma_len)) { stats->err++;return false; } - ds_cnt =MLX5E_TX_WQE_EMPTY_DS_COUNT +1 ; + inline_hdr_sz =0 ;if (sq->min_inline_mode !=MLX5_INLINE_MODE_NONE ) - ds_cnt++; + inline_hdr_sz =MLX5E_XDP_MIN_INLINE ; + + linear = !!(dma_len - inline_hdr_sz); + ds_cnt =MLX5E_TX_WQE_EMPTY_DS_COUNT + linear + !!inline_hdr_sz;/* check_result must be 0 if sinfo is passed. */ if (!check_result) {
@@ -517,22 +525,23 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, eseg = &wqe->eth; dseg = wqe->data; - inline_hdr_sz =0 ; -/* copy the inline part if required */ -if (sq->min_inline_mode !=MLX5_INLINE_MODE_NONE ) { +if (inline_hdr_sz) {memcpy (eseg->inline_hdr.start, xdptxd->data,sizeof (eseg->inline_hdr.start));memcpy (dseg, xdptxd->data +sizeof (eseg->inline_hdr.start), -MLX5E_XDP_MIN_INLINE -sizeof (eseg->inline_hdr.start)); - dma_len -=MLX5E_XDP_MIN_INLINE ; - dma_addr +=MLX5E_XDP_MIN_INLINE ; - inline_hdr_sz =MLX5E_XDP_MIN_INLINE ; + inline_hdr_sz -sizeof (eseg->inline_hdr.start)); + dma_len -= inline_hdr_sz; + dma_addr += inline_hdr_sz; dseg++; }/* write the dma part */ - dseg->addr =cpu_to_be64 (dma_addr); - dseg->byte_count =cpu_to_be32 (dma_len); +if (linear) { + dseg->addr =cpu_to_be64 (dma_addr); + dseg->byte_count =cpu_to_be32 (dma_len); + dseg->lkey = sq->mkey_be; + dseg++; + } cseg->opmod_idx_opcode =cpu_to_be32 ((sq->pc <<8 ) |MLX5_OPCODE_SEND );
@@ -543,7 +552,6 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,memset (eseg,0 ,sizeof (*eseg) -sizeof (eseg->trailer)); eseg->inline_hdr.sz =cpu_to_be16 (inline_hdr_sz); - dseg->lkey = sq->mkey_be;for (i =0 ; i < num_frags; i++) {skb_frag_t *frag = &xdptxdf->sinfo->frags[i];
@@ -553,10 +561,10 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd,page_pool_get_dma_addr (skb_frag_page (frag)) +skb_frag_off (frag); - dseg++; dseg->addr =cpu_to_be64 (addr); dseg->byte_count =cpu_to_be32 (skb_frag_size (frag)); dseg->lkey = sq->mkey_be; + dseg++; } cseg->qpn_ds =cpu_to_be32 ((sq->sqn <<8 ) | ds_cnt);
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0 -4
@@ -1886,7 +1886,6 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,struct mlx5e_tx_wqe *wqe =mlx5_wq_cyc_get_wqe (&sq->wq, i);struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;struct mlx5_wqe_eth_seg *eseg = &wqe->eth; -struct mlx5_wqe_data_seg *dseg; sq->db.wqe_info[i] = (struct mlx5e_xdp_wqe_info ) { .num_wqebbs =1 ,
@@ -1895,9 +1894,6 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params, cseg->qpn_ds =cpu_to_be32 ((sq->sqn <<8 ) | ds_cnt); eseg->inline_hdr.sz =cpu_to_be16 (inline_hdr_sz); - - dseg = (struct mlx5_wqe_data_seg *)cseg + (ds_cnt -1 ); - dseg->lkey = sq->mkey_be; } }
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 11/15 net/mlx5e: XDP, Allow non-linear single-segment frames in XDP TX MPWQE
In reply to Tariq ToukanUnder a few restrictions, TX MPWQE feature can serve multiple TX packets in a single TX descriptor. It requires each of the packets to have a single scatter entry / segment.
Today we allow only linear frames to use this feature, although there's no real problem with non-linear ones where the whole packet reside in the first fragment.
Expand the XDP TX MPWQE feature support to include such frames. This is in preparation for the downstream patch, in which we will generate such non-linear frames.
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 35 ++++++++++++++----- 1 file changed, 26 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +26 -9
@@ -405,18 +405,35 @@ mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptx {struct mlx5e_tx_mpwqe *session = &sq->mpwqe;struct mlx5e_xdpsq_stats *stats = sq->stats; +struct mlx5e_xmit_data *p = xdptxd; +struct mlx5e_xmit_data tmp;if (xdptxd->has_frags) { -/* MPWQE is enabled, but a multi-buffer packet is queued for -* transmission. MPWQE can't send fragmented packets, so close -* the current session and fall back to a regular WQE. -*/ -if (unlikely (sq->mpwqe.wqe)) -mlx5e_xdp_mpwqe_complete (sq); -return mlx5e_xmit_xdp_frame (sq, xdptxd,0 ); +struct mlx5e_xmit_data_frags *xdptxdf = + container_of(xdptxd, struct mlx5e_xmit_data_frags, xd); + +if (!!xdptxd->len + xdptxdf->sinfo->nr_frags >1 ) { +/* MPWQE is enabled, but a multi-buffer packet is queued for +* transmission. MPWQE can't send fragmented packets, so close +* the current session and fall back to a regular WQE. +*/ +if (unlikely (sq->mpwqe.wqe)) +mlx5e_xdp_mpwqe_complete (sq); +return mlx5e_xmit_xdp_frame (sq, xdptxd,0 ); + } +if (!xdptxd->len) { +skb_frag_t *frag = &xdptxdf->sinfo->frags[0 ]; + + tmp.data =skb_frag_address (frag); + tmp.len =skb_frag_size (frag); + tmp.dma_addr = xdptxdf->dma_arr ? xdptxdf->dma_arr[0 ] : +page_pool_get_dma_addr (skb_frag_page (frag)) + +skb_frag_off (frag); + p = &tmp; + } } -if (unlikely (xdptxd->len > sq->hw_mtu)) { +if (unlikely (p->len > sq->hw_mtu)) { stats->err++;return false; }
@@ -434,7 +451,7 @@ mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxmlx5e_xdp_mpwqe_session_start (sq); } -mlx5e_xdp_mpwqe_add_dseg (sq, xdptxd, stats); +mlx5e_xdp_mpwqe_add_dseg (sq, p, stats);if (unlikely (mlx5e_xdp_mpwqe_is_full (session, sq->max_sq_mpw_wqebbs)))mlx5e_xdp_mpwqe_complete (sq);
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 12/15 net/mlx5e: RX, Take shared info fragment addition into a function
In reply to Tariq ToukanIntroduce mlx5e_add_skb_shared_info_frag(), a function dedicated for adding a fragment into a struct skb_shared_info object.
Use it in the Legacy RQ flow. Similar usage will be added in a downstream patch by the corresponding Striding RQ flow.
--- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 56 ++++++++++--------- 1 file changed, 31 insertions(+), 25 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +31 -25
@@ -471,6 +471,35 @@ static int mlx5e_refill_rx_wqes(struct mlx5e_rq *rq, u16 ix, int wqe_bulk)return i; } +static void +mlx5e_add_skb_shared_info_frag (struct mlx5e_rq *rq,struct skb_shared_info *sinfo, +struct xdp_buff *xdp,struct mlx5e_frag_page *frag_page, +u32 frag_offset,u32 len) +{ +skb_frag_t *frag; + +dma_addr_t addr =page_pool_get_dma_addr (frag_page->page); + +dma_sync_single_for_cpu (rq->pdev, addr + frag_offset, len, rq->buff.map_dir); +if (!xdp_buff_has_frags (xdp)) { +/* Init on the first fragment to avoid cold cache access +* when possible. +*/ + sinfo->nr_frags =0 ; + sinfo->xdp_frags_size =0 ; +xdp_buff_set_frags_flag (xdp); + } + + frag = &sinfo->frags[sinfo->nr_frags++]; +__skb_frag_set_page (frag, frag_page->page); +skb_frag_off_set (frag, frag_offset); +skb_frag_size_set (frag, len); + +if (page_is_pfmemalloc (frag_page->page)) +xdp_buff_set_frag_pfmemalloc (xdp); + sinfo->xdp_frags_size += len; +} + static inline void mlx5e_add_skb_frag(struct mlx5e_rq *rq, struct sk_buff *skb, struct page *page, u32 frag_offset, u32 len,
@@ -1694,35 +1723,12 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi wi++;while (cqe_bcnt) { -skb_frag_t *frag; - frag_page = wi->frag_page; frag_consumed_bytes =min_t (u32, frag_info->frag_size, cqe_bcnt); - addr =page_pool_get_dma_addr (frag_page->page); -dma_sync_single_for_cpu (rq->pdev, addr + wi->offset, - frag_consumed_bytes, rq->buff.map_dir); - -if (!xdp_buff_has_frags (&mxbuf.xdp)) { -/* Init on the first fragment to avoid cold cache access -* when possible. -*/ - sinfo->nr_frags =0 ; - sinfo->xdp_frags_size =0 ; -xdp_buff_set_frags_flag (&mxbuf.xdp); - } - - frag = &sinfo->frags[sinfo->nr_frags++]; - -__skb_frag_set_page (frag, frag_page->page); -skb_frag_off_set (frag, wi->offset); -skb_frag_size_set (frag, frag_consumed_bytes); - -if (page_is_pfmemalloc (frag_page->page)) -xdp_buff_set_frag_pfmemalloc (&mxbuf.xdp); - - sinfo->xdp_frags_size += frag_consumed_bytes; +mlx5e_add_skb_shared_info_frag (rq, sinfo, &mxbuf.xdp, frag_page, + wi->offset, frag_consumed_bytes); truesize += frag_info->frag_stride; cqe_bcnt -= frag_consumed_bytes;
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 13/15 net/mlx5e: RX, Generalize mlx5e_fill_mxbuf()
In reply to Tariq ToukanMake the function more generic. Let it get an additional frame_sz parameter instead of deriving it from the RQ struct.
No functional change here, just a preparation for a downstream patch.
--- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +8 -5
@@ -1630,10 +1630,10 @@ struct sk_buff *mlx5e_build_linear_skb(struct mlx5e_rq *rq, void *va, }static void mlx5e_fill_mxbuf (struct mlx5e_rq *rq,struct mlx5_cqe64 *cqe, -void *va,u16 headroom,u32 len, +void *va,u16 headroom,u32 frame_sz,u32 len,struct mlx5e_xdp_buff *mxbuf) { -xdp_init_buff (&mxbuf->xdp, rq->buff.frame0_sz, &rq->xdp_rxq); +xdp_init_buff (&mxbuf->xdp, frame_sz, &rq->xdp_rxq);xdp_prepare_buff (&mxbuf->xdp, va, headroom, len, true); mxbuf->cqe = cqe; mxbuf->rq = rq;
@@ -1666,7 +1666,8 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi,struct mlx5e_xdp_buff mxbuf;net_prefetchw (va);/* xdp_frame data area */ -mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, cqe_bcnt, &mxbuf); +mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, rq->buff.frame0_sz, + cqe_bcnt, &mxbuf);if (mlx5e_xdp_handle (rq, prog, &mxbuf))return NULL ;/* page/packet was consumed by XDP */
@@ -1714,7 +1715,8 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *winet_prefetchw (va);/* xdp_frame data area */ net_prefetch (va + rx_headroom); -mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, frag_consumed_bytes, &mxbuf); +mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, rq->buff.frame0_sz, + frag_consumed_bytes, &mxbuf); sinfo =xdp_get_shared_info_from_buff (&mxbuf.xdp); truesize =0 ;
@@ -2042,7 +2044,8 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,struct mlx5e_xdp_buff mxbuf;net_prefetchw (va);/* xdp_frame data area */ -mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, cqe_bcnt, &mxbuf); +mlx5e_fill_mxbuf (rq, cqe, va, rx_headroom, rq->buff.frame0_sz, + cqe_bcnt, &mxbuf);if (mlx5e_xdp_handle (rq, prog, &mxbuf)) {if (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags))__set_bit (page_idx, wi->skip_release_bitmap);/* non-atomic */
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 14/15 net/mlx5e: RX, Prepare non-linear striding RQ for XDP multi-buffer support
In reply to Tariq ToukanIn preparation for supporting XDP multi-buffer in striding RQ, use xdp_buff struct to describe the packet. Make its skb_shared_info collide the one of the allocated SKB, then add the fragments using the xdp_buff API.
--- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 51 +++++++++++++++++-- 1 file changed, 47 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +47 -4
@@ -1977,10 +1977,17 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wstruct mlx5e_frag_page *frag_page = &wi->alloc_units.frag_pages[page_idx];u16 headlen =min_t (u16,MLX5E_RX_MAX_HEAD , cqe_bcnt);struct mlx5e_frag_page *head_page = frag_page; -u32 frag_offset = head_offset + headlen; -u32 byte_cnt = cqe_bcnt - headlen; +u32 frag_offset = head_offset; +u32 byte_cnt = cqe_bcnt; +struct skb_shared_info *sinfo; +struct mlx5e_xdp_buff mxbuf; +unsigned int truesize =0 ;struct sk_buff *skb; +u32 linear_frame_sz; +u16 linear_data_len;dma_addr_t addr; +u16 linear_hr; +void *va; skb =napi_alloc_skb (rq->cq.napi,ALIGN (MLX5E_RX_MAX_HEAD ,sizeof (long )));
@@ -1989,16 +1996,52 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wreturn NULL ; } + va = skb->head;net_prefetchw (skb->data); -/* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */ + frag_offset += headlen; + byte_cnt -= headlen; + linear_hr =skb_headroom (skb); + linear_data_len = headlen; + linear_frame_sz =MLX5_SKB_FRAG_SZ (skb_end_offset (skb));if (unlikely (frag_offset >=PAGE_SIZE )) { frag_page++; frag_offset -=PAGE_SIZE ; }skb_mark_for_recycle (skb); -mlx5e_fill_skb_data (skb, rq, frag_page, byte_cnt, frag_offset); +mlx5e_fill_mxbuf (rq, cqe, va, linear_hr, linear_frame_sz, linear_data_len, &mxbuf); +net_prefetch (mxbuf.xdp.data); + + sinfo =xdp_get_shared_info_from_buff (&mxbuf.xdp); + +while (byte_cnt) { +/* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */ +u32 pg_consumed_bytes =min_t (u32,PAGE_SIZE - frag_offset, byte_cnt); + +if (test_bit (MLX5E_RQ_STATE_SHAMPO , &rq->state)) + truesize += pg_consumed_bytes; +else + truesize +=ALIGN (pg_consumed_bytes,BIT (rq->mpwqe.log_stride_sz)); + +mlx5e_add_skb_shared_info_frag (rq, sinfo, &mxbuf.xdp, frag_page, frag_offset, + pg_consumed_bytes); + byte_cnt -= pg_consumed_bytes; + frag_offset =0 ; + frag_page++; + } +if (xdp_buff_has_frags (&mxbuf.xdp)) { +struct mlx5e_frag_page *pagep; + +xdp_update_skb_shared_info (skb, sinfo->nr_frags, + sinfo->xdp_frags_size, truesize, +xdp_buff_is_frag_pfmemalloc (&mxbuf.xdp)); + + pagep = frag_page - sinfo->nr_frags; +do + pagep->frags++; +while (++pagep < frag_page); + }/* copy header */ addr =page_pool_get_dma_addr (head_page->page); mlx5e_copy_skb_header(rq, skb, head_page->page, addr,
Signature · 1 line
2.34.1
Tariq Toukan
maintainerPATCH 15/15 net/mlx5e: RX, Add XDP multi-buffer support in Striding RQ
In reply to Tariq ToukanHere we add support for multi-buffer XDP handling in Striding RQ, which is our default out-of-the-box RQ type. Before this series, loading such an XDP program would fail, until you switch to the legacy RQ (by unsetting the rx_striding_rq priv-flag).
To overcome the lack of headroom and tailroom between the strides, we allocate a side page to be used for the descriptor (xdp_buff / skb) and the linear part. When an XDP program is attached, we structure the xdp_buff so that it contains no data in the linear part, and the whole packet resides in the fragments.
In case of XDP_PASS, where an SKB still needs to be created, we copy up to 256 bytes to its linear part, to match the current behavior, and satisfy functions that assume finding the packet headers in the SKB linear part (like eth_type_trans).
Performance testing:
Packet rate test, 64 bytes, 32 channels, MTU 9000 bytes.
CPU: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz.
NIC: ConnectX-6 Dx, at 100 Gbps.
+----------+-------------+-------------+---------+ | Test | Legacy RQ | Striding RQ | Speedup | +----------+-------------+-------------+---------+ | XDP_DROP | 101,615,544 | 117,191,020 | +15% | +----------+-------------+-------------+---------+ | XDP_TX | 95,608,169 | 117,043,422 | +22% | +----------+-------------+-------------+---------+
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../ethernet/mellanox/mlx5/core/en/params.c | 21 ++- .../ethernet/mellanox/mlx5/core/en/params.h | 3 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 37 +++-- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 135 +++++++++++++----- 5 files changed, 138 insertions(+), 59 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -587,6 +587,7 @@ union mlx5e_alloc_units {struct mlx5e_mpw_info {u16 consumed_strides; DECLARE_BITMAP(skip_release_bitmap, MLX5_MPWRQ_MAX_PAGES_PER_WQE); +struct mlx5e_frag_page linear_page;union mlx5e_alloc_units alloc_units; };
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +18 -3
@@ -323,6 +323,20 @@ static bool mlx5e_verify_rx_mpwqe_strides(struct mlx5_core_dev *mdev,return log_num_strides >=MLX5_MPWQE_LOG_NUM_STRIDES_BASE ; } +bool mlx5e_verify_params_rx_mpwqe_strides (struct mlx5_core_dev *mdev, +struct mlx5e_params *params, +struct mlx5e_xsk_param *xsk) +{ +u8 log_wqe_num_of_strides =mlx5e_mpwqe_get_log_num_strides (mdev, params, xsk); +u8 log_wqe_stride_size =mlx5e_mpwqe_get_log_stride_size (mdev, params, xsk); +enum mlx5e_mpwrq_umr_mode umr_mode =mlx5e_mpwrq_umr_mode (mdev, xsk); +u8 page_shift =mlx5e_mpwrq_page_shift (mdev, xsk); + +return mlx5e_verify_rx_mpwqe_strides (mdev, log_wqe_stride_size, + log_wqe_num_of_strides, + page_shift, umr_mode); +} +bool mlx5e_rx_mpwqe_is_linear_skb (struct mlx5_core_dev *mdev,struct mlx5e_params *params,struct mlx5e_xsk_param *xsk)
@@ -405,6 +419,10 @@ u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev,if (mlx5e_rx_mpwqe_is_linear_skb (mdev, params, xsk))return order_base_2 (mlx5e_rx_get_linear_stride_sz (mdev, params, xsk, true)); +/* XDP in mlx5e doesn't support multiple packets per page. */ +if (params->xdp_prog) +return PAGE_SHIFT ; +return MLX5_MPWRQ_DEF_LOG_STRIDE_SZ (mdev); }
@@ -575,9 +593,6 @@ int mlx5e_mpwrq_validate_regular(struct mlx5_core_dev *mdev, struct mlx5e_paramsif (!mlx5e_check_fragmented_striding_rq_cap (mdev, page_shift, umr_mode))return -EOPNOTSUPP ; -if (params->xdp_prog && !mlx5e_rx_mpwqe_is_linear_skb (mdev, params,NULL )) -return -EINVAL ; -return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +3 -0
@@ -153,6 +153,9 @@ int mlx5e_build_channel_param(struct mlx5_core_dev *mdev,u16 mlx5e_calc_sq_stop_room (struct mlx5_core_dev *mdev,struct mlx5e_params *params);int mlx5e_validate_params (struct mlx5_core_dev *mdev,struct mlx5e_params *params); +bool mlx5e_verify_params_rx_mpwqe_strides (struct mlx5_core_dev *mdev, +struct mlx5e_params *params, +struct mlx5e_xsk_param *xsk);static inline void mlx5e_params_print_info (struct mlx5_core_dev *mdev,struct mlx5e_params *params)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +18 -19
@@ -803,6 +803,9 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params, pool_size = rq->mpwqe.pages_per_wqe <<mlx5e_mpwqe_get_log_rq_size (mdev, params, xsk); +if (!mlx5e_rx_mpwqe_is_linear_skb (mdev, params, xsk) && params->xdp_prog) + pool_size *=2 ;/* additional page per packet for the linear part */ + rq->mpwqe.log_stride_sz =mlx5e_mpwqe_get_log_stride_size (mdev, params, xsk); rq->mpwqe.num_strides =BIT (mlx5e_mpwqe_get_log_num_strides (mdev, params, xsk));
@@ -4060,10 +4063,9 @@ void mlx5e_set_xdp_feature(struct net_device *netdev) val =NETDEV_XDP_ACT_BASIC |NETDEV_XDP_ACT_REDIRECT |NETDEV_XDP_ACT_XSK_ZEROCOPY | +NETDEV_XDP_ACT_RX_SG |NETDEV_XDP_ACT_NDO_XMIT |NETDEV_XDP_ACT_NDO_XMIT_SG ; -if (params->rq_wq_type ==MLX5_WQ_TYPE_CYCLIC ) - val |=NETDEV_XDP_ACT_RX_SG ;xdp_set_features_flag (netdev, val); }
@@ -4261,23 +4263,20 @@ static bool mlx5e_params_validate_xdp(struct net_device *netdev, mlx5e_rx_is_linear_skb(mdev, params, NULL) :mlx5e_rx_mpwqe_is_linear_skb (mdev, params,NULL ); -/* XDP affects striding RQ parameters. Block XDP if striding RQ won't be -* supported with the new parameters: if PAGE_SIZE is bigger than -* MLX5_MPWQE_LOG_STRIDE_SZ_MAX, striding RQ can't be used, even though -* the MTU is small enough for the linear mode, because XDP uses strides -* of PAGE_SIZE on regular RQs. -*/ -if (!is_linear && params->rq_wq_type !=MLX5_WQ_TYPE_CYCLIC ) { -netdev_warn (netdev,"XDP is not allowed with striding RQ and MTU(%d) > %d\n" , - params->sw_mtu, -mlx5e_xdp_max_mtu (params,NULL )); -return false; - } -if (!is_linear && !params->xdp_prog->aux->xdp_has_frags) { -netdev_warn (netdev,"MTU(%d) > %d, too big for an XDP program not aware of multi buffer\n" , - params->sw_mtu, -mlx5e_xdp_max_mtu (params,NULL )); -return false; +if (!is_linear) { +if (!params->xdp_prog->aux->xdp_has_frags) { +netdev_warn (netdev,"MTU(%d) > %d, too big for an XDP program not aware of multi buffer\n" , + params->sw_mtu, +mlx5e_xdp_max_mtu (params,NULL )); +return false; + } +if (params->rq_wq_type ==MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ && + !mlx5e_verify_params_rx_mpwqe_strides (mdev, params,NULL )) { +netdev_warn (netdev,"XDP is not allowed with striding RQ and MTU(%d) > %d\n" , + params->sw_mtu, +mlx5e_xdp_max_mtu (params,NULL )); +return false; + } }return true;
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +98 -37
159 lines not on this page · raw message
Signature · 1 line
2.34.1
Jakub Kicinski
maintainerQuoted 8 lines · On Mon, 17 Apr 2023 15:18:48 +0300 Tariq Toukan wrote:
This series extends the XDP multi-buffer support in the mlx5e driver.
Patchset breakdown:
- Infrastructural changes and preparations.
- Add XDP multi-buffer support for XDP redirect-in.
- Use TX MPWQE (multi-packet WQE) HW feature for non-linear single-segmented XDP frames.
- Add XDP multi-buffer support for striding RQ.
This is cd02a1a24897 in net-next, thanks!