Refactor UMR post send logic
Subsystems (2)
cover Refactor UMR post send logic
From: Leon Romanovsky <[email protected]>
Hi,
UMR are special QPs that require slightly different logic than other QPs. This series from Aharon refactors the logic around UMR QP to separate file and functions to clean the post send flow.
Thanks
Aharon Landau (12):
RDMA/mlx5: Move init and cleanup of UMR to umr.c RDMA/mlx5: Move umr checks to umr.h
RDMA/mlx5: Move mkey ctrl segment logic to umr.c RDMA/mlx5: Simplify get_umr_update_access_mask() RDMA/mlx5: Expose wqe posting helpers outside of wr.c RDMA/mlx5: Introduce mlx5_umr_post_send_wait() RDMA/mlx5: Use mlx5_umr_post_send_wait() to revoke MRs RDMA/mlx5: Use mlx5_umr_post_send_wait() to rereg pd access RDMA/mlx5: Move creation and free of translation tables to umr.c RDMA/mlx5: Use mlx5_umr_post_send_wait() to update MR pas RDMA/mlx5: Use mlx5_umr_post_send_wait() to update xlt RDMA/mlx5: Clean UMR QP type flow from mlx5_ib_post_send()
drivers/infiniband/hw/mlx5/Makefile | 1 + drivers/infiniband/hw/mlx5/main.c | 109 +---- drivers/infiniband/hw/mlx5/mlx5_ib.h | 89 +--- drivers/infiniband/hw/mlx5/mr.c | 421 +--------------- drivers/infiniband/hw/mlx5/odp.c | 63 ++- drivers/infiniband/hw/mlx5/qp.c | 1 + drivers/infiniband/hw/mlx5/umr.c | 700 +++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 97 ++++ drivers/infiniband/hw/mlx5/wr.c | 377 +++------------ drivers/infiniband/hw/mlx5/wr.h | 60 +++ 10 files changed, 976 insertions(+), 942 deletions(-) create mode 100644 drivers/infiniband/hw/mlx5/umr.c create mode 100644 drivers/infiniband/hw/mlx5/umr.h
Signature · 1 line
2.35.1
PATCH 6/12 RDMA/mlx5: Introduce mlx5_umr_post_send_wait()
From: Aharon Landau <[email protected]>
Introduce mlx5_umr_post_send_wait() that uses a UMR adjusted flow for posting WQEs. The next patches will gradually move UMR operations to use this flow. Once done, will get rid of mlx5_ib_post_send_wait().
mlx5_umr_post_send_wait gets already written WQE segments and will only memcpy it to the SQ. This way, we avoid packing all the data in a WR just to unpack it into the WQE.
--- drivers/infiniband/hw/mlx5/umr.c | 92 ++++++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 12 +++++ 2 files changed, 104 insertions(+)
drivers/infiniband/hw/mlx5/umr.c +92 -0
@@ -3,6 +3,7 @@#include "mlx5_ib.h" #include "umr.h" +#include "wr.h" static __be64 get_umr_enable_mr_mask (void ) {
@@ -228,3 +229,94 @@ void mlx5r_umr_resource_cleanup(struct mlx5_ib_dev *dev)ib_free_cq (dev->umrc.cq);ib_dealloc_pd (dev->umrc.pd); } + +static int mlx5r_umr_post_send (struct ib_qp *ibqp,u32 mkey,struct ib_cqe *cqe, +struct mlx5r_umr_wqe *wqe,bool with_data) +{ +unsigned int wqe_size = + with_data ?sizeof (struct mlx5r_umr_wqe ) : +sizeof (struct mlx5r_umr_wqe ) - +sizeof (struct mlx5_wqe_data_seg ); +struct mlx5_ib_dev *dev =to_mdev (ibqp->device); +struct mlx5_core_dev *mdev = dev->mdev; +struct mlx5_ib_qp *qp =to_mqp (ibqp); +struct mlx5_wqe_ctrl_seg *ctrl; +union { +struct ib_cqe *ib_cqe; +u64 wr_id; + } id; +void *cur_edge, *seg; +unsigned long flags; +unsigned int idx; +int size, err; + +if (unlikely (mdev->state ==MLX5_DEVICE_STATE_INTERNAL_ERROR )) +return -EIO ; + +spin_lock_irqsave (&qp->sq.lock, flags); + + err =mlx5r_begin_wqe (qp, &seg, &ctrl, &idx, &size, &cur_edge,0 , +cpu_to_be32 (mkey), false, false); +if (WARN_ON (err)) + goto out; + + qp->sq.wr_data[idx] =MLX5_IB_WR_UMR ; + +mlx5r_memcpy_send_wqe (&qp->sq, &cur_edge, &seg, &size, wqe, wqe_size); + + id.ib_cqe = cqe; +mlx5r_finish_wqe (qp, ctrl, seg, size, cur_edge, idx, id.wr_id,0 , +MLX5_FENCE_MODE_NONE ,MLX5_OPCODE_UMR ); + +mlx5r_ring_db (qp,1 , ctrl); + +out: +spin_unlock_irqrestore (&qp->sq.lock, flags); + +return err; +} + +static void mlx5r_umr_done (struct ib_cq *cq,struct ib_wc *wc) +{ +struct mlx5_ib_umr_context *context = + container_of(wc->wr_cqe, struct mlx5_ib_umr_context, cqe); + + context->status = wc->status; +complete (&context->done); +} + +static inline void mlx5r_umr_init_context (struct mlx5r_umr_context *context) +{ + context->cqe.done = mlx5r_umr_done; +init_completion (&context->done); +} + +static int mlx5r_umr_post_send_wait (struct mlx5_ib_dev *dev,u32 mkey, +struct mlx5r_umr_wqe *wqe,bool with_data) +{ +struct umr_common *umrc = &dev->umrc; +struct mlx5r_umr_context umr_context; +int err; + + err =umr_check_mkey_mask (dev,be64_to_cpu (wqe->ctrl_seg.mkey_mask)); +if (WARN_ON (err)) +return err; + +mlx5r_umr_init_context (&umr_context); + +down (&umrc->sem); + err =mlx5r_umr_post_send (umrc->qp, mkey, &umr_context.cqe, wqe, + with_data); +if (err) +mlx5_ib_warn (dev,"UMR post send failed, err %d\n" , err); +else { +wait_for_completion (&umr_context.done); +if (umr_context.status !=IB_WC_SUCCESS ) { +mlx5_ib_warn (dev,"reg umr failed (%u)\n" , + umr_context.status); + err = -EFAULT ; + } + } +up (&umrc->sem); +return err; +}
drivers/infiniband/hw/mlx5/umr.h +12 -0
@@ -79,4 +79,16 @@ int mlx5r_umr_set_umr_ctrl_seg(struct mlx5_ib_dev *dev,struct mlx5_wqe_umr_ctrl_seg *umr, const struct ib_send_wr *wr); +struct mlx5r_umr_context { +struct ib_cqe cqe; +enum ib_wc_status status; +struct completion done; +}; + +struct mlx5r_umr_wqe { +struct mlx5_wqe_umr_ctrl_seg ctrl_seg; +struct mlx5_mkey_seg mkey_seg; +struct mlx5_wqe_data_seg data_seg; +}; +#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 1/12 RDMA/mlx5: Move init and cleanup of UMR to umr.c
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
The first patch in a series to split UMR logic to a dedicated file. As a start, move the init and cleanup of UMR resources to umr.c.
--- drivers/infiniband/hw/mlx5/Makefile | 1 + drivers/infiniband/hw/mlx5/main.c | 109 ++-------------------------- drivers/infiniband/hw/mlx5/umr.c | 106 +++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 12 +++ 4 files changed, 125 insertions(+), 103 deletions(-) create mode 100644 drivers/infiniband/hw/mlx5/umr.c create mode 100644 drivers/infiniband/hw/mlx5/umr.h
drivers/infiniband/hw/mlx5/Makefile +1 -0
@@ -19,6 +19,7 @@ mlx5_ib-y := ah.o \ restrack.o \ srq.o \ srq_cmd.o \ + umr.o \ wr.o mlx5_ib-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += odp.o
drivers/infiniband/hw/mlx5/main.c +6 -103
136 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.c new file +106 -0
106 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */ + +#ifndef _MLX5_IB_UMR_H +#define _MLX5_IB_UMR_H + +#include "mlx5_ib.h" + +int mlx5r_umr_resource_init (struct mlx5_ib_dev *dev); +void mlx5r_umr_resource_cleanup (struct mlx5_ib_dev *dev); + +#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 5/12 RDMA/mlx5: Expose wqe posting helpers outside of wr.c
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Split posting WQEs logic to helpers, generalize it and expose for future use in the UMR post send.
--- drivers/infiniband/hw/mlx5/wr.c | 172 +++++++++++--------------------- drivers/infiniband/hw/mlx5/wr.h | 60 +++++++++++ 2 files changed, 121 insertions(+), 111 deletions(-)
drivers/infiniband/hw/mlx5/wr.c +61 -111
@@ -26,58 +26,7 @@ static const u32 mlx5_ib_opcode[] = { [MLX5_IB_WR_UMR] = MLX5_OPCODE_UMR, }; -/* handle_post_send_edge - Check if we get to SQ edge. If yes, update to the -* next nearby edge and get new address translation for current WQE position. -* @sq - SQ buffer. -* @seg: Current WQE position (16B aligned). -* @wqe_sz: Total current WQE size [16B]. -* @cur_edge: Updated current edge. -*/ -static inline void handle_post_send_edge (struct mlx5_ib_wq *sq,void **seg, -u32 wqe_sz,void **cur_edge) -{ -u32 idx; - -if (likely (*seg != *cur_edge)) -return ; - - idx = (sq->cur_post + (wqe_sz >>2 )) & (sq->wqe_cnt -1 ); - *cur_edge =get_sq_edge (sq, idx); - - *seg =mlx5_frag_buf_get_wqe (&sq->fbc, idx); -} - -/* memcpy_send_wqe - copy data from src to WQE and update the relevant WQ's -* pointers. At the end @seg is aligned to 16B regardless the copied size. -* @sq - SQ buffer. -* @cur_edge: Updated current edge. -* @seg: Current WQE position (16B aligned). -* @wqe_sz: Total current WQE size [16B]. -* @src: Pointer to copy from. -* @n: Number of bytes to copy. -*/ -static inline void memcpy_send_wqe (struct mlx5_ib_wq *sq,void **cur_edge, -void **seg,u32 *wqe_sz,const void *src, -size_t n) -{ -while (likely (n)) { -size_t leftlen = *cur_edge - *seg; -size_t copysz =min_t (size_t, leftlen, n); -size_t stride; - -memcpy (*seg, src, copysz); - - n -= copysz; - src += copysz; - stride = !n ?ALIGN (copysz,16 ) : copysz; - *seg += stride; - *wqe_sz += stride >>4 ; -handle_post_send_edge (sq, seg, *wqe_sz, cur_edge); - } -} - -static int mlx5_wq_overflow (struct mlx5_ib_wq *wq,int nreq, -struct ib_cq *ib_cq) +int mlx5r_wq_overflow (struct mlx5_ib_wq *wq,int nreq,struct ib_cq *ib_cq) {struct mlx5_ib_cq *cq;unsigned int cur;
@@ -123,9 +72,9 @@ static void set_eth_seg(const struct ib_send_wr *wr, struct mlx5_ib_qp *qp, eseg->mss =cpu_to_be16 (ud_wr->mss); eseg->inline_hdr.sz =cpu_to_be16 (left); -/* memcpy_send_wqe should get a 16B align address. Hence, we -* first copy up to the current edge and then, if needed, -* continue to memcpy_send_wqe. +/* mlx5r_memcpy_send_wqe should get a 16B align address. Hence, +* we first copy up to the current edge and then, if needed, +* continue to mlx5r_memcpy_send_wqe. */ copysz =min_t (u64, *cur_edge - (void *)eseg->inline_hdr.start, left);
@@ -139,8 +88,8 @@ static void set_eth_seg(const struct ib_send_wr *wr, struct mlx5_ib_qp *qp,handle_post_send_edge (&qp->sq, seg, *size, cur_edge); left -= copysz; pdata += copysz; -memcpy_send_wqe (&qp->sq, cur_edge, seg, size, pdata, - left); +mlx5r_memcpy_send_wqe (&qp->sq, cur_edge, seg, size, + pdata, left); }return ;
@@ -766,8 +715,8 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,handle_post_send_edge (&qp->sq, seg, *size, cur_edge);if (umr_inline) { -memcpy_send_wqe (&qp->sq, cur_edge, seg, size, mr->descs, - mr_list_size); +mlx5r_memcpy_send_wqe (&qp->sq, cur_edge, seg, size, mr->descs, + mr_list_size); *size =ALIGN (*size,MLX5_SEND_WQE_BB >>4 ); }else {set_reg_data_seg (*seg, mr, pd);
@@ -809,23 +758,22 @@ static void dump_wqe(struct mlx5_ib_qp *qp, u32 idx, int size_16) } } -static int __begin_wqe (struct mlx5_ib_qp *qp,void **seg, -struct mlx5_wqe_ctrl_seg **ctrl, -const struct ib_send_wr *wr,unsigned int *idx, -int *size,void **cur_edge,int nreq, -bool send_signaled,bool solicited) +int mlx5r_begin_wqe (struct mlx5_ib_qp *qp,void **seg, +struct mlx5_wqe_ctrl_seg **ctrl,unsigned int *idx, +int *size,void **cur_edge,int nreq,__be32 general_id, +bool send_signaled,bool solicited) { -if (unlikely (mlx5_wq_overflow (&qp->sq, nreq, qp->ibqp.send_cq))) +if (unlikely (mlx5r_wq_overflow (&qp->sq, nreq, qp->ibqp.send_cq)))return -ENOMEM ; *idx = qp->sq.cur_post & (qp->sq.wqe_cnt -1 ); *seg =mlx5_frag_buf_get_wqe (&qp->sq.fbc, *idx); *ctrl = *seg; *(uint32_t *)(*seg +8 ) =0 ; - (*ctrl)->imm =send_ieth (wr); + (*ctrl)->general_id = general_id; (*ctrl)->fm_ce_se = qp->sq_signal_bits | - (send_signaled ?MLX5_WQE_CTRL_CQ_UPDATE :0 ) | - (solicited ?MLX5_WQE_CTRL_SOLICITED :0 ); + (send_signaled ?MLX5_WQE_CTRL_CQ_UPDATE :0 ) | + (solicited ?MLX5_WQE_CTRL_SOLICITED :0 ); *seg +=sizeof (**ctrl); *size =sizeof (**ctrl) /16 ;
@@ -839,16 +787,14 @@ static int begin_wqe(struct mlx5_ib_qp *qp, void **seg, const struct ib_send_wr *wr, unsigned int *idx, int *size, void **cur_edge, int nreq) { - return __begin_wqe(qp, seg, ctrl, wr, idx, size, cur_edge, nreq, - wr->send_flags & IB_SEND_SIGNALED, - wr->send_flags & IB_SEND_SOLICITED); + return mlx5r_begin_wqe(qp, seg, ctrl, idx, size, cur_edge, nreq, + send_ieth(wr), wr->send_flags & IB_SEND_SIGNALED, + wr->send_flags & IB_SEND_SOLICITED); } -static void finish_wqe (struct mlx5_ib_qp *qp, -struct mlx5_wqe_ctrl_seg *ctrl, -void *seg,u8 size,void *cur_edge, -unsigned int idx,u64 wr_id,int nreq,u8 fence, -u32 mlx5_opcode) +void mlx5r_finish_wqe (struct mlx5_ib_qp *qp,struct mlx5_wqe_ctrl_seg *ctrl, +void *seg,u8 size,void *cur_edge,unsigned int idx, +u64 wr_id,int nreq,u8 fence,u32 mlx5_opcode) {u8 opmod =0 ;
@@ -912,8 +858,8 @@ static int handle_psv(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,/* * SET_PSV WQEs are not signaled and solicited on error. */ - err =__begin_wqe (qp, seg, ctrl, wr, idx, size, cur_edge, nreq, - false, true); + err =mlx5r_begin_wqe (qp, seg, ctrl, idx, size, cur_edge, nreq, +send_ieth (wr), false, true);if (unlikely (err)) {mlx5_ib_warn (dev,"\n" ); err = -ENOMEM ;
@@ -924,8 +870,8 @@ static int handle_psv(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,mlx5_ib_warn (dev,"\n" ); goto out; } -finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, wr->wr_id, nreq, - next_fence,MLX5_OPCODE_SET_PSV ); +mlx5r_finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, wr->wr_id, + nreq, next_fence,MLX5_OPCODE_SET_PSV ); out:return err;
@@ -965,8 +911,8 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,if (unlikely (err)) goto out; -finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, wr->wr_id, - nreq, fence,MLX5_OPCODE_UMR ); +mlx5r_finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, + wr->wr_id, nreq, fence,MLX5_OPCODE_UMR ); err =begin_wqe (qp, seg, ctrl, wr, idx, size, cur_edge, nreq);if (unlikely (err)) {
@@ -997,8 +943,8 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,mlx5_ib_warn (dev,"\n" ); goto out; } -finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, wr->wr_id, nreq, - fence,MLX5_OPCODE_UMR ); +mlx5r_finish_wqe (qp, *ctrl, *seg, *size, *cur_edge, *idx, wr->wr_id, + nreq, fence,MLX5_OPCODE_UMR ); sig_attrs = mr->ibmr.sig_attrs; err = handle_psv(dev, qp, wr, ctrl, seg, size, cur_edge, idx, nreq,
@@ -1142,6 +1088,32 @@ static int handle_qpt_reg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,return err; } +void mlx5r_ring_db (struct mlx5_ib_qp *qp,unsigned int nreq, +struct mlx5_wqe_ctrl_seg *ctrl) +{ +struct mlx5_bf *bf = &qp->bf; + + qp->sq.head += nreq; + +/* Make sure that descriptors are written before +* updating doorbell record and ringing the doorbell +*/ +wmb (); + + qp->db.db[MLX5_SND_DBR ] =cpu_to_be32 (qp->sq.cur_post); + +/* Make sure doorbell record is visible to the HCA before +* we hit doorbell. +*/ +wmb (); + +mlx5_write64 ((__be32 *)ctrl, bf->bfreg->map + bf->offset); +/* Make sure doorbells don't leak out of SQ spinlock +* and reach the HCA out of order. +*/ + bf->offset ^= bf->buf_size; +} +int mlx5_ib_post_send (struct ib_qp *ibqp,const struct ib_send_wr *wr,const struct ib_send_wr **bad_wr,bool drain) {
@@ -1150,7 +1122,6 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,struct mlx5_core_dev *mdev = dev->mdev;struct mlx5_ib_qp *qp =to_mqp (ibqp);struct mlx5_wqe_xrc_seg *xrc; -struct mlx5_bf *bf;void *cur_edge;int size;unsigned long flags;
@@ -1172,8 +1143,6 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,if (qp->type ==IB_QPT_GSI )return mlx5_ib_gsi_post_send (ibqp, wr, bad_wr); - bf = &qp->bf; -spin_lock_irqsave (&qp->sq.lock, flags);for (nreq =0 ; wr; nreq++, wr = wr->next) {
@@ -1285,35 +1254,16 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, } qp->next_fence = next_fence; -finish_wqe (qp, ctrl, seg, size, cur_edge, idx, wr->wr_id, nreq, - fence, mlx5_ib_opcode[wr->opcode]); +mlx5r_finish_wqe (qp, ctrl, seg, size, cur_edge, idx, wr->wr_id, + nreq, fence, mlx5_ib_opcode[wr->opcode]); skip_psv:if (0 )dump_wqe (qp, idx, size); } out: -if (likely (nreq)) { - qp->sq.head += nreq; - -/* Make sure that descriptors are written before -* updating doorbell record and ringing the doorbell -*/ -wmb (); - - qp->db.db[MLX5_SND_DBR ] =cpu_to_be32 (qp->sq.cur_post); - -/* Make sure doorbell record is visible to the HCA before -* we hit doorbell. -*/ -wmb (); - -mlx5_write64 ((__be32 *)ctrl, bf->bfreg->map + bf->offset); -/* Make sure doorbells don't leak out of SQ spinlock -* and reach the HCA out of order. -*/ - bf->offset ^= bf->buf_size; - } +if (likely (nreq)) +mlx5r_ring_db (qp, nreq, ctrl);spin_unlock_irqrestore (&qp->sq.lock, flags);
@@ -1353,7 +1303,7 @@ int mlx5_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, ind = qp->rq.head & (qp->rq.wqe_cnt -1 );for (nreq =0 ; wr; nreq++, wr = wr->next) { -if (mlx5_wq_overflow (&qp->rq, nreq, qp->ibqp.recv_cq)) { +if (mlx5r_wq_overflow (&qp->rq, nreq, qp->ibqp.recv_cq)) { err = -ENOMEM ; *bad_wr = wr; goto out;
drivers/infiniband/hw/mlx5/wr.h +60 -0
66 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH 7/12 RDMA/mlx5: Use mlx5_umr_post_send_wait() to revoke MRs
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move the revoke_mr logic to umr.c, and using mlx5_umr_post_send_wait() instead of mlx5_ib_post_send_wait().
In the new implementation, do not zero out the access flags. Before reusing the MR, we will update it to the required access.
--- drivers/infiniband/hw/mlx5/mr.c | 31 +++---------------------------- drivers/infiniband/hw/mlx5/umr.c | 29 +++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 2 ++ 3 files changed, 34 insertions(+), 28 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +3 -28
@@ -1629,31 +1629,6 @@ struct ib_mr *mlx5_ib_reg_user_mr_dmabuf(struct ib_pd *pd, u64 offset,return ERR_PTR (err); } -/** -* revoke_mr - Fence all DMA on the MR -* @mr: The MR to fence -* -* Upon return the NIC will not be doing any DMA to the pages under the MR, -* and any DMA in progress will be completed. Failure of this function -* indicates the HW has failed catastrophically. -*/ -static int revoke_mr (struct mlx5_ib_mr *mr) -{ -struct mlx5_umr_wr umrwr = {}; - -if (mr_to_mdev (mr)->mdev->state ==MLX5_DEVICE_STATE_INTERNAL_ERROR ) -return 0 ; - - umrwr.wr.send_flags =MLX5_IB_SEND_UMR_DISABLE_MR | -MLX5_IB_SEND_UMR_UPDATE_PD_ACCESS ; - umrwr.wr.opcode =MLX5_IB_WR_UMR ; - umrwr.pd =mr_to_mdev (mr)->umrc.pd; - umrwr.mkey = mr->mmkey.key; - umrwr.ignore_free_state =1 ; - -return mlx5_ib_post_send_wait (mr_to_mdev (mr), &umrwr); -} -/* * True if the change in access flags can be done via UMR, only some access * flags can be updated.
@@ -1730,7 +1705,7 @@ static int umr_rereg_pas(struct mlx5_ib_mr *mr, struct ib_pd *pd,* with it. This ensure the change is atomic relative to any use of the * MR. */ - err =revoke_mr (mr); + err =mlx5r_umr_revoke_mr (mr);if (err)return err;
@@ -1808,7 +1783,7 @@ struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,* Only one active MR can refer to a umem at one time, revoke * the old MR before assigning the umem to the new one. */ - err =revoke_mr (mr); + err =mlx5r_umr_revoke_mr (mr);if (err)return ERR_PTR (err); umem = mr->umem;
@@ -1953,7 +1928,7 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)/* Stop DMA */ if (mr->cache_ent) { -if (revoke_mr (mr)) { +if (mlx5r_umr_revoke_mr (mr)) {spin_lock_irq (&mr->cache_ent->lock); mr->cache_ent->total_mrs--;spin_unlock_irq (&mr->cache_ent->lock);
drivers/infiniband/hw/mlx5/umr.c +29 -0
@@ -320,3 +320,32 @@ static int mlx5r_umr_post_send_wait(struct mlx5_ib_dev *dev, u32 mkey,up (&umrc->sem);return err; } + +/** +* mlx5r_umr_revoke_mr - Fence all DMA on the MR +* @mr: The MR to fence +* +* Upon return the NIC will not be doing any DMA to the pages under the MR, +* and any DMA in progress will be completed. Failure of this function +* indicates the HW has failed catastrophically. +*/ +int mlx5r_umr_revoke_mr (struct mlx5_ib_mr *mr) +{ +struct mlx5_ib_dev *dev =mr_to_mdev (mr); +struct mlx5r_umr_wqe wqe = {}; + +if (dev->mdev->state ==MLX5_DEVICE_STATE_INTERNAL_ERROR ) +return 0 ; + + wqe.ctrl_seg.mkey_mask |=get_umr_update_pd_mask (); + wqe.ctrl_seg.mkey_mask |=get_umr_disable_mr_mask (); + wqe.ctrl_seg.flags |=MLX5_UMR_INLINE ; + +MLX5_SET (mkc, &wqe.mkey_seg, free,1 ); +MLX5_SET (mkc, &wqe.mkey_seg, pd,to_mpd (dev->umrc.pd)->pdn); +MLX5_SET (mkc, &wqe.mkey_seg, qpn,0xffffff ); +MLX5_SET (mkc, &wqe.mkey_seg, mkey_7_0, +mlx5_mkey_variant (mr->mmkey.key)); + +return mlx5r_umr_post_send_wait (dev, mr->mmkey.key, &wqe, false); +}
drivers/infiniband/hw/mlx5/umr.h +2 -0
@@ -91,4 +91,6 @@ struct mlx5r_umr_wqe { struct mlx5_wqe_data_seg data_seg; }; +int mlx5r_umr_revoke_mr (struct mlx5_ib_mr *mr); +#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 8/12 RDMA/mlx5: Use mlx5_umr_post_send_wait() to rereg pd access
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move rereg_pd_access logic to umr.c, and use mlx5_umr_post_send_wait() instead of mlx5_ib_post_send_wait().
--- drivers/infiniband/hw/mlx5/mr.c | 27 ++------------------- drivers/infiniband/hw/mlx5/umr.c | 41 ++++++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 2 ++ 3 files changed, 45 insertions(+), 25 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +2 -25
@@ -1646,30 +1646,6 @@ static bool can_use_umr_rereg_access(struct mlx5_ib_dev *dev, target_access_flags); } -static int umr_rereg_pd_access (struct mlx5_ib_mr *mr,struct ib_pd *pd, -int access_flags) -{ -struct mlx5_ib_dev *dev =to_mdev (mr->ibmr.device); -struct mlx5_umr_wr umrwr = { - .wr = { - .send_flags =MLX5_IB_SEND_UMR_FAIL_IF_FREE | -MLX5_IB_SEND_UMR_UPDATE_PD_ACCESS , - .opcode =MLX5_IB_WR_UMR , - }, - .mkey = mr->mmkey.key, - .pd = pd, - .access_flags = access_flags, - }; -int err; - - err =mlx5_ib_post_send_wait (dev, &umrwr); -if (err) -return err; - - mr->access_flags = access_flags; -return 0 ; -} - static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr, struct ib_umem *new_umem, int new_access_flags, u64 iova,
@@ -1770,7 +1746,8 @@ struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,/* Fast path for PD/access change */ if (can_use_umr_rereg_access (dev, mr->access_flags, new_access_flags)) { - err =umr_rereg_pd_access (mr, new_pd, new_access_flags); + err =mlx5r_umr_rereg_pd_access (mr, new_pd, + new_access_flags);if (err)return ERR_PTR (err);return NULL ;
drivers/infiniband/hw/mlx5/umr.c +41 -0
44 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +2 -0
@@ -92,5 +92,7 @@ struct mlx5r_umr_wqe { };int mlx5r_umr_revoke_mr (struct mlx5_ib_mr *mr); +int mlx5r_umr_rereg_pd_access (struct mlx5_ib_mr *mr,struct ib_pd *pd, +int access_flags);#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 2/12 RDMA/mlx5: Move umr checks to umr.h
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move mlx5_ib_can_load_pas_with_umr() and mlx5_ib_can_reconfig_with_umr() to umr.h and rename them accordingly.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 56 --------------------------- drivers/infiniband/hw/mlx5/mr.c | 19 +++++----- drivers/infiniband/hw/mlx5/odp.c | 7 ++-- drivers/infiniband/hw/mlx5/umr.h | 57 ++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/wr.c | 3 +- 5 files changed, 72 insertions(+), 70 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -56
@@ -1471,9 +1471,6 @@ static inline int is_qp1(enum ib_qp_type qp_type)return qp_type ==MLX5_IB_QPT_HW_GSI || qp_type ==IB_QPT_GSI ; } -#define MLX5_MAX_UMR_SHIFT 16 -#define MLX5_MAX_UMR_PAGES (1 << MLX5_MAX_UMR_SHIFT) -static inline u32 check_cq_create_flags (u32 flags) {/*
@@ -1545,59 +1542,6 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi, u32 bfregn, bool dyn_bfreg); -static inline bool mlx5_ib_can_load_pas_with_umr (struct mlx5_ib_dev *dev, -size_t length) -{ -/* -* umr_check_mkey_mask() rejects MLX5_MKEY_MASK_PAGE_SIZE which is -* always set if MLX5_IB_SEND_UMR_UPDATE_TRANSLATION (aka -* MLX5_IB_UPD_XLT_ADDR and MLX5_IB_UPD_XLT_ENABLE) is set. Thus, a mkey -* can never be enabled without this capability. Simplify this weird -* quirky hardware by just saying it can't use PAS lists with UMR at -* all. -*/ -if (MLX5_CAP_GEN (dev->mdev, umr_modify_entity_size_disabled)) -return false; - -/* -* length is the size of the MR in bytes when mlx5_ib_update_xlt() is -* used. -*/ -if (!MLX5_CAP_GEN (dev->mdev, umr_extended_translation_offset) && - length >=MLX5_MAX_UMR_PAGES *PAGE_SIZE ) -return false; -return true; -} - -/* -* true if an existing MR can be reconfigured to new access_flags using UMR. -* Older HW cannot use UMR to update certain elements of the MKC. See -* umr_check_mkey_mask(), get_umr_update_access_mask() and umr_check_mkey_mask() -*/ -static inline bool mlx5_ib_can_reconfig_with_umr (struct mlx5_ib_dev *dev, -unsigned int current_access_flags, -unsigned int target_access_flags) -{ -unsigned int diffs = current_access_flags ^ target_access_flags; - -if ((diffs &IB_ACCESS_REMOTE_ATOMIC ) && -MLX5_CAP_GEN (dev->mdev, atomic) && -MLX5_CAP_GEN (dev->mdev, umr_modify_atomic_disabled)) -return false; - -if ((diffs &IB_ACCESS_RELAXED_ORDERING ) && -MLX5_CAP_GEN (dev->mdev, relaxed_ordering_write) && - !MLX5_CAP_GEN (dev->mdev, relaxed_ordering_write_umr)) -return false; - -if ((diffs &IB_ACCESS_RELAXED_ORDERING ) && -MLX5_CAP_GEN (dev->mdev, relaxed_ordering_read) && - !MLX5_CAP_GEN (dev->mdev, relaxed_ordering_read_umr)) -return false; - -return true; -} -static inline int mlx5r_store_odp_mkey (struct mlx5_ib_dev *dev,struct mlx5_ib_mkey *mmkey) {
drivers/infiniband/hw/mlx5/mr.c +10 -9
@@ -44,6 +44,7 @@#include <rdma/ib_verbs.h> #include "dm.h" #include "mlx5_ib.h" +#include "umr.h" /* * We can't use an array for xlt_emergency_page because dma_map_single doesn't
@@ -598,7 +599,7 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,struct mlx5_ib_mr *mr;/* Matches access in alloc_cache_mr() */ -if (!mlx5_ib_can_reconfig_with_umr (dev,0 , access_flags)) +if (!mlx5r_umr_can_reconfig (dev,0 , access_flags))return ERR_PTR (-EOPNOTSUPP );spin_lock_irq (&ent->lock);
@@ -738,7 +739,7 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev) ent->access_mode =MLX5_MKC_ACCESS_MODE_MTT ;if ((dev->mdev->profile.mask &MLX5_PROF_MASK_MR_CACHE ) && !dev->is_rep &&mlx5_core_is_pf (dev->mdev) && -mlx5_ib_can_load_pas_with_umr (dev,0 )) +mlx5r_umr_can_load_pas (dev,0 )) ent->limit = dev->mdev->profile.mr_cache[i].limit;else ent->limit =0 ;
@@ -946,7 +947,7 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,* cache then synchronously create an uncached one. */ if (!ent || ent->limit ==0 || - !mlx5_ib_can_reconfig_with_umr (dev,0 , access_flags)) { + !mlx5r_umr_can_reconfig (dev,0 , access_flags)) {mutex_lock (&dev->slow_path_mutex); mr =reg_create (pd, umem, iova, access_flags, page_size, false);mutex_unlock (&dev->slow_path_mutex);
@@ -1438,7 +1439,7 @@ static struct ib_mr *create_real_mr(struct ib_pd *pd, struct ib_umem *umem,bool xlt_with_umr;int err; - xlt_with_umr =mlx5_ib_can_load_pas_with_umr (dev, umem->length); + xlt_with_umr =mlx5r_umr_can_load_pas (dev, umem->length);if (xlt_with_umr) { mr =alloc_cacheable_mr (pd, umem, iova, access_flags); }else {
@@ -1501,7 +1502,7 @@ static struct ib_mr *create_user_odp_mr(struct ib_pd *pd, u64 start, u64 length, }/* ODP requires xlt update via umr to work. */ -if (!mlx5_ib_can_load_pas_with_umr (dev, length)) +if (!mlx5r_umr_can_load_pas (dev, length))return ERR_PTR (-EINVAL ); odp = ib_umem_odp_get(&dev->ib_dev, start, length, access_flags,
@@ -1591,7 +1592,7 @@ struct ib_mr *mlx5_ib_reg_user_mr_dmabuf(struct ib_pd *pd, u64 offset, offset, virt_addr, length, fd, access_flags);/* dmabuf requires xlt update via umr to work. */ -if (!mlx5_ib_can_load_pas_with_umr (dev, length)) +if (!mlx5r_umr_can_load_pas (dev, length))return ERR_PTR (-EINVAL ); umem_dmabuf = ib_umem_dmabuf_get(&dev->ib_dev, offset, length, fd,
@@ -1666,8 +1667,8 @@ static bool can_use_umr_rereg_access(struct mlx5_ib_dev *dev,if (diffs & ~(IB_ACCESS_LOCAL_WRITE |IB_ACCESS_REMOTE_WRITE |IB_ACCESS_REMOTE_READ |IB_ACCESS_RELAXED_ORDERING ))return false; -return mlx5_ib_can_reconfig_with_umr (dev, current_access_flags, - target_access_flags); +return mlx5r_umr_can_reconfig (dev, current_access_flags, + target_access_flags); } static int umr_rereg_pd_access(struct mlx5_ib_mr *mr, struct ib_pd *pd,
@@ -1704,7 +1705,7 @@ static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr,/* We only track the allocated sizes of MRs from the cache */ if (!mr->cache_ent)return false; -if (!mlx5_ib_can_load_pas_with_umr (dev, new_umem->length)) +if (!mlx5r_umr_can_load_pas (dev, new_umem->length))return false; *page_size =
drivers/infiniband/hw/mlx5/odp.c +3 -4
@@ -38,6 +38,7 @@#include "mlx5_ib.h" #include "cmd.h" +#include "umr.h" #include "qp.h" #include <linux/mlx5/eq.h>
@@ -323,8 +324,7 @@ static void internal_fill_odp_caps(struct mlx5_ib_dev *dev)memset (caps,0 ,sizeof (*caps)); -if (!MLX5_CAP_GEN (dev->mdev, pg) || - !mlx5_ib_can_load_pas_with_umr (dev,0 )) +if (!MLX5_CAP_GEN (dev->mdev, pg) || !mlx5r_umr_can_load_pas (dev,0 ))return ; caps->general_caps =IB_ODP_SUPPORT ;
@@ -487,8 +487,7 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,struct mlx5_ib_mr *imr;int err; -if (!mlx5_ib_can_load_pas_with_umr (dev, -MLX5_IMR_MTT_ENTRIES *PAGE_SIZE )) +if (!mlx5r_umr_can_load_pas (dev,MLX5_IMR_MTT_ENTRIES *PAGE_SIZE ))return ERR_PTR (-EOPNOTSUPP ); umem_odp =ib_umem_odp_alloc_implicit (&dev->ib_dev, access_flags);
drivers/infiniband/hw/mlx5/umr.h +57 -0
64 lines not on this page · raw message
drivers/infiniband/hw/mlx5/wr.c +2 -1
@@ -7,6 +7,7 @@#include <linux/mlx5/qp.h> #include <linux/mlx5/driver.h> #include "wr.h" +#include "umr.h" static const u32 mlx5_ib_opcode[] = { [IB_WR_SEND ] =MLX5_OPCODE_SEND ,
@@ -870,7 +871,7 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,* Relaxed Ordering is set implicitly in mlx5_set_umr_free_mkey() and * kernel ULPs are not aware of it, so we don't set it here. */ -if (!mlx5_ib_can_reconfig_with_umr (dev,0 , wr->access)) { +if (!mlx5r_umr_can_reconfig (dev,0 , wr->access)) {mlx5_ib_warn (to_mdev (qp->ibqp.device),"Fast update for MR access flags is not possible\n" );
Signature · 1 line
2.35.1
PATCH 3/12 RDMA/mlx5: Move mkey ctrl segment logic to umr.c
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move set_reg_umr_segment() and its helpers to umr.c.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 - drivers/infiniband/hw/mlx5/qp.c | 1 + drivers/infiniband/hw/mlx5/umr.c | 129 ++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 13 +++ drivers/infiniband/hw/mlx5/wr.c | 142 +-------------------------- 5 files changed, 147 insertions(+), 141 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -3
@@ -311,9 +311,6 @@ struct mlx5_ib_flow_db {#define MLX5_IB_QPT_DCT IB_QPT_RESERVED4#define MLX5_IB_WR_UMR IB_WR_RESERVED1 -#define MLX5_IB_UMR_OCTOWORD 16 -#define MLX5_IB_UMR_XLT_ALIGNMENT 64 -#define MLX5_IB_UPD_XLT_ZAP BIT(0)#define MLX5_IB_UPD_XLT_ENABLE BIT(1)#define MLX5_IB_UPD_XLT_ATOMIC BIT(2)
drivers/infiniband/hw/mlx5/qp.c +1 -0
@@ -40,6 +40,7 @@#include "ib_rep.h" #include "counters.h" #include "cmd.h" +#include "umr.h" #include "qp.h" #include "wr.h"
drivers/infiniband/hw/mlx5/umr.c +129 -0
135 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +13 -0
@@ -10,6 +10,9 @@#define MLX5_MAX_UMR_SHIFT 16#define MLX5_MAX_UMR_PAGES (1 << MLX5_MAX_UMR_SHIFT) +#define MLX5_IB_UMR_OCTOWORD 16 +#define MLX5_IB_UMR_XLT_ALIGNMENT 64 +int mlx5r_umr_resource_init (struct mlx5_ib_dev *dev);void mlx5r_umr_resource_cleanup (struct mlx5_ib_dev *dev);
@@ -66,4 +69,14 @@ static inline bool mlx5r_umr_can_reconfig(struct mlx5_ib_dev *dev, return true; } +static inline u64 mlx5r_umr_get_xlt_octo (u64 bytes) +{ +return ALIGN (bytes,MLX5_IB_UMR_XLT_ALIGNMENT ) / +MLX5_IB_UMR_OCTOWORD ; +} + +int mlx5r_umr_set_umr_ctrl_seg (struct mlx5_ib_dev *dev, +struct mlx5_wqe_umr_ctrl_seg *umr, +const struct ib_send_wr *wr); +#endif /* _MLX5_IB_UMR_H */
drivers/infiniband/hw/mlx5/wr.c +4 -138
178 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH 4/12 RDMA/mlx5: Simplify get_umr_update_access_mask()
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Instead of getting the update access capabilities each call to get_umr_update_access_mask(), pass struct mlx5_ib_dev and get the capabilities inside the function.
--- drivers/infiniband/hw/mlx5/umr.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/umr.c +5 -10
@@ -34,9 +34,7 @@ static __be64 get_umr_update_translation_mask(void)return cpu_to_be64 (result); } -static __be64 get_umr_update_access_mask (int atomic, -int relaxed_ordering_write, -int relaxed_ordering_read) +static __be64 get_umr_update_access_mask (struct mlx5_ib_dev *dev) {u64 result;
@@ -45,13 +43,13 @@ static __be64 get_umr_update_access_mask(int atomic,MLX5_MKEY_MASK_RR |MLX5_MKEY_MASK_RW ; -if (atomic) +if (MLX5_CAP_GEN (dev->mdev, atomic)) result |=MLX5_MKEY_MASK_A ; -if (relaxed_ordering_write) +if (MLX5_CAP_GEN (dev->mdev, relaxed_ordering_write_umr)) result |=MLX5_MKEY_MASK_RELAXED_ORDERING_WRITE ; -if (relaxed_ordering_read) +if (MLX5_CAP_GEN (dev->mdev, relaxed_ordering_read_umr)) result |=MLX5_MKEY_MASK_RELAXED_ORDERING_READ ;return cpu_to_be64 (result);
@@ -116,10 +114,7 @@ int mlx5r_umr_set_umr_ctrl_seg(struct mlx5_ib_dev *dev,if (wr->send_flags &MLX5_IB_SEND_UMR_UPDATE_TRANSLATION ) umr->mkey_mask |=get_umr_update_translation_mask ();if (wr->send_flags &MLX5_IB_SEND_UMR_UPDATE_PD_ACCESS ) { - umr->mkey_mask |=get_umr_update_access_mask ( - !!MLX5_CAP_GEN (dev->mdev, atomic), - !!MLX5_CAP_GEN (dev->mdev, relaxed_ordering_write_umr), - !!MLX5_CAP_GEN (dev->mdev, relaxed_ordering_read_umr)); + umr->mkey_mask |=get_umr_update_access_mask (dev); umr->mkey_mask |=get_umr_update_pd_mask (); }if (wr->send_flags &MLX5_IB_SEND_UMR_ENABLE_MR )
Signature · 1 line
2.35.1
PATCH 9/12 RDMA/mlx5: Move creation and free of translation tables to umr.c
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
The only use of the translation tables is to update the mkey translation by a UMR operation. Move the responsibility of creating and freeing them to umr.c
--- drivers/infiniband/hw/mlx5/mr.c | 103 +---------------------------- drivers/infiniband/hw/mlx5/umr.c | 109 +++++++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 5 ++ 3 files changed, 117 insertions(+), 100 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +3 -100
@@ -46,13 +46,6 @@#include "mlx5_ib.h" #include "umr.h" -/* -* We can't use an array for xlt_emergency_page because dma_map_single doesn't -* work on kernel modules memory -*/ -void *xlt_emergency_page; -static DEFINE_MUTEX (xlt_emergency_page_mutex); -enum {MAX_PENDING_REG_MR =8 , };
@@ -966,74 +959,6 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,return mr; } -#define MLX5_MAX_UMR_CHUNK ((1 << (MLX5_MAX_UMR_SHIFT + 4)) - \ - MLX5_UMR_MTT_ALIGNMENT) -#define MLX5_SPARE_UMR_CHUNK 0x10000 - -/* -* Allocate a temporary buffer to hold the per-page information to transfer to -* HW. For efficiency this should be as large as it can be, but buffer -* allocation failure is not allowed, so try smaller sizes. -*/ -static void *mlx5_ib_alloc_xlt (size_t *nents,size_t ent_size,gfp_t gfp_mask) -{ -const size_t xlt_chunk_align = -MLX5_UMR_MTT_ALIGNMENT / ent_size; -size_t size; -void *res =NULL ; - -static_assert (PAGE_SIZE %MLX5_UMR_MTT_ALIGNMENT ==0 ); - -/* -* MLX5_IB_UPD_XLT_ATOMIC doesn't signal an atomic context just that the -* allocation can't trigger any kind of reclaim. -*/ -might_sleep (); - - gfp_mask |= __GFP_ZERO | __GFP_NORETRY; - -/* -* If the system already has a suitable high order page then just use -* that, but don't try hard to create one. This max is about 1M, so a -* free x86 huge page will satisfy it. -*/ - size =min_t (size_t, ent_size *ALIGN (*nents, xlt_chunk_align), -MLX5_MAX_UMR_CHUNK ); - *nents = size / ent_size; - res = (void *)__get_free_pages (gfp_mask | __GFP_NOWARN, -get_order (size)); -if (res) -return res; - -if (size >MLX5_SPARE_UMR_CHUNK ) { - size =MLX5_SPARE_UMR_CHUNK ; - *nents = size / ent_size; - res = (void *)__get_free_pages (gfp_mask | __GFP_NOWARN, -get_order (size)); -if (res) -return res; - } - - *nents =PAGE_SIZE / ent_size; - res = (void *)__get_free_page (gfp_mask); -if (res) -return res; - -mutex_lock (&xlt_emergency_page_mutex); -memset (xlt_emergency_page,0 ,PAGE_SIZE ); -return xlt_emergency_page; -} - -static void mlx5_ib_free_xlt (void *xlt,size_t length) -{ -if (xlt == xlt_emergency_page) { -mutex_unlock (&xlt_emergency_page_mutex); -return ; - } - -free_pages ((unsigned long )xlt,get_order (length)); -} -/* * Create a MLX5_IB_SEND_UMR_UPDATE_XLT work request and XLT buffer ready for * submission.
@@ -1044,22 +969,9 @@ static void *mlx5_ib_create_xlt_wr(struct mlx5_ib_mr *mr, unsigned int flags) {struct mlx5_ib_dev *dev =mr_to_mdev (mr); -struct device *ddev = &dev->mdev->pdev->dev; -dma_addr_t dma;void *xlt; - xlt =mlx5_ib_alloc_xlt (&nents, ent_size, - flags &MLX5_IB_UPD_XLT_ATOMIC ?GFP_ATOMIC : -GFP_KERNEL ); - sg->length = nents * ent_size; - dma =dma_map_single (ddev, xlt, sg->length,DMA_TO_DEVICE ); -if (dma_mapping_error (ddev, dma)) { -mlx5_ib_err (dev,"unable to map DMA during XLT update.\n" ); -mlx5_ib_free_xlt (xlt, sg->length); -return NULL ; - } - sg->addr = dma; - sg->lkey = dev->umrc.pd->local_dma_lkey; + xlt =mlx5r_umr_create_xlt (dev, sg, nents, ent_size, flags);memset (wr,0 ,sizeof (*wr)); wr->wr.send_flags =MLX5_IB_SEND_UMR_UPDATE_XLT ;
@@ -1078,15 +990,6 @@ static void *mlx5_ib_create_xlt_wr(struct mlx5_ib_mr *mr,return xlt; } -static void mlx5_ib_unmap_free_xlt (struct mlx5_ib_dev *dev,void *xlt, -struct ib_sge *sg) -{ -struct device *ddev = &dev->mdev->pdev->dev; - -dma_unmap_single (ddev, sg->addr, sg->length,DMA_TO_DEVICE ); -mlx5_ib_free_xlt (xlt, sg->length); -} -static unsigned int xlt_wr_final_send_flags (unsigned int flags) {unsigned int res =0 ;
@@ -1175,7 +1078,7 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, err =mlx5_ib_post_send_wait (dev, &wr); } sg.length = orig_sg_length; -mlx5_ib_unmap_free_xlt (dev, xlt, &sg); +mlx5r_umr_unmap_free_xlt (dev, xlt, &sg);return err; }
@@ -1245,7 +1148,7 @@ int mlx5_ib_update_mr_pas(struct mlx5_ib_mr *mr, unsigned int flags) err: sg.length = orig_sg_length; -mlx5_ib_unmap_free_xlt (dev, mtt, &sg); +mlx5r_umr_unmap_free_xlt (dev, mtt, &sg);return err; }
drivers/infiniband/hw/mlx5/umr.c +109 -0
118 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +5 -0
@@ -94,5 +94,10 @@ struct mlx5r_umr_wqe {int mlx5r_umr_revoke_mr (struct mlx5_ib_mr *mr);int mlx5r_umr_rereg_pd_access (struct mlx5_ib_mr *mr,struct ib_pd *pd,int access_flags); +void *mlx5r_umr_create_xlt (struct mlx5_ib_dev *dev,struct ib_sge *sg, +size_t nents,size_t ent_size,unsigned int flags); +void mlx5r_umr_free_xlt (void *xlt,size_t length); +void mlx5r_umr_unmap_free_xlt (struct mlx5_ib_dev *dev,void *xlt, +struct ib_sge *sg);#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 10/12 RDMA/mlx5: Use mlx5_umr_post_send_wait() to update MR pas
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move mlx5_ib_update_mr_pas logic to umr.c, and use mlx5_umr_post_send_wait() instead of mlx5_ib_post_send_wait().
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 - drivers/infiniband/hw/mlx5/mr.c | 76 +------------ drivers/infiniband/hw/mlx5/odp.c | 2 +- drivers/infiniband/hw/mlx5/umr.c | 159 +++++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 1 + 5 files changed, 164 insertions(+), 75 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -1
@@ -1289,7 +1289,6 @@ int mlx5_ib_alloc_mw(struct ib_mw *mw, struct ib_udata *udata);int mlx5_ib_dealloc_mw (struct ib_mw *mw);int mlx5_ib_update_xlt (struct mlx5_ib_mr *mr,u64 idx,int npages,int page_shift,int flags); -int mlx5_ib_update_mr_pas (struct mlx5_ib_mr *mr,unsigned int flags);struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr (struct mlx5_ib_pd *pd,int access_flags);void mlx5_ib_free_implicit_mr (struct mlx5_ib_mr *mr);
drivers/infiniband/hw/mlx5/mr.c +3 -73
100 lines not on this page · raw message
drivers/infiniband/hw/mlx5/odp.c +1 -1
@@ -715,7 +715,7 @@ static int pagefault_dmabuf_mr(struct mlx5_ib_mr *mr, size_t bcnt,ib_umem_dmabuf_unmap_pages (umem_dmabuf); err = -EINVAL ; }else { - err =mlx5_ib_update_mr_pas (mr, xlt_flags); + err =mlx5r_umr_update_mr_pas (mr, xlt_flags); }dma_resv_unlock (umem_dmabuf->attach->dmabuf->resv);
drivers/infiniband/hw/mlx5/umr.c +159 -0
162 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +1 -0
@@ -99,5 +99,6 @@ void *mlx5r_umr_create_xlt(struct mlx5_ib_dev *dev, struct ib_sge *sg,void mlx5r_umr_free_xlt (void *xlt,size_t length);void mlx5r_umr_unmap_free_xlt (struct mlx5_ib_dev *dev,void *xlt,struct ib_sge *sg); +int mlx5r_umr_update_mr_pas (struct mlx5_ib_mr *mr,unsigned int flags);#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 11/12 RDMA/mlx5: Use mlx5_umr_post_send_wait() to update xlt
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
Move mlx5_ib_update_mr_pas logic to umr.c, and use mlx5_umr_post_send_wait() instead of mlx5_ib_post_send_wait().
Since it is the last use of mlx5_ib_post_send_wait(), remove it.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 - drivers/infiniband/hw/mlx5/mr.c | 171 --------------------------- drivers/infiniband/hw/mlx5/odp.c | 54 ++++----- drivers/infiniband/hw/mlx5/umr.c | 93 ++++++++++++++- drivers/infiniband/hw/mlx5/umr.h | 7 +- 5 files changed, 117 insertions(+), 210 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -2
@@ -1287,8 +1287,6 @@ int mlx5_ib_advise_mr(struct ib_pd *pd, struct uverbs_attr_bundle *attrs);int mlx5_ib_alloc_mw (struct ib_mw *mw,struct ib_udata *udata);int mlx5_ib_dealloc_mw (struct ib_mw *mw); -int mlx5_ib_update_xlt (struct mlx5_ib_mr *mr,u64 idx,int npages, -int page_shift,int flags);struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr (struct mlx5_ib_pd *pd,int access_flags);void mlx5_ib_free_implicit_mr (struct mlx5_ib_mr *mr);
drivers/infiniband/hw/mlx5/mr.c +0 -171
189 lines not on this page · raw message
drivers/infiniband/hw/mlx5/odp.c +27 -27
101 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.c +88 -5
120 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +2 -5
@@ -94,11 +94,8 @@ struct mlx5r_umr_wqe {int mlx5r_umr_revoke_mr (struct mlx5_ib_mr *mr);int mlx5r_umr_rereg_pd_access (struct mlx5_ib_mr *mr,struct ib_pd *pd,int access_flags); -void *mlx5r_umr_create_xlt (struct mlx5_ib_dev *dev,struct ib_sge *sg, -size_t nents,size_t ent_size,unsigned int flags); -void mlx5r_umr_free_xlt (void *xlt,size_t length); -void mlx5r_umr_unmap_free_xlt (struct mlx5_ib_dev *dev,void *xlt, -struct ib_sge *sg);int mlx5r_umr_update_mr_pas (struct mlx5_ib_mr *mr,unsigned int flags); +int mlx5r_umr_update_xlt (struct mlx5_ib_mr *mr,u64 idx,int npages, +int page_shift,int flags);#endif /* _MLX5_IB_UMR_H */
Signature · 1 line
2.35.1
PATCH 12/12 RDMA/mlx5: Clean UMR QP type flow from mlx5_ib_post_send()
In reply to Leon RomanovskyFrom: Aharon Landau <[email protected]>
No internal UMR operation is using mlx5_ib_post_send(), remove the UMR QP type logic from this function.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 27 +---------- drivers/infiniband/hw/mlx5/umr.c | 43 ----------------- drivers/infiniband/hw/mlx5/umr.h | 4 -- drivers/infiniband/hw/mlx5/wr.c | 72 ---------------------------- 4 files changed, 1 insertion(+), 145 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -26
@@ -291,16 +291,9 @@ struct mlx5_ib_flow_db { };/* Use macros here so that don't have to duplicate -* enum ib_send_flags and enum ib_qp_type for low-level driver +* enum ib_qp_type for low-level driver */ -#define MLX5_IB_SEND_UMR_ENABLE_MR (IB_SEND_RESERVED_START << 0) -#define MLX5_IB_SEND_UMR_DISABLE_MR (IB_SEND_RESERVED_START << 1) -#define MLX5_IB_SEND_UMR_FAIL_IF_FREE (IB_SEND_RESERVED_START << 2) -#define MLX5_IB_SEND_UMR_UPDATE_XLT (IB_SEND_RESERVED_START << 3) -#define MLX5_IB_SEND_UMR_UPDATE_TRANSLATION (IB_SEND_RESERVED_START << 4) -#define MLX5_IB_SEND_UMR_UPDATE_PD_ACCESS IB_SEND_RESERVED_END -#define MLX5_IB_QPT_REG_UMR IB_QPT_RESERVED1/* * IB_QPT_GSI creates the software wrapper around GSI, and MLX5_IB_QPT_HW_GSI
@@ -536,24 +529,6 @@ struct mlx5_ib_cq_buf {int nent; }; -struct mlx5_umr_wr { -struct ib_send_wr wr; -u64 virt_addr; -u64 offset; -struct ib_pd *pd; -unsigned int page_shift; -unsigned int xlt_size; -u64 length; -int access_flags; -u32 mkey; -u8 ignore_free_state:1 ; -}; - -static inline const struct mlx5_umr_wr *umr_wr (const struct ib_send_wr *wr) -{ - return container_of(wr, struct mlx5_umr_wr, wr); -} -enum mlx5_ib_cq_pr_flags {MLX5_IB_CQ_PR_FLAGS_CQE_128_PAD =1 <<0 ,MLX5_IB_CQ_PR_FLAGS_REAL_TIME_TS =1 <<1 ,
drivers/infiniband/hw/mlx5/umr.c +0 -43
49 lines not on this page · raw message
drivers/infiniband/hw/mlx5/umr.h +0 -4
@@ -75,10 +75,6 @@ static inline u64 mlx5r_umr_get_xlt_octo(u64 bytes)MLX5_IB_UMR_OCTOWORD ; } -int mlx5r_umr_set_umr_ctrl_seg (struct mlx5_ib_dev *dev, -struct mlx5_wqe_umr_ctrl_seg *umr, -const struct ib_send_wr *wr); -struct mlx5r_umr_context {struct ib_cqe cqe;enum ib_wc_status status;
drivers/infiniband/hw/mlx5/wr.c +0 -72
90 lines not on this page · raw message
Signature · 1 line
2.35.1
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 09/12] RDMA/mlx5: Move creation and free of translation tables to umr.c
In reply to Leon RomanovskyReview
Quoted 2 lines · Leon Romanovsky
(quoted diff) +2
-0+void mlx5r_umr_free_xlt(void *xlt, size_t length) +{
This should be made static in this patch, not in the later one
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 03/12] RDMA/mlx5: Move mkey ctrl segment logic to umr.c
In reply to Leon RomanovskyQuoted 64 lines · Leon Romanovsky
From: Aharon Landau <[email protected]>
Move
set_reg_umr_segment()and its helpers toumr.c.Signed-off-by Aharon Landau <[email protected]>Reviewed-by Michael Guralnik <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 - drivers/infiniband/hw/mlx5/qp.c | 1 + drivers/infiniband/hw/mlx5/umr.c | 129 ++++++++++++++++++++++++ drivers/infiniband/hw/mlx5/umr.h | 13 +++ drivers/infiniband/hw/mlx5/wr.c | 142 +-------------------------- 5 files changed, 147 insertions(+), 141 deletions(-)drivers/infiniband/hw/mlx5/mlx5_ib.h +1
-3+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h@@ -311,9 +311,6 @@ struct mlx5_ib_flow_db {#define MLX5_IB_QPT_DCT IB_QPT_RESERVED4#define MLX5_IB_WR_UMR IB_WR_RESERVED1 -#define MLX5_IB_UMR_OCTOWORD 16 -#define MLX5_IB_UMR_XLT_ALIGNMENT 64 -#define MLX5_IB_UPD_XLT_ZAP BIT(0)#define MLX5_IB_UPD_XLT_ENABLE BIT(1)#define MLX5_IB_UPD_XLT_ATOMIC BIT(2)drivers/infiniband/hw/mlx5/qp.c +2
-0+++ b/drivers/infiniband/hw/mlx5/qp.c@@ -40,6 +40,7 @@#include "ib_rep.h" #include "counters.h" #include "cmd.h" +#include "umr.h" #include "qp.h" #include "wr.h" drivers/infiniband/hw/mlx5/umr.c +20
-0+++ b/drivers/infiniband/hw/mlx5/umr.c@@ -4,6 +4,135 @@#include "mlx5_ib.h" #include "umr.h" +static __be64 get_umr_enable_mr_mask (void ) +{ +u64 result; + + result =MLX5_MKEY_MASK_KEY | +MLX5_MKEY_MASK_FREE ; + +return cpu_to_be64 (result); +} + +static __be64 get_umr_disable_mr_mask (void ) +{ +u64 result; + + result =MLX5_MKEY_MASK_FREE ; + +return cpu_to_be64 (result); +} +
drivers/infiniband/hw/mlx5/umr.c line 35
Quoted 10 lines
drivers/infiniband/hw/mlx5/umr.c +10
-0+static __be64 get_umr_update_translation_mask (void ) +{ +u64 result; + + result =MLX5_MKEY_MASK_LEN | +MLX5_MKEY_MASK_PAGE_SIZE | +MLX5_MKEY_MASK_START_ADDR ; + +return cpu_to_be64 (result); +}
This is pretty ugly, it is fine to copy it, but an add-on patch to remove these function wrappers would be nice
#define UMR_MR_MASK_DISABLE cpu_to_be64(MLX5_MKEY_MASK_FREE)
At worst (and arguably it can just be open coded)
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 10/12] RDMA/mlx5: Use mlx5_umr_post_send_wait() to update MR pas
In reply to Leon Romanovskydrivers/infiniband/hw/mlx5/umr.c void *mlx5r_umr_create_xlt(struct mlx5_ib_dev *dev, struct ib_sge *sg, · line 589
Quoted 7 lines · Leon Romanovsky
drivers/infiniband/hw/mlx5/umr.c +7
-0+/* +* Send the DMA list to the HW for a normal MR using UMR. +* Dmabuf MR is handled in a similar way, except that the MLX5_IB_UPD_XLT_ZAP +* flag may be used. +*/ +int mlx5r_umr_update_mr_pas (struct mlx5_ib_mr *mr,unsigned int flags) +{
I would prefer to see zap split into its own function, it shouldn't call rdma_for_each_block at all - the only reason it was structured this way in the first place was because everything had to be squeezed into a WR struct.
Then all the places calling
mlx5r_umr_update_xlt(.. MLX5_IB_UPD_XLT_ZAP)
should call this new function instead as well.
Zapping an ODP or zapping an normal mkey should just be the same logic.
Also, looking at it, it would be futher nice if the duplication between mlx5r_umr_update_xlt() and mlx5r_umr_update_mr_pas() could be removed and perhaps organized into a way to make it logical to eliminate the mlx5_odp_populate_xlt() "callback"
Which would give four entry points:
'umr fill xlt from sgl' (mlx5r_umr_update_mr_pas)
'umr fill xlt from dma_list' (populate_mtt)
'umr fill klm from xarray' (populate_klm)
'umr fill xlt/klm with zero' (zap)
I'd imagine a general construction to consolidate more stuff into mlx5r_umr_create_xlt(), probably having it return a struct, including setting up the initial wqe so the above are slimmer.
Maybe in a followup series though
Jason
Jason Gunthorpe
maintainerQuoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
UMR are special QPs that require slightly different logic than other QPs. This series from Aharon refactors the logic around UMR QP to separate file and functions to clean the post send flow.
Thanks
Quoted 13 lines
Aharon Landau (12):
RDMA/mlx5: Move init and cleanup of UMR toumr.cRDMA/mlx5: Move umr checks toumr.h
RDMA/mlx5: Move mkey ctrl segment logic toumr.cRDMA/mlx5: Simplifyget_umr_update_access_mask()RDMA/mlx5: Expose wqe posting helpers outside ofwr.cRDMA/mlx5: Introducemlx5_umr_post_send_wait()RDMA/mlx5: Usemlx5_umr_post_send_wait()to revoke MRs RDMA/mlx5: Usemlx5_umr_post_send_wait()to rereg pd access RDMA/mlx5: Move creation and free of translation tables toumr.cRDMA/mlx5: Usemlx5_umr_post_send_wait()to update MR pas RDMA/mlx5: Usemlx5_umr_post_send_wait()to update xlt RDMA/mlx5: Clean UMR QP type flow frommlx5_ib_post_send()
I moved the static to the earlier patch, but otherwise applied to for-next, thanks
Jason