RDMA/mana_ib: Implement RNIC CQs
Subsystems (2)
v2 cover RDMA/mana_ib: Implement RNIC CQs
From: Konstantin Taranov <[email protected]>
This patch series implements creation and destruction of CQs which can be used with RC QPs.
Patches with RC QPs will be sent in the next patch series.
To create a CQ for RNIC, mana_ib requires creation of EQs within mana_ib device. An EQ of mana ethernet cannot be used.
To make the implementation of create_cq cleaner, this series also introduces a helper to remove CQ callbacks.
Mana ethernet and mana_ib CQs are different entities which are created in different isolation zones (ethernet vs rnic). As a result, RNIC cannot use ethenet CQs and ethernet cannot use RNIC CQs.
That is why, we use existing udata request for creation of ethernet CQs. If the request has an extra flag, then we create an RNIC CQ. The kernel-level CQs will be RNIC CQs (in future patches).
To preserve backward and forward compatibility with RDMA-CORE, we will make the following changes to mana provider in RDMA-CORE:
The rdma-core will request RNIC CQs by default, with the proposed request format and the special flag.
If the mana has installed an allocator with manadv_set_context_attr, then the rdma-core understands that this is a DPDK use-case and requests an ethernet CQ, by not setting the flag.
If the user has a new RDMA-core and an old kernel, then the user can detect it as the response to create RNIC cq will not have queue id.
If the user has an old RDMA-core, then the flags will be 0 and ethernet CQ will be created (as expected by the user).
v1->v2:
1) removed patch that replace cqe with buf_size
2) added aditional check of queue id in the remove cb helper
3) removed buf_size from uapi request and added flags instead. It seems to be a better proposal that will not require to increase the ABI version.
Konstantin Taranov (5):
RDMA/mana_ib: create EQs for RNIC CQs RDMA/mana_ib: create and destroy RNIC cqs RDMA/mana_ib: introduce a helper to remove cq callbacks RDMA/mana_ib: boundary check before installing cq callbacks RDMA/mana_ib: implement uapi for creation of rnic cq
drivers/infiniband/hw/mana/cq.c | 74 +++++++++++++++++++---- drivers/infiniband/hw/mana/main.c | 88 +++++++++++++++++++++++++++- drivers/infiniband/hw/mana/mana_ib.h | 34 +++++++++++ drivers/infiniband/hw/mana/qp.c | 26 ++------ include/uapi/rdma/mana-abi.h | 12 ++++ 5 files changed, 200 insertions(+), 34 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/5 RDMA/mana_ib: create EQs for RNIC CQs
From: Konstantin Taranov <[email protected]>
Create EQs within mana_ib device. Such EQs are required for creation of RNIC CQs.
--- drivers/infiniband/hw/mana/main.c | 34 ++++++++++++++++++++++++++-- drivers/infiniband/hw/mana/mana_ib.h | 1 + 2 files changed, 33 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mana/main.c +32 -2
@@ -658,7 +658,7 @@ int mana_ib_create_eqs(struct mana_ib_dev *mdev) {struct gdma_context *gc =mdev_to_gc (mdev);struct gdma_queue_spec spec = {}; -int err; +int err, i; spec.type =GDMA_EQ ; spec.monitor_avl_buf = false;
@@ -672,12 +672,42 @@ int mana_ib_create_eqs(struct mana_ib_dev *mdev)if (err)return err; + mdev->eqs =kcalloc (mdev->ib_dev.num_comp_vectors,sizeof (struct gdma_queue *), +GFP_KERNEL ); +if (!mdev->eqs) { + err = -ENOMEM ; + goto destroy_fatal_eq; + } + +for (i =0 ; i < mdev->ib_dev.num_comp_vectors; i++) { + spec.eq.msix_index = (i +1 ) % gc->num_msix_usable; + err =mana_gd_create_mana_eq (mdev->gdma_dev, &spec, &mdev->eqs[i]); +if (err) + goto destroy_eqs; + } +return 0 ; + +destroy_eqs: +while (i-- >0 ) +mana_gd_destroy_queue (gc, mdev->eqs[i]); +kfree (mdev->eqs); +destroy_fatal_eq: +mana_gd_destroy_queue (gc, mdev->fatal_err_eq); +return err; }void mana_ib_destroy_eqs (struct mana_ib_dev *mdev) { -mana_gd_destroy_queue (mdev_to_gc (mdev), mdev->fatal_err_eq); +struct gdma_context *gc =mdev_to_gc (mdev); +int i; + +mana_gd_destroy_queue (gc, mdev->fatal_err_eq); + +for (i =0 ; i < mdev->ib_dev.num_comp_vectors; i++) +mana_gd_destroy_queue (gc, mdev->eqs[i]); + +kfree (mdev->eqs); }int mana_ib_gd_create_rnic_adapter (struct mana_ib_dev *mdev)
drivers/infiniband/hw/mana/mana_ib.h +1 -0
@@ -56,6 +56,7 @@ struct mana_ib_dev {struct gdma_dev *gdma_dev;mana_handle_t adapter_handle;struct gdma_queue *fatal_err_eq; +struct gdma_queue **eqs;struct mana_ib_adapter_caps adapter_caps; };
Signature · 1 line
2.43.0
PATCH v2 2/5 RDMA/mana_ib: create and destroy RNIC cqs
In reply to Konstantin TaranovFrom: Konstantin Taranov <[email protected]>
Implement RNIC requests for creation and destruction of RNIC CQs.
--- drivers/infiniband/hw/mana/main.c | 54 ++++++++++++++++++++++++++++ drivers/infiniband/hw/mana/mana_ib.h | 32 +++++++++++++++++ 2 files changed, 86 insertions(+)
drivers/infiniband/hw/mana/main.c +54 -0
@@ -834,3 +834,57 @@ int mana_ib_gd_config_mac(struct mana_ib_dev *mdev, enum mana_ib_addr_op op, u8return 0 ; } + +int mana_ib_gd_create_cq (struct mana_ib_dev *mdev,struct mana_ib_cq *cq,u32 doorbell) +{ +struct gdma_context *gc =mdev_to_gc (mdev); +struct mana_rnic_create_cq_resp resp = {}; +struct mana_rnic_create_cq_req req = {}; +int err; + +mana_gd_init_req_hdr (&req.hdr,MANA_IB_CREATE_CQ ,sizeof (req),sizeof (resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.gdma_region = cq->queue.gdma_region; + req.eq_id = mdev->eqs[cq->comp_vector]->id; + req.doorbell_page = doorbell; + + err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); + +if (err) { +ibdev_err (&mdev->ib_dev,"Failed to create cq err %d" , err); +return err; + } + + cq->queue.id = resp.cq_id; + cq->cq_handle = resp.cq_handle; +/* The GDMA region is now owned by the CQ handle */ + cq->queue.gdma_region =GDMA_INVALID_DMA_REGION ; + +return 0 ; +} + +int mana_ib_gd_destroy_cq (struct mana_ib_dev *mdev,struct mana_ib_cq *cq) +{ +struct gdma_context *gc =mdev_to_gc (mdev); +struct mana_rnic_destroy_cq_resp resp = {}; +struct mana_rnic_destroy_cq_req req = {}; +int err; + +if (cq->cq_handle ==INVALID_MANA_HANDLE ) +return 0 ; + +mana_gd_init_req_hdr (&req.hdr,MANA_IB_DESTROY_CQ ,sizeof (req),sizeof (resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.cq_handle = cq->cq_handle; + + err =mana_gd_send_request (gc,sizeof (req), &req,sizeof (resp), &resp); + +if (err) { +ibdev_err (&mdev->ib_dev,"Failed to destroy cq err %d" , err); +return err; + } + +return 0 ; +}
drivers/infiniband/hw/mana/mana_ib.h +32 -0
@@ -92,6 +92,7 @@ struct mana_ib_cq {struct mana_ib_queue queue;int cqe;u32 comp_vector; +mana_handle_t cq_handle; };struct mana_ib_qp {
@@ -119,6 +120,8 @@ enum mana_ib_command_code {MANA_IB_DESTROY_ADAPTER =0x30003 ,MANA_IB_CONFIG_IP_ADDR =0x30004 ,MANA_IB_CONFIG_MAC_ADDR =0x30005 , +MANA_IB_CREATE_CQ =0x30008 , + MANA_IB_DESTROY_CQ = 0x30009, }; struct mana_ib_query_adapter_caps_req {
@@ -202,6 +205,31 @@ struct mana_rnic_config_mac_addr_resp {struct gdma_resp_hdr hdr; };/* HW Data */ +struct mana_rnic_create_cq_req { +struct gdma_req_hdr hdr; +mana_handle_t adapter; +u64 gdma_region; +u32 eq_id; +u32 doorbell_page; +};/* HW Data */ + +struct mana_rnic_create_cq_resp { +struct gdma_resp_hdr hdr; +mana_handle_t cq_handle; +u32 cq_id; +u32 reserved; +};/* HW Data */ + +struct mana_rnic_destroy_cq_req { +struct gdma_req_hdr hdr; +mana_handle_t adapter; +mana_handle_t cq_handle; +};/* HW Data */ + +struct mana_rnic_destroy_cq_resp { +struct gdma_resp_hdr hdr; +};/* HW Data */ +static inline struct gdma_context *mdev_to_gc (struct mana_ib_dev *mdev) {return mdev->gdma_dev->gdma_context;
@@ -321,4 +349,8 @@ int mana_ib_gd_add_gid(const struct ib_gid_attr *attr, void **context);int mana_ib_gd_del_gid (const struct ib_gid_attr *attr,void **context);int mana_ib_gd_config_mac (struct mana_ib_dev *mdev,enum mana_ib_addr_op op,u8 *mac); + +int mana_ib_gd_create_cq (struct mana_ib_dev *mdev,struct mana_ib_cq *cq,u32 doorbell); + +int mana_ib_gd_destroy_cq (struct mana_ib_dev *mdev,struct mana_ib_cq *cq);#endif
Signature · 1 line
2.43.0
PATCH v2 3/5 RDMA/mana_ib: introduce a helper to remove cq callbacks
In reply to Konstantin TaranovFrom: Konstantin Taranov <[email protected]>
Intoduce the mana_ib_remove_cq_cb helper to remove cq callbacks. The helper removes code duplicates.
--- drivers/infiniband/hw/mana/cq.c | 19 ++++++++++++------- drivers/infiniband/hw/mana/mana_ib.h | 1 + drivers/infiniband/hw/mana/qp.c | 26 ++++---------------------- 3 files changed, 17 insertions(+), 29 deletions(-)
drivers/infiniband/hw/mana/cq.c +12 -7
@@ -48,16 +48,10 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq);struct ib_device *ibdev = ibcq->device;struct mana_ib_dev *mdev; -struct gdma_context *gc; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); - gc =mdev_to_gc (mdev); - -if (cq->queue.id !=INVALID_QUEUE_ID ) { -kfree (gc->cq_table[cq->queue.id]); - gc->cq_table[cq->queue.id] =NULL ; - } +mana_ib_remove_cq_cb (mdev, cq);mana_ib_destroy_queue (mdev, &cq->queue);return 0 ;
@@ -89,3 +83,14 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq) gc->cq_table[cq->queue.id] = gdma_cq;return 0 ; } + +void mana_ib_remove_cq_cb (struct mana_ib_dev *mdev,struct mana_ib_cq *cq) +{ +struct gdma_context *gc =mdev_to_gc (mdev); + +if (cq->queue.id >= gc->max_num_cqs || cq->queue.id ==INVALID_QUEUE_ID ) +return ; + +kfree (gc->cq_table[cq->queue.id]); + gc->cq_table[cq->queue.id] =NULL ; +}
drivers/infiniband/hw/mana/mana_ib.h +1 -0
@@ -255,6 +255,7 @@ static inline void copy_in_reverse(u8 *dst, const u8 *src, u32 size) }int mana_ib_install_cq_cb (struct mana_ib_dev *mdev,struct mana_ib_cq *cq); +void mana_ib_remove_cq_cb (struct mana_ib_dev *mdev,struct mana_ib_cq *cq);int mana_ib_create_zero_offset_dma_region (struct mana_ib_dev *dev,struct ib_umem *umem,mana_handle_t *gdma_region);
drivers/infiniband/hw/mana/qp.c +4 -22
@@ -95,11 +95,9 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);struct mana_ib_dev *mdev = container_of(pd->device, struct mana_ib_dev, ib_dev); -struct gdma_context *gc =mdev_to_gc (mdev);struct ib_rwq_ind_table *ind_tbl = attr->rwq_ind_tbl;struct mana_ib_create_qp_rss_resp resp = {};struct mana_ib_create_qp_rss ucmd = {}; -struct gdma_queue **gdma_cq_allocated;mana_handle_t *mana_ind_table;struct mana_port_context *mpc;unsigned int ind_tbl_size;
@@ -173,13 +171,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, goto fail; } - gdma_cq_allocated =kcalloc (ind_tbl_size,sizeof (*gdma_cq_allocated), -GFP_KERNEL ); -if (!gdma_cq_allocated) { - ret = -ENOMEM ; - goto fail; - } - qp->port = port;for (i =0 ; i < ind_tbl_size; i++) {
@@ -229,8 +220,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, ret =mana_ib_install_cq_cb (mdev, cq);if (ret) goto fail; - - gdma_cq_allocated[i] = gc->cq_table[cq->queue.id]; } resp.num_entries = i;
@@ -250,7 +239,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, goto fail; } -kfree (gdma_cq_allocated);kfree (mana_ind_table);return 0 ;
@@ -262,13 +250,10 @@ fail: wq = container_of(ibwq, struct mana_ib_wq, ibwq); cq = container_of(ibcq, struct mana_ib_cq, ibcq); - gc->cq_table[cq->queue.id] =NULL ; -kfree (gdma_cq_allocated[i]); - +mana_ib_remove_cq_cb (mdev, cq);mana_destroy_wq_obj (mpc,GDMA_RQ , wq->rx_object); } -kfree (gdma_cq_allocated);kfree (mana_ind_table);return ret;
@@ -287,10 +272,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,struct mana_ib_ucontext *mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, ibucontext); -struct gdma_context *gc =mdev_to_gc (mdev);struct mana_ib_create_qp_resp resp = {};struct mana_ib_create_qp ucmd = {}; -struct gdma_queue *gdma_cq =NULL ;struct mana_obj_spec wq_spec = {};struct mana_obj_spec cq_spec = {};struct mana_port_context *mpc;
@@ -395,14 +378,13 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,ibdev_dbg (&mdev->ib_dev,"Failed copy udata for create qp-raw, %d\n" , err); - goto err_release_gdma_cq; + goto err_remove_cq_cb; }return 0 ; -err_release_gdma_cq: -kfree (gdma_cq); - gc->cq_table[send_cq->queue.id] =NULL ; +err_remove_cq_cb: +mana_ib_remove_cq_cb (mdev, send_cq); err_destroy_wq_obj:mana_destroy_wq_obj (mpc,GDMA_SQ , qp->qp_handle);
Signature · 1 line
2.43.0
PATCH v2 4/5 RDMA/mana_ib: boundary check before installing cq callbacks
In reply to Konstantin TaranovFrom: Konstantin Taranov <[email protected]>
Add a boundary check inside mana_ib_install_cq_cb to prevent index overflow.
--- drivers/infiniband/hw/mana/cq.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/infiniband/hw/mana/cq.c +2 -0
@@ -70,6 +70,8 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq)struct gdma_context *gc =mdev_to_gc (mdev);struct gdma_queue *gdma_cq; +if (cq->queue.id >= gc->max_num_cqs) +return -EINVAL ;/* Create CQ table entry */ WARN_ON (gc->cq_table[cq->queue.id]); gdma_cq =kzalloc (sizeof (*gdma_cq),GFP_KERNEL );
Signature · 1 line
2.43.0
PATCH v2 5/5 RDMA/mana_ib: implement uapi for creation of rnic cq
In reply to Konstantin TaranovFrom: Konstantin Taranov <[email protected]>
Enable users to create RNIC CQs using a corresponding flag. With the previous request size, an ethernet CQ is created. As a response, return ID of the created CQ.
--- drivers/infiniband/hw/mana/cq.c | 55 ++++++++++++++++++++++++++++++--- include/uapi/rdma/mana-abi.h | 12 +++++++ 2 files changed, 63 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mana/cq.c +51 -4
@@ -9,17 +9,22 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, struct ib_udata *udata) { struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq); +struct mana_ib_create_cq_resp resp = {}; +struct mana_ib_ucontext *mana_ucontext;struct ib_device *ibdev = ibcq->device;struct mana_ib_create_cq ucmd = {};struct mana_ib_dev *mdev; +bool is_rnic_cq; +u32 doorbell;int err; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); -if (udata->inlen <sizeof (ucmd)) -return -EINVAL ; - cq->comp_vector = attr->comp_vector % ibdev->num_comp_vectors; + cq->cq_handle =INVALID_MANA_HANDLE ; + +if (udata->inlen < offsetof(struct mana_ib_create_cq , flags)) +return -EINVAL ; err =ib_copy_from_udata (&ucmd, udata,min (sizeof (ucmd), udata->inlen));if (err) {
@@ -28,7 +33,9 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,return err; } -if (attr->cqe > mdev->adapter_caps.max_qp_wr) { + is_rnic_cq = !!(ucmd.flags &MANA_IB_CREATE_RNIC_CQ ); + +if (!is_rnic_cq && attr->cqe > mdev->adapter_caps.max_qp_wr) {ibdev_dbg (ibdev,"CQE %d exceeding limit\n" , attr->cqe);return -EINVAL ; }
@@ -40,7 +47,41 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,return err; } + mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, + ibucontext); + doorbell = mana_ucontext->doorbell; + +if (is_rnic_cq) { + err =mana_ib_gd_create_cq (mdev, cq, doorbell); +if (err) { +ibdev_dbg (ibdev,"Failed to create RNIC cq, %d\n" , err); + goto err_destroy_queue; + } + + err =mana_ib_install_cq_cb (mdev, cq); +if (err) { +ibdev_dbg (ibdev,"Failed to install cq callback, %d\n" , err); + goto err_destroy_rnic_cq; + } + } + + resp.cqid = cq->queue.id; + err =ib_copy_to_udata (udata, &resp,min (sizeof (resp), udata->outlen)); +if (err) { +ibdev_dbg (&mdev->ib_dev,"Failed to copy to udata, %d\n" , err); + goto err_remove_cq_cb; + } +return 0 ; + +err_remove_cq_cb: +mana_ib_remove_cq_cb (mdev, cq); +err_destroy_rnic_cq: +mana_ib_gd_destroy_cq (mdev, cq); +err_destroy_queue: +mana_ib_destroy_queue (mdev, &cq->queue); + +return err; }int mana_ib_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata)
@@ -52,6 +93,12 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);mana_ib_remove_cq_cb (mdev, cq); + +/* Ignore return code as there is not much we can do about it. +* The error message is printed inside. +*/ +mana_ib_gd_destroy_cq (mdev, cq); +mana_ib_destroy_queue (mdev, &cq->queue);return 0 ;
include/uapi/rdma/mana-abi.h +12 -0
@@ -16,8 +16,20 @@#define MANA_IB_UVERBS_ABI_VERSION 1 +enum mana_ib_create_cq_flags { +MANA_IB_CREATE_RNIC_CQ =1 <<0 , +}; +struct mana_ib_create_cq {__aligned_u64 buf_addr; +__u16 flags; +__u16 reserved0; +__u32 reserved1; +}; + +struct mana_ib_create_cq_resp { +__u32 cqid; +__u32 reserved; };struct mana_ib_create_qp {
Signature · 1 line
2.43.0
RE: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
Quoted 143 lines
Subject: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
From: Konstantin Taranov <[email protected]>
Enable users to create RNIC CQs using a corresponding flag. With the previous request size, an ethernet CQ is created. As a response, return ID of the created CQ.
Signed-off-by Konstantin Taranov <[email protected]>--- drivers/infiniband/hw/mana/cq.c | 55 ++++++++++++++++++++++++++++++--- include/uapi/rdma/mana-abi.h | 12 +++++++ 2 files changed, 63 insertions(+), 4 deletions(-)drivers/infiniband/hw/mana/cq.c +8
-3@@ -9,17 +9,22 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, struct ib_udata *udata) { struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq); +struct mana_ib_create_cq_resp resp = {}; +struct mana_ib_ucontext *mana_ucontext;struct ib_device *ibdev = ibcq->device;struct mana_ib_create_cq ucmd = {};struct mana_ib_dev *mdev; +bool is_rnic_cq; +u32 doorbell;int err; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); -if (udata->inlen <sizeof (ucmd)) -return -EINVAL ; - cq->comp_vector = attr->comp_vector % ibdev->num_comp_vectors; + cq->cq_handle =INVALID_MANA_HANDLE ; + +if (udata->inlen < offsetof(struct mana_ib_create_cq , flags)) +return -EINVAL ; err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata-Quoted 1 line
inlen));
drivers/infiniband/hw/mana/cq.c +3
-1if (err) {@@ -28,7 +33,9 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, return err; } -if (attr->cqe > mdev->adapter_caps.max_qp_wr) { + is_rnic_cq = !!(ucmd.flags &MANA_IB_CREATE_RNIC_CQ ); + +if (!is_rnic_cq && attr->cqe > mdev->adapter_caps.max_qp_wr) {ibdev_dbg (ibdev,"CQE %d exceeding limit\n" , attr->cqe);return -EINVAL ;drivers/infiniband/hw/mana/cq.c +34
-0}@@ -40,7 +47,41 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, return err; } + mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, + ibucontext); + doorbell = mana_ucontext->doorbell; + +if (is_rnic_cq) { + err =mana_ib_gd_create_cq (mdev, cq, doorbell); +if (err) { +ibdev_dbg (ibdev,"Failed to create RNIC cq, %d\n" , err); + goto err_destroy_queue; + } + + err =mana_ib_install_cq_cb (mdev, cq); +if (err) { +ibdev_dbg (ibdev,"Failed to install cq callback, %d\n" , err); + goto err_destroy_rnic_cq; + } + } + + resp.cqid = cq->queue.id; + err =ib_copy_to_udata (udata, &resp,min (sizeof (resp), udata->outlen)); +if (err) { +ibdev_dbg (&mdev->ib_dev,"Failed to copy to udata, %d\n" , err); + goto err_remove_cq_cb; + } +return 0 ; + +err_remove_cq_cb: +mana_ib_remove_cq_cb (mdev, cq); +err_destroy_rnic_cq: +mana_ib_gd_destroy_cq (mdev, cq); +err_destroy_queue: +mana_ib_destroy_queue (mdev, &cq->queue); + +return err;}
int
mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)@@ -52,6
+93,12 @@ intmana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)drivers/infiniband/hw/mana/cq.c +6
-0mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);mana_ib_remove_cq_cb (mdev, cq); + +/* Ignore return code as there is not much we can do about it. +* The error message is printed inside. +*/ +mana_ib_gd_destroy_cq (mdev, cq); +mana_ib_destroy_queue (mdev, &cq->queue);return 0 ;include/uapi/rdma/mana-abi.h +12
-1@@ -16,8 +16,20 @@#define MANA_IB_UVERBS_ABI_VERSION 1 +enum mana_ib_create_cq_flags { +MANA_IB_CREATE_RNIC_CQ =1 <<0 , +}; +struct mana_ib_create_cq {__aligned_u64 buf_addr; +__u16 flags; +__u16 reserved0; +__u32 reserved1; +}; + +struct mana_ib_create_cq_resp { +__u32 cqid; +__u32 reserved; };struct mana_ib_create_qp { --
Quoted 1 line
2.43.0
For this review, it will be helpful if you can also post a link to the rdma-core changes.
Long
Konstantin Taranov
maintainerRE: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
Quoted 5 lines
For this review, it will be helpful if you can also post a link to the rdma-core changes.
Long
Here is the PR to rdma-core with the changes: https://github.com/linux-rdma/rdma-core/pull/1455
The code was tested in 4 variations (old + new kernels against old + new rdma-cores) to confirm compatibility.
Konstantin
RE: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
Quoted 12 lines
Here is the PR to rdma-core with the changes:
https://github.co/
m%2Flinux-rdma%2Frdma- core%2Fpull%2F1455&data=05%7C02%7Clongli%40microsoft.com%7C15661647 58b54b0d382508dc69e72a37%7C72f988bf86f141af91ab2d7cd011db47%7C1%7 C0%7C638501688834172236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C &sdata=lZPsMJvU%2BeoVBajN5HZBNZaADKnwOfROv5OqXL52%2FF4%3D&reser ved=0
The code was tested in 4 variations (old + new kernels against old + new rdma- cores) to confirm compatibility.
Quoted 1 line
Konstantin
There are minor issues with rdma-core change. This kernel patch looks good to me.
I have added "Reviewed-by" on this patch.
Long
RE: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
In reply to Konstantin TaranovQuoted 10 lines
Subject: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq
From: Konstantin Taranov <[email protected]>
Enable users to create RNIC CQs using a corresponding flag. With the previous request size, an ethernet CQ is created. As a response, return ID of the created CQ.
Signed-off-by Konstantin Taranov <[email protected]>
Quoted 134 lines
--- drivers/infiniband/hw/mana/cq.c | 55 ++++++++++++++++++++++++++++++--- include/uapi/rdma/mana-abi.h | 12 +++++++ 2 files changed, 63 insertions(+), 4 deletions(-)drivers/infiniband/hw/mana/cq.c +8
-3@@ -9,17 +9,22 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, struct ib_udata *udata) { struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq); +struct mana_ib_create_cq_resp resp = {}; +struct mana_ib_ucontext *mana_ucontext;struct ib_device *ibdev = ibcq->device;struct mana_ib_create_cq ucmd = {};struct mana_ib_dev *mdev; +bool is_rnic_cq; +u32 doorbell;int err; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); -if (udata->inlen <sizeof (ucmd)) -return -EINVAL ; - cq->comp_vector = attr->comp_vector % ibdev->num_comp_vectors; + cq->cq_handle =INVALID_MANA_HANDLE ; + +if (udata->inlen < offsetof(struct mana_ib_create_cq , flags)) +return -EINVAL ; err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata-Quoted 1 line
inlen));
drivers/infiniband/hw/mana/cq.c +3
-1if (err) {@@ -28,7 +33,9 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, return err; } -if (attr->cqe > mdev->adapter_caps.max_qp_wr) { + is_rnic_cq = !!(ucmd.flags &MANA_IB_CREATE_RNIC_CQ ); + +if (!is_rnic_cq && attr->cqe > mdev->adapter_caps.max_qp_wr) {ibdev_dbg (ibdev,"CQE %d exceeding limit\n" , attr->cqe);return -EINVAL ;drivers/infiniband/hw/mana/cq.c +34
-0}@@ -40,7 +47,41 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const structib_cq_init_attr *attr, return err; } + mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, + ibucontext); + doorbell = mana_ucontext->doorbell; + +if (is_rnic_cq) { + err =mana_ib_gd_create_cq (mdev, cq, doorbell); +if (err) { +ibdev_dbg (ibdev,"Failed to create RNIC cq, %d\n" , err); + goto err_destroy_queue; + } + + err =mana_ib_install_cq_cb (mdev, cq); +if (err) { +ibdev_dbg (ibdev,"Failed to install cq callback, %d\n" , err); + goto err_destroy_rnic_cq; + } + } + + resp.cqid = cq->queue.id; + err =ib_copy_to_udata (udata, &resp,min (sizeof (resp), udata->outlen)); +if (err) { +ibdev_dbg (&mdev->ib_dev,"Failed to copy to udata, %d\n" , err); + goto err_remove_cq_cb; + } +return 0 ; + +err_remove_cq_cb: +mana_ib_remove_cq_cb (mdev, cq); +err_destroy_rnic_cq: +mana_ib_gd_destroy_cq (mdev, cq); +err_destroy_queue: +mana_ib_destroy_queue (mdev, &cq->queue); + +return err;}
int
mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)@@ -52,6
+93,12 @@ intmana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)drivers/infiniband/hw/mana/cq.c +6
-0mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);mana_ib_remove_cq_cb (mdev, cq); + +/* Ignore return code as there is not much we can do about it. +* The error message is printed inside. +*/ +mana_ib_gd_destroy_cq (mdev, cq); +mana_ib_destroy_queue (mdev, &cq->queue);return 0 ;include/uapi/rdma/mana-abi.h +12
-1@@ -16,8 +16,20 @@#define MANA_IB_UVERBS_ABI_VERSION 1 +enum mana_ib_create_cq_flags { +MANA_IB_CREATE_RNIC_CQ =1 <<0 , +}; +struct mana_ib_create_cq {__aligned_u64 buf_addr; +__u16 flags; +__u16 reserved0; +__u32 reserved1; +}; + +struct mana_ib_create_cq_resp { +__u32 cqid; +__u32 reserved; };struct mana_ib_create_qp { --2.43.0
Leon Romanovsky
AppliedQuoted 8 lines · Konstantin Taranov
From: Konstantin Taranov <[email protected]>
This patch series implements creation and destruction of CQs which can be used with RC QPs.
Patches with RC QPs will be sent in the next patch series.
[...]
Applied, thanks!
[1/5] RDMA/mana_ib: create EQs for RNIC CQs
https://git.kernel.org/rdma/rdma/c/e73c882f0a0149 [2/5] RDMA/mana_ib: create and destroy RNIC cqs https://git.kernel.org/rdma/rdma/c/58434159168529 [3/5] RDMA/mana_ib: introduce a helper to remove cq callbacks https://git.kernel.org/rdma/rdma/c/3e41105263d5d7 [4/5] RDMA/mana_ib: boundary check before installing cq callbacks https://git.kernel.org/rdma/rdma/c/f79edef79b6a21 [5/5] RDMA/mana_ib: implement uapi for creation of rnic cq https://git.kernel.org/rdma/rdma/c/44b607ad4cdf23
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>