Restore failure of destroy commands
Subsystems (14)
cover Restore failure of destroy commands
From: Leon Romanovsky <[email protected]>
Hi,
This series restores the ability to fail on destroy commands, due to the fact that mlx5_ib DEVX implementation interleaved ib_core objects with FW objects without sharing reference counters.
In a retrospective, every part of the mlx5_ib flow is correct.
It started from IBTA which was written by HW engineers with HW in mind and they allowed to fail in destruction. FW implemented it with symmetrical interface like any other command and propagated error back to the kernel, which forwarded it to the libibverbs and kernel ULPs.
Libibverbs was designed with IBTA spec in hand putting destroy errors in stone. Up till mlx5_ib DEVX, it worked well, because the IB verbs objects are counted by the kernel and ib_core ensures that FW destroy will success by managing various reference counters on such objects.
The extension of the mlx5 driver changed this flow when allowed DEVX objects that are not managed by ib_core to be interleaved with the ones under ib_core responsibility.
The drivers that want to implement DEVX flows, must ensure that FW/HW destroys are performed as early as possible before any other internal cleanup. After HW destroys, drivers are not allowed to fail.
This series includes two patches (WQ and "potential race") that will require extra work in mlx5_ib, they both theoretical. WQ is not in use in DEVX, but is needed to make interface symmetrical to other objects. "Potential race" is in ULP flow that ensures that SRQ is destoyed in proper order.
Thanks
Leon Romanovsky (10):
RDMA: Restore ability to fail on PD deallocate
RDMA: Restore ability to fail on AH destroy RDMA/mlx5: Issue FW command to destroy SRQ on reentry RDMA/mlx5: Fix potential race between destroy and CQE poll
RDMA: Restore ability to fail on SRQ destroy RDMA/core: Delete function indirection for alloc/free kernel CQ
RDMA: Allow fail of destroy CQ
RDMA: Change XRCD destroy return value
RDMA: Restore ability to return error for destroy WQ
RDMA: Make counters destroy symmetrical
drivers/infiniband/core/cq.c | 36 +++--- drivers/infiniband/core/uverbs_std_types.c | 3 +- .../core/uverbs_std_types_counters.c | 4 +- drivers/infiniband/core/uverbs_std_types_wq.c | 2 +- drivers/infiniband/core/verbs.c | 56 +++++++--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 12 +- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 8 +- drivers/infiniband/hw/cxgb4/cq.c | 3 +- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 4 +- drivers/infiniband/hw/cxgb4/provider.c | 3 +- drivers/infiniband/hw/cxgb4/qp.c | 3 +- drivers/infiniband/hw/efa/efa.h | 6 +- drivers/infiniband/hw/efa/efa_verbs.c | 11 +- drivers/infiniband/hw/hns/hns_roce_ah.c | 5 - drivers/infiniband/hw/hns/hns_roce_cq.c | 3 +- drivers/infiniband/hw/hns/hns_roce_device.h | 13 ++- drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 3 +- drivers/infiniband/hw/hns/hns_roce_pd.c | 3 +- drivers/infiniband/hw/hns/hns_roce_srq.c | 3 +- drivers/infiniband/hw/i40iw/i40iw_verbs.c | 6 +- drivers/infiniband/hw/mlx4/ah.c | 5 - drivers/infiniband/hw/mlx4/cq.c | 3 +- drivers/infiniband/hw/mlx4/main.c | 6 +- drivers/infiniband/hw/mlx4/mlx4_ib.h | 11 +- drivers/infiniband/hw/mlx4/qp.c | 3 +- drivers/infiniband/hw/mlx4/srq.c | 3 +- drivers/infiniband/hw/mlx5/ah.c | 5 - drivers/infiniband/hw/mlx5/cmd.c | 4 +- drivers/infiniband/hw/mlx5/cmd.h | 2 +- drivers/infiniband/hw/mlx5/counters.c | 3 +- drivers/infiniband/hw/mlx5/cq.c | 21 ++-- drivers/infiniband/hw/mlx5/main.c | 4 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 13 ++- drivers/infiniband/hw/mlx5/qp.c | 12 +- drivers/infiniband/hw/mlx5/qp.h | 4 +- drivers/infiniband/hw/mlx5/qpc.c | 5 +- drivers/infiniband/hw/mlx5/srq.c | 26 ++--- drivers/infiniband/hw/mlx5/srq.h | 2 +- drivers/infiniband/hw/mlx5/srq_cmd.c | 22 +++- drivers/infiniband/hw/mthca/mthca_provider.c | 12 +- drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 3 +- drivers/infiniband/hw/ocrdma/ocrdma_ah.h | 2 +- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 11 +- drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 6 +- drivers/infiniband/hw/qedr/verbs.c | 14 ++- drivers/infiniband/hw/qedr/verbs.h | 8 +- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 7 +- drivers/infiniband/hw/usnic/usnic_ib_verbs.h | 4 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 3 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c | 3 +- .../infiniband/hw/vmw_pvrdma/pvrdma_verbs.c | 8 +- .../infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 8 +- drivers/infiniband/sw/rdmavt/ah.c | 3 +- drivers/infiniband/sw/rdmavt/ah.h | 2 +- drivers/infiniband/sw/rdmavt/cq.c | 3 +- drivers/infiniband/sw/rdmavt/cq.h | 2 +- drivers/infiniband/sw/rdmavt/pd.c | 3 +- drivers/infiniband/sw/rdmavt/pd.h | 2 +- drivers/infiniband/sw/rdmavt/srq.c | 3 +- drivers/infiniband/sw/rdmavt/srq.h | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 12 +- drivers/infiniband/sw/siw/siw_verbs.c | 9 +- drivers/infiniband/sw/siw/siw_verbs.h | 6 +- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 6 +- include/rdma/ib_verbs.h | 105 +++++------------- 65 files changed, 313 insertions(+), 275 deletions(-)
--
2.26.2
PATCH 4/10 RDMA/mlx5: Fix potential race between destroy and CQE poll
From: Leon Romanovsky <[email protected]>
The SRQ can be destroyed right before mlx5_cmd_get_srq is called. In such case the latter will return NULL instead of expected SRQ.
--- drivers/infiniband/hw/mlx5/cq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +3 -2
@@ -168,7 +168,7 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe, {enum rdma_link_layer ll =rdma_port_get_link_layer (qp->ibqp.device,1 );struct mlx5_ib_dev *dev =to_mdev (qp->ibqp.device); -struct mlx5_ib_srq *srq; +struct mlx5_ib_srq *srq =NULL ;struct mlx5_ib_wq *wq;u16 wqe_ctr;u8 roce_packet_type;
@@ -180,7 +180,8 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe,if (qp->ibqp.xrcd) { msrq =mlx5_cmd_get_srq (dev,be32_to_cpu (cqe->srqn)); - srq =to_mibsrq (msrq); +if (msrq) + srq =to_mibsrq (msrq); }else { srq =to_msrq (qp->ibqp.srq); }
Signature · 1 line
2.26.2
PATCH 3/10 RDMA/mlx5: Issue FW command to destroy SRQ on reentry
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The HW release can fail and leave the system in limbo state, where SRQ is removed from the table, but can't be destroyed later. In every reentry, the initial xa_erase_irq() check will fail.
Rewrite the erase logic to keep index, but don't store the entry itself. By doing it, we can safely reinsert entry back in the case of destroy failure and be safe from any xa_store_irq() error.
--- drivers/infiniband/hw/mlx5/srq_cmd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/srq_cmd.c +12 -3
@@ -596,13 +596,22 @@ void mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)struct mlx5_core_srq *tmp;int err; - tmp =xa_erase_irq (&table->array, srq->srqn); -if (!tmp || tmp != srq) +/* Delete entry, but leave index occupied */ + tmp =xa_store_irq (&table->array, srq->srqn,NULL ,0 ); +if (WARN_ON (!tmp || tmp != srq))return ; err =destroy_srq_split (dev, srq); -if (err) +if (err) { +/* +* We don't need to check returned result for an error, +* because we are storing in pre-allocated space xarray +* entry and it can't fail at this stage. +*/ +xa_store_irq (&table->array, srq->srqn, srq,0 );return ; + } +xa_erase_irq (&table->array, srq->srqn);mlx5_core_res_put (&srq->common);wait_for_completion (&srq->common.free);
Signature · 1 line
2.26.2
PATCH 2/10 RDMA: Restore ability to fail on AH destroy
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Like any other IB verbs objects, AH are refcounted by ib_core. The release of those objects is controlled by ib_core with promise that AH destroy can't fail.
Being SW object for now, this change makes dealloc_ah() to behave like any other destroy IB flows.
--- drivers/infiniband/core/verbs.c | 8 ++++++-- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 ++- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +- drivers/infiniband/hw/efa/efa.h | 2 +- drivers/infiniband/hw/efa/efa_verbs.c | 5 +++-- drivers/infiniband/hw/hns/hns_roce_ah.c | 5 ----- drivers/infiniband/hw/hns/hns_roce_device.h | 5 ++++- drivers/infiniband/hw/mlx4/ah.c | 5 ----- drivers/infiniband/hw/mlx4/mlx4_ib.h | 5 ++++- drivers/infiniband/hw/mlx5/ah.c | 5 ----- drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 ++++- drivers/infiniband/hw/mthca/mthca_provider.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_ah.h | 2 +- drivers/infiniband/hw/qedr/verbs.c | 3 ++- drivers/infiniband/hw/qedr/verbs.h | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c | 3 ++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 2 +- drivers/infiniband/sw/rdmavt/ah.c | 3 ++- drivers/infiniband/sw/rdmavt/ah.h | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- include/rdma/ib_verbs.h | 8 +++++--- 22 files changed, 46 insertions(+), 38 deletions(-)
drivers/infiniband/core/verbs.c +6 -2
@@ -968,18 +968,22 @@ int rdma_destroy_ah_user(struct ib_ah *ah, u32 flags, struct ib_udata *udata) {const struct ib_gid_attr *sgid_attr = ah->sgid_attr;struct ib_pd *pd; +int ret;might_sleep_if (flags &RDMA_DESTROY_AH_SLEEPABLE ); pd = ah->pd; - ah->device->ops.destroy_ah (ah, flags); + ret = ah->device->ops.destroy_ah (ah, flags); +if (ret && udata) +return ret; +atomic_dec (&pd->usecnt);if (sgid_attr)rdma_put_gid_attr (sgid_attr);kfree (ah); -return 0 ; +return ret; }EXPORT_SYMBOL (rdma_destroy_ah_user);
drivers/infiniband/hw/bnxt_re/ib_verbs.c +2 -1
@@ -602,13 +602,14 @@ int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) }/* Address Handles */ -void bnxt_re_destroy_ah (struct ib_ah *ib_ah,u32 flags) +int bnxt_re_destroy_ah (struct ib_ah *ib_ah,u32 flags) { struct bnxt_re_ah *ah = container_of(ib_ah, struct bnxt_re_ah, ib_ah);struct bnxt_re_dev *rdev = ah->rdev;bnxt_qplib_destroy_ah (&rdev->qplib_res, &ah->qplib_ah, !(flags &RDMA_DESTROY_AH_SLEEPABLE )); +return 0 ; }static u8 bnxt_re_stack_to_dev_nw_type (enum rdma_network_type ntype)
drivers/infiniband/hw/bnxt_re/ib_verbs.h +1 -1
@@ -168,7 +168,7 @@ int bnxt_re_create_ah(struct ib_ah *ah, struct rdma_ah_init_attr *init_attr, struct ib_udata *udata);int bnxt_re_modify_ah (struct ib_ah *ah,struct rdma_ah_attr *ah_attr);int bnxt_re_query_ah (struct ib_ah *ah,struct rdma_ah_attr *ah_attr); -void bnxt_re_destroy_ah (struct ib_ah *ah,u32 flags); +int bnxt_re_destroy_ah (struct ib_ah *ah,u32 flags);int bnxt_re_create_srq (struct ib_srq *srq,struct ib_srq_init_attr *srq_init_attr,struct ib_udata *udata);
drivers/infiniband/hw/efa/efa.h +1 -1
@@ -156,7 +156,7 @@ void efa_mmap_free(struct rdma_user_mmap_entry *rdma_entry);int efa_create_ah (struct ib_ah *ibah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata); -void efa_destroy_ah (struct ib_ah *ibah,u32 flags); +int efa_destroy_ah (struct ib_ah *ibah,u32 flags);int efa_modify_qp (struct ib_qp *ibqp,struct ib_qp_attr *qp_attr,int qp_attr_mask,struct ib_udata *udata); enum rdma_link_layer efa_port_link_layer(struct ib_device *ibdev,
drivers/infiniband/hw/efa/efa_verbs.c +3 -2
@@ -1875,7 +1875,7 @@ int efa_create_ah(struct ib_ah *ibah,return err; } -void efa_destroy_ah (struct ib_ah *ibah,u32 flags) +int efa_destroy_ah (struct ib_ah *ibah,u32 flags) {struct efa_dev *dev =to_edev (ibah->pd->device);struct efa_ah *ah =to_eah (ibah);
@@ -1885,10 +1885,11 @@ void efa_destroy_ah(struct ib_ah *ibah, u32 flags)if (!(flags &RDMA_DESTROY_AH_SLEEPABLE )) {ibdev_dbg (&dev->ibdev,"Destroy address handle is not supported in atomic context\n" ); -return ; +return -EINVAL ; }efa_ah_destroy (dev, ah); +return 0 ; }struct rdma_hw_stats *efa_alloc_hw_stats (struct ib_device *ibdev,u8 port_num)
drivers/infiniband/hw/hns/hns_roce_ah.c +0 -5
@@ -98,8 +98,3 @@ int hns_roce_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)return 0 ; } - -void hns_roce_destroy_ah (struct ib_ah *ah,u32 flags) -{ -return ; -}
drivers/infiniband/hw/hns/hns_roce_device.h +4 -1
@@ -1180,7 +1180,10 @@ void hns_roce_bitmap_free_range(struct hns_roce_bitmap *bitmap,int hns_roce_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata);int hns_roce_query_ah (struct ib_ah *ibah,struct rdma_ah_attr *ah_attr); -void hns_roce_destroy_ah (struct ib_ah *ah,u32 flags); +static inline int hns_roce_destroy_ah (struct ib_ah *ah,u32 flags) +{ +return 0 ; +}int hns_roce_alloc_pd (struct ib_pd *pd,struct ib_udata *udata);int hns_roce_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);
drivers/infiniband/hw/mlx4/ah.c +0 -5
@@ -232,8 +232,3 @@ int mlx4_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)return 0 ; } - -void mlx4_ib_destroy_ah (struct ib_ah *ah,u32 flags) -{ -return ; -}
drivers/infiniband/hw/mlx4/mlx4_ib.h +4 -1
@@ -753,7 +753,10 @@ int mlx4_ib_create_ah(struct ib_ah *ah, struct rdma_ah_init_attr *init_attr,int mlx4_ib_create_ah_slave (struct ib_ah *ah,struct rdma_ah_attr *ah_attr,int slave_sgid_index,u8 *s_mac,u16 vlan_tag);int mlx4_ib_query_ah (struct ib_ah *ibah,struct rdma_ah_attr *ah_attr); -void mlx4_ib_destroy_ah (struct ib_ah *ah,u32 flags); +static inline int mlx4_ib_destroy_ah (struct ib_ah *ah,u32 flags) +{ +return 0 ; +}int mlx4_ib_create_srq (struct ib_srq *srq,struct ib_srq_init_attr *init_attr,struct ib_udata *udata);
drivers/infiniband/hw/mlx5/ah.c +0 -5
@@ -147,8 +147,3 @@ int mlx5_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)return 0 ; } - -void mlx5_ib_destroy_ah (struct ib_ah *ah,u32 flags) -{ -return ; -}
drivers/infiniband/hw/mlx5/mlx5_ib.h +4 -1
@@ -1119,7 +1119,10 @@ void mlx5_ib_free_srq_wqe(struct mlx5_ib_srq *srq, int wqe_index);int mlx5_ib_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata);int mlx5_ib_query_ah (struct ib_ah *ibah,struct rdma_ah_attr *ah_attr); -void mlx5_ib_destroy_ah (struct ib_ah *ah,u32 flags); +static inline int mlx5_ib_destroy_ah (struct ib_ah *ah,u32 flags) +{ +return 0 ; +}int mlx5_ib_create_srq (struct ib_srq *srq,struct ib_srq_init_attr *init_attr,struct ib_udata *udata); int mlx5_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
drivers/infiniband/hw/mthca/mthca_provider.c +2 -1
@@ -390,9 +390,10 @@ static int mthca_ah_create(struct ib_ah *ibah, init_attr->ah_attr, ah); } -static void mthca_ah_destroy (struct ib_ah *ah,u32 flags) +static int mthca_ah_destroy (struct ib_ah *ah,u32 flags) {mthca_destroy_ah (to_mdev (ah->device),to_mah (ah)); +return 0 ; } static int mthca_create_srq(struct ib_srq *ibsrq,
drivers/infiniband/hw/ocrdma/ocrdma_ah.c +2 -1
@@ -215,12 +215,13 @@ int ocrdma_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,return status; } -void ocrdma_destroy_ah (struct ib_ah *ibah,u32 flags) +int ocrdma_destroy_ah (struct ib_ah *ibah,u32 flags) {struct ocrdma_ah *ah =get_ocrdma_ah (ibah);struct ocrdma_dev *dev =get_ocrdma_dev (ibah->device);ocrdma_free_av (dev, ah); +return 0 ; }int ocrdma_query_ah (struct ib_ah *ibah,struct rdma_ah_attr *attr)
drivers/infiniband/hw/ocrdma/ocrdma_ah.h +1 -1
@@ -53,7 +53,7 @@ enum {int ocrdma_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata); -void ocrdma_destroy_ah (struct ib_ah *ah,u32 flags); +int ocrdma_destroy_ah (struct ib_ah *ah,u32 flags);int ocrdma_query_ah (struct ib_ah *ah,struct rdma_ah_attr *ah_attr); int ocrdma_process_mad(struct ib_device *dev, int process_mad_flags,
drivers/infiniband/hw/qedr/verbs.c +2 -1
@@ -2767,11 +2767,12 @@ int qedr_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,return 0 ; } -void qedr_destroy_ah (struct ib_ah *ibah,u32 flags) +int qedr_destroy_ah (struct ib_ah *ibah,u32 flags) {struct qedr_ah *ah =get_qedr_ah (ibah);rdma_destroy_ah_attr (&ah->attr); +return 0 ; }static void free_mr_info (struct qedr_dev *dev,struct mr_info *info)
drivers/infiniband/hw/qedr/verbs.h +1 -1
@@ -72,7 +72,7 @@ int qedr_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr, const struct ib_recv_wr **bad_recv_wr);int qedr_create_ah (struct ib_ah *ibah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata); -void qedr_destroy_ah (struct ib_ah *ibah,u32 flags); +int qedr_destroy_ah (struct ib_ah *ibah,u32 flags);int qedr_dereg_mr (struct ib_mr *ib_mr,struct ib_udata *udata);struct ib_mr *qedr_get_dma_mr (struct ib_pd *,int acc);
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c +2 -1
@@ -549,9 +549,10 @@ int pvrdma_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,* @flags: destroy address handle flags (see enum rdma_destroy_ah_flags) * */ -void pvrdma_destroy_ah (struct ib_ah *ah,u32 flags) +int pvrdma_destroy_ah (struct ib_ah *ah,u32 flags) {struct pvrdma_dev *dev =to_vdev (ah->device);atomic_dec (&dev->num_ahs); +return 0 ; }
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h +1 -1
@@ -416,7 +416,7 @@ int pvrdma_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);int pvrdma_req_notify_cq (struct ib_cq *cq,enum ib_cq_notify_flags flags);int pvrdma_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata); -void pvrdma_destroy_ah (struct ib_ah *ah,u32 flags); +int pvrdma_destroy_ah (struct ib_ah *ah,u32 flags);int pvrdma_create_srq (struct ib_srq *srq,struct ib_srq_init_attr *init_attr,struct ib_udata *udata);
drivers/infiniband/sw/rdmavt/ah.c +2 -1
@@ -132,7 +132,7 @@ int rvt_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,* * Return: 0 on success */ -void rvt_destroy_ah (struct ib_ah *ibah,u32 destroy_flags) +int rvt_destroy_ah (struct ib_ah *ibah,u32 destroy_flags) {struct rvt_dev_info *dev =ib_to_rvt (ibah->device);struct rvt_ah *ah =ibah_to_rvtah (ibah);
@@ -143,6 +143,7 @@ void rvt_destroy_ah(struct ib_ah *ibah, u32 destroy_flags)spin_unlock_irqrestore (&dev->n_ahs_lock, flags);rdma_destroy_ah_attr (&ah->attr); +return 0 ; }/**
drivers/infiniband/sw/rdmavt/ah.h +1 -1
@@ -52,7 +52,7 @@int rvt_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata); -void rvt_destroy_ah (struct ib_ah *ibah,u32 destroy_flags); +int rvt_destroy_ah (struct ib_ah *ibah,u32 destroy_flags);int rvt_modify_ah (struct ib_ah *ibah,struct rdma_ah_attr *ah_attr);int rvt_query_ah (struct ib_ah *ibah,struct rdma_ah_attr *ah_attr);
drivers/infiniband/sw/rxe/rxe_verbs.c +2 -1
@@ -228,11 +228,12 @@ static int rxe_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr)return 0 ; } -static void rxe_destroy_ah (struct ib_ah *ibah,u32 flags) +static int rxe_destroy_ah (struct ib_ah *ibah,u32 flags) {struct rxe_ah *ah =to_rah (ibah);rxe_drop_ref (ah); +return 0 ; }static int post_one_recv (struct rxe_rq *rq,const struct ib_recv_wr *ibwr)
include/rdma/ib_verbs.h +5 -3
@@ -2409,7 +2409,7 @@ struct ib_device_ops { struct ib_udata *udata);int (*modify_ah)(struct ib_ah *ah,struct rdma_ah_attr *ah_attr);int (*query_ah)(struct ib_ah *ah,struct rdma_ah_attr *ah_attr); -void (*destroy_ah)(struct ib_ah *ah,u32 flags); +int (*destroy_ah)(struct ib_ah *ah,u32 flags);int (*create_srq)(struct ib_srq *srq,struct ib_srq_init_attr *srq_init_attr,struct ib_udata *udata);
@@ -3602,9 +3602,11 @@ int rdma_destroy_ah_user(struct ib_ah *ah, u32 flags, struct ib_udata *udata);* * NOTE: for user ah use rdma_destroy_ah_user with valid udata! */ -static inline int rdma_destroy_ah (struct ib_ah *ah,u32 flags) +static inline void rdma_destroy_ah (struct ib_ah *ah,u32 flags) { -return rdma_destroy_ah_user (ah, flags,NULL ); +int ret =rdma_destroy_ah_user (ah, flags,NULL ); + +WARN_ONCE (ret,"Destroy of kernel AH shouldn't fail" ); } struct ib_srq *ib_create_srq_user(struct ib_pd *pd,
--
2.26.2
PATCH 10/10 RDMA: Make counters destroy symmetrical
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Change counters to return failure like any other verbs destroy, however this flow shouldn't return error at all.
--- drivers/infiniband/core/uverbs_std_types_counters.c | 4 +++- drivers/infiniband/hw/mlx5/counters.c | 3 ++- include/rdma/ib_verbs.h | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-)
drivers/infiniband/core/uverbs_std_types_counters.c +3 -1
@@ -46,7 +46,9 @@ static int uverbs_free_counters(struct ib_uobject *uobject,if (ret)return ret; - counters->device->ops.destroy_counters (counters); + ret = counters->device->ops.destroy_counters (counters); +if (ret) +return ret;kfree (counters);return 0 ; }
drivers/infiniband/hw/mlx5/counters.c +2 -1
@@ -117,7 +117,7 @@ static int mlx5_ib_read_counters(struct ib_counters *counters,return ret; } -static void mlx5_ib_destroy_counters (struct ib_counters *counters) +static int mlx5_ib_destroy_counters (struct ib_counters *counters) {struct mlx5_ib_mcounters *mcounters =to_mcounters (counters);
@@ -125,6 +125,7 @@ static void mlx5_ib_destroy_counters(struct ib_counters *counters)if (mcounters->hw_cntrs_hndl)mlx5_fc_destroy (to_mdev (counters->device)->mdev, mcounters->hw_cntrs_hndl); +return 0 ; } static int mlx5_ib_create_counters(struct ib_counters *counters,
include/rdma/ib_verbs.h +1 -1
@@ -2504,7 +2504,7 @@ struct ib_device_ops { struct uverbs_attr_bundle *attrs);int (*create_counters)(struct ib_counters *counters,struct uverbs_attr_bundle *attrs); -void (*destroy_counters)(struct ib_counters *counters); +int (*destroy_counters)(struct ib_counters *counters);int (*read_counters)(struct ib_counters *counters,struct ib_counters_read_attr *counters_read_attr,struct uverbs_attr_bundle *attrs);
Signature · 1 line
2.26.2
PATCH 5/10 RDMA: Restore ability to fail on SRQ destroy
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
In similar way to other IB objects, restore ability to return error on SRQ destroy. Strictly saying, this change is not necessary, and provided here to ensure symmetrical interface to be like any other destroy command.
--- drivers/infiniband/core/verbs.c | 8 ++++-- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 ++- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 2 +- drivers/infiniband/hw/cxgb4/qp.c | 3 ++- drivers/infiniband/hw/hns/hns_roce_device.h | 2 +- drivers/infiniband/hw/hns/hns_roce_srq.c | 3 ++- drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +- drivers/infiniband/hw/mlx4/srq.c | 3 ++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/srq.c | 26 +++++++++---------- drivers/infiniband/hw/mlx5/srq.h | 2 +- drivers/infiniband/hw/mlx5/srq_cmd.c | 7 ++--- drivers/infiniband/hw/mthca/mthca_provider.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 2 +- drivers/infiniband/hw/qedr/verbs.c | 3 ++- drivers/infiniband/hw/qedr/verbs.h | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c | 3 ++- .../infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 2 +- drivers/infiniband/sw/rdmavt/srq.c | 3 ++- drivers/infiniband/sw/rdmavt/srq.h | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.h | 2 +- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 6 +---- include/rdma/ib_verbs.h | 8 +++--- 27 files changed, 62 insertions(+), 48 deletions(-)
drivers/infiniband/core/verbs.c +6 -2
@@ -1068,10 +1068,14 @@ EXPORT_SYMBOL(ib_query_srq);int ib_destroy_srq_user (struct ib_srq *srq,struct ib_udata *udata) { +int ret; +if (atomic_read (&srq->usecnt))return -EBUSY ; - srq->device->ops.destroy_srq (srq, udata); + ret = srq->device->ops.destroy_srq (srq, udata); +if (ret && udata) +return ret;atomic_dec (&srq->pd->usecnt);if (srq->srq_type ==IB_SRQT_XRC )
@@ -1080,7 +1084,7 @@ int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata)atomic_dec (&srq->ext.cq->usecnt);kfree (srq); -return 0 ; +return ret; }EXPORT_SYMBOL (ib_destroy_srq_user);
drivers/infiniband/hw/bnxt_re/ib_verbs.c +2 -1
@@ -1570,7 +1570,7 @@ static enum ib_mtu __to_ib_mtu(u32 mtu) }/* Shared Receive Queues */ -void bnxt_re_destroy_srq (struct ib_srq *ib_srq,struct ib_udata *udata) +int bnxt_re_destroy_srq (struct ib_srq *ib_srq,struct ib_udata *udata) { struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq, ib_srq);
@@ -1585,6 +1585,7 @@ void bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)atomic_dec (&rdev->srq_count);if (nq) nq->budget--; +return 0 ; } static int bnxt_re_init_user_srq(struct bnxt_re_dev *rdev,
drivers/infiniband/hw/bnxt_re/ib_verbs.h +1 -1
@@ -176,7 +176,7 @@ int bnxt_re_modify_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr,enum ib_srq_attr_mask srq_attr_mask, struct ib_udata *udata);int bnxt_re_query_srq (struct ib_srq *srq,struct ib_srq_attr *srq_attr); -void bnxt_re_destroy_srq (struct ib_srq *srq,struct ib_udata *udata); +int bnxt_re_destroy_srq (struct ib_srq *srq,struct ib_udata *udata);int bnxt_re_post_srq_recv (struct ib_srq *srq,const struct ib_recv_wr *recv_wr,const struct ib_recv_wr **bad_recv_wr); struct ib_qp *bnxt_re_create_qp(struct ib_pd *pd,
drivers/infiniband/hw/cxgb4/iw_cxgb4.h +1 -1
@@ -999,7 +999,7 @@ int c4iw_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);int c4iw_modify_srq (struct ib_srq *ib_srq,struct ib_srq_attr *attr,enum ib_srq_attr_mask srq_attr_mask,struct ib_udata *udata); -void c4iw_destroy_srq (struct ib_srq *ib_srq,struct ib_udata *udata); +int c4iw_destroy_srq (struct ib_srq *ib_srq,struct ib_udata *udata);int c4iw_create_srq (struct ib_srq *srq,struct ib_srq_init_attr *attrs,struct ib_udata *udata);int c4iw_destroy_qp (struct ib_qp *ib_qp,struct ib_udata *udata);
drivers/infiniband/hw/cxgb4/qp.c +2 -1
@@ -2797,7 +2797,7 @@ int c4iw_create_srq(struct ib_srq *ib_srq, struct ib_srq_init_attr *attrs,return ret; } -void c4iw_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +int c4iw_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct c4iw_dev *rhp;struct c4iw_srq *srq;
@@ -2813,4 +2813,5 @@ void c4iw_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata) srq->wr_waitp);c4iw_free_srq_idx (&rhp->rdev, srq->idx);c4iw_put_wr_wait (srq->wr_waitp); +return 0 ; }
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
@@ -1225,7 +1225,7 @@ int hns_roce_create_srq(struct ib_srq *srq,int hns_roce_modify_srq (struct ib_srq *ibsrq,struct ib_srq_attr *srq_attr,enum ib_srq_attr_mask srq_attr_mask,struct ib_udata *udata); -void hns_roce_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata); +int hns_roce_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata); struct ib_qp *hns_roce_create_qp(struct ib_pd *ib_pd, struct ib_qp_init_attr *init_attr,
drivers/infiniband/hw/hns/hns_roce_srq.c +2 -1
@@ -363,7 +363,7 @@ int hns_roce_create_srq(struct ib_srq *ib_srq,return ret; } -void hns_roce_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +int hns_roce_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct hns_roce_dev *hr_dev =to_hr_dev (ibsrq->device);struct hns_roce_srq *srq =to_hr_srq (ibsrq);
@@ -372,6 +372,7 @@ void hns_roce_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)free_srq_idx (hr_dev, srq);free_srq_wrid (srq);free_srq_buf (hr_dev, srq); +return 0 ; }int hns_roce_init_srq_table (struct hns_roce_dev *hr_dev)
drivers/infiniband/hw/mlx4/mlx4_ib.h +1 -1
@@ -763,7 +763,7 @@ int mlx4_ib_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,int mlx4_ib_modify_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr,enum ib_srq_attr_mask attr_mask,struct ib_udata *udata);int mlx4_ib_query_srq (struct ib_srq *srq,struct ib_srq_attr *srq_attr); -void mlx4_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata); +int mlx4_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata);void mlx4_ib_free_srq_wqe (struct mlx4_ib_srq *srq,int wqe_index);int mlx4_ib_post_srq_recv (struct ib_srq *ibsrq,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_wr);
drivers/infiniband/hw/mlx4/srq.c +2 -1
@@ -260,7 +260,7 @@ int mlx4_ib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr)return 0 ; } -void mlx4_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) +int mlx4_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) {struct mlx4_ib_dev *dev =to_mdev (srq->device);struct mlx4_ib_srq *msrq =to_msrq (srq);
@@ -282,6 +282,7 @@ void mlx4_ib_destroy_srq(struct ib_srq *srq, struct ib_udata *udata)mlx4_db_free (dev->dev, &msrq->db); }ib_umem_release (msrq->umem); +return 0 ; }void mlx4_ib_free_srq_wqe (struct mlx4_ib_srq *srq,int wqe_index)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -1128,7 +1128,7 @@ int mlx5_ib_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,int mlx5_ib_modify_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr,enum ib_srq_attr_mask attr_mask,struct ib_udata *udata);int mlx5_ib_query_srq (struct ib_srq *ibsrq,struct ib_srq_attr *srq_attr); -void mlx5_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata); +int mlx5_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata);int mlx5_ib_post_srq_recv (struct ib_srq *ibsrq,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_wr);int mlx5_ib_enable_lb (struct mlx5_ib_dev *dev,bool td,bool qp);
drivers/infiniband/hw/mlx5/srq.c +13 -13
@@ -389,24 +389,24 @@ int mlx5_ib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr)return ret; } -void mlx5_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) +int mlx5_ib_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) {struct mlx5_ib_dev *dev =to_mdev (srq->device);struct mlx5_ib_srq *msrq =to_msrq (srq); +int ret; + + ret =mlx5_cmd_destroy_srq (dev, &msrq->msrq); +if (ret && udata) +return ret; -mlx5_cmd_destroy_srq (dev, &msrq->msrq); - -if (srq->uobject) { -mlx5_ib_db_unmap_user ( -rdma_udata_to_drv_context ( -udata , -struct mlx5_ib_ucontext , - ibucontext), - &msrq->db); -ib_umem_release (msrq->umem); - }else { -destroy_srq_kernel (dev, msrq); +if (udata) { +destroy_srq_user (srq->pd, msrq, udata); +return 0 ; } + +/* We are cleaning kernel resources anyway */ +destroy_srq_kernel (dev, msrq); +return ret; }void mlx5_ib_free_srq_wqe (struct mlx5_ib_srq *srq,int wqe_index)
drivers/infiniband/hw/mlx5/srq.h +1 -1
@@ -56,7 +56,7 @@ struct mlx5_srq_table {int mlx5_cmd_create_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq,struct mlx5_srq_attr *in); -void mlx5_cmd_destroy_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq); +int mlx5_cmd_destroy_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq);int mlx5_cmd_query_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq,struct mlx5_srq_attr *out); int mlx5_cmd_arm_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
drivers/infiniband/hw/mlx5/srq_cmd.c +4 -3
@@ -590,7 +590,7 @@ int mlx5_cmd_create_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,return err; } -void mlx5_cmd_destroy_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq) +int mlx5_cmd_destroy_srq (struct mlx5_ib_dev *dev,struct mlx5_core_srq *srq) {struct mlx5_srq_table *table = &dev->srq_table;struct mlx5_core_srq *tmp;
@@ -599,7 +599,7 @@ void mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)/* Delete entry, but leave index occupied */ tmp =xa_store_irq (&table->array, srq->srqn,NULL ,0 );if (WARN_ON (!tmp || tmp != srq)) -return ; +return -EINVAL ; err =destroy_srq_split (dev, srq);if (err) {
@@ -609,12 +609,13 @@ void mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)* entry and it can't fail at this stage. */ xa_store_irq (&table->array, srq->srqn, srq,0 ); -return ; +return err; }xa_erase_irq (&table->array, srq->srqn);mlx5_core_res_put (&srq->common);wait_for_completion (&srq->common.free); +return 0 ; } int mlx5_cmd_query_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
drivers/infiniband/hw/mthca/mthca_provider.c +2 -1
@@ -442,7 +442,7 @@ static int mthca_create_srq(struct ib_srq *ibsrq,return 0 ; } -static void mthca_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) +static int mthca_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) {if (udata) { struct mthca_ucontext *context =
@@ -456,6 +456,7 @@ static void mthca_destroy_srq(struct ib_srq *srq, struct ib_udata *udata) }mthca_free_srq (to_mdev (srq->device),to_msrq (srq)); +return 0 ; } static struct ib_qp *mthca_create_qp(struct ib_pd *pd,
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +2 -1
@@ -1858,7 +1858,7 @@ int ocrdma_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr)return status; } -void ocrdma_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +int ocrdma_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct ocrdma_srq *srq;struct ocrdma_dev *dev =get_ocrdma_dev (ibsrq->device);
@@ -1873,6 +1873,7 @@ void ocrdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)kfree (srq->idx_bit_fields);kfree (srq->rqe_wr_id_tbl); +return 0 ; }/* unprivileged verbs and their support functions. */
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h +1 -1
@@ -92,7 +92,7 @@ int ocrdma_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *attr,int ocrdma_modify_srq (struct ib_srq *,struct ib_srq_attr *,enum ib_srq_attr_mask ,struct ib_udata *);int ocrdma_query_srq (struct ib_srq *,struct ib_srq_attr *); -void ocrdma_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata); +int ocrdma_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata);int ocrdma_post_srq_recv (struct ib_srq *,const struct ib_recv_wr *,const struct ib_recv_wr **bad_recv_wr);
drivers/infiniband/hw/qedr/verbs.c +2 -1
@@ -1592,7 +1592,7 @@ int qedr_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr,return -EFAULT ; } -void qedr_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +int qedr_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct qed_rdma_destroy_srq_in_params in_params = {};struct qedr_dev *dev =get_qedr_dev (ibsrq->device);
@@ -1610,6 +1610,7 @@ void qedr_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)DP_DEBUG (dev,QEDR_MSG_SRQ ,"destroy srq: destroyed srq with srq_id=0x%0x\n" , srq->srq_id); +return 0 ; } int qedr_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
drivers/infiniband/hw/qedr/verbs.h +1 -1
@@ -67,7 +67,7 @@ int qedr_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *attr,int qedr_modify_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr,enum ib_srq_attr_mask attr_mask,struct ib_udata *udata);int qedr_query_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr); -void qedr_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata); +int qedr_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata);int qedr_post_srq_recv (struct ib_srq *ibsrq,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_recv_wr); int qedr_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c +2 -1
@@ -240,7 +240,7 @@ static void pvrdma_free_srq(struct pvrdma_dev *dev, struct pvrdma_srq *srq)* * @return: 0 for success. */ -void pvrdma_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) +int pvrdma_destroy_srq (struct ib_srq *srq,struct ib_udata *udata) {struct pvrdma_srq *vsrq =to_vsrq (srq);union pvrdma_cmd_req req;
@@ -259,6 +259,7 @@ void pvrdma_destroy_srq(struct ib_srq *srq, struct ib_udata *udata) ret);pvrdma_free_srq (dev, vsrq); +return 0 ; }/**
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h +1 -1
@@ -423,7 +423,7 @@ int pvrdma_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,int pvrdma_modify_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr,enum ib_srq_attr_mask attr_mask,struct ib_udata *udata);int pvrdma_query_srq (struct ib_srq *srq,struct ib_srq_attr *srq_attr); -void pvrdma_destroy_srq (struct ib_srq *srq,struct ib_udata *udata); +int pvrdma_destroy_srq (struct ib_srq *srq,struct ib_udata *udata); struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, struct ib_qp_init_attr *init_attr,
drivers/infiniband/sw/rdmavt/srq.c +2 -1
@@ -332,7 +332,7 @@ int rvt_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)* @ibsrq: srq object to destroy * */ -void rvt_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +int rvt_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct rvt_srq *srq =ibsrq_to_rvtsrq (ibsrq);struct rvt_dev_info *dev =ib_to_rvt (ibsrq->device);
@@ -343,4 +343,5 @@ void rvt_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)if (srq->ip)kref_put (&srq->ip->ref, rvt_release_mmap_info);kvfree (srq->rq.kwq); +return 0 ; }
drivers/infiniband/sw/rdmavt/srq.h +1 -1
@@ -56,6 +56,6 @@ int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);int rvt_query_srq (struct ib_srq *ibsrq,struct ib_srq_attr *attr); -void rvt_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata); +int rvt_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata);#endif /* DEF_RVTSRQ_H */
drivers/infiniband/sw/rxe/rxe_verbs.c +2 -1
@@ -367,7 +367,7 @@ static int rxe_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)return 0 ; } -static void rxe_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) +static int rxe_destroy_srq (struct ib_srq *ibsrq,struct ib_udata *udata) {struct rxe_srq *srq =to_rsrq (ibsrq);
@@ -376,6 +376,7 @@ static void rxe_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)rxe_drop_ref (srq->pd);rxe_drop_ref (srq); +return 0 ; } static int rxe_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
drivers/infiniband/sw/siw/siw_verbs.c +2 -1
@@ -1691,7 +1691,7 @@ int siw_query_srq(struct ib_srq *base_srq, struct ib_srq_attr *attrs)* QP anymore - the code trusts the RDMA core environment to keep track * of QP references. */ -void siw_destroy_srq (struct ib_srq *base_srq,struct ib_udata *udata) +int siw_destroy_srq (struct ib_srq *base_srq,struct ib_udata *udata) {struct siw_srq *srq =to_siw_srq (base_srq);struct siw_device *sdev =to_siw_dev (base_srq->device);
@@ -1703,6 +1703,7 @@ void siw_destroy_srq(struct ib_srq *base_srq, struct ib_udata *udata)rdma_user_mmap_entry_remove (srq->srq_entry);vfree (srq->recvq);atomic_dec (&sdev->num_srq); +return 0 ; }/*
drivers/infiniband/sw/siw/siw_verbs.h +1 -1
@@ -78,7 +78,7 @@ int siw_create_srq(struct ib_srq *base_srq, struct ib_srq_init_attr *attr,int siw_modify_srq (struct ib_srq *base_srq,struct ib_srq_attr *attr,enum ib_srq_attr_mask mask,struct ib_udata *udata);int siw_query_srq (struct ib_srq *base_srq,struct ib_srq_attr *attr); -void siw_destroy_srq (struct ib_srq *base_srq,struct ib_udata *udata); +int siw_destroy_srq (struct ib_srq *base_srq,struct ib_udata *udata);int siw_post_srq_recv (struct ib_srq *base_srq,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_wr);int siw_mmap (struct ib_ucontext *ctx,struct vm_area_struct *vma);
drivers/infiniband/ulp/ipoib/ipoib_cm.c +1 -5
@@ -1647,17 +1647,13 @@ int ipoib_cm_dev_init(struct net_device *dev)void ipoib_cm_dev_cleanup (struct net_device *dev) {struct ipoib_dev_priv *priv =ipoib_priv (dev); -int ret;if (!priv->cm.srq)return ;ipoib_dbg (priv,"Cleanup ipoib connected mode.\n" ); - ret =ib_destroy_srq (priv->cm.srq); -if (ret) -ipoib_warn (priv,"ib_destroy_srq failed: %d\n" , ret); - +ib_destroy_srq (priv->cm.srq); priv->cm.srq =NULL ;if (!priv->cm.srq_ring)return ;
include/rdma/ib_verbs.h +5 -3
@@ -2417,7 +2417,7 @@ struct ib_device_ops {enum ib_srq_attr_mask srq_attr_mask, struct ib_udata *udata);int (*query_srq)(struct ib_srq *srq,struct ib_srq_attr *srq_attr); -void (*destroy_srq)(struct ib_srq *srq,struct ib_udata *udata); +int (*destroy_srq)(struct ib_srq *srq,struct ib_udata *udata);struct ib_qp *(*create_qp)(struct ib_pd *pd,struct ib_qp_init_attr *qp_init_attr,struct ib_udata *udata);
@@ -3660,9 +3660,11 @@ int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata);* * NOTE: for user srq use ib_destroy_srq_user with valid udata! */ -static inline int ib_destroy_srq (struct ib_srq *srq) +static inline void ib_destroy_srq (struct ib_srq *srq) { -return ib_destroy_srq_user (srq,NULL ); +int ret =ib_destroy_srq_user (srq,NULL ); + +WARN_ONCE (ret,"Destroy of kernel SRQ shouldn't fail" ); }/**
Signature · 1 line
2.26.2
PATCH 8/10 RDMA: Change XRCD destroy return value
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Update XRCD destroy flow to allow command failure.
--- drivers/infiniband/core/verbs.c | 8 ++++++-- drivers/infiniband/hw/mlx4/main.c | 3 ++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/qp.c | 4 ++-- include/rdma/ib_verbs.h | 2 +- 5 files changed, 12 insertions(+), 7 deletions(-)
drivers/infiniband/core/verbs.c +6 -2
@@ -2345,13 +2345,17 @@ EXPORT_SYMBOL(ib_alloc_xrcd_user);*/ int ib_dealloc_xrcd_user (struct ib_xrcd *xrcd,struct ib_udata *udata) { +int ret; +if (atomic_read (&xrcd->usecnt))return -EBUSY ;WARN_ON (!xa_empty (&xrcd->tgt_qps)); - xrcd->device->ops.dealloc_xrcd (xrcd, udata); + ret = xrcd->device->ops.dealloc_xrcd (xrcd, udata); +if (ret && udata) +return ret;kfree (xrcd); -return 0 ; +return ret; }EXPORT_SYMBOL (ib_dealloc_xrcd_user);
drivers/infiniband/hw/mlx4/main.c +2 -1
@@ -1256,11 +1256,12 @@ static int mlx4_ib_alloc_xrcd(struct ib_xrcd *ibxrcd, struct ib_udata *udata)return err; } -static void mlx4_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata) +static int mlx4_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata) {ib_destroy_cq (to_mxrcd (xrcd)->cq);ib_dealloc_pd (to_mxrcd (xrcd)->pd);mlx4_xrcd_free (to_mdev (xrcd->device)->dev,to_mxrcd (xrcd)->xrcdn); +return 0 ; }static int add_gid_entry (struct ib_qp *ibqp,union ib_gid *gid)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -1196,7 +1196,7 @@ int mlx5_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num, const struct ib_mad *in, struct ib_mad *out, size_t *out_mad_size, u16 *out_mad_pkey_index);int mlx5_ib_alloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata); -void mlx5_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata); +int mlx5_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata);int mlx5_ib_get_buf_offset (u64 addr,int page_shift,u32 *offset);int mlx5_query_ext_port_caps (struct mlx5_ib_dev *dev,u8 port); int mlx5_query_mad_ifc_smp_attr_node_info(struct ib_device *ibdev,
drivers/infiniband/hw/mlx5/qp.c +2 -2
@@ -4745,12 +4745,12 @@ int mlx5_ib_alloc_xrcd(struct ib_xrcd *ibxrcd, struct ib_udata *udata)return mlx5_cmd_xrcd_alloc (dev->mdev, &xrcd->xrcdn,0 ); } -void mlx5_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata) +int mlx5_ib_dealloc_xrcd (struct ib_xrcd *xrcd,struct ib_udata *udata) {struct mlx5_ib_dev *dev =to_mdev (xrcd->device);u32 xrcdn =to_mxrcd (xrcd)->xrcdn; -mlx5_cmd_xrcd_dealloc (dev->mdev, xrcdn,0 ); +return mlx5_cmd_xrcd_dealloc (dev->mdev, xrcdn,0 ); }static void mlx5_ib_wq_event (struct mlx5_core_qp *core_qp,int type)
include/rdma/ib_verbs.h +1 -1
@@ -2458,7 +2458,7 @@ struct ib_device_ops {int (*attach_mcast)(struct ib_qp *qp,union ib_gid *gid,u16 lid);int (*detach_mcast)(struct ib_qp *qp,union ib_gid *gid,u16 lid);int (*alloc_xrcd)(struct ib_xrcd *xrcd,struct ib_udata *udata); -void (*dealloc_xrcd)(struct ib_xrcd *xrcd,struct ib_udata *udata); +int (*dealloc_xrcd)(struct ib_xrcd *xrcd,struct ib_udata *udata);struct ib_flow *(*create_flow)(struct ib_qp *qp,struct ib_flow_attr *flow_attr,struct ib_udata *udata);
Signature · 1 line
2.26.2
PATCH 7/10 RDMA: Allow fail of destroy CQ
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Like any other verbs object, CQ shouldn't fail during destroy, but mlx5_ib didn't follow this contract while mixed IB verbs objects with DEVX. Such mix causes to the situation where FW and kernel are fully dependent on the reference counting of another side.
Kernel verbs and drivers that don't have DEVX flows shouldn't fail.
--- drivers/infiniband/core/cq.c | 5 ++++- drivers/infiniband/core/verbs.c | 9 +++++++-- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 ++- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +- drivers/infiniband/hw/cxgb4/cq.c | 3 ++- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 2 +- drivers/infiniband/hw/efa/efa.h | 2 +- drivers/infiniband/hw/efa/efa_verbs.c | 3 ++- drivers/infiniband/hw/hns/hns_roce_cq.c | 3 ++- drivers/infiniband/hw/hns/hns_roce_device.h | 4 ++-- drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 3 ++- drivers/infiniband/hw/i40iw/i40iw_verbs.c | 3 ++- drivers/infiniband/hw/mlx4/cq.c | 3 ++- drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +- drivers/infiniband/hw/mlx5/cq.c | 16 +++++++++++----- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mthca/mthca_provider.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 2 +- drivers/infiniband/hw/qedr/verbs.c | 5 +++-- drivers/infiniband/hw/qedr/verbs.h | 2 +- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 4 ++-- drivers/infiniband/hw/usnic/usnic_ib_verbs.h | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 3 ++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 2 +- drivers/infiniband/sw/rdmavt/cq.c | 3 ++- drivers/infiniband/sw/rdmavt/cq.h | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.h | 2 +- include/rdma/ib_verbs.h | 6 ++++-- 31 files changed, 70 insertions(+), 40 deletions(-)
drivers/infiniband/core/cq.c +4 -1
@@ -319,6 +319,8 @@ EXPORT_SYMBOL(__ib_alloc_cq_any);*/ void ib_free_cq (struct ib_cq *cq) { +int ret; +WARN_ON_ONCE (atomic_read (&cq->usecnt));WARN_ON_ONCE (cq->cqe_used);
@@ -338,8 +340,9 @@ void ib_free_cq(struct ib_cq *cq)rdma_dim_destroy (cq);trace_cq_free (cq); + ret = cq->device->ops.destroy_cq (cq,NULL ); +WARN_ONCE (ret,"Destroy of kernel CQ shouldn't fail" );rdma_restrack_del (&cq->res); - cq->device->ops.destroy_cq (cq,NULL );kfree (cq->wc);kfree (cq); }
drivers/infiniband/core/verbs.c +7 -2
@@ -2023,16 +2023,21 @@ EXPORT_SYMBOL(rdma_set_cq_moderation);int ib_destroy_cq_user (struct ib_cq *cq,struct ib_udata *udata) { +int ret; +if (WARN_ON_ONCE (cq->shared))return -EOPNOTSUPP ;if (atomic_read (&cq->usecnt))return -EBUSY ; + ret = cq->device->ops.destroy_cq (cq, udata); +if (ret && udata) +return ret; +rdma_restrack_del (&cq->res); - cq->device->ops.destroy_cq (cq, udata);kfree (cq); -return 0 ; +return ret; }EXPORT_SYMBOL (ib_destroy_cq_user);
drivers/infiniband/hw/bnxt_re/ib_verbs.c +2 -1
@@ -2803,7 +2803,7 @@ int bnxt_re_post_recv(struct ib_qp *ib_qp, const struct ib_recv_wr *wr, }/* Completion Queues */ -void bnxt_re_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) +int bnxt_re_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) {struct bnxt_re_cq *cq;struct bnxt_qplib_nq *nq;
@@ -2819,6 +2819,7 @@ void bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)atomic_dec (&rdev->cq_count); nq->budget--;kfree (cq->cql); +return 0 ; } int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
drivers/infiniband/hw/bnxt_re/ib_verbs.h +1 -1
@@ -193,7 +193,7 @@ int bnxt_re_post_recv(struct ib_qp *qp, const struct ib_recv_wr *recv_wr, const struct ib_recv_wr **bad_recv_wr);int bnxt_re_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void bnxt_re_destroy_cq (struct ib_cq *cq,struct ib_udata *udata); +int bnxt_re_destroy_cq (struct ib_cq *cq,struct ib_udata *udata);int bnxt_re_poll_cq (struct ib_cq *cq,int num_entries,struct ib_wc *wc);int bnxt_re_req_notify_cq (struct ib_cq *cq,enum ib_cq_notify_flags flags);struct ib_mr *bnxt_re_get_dma_mr (struct ib_pd *pd,int mr_access_flags);
drivers/infiniband/hw/cxgb4/cq.c +2 -1
@@ -967,7 +967,7 @@ int c4iw_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)return !err || err == -ENODATA ? npolled : err; } -void c4iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) +int c4iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) {struct c4iw_cq *chp;struct c4iw_ucontext *ucontext;
@@ -985,6 +985,7 @@ void c4iw_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata) ucontext ? &ucontext->uctx : &chp->cq.rdev->uctx, chp->destroy_skb, chp->wr_waitp);c4iw_put_wr_wait (chp->wr_waitp); +return 0 ; } int c4iw_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
drivers/infiniband/hw/cxgb4/iw_cxgb4.h +1 -1
@@ -992,7 +992,7 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, struct ib_udata *udata);struct ib_mr *c4iw_get_dma_mr (struct ib_pd *pd,int acc);int c4iw_dereg_mr (struct ib_mr *ib_mr,struct ib_udata *udata); -void c4iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata); +int c4iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata);int c4iw_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);int c4iw_arm_cq (struct ib_cq *ibcq,enum ib_cq_notify_flags flags);
drivers/infiniband/hw/efa/efa.h +1 -1
@@ -139,7 +139,7 @@ int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);struct ib_qp *efa_create_qp (struct ib_pd *ibpd,struct ib_qp_init_attr *init_attr,struct ib_udata *udata); -void efa_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata); +int efa_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata);int efa_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length,
drivers/infiniband/hw/efa/efa_verbs.c +2 -1
@@ -973,7 +973,7 @@ static int efa_destroy_cq_idx(struct efa_dev *dev, int cq_idx)return efa_com_destroy_cq (&dev->edev, ¶ms); } -void efa_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +int efa_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibcq->device);struct efa_cq *cq =to_ecq (ibcq);
@@ -986,6 +986,7 @@ void efa_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)efa_destroy_cq_idx (dev, cq->cq_idx);efa_free_mapped (dev, cq->cpu_addr, cq->dma_addr, cq->size,DMA_FROM_DEVICE ); +return 0 ; } static int cq_mmap_entries_setup(struct efa_dev *dev, struct efa_cq *cq,
drivers/infiniband/hw/hns/hns_roce_cq.c +2 -1
@@ -311,7 +311,7 @@ int hns_roce_create_cq(struct ib_cq *ib_cq, const struct ib_cq_init_attr *attr,return ret; } -void hns_roce_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) +int hns_roce_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) {struct hns_roce_dev *hr_dev =to_hr_dev (ib_cq->device);struct hns_roce_cq *hr_cq =to_hr_cq (ib_cq);
@@ -322,6 +322,7 @@ void hns_roce_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)free_cq_buf (hr_dev, hr_cq);free_cq_db (hr_dev, hr_cq, udata);free_cqc (hr_dev, hr_cq); +return 0 ; }void hns_roce_cq_completion (struct hns_roce_dev *hr_dev,u32 cqn)
drivers/infiniband/hw/hns/hns_roce_device.h +2 -2
@@ -932,7 +932,7 @@ struct hns_roce_hw {int (*poll_cq)(struct ib_cq *ibcq,int num_entries,struct ib_wc *wc);int (*dereg_mr)(struct hns_roce_dev *hr_dev,struct hns_roce_mr *mr,struct ib_udata *udata); -void (*destroy_cq)(struct ib_cq *ibcq,struct ib_udata *udata); +int (*destroy_cq)(struct ib_cq *ibcq,struct ib_udata *udata);int (*modify_cq)(struct ib_cq *cq,u16 cq_count,u16 cq_period);int (*init_eq)(struct hns_roce_dev *hr_dev);void (*cleanup_eq)(struct hns_roce_dev *hr_dev);
@@ -1252,7 +1252,7 @@ int to_hr_qp_type(int qp_type);int hns_roce_create_cq (struct ib_cq *ib_cq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void hns_roce_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata); +int hns_roce_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata);int hns_roce_db_map_user (struct hns_roce_ucontext *context,struct ib_udata *udata,unsigned long virt,struct hns_roce_db *db);
drivers/infiniband/hw/hns/hns_roce_hw_v1.c +2 -1
@@ -3572,7 +3572,7 @@ int hns_roce_v1_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)return 0 ; } -static void hns_roce_v1_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +static int hns_roce_v1_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct hns_roce_dev *hr_dev =to_hr_dev (ibcq->device);struct hns_roce_cq *hr_cq =to_hr_cq (ibcq);
@@ -3603,6 +3603,7 @@ static void hns_roce_v1_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) } wait_time++; } +return 0 ; }static void set_eq_cons_index_v1 (struct hns_roce_eq *eq,int req_not)
drivers/infiniband/hw/i40iw/i40iw_verbs.c +2 -1
@@ -1053,7 +1053,7 @@ void i40iw_cq_wq_destroy(struct i40iw_device *iwdev, struct i40iw_sc_cq *cq)* @ib_cq: cq pointer * @udata: user data or NULL for kernel object */ -static void i40iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) +static int i40iw_destroy_cq (struct ib_cq *ib_cq,struct ib_udata *udata) {struct i40iw_cq *iwcq;struct i40iw_device *iwdev;
@@ -1065,6 +1065,7 @@ static void i40iw_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)i40iw_cq_wq_destroy (iwdev, cq);cq_free_resources (iwdev, iwcq);i40iw_rem_devusecount (iwdev); +return 0 ; }/**
drivers/infiniband/hw/mlx4/cq.c +2 -1
@@ -475,7 +475,7 @@ int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata)return err; } -void mlx4_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) +int mlx4_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) {struct mlx4_ib_dev *dev =to_mdev (cq->device);struct mlx4_ib_cq *mcq =to_mcq (cq);
@@ -495,6 +495,7 @@ void mlx4_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)mlx4_db_free (dev->dev, &mcq->db); }ib_umem_release (mcq->umem); +return 0 ; }static void dump_cqe (void *cqe)
drivers/infiniband/hw/mlx4/mlx4_ib.h +1 -1
@@ -742,7 +742,7 @@ int mlx4_ib_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period);int mlx4_ib_resize_cq (struct ib_cq *ibcq,int entries,struct ib_udata *udata);int mlx4_ib_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void mlx4_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata); +int mlx4_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata);int mlx4_ib_poll_cq (struct ib_cq *ibcq,int num_entries,struct ib_wc *wc);int mlx4_ib_arm_cq (struct ib_cq *cq,enum ib_cq_notify_flags flags);void __mlx4_ib_cq_clean (struct mlx4_ib_cq *cq,u32 qpn,struct mlx4_ib_srq *srq);
drivers/infiniband/hw/mlx5/cq.c +11 -5
@@ -1024,16 +1024,22 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,return err; } -void mlx5_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) +int mlx5_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) {struct mlx5_ib_dev *dev =to_mdev (cq->device);struct mlx5_ib_cq *mcq =to_mcq (cq); +int ret; -mlx5_core_destroy_cq (dev->mdev, &mcq->mcq); -if (udata) + ret =mlx5_core_destroy_cq (dev->mdev, &mcq->mcq); +if (ret && udata) +return ret; + +if (udata) {destroy_cq_user (mcq, udata); -else -destroy_cq_kernel (dev, mcq); +return 0 ; + } +destroy_cq_kernel (dev, mcq); +return ret; }static int is_equal_rsn (struct mlx5_cqe64 *cqe64,u32 rsn)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -1151,7 +1151,7 @@ int mlx5_ib_read_wqe_srq(struct mlx5_ib_srq *srq, int wqe_index, void *buffer, size_t buflen, size_t *bc);int mlx5_ib_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void mlx5_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata); +int mlx5_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata);int mlx5_ib_poll_cq (struct ib_cq *ibcq,int num_entries,struct ib_wc *wc);int mlx5_ib_arm_cq (struct ib_cq *ibcq,enum ib_cq_notify_flags flags);int mlx5_ib_modify_cq (struct ib_cq *cq,u16 cq_count,u16 cq_period);
drivers/infiniband/hw/mthca/mthca_provider.c +2 -1
@@ -792,7 +792,7 @@ static int mthca_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udareturn ret; } -static void mthca_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) +static int mthca_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) {if (udata) { struct mthca_ucontext *context =
@@ -811,6 +811,7 @@ static void mthca_destroy_cq(struct ib_cq *cq, struct ib_udata *udata) to_mcq(cq)->set_ci_db_index); }mthca_free_cq (to_mdev (cq->device),to_mcq (cq)); +return 0 ; }static inline u32 convert_access (int acc)
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +2 -1
@@ -1057,7 +1057,7 @@ static void ocrdma_flush_cq(struct ocrdma_cq *cq)spin_unlock_irqrestore (&cq->cq_lock, flags); } -void ocrdma_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +int ocrdma_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct ocrdma_cq *cq =get_ocrdma_cq (ibcq);struct ocrdma_eq *eq =NULL ;
@@ -1082,6 +1082,7 @@ void ocrdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)ocrdma_get_db_addr (dev, pdid), dev->nic_info.db_page_size); } +return 0 ; }static int ocrdma_add_qpn_map (struct ocrdma_dev *dev,struct ocrdma_qp *qp)
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h +1 -1
@@ -72,7 +72,7 @@ int ocrdma_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);int ocrdma_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);int ocrdma_resize_cq (struct ib_cq *,int cqe,struct ib_udata *); -void ocrdma_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata); +int ocrdma_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata); struct ib_qp *ocrdma_create_qp(struct ib_pd *, struct ib_qp_init_attr *attrs,
drivers/infiniband/hw/qedr/verbs.c +3 -2
@@ -1052,7 +1052,7 @@ int qedr_resize_cq(struct ib_cq *ibcq, int new_cnt, struct ib_udata *udata)#define QEDR_DESTROY_CQ_MAX_ITERATIONS (10)#define QEDR_DESTROY_CQ_ITER_DURATION (10) -void qedr_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +int qedr_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct qedr_dev *dev =get_qedr_dev (ibcq->device);struct qed_rdma_destroy_cq_out_params oparams;
@@ -1067,7 +1067,7 @@ void qedr_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)/* GSIs CQs are handled by driver, so they don't exist in the FW */ if (cq->cq_type ==QEDR_CQ_TYPE_GSI ) {qedr_db_recovery_del (dev, cq->db_addr, &cq->db.data); -return ; +return 0 ; } iparams.icid = cq->icid;
@@ -1115,6 +1115,7 @@ void qedr_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)* Since the destroy CQ ramrod has also been received on the EQ we can * be certain that there's no event handler in process. */ +return 0 ; } static inline int get_gid_info_from_table(struct ib_qp *ibqp,
drivers/infiniband/hw/qedr/verbs.h +1 -1
@@ -52,7 +52,7 @@ int qedr_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);int qedr_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);int qedr_resize_cq (struct ib_cq *,int cqe,struct ib_udata *); -void qedr_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata); +int qedr_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata);int qedr_arm_cq (struct ib_cq *ibcq,enum ib_cq_notify_flags flags);struct ib_qp *qedr_create_qp (struct ib_pd *,struct ib_qp_init_attr *attrs,struct ib_udata *);
drivers/infiniband/hw/usnic/usnic_ib_verbs.c +2 -2
@@ -597,9 +597,9 @@ int usnic_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,return 0 ; } -void usnic_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) +int usnic_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) { -return ; +return 0 ; } struct ib_mr *usnic_ib_reg_mr(struct ib_pd *pd, u64 start, u64 length,
drivers/infiniband/hw/usnic/usnic_ib_verbs.h +1 -1
@@ -60,7 +60,7 @@ int usnic_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, struct ib_udata *udata);int usnic_ib_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void usnic_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata); +int usnic_ib_destroy_cq (struct ib_cq *cq,struct ib_udata *udata);struct ib_mr *usnic_ib_reg_mr (struct ib_pd *pd,u64 start,u64 length,u64 virt_addr,int access_flags,struct ib_udata *udata);
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c +2 -1
@@ -235,7 +235,7 @@ static void pvrdma_free_cq(struct pvrdma_dev *dev, struct pvrdma_cq *cq)* @cq: the completion queue to destroy. * @udata: user data or null for kernel object */ -void pvrdma_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) +int pvrdma_destroy_cq (struct ib_cq *cq,struct ib_udata *udata) {struct pvrdma_cq *vcq =to_vcq (cq);union pvrdma_cmd_req req;
@@ -261,6 +261,7 @@ void pvrdma_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)pvrdma_free_cq (dev, vcq);atomic_dec (&dev->num_cqs); +return 0 ; }static inline struct pvrdma_cqe *get_cqe (struct pvrdma_cq *cq,int i)
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h +1 -1
@@ -411,7 +411,7 @@ int pvrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, unsigned int *sg_offset);int pvrdma_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void pvrdma_destroy_cq (struct ib_cq *cq,struct ib_udata *udata); +int pvrdma_destroy_cq (struct ib_cq *cq,struct ib_udata *udata);int pvrdma_poll_cq (struct ib_cq *ibcq,int num_entries,struct ib_wc *wc);int pvrdma_req_notify_cq (struct ib_cq *cq,enum ib_cq_notify_flags flags); int pvrdma_create_ah(struct ib_ah *ah, struct rdma_ah_init_attr *init_attr,
drivers/infiniband/sw/rdmavt/cq.c +2 -1
@@ -315,7 +315,7 @@ int rvt_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,* * Called by ib_destroy_cq() in the generic verbs code. */ -void rvt_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +int rvt_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct rvt_cq *cq =ibcq_to_rvtcq (ibcq);struct rvt_dev_info *rdi = cq->rdi;
@@ -328,6 +328,7 @@ void rvt_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)kref_put (&cq->ip->ref, rvt_release_mmap_info); else vfree(cq->kqueue); +return 0 ; }/**
drivers/infiniband/sw/rdmavt/cq.h +1 -1
@@ -53,7 +53,7 @@int rvt_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata); -void rvt_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata); +int rvt_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata);int rvt_req_notify_cq (struct ib_cq *ibcq,enum ib_cq_notify_flags notify_flags);int rvt_resize_cq (struct ib_cq *ibcq,int cqe,struct ib_udata *udata);int rvt_poll_cq (struct ib_cq *ibcq,int num_entries,struct ib_wc *entry);
drivers/infiniband/sw/rxe/rxe_verbs.c +2 -1
@@ -806,13 +806,14 @@ static int rxe_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,return rxe_add_to_pool (&rxe->cq_pool, &cq->pelem); } -static void rxe_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) +static int rxe_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata) {struct rxe_cq *cq =to_rcq (ibcq);rxe_cq_disable (cq);rxe_drop_ref (cq); +return 0 ; }static int rxe_resize_cq (struct ib_cq *ibcq,int cqe,struct ib_udata *udata)
drivers/infiniband/sw/siw/siw_verbs.c +2 -1
@@ -1056,7 +1056,7 @@ int siw_post_receive(struct ib_qp *base_qp, const struct ib_recv_wr *wr,return rv >0 ?0 : rv; } -void siw_destroy_cq (struct ib_cq *base_cq,struct ib_udata *udata) +int siw_destroy_cq (struct ib_cq *base_cq,struct ib_udata *udata) {struct siw_cq *cq =to_siw_cq (base_cq);struct siw_device *sdev =to_siw_dev (base_cq->device);
@@ -1074,6 +1074,7 @@ void siw_destroy_cq(struct ib_cq *base_cq, struct ib_udata *udata)atomic_dec (&sdev->num_cq);vfree (cq->queue); +return 0 ; }/*
drivers/infiniband/sw/siw/siw_verbs.h +1 -1
@@ -62,7 +62,7 @@ int siw_post_send(struct ib_qp *base_qp, const struct ib_send_wr *wr, const struct ib_send_wr **bad_wr);int siw_post_receive (struct ib_qp *base_qp,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_wr); -void siw_destroy_cq (struct ib_cq *base_cq,struct ib_udata *udata); +int siw_destroy_cq (struct ib_cq *base_cq,struct ib_udata *udata);int siw_poll_cq (struct ib_cq *base_cq,int num_entries,struct ib_wc *wc);int siw_req_notify_cq (struct ib_cq *base_cq,enum ib_cq_notify_flags flags); struct ib_mr *siw_reg_user_mr(struct ib_pd *base_pd, u64 start, u64 len,
include/rdma/ib_verbs.h +4 -2
@@ -2429,7 +2429,7 @@ struct ib_device_ops {int (*create_cq)(struct ib_cq *cq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);int (*modify_cq)(struct ib_cq *cq,u16 cq_count,u16 cq_period); -void (*destroy_cq)(struct ib_cq *cq,struct ib_udata *udata); +int (*destroy_cq)(struct ib_cq *cq,struct ib_udata *udata);int (*resize_cq)(struct ib_cq *cq,int cqe,struct ib_udata *udata);struct ib_mr *(*get_dma_mr)(struct ib_pd *pd,int mr_access_flags); struct ib_mr *(*reg_user_mr)(struct ib_pd *pd, u64 start, u64 length,
@@ -3896,7 +3896,9 @@ int ib_destroy_cq_user(struct ib_cq *cq, struct ib_udata *udata);*/ static inline void ib_destroy_cq (struct ib_cq *cq) { -ib_destroy_cq_user (cq,NULL ); +int ret =ib_destroy_cq_user (cq,NULL ); + +WARN_ONCE (ret,"Destroy of kernel CQ shouldn't fail" ); }/**
--
2.26.2
PATCH 9/10 RDMA: Restore ability to return error for destroy WQ
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make this interface symmetrical to other destroy paths.
--- drivers/infiniband/core/uverbs_std_types_wq.c | 2 +- drivers/infiniband/core/verbs.c | 15 +++++++++------ drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +- drivers/infiniband/hw/mlx4/qp.c | 3 ++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/qp.c | 8 ++++++-- drivers/infiniband/hw/mlx5/qp.h | 4 ++-- drivers/infiniband/hw/mlx5/qpc.c | 5 +++-- include/rdma/ib_verbs.h | 4 ++-- 9 files changed, 27 insertions(+), 18 deletions(-)
drivers/infiniband/core/uverbs_std_types_wq.c +1 -1
@@ -16,7 +16,7 @@ static int uverbs_free_wq(struct ib_uobject *uobject, container_of(uobject, struct ib_uwq_object, uevent.uobject);int ret; - ret =ib_destroy_wq (wq, &attrs->driver_udata); + ret =ib_destroy_wq_user (wq, &attrs->driver_udata);if (ib_is_destroy_retryable (ret, why, uobject))return ret;
drivers/infiniband/core/verbs.c +9 -6
@@ -2399,25 +2399,28 @@ struct ib_wq *ib_create_wq(struct ib_pd *pd,EXPORT_SYMBOL (ib_create_wq);/** -* ib_destroy_wq - Destroys the specified user WQ. +* ib_destroy_wq_user - Destroys the specified user WQ. * @wq: The WQ to destroy. * @udata: Valid user data */ -int ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata) +int ib_destroy_wq_user (struct ib_wq *wq,struct ib_udata *udata) {struct ib_cq *cq = wq->cq;struct ib_pd *pd = wq->pd; +int ret;if (atomic_read (&wq->usecnt))return -EBUSY ; - wq->device->ops.destroy_wq (wq, udata); + ret = wq->device->ops.destroy_wq (wq, udata); +if (ret && udata) +return ret; +atomic_dec (&pd->usecnt);atomic_dec (&cq->usecnt); - -return 0 ; +return ret; } -EXPORT_SYMBOL (ib_destroy_wq); +EXPORT_SYMBOL (ib_destroy_wq_user);/** * ib_modify_wq - Modifies the specified WQ.
drivers/infiniband/hw/mlx4/mlx4_ib.h +1 -1
@@ -899,7 +899,7 @@ void mlx4_ib_sl2vl_update(struct mlx4_ib_dev *mdev, int port);struct ib_wq *mlx4_ib_create_wq (struct ib_pd *pd,struct ib_wq_init_attr *init_attr,struct ib_udata *udata); -void mlx4_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata); +int mlx4_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata);int mlx4_ib_modify_wq (struct ib_wq *wq,struct ib_wq_attr *wq_attr,u32 wq_attr_mask,struct ib_udata *udata);
drivers/infiniband/hw/mlx4/qp.c +2 -1
@@ -4327,7 +4327,7 @@ int mlx4_ib_modify_wq(struct ib_wq *ibwq, struct ib_wq_attr *wq_attr,return err; } -void mlx4_ib_destroy_wq (struct ib_wq *ibwq,struct ib_udata *udata) +int mlx4_ib_destroy_wq (struct ib_wq *ibwq,struct ib_udata *udata) {struct mlx4_ib_dev *dev =to_mdev (ibwq->device);struct mlx4_ib_qp *qp =to_mqp ((struct ib_qp *)ibwq);
@@ -4338,6 +4338,7 @@ void mlx4_ib_destroy_wq(struct ib_wq *ibwq, struct ib_udata *udata)destroy_qp_common (dev, qp,MLX4_IB_RWQ_SRC , udata);kfree (qp); +return 0 ; }struct ib_rwq_ind_table
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -1241,7 +1241,7 @@ int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask,struct ib_wq *mlx5_ib_create_wq (struct ib_pd *pd,struct ib_wq_init_attr *init_attr,struct ib_udata *udata); -void mlx5_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata); +int mlx5_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata);int mlx5_ib_modify_wq (struct ib_wq *wq,struct ib_wq_attr *wq_attr,u32 wq_attr_mask,struct ib_udata *udata); struct ib_rwq_ind_table *mlx5_ib_create_rwq_ind_table(struct ib_device *device,
drivers/infiniband/hw/mlx5/qp.c +6 -2
@@ -5085,14 +5085,18 @@ struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd,return ERR_PTR (err); } -void mlx5_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata) +int mlx5_ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata) {struct mlx5_ib_dev *dev =to_mdev (wq->device);struct mlx5_ib_rwq *rwq =to_mrwq (wq); +int ret; -mlx5_core_destroy_rq_tracked (dev, &rwq->core_qp); + ret =mlx5_core_destroy_rq_tracked (dev, &rwq->core_qp); +if (ret && udata) +return ret;destroy_user_rq (dev, wq->pd, rwq, udata);kfree (rwq); +return ret; } struct ib_rwq_ind_table *mlx5_ib_create_rwq_ind_table(struct ib_device *device,
drivers/infiniband/hw/mlx5/qp.h +2 -2
@@ -26,8 +26,8 @@ int mlx5_core_dct_query(struct mlx5_ib_dev *dev, struct mlx5_core_dct *dct,int mlx5_core_set_delay_drop (struct mlx5_ib_dev *dev,u32 timeout_usec); -void mlx5_core_destroy_rq_tracked (struct mlx5_ib_dev *dev, -struct mlx5_core_qp *rq); +int mlx5_core_destroy_rq_tracked (struct mlx5_ib_dev *dev, +struct mlx5_core_qp *rq);int mlx5_core_create_sq_tracked (struct mlx5_ib_dev *dev,u32 *in,int inlen,struct mlx5_core_qp *sq); void mlx5_core_destroy_sq_tracked(struct mlx5_ib_dev *dev,
drivers/infiniband/hw/mlx5/qpc.c +3 -2
@@ -576,11 +576,12 @@ int mlx5_core_create_rq_tracked(struct mlx5_ib_dev *dev, u32 *in, int inlen,return err; } -void mlx5_core_destroy_rq_tracked (struct mlx5_ib_dev *dev, -struct mlx5_core_qp *rq) +int mlx5_core_destroy_rq_tracked (struct mlx5_ib_dev *dev, +struct mlx5_core_qp *rq) {destroy_resource_common (dev, rq);destroy_rq_tracked (dev, rq->qpn, rq->uid); +return 0 ; }static void destroy_sq_tracked (struct mlx5_ib_dev *dev,u32 sqn,u16 uid)
include/rdma/ib_verbs.h +2 -2
@@ -2486,7 +2486,7 @@ struct ib_device_ops {struct ib_wq *(*create_wq)(struct ib_pd *pd,struct ib_wq_init_attr *init_attr,struct ib_udata *udata); -void (*destroy_wq)(struct ib_wq *wq,struct ib_udata *udata); +int (*destroy_wq)(struct ib_wq *wq,struct ib_udata *udata);int (*modify_wq)(struct ib_wq *wq,struct ib_wq_attr *attr,u32 wq_attr_mask,struct ib_udata *udata); struct ib_rwq_ind_table *(*create_rwq_ind_table)(
@@ -4322,7 +4322,7 @@ struct net_device *ib_device_netdev(struct ib_device *dev, u8 port);struct ib_wq *ib_create_wq (struct ib_pd *pd,struct ib_wq_init_attr *init_attr); -int ib_destroy_wq (struct ib_wq *wq,struct ib_udata *udata); +int ib_destroy_wq_user (struct ib_wq *wq,struct ib_udata *udata);int ib_modify_wq (struct ib_wq *wq,struct ib_wq_attr *attr,u32 wq_attr_mask);int ib_destroy_rwq_ind_table (struct ib_rwq_ind_table *wq_ind_table);
Signature · 1 line
2.26.2
PATCH 6/10 RDMA/core: Delete function indirection for alloc/free kernel CQ
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The ib_alloc_cq*() and ib_free_cq*() are solely kernel verbs to manage CQs and doesn't need extra indirection just to call same functions with constant parameter NULL as udata.
--- drivers/infiniband/core/cq.c | 33 ++++++++----------- include/rdma/ib_verbs.h | 62 ++++-------------------------------- 2 files changed, 20 insertions(+), 75 deletions(-)
drivers/infiniband/core/cq.c +14 -19
@@ -197,24 +197,22 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) }/** -* __ib_alloc_cq_user - allocate a completion queue +* __ib_alloc_cq allocate a completion queue * @dev: device to allocate the CQ for * @private: driver private data, accessible from cq->cq_context * @nr_cqe: number of CQEs to allocate * @comp_vector: HCA completion vectors for this CQ * @poll_ctx: context to poll the CQ from. * @caller: module owner name. -* @udata: Valid user data or NULL for kernel object * * This is the proper interface to allocate a CQ for in-kernel users. A * CQ allocated with this interface will automatically be polled from the * specified context. The ULP must use wr->wr_cqe instead of wr->wr_id * to use this CQ abstraction. */ -struct ib_cq *__ib_alloc_cq_user (struct ib_device *dev,void *private, -int nr_cqe,int comp_vector, -enum ib_poll_context poll_ctx, -const char *caller,struct ib_udata *udata) +struct ib_cq *__ib_alloc_cq (struct ib_device *dev,void *private,int nr_cqe, +int comp_vector,enum ib_poll_context poll_ctx, +const char *caller) {struct ib_cq_init_attr cq_attr = { .cqe = nr_cqe,
@@ -277,7 +275,7 @@ struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, out_destroy_cq:rdma_dim_destroy (cq);rdma_restrack_del (&cq->res); - cq->device->ops.destroy_cq (cq, udata); + cq->device->ops.destroy_cq (cq,NULL ); out_free_wc:kfree (cq->wc); out_free_cq:
@@ -285,7 +283,7 @@ struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private,trace_cq_alloc_error (nr_cqe, comp_vector, poll_ctx, ret);return ERR_PTR (ret); } -EXPORT_SYMBOL (__ib_alloc_cq_user); +EXPORT_SYMBOL (__ib_alloc_cq);/** * __ib_alloc_cq_any - allocate a completion queue
@@ -310,22 +308,19 @@ struct ib_cq *__ib_alloc_cq_any(struct ib_device *dev, void *private,atomic_inc_return (&counter) %min_t (int, dev->num_comp_vectors,num_online_cpus ()); -return __ib_alloc_cq_user (dev, private, nr_cqe, comp_vector, poll_ctx, - caller,NULL ); +return __ib_alloc_cq (dev, private, nr_cqe, comp_vector, poll_ctx, + caller); }EXPORT_SYMBOL (__ib_alloc_cq_any);/** -* ib_free_cq_user - free a completion queue +* ib_free_cq - free a completion queue * @cq: completion queue to free. -* @udata: User data or NULL for kernel object */ -void ib_free_cq_user (struct ib_cq *cq,struct ib_udata *udata) +void ib_free_cq (struct ib_cq *cq) { -if (WARN_ON_ONCE (atomic_read (&cq->usecnt))) -return ; -if (WARN_ON_ONCE (cq->cqe_used)) -return ; +WARN_ON_ONCE (atomic_read (&cq->usecnt)); +WARN_ON_ONCE (cq->cqe_used);switch (cq->poll_ctx) {case IB_POLL_DIRECT :
@@ -344,11 +339,11 @@ void ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata)rdma_dim_destroy (cq);trace_cq_free (cq);rdma_restrack_del (&cq->res); - cq->device->ops.destroy_cq (cq, udata); + cq->device->ops.destroy_cq (cq,NULL );kfree (cq->wc);kfree (cq); } -EXPORT_SYMBOL (ib_free_cq_user); +EXPORT_SYMBOL (ib_free_cq);void ib_cq_pool_init (struct ib_device *dev) {
include/rdma/ib_verbs.h +6 -56
78 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 1/10 RDMA: Restore ability to fail on PD deallocate
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The IB verbs objects are counted by the kernel and ib_core ensures that deallocate PD will success so it will be called once all other objects that depends on PD will be released. This is achieved by managing various reference counters on such objects.
The mlx5 driver didn't follow this standard flow when allowed DEVX objects that are not managed by ib_core to be interleaved with the ones under ib_core responsibility.
In such interleaved scenarios deallocate command can fail and ib_core will leave uobject in internal DB and attempt to clean it later to free resources anyway.
This change partially restores returned value from dealloc_pd() for all drivers, but keeping in mind that non-DEVX devices and kernel verbs paths shouldn't fail.
--- drivers/infiniband/core/uverbs_std_types.c | 3 +-- drivers/infiniband/core/verbs.c | 8 ++++++-- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 ++- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +- drivers/infiniband/hw/cxgb4/provider.c | 3 ++- drivers/infiniband/hw/efa/efa.h | 2 +- drivers/infiniband/hw/efa/efa_verbs.c | 3 ++- drivers/infiniband/hw/hns/hns_roce_device.h | 2 +- drivers/infiniband/hw/hns/hns_roce_pd.c | 3 ++- drivers/infiniband/hw/i40iw/i40iw_verbs.c | 3 ++- drivers/infiniband/hw/mlx4/main.c | 3 ++- drivers/infiniband/hw/mlx5/cmd.c | 4 ++-- drivers/infiniband/hw/mlx5/cmd.h | 2 +- drivers/infiniband/hw/mlx5/main.c | 4 ++-- drivers/infiniband/hw/mthca/mthca_provider.c | 3 ++- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 5 +++-- drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 2 +- drivers/infiniband/hw/qedr/verbs.c | 3 ++- drivers/infiniband/hw/qedr/verbs.h | 2 +- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 3 ++- drivers/infiniband/hw/usnic/usnic_ib_verbs.h | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c | 5 +++-- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 2 +- drivers/infiniband/sw/rdmavt/pd.c | 3 ++- drivers/infiniband/sw/rdmavt/pd.h | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.c | 3 ++- drivers/infiniband/sw/siw/siw_verbs.h | 2 +- include/rdma/ib_verbs.h | 13 +++++-------- 29 files changed, 56 insertions(+), 42 deletions(-)
drivers/infiniband/core/uverbs_std_types.c +1 -2
@@ -122,8 +122,7 @@ static int uverbs_free_pd(struct ib_uobject *uobject,if (ret)return ret; -ib_dealloc_pd_user (pd, &attrs->driver_udata); -return 0 ; +return ib_dealloc_pd_user (pd, &attrs->driver_udata); }void ib_uverbs_free_event_queue (struct ib_uverbs_event_queue *event_queue)
drivers/infiniband/core/verbs.c +6 -2
@@ -329,7 +329,7 @@ EXPORT_SYMBOL(__ib_alloc_pd);* exist. The caller is responsible to synchronously destroy them and * guarantee no new allocations will happen. */ -void ib_dealloc_pd_user (struct ib_pd *pd,struct ib_udata *udata) +int ib_dealloc_pd_user (struct ib_pd *pd,struct ib_udata *udata) {int ret;
@@ -343,9 +343,13 @@ void ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata)requires the caller to guarantee we can't race here. */ WARN_ON (atomic_read (&pd->usecnt)); + ret = pd->device->ops.dealloc_pd (pd, udata); +if (ret && udata) +return ret; +rdma_restrack_del (&pd->res); - pd->device->ops.dealloc_pd (pd, udata);kfree (pd); +return ret; }EXPORT_SYMBOL (ib_dealloc_pd_user);
drivers/infiniband/hw/bnxt_re/ib_verbs.c +2 -1
@@ -532,7 +532,7 @@ static int bnxt_re_create_fence_mr(struct bnxt_re_pd *pd) }/* Protection Domains */ -void bnxt_re_dealloc_pd (struct ib_pd *ib_pd,struct ib_udata *udata) +int bnxt_re_dealloc_pd (struct ib_pd *ib_pd,struct ib_udata *udata) { struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);struct bnxt_re_dev *rdev = pd->rdev;
@@ -542,6 +542,7 @@ void bnxt_re_dealloc_pd(struct ib_pd *ib_pd, struct ib_udata *udata)if (pd->qplib_pd.id)bnxt_qplib_dealloc_pd (&rdev->qplib_res, &rdev->qplib_res.pd_tbl, &pd->qplib_pd); +return 0 ; }int bnxt_re_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata)
drivers/infiniband/hw/bnxt_re/ib_verbs.h +1 -1
@@ -163,7 +163,7 @@ int bnxt_re_query_gid(struct ib_device *ibdev, u8 port_num,enum rdma_link_layer bnxt_re_get_link_layer (struct ib_device *ibdev,u8 port_num);int bnxt_re_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void bnxt_re_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata); +int bnxt_re_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);int bnxt_re_create_ah (struct ib_ah *ah,struct rdma_ah_init_attr *init_attr,struct ib_udata *udata);int bnxt_re_modify_ah (struct ib_ah *ah,struct rdma_ah_attr *ah_attr);
drivers/infiniband/hw/cxgb4/provider.c +2 -1
@@ -190,7 +190,7 @@ static int c4iw_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)return ret; } -static void c4iw_deallocate_pd (struct ib_pd *pd,struct ib_udata *udata) +static int c4iw_deallocate_pd (struct ib_pd *pd,struct ib_udata *udata) {struct c4iw_dev *rhp;struct c4iw_pd *php;
@@ -202,6 +202,7 @@ static void c4iw_deallocate_pd(struct ib_pd *pd, struct ib_udata *udata)mutex_lock (&rhp->rdev.stats.lock); rhp->rdev.stats.pd.cur--;mutex_unlock (&rhp->rdev.stats.lock); +return 0 ; }static int c4iw_allocate_pd (struct ib_pd *pd,struct ib_udata *udata)
drivers/infiniband/hw/efa/efa.h +1 -1
@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,
drivers/infiniband/hw/efa/efa_verbs.c +2 -1
@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; } -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }static int efa_destroy_qp_handle (struct efa_dev *dev,u32 qp_handle)
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
@@ -1183,7 +1183,7 @@ int hns_roce_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);void hns_roce_destroy_ah (struct ib_ah *ah,u32 flags);int hns_roce_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void hns_roce_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata); +int hns_roce_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);struct ib_mr *hns_roce_get_dma_mr (struct ib_pd *pd,int acc); struct ib_mr *hns_roce_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
drivers/infiniband/hw/hns/hns_roce_pd.c +2 -1
@@ -82,9 +82,10 @@ int hns_roce_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -void hns_roce_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +int hns_roce_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {hns_roce_pd_free (to_hr_dev (pd->device),to_hr_pd (pd)->pdn); +return 0 ; }int hns_roce_uar_alloc (struct hns_roce_dev *hr_dev,struct hns_roce_uar *uar)
drivers/infiniband/hw/i40iw/i40iw_verbs.c +2 -1
@@ -328,12 +328,13 @@ static int i40iw_alloc_pd(struct ib_pd *pd, struct ib_udata *udata)* @ibpd: ptr of pd to be deallocated * @udata: user data or null for kernel object */ -static void i40iw_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +static int i40iw_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct i40iw_pd *iwpd =to_iwpd (ibpd);struct i40iw_device *iwdev =to_iwdev (ibpd->device);i40iw_rem_pdusecount (iwpd, iwdev); +return 0 ; }/**
drivers/infiniband/hw/mlx4/main.c +2 -1
@@ -1214,9 +1214,10 @@ static int mlx4_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -static void mlx4_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +static int mlx4_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {mlx4_pd_free (to_mdev (pd->device)->dev,to_mpd (pd)->pdn); +return 0 ; }static int mlx4_ib_alloc_xrcd (struct ib_xrcd *ibxrcd,struct ib_udata *udata)
drivers/infiniband/hw/mlx5/cmd.c +2 -2
@@ -209,14 +209,14 @@ void mlx5_cmd_dealloc_transport_domain(struct mlx5_core_dev *dev, u32 tdn,mlx5_cmd_exec_in (dev, dealloc_transport_domain, in); } -void mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid) +int mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid) {u32 in[MLX5_ST_SZ_DW (dealloc_pd_in)] = {};MLX5_SET (dealloc_pd_in, in, opcode,MLX5_CMD_OP_DEALLOC_PD );MLX5_SET (dealloc_pd_in, in, pd, pdn);MLX5_SET (dealloc_pd_in, in, uid, uid); -mlx5_cmd_exec_in (dev, dealloc_pd, in); +return mlx5_cmd_exec_in (dev, dealloc_pd, in); } int mlx5_cmd_attach_mcg(struct mlx5_core_dev *dev, union ib_gid *mgid,
drivers/infiniband/hw/mlx5/cmd.h +1 -1
@@ -44,7 +44,7 @@ int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,int mlx5_cmd_alloc_memic (struct mlx5_dm *dm,phys_addr_t *addr,u64 length,u32 alignment);void mlx5_cmd_dealloc_memic (struct mlx5_dm *dm,phys_addr_t addr,u64 length); -void mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid); +int mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid);void mlx5_cmd_destroy_tir (struct mlx5_core_dev *dev,u32 tirn,u16 uid);void mlx5_cmd_destroy_tis (struct mlx5_core_dev *dev,u32 tisn,u16 uid);void mlx5_cmd_destroy_rqt (struct mlx5_core_dev *dev,u32 rqtn,u16 uid);
drivers/infiniband/hw/mlx5/main.c +2 -2
@@ -2569,12 +2569,12 @@ static int mlx5_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -static void mlx5_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +static int mlx5_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {struct mlx5_ib_dev *mdev =to_mdev (pd->device);struct mlx5_ib_pd *mpd =to_mpd (pd); -mlx5_cmd_dealloc_pd (mdev->mdev, mpd->pdn, mpd->uid); +return mlx5_cmd_dealloc_pd (mdev->mdev, mpd->pdn, mpd->uid); }static int mlx5_ib_mcg_attach (struct ib_qp *ibqp,union ib_gid *gid,u16 lid)
drivers/infiniband/hw/mthca/mthca_provider.c +2 -1
@@ -373,9 +373,10 @@ static int mthca_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -static void mthca_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +static int mthca_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {mthca_pd_free (to_mdev (pd->device),to_mpd (pd)); +return 0 ; } static int mthca_ah_create(struct ib_ah *ibah,
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +3 -2
@@ -664,7 +664,7 @@ int ocrdma_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return status; } -void ocrdma_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int ocrdma_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct ocrdma_pd *pd =get_ocrdma_pd (ibpd);struct ocrdma_dev *dev =get_ocrdma_dev (ibpd->device);
@@ -682,10 +682,11 @@ void ocrdma_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)if (is_ucontext_pd (uctx, pd)) {ocrdma_release_ucontext_pd (uctx); -return ; +return 0 ; } }_ocrdma_dealloc_pd (dev, pd); +return 0 ; } static int ocrdma_alloc_lkey(struct ocrdma_dev *dev, struct ocrdma_mr *mr,
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h +1 -1
@@ -67,7 +67,7 @@ void ocrdma_dealloc_ucontext(struct ib_ucontext *uctx);int ocrdma_mmap (struct ib_ucontext *,struct vm_area_struct *vma);int ocrdma_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void ocrdma_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata); +int ocrdma_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);int ocrdma_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);
drivers/infiniband/hw/qedr/verbs.c +2 -1
@@ -471,13 +471,14 @@ int qedr_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -void qedr_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int qedr_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct qedr_dev *dev =get_qedr_dev (ibpd->device);struct qedr_pd *pd =get_qedr_pd (ibpd);DP_DEBUG (dev,QEDR_MSG_INIT ,"Deallocating PD %d\n" , pd->pd_id); dev->ops->rdma_dealloc_pd (dev->rdma_ctx, pd->pd_id); +return 0 ; } static void qedr_free_pbl(struct qedr_dev *dev,
drivers/infiniband/hw/qedr/verbs.h +1 -1
@@ -47,7 +47,7 @@ void qedr_dealloc_ucontext(struct ib_ucontext *uctx);int qedr_mmap (struct ib_ucontext *ucontext,struct vm_area_struct *vma);void qedr_mmap_free (struct rdma_user_mmap_entry *rdma_entry);int qedr_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void qedr_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata); +int qedr_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);int qedr_create_cq (struct ib_cq *ibcq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);
drivers/infiniband/hw/usnic/usnic_ib_verbs.c +2 -1
@@ -460,9 +460,10 @@ int usnic_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return 0 ; } -void usnic_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +int usnic_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {usnic_uiom_dealloc_pd ((to_upd (pd))->umem_pd); +return 0 ; } struct ib_qp *usnic_ib_create_qp(struct ib_pd *pd,
drivers/infiniband/hw/usnic/usnic_ib_verbs.h +1 -1
@@ -51,7 +51,7 @@ int usnic_ib_query_gid(struct ib_device *ibdev, u8 port, int index,int usnic_ib_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int usnic_ib_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void usnic_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata); +int usnic_ib_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata);struct ib_qp *usnic_ib_create_qp (struct ib_pd *pd,struct ib_qp_init_attr *init_attr,struct ib_udata *udata);
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c +3 -2
@@ -479,9 +479,9 @@ int pvrdma_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)* @pd: the protection domain to be released * @udata: user data or null for kernel object * -* @return: 0 on success, otherwise errno. +* @return: Always 0 */ -void pvrdma_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +int pvrdma_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {struct pvrdma_dev *dev =to_vdev (pd->device);union pvrdma_cmd_req req = {};
@@ -498,6 +498,7 @@ void pvrdma_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata) ret);atomic_dec (&dev->num_pds); +return 0 ; }/**
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h +1 -1
@@ -399,7 +399,7 @@ int pvrdma_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);int pvrdma_alloc_ucontext (struct ib_ucontext *uctx,struct ib_udata *udata);void pvrdma_dealloc_ucontext (struct ib_ucontext *context);int pvrdma_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void pvrdma_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int pvrdma_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);struct ib_mr *pvrdma_get_dma_mr (struct ib_pd *pd,int acc); struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, u64 virt_addr, int access_flags,
drivers/infiniband/sw/rdmavt/pd.c +2 -1
@@ -95,11 +95,12 @@ int rvt_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)* * Return: always 0 */ -void rvt_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int rvt_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct rvt_dev_info *dev =ib_to_rvt (ibpd->device);spin_lock (&dev->n_pds_lock); dev->n_pds_allocated--;spin_unlock (&dev->n_pds_lock); +return 0 ; }
drivers/infiniband/sw/rdmavt/pd.h +1 -1
@@ -51,6 +51,6 @@#include <rdma/rdma_vt.h> int rvt_alloc_pd (struct ib_pd *pd,struct ib_udata *udata); -void rvt_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int rvt_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);#endif /* DEF_RDMAVTPD_H */
drivers/infiniband/sw/rxe/rxe_verbs.c +2 -1
@@ -175,11 +175,12 @@ static int rxe_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return rxe_add_to_pool (&rxe->pd_pool, &pd->pelem); } -static void rxe_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +static int rxe_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct rxe_pd *pd =to_rpd (ibpd);rxe_drop_ref (pd); +return 0 ; } static int rxe_create_ah(struct ib_ah *ibah,
drivers/infiniband/sw/siw/siw_verbs.c +2 -1
@@ -234,12 +234,13 @@ int siw_alloc_pd(struct ib_pd *pd, struct ib_udata *udata)return 0 ; } -void siw_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) +int siw_dealloc_pd (struct ib_pd *pd,struct ib_udata *udata) {struct siw_device *sdev =to_siw_dev (pd->device);siw_dbg_pd (pd,"free PD\n" );atomic_dec (&sdev->num_pd); +return 0 ; }void siw_qp_get_ref (struct ib_qp *base_qp)
drivers/infiniband/sw/siw/siw_verbs.h +1 -1
@@ -49,7 +49,7 @@ int siw_query_port(struct ib_device *base_dev, u8 port,int siw_query_gid (struct ib_device *base_dev,u8 port,int idx,union ib_gid *gid);int siw_alloc_pd (struct ib_pd *base_pd,struct ib_udata *udata); -void siw_dealloc_pd (struct ib_pd *base_pd,struct ib_udata *udata); +int siw_dealloc_pd (struct ib_pd *base_pd,struct ib_udata *udata);struct ib_qp *siw_create_qp (struct ib_pd *base_pd,struct ib_qp_init_attr *attr,struct ib_udata *udata);
include/rdma/ib_verbs.h +5 -8
@@ -2404,7 +2404,7 @@ struct ib_device_ops {void (*mmap_free)(struct rdma_user_mmap_entry *entry);void (*disassociate_ucontext)(struct ib_ucontext *ibcontext);int (*alloc_pd)(struct ib_pd *pd,struct ib_udata *udata); -void (*dealloc_pd)(struct ib_pd *pd,struct ib_udata *udata); +int (*dealloc_pd)(struct ib_pd *pd,struct ib_udata *udata);int (*create_ah)(struct ib_ah *ah,struct rdma_ah_init_attr *attr,struct ib_udata *udata);int (*modify_ah)(struct ib_ah *ah,struct rdma_ah_attr *ah_attr);
@@ -3462,12 +3462,7 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags,#define ib_alloc_pd (device, flags) \ __ib_alloc_pd((device), (flags), KBUILD_MODNAME) -/** -* ib_dealloc_pd_user - Deallocate kernel/user PD -* @pd: The protection domain -* @udata: Valid user data or NULL for kernel objects -*/ -void ib_dealloc_pd_user (struct ib_pd *pd,struct ib_udata *udata); +int ib_dealloc_pd_user (struct ib_pd *pd,struct ib_udata *udata);/** * ib_dealloc_pd - Deallocate kernel PD
@@ -3477,7 +3472,9 @@ void ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata);*/ static inline void ib_dealloc_pd (struct ib_pd *pd) { -ib_dealloc_pd_user (pd,NULL ); +int ret =ib_dealloc_pd_user (pd,NULL ); + +WARN_ONCE (ret,"Destroy of kernel PD shouldn't fail" ); }enum rdma_create_ah_flags {
--
2.26.2
Re: [PATCH rdma-next 02/10] RDMA: Restore ability to fail on AH destroy
In reply to Leon RomanovskyQuoted 8 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Like any other IB verbs objects, AH are refcounted by
ib_core. The release of those objects is controlled byib_corewith promise that AH destroy can't fail.Being SW object for now, this change makes
dealloc_ah()to behave like any other destroy IB flows.
Maybe I'm misreading this, but AH isn't necessarily a software object. It's a HW object as well in some of the drivers.
Quoted 14 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-1@@ -1875,7 +1875,7 @@ int efa_create_ah(struct ib_ah *ibah,return err; } -void efa_destroy_ah (struct ib_ah *ibah,u32 flags) +int efa_destroy_ah (struct ib_ah *ibah,u32 flags) {struct efa_dev *dev =to_edev (ibah->pd->device);struct efa_ah *ah =to_eah (ibah);
drivers/infiniband/hw/efa/efa_verbs.c void efa_destroy_ah(struct ib_ah *ibah, u32 flags) · line 1888
Quoted 5 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-1@@ -1885,10 +1885,11 @@ void efa_destroy_ah(struct ib_ah *ibah, u32 flags)if (!(flags &RDMA_DESTROY_AH_SLEEPABLE )) {ibdev_dbg (&dev->ibdev,"Destroy address handle is not supported in atomic context\n" ); -return ; +return -EINVAL ;
-EOPNOTSUPP.
Quoted 5 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-0}efa_ah_destroy (dev, ah); +return 0 ; }
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Leon RomanovskyQuoted 21 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +1
-1@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +0
-0@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; }
drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 10 lines
drivers/infiniband/hw/efa/efa_verbs.c +2
-1-void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }
Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Re: [PATCH rdma-next 02/10] RDMA: Restore ability to fail on AH destroy
In reply to Gal PressmanQuoted 8 lines · Gal Pressman
Quoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Like any other IB verbs objects, AH are refcounted by
ib_core. The release of those objects is controlled byib_corewith promise that AH destroy can't fail.
Quoted 5 lines
Quoted 2 lines
Being SW object for now, this change makes
dealloc_ah()to behave like any other destroy IB flows.Maybe I'm misreading this, but AH isn't necessarily a software object. It's a HW object as well in some of the drivers.
We are working to introduce AH HW object.
Quoted 9 lines
Quoted 9 lines
drivers/infiniband/hw/efa/efa_verbs.c +0
-0@@ -1875,7 +1875,7 @@ int efa_create_ah(struct ib_ah *ibah,return err; }
Quoted 13 lines
drivers/infiniband/hw/efa/efa_verbs.c void efa_destroy_ah(struct ib_ah *ibah, u32 flags) · line 1888
Quoted 11 lines
drivers/infiniband/hw/efa/efa_verbs.c +2
-2-void efa_destroy_ah (struct ib_ah *ibah,u32 flags) +int efa_destroy_ah (struct ib_ah *ibah,u32 flags) {struct efa_dev *dev =to_edev (ibah->pd->device);struct efa_ah *ah =to_eah (ibah);@@ -1885,10 +1885,11 @@ void efa_destroy_ah(struct ib_ah *ibah, u32 flags)if (!(flags &RDMA_DESTROY_AH_SLEEPABLE )) {ibdev_dbg (&dev->ibdev,"Destroy address handle is not supported in atomic context\n" ); -return ; +return -EINVAL ;
-EOPNOTSUPP.
Thanks, I'll fix.
Quoted 6 lines
Quoted 6 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-0}efa_ah_destroy (dev, ah); +return 0 ; }
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Gal PressmanQuoted 29 lines · Gal Pressman
Quoted 27 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +1
-1@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +1
-1@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; } -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);
Quoted 8 lines
drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 4 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-0ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
The problem with returning error is that ib_core will retrigger deallocation again,while it was supposed to succeed from the first try. See our discussion over bnxt_re patch.
https://lore.kernel.org/linux-rdma/[email protected]
Thanks
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Gal PressmanQuoted 26 lines · Gal Pressman
Quoted 25 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +2
-1+++ b/drivers/infiniband/hw/efa/efa.h@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +2
-1+++ b/drivers/infiniband/hw/efa/efa_verbs.c@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; } -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);
Quoted 8 lines
drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 4 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-0ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Drivers can't fail the destroy unless a future destroy will succeed. it breaks everything to do that.
That is why the return codes were removed.
Unfortunately there are cases where future destroy will succeed, so here we are :(
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
Jason
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 36 lines · Jason Gunthorpe
Quoted 26 lines · Gal Pressman
Quoted 25 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +2
-1+++ b/drivers/infiniband/hw/efa/efa.h@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +2
-1+++ b/drivers/infiniband/hw/efa/efa_verbs.c@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; } -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);Quoted 8 lines
drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 4 lines
drivers/infiniband/hw/efa/efa_verbs.c +1
-0ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Quoted 2 lines
Drivers can't fail the destroy unless a future destroy will succeed. it breaks everything to do that.
What does it break?
Quoted 4 lines
That is why the return codes were removed.
Unfortunately there are cases where future destroy will succeed, so here we are :(
I replied in the other thread as well, we can continue the discussion here.
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
These kinds of things aren't really expected so I don't mind what we end up with, but in the rare cases they do I would prefer not to ignore such errors, and yea, give the destroy command another shot when the process is cleaned up.
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 37 lines · Gal Pressman
Quoted 36 lines · Jason Gunthorpe
Quoted 34 lines · Gal Pressman
Quoted 19 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +2
-1+++ b/drivers/infiniband/hw/efa/efa.h@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +1
-0+++ b/drivers/infiniband/hw/efa/efa_verbs.c@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; }drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 10 lines
drivers/infiniband/hw/efa/efa_verbs.c +2
-1-void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Quoted 4 lines
Quoted 2 lines
Drivers can't fail the destroy unless a future destroy will succeed. it breaks everything to do that.
What does it break?
It will break ucontext cleanup flow exactly where you pointed, e.g. uverbs_destroy_ufile_hw(). We release all objects and memory associated with that ucontext to ensure that no memory leaks.
Quoted 11 lines
Quoted 5 lines
That is why the return codes were removed.
Unfortunately there are cases where future destroy will succeed, so here we are :(
I replied in the other thread as well, we can continue the discussion here.
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
Because DevX flow guarantees that this second call will succeed.
Quoted 9 lines
These kinds of things aren't really expected so I don't mind what we end up with, but in the rare cases they do I would prefer not to ignore such errors, and yea, give the destroy command another shot when the process is cleaned up.
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
I think that it is not different than any PCI error handling.
Thanks
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Gal PressmanQuoted 37 lines · Gal Pressman
Quoted 36 lines · Jason Gunthorpe
Quoted 34 lines · Gal Pressman
Quoted 19 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +2
-1+++ b/drivers/infiniband/hw/efa/efa.h@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +1
-0+++ b/drivers/infiniband/hw/efa/efa_verbs.c@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; }drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 10 lines
drivers/infiniband/hw/efa/efa_verbs.c +2
-1-void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Quoted 4 lines
Quoted 2 lines
Drivers can't fail the destroy unless a future destroy will succeed. it breaks everything to do that.
What does it break?
For uverbs it will go into an infinite loop in uverbs_destroy_ufile_hw() if destroy doesn't eventually succeed.
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
There is no way to propogate them.
All destroy must eventually succeed.
Quoted 4 lines
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
The zap stuff revokes the BAR mmaping, it triggerst device fatal to userspace and that is mostly it for userspace..
It is more complicated for kernel users
Jason
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 38 lines · Jason Gunthorpe
Quoted 37 lines · Gal Pressman
Quoted 36 lines · Jason Gunthorpe
Quoted 34 lines · Gal Pressman
Quoted 19 lines · Leon Romanovsky
drivers/infiniband/hw/efa/efa.h +2
-1+++ b/drivers/infiniband/hw/efa/efa.h@@ -134,7 +134,7 @@ int efa_query_gid(struct ib_device *ibdev, u8 port, int index,int efa_query_pkey (struct ib_device *ibdev,u8 port,u16 index,u16 *pkey);int efa_alloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); -void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata); +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata);int efa_destroy_qp (struct ib_qp *ibqp,struct ib_udata *udata); struct ib_qp *efa_create_qp(struct ib_pd *ibpd, struct ib_qp_init_attr *init_attr,drivers/infiniband/hw/efa/efa_verbs.c +1
-0+++ b/drivers/infiniband/hw/efa/efa_verbs.c@@ -383,13 +383,14 @@ int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)return err; }drivers/infiniband/hw/efa/efa_verbs.c int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) · line 394
Quoted 10 lines
drivers/infiniband/hw/efa/efa_verbs.c +2
-1-void efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) +int efa_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata) {struct efa_dev *dev =to_edev (ibpd->device);struct efa_pd *pd =to_epd (ibpd);ibdev_dbg (&dev->ibdev,"Dealloc pd[%d]\n" , pd->pdn);efa_pd_dealloc (dev, pd->pdn); +return 0 ; }Nice change, thanks Leon.
At least for EFA, I prefer to return the return value of the destroy command instead of silently ignoring it (same for the other patches).
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Drivers can't fail the destroy unless a future destroy will succeed. it breaks everything to do that.
What does it break?
For uverbs it will go into an infinite loop in
uverbs_destroy_ufile_hw()if destroy doesn't eventually succeed.
The code breaks the loop in such cases, why infinite loop?
Quoted 9 lines
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
There is no way to propogate them.
All destroy must eventually succeed.
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation. The cleanup flow can either ignore the return value, or we can add another parameter that explicitly means the call shouldn't fail and all allocated memory/state should be freed.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
The zap stuff revokes the BAR mmaping, it triggerst device fatal to userspace and that is mostly it for userspace..
Interesting, is there a reference driver that does that today?
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 4 lines · Gal Pressman
Quoted 2 lines
For uverbs it will go into an infinite loop in
uverbs_destroy_ufile_hw()if destroy doesn't eventually succeed.The code breaks the loop in such cases, why infinite loop?
Oh, that is a bug, it should WARN_ON when that happens, because the driver has triggered a permanent memory leak.
Quoted 6 lines
Quoted 6 lines
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
Quoted 7 lines
Quoted 3 lines
There is no way to propogate them.
All destroy must eventually succeed.
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation.
It is sort of OK for a driver to fail during RDMA_REMOVE_DESTROY.
All other reason codes must eventually succeed.
Quoted 3 lines
The cleanup flow can either ignore the return value, or we can add another parameter that explicitly means the call shouldn't fail and all allocated memory/state should be freed.
I don't really see the value to return the error code to userspace, it would require churning all the drivers and all the destroy functions to pass the existing reason in.
Since all the details of the FW failure reason are lost to some EINVAL (or already logged to dmesg) I don't see much point.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
The zap stuff revokes the BAR mmaping, it triggerst device fatal to userspace and that is mostly it for userspace..
Interesting, is there a reference driver that does that today?
I think both mlx drivers and hns do? Seeuverbs_user_mmap_disassociate()
Jason
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 7 lines · Jason Gunthorpe
Quoted 3 lines
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation.
It is sort of OK for a driver to fail during
RDMA_REMOVE_DESTROY.All other reason codes must eventually succeed.
Actually 'eventually succeed' is not even quite right
There are situations where a driver object will be created and then destroyed during error unwind of the syscall eg via RDMA_REMOVE_ABORT_HWOBJ
In which case, again, destroy must succeed or memory gets leaked and a WARN_ON trigger. This might be a problem still in the devx stuff :\
Jason
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Jason GunthorpeQuoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Gal Pressman
Quoted 2 lines
For uverbs it will go into an infinite loop in
uverbs_destroy_ufile_hw()if destroy doesn't eventually succeed.The code breaks the loop in such cases, why infinite loop?
Oh, that is a bug, it should
WARN_ONwhen that happens, because the driver has triggered a permanent memory leak.
Well, a WARN_ON won't do much good if you're stuck in an infinite loop :), the break is definitely needed there.
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the drivers?
Quoted 7 lines
Quoted 3 lines
There is no way to propogate them.
All destroy must eventually succeed.
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation.
It is sort of OK for a driver to fail during
RDMA_REMOVE_DESTROY.All other reason codes must eventually succeed.
Quoted 3 lines
The cleanup flow can either ignore the return value, or we can add another parameter that explicitly means the call shouldn't fail and all allocated memory/state should be freed.
Quoted 6 lines
I don't really see the value to return the error code to userspace, it would require churning all the drivers and all the destroy functions to pass the existing reason in.
Since all the details of the FW failure reason are lost to some EINVAL (or already logged to dmesg) I don't see much point.
Right, as always, the error code would probably not contain much information, but there's a big difference between returning error code X/Y vs returning success instead of an error. To me that just feels wrong, at least in cases where we can prevent that.
Won't argue about the churn, it's a lot of work for a "small" benefit in the rare error cases. Can't we leave it up for the individual driver to decide whether it chooses to support that or not? i.e. default behavior would be that failures are not allowed, and a driver can opt to respect the extra parameter (kinda like how DEVX "supports" failing)?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
If the chip fails a destroy when it should not then it has failed and should be disabled at PCI and reset, continuing to free anyhow.
How do we reset the device when there are active apps using it?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The zap stuff revokes the BAR mmaping, it triggerst device fatal to userspace and that is mostly it for userspace..
Interesting, is there a reference driver that does that today?
I think both mlx drivers and hns do? See
uverbs_user_mmap_disassociate()
Thanks!
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 4 lines · Gal Pressman
Right, as always, the error code would probably not contain much information, but there's a big difference between returning error code X/Y vs returning success instead of an error. To me that just feels wrong, at least in cases where we can prevent that.
From user perspective it is a success. The purpose of the destroy kernel command is to allow userspace to release the memory underlying the object.
The only valid reason to return a failure from destroy to userspace is if userspace has a programming error - eg destroying a PD while QPs exist does fail today for all drivers.
Userspace should treat destroy failure as a serious error and crash - there is no possible recovery from it beyond leaking memory.
Userspace should NOT be exposed to device failure via destroy. Failed devices trigger an ASYNC EVENT and destroy MUST succeed after the device fatal event.
Jason
RE: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Gal PressmanQuoted 43 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Gal Pressman
Quoted 2 lines
For uverbs it will go into an infinite loop in
uverbs_destroy_ufile_hw()if destroy doesn't eventually succeed.The code breaks the loop in such cases, why infinite loop?
Oh, that is a bug, it should
WARN_ONwhen that happens, because the driver has triggered a permanent memory leak.Well, a
WARN_ONwon't do much good if you're stuck in an infinite loop :), the break is definitely needed there.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the
drivers?
Quoted 21 lines
Quoted 8 lines
Quoted 4 lines
There is no way to propogate them.
All destroy must eventually succeed.
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation.
It is sort of OK for a driver to fail during
RDMA_REMOVE_DESTROY.All other reason codes must eventually succeed.
Quoted 3 lines
The cleanup flow can either ignore the return value, or we can add another parameter that explicitly means the call shouldn't fail and all allocated memory/state should be freed.
I don't really see the value to return the error code to userspace, it would require churning all the drivers and all the destroy functions to pass the existing reason in.
Quoted 8 lines
Quoted 2 lines
Since all the details of the FW failure reason are lost to some EINVAL (or already logged to dmesg) I don't see much point.
Right, as always, the error code would probably not contain much information, but there's a big difference between returning error code X/Y vs returning success instead of an error. To me that just feels wrong, at least in cases where we can prevent that.
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
Do we really want to have mixed/ambiguous definition of the API to support the quirks of one type of device?
Shiraz
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 52 lines · Shiraz
Quoted 43 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Gal Pressman
Quoted 2 lines
For uverbs it will go into an infinite loop in
uverbs_destroy_ufile_hw()if destroy doesn't eventually succeed.The code breaks the loop in such cases, why infinite loop?
Oh, that is a bug, it should
WARN_ONwhen that happens, because the driver has triggered a permanent memory leak.Well, a
WARN_ONwon't do much good if you're stuck in an infinite loop :), the break is definitely needed there.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
For kernel it will trigger WARN_ON's and then a permanent memory leak.
Quoted 3 lines
I agree that drivers shouldn't fail destroy commands, but you know.. bugs/errors happen (especially when dealing with hardware), and we have a way to propagate them, why do it for only some of the
drivers?
Quoted 21 lines
Quoted 8 lines
Quoted 4 lines
There is no way to propogate them.
All destroy must eventually succeed.
There is no way to propagate them on process cleanup, but the destroy verbs have a return code all the way back to libibverbs, which we can use for error propagation.
It is sort of OK for a driver to fail during
RDMA_REMOVE_DESTROY.All other reason codes must eventually succeed.
Quoted 3 lines
The cleanup flow can either ignore the return value, or we can add another parameter that explicitly means the call shouldn't fail and all allocated memory/state should be freed.
I don't really see the value to return the error code to userspace, it would require churning all the drivers and all the destroy functions to pass the existing reason in.
Quoted 8 lines
Quoted 2 lines
Since all the details of the FW failure reason are lost to some EINVAL (or already logged to dmesg) I don't see much point.
Right, as always, the error code would probably not contain much information, but there's a big difference between returning error code X/Y vs returning success instead of an error. To me that just feels wrong, at least in cases where we can prevent that.
Quoted 4 lines
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
Do we really want to have mixed/ambiguous definition of the API to support the quirks of one type of device?
This is in-kernel API and it can be imperfect, because we are not bounded to not-break-userspace rule.
I don't like the current situation either, just don't know how to solve it differently.
Thanks
Quoted 3 lines
Shiraz
Jason Gunthorpe
maintainerRe: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Saleem, ShirazQuoted 3 lines · Shiraz
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
More or less, drivers can only return -EAGAIN with the idea that a future call during the close process will eventually succeed.
Any permanent failure will trigger WARN_ON and a memory leak
Maybe we should switch the return code to bool or something to be a little clearer that it is request to retry, not a failure?
Jason
RE: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 9 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 3 lines · Shiraz
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
Quoted 8 lines
More or less, drivers can only return
-EAGAINwith the idea that a future call during the close process will eventually succeed.Any permanent failure will trigger
WARN_ONand a memory leakMaybe we should switch the return code to bool or something to be a little clearer that it is request to retry, not a failure?
There is no retry for kernel object destroy right? So not sure bool is making It clearer.
I am not very familiar with devx flows but doesn’t it bypass the ib verbs layer altogether? i.e. mlx5_ib_dealloc_pd isn’t directly called in the devx flows no? So why changes its return type and other provider destroy callbacks?
But lets go down the path that we really need a return code in the destroy APIs to solve this problem. For one I don’t see how we can say its meant exclusively for devx drivers to use for a fail. Also can we really claim the API contract is that driver can fail a destroy given a future destroy will succeed? Since the kernel destroy has no retry.
Which then boils down do we just keep a simpler definition of the API contract -- driver can just return whatever the true error code is? i.e. if it wants a retry, use EAGAIN. If it has a non recoverable device error, then reset the device, clean up the resources but return ENOTRECOVERABLE. ib_core can enable the retry logic for EAGAIN _only_. For other error codes, Ib_core can trigger a warn_on or something to indicate permanent failure. It can also pass on ret_code to user-space as its doing today?
Shiraz
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 12 lines · Shiraz
Quoted 12 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 3 lines · Shiraz
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
More or less, drivers can only return
-EAGAINwith the idea that a future call during the close process will eventually succeed.
Quoted 6 lines
Quoted 5 lines
Any permanent failure will trigger
WARN_ONand a memory leakMaybe we should switch the return code to bool or something to be a little clearer that it is request to retry, not a failure?
There is no retry for kernel object destroy right? So not sure bool is making It clearer.
Right, kernel verbs users don't know how to deal with destroy failure and designed to ensure that destroy always success.
Quoted 4 lines
I am not very familiar with devx flows but doesn’t it bypass the ib verbs layer altogether? i.e.
mlx5_ib_dealloc_pdisn’t directly called in the devx flows no? So why changes its return type and other provider destroy callbacks?
DevX itself indeed bypasses ib_core and as a standalone feature doesn't need any changes in destroys. The problem arises when ib_core object is created with ibv_create_XXX() and forwarded later to DevX context.
FW counts DevX accesses and elevates internal reference counters to ensure that user will get proper error if he tries to destroy in-use resources.
This error is returned to mlx5_ib_dealloc_pd() too if DevX is not cleaned. This call can be executed by user anytime, for example if he decided to skip DevX cleanup and the ib_core/mlx5_ib can't prevent call to mlx5_ib_dealloc_pd() at this stage.
The difference between mlx5 device from other providers that HW/FW guarantees full cleanup during file close.
Quoted 5 lines
But lets go down the path that we really need a return code in the destroy APIs to solve this problem. For one I don’t see how we can say its meant exclusively for devx drivers to use for a fail. Also can we really claim the API contract is that driver can fail a destroy given a future destroy will succeed? Since the kernel destroy has no retry.
This is why we have special calls for kernel users with WARN_ON() and forced cleanup of ib_core resources.
Quoted 4 lines
Which then boils down do we just keep a simpler definition of the API contract -- driver can just return whatever the true error code is? i.e. if it wants a retry, use EAGAIN. If it has a non recoverable device error, then reset the device, clean up the resources but return ENOTRECOVERABLE.
ib_corecan enable the retry logic for EAGAIN _only_. For other error codes,Ib_corecan trigger awarn_onor something to indicate permanent failure.
We can, but drivers should implement this EAGAIN/ENOTRECOVERABLE logic, this is why in initial phase we are returning always success.
Quoted 7 lines
It can also pass on
ret_codeto user-space as its doing today?Shiraz
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Saleem, ShirazQuoted 3 lines · Shiraz
Which then boils down do we just keep a simpler definition of the API contract -- driver can just return whatever the true error code is?
No, that was always wrong. In almost every case returning codes from destroy is a driver bug, flat out. It causes kernel leaking memory/worse and unrecoverable userspace failures.
Jason
RE: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Leon RomanovskyQuoted 61 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 12 lines · Shiraz
Quoted 12 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 3 lines · Shiraz
The API is quite confusing now. If drivers are not expected to fail the destroy and there is no way to propagate the device failures, then the return type should be a void.
More or less, drivers can only return
-EAGAINwith the idea that a future call during the close process will eventually succeed.Quoted 6 lines
Quoted 5 lines
Any permanent failure will trigger
WARN_ONand a memory leakMaybe we should switch the return code to bool or something to be a little clearer that it is request to retry, not a failure?
There is no retry for kernel object destroy right? So not sure bool is making It
clearer.
Right, kernel verbs users don't know how to deal with destroy failure and designed to ensure that destroy always success.
Quoted 2 lines
I am not very familiar with devx flows but doesn’t it bypass the ib verbs layer
altogether?
Quoted 2 lines
i.e.
mlx5_ib_dealloc_pdisn’t directly called in the devx flows no? So why changes its return type and other provider destroy callbacks?DevX itself indeed bypasses
ib_coreand as a standalone feature doesn't need any changes in destroys. The problem arises whenib_coreobject is created withibv_create_XXX()and forwarded later to DevX context.FW counts DevX accesses and elevates internal reference counters to ensure that user will get proper error if he tries to destroy in-use resources.
This error is returned to
mlx5_ib_dealloc_pd()too if DevX is not cleaned. This call can be executed by user anytime, for example if he decided to skip DevX cleanup and the ib_core/mlx5_ib can't prevent call tomlx5_ib_dealloc_pd()at this stage.The difference between mlx5 device from other providers that HW/FW guarantees full cleanup during file close.
Quoted 2 lines
But lets go down the path that we really need a return code in the destroy APIs
to solve this problem.
Quoted 1 line
For one I don’t see how we can say its meant exclusively for devx drivers to use
for a fail.
Quoted 1 line
Also can we really claim the API contract is that driver can fail a destroy given a
future destroy will succeed?
Quoted 1 line
Since the kernel destroy has no retry.
This is why we have special calls for kernel users with
WARN_ON()and forced cleanup ofib_coreresources.Quoted 2 lines
Which then boils down do we just keep a simpler definition of the API contract --
Quoted 9 lines
driver can just return whatever the true error code is?
Quoted 1 line
i.e. if it wants a retry, use EAGAIN. If it has a non recoverable device error, then
reset the device, clean up the resources but return ENOTRECOVERABLE.
Quoted 1 line
ib_corecan enable the retry logic for EAGAIN _only_. For other error codes,
Ib_corecan trigger awarn_onor something to indicate permanent failure.We can, but drivers should implement this EAGAIN/ENOTRECOVERABLE logic, this is why in initial phase we are returning always success.
I don’t think Jason is entertaining this. So...
RE: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
In reply to Jason GunthorpeQuoted 12 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 2 lines · Shiraz
Which then boils down do we just keep a simpler definition of the API contract -- driver can just return whatever the true error code is?
No, that was always wrong. In almost every case returning codes from destroy is a driver bug, flat out. It causes kernel leaking memory/worse and unrecoverable userspace failures.
seems like we are opening a can then.
I can see a new provider seeing the int return type and returning error codes. And maybe being stumped by seeing some providers ignoring device errors and faking a success. And one provider returning error codes.
More so, how are we going to document the ambiguous definition of this API, and who can and cannot report error codes?
Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
Quoted 5 lines · Shiraz
Quoted 5 lines
Subject: Re: [PATCH rdma-next 01/10] RDMA: Restore ability to fail on PD deallocate
On Thu, Aug 27, 2020 at 02:06:03AM +0000, Saleem, Shiraz wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Which then boils down do we just keep a simpler definition of the API contract -- driver can just return whatever the true error code is?
No, that was always wrong. In almost every case returning codes from destroy is a driver bug, flat out. It causes kernel leaking memory/worse and unrecoverable userspace failures.
seems like we are opening a can then.
It is not something new, it has always been like this, with these rules. The effort to remove the return codes simply failed :(
Quoted 3 lines
I can see a new provider seeing the int return type and returning error codes. And maybe being stumped by seeing some providers ignoring device errors and faking a success. And one provider returning error codes.
No, things can't ignore device failures. If the provider can't shutdown a rouge device then it must return error, leak memory and accept the WARN_ON. Otherwise the device will cause memory corruption by DMA'ing to memory that has been freed.
Having a RDMA driver that can do recovery from HW errors via device reset is really required to close these edge cases.
I suspect no RDMA driver gets this all right today.
Jason