Add SRQ and XRC support for ODP MRs
Subsystems (5)
cover Add SRQ and XRC support for ODP MRs
From: Leon Romanovsky <[email protected]>
Hi,
This series extend ODP to work with SRQ and XRC. Being both per-operation (e.g. RDMA write, RDMA read and atomic) and per-transport (e.g. RC, UD and XRC), we extend IB/core and mlx5 driver to provide needed information to user space.
Thanks
Moni Shoua (12):
IB/mlx5: Fix locking SRQ object in ODP event IB/core: Allocate bit for SRQ ODP support
IB/uverbs: Expose XRC ODP device capabilities IB/mlx5: Remove useless check in ODP handler IB/mlx5: Clean mlx5_ib_mr_responder_pfault_handler() signature IB/mlx5: Add XRC initiator ODP support
IB/mlx5: Let read user wqe also from SRQ buffer IB/mlx5: Add ODP SRQ support
IB/mlx5: Advertise SRQ ODP support for supported transports net/mlx5: Add XRC transport to ODP device capabilities layout IB/mlx5: Advertise XRC ODP support
net/mlx5: Set ODP SRQ support in firmware
drivers/infiniband/core/uverbs_cmd.c | 1 + drivers/infiniband/hw/mlx5/cq.c | 4 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +- drivers/infiniband/hw/mlx5/odp.c | 138 ++++++++++--- drivers/infiniband/hw/mlx5/qp.c | 192 +++++++++++++----- drivers/infiniband/hw/mlx5/srq.h | 2 - drivers/infiniband/hw/mlx5/srq_cmd.c | 16 +- .../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++ include/linux/mlx5/device.h | 3 + include/linux/mlx5/mlx5_ifc.h | 5 +- include/rdma/ib_verbs.h | 2 + include/uapi/rdma/ib_user_verbs.h | 2 + 12 files changed, 331 insertions(+), 96 deletions(-)
PATCH 1/12 IB/mlx5: Fix locking SRQ object in ODP event
From: Moni Shoua <[email protected]>
QP and SRQ objects are stored in different containers so the action to get and lock a common resource during ODP event needs to address that.
While that get rid of 'refcount' and 'free' fields in mlx5_core_srq struct and use the fields with same semantics in common structure.
--- drivers/infiniband/hw/mlx5/cq.c | 4 ++-- drivers/infiniband/hw/mlx5/odp.c | 13 ++++++++----- drivers/infiniband/hw/mlx5/srq.h | 2 -- drivers/infiniband/hw/mlx5/srq_cmd.c | 16 +++++++--------- 4 files changed, 17 insertions(+), 18 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +2 -2
@@ -187,8 +187,8 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe, wqe_ctr =be16_to_cpu (cqe->wqe_counter); wc->wr_id = srq->wrid[wqe_ctr];mlx5_ib_free_srq_wqe (srq, wqe_ctr); -if (msrq &&atomic_dec_and_test (&msrq->refcount)) -complete (&msrq->free); +if (msrq) +mlx5_core_res_put (&msrq->common); } }else { wq = &qp->rq;
drivers/infiniband/hw/mlx5/odp.c +8 -5
@@ -1115,22 +1115,25 @@ static int mlx5_ib_mr_responder_pfault_handler(static inline struct mlx5_core_rsc_common *odp_get_rsc (struct mlx5_ib_dev *dev,u32 wq_num,int pf_type) { -enum mlx5_res_type res_type; +struct mlx5_core_rsc_common *common =NULL ; +struct mlx5_core_srq *srq;switch (pf_type) {case MLX5_WQE_PF_TYPE_RMP : - res_type =MLX5_RES_SRQ ; + srq =mlx5_cmd_get_srq (dev, wq_num); +if (srq) + common = &srq->common;break ;case MLX5_WQE_PF_TYPE_REQ_SEND_OR_WRITE :case MLX5_WQE_PF_TYPE_RESP :case MLX5_WQE_PF_TYPE_REQ_READ_OR_ATOMIC : - res_type =MLX5_RES_QP ; + common =mlx5_core_res_hold (dev->mdev, wq_num,MLX5_RES_QP );break ;default : -return NULL ; +break ; } -return mlx5_core_res_hold (dev->mdev, wq_num, res_type); +return common; }static inline struct mlx5_ib_qp *res_to_qp (struct mlx5_core_rsc_common *res)
drivers/infiniband/hw/mlx5/srq.h +0 -2
@@ -46,8 +46,6 @@ struct mlx5_core_srq {int wqe_shift;void (*event)(struct mlx5_core_srq *srq,enum mlx5_event e); -atomic_t refcount; -struct completion free;u16 uid; };
drivers/infiniband/hw/mlx5/srq_cmd.c +7 -9
@@ -87,7 +87,7 @@ struct mlx5_core_srq *mlx5_cmd_get_srq(struct mlx5_ib_dev *dev, u32 srqn) srq =radix_tree_lookup (&table->tree, srqn);if (srq) -atomic_inc (&srq->refcount); +atomic_inc (&srq->common.refcount);spin_unlock (&table->lock);
@@ -594,8 +594,8 @@ int mlx5_cmd_create_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,if (err)return err; -atomic_set (&srq->refcount,1 ); -init_completion (&srq->free); +atomic_set (&srq->common.refcount,1 ); +init_completion (&srq->common.free);spin_lock_irq (&table->lock); err =radix_tree_insert (&table->tree, srq->srqn, srq);
@@ -627,9 +627,8 @@ int mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)if (err)return err; -if (atomic_dec_and_test (&srq->refcount)) -complete (&srq->free); -wait_for_completion (&srq->free); +mlx5_core_res_put (&srq->common); +wait_for_completion (&srq->common.free);return 0 ; }
@@ -685,7 +684,7 @@ static int srq_event_notifier(struct notifier_block *nb, srq =radix_tree_lookup (&table->tree, srqn);if (srq) -atomic_inc (&srq->refcount); +atomic_inc (&srq->common.refcount);spin_unlock (&table->lock);
@@ -694,8 +693,7 @@ static int srq_event_notifier(struct notifier_block *nb, srq->event (srq, eqe->type); -if (atomic_dec_and_test (&srq->refcount)) -complete (&srq->free); +mlx5_core_res_put (&srq->common);return NOTIFY_OK ; }
Signature · 1 line
2.19.1
PATCH 2/12 IB/core: Allocate bit for SRQ ODP support
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
ODP support matrix is per operation and per transport. The support for each transport (RC, UD, etc.) is described with a bit field.
ODP for SRQ WQEs is considered a different kind of support from ODP for RQ WQs and therefore need a different capability bit.
--- include/rdma/ib_verbs.h | 1 + 1 file changed, 1 insertion(+)
include/rdma/ib_verbs.h +1 -0
@@ -268,6 +268,7 @@ enum ib_odp_transport_cap_bits {IB_ODP_SUPPORT_WRITE =1 <<2 ,IB_ODP_SUPPORT_READ =1 <<3 ,IB_ODP_SUPPORT_ATOMIC =1 <<4 , + IB_ODP_SUPPORT_SRQ_RECV = 1 << 5, }; struct ib_odp_caps {
Signature · 1 line
2.19.1
PATCH 3/12 IB/uverbs: Expose XRC ODP device capabilities
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
Expose XRC ODP capabilities as part of the extended device capabilities.
--- drivers/infiniband/core/uverbs_cmd.c | 1 + include/rdma/ib_verbs.h | 1 + include/uapi/rdma/ib_user_verbs.h | 2 ++ 3 files changed, 4 insertions(+)
drivers/infiniband/core/uverbs_cmd.c +1 -0
@@ -3613,6 +3613,7 @@ static int ib_uverbs_ex_query_device(struct uverbs_attr_bundle *attrs) attr.odp_caps.per_transport_caps.uc_odp_caps; resp.odp_caps.per_transport_caps.ud_odp_caps = attr.odp_caps.per_transport_caps.ud_odp_caps; + resp.xrc_odp_caps = attr.odp_caps.per_transport_caps.xrc_odp_caps; resp.timestamp_mask = attr.timestamp_mask; resp.hca_core_clock = attr.hca_core_clock;
include/rdma/ib_verbs.h +1 -0
@@ -277,6 +277,7 @@ struct ib_odp_caps {uint32_t rc_odp_caps;uint32_t uc_odp_caps;uint32_t ud_odp_caps; +uint32_t xrc_odp_caps; } per_transport_caps; };
include/uapi/rdma/ib_user_verbs.h +2 -0
@@ -270,6 +270,8 @@ struct ib_uverbs_ex_query_device_resp {struct ib_uverbs_tm_caps tm_caps;struct ib_uverbs_cq_moderation_caps cq_moderation_caps;__aligned_u64 max_dm_size; +__u32 xrc_odp_caps; +__u32 reserved; };struct ib_uverbs_query_port {
Signature · 1 line
2.19.1
PATCH 4/12 IB/mlx5: Remove useless check in ODP handler
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
When handling an ODP event for a revive WQE in SRQ the target QP is unknown. Therefore, it is wrong to ask if QP has a SRQ in the page-fault handler.
--- drivers/infiniband/hw/mlx5/odp.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +2 -7
@@ -869,7 +869,6 @@ static int pagefault_single_data_segment(struct mlx5_ib_dev *dev, u32 key,/** * Parse a series of data segments for page fault handling. * -* @qp the QP on which the fault occurred. * @pfault contains page fault information. * @wqe points at the first data segment in the WQE. * @wqe_end points after the end of the WQE.
@@ -886,7 +885,7 @@ static int pagefault_single_data_segment(struct mlx5_ib_dev *dev, u32 key,*/ static int pagefault_data_segments (struct mlx5_ib_dev *dev,struct mlx5_pagefault *pfault, -struct mlx5_ib_qp *qp,void *wqe, +void *wqe,void *wqe_end,u32 *bytes_mapped,u32 *total_wqe_bytes,int receive_queue) {
@@ -897,10 +896,6 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,size_t bcnt;int inline_segment; -/* Skip SRQ next-WQE segment. */ -if (receive_queue && qp->ibqp.srq) - wqe +=sizeof (struct mlx5_wqe_srq_next_seg ); -if (bytes_mapped) *bytes_mapped =0 ;if (total_wqe_bytes)
@@ -1200,7 +1195,7 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, goto resolve_page_fault; } - ret =pagefault_data_segments (dev, pfault, qp, wqe, wqe_end, + ret =pagefault_data_segments (dev, pfault, wqe, wqe_end, &bytes_mapped, &total_wqe_bytes, !requestor);if (ret == -EAGAIN ) {
Signature · 1 line
2.19.1
PATCH 5/12 IB/mlx5: Clean mlx5_ib_mr_responder_pfault_handler() signature
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
In the function mlx5_ib_mr_responder_pfault_handler()
1. The parameter wqe is used as read-only so there is no need to pass it by reference.
2. Remove the unused argument pfault from list of arguments.
--- drivers/infiniband/hw/mlx5/odp.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +7 -7
@@ -1064,12 +1064,12 @@ static int mlx5_ib_mr_initiator_pfault_handler( }/* -* Parse responder WQE. Advances the wqe pointer to point at the -* scatter-gather list, and set wqe_end to the end of the WQE. +* Parse responder WQE and set wqe_end to the end of the WQE. */ -static int mlx5_ib_mr_responder_pfault_handler ( -struct mlx5_ib_dev *dev,struct mlx5_pagefault *pfault, -struct mlx5_ib_qp *qp,void **wqe,void **wqe_end,int wqe_length) +static int mlx5_ib_mr_responder_pfault_handler (struct mlx5_ib_dev *dev, +struct mlx5_ib_qp *qp,void *wqe, +void **wqe_end, +int wqe_length) {struct mlx5_ib_wq *wq = &qp->rq;int wqe_size =1 << wq->wqe_shift;
@@ -1102,7 +1102,7 @@ static int mlx5_ib_mr_responder_pfault_handler(return -EFAULT ; } - *wqe_end = *wqe + wqe_size; + *wqe_end = wqe + wqe_size;return 0 ; }
@@ -1185,7 +1185,7 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, ret =mlx5_ib_mr_initiator_pfault_handler (dev, pfault, qp, &wqe, &wqe_end, ret); else - ret =mlx5_ib_mr_responder_pfault_handler (dev, pfault, qp, &wqe, + ret =mlx5_ib_mr_responder_pfault_handler (dev, qp, wqe, &wqe_end, ret);if (ret <0 ) goto resolve_page_fault;
Signature · 1 line
2.19.1
PATCH 6/12 IB/mlx5: Add XRC initiator ODP support
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
Skip XRC segment in the beginning of a send WQE and fetch ODP XRC capabilities when QP type is IB_QPT_XRC_INI. The rest of the handling is the same as in RC QP.
--- drivers/infiniband/hw/mlx5/odp.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/odp.c +5 -1
@@ -1020,6 +1020,10 @@ static int mlx5_ib_mr_initiator_pfault_handler(MLX5_WQE_CTRL_OPCODE_MASK ;switch (qp->ibqp.qp_type) { +case IB_QPT_XRC_INI : + *wqe +=sizeof (struct mlx5_wqe_xrc_seg ); + transport_caps = dev->odp_caps.per_transport_caps.xrc_odp_caps; +break ;case IB_QPT_RC : transport_caps = dev->odp_caps.per_transport_caps.rc_odp_caps;break ;
@@ -1039,7 +1043,7 @@ static int mlx5_ib_mr_initiator_pfault_handler(return -EFAULT ; } -if (qp->ibqp.qp_type !=IB_QPT_RC ) { +if (qp->ibqp.qp_type ==IB_QPT_UD ) { av = *wqe;if (av->dqp_dct &cpu_to_be32 (MLX5_EXTENDED_UD_AV )) *wqe +=sizeof (struct mlx5_av );
Signature · 1 line
2.19.1
PATCH 7/12 IB/mlx5: Let read user wqe also from SRQ buffer
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
Reading a WQE from SRQ is almost identical to reading from regular RQ. The differences are the size of the queue, the size of a WQE and buffer location.
Make necessary changes to mlx5_ib_read_user_wqe() to let it read a WQE from a SRQ or RQ by caller choice.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +- drivers/infiniband/hw/mlx5/odp.c | 20 ++- drivers/infiniband/hw/mlx5/qp.c | 192 ++++++++++++++++++++------- 3 files changed, 166 insertions(+), 55 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +6 -3
@@ -1076,9 +1076,12 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, const struct ib_send_wr **bad_wr);int mlx5_ib_post_recv (struct ib_qp *ibqp,const struct ib_recv_wr *wr,const struct ib_recv_wr **bad_wr); -int mlx5_ib_read_user_wqe (struct mlx5_ib_qp *qp,int send,int wqe_index, -void *buffer,u32 length, -struct mlx5_ib_qp_base *base); +int mlx5_ib_read_user_wqe_sq (struct mlx5_ib_qp *qp,int wqe_index,void *buffer, +int buflen,size_t *bc); +int mlx5_ib_read_user_wqe_rq (struct mlx5_ib_qp *qp,int wqe_index,void *buffer, +int buflen,size_t *bc); +int mlx5_ib_read_user_wqe_srq (struct mlx5_ib_srq *srq,int wqe_index, +void *buffer,int buflen,size_t *bc); struct ib_cq *mlx5_ib_create_cq(struct ib_device *ibdev, const struct ib_cq_init_attr *attr, struct ib_ucontext *context,
drivers/infiniband/hw/mlx5/odp.c +15 -5
@@ -1154,6 +1154,7 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev,int requestor = pfault->type &MLX5_PFAULT_REQUESTOR ;struct mlx5_core_rsc_common *res;struct mlx5_ib_qp *qp; +size_t bytes_copied; res =odp_get_rsc (dev, pfault->wqe.wq_num, pfault->type);if (!res) {
@@ -1176,9 +1177,16 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, goto resolve_page_fault; } - ret =mlx5_ib_read_user_wqe (qp, requestor, wqe_index, buffer, -PAGE_SIZE , &qp->trans_qp.base); -if (ret <0 ) { +if (requestor) + ret =mlx5_ib_read_user_wqe_sq (qp, wqe_index, + buffer,PAGE_SIZE , + &bytes_copied); +else + ret =mlx5_ib_read_user_wqe_rq (qp, wqe_index, + buffer,PAGE_SIZE , + &bytes_copied); + +if (ret) {mlx5_ib_err (dev,"Failed reading a WQE following page fault, error=%d, wqe_index=%x, qpn=%x\n" , ret, wqe_index, pfault->token); goto resolve_page_fault;
@@ -1187,10 +1195,12 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, wqe = buffer;if (requestor) ret =mlx5_ib_mr_initiator_pfault_handler (dev, pfault, qp, &wqe, - &wqe_end, ret); + &wqe_end, + bytes_copied);else ret =mlx5_ib_mr_responder_pfault_handler (dev, qp, wqe, - &wqe_end, ret); + &wqe_end, + bytes_copied);if (ret <0 ) goto resolve_page_fault;
drivers/infiniband/hw/mlx5/qp.c +145 -47
220 lines not on this page · raw message
Signature · 1 line
2.19.1
PATCH 8/12 IB/mlx5: Add ODP SRQ support
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
Add changes to the WQE page-fault handler to
1. Identify that the event is for a SRQ WQE
2. Pass SRQ object instead of a QP to the function that reads the WQE
3. Parse the SRQ WQE with respect to its structure
The rest is handled as for regular RQ WQE.
--- drivers/infiniband/hw/mlx5/odp.c | 84 +++++++++++++++++++++++--------- 1 file changed, 61 insertions(+), 23 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +61 -23
@@ -1070,19 +1070,32 @@ static int mlx5_ib_mr_initiator_pfault_handler(/* * Parse responder WQE and set wqe_end to the end of the WQE. */ -static int mlx5_ib_mr_responder_pfault_handler (struct mlx5_ib_dev *dev, -struct mlx5_ib_qp *qp,void *wqe, -void **wqe_end, -int wqe_length) +static int mlx5_ib_mr_responder_pfault_handler_srq (struct mlx5_ib_dev *dev, +struct mlx5_ib_srq *srq, +void **wqe,void **wqe_end, +int wqe_length) { -struct mlx5_ib_wq *wq = &qp->rq; -int wqe_size =1 << wq->wqe_shift; +int wqe_size =1 << srq->msrq.wqe_shift; -if (qp->ibqp.srq) { -mlx5_ib_err (dev,"ODP fault on SRQ is not supported\n" ); +if (wqe_size > wqe_length) { +mlx5_ib_err (dev,"Couldn't read all of the receive WQE's content\n" );return -EFAULT ; } + *wqe_end = *wqe + wqe_size; + *wqe +=sizeof (struct mlx5_wqe_srq_next_seg ); + +return 0 ; +} + +static int mlx5_ib_mr_responder_pfault_handler_rq (struct mlx5_ib_dev *dev, +struct mlx5_ib_qp *qp, +void *wqe,void **wqe_end, +int wqe_length) +{ +struct mlx5_ib_wq *wq = &qp->rq; +int wqe_size =1 << wq->wqe_shift; +if (qp->wq_sig) {mlx5_ib_err (dev,"ODP fault with WQE signatures is not supported\n" );return -EFAULT ;
@@ -1142,6 +1155,14 @@ static inline struct mlx5_ib_qp *res_to_qp(struct mlx5_core_rsc_common *res)return to_mibqp (mqp); } +static inline struct mlx5_ib_srq *res_to_srq (struct mlx5_core_rsc_common *res) +{ +struct mlx5_core_srq *msrq = + container_of(res, struct mlx5_core_srq, common); + +return to_mibsrq (msrq); +} +static void mlx5_ib_mr_wqe_pfault_handler (struct mlx5_ib_dev *dev,struct mlx5_pagefault *pfault) {
@@ -1152,8 +1173,9 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev,int resume_with_error =1 ;u16 wqe_index = pfault->wqe.wqe_index;int requestor = pfault->type &MLX5_PFAULT_REQUESTOR ; -struct mlx5_core_rsc_common *res; -struct mlx5_ib_qp *qp; +struct mlx5_core_rsc_common *res =NULL ; +struct mlx5_ib_qp *qp =NULL ; +struct mlx5_ib_srq *srq =NULL ;size_t bytes_copied; res =odp_get_rsc (dev, pfault->wqe.wq_num, pfault->type);
@@ -1166,6 +1188,10 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev,case MLX5_RES_QP : qp =res_to_qp (res);break ; +case MLX5_RES_SRQ : +case MLX5_RES_XSRQ : + srq =res_to_srq (res); +break ;default :mlx5_ib_err (dev,"wqe page fault for unsupported type %d\n" , pfault->type); goto resolve_page_fault;
@@ -1177,14 +1203,21 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, goto resolve_page_fault; } -if (requestor) - ret =mlx5_ib_read_user_wqe_sq (qp, wqe_index, - buffer,PAGE_SIZE , - &bytes_copied); -else - ret =mlx5_ib_read_user_wqe_rq (qp, wqe_index, - buffer,PAGE_SIZE , - &bytes_copied); +if (qp) { +if (requestor) { + ret =mlx5_ib_read_user_wqe_sq (qp, wqe_index, + buffer,PAGE_SIZE , + &bytes_copied); + }else { + ret =mlx5_ib_read_user_wqe_rq (qp, wqe_index, + buffer,PAGE_SIZE , + &bytes_copied); + } + }else { + ret =mlx5_ib_read_user_wqe_srq (srq, wqe_index, + buffer,PAGE_SIZE , + &bytes_copied); + }if (ret) { mlx5_ib_err(dev, "Failed reading a WQE following page fault, error=%d, wqe_index=%x, qpn=%x\n",
@@ -1194,13 +1227,18 @@ static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev, wqe = buffer;if (requestor) - ret =mlx5_ib_mr_initiator_pfault_handler (dev, pfault, qp, &wqe, - &wqe_end, + ret =mlx5_ib_mr_initiator_pfault_handler (dev, pfault, qp, + &wqe, &wqe_end, bytes_copied); +else if (qp) + ret =mlx5_ib_mr_responder_pfault_handler_rq (dev, qp, + wqe, &wqe_end, + bytes_copied);else - ret =mlx5_ib_mr_responder_pfault_handler (dev, qp, wqe, - &wqe_end, - bytes_copied); + ret =mlx5_ib_mr_responder_pfault_handler_srq (dev, srq, + &wqe, &wqe_end, + bytes_copied); +if (ret <0 ) goto resolve_page_fault;
Signature · 1 line
2.19.1
PATCH 9/12 IB/mlx5: Advertise SRQ ODP support for supported transports
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
ODP support in SRQ is per transport capability. Based on device capabilities set this flag in device structure for future queries.
--- drivers/infiniband/hw/mlx5/odp.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/infiniband/hw/mlx5/odp.c +6 -0
@@ -315,6 +315,9 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)if (MLX5_CAP_ODP (dev->mdev, ud_odp_caps.send)) caps->per_transport_caps.ud_odp_caps |=IB_ODP_SUPPORT_SEND ; +if (MLX5_CAP_ODP (dev->mdev, ud_odp_caps.srq_receive)) + caps->per_transport_caps.ud_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; +if (MLX5_CAP_ODP (dev->mdev, rc_odp_caps.send)) caps->per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SEND ;
@@ -330,6 +333,9 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)if (MLX5_CAP_ODP (dev->mdev, rc_odp_caps.atomic)) caps->per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_ATOMIC ; +if (MLX5_CAP_ODP (dev->mdev, rc_odp_caps.srq_receive)) + caps->per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; +if (MLX5_CAP_GEN (dev->mdev, fixed_buffer_size) &&MLX5_CAP_GEN (dev->mdev, null_mkey) &&MLX5_CAP_GEN (dev->mdev, umr_extended_translation_offset))
Signature · 1 line
2.19.1
PATCH 10/12 net/mlx5: Add XRC transport to ODP device capabilities layout
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
The device capabilities for ODP structure was missing the field for XRC transport so add it here.
--- include/linux/mlx5/mlx5_ifc.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/mlx5/mlx5_ifc.h +3 -1
@@ -831,7 +831,9 @@ struct mlx5_ifc_odp_cap_bits {struct mlx5_ifc_odp_per_transport_service_cap_bits ud_odp_caps; -u8 reserved_at_e0[0x720 ]; +struct mlx5_ifc_odp_per_transport_service_cap_bits xrc_odp_caps; + +u8 reserved_at_100[0x700 ]; };struct mlx5_ifc_calc_op {
Signature · 1 line
2.19.1
PATCH 11/12 IB/mlx5: Advertise XRC ODP support
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
Query all per transport caps for XRC and set the appropriate bits in the per transport field of the advertised struct.
--- drivers/infiniband/hw/mlx5/odp.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
drivers/infiniband/hw/mlx5/odp.c +18 -0
@@ -336,6 +336,24 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)if (MLX5_CAP_ODP (dev->mdev, rc_odp_caps.srq_receive)) caps->per_transport_caps.rc_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.send)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_SEND ; + +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.receive)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_RECV ; + +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.write)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_WRITE ; + +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.read)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_READ ; + +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.atomic)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_ATOMIC ; + +if (MLX5_CAP_ODP (dev->mdev, xrc_odp_caps.srq_receive)) + caps->per_transport_caps.xrc_odp_caps |=IB_ODP_SUPPORT_SRQ_RECV ; +if (MLX5_CAP_GEN (dev->mdev, fixed_buffer_size) &&MLX5_CAP_GEN (dev->mdev, null_mkey) &&MLX5_CAP_GEN (dev->mdev, umr_extended_translation_offset))
Signature · 1 line
2.19.1
PATCH 12/12 net/mlx5: Set ODP SRQ support in firmware
In reply to Leon RomanovskyFrom: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +53 -0
@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; } +static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; + +if (!MLX5_CAP_GEN (dev, pg)) +return 0 ; + + err =mlx5_core_get_caps (dev,MLX5_CAP_ODP ); +if (err) +return err; + +/** +* If all bits are cleared we shouldn't try to set it +* or we might fail while trying to access a reserved bit. +*/ +if (!(MLX5_CAP_ODP_MAX (dev, ud_odp_caps.srq_receive) || +MLX5_CAP_ODP_MAX (dev, rc_odp_caps.srq_receive) || +MLX5_CAP_ODP_MAX (dev, xrc_odp_caps.srq_receive))) +return 0 ; + + set_ctx =kzalloc (set_sz,GFP_KERNEL ); +if (!set_ctx) +return -ENOMEM ; + + set_hca_cap =MLX5_ADDR_OF (set_hca_cap_in, set_ctx, capability); +memcpy (set_hca_cap, dev->caps.hca_cur[MLX5_CAP_ODP ], +MLX5_ST_SZ_BYTES (odp_cap)); + +/* set ODP SRQ support for RC/UD and XRC transports */ +MLX5_SET (odp_cap, set_hca_cap, ud_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, ud_odp_caps.srq_receive))); + +MLX5_SET (odp_cap, set_hca_cap, rc_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, rc_odp_caps.srq_receive))); + +MLX5_SET (odp_cap, set_hca_cap, xrc_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, xrc_odp_caps.srq_receive))); + + err =set_caps (dev, set_ctx, set_sz,MLX5_SET_HCA_CAP_OP_MOD_ODP ); + +kfree (set_ctx); +return err; +} +static int handle_hca_cap (struct mlx5_core_dev *dev) {void *set_ctx =NULL ;
@@ -926,6 +973,12 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, goto reclaim_boot_pages; } + err =handle_hca_cap_odp (dev); +if (err) { +dev_err (&pdev->dev,"handle_hca_cap_odp failed\n" ); + goto reclaim_boot_pages; + } + err =mlx5_satisfy_startup_pages (dev,0 );if (err) {dev_err (&pdev->dev,"failed to allocate init pages\n" );
include/linux/mlx5/device.h +3 -0
@@ -1201,6 +1201,9 @@ enum mlx5_qcam_feature_groups {#define MLX5_CAP_ODP (mdev, cap)\ MLX5_GET(odp_cap, mdev->caps.hca_cur[MLX5_CAP_ODP], cap) +#define MLX5_CAP_ODP_MAX (mdev, cap)\ + MLX5_GET(odp_cap, mdev->caps.hca_max[MLX5_CAP_ODP], cap) +#define MLX5_CAP_VECTOR_CALC (mdev, cap) \ MLX5_GET(vector_calc_cap, \ mdev->caps.hca_cur[MLX5_CAP_VECTOR_CALC], cap)
include/linux/mlx5/mlx5_ifc.h +1 -0
@@ -72,6 +72,7 @@ enum {enum {MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE =0x0 , +MLX5_SET_HCA_CAP_OP_MOD_ODP =0x2 ,MLX5_SET_HCA_CAP_OP_MOD_ATOMIC =0x3 , };
Signature · 1 line
2.19.1
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
Quoted 19 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +1
-0+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; }
drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 470
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +9
-0+static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; + +if (!MLX5_CAP_GEN (dev, pg)) +return 0 ;
Should a
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING))
return 0;Be here?
Jason
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
Quoted 33 lines · Jason Gunthorpe
Quoted 19 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +1
-0+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; }drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 470
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +9
-0+static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; + +if (!MLX5_CAP_GEN (dev, pg)) +return 0 ;Should a
Quoted 4 lines
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) return 0;Be here?
We had similar discussion in mlx5_ib main.c, but here we are talking about mlx5_core code, which from my point of view should represent the real HW capabilities without relation to kernel compilation mode.
Thanks
Quoted 2 lines
Jason
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
Quoted 34 lines · Leon Romanovsky
Quoted 33 lines · Jason Gunthorpe
Quoted 19 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +1
-0+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; }drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 470
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +9
-0+static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; + +if (!MLX5_CAP_GEN (dev, pg)) +return 0 ;Should a
Quoted 8 lines
Quoted 4 lines
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) return 0;Be here?
We had similar discussion in
mlx5_ibmain.c, but here we are talking aboutmlx5_corecode, which from my point of view should represent the real HW capabilities without relation to kernel compilation mode.
This switch is to tell the FW that the mlx5_ib module supports the new protocol - so having it in core code at all is really weird. I assume there is some startup sequence reason?
Since the modularity is already wrecked it seems like an odd reason not to add the if..
Jason
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
Quoted 50 lines · Jason Gunthorpe
Quoted 41 lines · Leon Romanovsky
Quoted 40 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't
allow SRQ to work with ODP unless kernel asks for it.Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>.../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +1
-0+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; }drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct · line 471
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +9
-0+static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; + +if (!MLX5_CAP_GEN (dev, pg)) +return 0 ;Should a
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) return 0;Quoted 7 lines
Quoted 1 line
Be here?
We had similar discussion in
mlx5_ibmain.c, but here we are talking
aboutmlx5_corecode, which from my point of view should represent the
real HW capabilities without relation to kernel compilation mode.
Quoted 5 lines
This switch is to tell the FW that the
mlx5_ibmodule supports the new
protocol - so having it in core code at all is really weird. I assume there is some startup sequence reason?
Yes, sadly this must be in startup, set_hca_cap requests must come prior to init_hca command.
Quoted 3 lines
Since the modularity is already wrecked it seems like an odd reason not to add the if..
Agree, even better, let's compile out the whole function. I would even consider having a separate file in mlx5/core for IB related start-up procedures :).
Quoted 1 line
Jason
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
In reply to Leon RomanovskyQuoted 22 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +0
-0@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)
drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct · line 469
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +7
-0return err; } +static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; +
reversed xmas tree.
drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 479
Quoted 11 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +11
-0+if (!MLX5_CAP_GEN (dev, pg)) +return 0 ; + + err =mlx5_core_get_caps (dev,MLX5_CAP_ODP ); +if (err) +return err; + +/** +* If all bits are cleared we shouldn't try to set it +* or we might fail while trying to access a reserved bit. +*/
"set them" not "set it" ?
to me this is a redundant comment, the code is self explanatory.
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +13
-0+if (!(MLX5_CAP_ODP_MAX (dev, ud_odp_caps.srq_receive) || +MLX5_CAP_ODP_MAX (dev, rc_odp_caps.srq_receive) || +MLX5_CAP_ODP_MAX (dev, xrc_odp_caps.srq_receive))) +return 0 ; + + set_ctx =kzalloc (set_sz,GFP_KERNEL ); +if (!set_ctx) +return -ENOMEM ; + + set_hca_cap =MLX5_ADDR_OF (set_hca_cap_in, set_ctx, capability); +memcpy (set_hca_cap, dev->caps.hca_cur[MLX5_CAP_ODP ], +MLX5_ST_SZ_BYTES (odp_cap)); +
drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 502
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +10
-0+/* set ODP SRQ support for RC/UD and XRC transports */ +MLX5_SET (odp_cap, set_hca_cap, ud_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, ud_odp_caps.srq_receive))); + +MLX5_SET (odp_cap, set_hca_cap, rc_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, rc_odp_caps.srq_receive))); + +MLX5_SET (odp_cap, set_hca_cap, xrc_odp_caps.srq_receive, + (MLX5_CAP_ODP_MAX (dev, xrc_odp_caps.srq_receive))); +
Redundant parentheses around the 3rd parameter.
Quoted 52 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +6
-0+ err =set_caps (dev, set_ctx, set_sz,MLX5_SET_HCA_CAP_OP_MOD_ODP ); + +kfree (set_ctx); +return err; +} +static int
handle_hca_cap(struct mlx5_core_dev *dev){drivers/net/ethernet/mellanox/mlx5/core/main.c +6
-0void *set_ctx =NULL ;@@ -926,6 +973,12 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, goto reclaim_boot_pages; } + err =handle_hca_cap_odp (dev); +if (err) { +dev_err (&pdev->dev,"handle_hca_cap_odp failed\n" ); + goto reclaim_boot_pages; + } + err =mlx5_satisfy_startup_pages (dev,0 );if (err) {drivers/net/ethernet/mellanox/mlx5/core/main.c +0
-0dev_err (&pdev->dev,"failed to allocate init pages\n" );include/linux/mlx5/device.h +3
-0@@ -1201,6 +1201,9 @@ enum mlx5_qcam_feature_groups {#define MLX5_CAP_ODP (mdev, cap)\ MLX5_GET(odp_cap, mdev->caps.hca_cur[MLX5_CAP_ODP], cap) +#define MLX5_CAP_ODP_MAX (mdev, cap)\ + MLX5_GET(odp_cap, mdev->caps.hca_max[MLX5_CAP_ODP], cap) +#define MLX5_CAP_VECTOR_CALC (mdev, cap) \ MLX5_GET(vector_calc_cap, \ mdev->caps.hca_cur[MLX5_CAP_VECTOR_CALC], cap)include/linux/mlx5/mlx5_ifc.h +1
-0@@ -72,6 +72,7 @@ enum {enum {MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE =0x0 , +MLX5_SET_HCA_CAP_OP_MOD_ODP =0x2 ,MLX5_SET_HCA_CAP_OP_MOD_ATOMIC =0x3 , };
Re: [PATCH mlx5-next 12/12] net/mlx5: Set ODP SRQ support in firmware
Quoted 26 lines · Saeed Mahameed
Quoted 25 lines · Leon Romanovsky
From: Moni Shoua <[email protected]>
To avoid compatibility issue with older kernels the firmware doesn't allow SRQ to work with ODP unless kernel asks for it.
Signed-off-by Moni Shoua <[email protected]>Reviewed-by Majd Dibbiny <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- .../net/ethernet/mellanox/mlx5/core/main.c | 53 +++++++++++++++++++ include/linux/mlx5/device.h | 3 ++ include/linux/mlx5/mlx5_ifc.h | 1 + 3 files changed, 57 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +0
-0@@ -459,6 +459,53 @@ static int handle_hca_cap_atomic(struct mlx5_core_dev *dev)return err; }
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c static int handle_hca_cap_atomic(struct mlx5_core_dev *dev) · line 468
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c +7
-0+static int handle_hca_cap_odp (struct mlx5_core_dev *dev) +{ +void *set_ctx; +void *set_hca_cap; +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +int err; +reversed xmas tree.
I'll send followup to address your comments.
Thanks
Quoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This series extend ODP to work with SRQ and XRC. Being both per-operation (e.g. RDMA write, RDMA read and atomic) and per-transport (e.g. RC, UD and XRC), we extend IB/core and mlx5 driver to provide needed information to user space.
Thanks
Quoted 13 lines
Moni Shoua (12):
IB/mlx5: Fix locking SRQ object in ODP event IB/core: Allocate bit for SRQ ODP support
IB/uverbs: Expose XRC ODP device capabilities IB/mlx5: Remove useless check in ODP handler IB/mlx5: Cleanmlx5_ib_mr_responder_pfault_handler()signature IB/mlx5: Add XRC initiator ODP support
IB/mlx5: Let read user wqe also from SRQ buffer IB/mlx5: Add ODP SRQ support
IB/mlx5: Advertise SRQ ODP support for supported transports net/mlx5: Add XRC transport to ODP device capabilities layout IB/mlx5: Advertise XRC ODP support
net/mlx5: Set ODP SRQ support in firmware
Doug, Jason,
Please pay attention that patches below should go to shared branch:
net/mlx5: Add XRC transport to ODP device capabilities layout net/mlx5: Set ODP SRQ support in firmware
Thanks
Jason Gunthorpe
AppliedQuoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This series extend ODP to work with SRQ and XRC. Being both per-operation (e.g. RDMA write, RDMA read and atomic) and per-transport (e.g. RC, UD and XRC), we extend IB/core and mlx5 driver to provide needed information to user space.
Thanks
Quoted 10 lines
Moni Shoua (12):
IB/mlx5: Fix locking SRQ object in ODP event IB/core: Allocate bit for SRQ ODP support
IB/uverbs: Expose XRC ODP device capabilities IB/mlx5: Remove useless check in ODP handler IB/mlx5: Cleanmlx5_ib_mr_responder_pfault_handler()signature IB/mlx5: Add XRC initiator ODP support
IB/mlx5: Let read user wqe also from SRQ buffer IB/mlx5: Add ODP SRQ support
IB/mlx5: Advertise SRQ ODP support for supported transports
I applied these patches to for-next
Quoted 3 lines
net/mlx5: Add XRC transport to ODP device capabilities layout IB/mlx5: Advertise XRC ODP support
net/mlx5: Set ODP SRQ support in firmware
This might need some re-organizing - the last patch could be split (possibly merge with the first) so the header changes can go to the shared branch, but the handle_hca_cap_odp() stuff must only be applied to the rdma tree.
I'm fine either way, if you don't want to split it send a commit ID for the first patch on mlx5-next.
Thanks,
Jason
Quoted 28 lines · Jason Gunthorpe
Quoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This series extend ODP to work with SRQ and XRC. Being both per-operation (e.g. RDMA write, RDMA read and atomic) and per-transport (e.g. RC, UD and XRC), we extend IB/core and mlx5 driver to provide needed information to user space.
Thanks
Quoted 10 lines
Moni Shoua (12):
IB/mlx5: Fix locking SRQ object in ODP event IB/core: Allocate bit for SRQ ODP support
IB/uverbs: Expose XRC ODP device capabilities IB/mlx5: Remove useless check in ODP handler IB/mlx5: Cleanmlx5_ib_mr_responder_pfault_handler()signature IB/mlx5: Add XRC initiator ODP support
IB/mlx5: Let read user wqe also from SRQ buffer IB/mlx5: Add ODP SRQ support
IB/mlx5: Advertise SRQ ODP support for supported transportsI applied these patches to for-next
Quoted 3 lines
net/mlx5: Add XRC transport to ODP device capabilities layout IB/mlx5: Advertise XRC ODP support
net/mlx5: Set ODP SRQ support in firmware
Quoted 7 lines
This might need some re-organizing - the last patch could be split (possibly merge with the first) so the header changes can go to the shared branch, but the
handle_hca_cap_odp()stuff must only be applied to the rdma tree.I'm fine either way, if you don't want to split it send a commit ID for the first patch on mlx5-next.
I applied two following patches,
46861e3e88be net/mlx5: Set ODP SRQ support in firmware dda7a817f287 net/mlx5: Add XRC transport to ODP device capabilities layout
Thanks
Quoted 3 lines
Thanks,
Jason
Quoted 34 lines · Leon Romanovsky
Quoted 33 lines · Jason Gunthorpe
Quoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This series extend ODP to work with SRQ and XRC. Being both per-operation (e.g. RDMA write, RDMA read and atomic) and per-transport (e.g. RC, UD and XRC), we extend IB/core and mlx5 driver to provide needed information to user space.
Thanks
Quoted 10 lines
Moni Shoua (12):
IB/mlx5: Fix locking SRQ object in ODP event IB/core: Allocate bit for SRQ ODP support
IB/uverbs: Expose XRC ODP device capabilities IB/mlx5: Remove useless check in ODP handler IB/mlx5: Cleanmlx5_ib_mr_responder_pfault_handler()signature IB/mlx5: Add XRC initiator ODP support
IB/mlx5: Let read user wqe also from SRQ buffer IB/mlx5: Add ODP SRQ support
IB/mlx5: Advertise SRQ ODP support for supported transportsI applied these patches to for-next
Quoted 3 lines
net/mlx5: Add XRC transport to ODP device capabilities layout IB/mlx5: Advertise XRC ODP support
net/mlx5: Set ODP SRQ support in firmwareThis might need some re-organizing - the last patch could be split (possibly merge with the first) so the header changes can go to the shared branch, but the
handle_hca_cap_odp()stuff must only be applied to the rdma tree.
Quoted 7 lines
Quoted 2 lines
I'm fine either way, if you don't want to split it send a commit ID for the first patch on mlx5-next.
I applied two following patches,
46861e3e88benet/mlx5: Set ODP SRQ support in firmwaredda7a817f287net/mlx5: Add XRC transport to ODP device capabilities layout
Okay, done..
Thanks,
Jason