Use max_sge_rd device capability
Subsystems (6)
cover Use max_sge_rd device capability
Resending because I forgot to cc linux-rdma :(
Some devices were not setting this capability, so fix those devices, and svcrdma to use max_sge_rd. Also remove rdma_cap_read_multi_sge() since it isn't needed.
These patches were originally part of:
http://www.spinics.net/lists/linux-rdma/msg27436.html
They really aren't part of iSER/iWARP at all, so I've split them out.
Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly
Steve Wise (3):
RDMA/Core: remove rdma_cap_read_multi_sge() helper
svcrdma: Use max_sge_rd for destination read depths ipath,qib: Expose max_sge_rd correctly
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + include/linux/sunrpc/svc_rdma.h | 1 + include/rdma/ib_verbs.h | 28 -------------------------- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +---------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 4 ++++ 9 files changed, 11 insertions(+), 39 deletions(-)
Signature · 5 lines
Steve -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH RESEND 1/4 mlx4, mlx5, mthca: Expose max_sge_rd correctly
From: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/[email protected]>
Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
---
drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + 3 files changed, 3 insertions(+), 0 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -0
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, props->max_qp_wr = dev->dev->caps.max_wqes -MLX4_IB_SQ_MAX_SPARE ; props->max_sge =min (dev->dev->caps.max_sq_sg, dev->dev->caps.max_rq_sg); + props->max_sge_rd = props->max_sge; props->max_cq = dev->dev->quotas.cq; props->max_cqe = dev->dev->caps.max_cqes; props->max_mr = dev->dev->quotas.mpt;
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -273,6 +273,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, sizeof(struct mlx5_wqe_ctrl_seg)) /sizeof (struct mlx5_wqe_data_seg ); props->max_sge =min (max_rq_sg, max_sq_sg); + props->max_sge_rd = props->max_sge; props->max_cq =1 <<MLX5_CAP_GEN (mdev, log_max_cq); props->max_cqe = (1 <<MLX5_CAP_GEN (mdev, log_max_eq_sz)) -1 ; props->max_mr =1 <<MLX5_CAP_GEN (mdev, log_max_mkey);
drivers/infiniband/hw/mthca/mthca_provider.c +1 -0
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = mdev->limits.num_qps - mdev->limits.reserved_qps; props->max_qp_wr = mdev->limits.max_wqes; props->max_sge = mdev->limits.max_sg; + props->max_sge_rd = props->max_sge; props->max_cq = mdev->limits.num_cqs - mdev->limits.reserved_cqs; props->max_cqe = mdev->limits.max_cqes; props->max_mr = mdev->limits.num_mpts - mdev->limits.reserved_mrws;
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH RESEND 2/4 ipath,qib: Expose max_sge_rd correctly
In reply to Steve WiseApplications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
---
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + 2 files changed, 2 insertions(+), 0 deletions(-)
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = ib_ipath_max_qps; props->max_qp_wr = ib_ipath_max_qp_wrs; props->max_sge = ib_ipath_max_sges; + props->max_sge_rd = ib_ipath_max_sges; props->max_cq = ib_ipath_max_cqs; props->max_ah = ib_ipath_max_ahs; props->max_cqe = ib_ipath_max_cqes;
drivers/infiniband/hw/qib/qib_verbs.c +1 -0
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop props->max_qp = ib_qib_max_qps; props->max_qp_wr = ib_qib_max_qp_wrs; props->max_sge = ib_qib_max_sges; + props->max_sge_rd = ib_qib_max_sges; props->max_cq = ib_qib_max_cqs; props->max_ah = ib_qib_max_ahs; props->max_cqe = ib_qib_max_cqes;
PATCH RESEND 3/4 svcrdma: Use max_sge_rd for destination read depths
In reply to Steve Wise---
include/linux/sunrpc/svc_rdma.h | 1 + net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +----------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 4 ++++ 3 files changed, 6 insertions(+), 11 deletions(-)
include/linux/sunrpc/svc_rdma.h +1 -0
@@ -132,6 +132,7 @@ struct svcxprt_rdma {struct list_head sc_accept_q;/* Conn. waiting accept */ int sc_ord;/* RDMA read limit */ int sc_max_sge; +int sc_max_sge_rd;/* max sge for read target */ int sc_sq_depth;/* Depth of SQ */ atomic_t sc_sq_count;/* Number of SQ WR on queue */
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +1 -11
@@ -115,15 +115,6 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp, rqstp->rq_arg.tail[0 ].iov_len =0 ; } -static int rdma_read_max_sge (struct svcxprt_rdma *xprt,int sge_count) -{ -if (!rdma_cap_read_multi_sge (xprt->sc_cm_id->device, - xprt->sc_cm_id->port_num)) -return 1 ; -else -return min_t (int, sge_count, xprt->sc_max_sge); -} -/* Issue an RDMA_READ using the local lkey to map the data sink */ int rdma_read_chunk_lcl(struct svcxprt_rdma *xprt, struct svc_rqst *rqstp,
@@ -144,8 +135,7 @@ int rdma_read_chunk_lcl(struct svcxprt_rdma *xprt, ctxt->direction =DMA_FROM_DEVICE ; ctxt->read_hdr = head; - pages_needed = -min_t (int, pages_needed,rdma_read_max_sge (xprt, pages_needed)); + pages_needed =min_t (int, pages_needed, xprt->sc_max_sge_rd); read =min_t (int, pages_needed <<PAGE_SHIFT , rs_length);for (pno =0 ; pno < pages_needed; pno++) {
net/sunrpc/xprtrdma/svc_rdma_transport.c +4 -0
@@ -872,6 +872,8 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)* capabilities of this particular device */ newxprt->sc_max_sge =min ((size_t )devattr.max_sge, (size_t )RPCSVC_MAXPAGES ); + newxprt->sc_max_sge_rd =min_t (size_t, devattr.max_sge_rd, +RPCSVC_MAXPAGES ); newxprt->sc_max_requests =min ((size_t )devattr.max_qp_wr, (size_t )svcrdma_max_requests); newxprt->sc_sq_depth =RPCRDMA_SQ_DEPTH_MULT * newxprt->sc_max_requests;
@@ -1046,6 +1048,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)" remote_ip : %pI4\n" " remote_port : %d\n" " max_sge : %d\n" +" max_sge_rd : %d\n" " sq_depth : %d\n" " max_requests : %d\n" " ord : %d\n",
@@ -1059,6 +1062,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)ntohs (((struct sockaddr_in *)&newxprt->sc_cm_id-> route.addr.dst_addr)->sin_port), newxprt->sc_max_sge, + newxprt->sc_max_sge_rd, newxprt->sc_sq_depth, newxprt->sc_max_requests, newxprt->sc_ord);
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH RESEND 4/4 RDMA/Core: remove rdma_cap_read_multi_sge() helper
In reply to Steve WiseThis functionality already exists via the max_sge_rd device capability.
---
include/rdma/ib_verbs.h | 28 ---------------------------- 1 files changed, 0 insertions(+), 28 deletions(-)
include/rdma/ib_verbs.h +0 -28
@@ -2071,34 +2071,6 @@ static inline bool rdma_cap_eth_ah(const struct ib_device *device, u8 port_num) }/** -* rdma_cap_read_multi_sge - Check if the port of device has the capability -* RDMA Read Multiple Scatter-Gather Entries. -* @device: Device to check -* @port_num: Port number to check -* -* iWARP has a restriction that RDMA READ requests may only have a single -* Scatter/Gather Entry (SGE) in the work request. -* -* NOTE: although the linux kernel currently assumes all devices are either -* single SGE RDMA READ devices or identical SGE maximums for RDMA READs and -* WRITEs, according to Tom Talpey, this is not accurate. There are some -* devices out there that support more than a single SGE on RDMA READ -* requests, but do not support the same number of SGEs as they do on -* RDMA WRITE requests. The linux kernel would need rearchitecting to -* support these imbalanced READ/WRITE SGEs allowed devices. So, for now, -* suffice with either the device supports the same READ/WRITE SGEs, or -* it only gets one READ sge. -* -* Return: true for any device that allows more than one SGE in RDMA READ -* requests. -*/ -static inline bool rdma_cap_read_multi_sge (struct ib_device *device, -u8 port_num) -{ -return !(device->port_immutable[port_num].core_cap_flags &RDMA_CORE_CAP_PROT_IWARP ); -} - -/** * rdma_max_mad_size - Return the max MAD size required by this RDMA Port. * * @device: Device
Quoted 1 line · Steve Wise
Resending because I forgot to cc linux-rdma :(
All looks sane to me:
Thanks,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Steve Wise
Resending because I forgot to cc linux-rdma :(
Some devices were not setting this capability, so fix those devices, and svcrdma to use
max_sge_rd. Also removerdma_cap_read_multi_sge()since it isn't needed.These patches were originally part of:
Quoted 7 lines
http://www.spinics.net/lists/linux-rdma/msg27436.html
They really aren't part of iSER/iWARP at all, so I've split them out.
Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.
Looks good to me. Thanks for cleaning this cap helper up.
Quoted 12 lines
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Exposemax_sge_rdcorrectlySteve Wise (3):
RDMA/Core: removerdma_cap_read_multi_sge()helper
svcrdma: Usemax_sge_rdfor destination read depths ipath,qib: Exposemax_sge_rdcorrectly
Quoted 18 lines
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + include/linux/sunrpc/svc_rdma.h | 1 + include/rdma/ib_verbs.h | 28 -------------------------- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +---------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 4 ++++ 9 files changed, 11 insertions(+), 39 deletions(-)Signature · 5 lines
Steve -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Steve Wise
Resending because I forgot to cc linux-rdma :(
Some devices were not setting this capability, so fix those devices, and svcrdma to use
max_sge_rd. Also removerdma_cap_read_multi_sge()since it isn't needed.These patches were originally part of:
Quoted 7 lines
They really aren't part of iSER/iWARP at all, so I've split them out.
Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.
---
Looks good,
Quoted 8 lines · Steve Wise
Resending because I forgot to cc linux-rdma :(
Some devices were not setting this capability, so fix those devices, and svcrdma to use
max_sge_rd. Also removerdma_cap_read_multi_sge()since it isn't needed.These patches were originally part of:
Quoted 7 lines
http://www.spinics.net/lists/linux-rdma/msg27436.html
They really aren't part of iSER/iWARP at all, so I've split them out.
Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.
Sounds fine. I've got some changes to those files in my for-next as well, but they don't appear to conflict.
Feel free to add my
to patch 3/4 if it helps.
--b.
Quoted 12 lines
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Exposemax_sge_rdcorrectlySteve Wise (3):
RDMA/Core: removerdma_cap_read_multi_sge()helper
svcrdma: Usemax_sge_rdfor destination read depths ipath,qib: Exposemax_sge_rdcorrectly
Quoted 14 lines
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + include/linux/sunrpc/svc_rdma.h | 1 + include/rdma/ib_verbs.h | 28 -------------------------- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +---------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 4 ++++ 9 files changed, 11 insertions(+), 39 deletions(-)Signature · 1 line
Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 27 lines · Steve Wise
Resending because I forgot to cc linux-rdma :(
Some devices were not setting this capability, so fix those devices, and svcrdma to use
max_sge_rd. Also removerdma_cap_read_multi_sge()since it isn't needed.These patches were originally part of:
http://www.spinics.net/lists/linux-rdma/msg27436.html
They really aren't part of iSER/iWARP at all, so I've split them out.
Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Exposemax_sge_rdcorrectlySteve Wise (3):
RDMA/Core: removerdma_cap_read_multi_sge()helper
svcrdma: Usemax_sge_rdfor destination read depths ipath,qib: Exposemax_sge_rdcorrectly
Quoted 11 lines
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + include/linux/sunrpc/svc_rdma.h | 1 + include/rdma/ib_verbs.h | 28 -------------------------- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +---------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 4 ++++ 9 files changed, 11 insertions(+), 39 deletions(-)
Thanks Steve, I'm picking this up for my for-next work today.
Signature · 2 lines
Doug Ledford <[email protected]> GPG KeyID: 0E572FDD