Use max_sge_rd device capability

PATCH Superseded Steve Wise 1 participant 5 messages Jul 27, 2015

Subsystems (6)

Networking NFS client NFS server InfiniBand +2 more subsystems

lore

Share & tools

lore

Steve Wise

author

2015-07-27 22:57 UTC #0 lore raw

cover Use max_sge_rd device capability

Some devices were not setting this capability, so fix those devices, and svcrdma to use max_sge_rd. Also remove rdma_cap_read_multi_sge() since it isn't needed.

These patches were originally part of:

http://www.spinics.net/lists/linux-rdma/msg27436.html

They really aren't part of iSER/iWARP at all, so I've split them out.

Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems.

---

Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly

Steve Wise (3):
RDMA/Core: remove rdma_cap_read_multi_sge() helper
svcrdma: Use max_sge_rd for destination read depths ipath,qib: Expose max_sge_rd correctly

 drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 +
 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c        |    1 +
 include/linux/sunrpc/svc_rdma.h              |    1 +
 include/rdma/ib_verbs.h                      |   28 --------------------------
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c      |   12 +----------
 net/sunrpc/xprtrdma/svc_rdma_transport.c     |    4 ++++
 9 files changed, 11 insertions(+), 39 deletions(-)

Signature · 1 line

Steve.

Steve Wise

author

2015-07-27 22:57 UTC #1 lore raw

PATCH 1/4 mlx4, mlx5, mthca: Expose max_sge_rd correctly

From: Sagi Grimberg <[email protected]>

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

Reported-by Steve Wise <[email protected]>

Signed-off-by Sagi Grimberg <[email protected]>

---

 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 3 files changed, 3 insertions(+), 0 deletions(-)

drivers/infiniband/hw/mlx4/main.c +1 -0

index 8be6db8..05166b7 100644
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 	props->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	props->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = dev->dev->quotas.cq;
 	props->max_cqe		   = dev->dev->caps.max_cqes;
 	props->max_mr		   = dev->dev->quotas.mpt;

drivers/infiniband/hw/mlx5/main.c +1 -0

index 085c24b..bf27f21 100644
@@ -273,6 +273,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 		     sizeof(struct mlx5_wqe_ctrl_seg)) /
 		     sizeof(struct mlx5_wqe_data_seg);
 	props->max_sge = min(max_rq_sg, max_sq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = 1 << MLX5_CAP_GEN(mdev, log_max_cq);
 	props->max_cqe = (1 << MLX5_CAP_GEN(mdev, log_max_eq_sz)) - 1;
 	props->max_mr		   = 1 << MLX5_CAP_GEN(mdev, log_max_mkey);

drivers/infiniband/hw/mthca/mthca_provider.c +1 -0

index 93ae51d..dc2d48c 100644
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp              = mdev->limits.num_qps - mdev->limits.reserved_qps;
 	props->max_qp_wr           = mdev->limits.max_wqes;
 	props->max_sge             = mdev->limits.max_sg;
+	props->max_sge_rd          = props->max_sge;
 	props->max_cq              = mdev->limits.num_cqs - mdev->limits.reserved_cqs;
 	props->max_cqe             = mdev->limits.max_cqes;
 	props->max_mr              = mdev->limits.num_mpts - mdev->limits.reserved_mrws;

Steve Wise

author

2015-07-27 22:57 UTC #2 lore raw

PATCH 2/4 ipath,qib: Expose max_sge_rd correctly

In reply to Steve Wise

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

Signed-off-by Steve Wise <[email protected]>

Acked-by Mike Marciniszyn <[email protected]>

---

 drivers/infiniband/hw/ipath/ipath_verbs.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c     |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0

index 30ba49c..ed2bbc2 100644
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp = ib_ipath_max_qps;
 	props->max_qp_wr = ib_ipath_max_qp_wrs;
 	props->max_sge = ib_ipath_max_sges;
+	props->max_sge_rd = ib_ipath_max_sges;
 	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = ib_ipath_max_cqes;

drivers/infiniband/hw/qib/qib_verbs.c +1 -0

index a05d1a3..bc723b5 100644
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop
 	props->max_qp = ib_qib_max_qps;
 	props->max_qp_wr = ib_qib_max_qp_wrs;
 	props->max_sge = ib_qib_max_sges;
+	props->max_sge_rd = ib_qib_max_sges;
 	props->max_cq = ib_qib_max_cqs;
 	props->max_ah = ib_qib_max_ahs;
 	props->max_cqe = ib_qib_max_cqes;

Steve Wise

author

2015-07-27 22:58 UTC #3 lore raw

PATCH 3/4 svcrdma: Use max_sge_rd for destination read depths

In reply to Steve Wise

Signed-off-by Steve Wise <[email protected]>

---

 include/linux/sunrpc/svc_rdma.h          |    1 +
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c  |   12 +-----------
 net/sunrpc/xprtrdma/svc_rdma_transport.c |    4 ++++
 3 files changed, 6 insertions(+), 11 deletions(-)

include/linux/sunrpc/svc_rdma.h +1 -0

index cb94ee4..83211bc 100644
@@ -132,6 +132,7 @@ struct svcxprt_rdma {
 	struct list_head     sc_accept_q;	/* Conn. waiting accept */
 	int		     sc_ord;		/* RDMA read limit */
 	int                  sc_max_sge;
+	int                  sc_max_sge_rd;	/* max sge for read target */
 
 	int                  sc_sq_depth;	/* Depth of SQ */
 	atomic_t             sc_sq_count;	/* Number of SQ WR on queue */

net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +1 ~~-11~~

index 2e1348b..cb51742 100644
@@ -115,15 +115,6 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp,
 	rqstp->rq_arg.tail[0].iov_len = 0;
 }
 
-static int rdma_read_max_sge(struct svcxprt_rdma *xprt, int sge_count)
-{
-	if (!rdma_cap_read_multi_sge(xprt->sc_cm_id->device,
-				     xprt->sc_cm_id->port_num))
-		return 1;
-	else
-		return min_t(int, sge_count, xprt->sc_max_sge);
-}
-
 /* Issue an RDMA_READ using the local lkey to map the data sink */
 int rdma_read_chunk_lcl(struct svcxprt_rdma *xprt,
 			struct svc_rqst *rqstp,

@@ -144,8 +135,7 @@ int rdma_read_chunk_lcl(struct svcxprt_rdma *xprt,
 
 	ctxt->direction = DMA_FROM_DEVICE;
 	ctxt->read_hdr = head;
-	pages_needed =
-		min_t(int, pages_needed, rdma_read_max_sge(xprt, pages_needed));
+	pages_needed = min_t(int, pages_needed, xprt->sc_max_sge_rd);
 	read = min_t(int, pages_needed << PAGE_SHIFT, rs_length);
 
 	for (pno = 0; pno < pages_needed; pno++) {

net/sunrpc/xprtrdma/svc_rdma_transport.c +4 -0

index 6b36279..fdc850f 100644
@@ -872,6 +872,8 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 	 * capabilities of this particular device */
 	newxprt->sc_max_sge = min((size_t)devattr.max_sge,
 				  (size_t)RPCSVC_MAXPAGES);
+	newxprt->sc_max_sge_rd = min_t(size_t, devattr.max_sge_rd,
+				       RPCSVC_MAXPAGES);
 	newxprt->sc_max_requests = min((size_t)devattr.max_qp_wr,
 				   (size_t)svcrdma_max_requests);
 	newxprt->sc_sq_depth = RPCRDMA_SQ_DEPTH_MULT * newxprt->sc_max_requests;

@@ -1046,6 +1048,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 		"    remote_ip       : %pI4\n"
 		"    remote_port     : %d\n"
 		"    max_sge         : %d\n"
+		"    max_sge_rd      : %d\n"
 		"    sq_depth        : %d\n"
 		"    max_requests    : %d\n"
 		"    ord             : %d\n",

@@ -1059,6 +1062,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 		ntohs(((struct sockaddr_in *)&newxprt->sc_cm_id->
 		       route.addr.dst_addr)->sin_port),
 		newxprt->sc_max_sge,
+		newxprt->sc_max_sge_rd,
 		newxprt->sc_sq_depth,
 		newxprt->sc_max_requests,
 		newxprt->sc_ord);

Steve Wise

author

2015-07-27 22:58 UTC #4 lore raw

PATCH 4/4 RDMA/Core: remove rdma_cap_read_multi_sge() helper

In reply to Steve Wise

This functionality already exists via the max_sge_rd device capability.

Signed-off-by Steve Wise <[email protected]>

---

 include/rdma/ib_verbs.h |   28 ----------------------------
 1 files changed, 0 insertions(+), 28 deletions(-)

include/rdma/ib_verbs.h +0 ~~-28~~

index b0f898e..7448a27 100644
@@ -2071,34 +2071,6 @@ static inline bool rdma_cap_eth_ah(const struct ib_device *device, u8 port_num)
 }
 
 /**
- * rdma_cap_read_multi_sge - Check if the port of device has the capability
- * RDMA Read Multiple Scatter-Gather Entries.
- * @device: Device to check
- * @port_num: Port number to check
- *
- * iWARP has a restriction that RDMA READ requests may only have a single
- * Scatter/Gather Entry (SGE) in the work request.
- *
- * NOTE: although the linux kernel currently assumes all devices are either
- * single SGE RDMA READ devices or identical SGE maximums for RDMA READs and
- * WRITEs, according to Tom Talpey, this is not accurate.  There are some
- * devices out there that support more than a single SGE on RDMA READ
- * requests, but do not support the same number of SGEs as they do on
- * RDMA WRITE requests.  The linux kernel would need rearchitecting to
- * support these imbalanced READ/WRITE SGEs allowed devices.  So, for now,
- * suffice with either the device supports the same READ/WRITE SGEs, or
- * it only gets one READ sge.
- *
- * Return: true for any device that allows more than one SGE in RDMA READ
- * requests.
- */
-static inline bool rdma_cap_read_multi_sge(struct ib_device *device,
-					   u8 port_num)
-{
-	return !(device->port_immutable[port_num].core_cap_flags & RDMA_CORE_CAP_PROT_IWARP);
-}
-
-/**
  * rdma_max_mad_size - Return the max MAD size required by this RDMA Port.
  *
  * @device: Device