Skip to contentSkip to conversation

iSER support for iWARP

PATCH Superseded Steve Wise 8 participants 23 messages Jul 5, 2015 – Jul 14, 2015 7 messages not in the archive

Subsystems (5)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Steve Wise

author
#0 lore raw

v5 cover iSER support for iWARP

The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.

I've tested this on cxgb4 and mlx4 hardware.

Changes since V4:

iser: fixedcompiler warning

isert: back to setting REMOTE_WRITE only for iWARP devices

Changes since V3:

Fixed commit messages based on feedback.

iser: adjust max_sectors

isert: split into 2 patches

isert: always set REMOTE_WRITE for dma mrs

Changes since V2:

The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.

Changes since V1:

Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.

Correctly set the device max_sge_rd capability in several rdma device drivers.

isert: use device capability max_sge_rd for the read sge depth.

isert: change max_sge to max_write_sge in struct isert_conn.

---

Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly

Steve Wise (4):
RDMA/isert: Limit read depth based on the device max_sge_rd capability RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth ipath,qib: Expose max_sge_rd correctly

 drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 
 drivers/infiniband/hw/mlx4/main.c            |    1 
 drivers/infiniband/hw/mlx5/main.c            |    1 
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 
 drivers/infiniband/hw/qib/qib_verbs.c        |    1 
 drivers/infiniband/ulp/iser/iscsi_iser.c     |    9 ++++
 drivers/infiniband/ulp/isert/ib_isert.c      |   53 ++++++++++++++++++++++----
 drivers/infiniband/ulp/isert/ib_isert.h      |    3 +
 8 files changed, 60 insertions(+), 10 deletions(-)
Signature · 1 line
Steve.
author
#1 lore raw

PATCH v5 1/5 mlx4, mlx5, mthca: Expose max_sge_rd correctly

From: Sagi Grimberg <[email protected]>

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

---

 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 3 files changed, 3 insertions(+), 0 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -0
index 166da78..81c342f 100644
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 	props->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	props->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = dev->dev->quotas.cq;
 	props->max_cqe		   = dev->dev->caps.max_cqes;
 	props->max_mr		   = dev->dev->quotas.mpt;
drivers/infiniband/hw/mlx5/main.c +1 -0
index c6cb26e..a2bcd96 100644
@@ -137,6 +137,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	max_sq_sg = (gen->max_sq_desc_sz - sizeof(struct mlx5_wqe_ctrl_seg)) /
 		sizeof(struct mlx5_wqe_data_seg);
 	props->max_sge = min(max_rq_sg, max_sq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = 1 << gen->log_max_cq;
 	props->max_cqe		   = gen->max_cqes - 1;
 	props->max_mr		   = 1 << gen->log_max_mkey;
drivers/infiniband/hw/mthca/mthca_provider.c +1 -0
index 93ae51d..dc2d48c 100644
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp              = mdev->limits.num_qps - mdev->limits.reserved_qps;
 	props->max_qp_wr           = mdev->limits.max_wqes;
 	props->max_sge             = mdev->limits.max_sg;
+	props->max_sge_rd          = props->max_sge;
 	props->max_cq              = mdev->limits.num_cqs - mdev->limits.reserved_cqs;
 	props->max_cqe             = mdev->limits.max_cqes;
 	props->max_mr              = mdev->limits.num_mpts - mdev->limits.reserved_mrws;
author
#2 lore raw

PATCH v5 2/5 ipath,qib: Expose max_sge_rd correctly

In reply to Steve Wise

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

---

 drivers/infiniband/hw/ipath/ipath_verbs.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c     |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
index 48253b8..d958236 100644
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp = ib_ipath_max_qps;
 	props->max_qp_wr = ib_ipath_max_qp_wrs;
 	props->max_sge = ib_ipath_max_sges;
+	props->max_sge_rd = ib_ipath_max_sges;
 	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = ib_ipath_max_cqes;
drivers/infiniband/hw/qib/qib_verbs.c +1 -0
index a05d1a3..bc723b5 100644
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop
 	props->max_qp = ib_qib_max_qps;
 	props->max_qp_wr = ib_qib_max_qp_wrs;
 	props->max_sge = ib_qib_max_sges;
+	props->max_sge_rd = ib_qib_max_sges;
 	props->max_cq = ib_qib_max_cqs;
 	props->max_ah = ib_qib_max_ahs;
 	props->max_cqe = ib_qib_max_cqes;
author
#3 lore raw

PATCH v5 3/5 RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

In reply to Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---

 drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
 						   SHOST_DIX_GUARD_CRC);
 		}
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+
 		if (iscsi_host_add(shost,
 				   ib_conn->device->ib_device->dma_device)) {
 			mutex_unlock(&iser_conn->state_mutex);
author
#13 lore raw

PATCH v5 4/5 RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices

In reply to Steve Wise

iWARP devices require REMOTE_WRITE for MRs used as the destination of an RDMA READ. So if the device protocol is iWARP, then set REMOTE_WRITE when allocating the DMA MR.

---

 drivers/infiniband/ulp/isert/ib_isert.c |   22 +++++++++++++++++++++-
 1 files changed, 21 insertions(+), 1 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +21 -1
index 9e7b492..ee40478 100644
@@ -348,6 +348,17 @@ out_cq:
 	return ret;
 }
 
+static int any_port_is_iwarp(struct isert_device *device)
+{
+	int i;
+
+	for (i = rdma_start_port(device->ib_device);
+	     i <= rdma_end_port(device->ib_device); i++)
+		if (rdma_protocol_iwarp(device->ib_device, i))
+			return 1;
+	return 0;
+}
+
 static int
 isert_create_device_ib_res(struct isert_device *device)
 {
@@ -383,7 +394,16 @@ isert_create_device_ib_res(struct isert_device *device)
 		goto out_cq;
 	}
 
-	device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE);
+	/*
+	 * IWARP transports need REMOTE_WRITE for MRs used as the target of
+	 * an RDMA_READ.  Since the DMA MR is used for all ports, then if
+	 * any port is running IWARP, add REMOTE_WRITE.
+	 */
+	if (any_port_is_iwarp(device))
+		device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE |
+						       IB_ACCESS_REMOTE_WRITE);
+	else
+		device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(device->mr)) {
 		ret = PTR_ERR(device->mr);
 		isert_err("failed to create dma mr, device %p, ret=%d\n",
author
#16 lore raw

PATCH v5 5/5 RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to Steve Wise

Use the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.

---

 drivers/infiniband/ulp/isert/ib_isert.c |   31 +++++++++++++++++++++++--------
 drivers/infiniband/ulp/isert/ib_isert.h |    3 ++-
 2 files changed, 25 insertions(+), 9 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +23 -8
index ee40478..35015c9 100644
@@ -163,7 +163,9 @@ isert_create_qp(struct isert_conn *isert_conn,
 	 * outgoing control PDU responses.
 	 */
 	attr.cap.max_send_sge = max(2, device->dev_attr.max_sge - 2);
-	isert_conn->max_sge = attr.cap.max_send_sge;
+	isert_conn->max_write_sge = attr.cap.max_send_sge;
+	isert_conn->max_read_sge = min_t(u32, device->dev_attr.max_sge_rd,
+					 attr.cap.max_send_sge);
 
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
@@ -2395,7 +2397,7 @@ isert_put_text_rsp(struct iscsi_cmd *cmd, struct iscsi_conn *conn)
 static int
 isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 		    struct ib_sge *ib_sge, struct ib_send_wr *send_wr,
-		    u32 data_left, u32 offset)
+		    u32 data_left, u32 offset, u32 max_sge)
 {
 	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
 	struct scatterlist *sg_start, *tmp_sg;
@@ -2406,7 +2408,7 @@ isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 
 	sg_off = offset / PAGE_SIZE;
 	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, isert_conn->max_sge);
+	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, max_sge);
 	page_off = offset % PAGE_SIZE;
 
 	send_wr->sg_list = ib_sge;
@@ -2450,8 +2452,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct isert_data_buf *data = &wr->data;
 	struct ib_send_wr *send_wr;
 	struct ib_sge *ib_sge;
-	u32 offset, data_len, data_left, rdma_write_max, va_offset = 0;
+	u32 offset, data_len, data_left, rdma_max_len, va_offset = 0;
 	int ret = 0, i, ib_sge_cnt;
+	u32 max_sge;
 
 	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 
@@ -2473,7 +2476,12 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 	wr->ib_sge = ib_sge;
 
-	wr->send_wr_num = DIV_ROUND_UP(data->nents, isert_conn->max_sge);
+	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE)
+		max_sge = isert_conn->max_write_sge;
+	else
+		max_sge =  isert_conn->max_read_sge;
+
+	wr->send_wr_num = DIV_ROUND_UP(data->nents, max_sge);
 	wr->send_wr = kzalloc(sizeof(struct ib_send_wr) * wr->send_wr_num,
 				GFP_KERNEL);
 	if (!wr->send_wr) {
@@ -2483,11 +2491,11 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 
 	wr->isert_cmd = isert_cmd;
-	rdma_write_max = isert_conn->max_sge * PAGE_SIZE;
+	rdma_max_len = max_sge * PAGE_SIZE;
 
 	for (i = 0; i < wr->send_wr_num; i++) {
 		send_wr = &isert_cmd->rdma_wr.send_wr[i];
-		data_len = min(data_left, rdma_write_max);
+		data_len = min(data_left, rdma_max_len);
 
 		send_wr->send_flags = 0;
 		if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2509,7 +2517,7 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		}
 
 		ib_sge_cnt = isert_build_rdma_wr(isert_conn, isert_cmd, ib_sge,
-					send_wr, data_len, offset);
+					send_wr, data_len, offset, max_sge);
 		ib_sge += ib_sge_cnt;
 
 		offset += data_len;
@@ -2638,6 +2646,13 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 	fr_wr.wr.fast_reg.rkey = mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
+	/*
+	 * IWARP transports need REMOTE_WRITE for MRs used as the target of
+	 * an RDMA_READ.
+	 */
+	if (rdma_protocol_iwarp(ib_dev, isert_conn->cm_id->port_num))
+		fr_wr.wr.fast_reg.access_flags |= IB_ACCESS_REMOTE_WRITE;
+
 	if (!wr)
 		wr = &fr_wr;
 	else
drivers/infiniband/ulp/isert/ib_isert.h +2 -1
index 9ec23a7..29fde27 100644
@@ -152,7 +152,8 @@ struct isert_conn {
 	u32			responder_resources;
 	u32			initiator_depth;
 	bool			pi_support;
-	u32			max_sge;
+	u32			max_write_sge;
+	u32			max_read_sge;
 	char			*login_buf;
 	char			*login_req_buf;
 	char			*login_rsp_buf;
#5 lore raw

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

In reply to a message not in the archive
Quoted 18 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---

  drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
  1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
  						   SHOST_DIX_GUARD_CRC);
  		}

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Reviewed-by
#15 lore raw

Re: [PATCH V5 4/5] RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices

In reply to a message not in the archive
Quoted 5 lines · Steve Wise

iWARP devices require REMOTE_WRITE for MRs used as the destination of an RDMA READ. So if the device protocol is iWARP, then set REMOTE_WRITE when allocating the DMA MR.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Reviewed-by
#18 lore raw

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to a message not in the archive
Quoted 5 lines · Steve Wise

Use the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

In reply to a message not in the archive
Quoted 30 lines

-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 19 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---

  drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
  1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
  						   SHOST_DIX_GUARD_CRC);
  		}
 
Quoted 13 lines

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#29 lore raw
In reply to a message not in the archive

Hi Steve & Co,

Quoted 51 lines · Steve Wise

The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.

I've tested this on cxgb4 and mlx4 hardware.

Changes since V4:

iser: fixedcompiler warning

isert: back to setting REMOTE_WRITE only for iWARP devices

Changes since V3:

Fixed commit messages based on feedback.

iser: adjust max_sectors

isert: split into 2 patches

isert: always set REMOTE_WRITE for dma mrs

Changes since V2:

The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.

Changes since V1:

Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.

Correctly set the device max_sge_rd capability in several rdma device drivers.

isert: use device capability max_sge_rd for the read sge depth.

isert: change max_sge to max_write_sge in struct isert_conn.

---

Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly

Steve Wise (4):
RDMA/isert: Limit read depth based on the device max_sge_rd capability RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth ipath,qib: Expose max_sge_rd correctly

Quoted 10 lines
 drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 
 drivers/infiniband/hw/mlx4/main.c            |    1 
 drivers/infiniband/hw/mlx5/main.c            |    1 
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 
 drivers/infiniband/hw/qib/qib_verbs.c        |    1 
 drivers/infiniband/ulp/iser/iscsi_iser.c     |    9 ++++
 drivers/infiniband/ulp/isert/ib_isert.c      |   53 ++++++++++++++++++++++----
 drivers/infiniband/ulp/isert/ib_isert.h      |    3 +
 8 files changed, 60 insertions(+), 10 deletions(-)

Very excited to see iWARP support for iser-initiator + iser-target. ;)

No objections to taking the iser-target changes through Doug's infiniband.git. For the target pieces:

Also Doug, since this series is sufficiently small enough, and allows a class of hardware that people are expecting to function with iser to 'just work', I'd recommend to go ahead and push this series for v4.2-rc code.

Thank you,

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

In reply to Steve Wise
Quoted 44 lines · Steve Wise
Quoted 30 lines

-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 19 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---

   drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
   1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
   						   SHOST_DIX_GUARD_CRC);
   		}
 
Quoted 13 lines

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

Quoted 2 lines

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).

author
#9 lore raw

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 55 lines

-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Tuesday, July 07, 2015 1:27 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 44 lines · Steve Wise
Quoted 43 lines

-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 18 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---

   drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
   1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
   						   SHOST_DIX_GUARD_CRC);
   		}

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

Quoted 5 lines
Quoted 2 lines

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can remove the min_t() though.

Hey Or, thoughts?

Steve.

#10 lore raw

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
   						   SHOST_DIX_GUARD_CRC);
   		}
 
Quoted 9 lines

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can remove the min_t() though.

Hey Or, thoughts?

Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.

Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the max_sectors to be <= 1024

If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#11 lore raw

RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 47 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Or Gerlitz
Sent: Tuesday, July 07, 2015 9:32 AM
To: Steve Wise; 'Sagi Grimberg'
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; target- [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
   						   SHOST_DIX_GUARD_CRC);
   		}
 
Quoted 9 lines

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can

remove the min_t() though.

Quoted 2 lines

Hey Or, thoughts?

Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.

Quoted 8 lines

Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the max_sectors to be <= 1024

If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.

I'm not sure... you guys are the iSER experts. :) But considering a 64K PAGE_SIZE and an adjusted sg_tablesize of say, 32, w/o the min() we get (32 * 65536) >> 9 == 4096. So if the requirement is that max_sectors always be <= 1024, then we need the min()...

Steve.

#12 lore raw

Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 52 lines · Steve Wise
Quoted 52 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Or Gerlitz
Sent: Tuesday, July 07, 2015 9:32 AM
To: Steve Wise; 'Sagi Grimberg'
Cc: [email protected]; [email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; target- [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..de8730d 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
    						   SHOST_DIX_GUARD_CRC);
    		}
 
Quoted 9 lines

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors = min_t(unsigned int,
+			1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.

Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).

Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can

remove the min_t() though.

Quoted 2 lines

Hey Or, thoughts?

Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.

Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the max_sectors to be <= 1024

Quoted 6 lines
Quoted 3 lines

If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.

I'm not sure... you guys are the iSER experts. :) But considering a 64K PAGE_SIZE and an adjusted sg_tablesize of say, 32, w/o the min() we get (32 * 65536) >> 9 == 4096. So if the requirement is that max_sectors always be <= 1024, then we need the min()...

I guess the min is fine given that I have a patchset for 8MB support, so we'll get it right then.

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#19 lore raw

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to a message not in the archive
Quoted 3 lines · Steve Wise

Use the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to a message not in the archive
Quoted 10 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Christoph Hellwig
Sent: Tuesday, July 14, 2015 3:27 AM
To: Steve Wise
Cc: [email protected]; [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; linux- [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 7 lines
Quoted 3 lines · Steve Wise

Use the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Chuck, can you add this to your cleanup list?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#22 lore raw

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 4 lines · Steve Wise
Quoted 2 lines

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to a message not in the archive
Quoted 11 lines

-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

On Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.

Ok, I will do this.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#25 lore raw

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 14 lines · Steve Wise
Quoted 14 lines

-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 3 lines · Steve Wise
Quoted 2 lines

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.

Ok, I will do this.

Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:

http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190

Which changes the way xprtrdma uses max_sge.

--
Chuck Lever

author
#26 lore raw

RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 7 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chuck Lever
Sent: Tuesday, July 14, 2015 1:47 PM
To: Steve Wise
Cc: Christoph Hellwig; Doug Ledford; [email protected]; Sagi Grimberg; Or Gerlitz; [email protected]; linux-rdma; Eli Cohen;

target-

Quoted 26 lines

[email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 20 lines · Steve Wise
Quoted 19 lines

-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 4 lines · Steve Wise
Quoted 2 lines

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.

Quoted 9 lines
Quoted 1 line

Ok, I will do this.

Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:

http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190

Which changes the way xprtrdma uses max_sge.

Sure, but xprtrdma doesn't issue reads, so max_sge_rd isn't needed. The change Christoph wants is actually in svcrdma. I will change the server to use max_sge_rd instead of rdma_cap_read_multi_sge().

#27 lore raw

Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 39 lines · Steve Wise
Quoted 7 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chuck Lever
Sent: Tuesday, July 14, 2015 1:47 PM
To: Steve Wise
Cc: Christoph Hellwig; Doug Ledford; [email protected]; Sagi Grimberg; Or Gerlitz; [email protected]; linux-rdma; Eli Cohen;

target-

Quoted 31 lines

[email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 14 lines · Steve Wise
Quoted 14 lines

-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability

Quoted 3 lines · Steve Wise
Quoted 2 lines

Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Chuck, can you add this to your cleanup list?

It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.

Ok, I will do this.

Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:

Quoted 7 lines
Quoted 4 lines

http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190

Which changes the way xprtrdma uses max_sge.

Sure, but xprtrdma doesn't issue reads, so max_sge_rd isn't needed. The change Christoph wants is actually in svcrdma. I will change the server to use max_sge_rd instead of rdma_cap_read_multi_sge().

OK, glad the conflict has been avoided.

--
Chuck Lever