Skip to contentSkip to conversation

iSER support for iWARP

PATCH Superseded Steve Wise 3 participants 8 messages Jul 2, 2015 – Jul 5, 2015 3 messages not in the archive

Subsystems (5)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Steve Wise

author
#0 lore raw

v4 cover iSER support for iWARP

The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.

I've tested this on cxgb4 and mlx4 hardware.

Changes since V3:

Fixed commit messages based on feedback.

iser: adjust max_sectors

isert: split into 2 patches

isert: always set REMOTE_WRITE for dma mrs

Changes since V2:

The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.

Changes since V1:

Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.

Correctly set the device max_sge_rd capability in several rdma device drivers.

isert: use device capability max_sge_rd for the read sge depth.

isert: change max_sge to max_write_sge in struct isert_conn.

---

Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly

Steve Wise (4):
ipath,qib: Expose max_sge_rd correctly
RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices RDMA/isert: Limit read depth based on the device max_sge_rd capability

 drivers/infiniband/hw/ipath/ipath_verbs.c    |    1 +
 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c        |    1 +
 drivers/infiniband/ulp/iser/iscsi_iser.c     |    9 +++++++
 drivers/infiniband/ulp/isert/ib_isert.c      |   34 +++++++++++++++++++-------
 drivers/infiniband/ulp/isert/ib_isert.h      |    3 ++
 8 files changed, 41 insertions(+), 10 deletions(-)

--
Steve.

author
#1 lore raw

PATCH v4 1/5 mlx4, mlx5, mthca: Expose max_sge_rd correctly

From: Sagi Grimberg <[email protected]>

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

---
 drivers/infiniband/hw/mlx4/main.c            |    1 +
 drivers/infiniband/hw/mlx5/main.c            |    1 +
 drivers/infiniband/hw/mthca/mthca_provider.c |    1 +
 3 files changed, 3 insertions(+), 0 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -0
index 166da78..81c342f 100644
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 	props->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	props->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = dev->dev->quotas.cq;
 	props->max_cqe		   = dev->dev->caps.max_cqes;
 	props->max_mr		   = dev->dev->quotas.mpt;
drivers/infiniband/hw/mlx5/main.c +1 -0
index c6cb26e..a2bcd96 100644
@@ -137,6 +137,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	max_sq_sg = (gen->max_sq_desc_sz - sizeof(struct mlx5_wqe_ctrl_seg)) /
 		sizeof(struct mlx5_wqe_data_seg);
 	props->max_sge = min(max_rq_sg, max_sq_sg);
+	props->max_sge_rd = props->max_sge;
 	props->max_cq		   = 1 << gen->log_max_cq;
 	props->max_cqe		   = gen->max_cqes - 1;
 	props->max_mr		   = 1 << gen->log_max_mkey;
drivers/infiniband/hw/mthca/mthca_provider.c +1 -0
index 93ae51d..dc2d48c 100644
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp              = mdev->limits.num_qps - mdev->limits.reserved_qps;
 	props->max_qp_wr           = mdev->limits.max_wqes;
 	props->max_sge             = mdev->limits.max_sg;
+	props->max_sge_rd          = props->max_sge;
 	props->max_cq              = mdev->limits.num_cqs - mdev->limits.reserved_cqs;
 	props->max_cqe             = mdev->limits.max_cqes;
 	props->max_mr              = mdev->limits.num_mpts - mdev->limits.reserved_mrws;

A message that is not in the archive (sent off-list or not yet received): 20150702151508.12946.29133.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org

#3 lore raw

PATCH v4 2/5 ipath,qib: Expose max_sge_rd correctly

Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.

---
 drivers/infiniband/hw/ipath/ipath_verbs.c |    1 +
 drivers/infiniband/hw/qib/qib_verbs.c     |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
index 48253b8..d958236 100644
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr
 	props->max_qp = ib_ipath_max_qps;
 	props->max_qp_wr = ib_ipath_max_qp_wrs;
 	props->max_sge = ib_ipath_max_sges;
+	props->max_sge_rd = ib_ipath_max_sges;
 	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = ib_ipath_max_cqes;
drivers/infiniband/hw/qib/qib_verbs.c +1 -0
index a05d1a3..bc723b5 100644
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop
 	props->max_qp = ib_qib_max_qps;
 	props->max_qp_wr = ib_qib_max_qp_wrs;
 	props->max_sge = ib_qib_max_sges;
+	props->max_sge_rd = ib_qib_max_sges;
 	props->max_cq = ib_qib_max_cqs;
 	props->max_ah = ib_qib_max_ahs;
 	props->max_cqe = ib_qib_max_cqes;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

PATCH v4 3/5 RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

In reply to a message not in the archive

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---
 drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
index 6a594aa..0ed8638 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
 						   SHOST_DIX_GUARD_CRC);
 		}
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(u32, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors =
+			min(1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+
 		if (iscsi_host_add(shost,
 				   ib_conn->device->ib_device->dma_device)) {
 			mutex_unlock(&iser_conn->state_mutex);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20150702151619.12946.79500.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org

#6 lore raw

Re: [PATCH V4 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 18 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---
  drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
  1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..0ed8638 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep,
  						   SHOST_DIX_GUARD_CRC);
  		}

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651

Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
 
+		/*
+		 * Limit the sg_tablesize and max_sectors based on the device
+		 * max fastreg page list length.
+		 */
+		shost->sg_tablesize = min_t(u32, shost->sg_tablesize,
+			ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+		shost->max_sectors =
+			min(1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

Steve,

This triggers a compilation warning:
drivers/infiniband/ulp/iser/iscsi_iser.c: In function iscsi_iser_session_create:
drivers/infiniband/ulp/iser/iscsi_iser.c:653: warning: comparison of distinct pointer types lacks a cast

For next time, let's remove the reviewed-by tags and I'll re-review the patches.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 5598FFA3.5020508-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

#8 lore raw

Re: [PATCH V4 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

Quoted 33 lines · Sagi Grimberg
Quoted 19 lines · Steve Wise

Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.

---
  drivers/infiniband/ulp/iser/iscsi_iser.c |    9 +++++++++
  1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +0 -0
index 6a594aa..0ed8638 100644
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint 
 *ep,
                             SHOST_DIX_GUARD_CRC);
          }

drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint · line 652

Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
 
+        /*
+         * Limit the sg_tablesize and max_sectors based on the device
+         * max fastreg page list length.
+         */
+        shost->sg_tablesize = min_t(u32, shost->sg_tablesize,
+ ib_conn->device->dev_attr.max_fast_reg_page_list_len);
+        shost->max_sectors =
+            min(1024, (shost->sg_tablesize * PAGE_SIZE) >> 9);
+

Steve,

Quoted 6 lines

This triggers a compilation warning:
drivers/infiniband/ulp/iser/iscsi_iser.c: In function iscsi_iser_session_create:
drivers/infiniband/ulp/iser/iscsi_iser.c:653: warning: comparison of distinct pointer types lacks a cast

For next time, let's remove the reviewed-by tags and I'll re-review the patches.

Oops. My bad.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw

PATCH v4 5/5 RDMA/isert: Limit read depth based on the device max_sge_rd capability

In reply to a message not in the archive

Use the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.

---
 drivers/infiniband/ulp/isert/ib_isert.c |   31 +++++++++++++++++++++++--------
 drivers/infiniband/ulp/isert/ib_isert.h |    3 ++-
 2 files changed, 25 insertions(+), 9 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +23 -8
index 2737ade..c2c7037 100644
@@ -163,7 +163,9 @@ isert_create_qp(struct isert_conn *isert_conn,
 	 * outgoing control PDU responses.
 	 */
 	attr.cap.max_send_sge = max(2, device->dev_attr.max_sge - 2);
-	isert_conn->max_sge = attr.cap.max_send_sge;
+	isert_conn->max_write_sge = attr.cap.max_send_sge;
+	isert_conn->max_read_sge = min_t(u32, device->dev_attr.max_sge_rd,
+					 attr.cap.max_send_sge);
 
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
@@ -2376,7 +2378,7 @@ isert_put_text_rsp(struct iscsi_cmd *cmd, struct iscsi_conn *conn)
 static int
 isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 		    struct ib_sge *ib_sge, struct ib_send_wr *send_wr,
-		    u32 data_left, u32 offset)
+		    u32 data_left, u32 offset, u32 max_sge)
 {
 	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
 	struct scatterlist *sg_start, *tmp_sg;
@@ -2387,7 +2389,7 @@ isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
 
 	sg_off = offset / PAGE_SIZE;
 	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, isert_conn->max_sge);
+	sg_nents = min(cmd->se_cmd.t_data_nents - sg_off, max_sge);
 	page_off = offset % PAGE_SIZE;
 
 	send_wr->sg_list = ib_sge;
@@ -2431,8 +2433,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct isert_data_buf *data = &wr->data;
 	struct ib_send_wr *send_wr;
 	struct ib_sge *ib_sge;
-	u32 offset, data_len, data_left, rdma_write_max, va_offset = 0;
+	u32 offset, data_len, data_left, rdma_max_len, va_offset = 0;
 	int ret = 0, i, ib_sge_cnt;
+	u32 max_sge;
 
 	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 
@@ -2454,7 +2457,12 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 	wr->ib_sge = ib_sge;
 
-	wr->send_wr_num = DIV_ROUND_UP(data->nents, isert_conn->max_sge);
+	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE)
+		max_sge = isert_conn->max_write_sge;
+	else
+		max_sge =  isert_conn->max_read_sge;
+
+	wr->send_wr_num = DIV_ROUND_UP(data->nents, max_sge);
 	wr->send_wr = kzalloc(sizeof(struct ib_send_wr) * wr->send_wr_num,
 				GFP_KERNEL);
 	if (!wr->send_wr) {
@@ -2464,11 +2472,11 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 
 	wr->isert_cmd = isert_cmd;
-	rdma_write_max = isert_conn->max_sge * PAGE_SIZE;
+	rdma_max_len = max_sge * PAGE_SIZE;
 
 	for (i = 0; i < wr->send_wr_num; i++) {
 		send_wr = &isert_cmd->rdma_wr.send_wr[i];
-		data_len = min(data_left, rdma_write_max);
+		data_len = min(data_left, rdma_max_len);
 
 		send_wr->send_flags = 0;
 		if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2490,7 +2498,7 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		}
 
 		ib_sge_cnt = isert_build_rdma_wr(isert_conn, isert_cmd, ib_sge,
-					send_wr, data_len, offset);
+					send_wr, data_len, offset, max_sge);
 		ib_sge += ib_sge_cnt;
 
 		offset += data_len;
@@ -2619,6 +2627,13 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 	fr_wr.wr.fast_reg.rkey = mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
+	/*
+	 * IWARP transports need REMOTE_WRITE for MRs used as the target of
+	 * an RDMA_READ.
+	 */
+	if (rdma_protocol_iwarp(ib_dev, isert_conn->cm_id->port_num))
+		fr_wr.wr.fast_reg.access_flags |= IB_ACCESS_REMOTE_WRITE;
+
 	if (!wr)
 		wr = &fr_wr;
 	else
drivers/infiniband/ulp/isert/ib_isert.h +2 -1
index 9ec23a7..29fde27 100644
@@ -152,7 +152,8 @@ struct isert_conn {
 	u32			responder_resources;
 	u32			initiator_depth;
 	bool			pi_support;
-	u32			max_sge;
+	u32			max_write_sge;
+	u32			max_read_sge;
 	char			*login_buf;
 	char			*login_req_buf;
 	char			*login_rsp_buf;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw

PATCH v4 4/5 RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices

In reply to Steve Wise

iWARP devices require REMOTE_WRITE for MRs used as the destination of an RDMA READ. IB does not require this. Since isert never advertises the rkey of its DMA MRs to the peer, just add REMOTE_WRITE to the access flags when allocating DMA MRs for all devices.

---
 drivers/infiniband/ulp/isert/ib_isert.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +2 -1
index 9e7b492..2737ade 100644
@@ -383,7 +383,8 @@ isert_create_device_ib_res(struct isert_device *device)
 		goto out_cq;
 	}
 
-	device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE);
+	device->mr = ib_get_dma_mr(device->pd, IB_ACCESS_LOCAL_WRITE |
+					       IB_ACCESS_REMOTE_WRITE);
 	if (IS_ERR(device->mr)) {
 		ret = PTR_ERR(device->mr);
 		isert_err("failed to create dma mr, device %p, ret=%d\n",