iSER support for iWARP
Subsystems (5)
v4 cover iSER support for iWARP
The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.
I've tested this on cxgb4 and mlx4 hardware.
Changes since V3:
Fixed commit messages based on feedback.
iser: adjust max_sectors
isert: split into 2 patches
isert: always set REMOTE_WRITE for dma mrs
Changes since V2:
The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.
Changes since V1:
Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.
Correctly set the device max_sge_rd capability in several rdma device drivers.
isert: use device capability max_sge_rd for the read sge depth.
isert: change max_sge to max_write_sge in struct isert_conn.
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly
Steve Wise (4):
ipath,qib: Expose max_sge_rd correctly
RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices RDMA/isert: Limit read depth based on the device max_sge_rd capability
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++ drivers/infiniband/ulp/isert/ib_isert.c | 34 +++++++++++++++++++------- drivers/infiniband/ulp/isert/ib_isert.h | 3 ++ 8 files changed, 41 insertions(+), 10 deletions(-)
--
Steve.
PATCH v4 1/5 mlx4, mlx5, mthca: Expose max_sge_rd correctly
From: Sagi Grimberg <[email protected]>
Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
--- drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + 3 files changed, 3 insertions(+), 0 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -0
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, props->max_qp_wr = dev->dev->caps.max_wqes -MLX4_IB_SQ_MAX_SPARE ; props->max_sge =min (dev->dev->caps.max_sq_sg, dev->dev->caps.max_rq_sg); + props->max_sge_rd = props->max_sge; props->max_cq = dev->dev->quotas.cq; props->max_cqe = dev->dev->caps.max_cqes; props->max_mr = dev->dev->quotas.mpt;
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -137,6 +137,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, max_sq_sg = (gen->max_sq_desc_sz -sizeof (struct mlx5_wqe_ctrl_seg )) /sizeof (struct mlx5_wqe_data_seg ); props->max_sge =min (max_rq_sg, max_sq_sg); + props->max_sge_rd = props->max_sge; props->max_cq =1 << gen->log_max_cq; props->max_cqe = gen->max_cqes -1 ; props->max_mr =1 << gen->log_max_mkey;
drivers/infiniband/hw/mthca/mthca_provider.c +1 -0
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = mdev->limits.num_qps - mdev->limits.reserved_qps; props->max_qp_wr = mdev->limits.max_wqes; props->max_sge = mdev->limits.max_sg; + props->max_sge_rd = props->max_sge; props->max_cq = mdev->limits.num_cqs - mdev->limits.reserved_cqs; props->max_cqe = mdev->limits.max_cqes; props->max_mr = mdev->limits.num_mpts - mdev->limits.reserved_mrws;
A message that is not in the archive (sent off-list or not yet received): 20150702151508.12946.29133.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org
PATCH v4 2/5 ipath,qib: Expose max_sge_rd correctly
Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
--- drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + 2 files changed, 2 insertions(+), 0 deletions(-)
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = ib_ipath_max_qps; props->max_qp_wr = ib_ipath_max_qp_wrs; props->max_sge = ib_ipath_max_sges; + props->max_sge_rd = ib_ipath_max_sges; props->max_cq = ib_ipath_max_cqs; props->max_ah = ib_ipath_max_ahs; props->max_cqe = ib_ipath_max_cqes;
drivers/infiniband/hw/qib/qib_verbs.c +1 -0
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop props->max_qp = ib_qib_max_qps; props->max_qp_wr = ib_qib_max_qp_wrs; props->max_sge = ib_qib_max_sges; + props->max_sge_rd = ib_qib_max_sges; props->max_cq = ib_qib_max_cqs; props->max_ah = ib_qib_max_ahs; props->max_cqe = ib_qib_max_cqes;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v4 3/5 RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
In reply to a message not in the archiveCurrently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.
--- drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); } +/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize =min_t (u32, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = +min (1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +if (iscsi_host_add (shost, ib_conn->device->ib_device->dma_device)) {mutex_unlock (&iser_conn->state_mutex);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20150702151619.12946.79500.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org
Re: [PATCH V4 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 18 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>--- drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize =min_t (u32, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = +min (1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +
Steve,
This triggers a compilation warning:drivers/infiniband/ulp/iser/iscsi_iser.c: In function iscsi_iser_session_create:drivers/infiniband/ulp/iser/iscsi_iser.c:653: warning: comparison of distinct pointer types lacks a cast
For next time, let's remove the reviewed-by tags and I'll re-review the patches.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 5598FFA3.5020508-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
Re: [PATCH V4 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 33 lines · Sagi Grimberg
Quoted 19 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>--- drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint · line 652
Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize =min_t (u32, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = +min (1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +Steve,
Quoted 6 lines
This triggers a compilation warning:
drivers/infiniband/ulp/iser/iscsi_iser.c: In functioniscsi_iser_session_create:drivers/infiniband/ulp/iser/iscsi_iser.c:653: warning: comparison of distinct pointer types lacks a castFor next time, let's remove the reviewed-by tags and I'll re-review the patches.
Oops. My bad.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v4 5/5 RDMA/isert: Limit read depth based on the device max_sge_rd capability
In reply to a message not in the archiveUse the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.
--- drivers/infiniband/ulp/isert/ib_isert.c | 31 +++++++++++++++++++++++-------- drivers/infiniband/ulp/isert/ib_isert.h | 3 ++- 2 files changed, 25 insertions(+), 9 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +23 -8
@@ -163,7 +163,9 @@ isert_create_qp(struct isert_conn *isert_conn,* outgoing control PDU responses. */ attr.cap.max_send_sge =max (2 , device->dev_attr.max_sge -2 ); - isert_conn->max_sge = attr.cap.max_send_sge; + isert_conn->max_write_sge = attr.cap.max_send_sge; + isert_conn->max_read_sge =min_t (u32, device->dev_attr.max_sge_rd, + attr.cap.max_send_sge); attr.cap.max_recv_sge =1 ; attr.sq_sig_type =IB_SIGNAL_REQ_WR ;
@@ -2376,7 +2378,7 @@ isert_put_text_rsp(struct iscsi_cmd *cmd, struct iscsi_conn *conn)static int isert_build_rdma_wr (struct isert_conn *isert_conn,struct isert_cmd *isert_cmd,struct ib_sge *ib_sge,struct ib_send_wr *send_wr, -u32 data_left,u32 offset) +u32 data_left,u32 offset,u32 max_sge) {struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;struct scatterlist *sg_start, *tmp_sg;
@@ -2387,7 +2389,7 @@ isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd, sg_off = offset /PAGE_SIZE ; sg_start = &cmd->se_cmd.t_data_sg[sg_off]; - sg_nents =min (cmd->se_cmd.t_data_nents - sg_off, isert_conn->max_sge); + sg_nents =min (cmd->se_cmd.t_data_nents - sg_off, max_sge); page_off = offset %PAGE_SIZE ; send_wr->sg_list = ib_sge;
@@ -2431,8 +2433,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,struct isert_data_buf *data = &wr->data;struct ib_send_wr *send_wr;struct ib_sge *ib_sge; -u32 offset, data_len, data_left, rdma_write_max, va_offset =0 ; +u32 offset, data_len, data_left, rdma_max_len, va_offset =0 ;int ret =0 , i, ib_sge_cnt; +u32 max_sge; isert_cmd->tx_desc.isert_cmd = isert_cmd;
@@ -2454,7 +2457,12 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } wr->ib_sge = ib_sge; - wr->send_wr_num =DIV_ROUND_UP (data->nents, isert_conn->max_sge); +if (wr->iser_ib_op ==ISER_IB_RDMA_WRITE ) + max_sge = isert_conn->max_write_sge; +else + max_sge = isert_conn->max_read_sge; + + wr->send_wr_num =DIV_ROUND_UP (data->nents, max_sge); wr->send_wr =kzalloc (sizeof (struct ib_send_wr ) * wr->send_wr_num,GFP_KERNEL );if (!wr->send_wr) {
@@ -2464,11 +2472,11 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } wr->isert_cmd = isert_cmd; - rdma_write_max = isert_conn->max_sge *PAGE_SIZE ; + rdma_max_len = max_sge *PAGE_SIZE ;for (i =0 ; i < wr->send_wr_num; i++) { send_wr = &isert_cmd->rdma_wr.send_wr[i]; - data_len =min (data_left, rdma_write_max); + data_len =min (data_left, rdma_max_len); send_wr->send_flags =0 ;if (wr->iser_ib_op ==ISER_IB_RDMA_WRITE ) {
@@ -2490,7 +2498,7 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } ib_sge_cnt =isert_build_rdma_wr (isert_conn, isert_cmd, ib_sge, - send_wr, data_len, offset); + send_wr, data_len, offset, max_sge); ib_sge += ib_sge_cnt; offset += data_len;
@@ -2619,6 +2627,13 @@ isert_fast_reg_mr(struct isert_conn *isert_conn, fr_wr.wr.fast_reg.rkey = mr->rkey; fr_wr.wr.fast_reg.access_flags =IB_ACCESS_LOCAL_WRITE ; +/* +* IWARP transports need REMOTE_WRITE for MRs used as the target of +* an RDMA_READ. +*/ +if (rdma_protocol_iwarp (ib_dev, isert_conn->cm_id->port_num)) + fr_wr.wr.fast_reg.access_flags |=IB_ACCESS_REMOTE_WRITE ; +if (!wr) wr = &fr_wr;else
drivers/infiniband/ulp/isert/ib_isert.h +2 -1
@@ -152,7 +152,8 @@ struct isert_conn {u32 responder_resources;u32 initiator_depth;bool pi_support; -u32 max_sge; +u32 max_write_sge; +u32 max_read_sge;char *login_buf;char *login_req_buf;char *login_rsp_buf;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v4 4/5 RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices
In reply to Steve WiseiWARP devices require REMOTE_WRITE for MRs used as the destination of an RDMA READ. IB does not require this. Since isert never advertises the rkey of its DMA MRs to the peer, just add REMOTE_WRITE to the access flags when allocating DMA MRs for all devices.
--- drivers/infiniband/ulp/isert/ib_isert.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +2 -1
@@ -383,7 +383,8 @@ isert_create_device_ib_res(struct isert_device *device) goto out_cq; } - device->mr =ib_get_dma_mr (device->pd,IB_ACCESS_LOCAL_WRITE ); + device->mr =ib_get_dma_mr (device->pd,IB_ACCESS_LOCAL_WRITE | +IB_ACCESS_REMOTE_WRITE );if (IS_ERR (device->mr)) { ret =PTR_ERR (device->mr); isert_err("failed to create dma mr, device %p, ret=%d\n",