iSER support for iWARP
Subsystems (5)
v5 cover iSER support for iWARP
The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.
I've tested this on cxgb4 and mlx4 hardware.
Changes since V4:
iser: fixedcompiler warning
isert: back to setting REMOTE_WRITE only for iWARP devices
Changes since V3:
Fixed commit messages based on feedback.
iser: adjust max_sectors
isert: split into 2 patches
isert: always set REMOTE_WRITE for dma mrs
Changes since V2:
The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.
Changes since V1:
Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.
Correctly set the device max_sge_rd capability in several rdma device drivers.
isert: use device capability max_sge_rd for the read sge depth.
isert: change max_sge to max_write_sge in struct isert_conn.
---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Expose max_sge_rd correctly
Steve Wise (4):
RDMA/isert: Limit read depth based on the device max_sge_rd capability RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth ipath,qib: Expose max_sge_rd correctly
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 drivers/infiniband/hw/mlx4/main.c | 1 drivers/infiniband/hw/mlx5/main.c | 1 drivers/infiniband/hw/mthca/mthca_provider.c | 1 drivers/infiniband/hw/qib/qib_verbs.c | 1 drivers/infiniband/ulp/iser/iscsi_iser.c | 9 ++++ drivers/infiniband/ulp/isert/ib_isert.c | 53 ++++++++++++++++++++++---- drivers/infiniband/ulp/isert/ib_isert.h | 3 + 8 files changed, 60 insertions(+), 10 deletions(-)
Signature · 1 line
Steve.
PATCH v5 1/5 mlx4, mlx5, mthca: Expose max_sge_rd correctly
From: Sagi Grimberg <[email protected]>
Applications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
---
drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + 3 files changed, 3 insertions(+), 0 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -0
@@ -229,6 +229,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, props->max_qp_wr = dev->dev->caps.max_wqes -MLX4_IB_SQ_MAX_SPARE ; props->max_sge =min (dev->dev->caps.max_sq_sg, dev->dev->caps.max_rq_sg); + props->max_sge_rd = props->max_sge; props->max_cq = dev->dev->quotas.cq; props->max_cqe = dev->dev->caps.max_cqes; props->max_mr = dev->dev->quotas.mpt;
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -137,6 +137,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, max_sq_sg = (gen->max_sq_desc_sz -sizeof (struct mlx5_wqe_ctrl_seg )) /sizeof (struct mlx5_wqe_data_seg ); props->max_sge =min (max_rq_sg, max_sq_sg); + props->max_sge_rd = props->max_sge; props->max_cq =1 << gen->log_max_cq; props->max_cqe = gen->max_cqes -1 ; props->max_mr =1 << gen->log_max_mkey;
drivers/infiniband/hw/mthca/mthca_provider.c +1 -0
@@ -97,6 +97,7 @@ static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = mdev->limits.num_qps - mdev->limits.reserved_qps; props->max_qp_wr = mdev->limits.max_wqes; props->max_sge = mdev->limits.max_sg; + props->max_sge_rd = props->max_sge; props->max_cq = mdev->limits.num_cqs - mdev->limits.reserved_cqs; props->max_cqe = mdev->limits.max_cqes; props->max_mr = mdev->limits.num_mpts - mdev->limits.reserved_mrws;
PATCH v5 2/5 ipath,qib: Expose max_sge_rd correctly
In reply to Steve WiseApplications must not assume that max_sge and max_sge_rd are the same, Hence expose max_sge_rd correctly as well.
---
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 + drivers/infiniband/hw/qib/qib_verbs.c | 1 + 2 files changed, 2 insertions(+), 0 deletions(-)
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
@@ -1521,6 +1521,7 @@ static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *pr props->max_qp = ib_ipath_max_qps; props->max_qp_wr = ib_ipath_max_qp_wrs; props->max_sge = ib_ipath_max_sges; + props->max_sge_rd = ib_ipath_max_sges; props->max_cq = ib_ipath_max_cqs; props->max_ah = ib_ipath_max_ahs; props->max_cqe = ib_ipath_max_cqes;
drivers/infiniband/hw/qib/qib_verbs.c +1 -0
@@ -1574,6 +1574,7 @@ static int qib_query_device(struct ib_device *ibdev, struct ib_device_attr *prop props->max_qp = ib_qib_max_qps; props->max_qp_wr = ib_qib_max_qp_wrs; props->max_sge = ib_qib_max_sges; + props->max_sge_rd = ib_qib_max_sges; props->max_cq = ib_qib_max_cqs; props->max_ah = ib_qib_max_ahs; props->max_cqe = ib_qib_max_cqes;
PATCH v5 3/5 RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
In reply to Steve WiseCurrently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the max_sectors based on the resulting sg tablesize.
---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/infiniband/ulp/iser/iscsi_iser.c +9 -0
@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); } +/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +if (iscsi_host_add (shost, ib_conn->device->ib_device->dma_device)) {mutex_unlock (&iser_conn->state_mutex);
A message that is not in the archive (sent off-list or not yet received): 20150705174452.10042.44695.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org
Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 18 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +
The min statement is meaningless for max_sectors - you do a min between default sg_tablesize and frpl length - so the maximum sg_tablesize is 128 which is 1024 max_sectors.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 30 lines
-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 19 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }
Quoted 13 lines
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.
I'm not following. What if ib_conn->device->dev_attr.max_fast_reg_page_list_len is say, 32? Then shost->sg_tablesize is set to 32, and max_sectors is set to (32*4K) >> 9 == 256 512B sectors.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 44 lines · Steve Wise
Quoted 30 lines
-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 19 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }Quoted 13 lines
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.
Quoted 2 lines
I'm not following. What if
ib_conn->device->dev_attr.max_fast_reg_page_list_lenis say, 32? Thenshost->sg_tablesizeis set to 32, andmax_sectorsis set to (32*4K) >> 9 == 256 512B sectors.
Correct - but it cannot exceed 1024 (as it is derived from sg_tablesize which is maximum 128).
RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 55 lines
-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Tuesday, July 07, 2015 1:27 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 44 lines · Steve Wise
Quoted 43 lines
-----Original Message-----
From: Sagi Grimberg [mailto:[email protected]]
Sent: Monday, July 06, 2015 2:51 AM
To: Steve Wise; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 18 lines · Steve Wise
Currently the sg tablesize, which dictates fast register page list depth to use, does not take into account the limits of the rdma device. So adjust it once we discover the device fastreg max depth limit. Also adjust the
max_sectorsbased on the resulting sg tablesize.Signed-off-by Steve Wise <[email protected]>---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 10 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.
Quoted 5 lines
Quoted 2 lines
I'm not following. What if
ib_conn->device->dev_attr.max_fast_reg_page_list_lenis say, 32? Thenshost->sg_tablesizeis set to 32, andmax_sectorsis set to (32*4K) >> 9 == 256 512B sectors.Correct - but it cannot exceed 1024 (as it is derived from
sg_tablesizewhich is maximum 128).
Actually it is initialized to 1024 in iscsi_iser_sht / iscsi_iser.c, so it isn't derived from sg_tables (although it probably should be). I can remove the min_t() though.
Hey Or, thoughts?
Steve.
Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
I'm not following. What if
ib_conn->device->dev_attr.max_fast_reg_page_list_lenis say, 32? Thenshost->sg_tablesizeis set to 32, andmax_sectorsis set to (32*4K) >> 9 == 256 512B sectors.Correct - but it cannot exceed 1024 (as it is derived from
sg_tablesizewhich is maximum 128).Actually it is initialized to 1024 in
iscsi_iser_sht/iscsi_iser.c, so it isn't derived fromsg_tables(although it probably should be). I can remove themin_t()though.Hey Or, thoughts?
Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.
Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the max_sectors to be <= 1024
If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 47 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Or Gerlitz
Sent: Tuesday, July 07, 2015 9:32 AM
To: Steve Wise; 'Sagi Grimberg'
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; target- [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
I'm not following. What if
ib_conn->device->dev_attr.max_fast_reg_page_list_lenis say, 32? Thenshost->sg_tablesizeis set to 32, andmax_sectorsis set to (32*4K) >> 9 == 256 512B sectors.Correct - but it cannot exceed 1024 (as it is derived from
sg_tablesizewhich is maximum 128).Actually it is initialized to 1024 in
iscsi_iser_sht/iscsi_iser.c, so it isn't derived fromsg_tables(although it probably should be). I canremove the
min_t()though.Quoted 2 lines
Hey Or, thoughts?
Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.
Quoted 8 lines
Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the
max_sectorsto be <= 1024If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.
I'm not sure... you guys are the iSER experts. :) But considering a 64K PAGE_SIZE and an adjusted sg_tablesize of say, 32, w/o the min() we get (32 * 65536) >> 9 == 4096. So if the requirement is that max_sectors always be <= 1024, then we need the min()...
Steve.
Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth
Quoted 52 lines · Steve Wise
Quoted 52 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Or Gerlitz
Sent: Tuesday, July 07, 2015 9:32 AM
To: Steve Wise; 'Sagi Grimberg'
Cc: [email protected]; [email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; target- [email protected]
Subject: Re: [PATCH V5 3/5] RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depthQuoted 22 lines · Steve Wise
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +0
-0@@ -640,6 +640,15 @@ iscsi_iser_session_create(struct iscsi_endpoint *ep, SHOST_DIX_GUARD_CRC); }Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c iscsi_iser_session_create(struct iscsi_endpoint *ep, · line 651
Quoted 9 lines
drivers/infiniband/ulp/iser/iscsi_iser.c +9
-0+/* +* Limit the sg_tablesize and max_sectors based on the device +* max fastreg page list length. +*/ + shost->sg_tablesize = min_t(unsigned short, shost->sg_tablesize, + ib_conn->device->dev_attr.max_fast_reg_page_list_len); + shost->max_sectors = min_t(unsigned int, +1024 , (shost->sg_tablesize *PAGE_SIZE ) >>9 ); +The min statement is meaningless for
max_sectors- you do a min between defaultsg_tablesizeand frpl length - so the maximumsg_tablesizeis 128 which is 1024max_sectors.Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
I'm not following. What if
ib_conn->device->dev_attr.max_fast_reg_page_list_lenis say, 32? Thenshost->sg_tablesizeis set to 32, andmax_sectorsis set to (32*4K) >> 9 == 256 512B sectors.Correct - but it cannot exceed 1024 (as it is derived from
sg_tablesizewhich is maximum 128).Actually it is initialized to 1024 in
iscsi_iser_sht/iscsi_iser.c, so it isn't derived fromsg_tables(although it probably should be). I canremove the
min_t()though.Quoted 2 lines
Hey Or, thoughts?
Originally, we've put the double restriction of 128 SG entries AND 1024 sectors to make sure that whatever SG is up there, it'snot spanning > 512KB.
Think on SG whose one/some of their element/s is > one page or on systems with > 4KB page size or others examples... since your patch touched the number of SG entries I was thinking you need to make sure no regression was introduced re the
max_sectorsto be <= 1024
Quoted 6 lines
Quoted 3 lines
If U2 are @ consensus that this is the case with the original patch w.o further changes, let it be.
I'm not sure... you guys are the iSER experts. :) But considering a 64K
PAGE_SIZEand an adjustedsg_tablesizeof say, 32, w/o themin()we get (32 * 65536) >> 9 == 4096. So if the requirement is thatmax_sectorsalways be <= 1024, then we need themin()...
I guess the min is fine given that I have a patchset for 8MB support, so we'll get it right then.
Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v5 4/5 RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices
In reply to Steve WiseiWARP devices require REMOTE_WRITE for MRs used as the destination of an RDMA READ. So if the device protocol is iWARP, then set REMOTE_WRITE when allocating the DMA MR.
---
drivers/infiniband/ulp/isert/ib_isert.c | 22 +++++++++++++++++++++- 1 files changed, 21 insertions(+), 1 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +21 -1
@@ -348,6 +348,17 @@ out_cq:return ret; } +static int any_port_is_iwarp (struct isert_device *device) +{ +int i; + +for (i =rdma_start_port (device->ib_device); + i <=rdma_end_port (device->ib_device); i++) +if (rdma_protocol_iwarp (device->ib_device, i)) +return 1 ; +return 0 ; +} +static int isert_create_device_ib_res (struct isert_device *device) {
@@ -383,7 +394,16 @@ isert_create_device_ib_res(struct isert_device *device) goto out_cq; } - device->mr =ib_get_dma_mr (device->pd,IB_ACCESS_LOCAL_WRITE ); +/* +* IWARP transports need REMOTE_WRITE for MRs used as the target of +* an RDMA_READ. Since the DMA MR is used for all ports, then if +* any port is running IWARP, add REMOTE_WRITE. +*/ +if (any_port_is_iwarp (device)) + device->mr =ib_get_dma_mr (device->pd,IB_ACCESS_LOCAL_WRITE | +IB_ACCESS_REMOTE_WRITE ); +else + device->mr =ib_get_dma_mr (device->pd,IB_ACCESS_LOCAL_WRITE );if (IS_ERR (device->mr)) { ret =PTR_ERR (device->mr); isert_err("failed to create dma mr, device %p, ret=%d\n",
A message that is not in the archive (sent off-list or not yet received): 20150705174459.10042.37298.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org
Sagi Grimberg
Reviewed-byRe: [PATCH V5 4/5] RDMA/isert: Set REMOTE_WRITE on DMA MRs to support iWARP devices
Quoted 5 lines · Steve Wise
iWARP devices require
REMOTE_WRITEfor MRs used as the destination of an RDMA READ. So if the device protocol is iWARP, then setREMOTE_WRITEwhen allocating the DMA MR.Signed-off-by Steve Wise <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v5 5/5 RDMA/isert: Limit read depth based on the device max_sge_rd capability
In reply to Steve WiseUse the device's max_sge_rd capability to compute the target's read sge depth. Save both the read and write max_sge values in the isert_conn struct, and use these when creating RDMA_WRITE/READ work requests.
---
drivers/infiniband/ulp/isert/ib_isert.c | 31 +++++++++++++++++++++++-------- drivers/infiniband/ulp/isert/ib_isert.h | 3 ++- 2 files changed, 25 insertions(+), 9 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +23 -8
@@ -163,7 +163,9 @@ isert_create_qp(struct isert_conn *isert_conn,* outgoing control PDU responses. */ attr.cap.max_send_sge =max (2 , device->dev_attr.max_sge -2 ); - isert_conn->max_sge = attr.cap.max_send_sge; + isert_conn->max_write_sge = attr.cap.max_send_sge; + isert_conn->max_read_sge =min_t (u32, device->dev_attr.max_sge_rd, + attr.cap.max_send_sge); attr.cap.max_recv_sge =1 ; attr.sq_sig_type =IB_SIGNAL_REQ_WR ;
@@ -2395,7 +2397,7 @@ isert_put_text_rsp(struct iscsi_cmd *cmd, struct iscsi_conn *conn)static int isert_build_rdma_wr (struct isert_conn *isert_conn,struct isert_cmd *isert_cmd,struct ib_sge *ib_sge,struct ib_send_wr *send_wr, -u32 data_left,u32 offset) +u32 data_left,u32 offset,u32 max_sge) {struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;struct scatterlist *sg_start, *tmp_sg;
@@ -2406,7 +2408,7 @@ isert_build_rdma_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd, sg_off = offset /PAGE_SIZE ; sg_start = &cmd->se_cmd.t_data_sg[sg_off]; - sg_nents =min (cmd->se_cmd.t_data_nents - sg_off, isert_conn->max_sge); + sg_nents =min (cmd->se_cmd.t_data_nents - sg_off, max_sge); page_off = offset %PAGE_SIZE ; send_wr->sg_list = ib_sge;
@@ -2450,8 +2452,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,struct isert_data_buf *data = &wr->data;struct ib_send_wr *send_wr;struct ib_sge *ib_sge; -u32 offset, data_len, data_left, rdma_write_max, va_offset =0 ; +u32 offset, data_len, data_left, rdma_max_len, va_offset =0 ;int ret =0 , i, ib_sge_cnt; +u32 max_sge; isert_cmd->tx_desc.isert_cmd = isert_cmd;
@@ -2473,7 +2476,12 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } wr->ib_sge = ib_sge; - wr->send_wr_num =DIV_ROUND_UP (data->nents, isert_conn->max_sge); +if (wr->iser_ib_op ==ISER_IB_RDMA_WRITE ) + max_sge = isert_conn->max_write_sge; +else + max_sge = isert_conn->max_read_sge; + + wr->send_wr_num =DIV_ROUND_UP (data->nents, max_sge); wr->send_wr =kzalloc (sizeof (struct ib_send_wr ) * wr->send_wr_num,GFP_KERNEL );if (!wr->send_wr) {
@@ -2483,11 +2491,11 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } wr->isert_cmd = isert_cmd; - rdma_write_max = isert_conn->max_sge *PAGE_SIZE ; + rdma_max_len = max_sge *PAGE_SIZE ;for (i =0 ; i < wr->send_wr_num; i++) { send_wr = &isert_cmd->rdma_wr.send_wr[i]; - data_len =min (data_left, rdma_write_max); + data_len =min (data_left, rdma_max_len); send_wr->send_flags =0 ;if (wr->iser_ib_op ==ISER_IB_RDMA_WRITE ) {
@@ -2509,7 +2517,7 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd, } ib_sge_cnt =isert_build_rdma_wr (isert_conn, isert_cmd, ib_sge, - send_wr, data_len, offset); + send_wr, data_len, offset, max_sge); ib_sge += ib_sge_cnt; offset += data_len;
@@ -2638,6 +2646,13 @@ isert_fast_reg_mr(struct isert_conn *isert_conn, fr_wr.wr.fast_reg.rkey = mr->rkey; fr_wr.wr.fast_reg.access_flags =IB_ACCESS_LOCAL_WRITE ; +/* +* IWARP transports need REMOTE_WRITE for MRs used as the target of +* an RDMA_READ. +*/ +if (rdma_protocol_iwarp (ib_dev, isert_conn->cm_id->port_num)) + fr_wr.wr.fast_reg.access_flags |=IB_ACCESS_REMOTE_WRITE ; +if (!wr) wr = &fr_wr;else
drivers/infiniband/ulp/isert/ib_isert.h +2 -1
@@ -152,7 +152,8 @@ struct isert_conn {u32 responder_resources;u32 initiator_depth;bool pi_support; -u32 max_sge; +u32 max_write_sge; +u32 max_read_sge;char *login_buf;char *login_req_buf;char *login_rsp_buf;
A message that is not in the archive (sent off-list or not yet received): 20150705174505.10042.28442.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org
Sagi Grimberg
Reviewed-byRe: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 5 lines · Steve Wise
Use the device's
max_sge_rdcapability to compute the target's read sge depth. Save both the read and writemax_sgevalues in theisert_connstruct, and use these when creating RDMA_WRITE/READ work requests.Signed-off-by Steve Wise <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
In reply to a message not in the archiveQuoted 3 lines · Steve Wise
Use the device's
max_sge_rdcapability to compute the target's read sge depth. Save both the read and writemax_sgevalues in theisert_connstruct, and use these when creating RDMA_WRITE/READ work requests.
Btw, any hance to make the NFS client use these values as well instead of the current rdma_read_max_sge() hack?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 10 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Christoph Hellwig
Sent: Tuesday, July 14, 2015 3:27 AM
To: Steve Wise
Cc: [email protected]; [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; linux- [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapability
Quoted 7 lines
Quoted 3 lines · Steve Wise
Use the device's
max_sge_rdcapability to compute the target's read sge depth. Save both the read and writemax_sgevalues in theisert_connstruct, and use these when creating RDMA_WRITE/READ work requests.Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?
Chuck, can you add this to your cleanup list?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 4 lines · Steve Wise
Quoted 2 lines
Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?Chuck, can you add this to your cleanup list?
It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 11 lines
-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; sagig-VPRAkNaXOzVWk0Htik3J/[email protected]; ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]; roid-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]; eli-VPRAkNaXOzVWk0Htik3J/[email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityOn Tue, Jul 14, 2015 at 09:41:00AM -0500, Steve Wise wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?Chuck, can you add this to your cleanup list?
It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
Ok, I will do this.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 14 lines · Steve Wise
Quoted 14 lines
-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityQuoted 3 lines · Steve Wise
Quoted 2 lines
Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Chuck, can you add this to your cleanup list?
It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
Ok, I will do this.
Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:
http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190
Which changes the way xprtrdma uses max_sge.
--
Chuck Lever
RE: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 7 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chuck Lever
Sent: Tuesday, July 14, 2015 1:47 PM
To: Steve Wise
Cc: Christoph Hellwig; Doug Ledford; [email protected]; Sagi Grimberg; Or Gerlitz; [email protected]; linux-rdma; Eli Cohen;
target-
Quoted 26 lines
[email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityQuoted 20 lines · Steve Wise
Quoted 19 lines
-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityQuoted 4 lines · Steve Wise
Quoted 2 lines
Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?Chuck, can you add this to your cleanup list?
It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
Quoted 9 lines
Quoted 1 line
Ok, I will do this.
Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:
http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190
Which changes the way xprtrdma uses
max_sge.
Sure, but xprtrdma doesn't issue reads, so max_sge_rd isn't needed. The change Christoph wants is actually in svcrdma. I will change the server to use max_sge_rd instead of rdma_cap_read_multi_sge().
Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the device max_sge_rd capability
Quoted 39 lines · Steve Wise
Quoted 7 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chuck Lever
Sent: Tuesday, July 14, 2015 1:47 PM
To: Steve Wise
Cc: Christoph Hellwig; Doug Ledford; [email protected]; Sagi Grimberg; Or Gerlitz; [email protected]; linux-rdma; Eli Cohen;target-
Quoted 31 lines
[email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityQuoted 14 lines · Steve Wise
Quoted 14 lines
-----Original Message-----
From: 'Christoph Hellwig' [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 10:42 AM
To: Steve Wise
Cc: 'Christoph Hellwig'; Chuck Lever; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V5 5/5] RDMA/isert: Limit read depth based on the devicemax_sge_rdcapabilityQuoted 3 lines · Steve Wise
Quoted 2 lines
Btw, any hance to make the NFS client use these values as well instead of the current
rdma_read_max_sge()hack?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Chuck, can you add this to your cleanup list?
It would be useful to add this to your series so we can get rid of it for the next merge window instead of introducing a depenency that would defer it to the next merge window.
Ok, I will do this.
Steve, can you review v2 of the nfs-rdma-for-4.3 series I posted yesterday? Specifically:
Quoted 7 lines
Quoted 4 lines
http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;h=6abafb636e03fbb7f93a26796223833581a70190
Which changes the way xprtrdma uses
max_sge.Sure, but xprtrdma doesn't issue reads, so
max_sge_rdisn't needed. The change Christoph wants is actually in svcrdma. I will change the server to usemax_sge_rdinstead ofrdma_cap_read_multi_sge().
OK, glad the conflict has been avoided.
--
Chuck Lever
A message that is not in the archive (sent off-list or not yet received): 20150705174353.10042.39648.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org
Nicholas A. Bellinger
Acked-byHi Steve & Co,
Quoted 51 lines · Steve Wise
The following series implements support for iWARP transports in the iSER initiator and target. This is based on Doug's k.o/for-4.2 branch.
I've tested this on cxgb4 and mlx4 hardware.
Changes since V4:
iser: fixedcompiler warning
isert: back to setting
REMOTE_WRITEonly for iWARP devicesChanges since V3:
Fixed commit messages based on feedback.
iser: adjust
max_sectorsisert: split into 2 patches
isert: always set
REMOTE_WRITEfor dma mrsChanges since V2:
The transport independent work is removed from this series and will be submitted in a subsequent series. This V3 series now enables iWARP using existing core services.
Changes since V1:
Introduce and use transport-independent RDMA core services for allocating DMA MRs and computing fast register access flags.
Correctly set the device
max_sge_rdcapability in several rdma device drivers.isert: use device capability
max_sge_rdfor the read sge depth.isert: change
max_sgetomax_write_sgein structisert_conn.---
Sagi Grimberg (1):
mlx4, mlx5, mthca: Exposemax_sge_rdcorrectlySteve Wise (4):
RDMA/isert: Limit read depth based on the devicemax_sge_rdcapability RDMA/isert: SetREMOTE_WRITEon DMA MRs to support iWARP devices RDMA/iser: Limit sg tablesize andmax_sectorsto device fastreg max depth ipath,qib: Exposemax_sge_rdcorrectly
Quoted 10 lines
drivers/infiniband/hw/ipath/ipath_verbs.c | 1 drivers/infiniband/hw/mlx4/main.c | 1 drivers/infiniband/hw/mlx5/main.c | 1 drivers/infiniband/hw/mthca/mthca_provider.c | 1 drivers/infiniband/hw/qib/qib_verbs.c | 1 drivers/infiniband/ulp/iser/iscsi_iser.c | 9 ++++ drivers/infiniband/ulp/isert/ib_isert.c | 53 ++++++++++++++++++++++---- drivers/infiniband/ulp/isert/ib_isert.h | 3 + 8 files changed, 60 insertions(+), 10 deletions(-)
Very excited to see iWARP support for iser-initiator + iser-target. ;)
No objections to taking the iser-target changes through Doug's infiniband.git. For the target pieces:
Also Doug, since this series is sufficiently small enough, and allows a class of hardware that people are expecting to function with iser to 'just work', I'd recommend to go ahead and push this series for v4.2-rc code.
Thank you,
--nab
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html