Skip to contentSkip to conversation

IB SRP initiator patches for kernel 3.11

PATCH Superseded Bart Van Assche 6 participants 47 messages Jun 28, 2013 – Jul 3, 2013 25 messages not in the archive

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover IB SRP initiator patches for kernel 3.11

The purpose of this InfiniBand SRP initiator patch series is as follows:
- Make the SRP initiator driver better suited for use in a H.A. setup. Speed up failover by reducing the IB RC retry count and by notifying multipathd faster about transport layer failures by adding fast_io_fail_tmo and dev_loss_tmo parameters.
- Make the SRP initiator better suited for use on NUMA systems by making the HCA completion vector configurable.

Changes since the v1 of the "IB SRP initiator patches for kernel 3.11" patch series:
- scsi_transport_srp: Allowed both fast_io_fail and dev_loss timeouts to be disabled.
- scsi_transport_srp, srp_reconnect_rport(): switched from scsi_block_requests() to scsi_target_block() for blocking SCSI command processing temporarily.
- scsi_transport_srp, srp_start_tl_fail_timers(): only block SCSI device command processing if the fast_io_fail timer is enabled.
- Changed srp_abort() such that upon transport offline the value FAST_IO_FAIL is returned instead of SUCCESS.
- Fixed a race condition in the "maintain single connection" patch: a new login after removal had started but before removal had finished still could create a duplicate connection. Fixed this by deferring removal from the target list until removal has finished.
- Modified the error message in the same patch for reporting that a duplicate connection has been rejected.
- Modified patch 2/15 such that all possible race conditions with srp_claim_req() are addressed.
- Documented the comp_vector and tl_retry_count login string parameters.
- Updated dev_loss_tmo and fast_io_fail_tmo documentation - mentioned "off" is a valid choice.

Changes compared to v5 of the "Make ib_srp better suited for H.A. purposes" patch series:
- Left out patches that are already upstream.
- Made it possible to set dev_loss_tmo to "off". This is useful in a
setup using initiator side mirroring to avoid that new /dev/sd* names
are reassigned after a failover or cable pull and reinsert.
- Added kernel module parameters to ib_srp for configuring default
values of the fast_io_fail_tmo and dev_loss_tmo parameters.
- Added a patch from Dotan Barak that fixes a kernel oops during rmmod triggered by resource allocation failure at module load time.
- Avoid duplicate connections by refusing relogins instead of dropping duplicate connections, as proposed by Sebastian Riemer.
- Added a patch from Sebastian Riemer for failing SCSI commands
silently.
- Added a patch from Vu Pham to make the transport layer (IB RC) retry count configurable.
- Made HCA completion vector configurable.

Changes since v4:
- Added a patch for removing SCSI devices upon a port down event

Changes since v3:
- Restored the dev_loss_tmo and fast_io_fail_tmo sysfs attributes.
- Included a patch to fix an ib_srp crash that could be triggered by cable pulling.

Changes since v2:
- Addressed the v2 review comments.
- Dropped the patches that have already been merged.
- Dropped the patches for integration with multipathd.
- Dropped the micro-optimization of the IB completion handlers.

The individual patches in this series are as follows:
0001-IB-srp-Fix-remove_one-crash-due-to-resource-exhausti.patch 0002-IB-srp-Fix-race-between-srp_queuecommand-and-srp_cla.patch 0003-IB-srp-Avoid-that-srp_reset_host-is-skipped-after-a-.patch 0004-IB-srp-Fail-I-O-fast-if-target-offline.patch 0005-IB-srp-Skip-host-settle-delay.patch 0006-IB-srp-Maintain-a-single-connection-per-I_T-nexus.patch 0007-IB-srp-Keep-rport-as-long-as-the-IB-transport-layer.patch 0008-scsi_transport_srp-Add-transport-layer-error-handlin.patch 0009-IB-srp-Add-srp_terminate_io.patch 0010-IB-srp-Use-SRP-transport-layer-error-recovery.patch 0011-IB-srp-Start-timers-if-a-transport-layer-error-occur.patch 0012-IB-srp-Fail-SCSI-commands-silently.patch 0013-IB-srp-Make-HCA-completion-vector-configurable.patch 0014-IB-srp-Make-transport-layer-retry-count-configurable.patch 0015-IB-srp-Bump-driver-version-and-release-date.patch
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

PATCH v2 1/15 IB/srp: Fix remove_one crash due to resource exhaustion

In reply to a message not in the archive

From: Dotan Barak <[email protected]>

If the add_one callback fails during driver load no resources are allocated so there isn't a need to release any resources. Trying to clean the resource may lead to the following kernel panic:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffffa0132331>] srp_remove_one+0x31/0x240 [ib_srp]
RIP: 0010:[<ffffffffa0132331>]  [<ffffffffa0132331>] srp_remove_one+0x31/0x240 [ib_srp]
Process rmmod (pid: 4562, threadinfo ffff8800dd738000, task ffff8801167e60c0)
Call Trace:
 [<ffffffffa024500e>] ib_unregister_client+0x4e/0x120 [ib_core]
 [<ffffffffa01361bd>] srp_cleanup_module+0x15/0x71 [ib_srp]
 [<ffffffff810ac6a4>] sys_delete_module+0x194/0x260
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
[bvanassche: Shortened patch description]
Cc Roland Dreier <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
index 7ccf328..368d160 100644
@@ -2507,6 +2507,8 @@ static void srp_remove_one(struct ib_device *device)
 	struct srp_target_port *target;
 
 	srp_dev = ib_get_client_data(device, &srp_client);
+	if (!srp_dev)
+		return;
 
 	list_for_each_entry_safe(host, tmp_host, &srp_dev->dev_list, list) {
 		device_unregister(&host->dev);
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw

PATCH v2 2/15 IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive

Avoid that srp_claim_command() can claim a command while srp_queuecommand() is still busy queueing the same command. Found this via source reading.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +8 -6
index 368d160..035167b 100644
@@ -1047,13 +1047,11 @@ map_complete:
 static void srp_put_tx_iu(struct srp_target_port *target, struct srp_iu *iu,
 			  enum srp_iu_type iu_type)
 {
-	unsigned long flags;
+	lockdep_assert_held(&target->lock);
 
-	spin_lock_irqsave(&target->lock, flags);
 	list_add(&iu->list, &target->free_tx);
 	if (iu_type != SRP_IU_RSP)
 		++target->req_lim;
-	spin_unlock_irqrestore(&target->lock, flags);
 }
 
 /*
@@ -1210,7 +1208,10 @@ static int srp_response_common(struct srp_target_port *target, s32 req_delta,
 	if (err) {
 		shost_printk(KERN_ERR, target->scsi_host, PFX
 			     "unable to post response: %d\n", err);
+
+		spin_lock_irqsave(&target->lock, flags);
 		srp_put_tx_iu(target, iu, SRP_IU_RSP);
+		spin_unlock_irqrestore(&target->lock, flags);
 	}
 
 	return err;
@@ -1367,7 +1368,6 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd)
 
 	req = list_first_entry(&target->free_reqs, struct srp_request, list);
 	list_del(&req->list);
-	spin_unlock_irqrestore(&target->lock, flags);
 
 	dev = target->srp_host->srp_dev->dev;
 	ib_dma_sync_single_for_cpu(dev, iu->dma, target->max_iu_len,
@@ -1401,6 +1401,7 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd)
 		shost_printk(KERN_ERR, target->scsi_host, PFX "Send failed\n");
 		goto err_unmap;
 	}
+	spin_unlock_irqrestore(&target->lock, flags);
 
 	return 0;
 
@@ -1409,8 +1410,6 @@ err_unmap:
 
 err_iu:
 	srp_put_tx_iu(target, iu, SRP_IU_CMD);
-
-	spin_lock_irqsave(&target->lock, flags);
 	list_add(&req->list, &target->free_reqs);
 
 err_unlock:
@@ -1729,7 +1728,10 @@ static int srp_send_tsk_mgmt(struct srp_target_port *target,
 	ib_dma_sync_single_for_device(dev, iu->dma, sizeof *tsk_mgmt,
 				      DMA_TO_DEVICE);
 	if (srp_post_send(target, iu, sizeof *tsk_mgmt)) {
+		spin_lock_irq(&target->lock);
 		srp_put_tx_iu(target, iu, SRP_IU_TSK_MGMT);
+		spin_unlock_irq(&target->lock);
+
 		return -1;
 	}
 
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#13 lore raw

PATCH v2 3/15 IB/srp: Avoid that srp_reset_host() is skipped after a TL error

In reply to a message not in the archive

The SCSI error handler assumes that the transport layer is operational if an eh_abort_handler() returns SUCCESS. Hence let srp_abort() only return SUCCESS if sending the ABORT TASK task management function succeeded. This patch avoids that the SCSI error handler skips the srp_reset_host() call after a transport layer error.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +7 -3
index 035167b..8c95262 100644
@@ -1746,18 +1746,22 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 {
 	struct srp_target_port *target = host_to_target(scmnd->device->host);
 	struct srp_request *req = (struct srp_request *) scmnd->host_scribble;
+	int ret;
 
 	shost_printk(KERN_ERR, target->scsi_host, "SRP abort called\n");
 
 	if (!req || !srp_claim_req(target, req, scmnd))
 		return FAILED;
-	srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
-			  SRP_TSK_ABORT_TASK);
+	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
+			      SRP_TSK_ABORT_TASK) == 0)
+		ret = SUCCESS;
+	else
+		ret = FAILED;
 	srp_free_req(target, req, scmnd, 0);
 	scmnd->result = DID_ABORT << 16;
 	scmnd->scsi_done(scmnd);
 
-	return SUCCESS;
+	return ret;
 }
 
 static int srp_reset_device(struct scsi_cmnd *scmnd)
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#16 lore raw

PATCH v2 4/15 IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Sebastian Riemer <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
index 8c95262..5c91521 100644
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
 			      SRP_TSK_ABORT_TASK) == 0)
 		ret = SUCCESS;
+	else if (target->transport_offline)
+		ret = FAST_IO_FAIL;
 	else
 		ret = FAILED;
 	srp_free_req(target, req, scmnd, 0);
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#32 lore raw

PATCH v2 5/15 IB/srp: Skip host settle delay

In reply to a message not in the archive

The SRP initiator implements host reset by reconnecting to the SRP target. That means that communication with the target is possible as soon as host reset finished. Hence skip the host settle delay.

Cc Roland Dreier <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    1 +
 1 file changed, 1 insertion(+)
drivers/infiniband/ulp/srp/ib_srp.c +1 -0
index 5c91521..01212c9 100644
@@ -1954,6 +1954,7 @@ static struct scsi_host_template srp_template = {
 	.eh_abort_handler		= srp_abort,
 	.eh_device_reset_handler	= srp_reset_device,
 	.eh_host_reset_handler		= srp_reset_host,
+	.skip_settle_delay		= true,
 	.sg_tablesize			= SRP_DEF_SG_TABLESIZE,
 	.can_queue			= SRP_CMD_SQ_SIZE,
 	.this_id			= -1,
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#33 lore raw

PATCH v2 6/15 IB/srp: Maintain a single connection per I_T nexus

In reply to a message not in the archive

An SRP target is required to maintain a single connection between initiator and target. This means that if the 'add_target' attribute is used to create a second connection to a target that the first connection will be logged out and that the SCSI error handler will kick in. The SCSI error handler will cause the SRP initiator to reconnect, which will cause I/O over the second connection to fail. Avoid such ping-pong behavior by disabling relogins. Note: if reconnecting manually is necessary, that is possible by deleting and recreating an rport via sysfs.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |   44 +++++++++++++++++++++++++++++++++--
 1 file changed, 42 insertions(+), 2 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +42 -2
index 01212c9..4eba1f7 100644
@@ -542,11 +542,11 @@ static void srp_remove_work(struct work_struct *work)
 
 	WARN_ON_ONCE(target->state != SRP_TARGET_REMOVED);
 
+	srp_remove_target(target);
+
 	spin_lock(&target->srp_host->target_lock);
 	list_del(&target->list);
 	spin_unlock(&target->srp_host->target_lock);
-
-	srp_remove_target(target);
 }
 
 static void srp_rport_delete(struct srp_rport *rport)
@@ -2010,6 +2010,36 @@ static struct class srp_class = {
 	.dev_release = srp_release_dev
 };
 
+/**
+ * srp_conn_unique() - check whether the connection to a target is unique
+ */
+static bool srp_conn_unique(struct srp_host *host,
+			    struct srp_target_port *target)
+{
+	struct srp_target_port *t;
+	bool ret = false;
+
+	if (target->state == SRP_TARGET_REMOVED)
+		goto out;
+
+	ret = true;
+
+	spin_lock(&host->target_lock);
+	list_for_each_entry(t, &host->target_list, list) {
+		if (t != target &&
+		    target->id_ext == t->id_ext &&
+		    target->ioc_guid == t->ioc_guid &&
+		    target->initiator_ext == t->initiator_ext) {
+			ret = false;
+			break;
+		}
+	}
+	spin_unlock(&host->target_lock);
+
+out:
+	return ret;
+}
+
 /*
  * Target ports are added by writing
  *
@@ -2266,6 +2296,16 @@ static ssize_t srp_create_target(struct device *dev,
 	if (ret)
 		goto err;
 
+	if (!srp_conn_unique(target->srp_host, target)) {
+		shost_printk(KERN_INFO, target->scsi_host,
+			     PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;initiator_ext=%016llx\n",
+			     be64_to_cpu(target->id_ext),
+			     be64_to_cpu(target->ioc_guid),
+			     be64_to_cpu(target->initiator_ext));
+		ret = -EEXIST;
+		goto err;
+	}
+
 	if (!host->srp_dev->fmr_pool && !target->allow_ext_sg &&
 				target->cmd_sg_cnt < target->sg_tablesize) {
 		pr_warn("No FMR pool and no external indirect descriptors, limiting sg_tablesize to cmd_sg_cnt\n");
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#36 lore raw

PATCH v2 7/15 IB/srp: Keep rport as long as the IB transport layer

In reply to a message not in the archive

Keep the rport data structure around after srp_remove_host() has finished until cleanup of the IB transport layer has finished completely. This is necessary because later patches use the rport pointer inside the queuecommand callback. Without this patch accessing the rport from inside a queuecommand callback is racy because srp_remove_host() must be invoked before scsi_remove_host() and because the queuecommand callback may get invoked after srp_remove_host() has finished.

Cc Roland Dreier <[email protected]>
Cc James Bottomley <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    3 +++
 drivers/infiniband/ulp/srp/ib_srp.h |    1 +
 drivers/scsi/scsi_transport_srp.c   |   18 ++++++++++++++++++
 include/scsi/scsi_transport_srp.h   |    2 ++
 4 files changed, 24 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +3 -0
index 4eba1f7..cb86be5 100644
@@ -526,11 +526,13 @@ static void srp_remove_target(struct srp_target_port *target)
 	WARN_ON_ONCE(target->state != SRP_TARGET_REMOVED);
 
 	srp_del_scsi_host_attr(target->scsi_host);
+	srp_rport_get(target->rport);
 	srp_remove_host(target->scsi_host);
 	scsi_remove_host(target->scsi_host);
 	srp_disconnect_target(target);
 	ib_destroy_cm_id(target->cm_id);
 	srp_free_target_ib(target);
+	srp_rport_put(target->rport);
 	srp_free_req_data(target);
 	scsi_host_put(target->scsi_host);
 }
@@ -1984,6 +1986,7 @@ static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
 	}
 
 	rport->lld_data = target;
+	target->rport = rport;
 
 	spin_lock(&host->target_lock);
 	list_add_tail(&target->list, &host->target_list);
drivers/infiniband/ulp/srp/ib_srp.h +1 -0
index 66fbedd..1817ed5 100644
@@ -153,6 +153,7 @@ struct srp_target_port {
 	u16			io_class;
 	struct srp_host	       *srp_host;
 	struct Scsi_Host       *scsi_host;
+	struct srp_rport       *rport;
 	char			target_name[32];
 	unsigned int		scsi_id;
 	unsigned int		sg_tablesize;
drivers/scsi/scsi_transport_srp.c +18 -0
index f379c7f..f7ba94a 100644
@@ -185,6 +185,24 @@ static int srp_host_match(struct attribute_container *cont, struct device *dev)
 }
 
 /**
+ * srp_rport_get() - increment rport reference count
+ */
+void srp_rport_get(struct srp_rport *rport)
+{
+	get_device(&rport->dev);
+}
+EXPORT_SYMBOL(srp_rport_get);
+
+/**
+ * srp_rport_put() - decrement rport reference count
+ */
+void srp_rport_put(struct srp_rport *rport)
+{
+	put_device(&rport->dev);
+}
+EXPORT_SYMBOL(srp_rport_put);
+
+/**
  * srp_rport_add - add a SRP remote port to the device hierarchy
  * @shost:	scsi host the remote port is connected to.
  * @ids:	The port id for the remote port.
include/scsi/scsi_transport_srp.h +2 -0
index ff0f04a..5a2d2d1 100644
@@ -38,6 +38,8 @@ extern struct scsi_transport_template *
 srp_attach_transport(struct srp_function_template *);
 extern void srp_release_transport(struct scsi_transport_template *);
 
+extern void srp_rport_get(struct srp_rport *rport);
+extern void srp_rport_put(struct srp_rport *rport);
 extern struct srp_rport *srp_rport_add(struct Scsi_Host *,
 				       struct srp_rport_identifiers *);
 extern void srp_rport_del(struct srp_rport *);
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
maintainer
#63 lore raw

PATCH v2 8/15 scsi_transport_srp: Add transport layer error handling

In reply to Bart Van Assche

Add the necessary functions in the SRP transport module to allow an SRP initiator driver to implement transport layer error handling similar to the functionality already provided by the FC transport layer. This includes:
- Support for implementing fast_io_fail_tmo, the time that should elapse after having detected a transport layer problem and before failing I/O.
- Support for implementing dev_loss_tmo, the time that should elapse after having detected a transport layer problem and before removing a remote port.

Cc Roland Dreier <[email protected]>
Cc James Bottomley <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 Documentation/ABI/stable/sysfs-transport-srp |   37 ++
 drivers/scsi/scsi_transport_srp.c            |  477 +++++++++++++++++++++++++-
 include/scsi/scsi_transport_srp.h            |   62 +++-
 3 files changed, 573 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-transport-srp +37 -0
index b36fb0d..6a4d651 100644
@@ -5,6 +5,24 @@ Contact:	[email protected], [email protected]
 Description:	Instructs an SRP initiator to disconnect from a target and to
 		remove all LUNs imported from that target.
 
+What:		/sys/class/srp_remote_ports/port-<h>:<n>/dev_loss_tmo
+Date:		October 1, 2013
+KernelVersion:	3.11
+Contact:	[email protected], [email protected]
+Description:	Number of seconds the SCSI layer will wait after a transport
+		layer error has been observed before removing a target port.
+		Zero means immediate removal. Setting this attribute to "off"
+		will disable this behavior.
+
+What:		/sys/class/srp_remote_ports/port-<h>:<n>/fast_io_fail_tmo
+Date:		October 1, 2013
+KernelVersion:	3.11
+Contact:	[email protected], [email protected]
+Description:	Number of seconds the SCSI layer will wait after a transport
+		layer error has been observed before failing I/O. Zero means
+		immediate removal. Setting this attribute to "off" will
+		disable this behavior.
+
 What:		/sys/class/srp_remote_ports/port-<h>:<n>/port_id
 Date:		June 27, 2007
 KernelVersion:	2.6.24
@@ -12,8 +30,27 @@ Contact:	[email protected]
 Description:	16-byte local SRP port identifier in hexadecimal format. An
 		example: 4c:49:4e:55:58:20:56:49:4f:00:00:00:00:00:00:00.
 
+What:		/sys/class/srp_remote_ports/port-<h>:<n>/reconnect_delay
+Date:		October 1, 2013
+KernelVersion:	3.11
+Contact:	[email protected], [email protected]
+Description:	Number of seconds the SCSI layer will wait after a reconnect
+		attempt failed before retrying.
+
 What:		/sys/class/srp_remote_ports/port-<h>:<n>/roles
 Date:		June 27, 2007
 KernelVersion:	2.6.24
 Contact:	[email protected]
 Description:	Role of the remote port. Either "SRP Initiator" or "SRP Target".
+
+What:		/sys/class/srp_remote_ports/port-<h>:<n>/state
+Date:		October 1, 2013
+KernelVersion:	3.11
+Contact:	[email protected], [email protected]
+Description:	State of the transport layer to the remote port. "running" if
+		the transport layer is operational; "blocked" if a transport
+		layer error has been encountered but the fail_io_fast_tmo
+		timer has not yet fired; "fail-fast" after the
+		fail_io_fast_tmo timer has fired and before the "dev_loss_tmo"
+		timer has fired; "lost" after the "dev_loss_tmo" timer has
+		fired and before the port is finally removed.
drivers/scsi/scsi_transport_srp.c +476 -1

546 lines not on this page · raw message

include/scsi/scsi_transport_srp.h +60 -2

84 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#38 lore raw

PATCH v2 9/15 IB/srp: Add srp_terminate_io()

In reply to a message not in the archive

Finish all outstanding I/O requests after fast_io_fail_tmo expired, which speeds up failover in a multipath setup. This patch is a reworked version of a patch from Sebastian Riemer.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |   22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +17 -5
index cb86be5..16c3612 100644
@@ -686,17 +686,29 @@ static void srp_free_req(struct srp_target_port *target,
 	spin_unlock_irqrestore(&target->lock, flags);
 }
 
-static void srp_reset_req(struct srp_target_port *target, struct srp_request *req)
+static void srp_finish_req(struct srp_target_port *target,
+			   struct srp_request *req, int result)
 {
 	struct scsi_cmnd *scmnd = srp_claim_req(target, req, NULL);
 
 	if (scmnd) {
 		srp_free_req(target, req, scmnd, 0);
-		scmnd->result = DID_RESET << 16;
+		scmnd->result = result;
 		scmnd->scsi_done(scmnd);
 	}
 }
 
+static void srp_terminate_io(struct srp_rport *rport)
+{
+	struct srp_target_port *target = rport->lld_data;
+	int i;
+
+	for (i = 0; i < SRP_CMD_SQ_SIZE; ++i) {
+		struct srp_request *req = &target->req_ring[i];
+		srp_finish_req(target, req, DID_TRANSPORT_FAILFAST << 16);
+	}
+}
+
 static int srp_reconnect_target(struct srp_target_port *target)
 {
 	struct Scsi_Host *shost = target->scsi_host;
@@ -723,8 +735,7 @@ static int srp_reconnect_target(struct srp_target_port *target)
 
 	for (i = 0; i < SRP_CMD_SQ_SIZE; ++i) {
 		struct srp_request *req = &target->req_ring[i];
-		if (req->scmnd)
-			srp_reset_req(target, req);
+		srp_finish_req(target, req, DID_RESET << 16);
 	}
 
 	INIT_LIST_HEAD(&target->free_tx);
@@ -1784,7 +1795,7 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
 	for (i = 0; i < SRP_CMD_SQ_SIZE; ++i) {
 		struct srp_request *req = &target->req_ring[i];
 		if (req->scmnd && req->scmnd->device == scmnd->device)
-			srp_reset_req(target, req);
+			srp_finish_req(target, req, DID_RESET << 16);
 	}
 
 	return SUCCESS;
@@ -2596,6 +2607,7 @@ static void srp_remove_one(struct ib_device *device)
 
 static struct srp_function_template ib_srp_transport_functions = {
 	.rport_delete		 = srp_rport_delete,
+	.terminate_rport_io	 = srp_terminate_io,
 };
 
 static int __init srp_init_module(void)
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#41 lore raw

PATCH v2 10/15 IB/srp: Use SRP transport layer error recovery

In reply to a message not in the archive

Enable reconnect_delay, fast_io_fail_tmo and dev_loss_tmo functionality for the IB SRP initiator. Add kernel module parameters that allow to specify default values for these three parameters.

Cc David Dillow <[email protected]>
Cc Roland Dreier <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |  123 +++++++++++++++++++++++++----------
 drivers/infiniband/ulp/srp/ib_srp.h |    1 -
 2 files changed, 88 insertions(+), 36 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +88 -35
index 16c3612..197b310 100644
@@ -86,6 +86,31 @@ module_param(topspin_workarounds, int, 0444);
 MODULE_PARM_DESC(topspin_workarounds,
 		 "Enable workarounds for Topspin/Cisco SRP target bugs if != 0");
 
+static int srp_reconnect_delay = 10;
+module_param_named(reconnect_delay, srp_reconnect_delay, int, S_IRUGO|S_IWUSR);
+MODULE_PARM_DESC(reconnect_delay, "Time between successive reconnect attempts");
+
+static struct kernel_param_ops srp_tmo_ops;
+
+static int srp_fast_io_fail_tmo = 15;
+module_param_cb(fast_io_fail_tmo, &srp_tmo_ops, &srp_fast_io_fail_tmo,
+		S_IRUGO | S_IWUSR);
+MODULE_PARM_DESC(fast_io_fail_tmo,
+		 "Number of seconds between the observation of a transport"
+		 " layer error and failing all I/O. \"off\" means that this"
+		 " functionality is disabled.");
+
+static int srp_dev_loss_tmo = 600;
+module_param_cb(dev_loss_tmo, &srp_tmo_ops, &srp_dev_loss_tmo,
+		S_IRUGO | S_IWUSR);
+MODULE_PARM_DESC(dev_loss_tmo,
+		 "Maximum number of seconds that the SRP transport should"
+		 " insulate transport layer errors. After this time has been"
+		 " exceeded the SCSI target is removed. Should be"
+		 " between 1 and " __stringify(SCSI_DEVICE_BLOCK_MAX_TIMEOUT)
+		 " if fast_io_fail_tmo has not been set. \"off\" means that"
+		 " this functionality is disabled.");
+
 static void srp_add_one(struct ib_device *device);
 static void srp_remove_one(struct ib_device *device);
 static void srp_recv_completion(struct ib_cq *cq, void *target_ptr);
@@ -102,6 +127,44 @@ static struct ib_client srp_client = {
 
 static struct ib_sa_client srp_sa_client;
 
+static int srp_tmo_get(char *buffer, const struct kernel_param *kp)
+{
+	int tmo = *(int *)kp->arg;
+
+	if (tmo >= 0)
+		return sprintf(buffer, "%d", tmo);
+	else
+		return sprintf(buffer, "off");
+}
+
+static int srp_tmo_set(const char *val, const struct kernel_param *kp)
+{
+	int tmo, res;
+
+	if (strncmp(val, "off", 3) != 0) {
+		res = kstrtoint(val, 0, &tmo);
+		if (res)
+			goto out;
+	} else {
+		tmo = -1;
+	}
+	if (kp->arg == &srp_fast_io_fail_tmo)
+		res = srp_tmo_valid(tmo, srp_dev_loss_tmo);
+	else
+		res = srp_tmo_valid(srp_fast_io_fail_tmo, tmo);
+	if (res)
+		goto out;
+	*(int *)kp->arg = tmo;
+
+out:
+	return res;
+}
+
+static struct kernel_param_ops srp_tmo_ops = {
+	.get = srp_tmo_get,
+	.set = srp_tmo_set,
+};
+
 static inline struct srp_target_port *host_to_target(struct Scsi_Host *host)
 {
 	return (struct srp_target_port *) host->hostdata;
@@ -709,13 +772,20 @@ static void srp_terminate_io(struct srp_rport *rport)
 	}
 }
 
-static int srp_reconnect_target(struct srp_target_port *target)
+/*
+ * It is up to the caller to ensure that srp_rport_reconnect() calls are
+ * serialized and that no concurrent srp_queuecommand(), srp_abort(),
+ * srp_reset_device() or srp_reset_host() calls will occur while this function
+ * is in progress. One way to realize that is not to call this function
+ * directly but to call srp_reconnect_rport() instead since that last function
+ * serializes calls of this function via rport->mutex and also blocks
+ * srp_queuecommand() calls before invoking this function.
+ */
+static int srp_rport_reconnect(struct srp_rport *rport)
 {
-	struct Scsi_Host *shost = target->scsi_host;
+	struct srp_target_port *target = rport->lld_data;
 	int i, ret;
 
-	scsi_target_block(&shost->shost_gendev);
-
 	srp_disconnect_target(target);
 	/*
 	 * Now get a new local CM ID so that we avoid confusing the target in
@@ -745,28 +815,9 @@ static int srp_reconnect_target(struct srp_target_port *target)
 	if (ret == 0)
 		ret = srp_connect_target(target);
 
-	scsi_target_unblock(&shost->shost_gendev, ret == 0 ? SDEV_RUNNING :
-			    SDEV_TRANSPORT_OFFLINE);
-	target->transport_offline = !!ret;
-
-	if (ret)
-		goto err;
-
-	shost_printk(KERN_INFO, target->scsi_host, PFX "reconnect succeeded\n");
-
-	return ret;
-
-err:
-	shost_printk(KERN_ERR, target->scsi_host,
-		     PFX "reconnect failed (%d), removing target port.\n", ret);
-
-	/*
-	 * We couldn't reconnect, so kill our target port off.
-	 * However, we have to defer the real removal because we
-	 * are in the context of the SCSI error handler now, which
-	 * will deadlock if we call scsi_remove_host().
-	 */
-	srp_queue_remove_work(target);
+	if (ret == 0)
+		shost_printk(KERN_INFO, target->scsi_host,
+			     PFX "reconnect succeeded\n");
 
 	return ret;
 }
@@ -1366,10 +1417,11 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd)
 	struct srp_cmd *cmd;
 	struct ib_device *dev;
 	unsigned long flags;
-	int len;
+	int len, result;
 
-	if (unlikely(target->transport_offline)) {
-		scmnd->result = DID_NO_CONNECT << 16;
+	result = srp_chkready(target->rport);
+	if (unlikely(result)) {
+		scmnd->result = result;
 		scmnd->scsi_done(scmnd);
 		return 0;
 	}
@@ -1768,7 +1820,7 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
 			      SRP_TSK_ABORT_TASK) == 0)
 		ret = SUCCESS;
-	else if (target->transport_offline)
+	else if (target->rport->state == SRP_RPORT_LOST)
 		ret = FAST_IO_FAIL;
 	else
 		ret = FAILED;
@@ -1804,14 +1856,10 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
 static int srp_reset_host(struct scsi_cmnd *scmnd)
 {
 	struct srp_target_port *target = host_to_target(scmnd->device->host);
-	int ret = FAILED;
 
 	shost_printk(KERN_ERR, target->scsi_host, PFX "SRP reset_host called\n");
 
-	if (!srp_reconnect_target(target))
-		ret = SUCCESS;
-
-	return ret;
+	return srp_reconnect_rport(target->rport) == 0 ? SUCCESS : FAILED;
 }
 
 static int srp_slave_configure(struct scsi_device *sdev)
@@ -2606,6 +2654,11 @@ static void srp_remove_one(struct ib_device *device)
 }
 
 static struct srp_function_template ib_srp_transport_functions = {
+	.has_rport_state	 = true,
+	.reconnect_delay	 = &srp_reconnect_delay,
+	.fast_io_fail_tmo	 = &srp_fast_io_fail_tmo,
+	.dev_loss_tmo		 = &srp_dev_loss_tmo,
+	.reconnect		 = srp_rport_reconnect,
 	.rport_delete		 = srp_rport_delete,
 	.terminate_rport_io	 = srp_terminate_io,
 };
drivers/infiniband/ulp/srp/ib_srp.h +0 -1
index 1817ed5..fda82f7 100644
@@ -140,7 +140,6 @@ struct srp_target_port {
 	unsigned int		cmd_sg_cnt;
 	unsigned int		indirect_size;
 	bool			allow_ext_sg;
-	bool			transport_offline;
 
 	/* Everything above this point is used in the hot path of
 	 * command processing. Try to keep them packed into cachelines.
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#44 lore raw

PATCH v2 11/15 IB/srp: Start timers if a transport layer error occurs

In reply to a message not in the archive

Start the reconnect timer, fast_io_fail timer and dev_loss timers if a transport layer error occurs.

Cc David Dillow <[email protected]>
Cc Roland Dreier <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |   19 +++++++++++++++++++
 drivers/infiniband/ulp/srp/ib_srp.h |    1 +
 2 files changed, 20 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +19 -0
index 197b310..a8cc427 100644
@@ -595,6 +595,7 @@ static void srp_remove_target(struct srp_target_port *target)
 	srp_disconnect_target(target);
 	ib_destroy_cm_id(target->cm_id);
 	srp_free_target_ib(target);
+	cancel_work_sync(&target->tl_err_work);
 	srp_rport_put(target->rport);
 	srp_free_req_data(target);
 	scsi_host_put(target->scsi_host);
@@ -1365,6 +1366,21 @@ static void srp_handle_recv(struct srp_target_port *target, struct ib_wc *wc)
 			     PFX "Recv failed with error code %d\n", res);
 }
 
+/**
+ * srp_tl_err_work() - handle a transport layer error
+ *
+ * Note: This function may get invoked before the rport has been created,
+ * hence the target->rport test.
+ */
+static void srp_tl_err_work(struct work_struct *work)
+{
+	struct srp_target_port *target;
+
+	target = container_of(work, struct srp_target_port, tl_err_work);
+	if (target->rport)
+		srp_start_tl_fail_timers(target->rport);
+}
+
 static void srp_handle_qp_err(enum ib_wc_status wc_status,
 			      enum ib_wc_opcode wc_opcode,
 			      struct srp_target_port *target)
@@ -1374,6 +1390,7 @@ static void srp_handle_qp_err(enum ib_wc_status wc_status,
 			     PFX "failed %s status %d\n",
 			     wc_opcode & IB_WC_RECV ? "receive" : "send",
 			     wc_status);
+		queue_work(system_long_wq, &target->tl_err_work);
 	}
 	target->qp_in_error = true;
 }
@@ -1734,6 +1751,7 @@ static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 		if (ib_send_cm_drep(cm_id, NULL, 0))
 			shost_printk(KERN_ERR, target->scsi_host,
 				     PFX "Sending CM DREP failed\n");
+		queue_work(system_long_wq, &target->tl_err_work);
 		break;
 
 	case IB_CM_TIMEWAIT_EXIT:
@@ -2381,6 +2399,7 @@ static ssize_t srp_create_target(struct device *dev,
 			     sizeof (struct srp_indirect_buf) +
 			     target->cmd_sg_cnt * sizeof (struct srp_direct_buf);
 
+	INIT_WORK(&target->tl_err_work, srp_tl_err_work);
 	INIT_WORK(&target->remove_work, srp_remove_work);
 	spin_lock_init(&target->lock);
 	INIT_LIST_HEAD(&target->free_tx);
drivers/infiniband/ulp/srp/ib_srp.h +1 -0
index fda82f7..e45d9d0 100644
@@ -175,6 +175,7 @@ struct srp_target_port {
 	struct srp_iu	       *rx_ring[SRP_RQ_SIZE];
 	struct srp_request	req_ring[SRP_CMD_SQ_SIZE];
 
+	struct work_struct	tl_err_work;
 	struct work_struct	remove_work;
 
 	struct list_head	list;
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
maintainer
#69 lore raw

PATCH v2 12/15 IB/srp: Fail SCSI commands silently

In reply to Bart Van Assche

From: Sebastian Riemer <[email protected]>

Avoid that path failover in a multipath setup causes the SCSI layer to generate kernel messages about SCSI command failures. This patch speeds up SRP initiator operation significantly when monitoring kernel messages over a serial port.

[bvanassche: Changed patch description]
Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +3 -0
index a8cc427..e77176e 100644
@@ -757,6 +757,7 @@ static void srp_finish_req(struct srp_target_port *target,
 
 	if (scmnd) {
 		srp_free_req(target, req, scmnd, 0);
+		scmnd->request->cmd_flags |= REQ_QUIET;
 		scmnd->result = result;
 		scmnd->scsi_done(scmnd);
 	}
@@ -1438,6 +1439,7 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd)
 
 	result = srp_chkready(target->rport);
 	if (unlikely(result)) {
+		scmnd->request->cmd_flags |= REQ_QUIET;
 		scmnd->result = result;
 		scmnd->scsi_done(scmnd);
 		return 0;
@@ -1843,6 +1845,7 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	else
 		ret = FAILED;
 	srp_free_req(target, req, scmnd, 0);
+	scmnd->request->cmd_flags |= REQ_QUIET;
 	scmnd->result = DID_ABORT << 16;
 	scmnd->scsi_done(scmnd);
 
Signature · 1 line
1.7.10.4
#47 lore raw

PATCH v2 13/15 IB/srp: Make HCA completion vector configurable

In reply to a message not in the archive

Several InfiniBand HCA's allow to configure the completion vector per queue pair. This allows to spread the workload created by IB completion interrupts over multiple MSI-X vectors and hence over multiple CPU cores. In other words, configuring the completion vector properly not only allows to reduce latency on an initiator connected to multiple SRP targets but also allows to improve throughput.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 Documentation/ABI/stable/sysfs-driver-ib_srp |    7 +++++++
 drivers/infiniband/ulp/srp/ib_srp.c          |   26 ++++++++++++++++++++++++--
 drivers/infiniband/ulp/srp/ib_srp.h          |    1 +
 3 files changed, 32 insertions(+), 2 deletions(-)
Documentation/ABI/stable/sysfs-driver-ib_srp +7 -0
index 481aae9..5c53d28 100644
@@ -54,6 +54,13 @@ Description:	Interface for making ib_srp connect to a new target.
 		  ib_srp. Specifying a value that exceeds cmd_sg_entries is
 		  only safe with partial memory descriptor list support enabled
 		  (allow_ext_sg=1).
+		* comp_vector, a number in the range 0..n-1 specifying the
+		  MSI-X completion vector. Some HCA's allocate multiple (n)
+		  MSI-X vectors per HCA port. If the IRQ affinity masks of
+		  these interrupts have been configured such that each MSI-X
+		  interrupt is handled by a different CPU then the comp_vector
+		  parameter can be used to spread the SRP completion workload
+		  over multiple CPU's.
 
 What:		/sys/class/infiniband_srp/srp-<hca>-<port_number>/ibdev
 Date:		January 2, 2006
drivers/infiniband/ulp/srp/ib_srp.c +24 -2
index e77176e..93cb101 100644
@@ -294,14 +294,16 @@ static int srp_create_target_ib(struct srp_target_port *target)
 		return -ENOMEM;
 
 	recv_cq = ib_create_cq(target->srp_host->srp_dev->dev,
-			       srp_recv_completion, NULL, target, SRP_RQ_SIZE, 0);
+			       srp_recv_completion, NULL, target, SRP_RQ_SIZE,
+			       target->comp_vector);
 	if (IS_ERR(recv_cq)) {
 		ret = PTR_ERR(recv_cq);
 		goto err;
 	}
 
 	send_cq = ib_create_cq(target->srp_host->srp_dev->dev,
-			       srp_send_completion, NULL, target, SRP_SQ_SIZE, 0);
+			       srp_send_completion, NULL, target, SRP_SQ_SIZE,
+			       target->comp_vector);
 	if (IS_ERR(send_cq)) {
 		ret = PTR_ERR(send_cq);
 		goto err_recv_cq;
@@ -1981,6 +1983,14 @@ static ssize_t show_local_ib_device(struct device *dev,
 	return sprintf(buf, "%s\n", target->srp_host->srp_dev->dev->name);
 }
 
+static ssize_t show_comp_vector(struct device *dev,
+				struct device_attribute *attr, char *buf)
+{
+	struct srp_target_port *target = host_to_target(class_to_shost(dev));
+
+	return sprintf(buf, "%d\n", target->comp_vector);
+}
+
 static ssize_t show_cmd_sg_entries(struct device *dev,
 				   struct device_attribute *attr, char *buf)
 {
@@ -2007,6 +2017,7 @@ static DEVICE_ATTR(req_lim,         S_IRUGO, show_req_lim,         NULL);
 static DEVICE_ATTR(zero_req_lim,    S_IRUGO, show_zero_req_lim,	   NULL);
 static DEVICE_ATTR(local_ib_port,   S_IRUGO, show_local_ib_port,   NULL);
 static DEVICE_ATTR(local_ib_device, S_IRUGO, show_local_ib_device, NULL);
+static DEVICE_ATTR(comp_vector,     S_IRUGO, show_comp_vector,     NULL);
 static DEVICE_ATTR(cmd_sg_entries,  S_IRUGO, show_cmd_sg_entries,  NULL);
 static DEVICE_ATTR(allow_ext_sg,    S_IRUGO, show_allow_ext_sg,    NULL);
 
@@ -2021,6 +2032,7 @@ static struct device_attribute *srp_host_attrs[] = {
 	&dev_attr_zero_req_lim,
 	&dev_attr_local_ib_port,
 	&dev_attr_local_ib_device,
+	&dev_attr_comp_vector,
 	&dev_attr_cmd_sg_entries,
 	&dev_attr_allow_ext_sg,
 	NULL
@@ -2145,6 +2157,7 @@ enum {
 	SRP_OPT_CMD_SG_ENTRIES	= 1 << 9,
 	SRP_OPT_ALLOW_EXT_SG	= 1 << 10,
 	SRP_OPT_SG_TABLESIZE	= 1 << 11,
+	SRP_OPT_COMP_VECTOR	= 1 << 12,
 	SRP_OPT_ALL		= (SRP_OPT_ID_EXT	|
 				   SRP_OPT_IOC_GUID	|
 				   SRP_OPT_DGID		|
@@ -2165,6 +2178,7 @@ static const match_table_t srp_opt_tokens = {
 	{ SRP_OPT_CMD_SG_ENTRIES,	"cmd_sg_entries=%u"	},
 	{ SRP_OPT_ALLOW_EXT_SG,		"allow_ext_sg=%u"	},
 	{ SRP_OPT_SG_TABLESIZE,		"sg_tablesize=%u"	},
+	{ SRP_OPT_COMP_VECTOR,		"comp_vector=%u"	},
 	{ SRP_OPT_ERR,			NULL 			}
 };
 
@@ -2320,6 +2334,14 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
 			target->sg_tablesize = token;
 			break;
 
+		case SRP_OPT_COMP_VECTOR:
+			if (match_int(args, &token) || token < 0) {
+				pr_warn("bad comp_vector parameter '%s'\n", p);
+				goto out;
+			}
+			target->comp_vector = token;
+			break;
+
 		default:
 			pr_warn("unknown parameter or missing value '%s' in target creation request\n",
 				p);
drivers/infiniband/ulp/srp/ib_srp.h +1 -0
index e45d9d0..cbc0b14 100644
@@ -156,6 +156,7 @@ struct srp_target_port {
 	char			target_name[32];
 	unsigned int		scsi_id;
 	unsigned int		sg_tablesize;
+	int			comp_vector;
 
 	struct ib_sa_path_rec	path;
 	__be16			orig_dgid[8];
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#50 lore raw

PATCH v2 14/15 IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive

From: Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Allow the InfiniBand RC retry count to be configured by the user as an option in the target login string. The transport layer timeout in nanoseconds is computed as follows from the retry count:

rc_timeout = rc_retry_count * 4 * 4096 * (1 << qp->timeout)

The default value for tl_retry_count is changed from 7 into 2. Hence with a qp->timedout value of 19 this patch reduces the default transport layer timeout from about 60s to about 17s. The purpose of this patch is to reduce the time needed for SCSI error handling significantly and at the same time to avoid activating the SCSI error handler on an IB path with a regular BER or due to brief IB network congestion.

[bvanassche: Rewrote patch description]
Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 Documentation/ABI/stable/sysfs-driver-ib_srp |    2 ++
 drivers/infiniband/ulp/srp/ib_srp.c          |   24 +++++++++++++++++++++++-
 drivers/infiniband/ulp/srp/ib_srp.h          |    1 +
 3 files changed, 26 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-driver-ib_srp +2 -0
index 5c53d28..18e9b27 100644
@@ -61,6 +61,8 @@ Description:	Interface for making ib_srp connect to a new target.
 		  interrupt is handled by a different CPU then the comp_vector
 		  parameter can be used to spread the SRP completion workload
 		  over multiple CPU's.
+		* tl_retry_count, a number in the range 2..7 specifying the
+		  IB RC retry count.
 
 What:		/sys/class/infiniband_srp/srp-<hca>-<port_number>/ibdev
 Date:		January 2, 2006
drivers/infiniband/ulp/srp/ib_srp.c +23 -1
index 93cb101..58a6af9 100644
@@ -453,7 +453,7 @@ static int srp_send_req(struct srp_target_port *target)
 	req->param.responder_resources	      = 4;
 	req->param.remote_cm_response_timeout = 20;
 	req->param.local_cm_response_timeout  = 20;
-	req->param.retry_count 		      = 7;
+	req->param.retry_count                = target->tl_retry_count;
 	req->param.rnr_retry_count 	      = 7;
 	req->param.max_cm_retries 	      = 15;
 
@@ -1991,6 +1991,14 @@ static ssize_t show_comp_vector(struct device *dev,
 	return sprintf(buf, "%d\n", target->comp_vector);
 }
 
+static ssize_t show_tl_retry_count(struct device *dev,
+				   struct device_attribute *attr, char *buf)
+{
+	struct srp_target_port *target = host_to_target(class_to_shost(dev));
+
+	return sprintf(buf, "%d\n", target->tl_retry_count);
+}
+
 static ssize_t show_cmd_sg_entries(struct device *dev,
 				   struct device_attribute *attr, char *buf)
 {
@@ -2018,6 +2026,7 @@ static DEVICE_ATTR(zero_req_lim,    S_IRUGO, show_zero_req_lim,	   NULL);
 static DEVICE_ATTR(local_ib_port,   S_IRUGO, show_local_ib_port,   NULL);
 static DEVICE_ATTR(local_ib_device, S_IRUGO, show_local_ib_device, NULL);
 static DEVICE_ATTR(comp_vector,     S_IRUGO, show_comp_vector,     NULL);
+static DEVICE_ATTR(tl_retry_count,  S_IRUGO, show_tl_retry_count,  NULL);
 static DEVICE_ATTR(cmd_sg_entries,  S_IRUGO, show_cmd_sg_entries,  NULL);
 static DEVICE_ATTR(allow_ext_sg,    S_IRUGO, show_allow_ext_sg,    NULL);
 
@@ -2033,6 +2042,7 @@ static struct device_attribute *srp_host_attrs[] = {
 	&dev_attr_local_ib_port,
 	&dev_attr_local_ib_device,
 	&dev_attr_comp_vector,
+	&dev_attr_tl_retry_count,
 	&dev_attr_cmd_sg_entries,
 	&dev_attr_allow_ext_sg,
 	NULL
@@ -2158,6 +2168,7 @@ enum {
 	SRP_OPT_ALLOW_EXT_SG	= 1 << 10,
 	SRP_OPT_SG_TABLESIZE	= 1 << 11,
 	SRP_OPT_COMP_VECTOR	= 1 << 12,
+	SRP_OPT_TL_RETRY_COUNT	= 1 << 13,
 	SRP_OPT_ALL		= (SRP_OPT_ID_EXT	|
 				   SRP_OPT_IOC_GUID	|
 				   SRP_OPT_DGID		|
@@ -2179,6 +2190,7 @@ static const match_table_t srp_opt_tokens = {
 	{ SRP_OPT_ALLOW_EXT_SG,		"allow_ext_sg=%u"	},
 	{ SRP_OPT_SG_TABLESIZE,		"sg_tablesize=%u"	},
 	{ SRP_OPT_COMP_VECTOR,		"comp_vector=%u"	},
+	{ SRP_OPT_TL_RETRY_COUNT,	"tl_retry_count=%u"	},
 	{ SRP_OPT_ERR,			NULL 			}
 };
 
@@ -2342,6 +2354,15 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
 			target->comp_vector = token;
 			break;
 
+		case SRP_OPT_TL_RETRY_COUNT:
+			if (match_int(args, &token) || token < 2 || token > 7) {
+				pr_warn("bad tl_retry_count parameter '%s' (must be a number between 2 and 7)\n",
+					p);
+				goto out;
+			}
+			target->tl_retry_count = token;
+			break;
+
 		default:
 			pr_warn("unknown parameter or missing value '%s' in target creation request\n",
 				p);
@@ -2396,6 +2417,7 @@ static ssize_t srp_create_target(struct device *dev,
 	target->cmd_sg_cnt	= cmd_sg_entries;
 	target->sg_tablesize	= indirect_sg_entries ? : cmd_sg_entries;
 	target->allow_ext_sg	= allow_ext_sg;
+	target->tl_retry_count	= 2;
 
 	ret = srp_parse_options(buf, target);
 	if (ret)
drivers/infiniband/ulp/srp/ib_srp.h +1 -0
index cbc0b14..446b045 100644
@@ -157,6 +157,7 @@ struct srp_target_port {
 	unsigned int		scsi_id;
 	unsigned int		sg_tablesize;
 	int			comp_vector;
+	int			tl_retry_count;
 
 	struct ib_sa_path_rec	path;
 	__be16			orig_dgid[8];
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#62 lore raw

PATCH v2 15/15 IB/srp: Bump driver version and release date

In reply to a message not in the archive

From: Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]>

---
 drivers/infiniband/ulp/srp/ib_srp.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +2 -2
index 58a6af9..4fefbde 100644
@@ -53,8 +53,8 @@
 
 #define DRV_NAME	"ib_srp"
 #define PFX		DRV_NAME ": "
-#define DRV_VERSION	"0.2"
-#define DRV_RELDATE	"November 1, 2005"
+#define DRV_VERSION	"1.0"
+#define DRV_RELDATE	"July 1, 2013"
 
 MODULE_AUTHOR("Roland Dreier");
 MODULE_DESCRIPTION("InfiniBand SCSI RDMA Protocol initiator "
Signature · 6 lines
1.7.10.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw

Re: [PATCH v2 02/15] IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive
Quoted 3 lines · Bart Van Assche

Avoid that srp_claim_command() can claim a command while srp_queuecommand() is still busy queueing the same command. Found this via source reading.

Nice, that's much less re-acquiring of the target lock in error case in srp_queuecommand().
But if we have to change that many locations for srp_put_tx_iu() anyway, wouldn't it make sense to rename it into __srp_put_tx_iu() as well?

Then we can also put a little description to it and it looks familiar compared to __srp_get_tx_iu().

The description could look like follows:
/*
* Return an IU and possible credit to the free pool *
* Must be called with target->lock held to protect free_tx. */

I'm not sure if we still need that lockdep_assert_held() then. There is no other location with lock debugging in ib_srp.

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw

Re: [PATCH v2 02/15] IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive
Quoted 20 lines · Sebastian Riemer
Quoted 3 lines · Bart Van Assche

Avoid that srp_claim_command() can claim a command while srp_queuecommand() is still busy queueing the same command. Found this via source reading.

Nice, that's much less re-acquiring of the target lock in error case in srp_queuecommand().
But if we have to change that many locations for srp_put_tx_iu() anyway, wouldn't it make sense to rename it into __srp_put_tx_iu() as well?

Then we can also put a little description to it and it looks familiar compared to __srp_get_tx_iu().

The description could look like follows:
/*
* Return an IU and possible credit to the free pool *
* Must be called with target->lock held to protect free_tx. */

Quoted 2 lines

I'm not sure if we still need that lockdep_assert_held() then. There is no other location with lock debugging in ib_srp.

Hello Sebastian,

I don't have a strong opinion about either of these two topics.

If a function like __srp_get_tx_iu() would be introduced that would allow to drop only two spin_lock/spin_unlock call pairs. So introducing that function would probably add more lines of code than adding the spin_lock/spin_unlock pairs. Hence my choice not to introduce __srp_get_tx_iu().

Regarding the lockdep_assert_held() statement: the reason I introduced it instead of adding a comment above the function telling which locking is required is because a lockdep_assert_held() statement is verified at runtime on a system with a kernel in which lockdep support has been enabled.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw

Re: [PATCH v2 02/15] IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive
Quoted 28 lines · Bart Van Assche
Quoted 15 lines

Nice, that's much less re-acquiring of the target lock in error case in srp_queuecommand().
But if we have to change that many locations for srp_put_tx_iu() anyway, wouldn't it make sense to rename it into __srp_put_tx_iu() as well?

Then we can also put a little description to it and it looks familiar compared to __srp_get_tx_iu().

The description could look like follows:
/*
* Return an IU and possible credit to the free pool *
* Must be called with target->lock held to protect free_tx. */

Quoted 2 lines

I'm not sure if we still need that lockdep_assert_held() then. There is no other location with lock debugging in ib_srp.

Hello Sebastian,

I don't have a strong opinion about either of these two topics.

If a function like __srp_get_tx_iu() would be introduced that would allow to drop only two spin_lock/spin_unlock call pairs. So introducing that function would probably add more lines of code than adding the spin_lock/spin_unlock pairs. Hence my choice not to introduce __srp_get_tx_iu().

Quoted 5 lines

Regarding the lockdep_assert_held() statement: the reason I introduced it instead of adding a comment above the function telling which locking is required is because a lockdep_assert_held() statement is verified at runtime on a system with a kernel in which lockdep support has been enabled.

Hi Bart,

I just meant a rename into "__srp_put_tx_iu()" to show that locking is required and not introducing a further wrapper function.

The other function of that kind "__srp_get_tx_iu()" also doesn't have a wrapper function "srp_get_tx_iu()".

For me it doesn't make much difference how it is marked that locking is required. I just wanted to point out that this method is new to ib_srp.

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

Re: [PATCH v2 02/15] IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive
Quoted 1 line · Bart Van Assche

Avoid that srp_claim_command() can claim a command while

I think you meant srp_claim_req() ?

Quoted 2 lines

srp_queuecommand() is still busy queueing the same command. Found this via source reading.

The main issue seems to be a broken, possibly malicious, target? I'm not sure I see another path to get into this race, as the scsi layer should protect us from queue submission and error handling at the same time, correct?

I'm not keen on giving up the command submission concurrency we worked so hard to get -- you completely serialize command submission again. If we really need to protect against this race -- and we probably should, to guard against broken targets -- I think we can achieve the concurrency and safety by improving the receive and error handling sides -- especially if we can push it into error handling, as that's already a slow path.

Let's push this patch to the back of the pile for any respin; I don't think the others rely on it (haven't gotten there yet) and would rather not hold them up for this.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#15 lore raw

Re: [PATCH v2 03/15] IB/srp: Avoid that srp_reset_host() is skipped after a TL error

In reply to a message not in the archive
Quoted 6 lines · Bart Van Assche

The SCSI error handler assumes that the transport layer is operational if an eh_abort_handler() returns SUCCESS. Hence let srp_abort() only return SUCCESS if sending the ABORT TASK task management function succeeded. This patch avoids that the SCSI error handler skips the srp_reset_host() call after a transport layer error.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#18 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 3 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#35 lore raw

Re: [PATCH v2 06/15] IB/srp: Maintain a single connection per I_T nexus

In reply to a message not in the archive
Quoted 9 lines · Bart Van Assche

An SRP target is required to maintain a single connection between initiator and target. This means that if the 'add_target' attribute is used to create a second connection to a target that the first connection will be logged out and that the SCSI error handler will kick in. The SCSI error handler will cause the SRP initiator to reconnect, which will cause I/O over the second connection to fail. Avoid such ping-pong behavior by disabling relogins. Note: if reconnecting manually is necessary, that is possible by deleting and recreating an rport via sysfs.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#64 lore raw

Re: [PATCH v2 08/15] scsi_transport_srp: Add transport layer error handling

In reply to Bart Van Assche
Quoted 26 lines · Bart Van Assche

Add the necessary functions in the SRP transport module to allow an SRP initiator driver to implement transport layer error handling similar to the functionality already provided by the FC transport layer. This includes:
- Support for implementing fast_io_fail_tmo, the time that should elapse after having detected a transport layer problem and before failing I/O.
- Support for implementing dev_loss_tmo, the time that should elapse after having detected a transport layer problem and before removing a remote port.

Cc Roland Dreier <[email protected]>
Cc James Bottomley <[email protected]>
Cc David Dillow <[email protected]>
Cc Vu Pham <[email protected]>
Cc Sebastian Riemer <[email protected]>
---
 Documentation/ABI/stable/sysfs-transport-srp |   37 ++
 drivers/scsi/scsi_transport_srp.c            |  477 +++++++++++++++++++++++++-
 include/scsi/scsi_transport_srp.h            |   62 +++-
 3 files changed, 573 insertions(+), 3 deletions(-)
drivers/scsi/scsi_transport_srp.c +3 -0
+/**
+ * srp_tmo_valid() - check timeout combination validity
+ *

drivers/scsi/scsi_transport_srp.c struct srp_internal { · line 76

Quoted 10 lines
drivers/scsi/scsi_transport_srp.c +10 -0
+ * If both a fast I/O fail and a device loss timeout have been configured then
+ * the fast I/O fail timeout must be below the device loss timeout.
+ */
+int srp_tmo_valid(int fast_io_fail_tmo, int dev_loss_tmo)
+{
+	return (fast_io_fail_tmo < 0 || dev_loss_tmo < 0 ||
+		fast_io_fail_tmo < dev_loss_tmo) &&
+		fast_io_fail_tmo < LONG_MAX / HZ &&
+		dev_loss_tmo < LONG_MAX / HZ ? 0 : -EINVAL;
+}

There's really no need to fit it in one line to save space -- it makes it much easier to read if you unpack it and return -EINVAL for the problem cases.

They should also be capped by SCSI_DEVICE_BLOCK_MAX_TIMEOUT instead of LONG_MAX / HZ, I think.

drivers/scsi/scsi_transport_srp.c static ssize_t store_srp_rport_delete(struct device *dev, · line 197

Quoted 12 lines
drivers/scsi/scsi_transport_srp.c +12 -0
+static ssize_t store_reconnect_delay(struct device *dev,
+				     struct device_attribute *attr,
+				     const char *buf, const size_t count)
+{
+	struct srp_rport *rport = transport_class_to_srp_rport(dev);
+	char ch[16];
+	int res, delay;
+
+	sprintf(ch, "%.*s", min_t(int, sizeof(ch) - 1, count), buf);
+	res = kstrtoint(ch, 0, &delay);
+	if (res)
+		goto out;

I don't see the need for the sprintf? kstrtoint() will give you -ERANGE if the value doesn't fit in an int...

Ah, NULL terminated string... except sysfs guarantees this for us for the store method -- see fill_write_buffer() in fs/sysfs.c.

drivers/scsi/scsi_transport_srp.c static ssize_t store_srp_rport_delete(struct device *dev, · line 255

Quoted 13 lines
drivers/scsi/scsi_transport_srp.c +13 -0
+static ssize_t store_srp_rport_fast_io_fail_tmo(struct device *dev,
+						struct device_attribute *attr,
+						const char *buf, size_t count)
+{
+	struct srp_rport *rport = transport_class_to_srp_rport(dev);
+	char ch[16], *p;
+	int res;
+	int fast_io_fail_tmo;
+
+	sprintf(ch, "%.*s", min_t(int, sizeof(ch) - 1, count), buf);
+	p = strchr(ch, '\n');
+	if (p)
+		*p = '\0';

Again, no need for the sprintf if you don't modify the buffer? Instead of using strchr() to make the strcmp() work with newlines, just do

if (!strcmp(buf, "off") || !strcmp(buf, "off\n")) {
	fast_io_fail_tmo = 1;
} else {
	res = kstrtoint(buf, 0, &fast_io_fail_tmo);

...

instead?

Same comment applies for store_srp_rport_dev_loss_tmo().

I haven't fully gone over the logic just yet, so I'm relying a bit on Hannes and Mike's discussion from the last round. This looks to be pretty reasonably sorted, though, based on the level of review I've been able to do so far...

Acked-by on v2 7/15
#37 lore raw

Re: [PATCH v2 07/15] IB/srp: Keep rport as long as the IB transport layer

In reply to Bart Van Assche
Quoted 8 lines · Bart Van Assche

Keep the rport data structure around after srp_remove_host() has finished until cleanup of the IB transport layer has finished completely. This is necessary because later patches use the rport pointer inside the queuecommand callback. Without this patch accessing the rport from inside a queuecommand callback is racy because srp_remove_host() must be invoked before scsi_remove_host() and because the queuecommand callback may get invoked after srp_remove_host() has finished.

Assuming the patches that require this are eventually accepted,

#40 lore raw

Re: [PATCH v2 09/15] IB/srp: Add srp_terminate_io()

In reply to a message not in the archive
Quoted 4 lines · Bart Van Assche

Finish all outstanding I/O requests after fast_io_fail_tmo expired, which speeds up failover in a multipath setup. This patch is a reworked version of a patch from Sebastian Riemer.

drivers/infiniband/ulp/srp/ib_srp.c static void srp_free_req(struct srp_target_port *target, · line 696

Quoted 10 lines
drivers/infiniband/ulp/srp/ib_srp.c +3 -2
-static void srp_reset_req(struct srp_target_port *target, struct srp_request *req)
+static void srp_finish_req(struct srp_target_port *target,
+			   struct srp_request *req, int result)
 {
 	struct scsi_cmnd *scmnd = srp_claim_req(target, req, NULL);
 
 	if (scmnd) {
 		srp_free_req(target, req, scmnd, 0);
-		scmnd->result = DID_RESET << 16;
+		scmnd->result = result;

Rather than have all the callers do the shift, just have srp_finish_req() do it.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#43 lore raw

Re: [PATCH v2 10/15] IB/srp: Use SRP transport layer error recovery

In reply to a message not in the archive

drivers/infiniband/ulp/srp/ib_srp.c static struct ib_client srp_client = { · line 144

Quoted 10 lines · Bart Van Assche

Enable reconnect_delay, fast_io_fail_tmo and dev_loss_tmo functionality for the IB SRP initiator. Add kernel module parameters that allow to specify default values for these three parameters.

drivers/infiniband/ulp/srp/ib_srp.c +5 -0
+static int srp_tmo_set(const char *val, const struct kernel_param *kp)
+{
+	int tmo, res;
+
+	if (strncmp(val, "off", 3) != 0) {

Trivial nit: this allows 'offended' to turn off the functionality. Perhaps use two strcmp() to deal with the possible newline? Of course, I think everyone else follows this pattern so, as I said trvial nit.

Assuming we accept patch 8,

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#46 lore raw

Re: [PATCH v2 11/15] IB/srp: Start timers if a transport layer error occurs

In reply to a message not in the archive
Quoted 2 lines · Bart Van Assche

Start the reconnect timer, fast_io_fail timer and dev_loss timers if a transport layer error occurs.

Assuming 8 is applied,

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#71 lore raw

Re: [PATCH v2 12/15] IB/srp: Fail SCSI commands silently

In reply to a message not in the archive
Quoted 6 lines · Bart Van Assche

From: Sebastian Riemer <[email protected]>

Avoid that path failover in a multipath setup causes the SCSI layer to generate kernel messages about SCSI command failures. This patch speeds up SRP initiator operation significantly when monitoring kernel messages over a serial port.

I'm not sure this should be fixed in the drivers -- this seems like an issue that should be handled/configured in the core based on the host result code.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by
#49 lore raw

Re: [PATCH v2 13/15] IB/srp: Make HCA completion vector configurable

In reply to a message not in the archive
Quoted 7 lines · Bart Van Assche

Several InfiniBand HCA's allow to configure the completion vector per queue pair. This allows to spread the workload created by IB completion interrupts over multiple MSI-X vectors and hence over multiple CPU cores. In other words, configuring the completion vector properly not only allows to reduce latency on an initiator connected to multiple SRP targets but also allows to improve throughput.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#52 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 8 lines · Bart Van Assche

From: Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Allow the InfiniBand RC retry count to be configured by the user as an option in the target login string. The transport layer timeout in nanoseconds is computed as follows from the retry count:

rc_timeout = rc_retry_count * 4 * 4096 * (1 << qp->timeout)

Quoted 7 lines

The default value for tl_retry_count is changed from 7 into 2. Hence with a qp->timedout value of 19 this patch reduces the default transport layer timeout from about 60s to about 17s. The purpose of this patch is to reduce the time needed for SCSI error handling significantly and at the same time to avoid activating the SCSI error handler on an IB path with a regular BER or due to brief IB network congestion.

I keep vacillating between preserving the default of 7 and opting for easier/optimized configuration for the common case. It my internal argument over this today, I wondered about changing the QP timeout instead -- doesn't that achieve your goals of allowing for errors and network congestion while optimizing for a reasonable fabric? Going from 19 to 17 drops the timeout by about the same amount, while allowing for more errors.

I agree that one or both of the items should be configurable, but I'm still worried about changing the defaults, given the feed back from those that want to use IB over the WAN.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#66 lore raw

Re: [PATCH v2 08/15] scsi_transport_srp: Add transport layer error handling

In reply to a message not in the archive
Quoted 11 lines · David Dillow

drivers/scsi/scsi_transport_srp.c struct srp_internal { · line 76

Quoted 7 lines · Bart Van Assche
drivers/scsi/scsi_transport_srp.c +7 -0
+int srp_tmo_valid(int fast_io_fail_tmo, int dev_loss_tmo)
+{
+	return (fast_io_fail_tmo < 0 || dev_loss_tmo < 0 ||
+		fast_io_fail_tmo < dev_loss_tmo) &&
+		fast_io_fail_tmo < LONG_MAX / HZ &&
+		dev_loss_tmo < LONG_MAX / HZ ? 0 : -EINVAL;
+}

They should also be capped by SCSI_DEVICE_BLOCK_MAX_TIMEOUT instead of LONG_MAX / HZ, I think.

The fast_io_fail_tmo should indeed be capped by that value. However, I'm not sure about dev_loss_tmo. I think there are several use cases (e.g. initiator-side mirroring) where it's useful to set dev_loss_tmo to a larger value than ten minutes.

Quoted 21 lines

drivers/scsi/scsi_transport_srp.c static ssize_t store_srp_rport_delete(struct device *dev, · line 255

Quoted 13 lines
drivers/scsi/scsi_transport_srp.c +13 -0
+static ssize_t store_srp_rport_fast_io_fail_tmo(struct device *dev,
+						struct device_attribute *attr,
+						const char *buf, size_t count)
+{
+	struct srp_rport *rport = transport_class_to_srp_rport(dev);
+	char ch[16], *p;
+	int res;
+	int fast_io_fail_tmo;
+
+	sprintf(ch, "%.*s", min_t(int, sizeof(ch) - 1, count), buf);
+	p = strchr(ch, '\n');
+	if (p)
+		*p = '\0';

Again, no need for the sprintf if you don't modify the buffer? Instead of using strchr() to make the strcmp() work with newlines, just do

if (!strcmp(buf, "off") || !strcmp(buf, "off\n")) {
	fast_io_fail_tmo = 1;
} else {
	res = kstrtoint(buf, 0, &fast_io_fail_tmo);
Quoted 5 lines

...

instead?

Same comment applies for store_srp_rport_dev_loss_tmo().

OK, will remove the temporary char arrays.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw

Re: [PATCH v2 02/15] IB/srp: Fix race between srp_queuecommand() and srp_claim_req()

In reply to a message not in the archive
Quoted 3 lines · David Dillow

Let's push this patch to the back of the pile for any respin; I don't think the others rely on it (haven't gotten there yet) and would rather not hold them up for this.

I'll drop this patch for now. It's a race I have not yet been able to trigger anyway.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#54 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 25 lines · David Dillow
Quoted 8 lines · Bart Van Assche

From: Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Allow the InfiniBand RC retry count to be configured by the user as an option in the target login string. The transport layer timeout in nanoseconds is computed as follows from the retry count:

rc_timeout = rc_retry_count * 4 * 4096 * (1 << qp->timeout)

Quoted 7 lines

The default value for tl_retry_count is changed from 7 into 2. Hence with a qp->timedout value of 19 this patch reduces the default transport layer timeout from about 60s to about 17s. The purpose of this patch is to reduce the time needed for SCSI error handling significantly and at the same time to avoid activating the SCSI error handler on an IB path with a regular BER or due to brief IB network congestion.

I keep vacillating between preserving the default of 7 and opting for easier/optimized configuration for the common case. It my internal argument over this today, I wondered about changing the QP timeout instead -- doesn't that achieve your goals of allowing for errors and network congestion while optimizing for a reasonable fabric? Going from 19 to 17 drops the timeout by about the same amount, while allowing for more errors.

Quoted 3 lines

I agree that one or both of the items should be configurable, but I'm still worried about changing the defaults, given the feed back from those that want to use IB over the WAN.

The InfiniBand specification mentions the following about differential receiver inputs (C6-11.2.1): "A BER of 10^-12 shall be achieved when
connected to the worst case transmitter through any compliant channel". The maximum packet size for an InfiniBand packet is about 4 KB (see also section 7.7.8 in the spec). This means that with an 8b/10b encoding the chance to lose a packet over a single link due to bit errors is about 4*10^-8. So the chance to lose a packet over a network consisting of n links with retry count r is about (n*4*10^-8)^r. With r=2 that results already in a really low value, even with multiple links. Since lowering the QP timeout might make congestion worse my preference is to lower the retry count.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#19 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 19 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Sebastian Riemer <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +0 -0
index 8c95262..5c91521 100644

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 8 lines
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
 			      SRP_TSK_ABORT_TASK) == 0)
 		ret = SUCCESS;
+	else if (target->transport_offline)
+		ret = FAST_IO_FAIL;
 	else
 		ret = FAILED;
 	srp_free_req(target, req, scmnd, 0);

This doesn't give us much speed advantage IMHO. The check for target->transport_offline should be before calling srp_send_tsk_mgmt().

This way it would also match the patch description better.

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 19 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Sebastian Riemer <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
 drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +0 -0
index 8c95262..5c91521 100644

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 8 lines
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
 			      SRP_TSK_ABORT_TASK) == 0)
 		ret = SUCCESS;
+	else if (target->transport_offline)
+		ret = FAST_IO_FAIL;
 	else
 		ret = FAILED;
 	srp_free_req(target, req, scmnd, 0);

I'm also missing the concept for srp_reset_device(). There is a very common case that the SCSI error handling and the transport layer error handling run in parallel: Congestion.

In congestion some LUNs are blocked while others can still transmit. A little bit later the QP timeout triggers in the middle of the SCSI error handling in srp_abort(), srp_reset_device() or less likely in srp_reset_host().

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#68 lore raw

Re: [PATCH v2 08/15] scsi_transport_srp: Add transport layer error handling

In reply to a message not in the archive
Quoted 13 lines · Bart Van Assche
Quoted 11 lines · David Dillow

drivers/scsi/scsi_transport_srp.c struct srp_internal { · line 76

Quoted 7 lines · Bart Van Assche
drivers/scsi/scsi_transport_srp.c +7 -0
+int srp_tmo_valid(int fast_io_fail_tmo, int dev_loss_tmo)
+{
+	return (fast_io_fail_tmo < 0 || dev_loss_tmo < 0 ||
+		fast_io_fail_tmo < dev_loss_tmo) &&
+		fast_io_fail_tmo < LONG_MAX / HZ &&
+		dev_loss_tmo < LONG_MAX / HZ ? 0 : -EINVAL;
+}

They should also be capped by SCSI_DEVICE_BLOCK_MAX_TIMEOUT instead of LONG_MAX / HZ, I think.

Quoted 4 lines

The fast_io_fail_tmo should indeed be capped by that value. However, I'm not sure about dev_loss_tmo. I think there are several use cases (e.g. initiator-side mirroring) where it's useful to set dev_loss_tmo to a larger value than ten minutes.

Ah, yes, very good point.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#56 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 26 lines · Bart Van Assche
Quoted 25 lines · David Dillow
Quoted 8 lines · Bart Van Assche

From: Vu Pham <vuhuong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Allow the InfiniBand RC retry count to be configured by the user as an option in the target login string. The transport layer timeout in nanoseconds is computed as follows from the retry count:

rc_timeout = rc_retry_count * 4 * 4096 * (1 << qp->timeout)

Quoted 7 lines

The default value for tl_retry_count is changed from 7 into 2. Hence with a qp->timedout value of 19 this patch reduces the default transport layer timeout from about 60s to about 17s. The purpose of this patch is to reduce the time needed for SCSI error handling significantly and at the same time to avoid activating the SCSI error handler on an IB path with a regular BER or due to brief IB network congestion.

I keep vacillating between preserving the default of 7 and opting for easier/optimized configuration for the common case. It my internal argument over this today, I wondered about changing the QP timeout instead -- doesn't that achieve your goals of allowing for errors and network congestion while optimizing for a reasonable fabric? Going from 19 to 17 drops the timeout by about the same amount, while allowing for more errors.

Quoted 7 lines
Quoted 3 lines

I agree that one or both of the items should be configurable, but I'm still worried about changing the defaults, given the feed back from those that want to use IB over the WAN.

The InfiniBand specification mentions the following about differential receiver inputs (C6-11.2.1): "A BER of 10^-12 shall be achieved when connected to the worst case transmitter through any compliant channel".

There are a _lot_ of networks out there with non-compliant channels, then. The spec says one thing, but the reality matters. Also, perhaps my math is wrong, but that seems to be one error every 18 seconds or so at FDR speeds?

Quoted 8 lines

The maximum packet size for an InfiniBand packet is about 4 KB (see also section 7.7.8 in the spec). This means that with an 8b/10b encoding the chance to lose a packet over a single link due to bit errors is about 4*10^-8. So the chance to lose a packet over a network consisting of n links with retry count r is about (n*4*10^-8)^r. With r=2 that results already in a really low value, even with multiple links. Since lowering the QP timeout might make congestion worse my preference is to lower the retry count.

You assume independent failures, which is suspect -- many times these are data-dependent, or so I tend to think. Jason, do you have any insight on this (overall) topic you could share?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 32 lines · Sebastian Riemer
Quoted 19 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Sebastian Riemer <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
  drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
  1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +0 -0
index 8c95262..5c91521 100644

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 8 lines
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
  	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
  			      SRP_TSK_ABORT_TASK) == 0)
  		ret = SUCCESS;
+	else if (target->transport_offline)
+		ret = FAST_IO_FAIL;
  	else
  		ret = FAILED;
  	srp_free_req(target, req, scmnd, 0);

This doesn't give us much speed advantage IMHO. The check for target->transport_offline should be before calling srp_send_tsk_mgmt().

Quoted 1 line

This way it would also match the patch description better.

Hello Sebastian,

Had you perhaps overlooked the following code at the start of srp_send_tsk_mgmt() ?

	if (!target->connected || target->qp_in_error)
		return -1;

Given this I don't think it matters whether the transport_offline check occurs before or after the srp_send_tsk_mgmt() call.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#26 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 23 lines · Sebastian Riemer
Quoted 22 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

Cc Roland Dreier <[email protected]>
Cc David Dillow <[email protected]>
Cc Sebastian Riemer <[email protected]>
Cc Vu Pham <vu-VPRAkNaXOzVWk0Htik3J/[email protected]>
---
  drivers/infiniband/ulp/srp/ib_srp.c |    2 ++
  1 file changed, 2 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +1 -0
index 8c95262..5c91521 100644
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
  	if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
  			      SRP_TSK_ABORT_TASK) == 0)
  		ret = SUCCESS;
+	else if (target->transport_offline)
Quoted 8 lines

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 4 lines
drivers/infiniband/ulp/srp/ib_srp.c +1 -0
+		ret = FAST_IO_FAIL;
  	else
  		ret = FAILED;
  	srp_free_req(target, req, scmnd, 0);

I'm also missing the concept for srp_reset_device(). There is a very common case that the SCSI error handling and the transport layer error handling run in parallel: Congestion.

Can you explain this comment further, and also how this comment relates to patch 04/15 ?

Quoted 4 lines

In congestion some LUNs are blocked while others can still transmit. A little bit later the QP timeout triggers in the middle of the SCSI error handling in srp_abort(), srp_reset_device() or less likely in srp_reset_host().

I am aware this can result in concurrent srp_reconnect_rport() calls. However, such concurrent calls are serialized via rport->mutex.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#58 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 9 lines · David Dillow
Quoted 3 lines · Bart Van Assche

The InfiniBand specification mentions the following about differential receiver inputs (C6-11.2.1): "A BER of 10^-12 shall be achieved when connected to the worst case transmitter through any compliant channel".

There are a _lot_ of networks out there with non-compliant channels, then. The spec says one thing, but the reality matters. Also, perhaps my math is wrong, but that seems to be one error every 18 seconds or so at FDR speeds?

If data is transferred at line rate on an FDR network, I agree that about one error is expected about every 18 seconds.

Sorry but it's not clear to me what makes you think that there are a lot of IB networks with non-compliant channels ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#23 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 20 lines · Bart Van Assche
Quoted 16 lines

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 12 lines
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
      if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
                    SRP_TSK_ABORT_TASK) == 0)
          ret = SUCCESS;
+    else if (target->transport_offline)
+        ret = FAST_IO_FAIL;
      else
          ret = FAILED;
      srp_free_req(target, req, scmnd, 0);

This doesn't give us much speed advantage IMHO. The check for target->transport_offline should be before calling srp_send_tsk_mgmt().

Quoted 1 line

This way it would also match the patch description better.

Hello Sebastian,

Quoted 8 lines

Had you perhaps overlooked the following code at the start of srp_send_tsk_mgmt() ?

    if (!target->connected || target->qp_in_error)
        return -1;

Given this I don't think it matters whether the transport_offline check occurs before or after the srp_send_tsk_mgmt() call.

Hi Bart,

okay, right. So you get an error due to the connected and qp_in_error state first. Yes, I've overlooked that. Thanks!

Cheers,
Sebastian

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 16 lines · Bart Van Assche
Quoted 7 lines
Quoted 7 lines
drivers/infiniband/ulp/srp/ib_srp.c +1 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
      if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
                    SRP_TSK_ABORT_TASK) == 0)
          ret = SUCCESS;
+    else if (target->transport_offline)
Quoted 8 lines

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 4 lines
drivers/infiniband/ulp/srp/ib_srp.c +1 -0
+        ret = FAST_IO_FAIL;
      else
          ret = FAILED;
      srp_free_req(target, req, scmnd, 0);

I'm also missing the concept for srp_reset_device(). There is a very common case that the SCSI error handling and the transport layer error handling run in parallel: Congestion.

Quoted 2 lines

Can you explain this comment further, and also how this comment relates to patch 04/15 ?

Sorry, found it. Even if only one srp_reset_device() fails, then srp_reset_host() is called anyway. So there this check + returning FAST_IO_FAIL doesn't make so much sense.

Quoted 7 lines
Quoted 4 lines

In congestion some LUNs are blocked while others can still transmit. A little bit later the QP timeout triggers in the middle of the SCSI error handling in srp_abort(), srp_reset_device() or less likely in srp_reset_host().

I am aware this can result in concurrent srp_reconnect_rport() calls. However, such concurrent calls are serialized via rport->mutex.

I put my comment regarding this to patch 10.

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#30 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 17 lines · Sebastian Riemer
Quoted 16 lines · Bart Van Assche
Quoted 15 lines

drivers/infiniband/ulp/srp/ib_srp.c static int srp_abort(struct scsi_cmnd *scmnd) · line 1762

Quoted 11 lines
drivers/infiniband/ulp/srp/ib_srp.c +2 -0
@@ -1755,6 +1755,8 @@ static int srp_abort(struct scsi_cmnd *scmnd)
       if (srp_send_tsk_mgmt(target, req->index, scmnd->device->lun,
                     SRP_TSK_ABORT_TASK) == 0)
           ret = SUCCESS;
+    else if (target->transport_offline)
+        ret = FAST_IO_FAIL;
       else
           ret = FAILED;
       srp_free_req(target, req, scmnd, 0);

I'm also missing the concept for srp_reset_device(). There is a very common case that the SCSI error handling and the transport layer error handling run in parallel: Congestion.

Quoted 6 lines
Quoted 2 lines

Can you explain this comment further, and also how this comment relates to patch 04/15 ?

Sorry, found it. Even if only one srp_reset_device() fails, then srp_reset_host() is called anyway. So there this check + returning FAST_IO_FAIL doesn't make so much sense.

Hello Sebastian,

I agree that if one or more srp_abort() calls return FAILED that srp_reset_host() will be called anyway. However, this patch helps if the first call of srp_abort() occurs after a reconnect failure, which is likely with the default reconnnect_delay and SCSI timeout settings. In that case all srp_abort() calls will return FAST_IO_FAIL, scsi_eh_abort_cmds() will terminate the pending commands and hence the host reset will be skipped.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#31 lore raw

Re: [PATCH v2 04/15] IB/srp: Fail I/O fast if target offline

In reply to a message not in the archive
Quoted 3 lines · Bart Van Assche

If reconnecting failed we know that no command completion will be received anymore. Hence let the SCSI error handler fail such commands immediately.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#59 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 3 lines · David Dillow
Quoted 3 lines

The InfiniBand specification mentions the following about differential receiver inputs (C6-11.2.1): "A BER of 10^-12 shall be achieved when connected to the worst case transmitter through any compliant channel".

This test condition is under 'stressed conditions' for the higher rates, which doesn't directly translate into an error every 18 seconds under normal conditions.

A properly functioning IB link should not be seeing link errors with any frequency (eg we expect/observe no errors on single links over month long periods).

Naturally, a large number of IB links, will on aggregate, have error rates significantly higher..

.. and this whole area is very challenging and installations with defective cables/defective end points/non-compliance/etc are not uncommon, so some sites see higher error rates. :(

However, IIRC, one large installation I know of, had the achieved target of nearly no errors on any link during typical operation.

In a practical sense, IB just doesn't work with packet loss. You need to have low error rate signalling or your network doesn't work/won't perform.

Quoted 9 lines
Quoted 8 lines

The maximum packet size for an InfiniBand packet is about 4 KB (see also section 7.7.8 in the spec). This means that with an 8b/10b encoding the chance to lose a packet over a single link due to bit errors is about 4*10^-8. So the chance to lose a packet over a network consisting of n links with retry count r is about (n*4*10^-8)^r. With r=2 that results already in a really low value, even with multiple links. Since lowering the QP timeout might make congestion worse my preference is to lower the retry count.

Quoted 3 lines

You assume independent failures, which is suspect -- many times these are data-dependent, or so I tend to think. Jason, do you have any insight on this (overall) topic you could share?

All data transmitted on modern serial links is 'whitened' somehow. This is does independently on a link-by-link basis either with 8b/10b coding or with the 64b/66b scrambler. So the idea of a high level 'magic packet' that causes data-dependent errors is not statistically likely.

Errors in properly functioning modern serial links are pure-random, from the perspective of SRP.

It is best to use all the information the SM provides when setting up the path, however I don't think there is a best practice idea yet for how to setup the retry count though..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#61 lore raw

Re: [PATCH v2 14/15] IB/srp: Make transport layer retry count configurable

In reply to a message not in the archive
Quoted 10 lines · Jason Gunthorpe
Quoted 3 lines · David Dillow

You assume independent failures, which is suspect -- many times these are data-dependent, or so I tend to think. Jason, do you have any insight on this (overall) topic you could share?

All data transmitted on modern serial links is 'whitened' somehow. This is does independently on a link-by-link basis either with 8b/10b coding or with the 64b/66b scrambler. So the idea of a high level 'magic packet' that causes data-dependent errors is not statistically likely.

My thought was that if we hit a statistically unlikely pattern that caused an issue, the retransmission is likely to also hit the issue given the deterministic scrambling. But I didn't think about the fact that the signal stream was being whitened.

Quoted 3 lines

It is best to use all the information the SM provides when setting up the path, however I don't think there is a best practice idea yet for how to setup the retry count though..

Hmm, that would be a useful presentation for the workshop; I'll have to see if I can get some people interested here.

Thanks for the information,
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html