threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

DISCUSSION Konstantin Krotov 2 participants 8 messages Aug 10, 2015 – Aug 14, 2015

Subsystems (3)

SCSI InfiniBand SCSI RDMA protocol (SRP) initiator

lore

Share & tools

lore

Konstantin Krotov

author

2015-08-10 15:18 UTC #0 lore raw

Hello list!

I was faced with the following problem:

Im exported to compute-node scsi disks through ib_srp from storages. And if reconnect occurs for any reason, eg

"[1201117.626498] scsi host62: ib_srp: REJ received

[1201117.626507] scsi host62: ib_srp: SRP LOGIN from fe80:0000:0000:0000:0002:c903:0009:1796 to fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"

i find threads in output ps:
"root 28047 0.0 0.0 0 0 ? S 18:04 0:00 [scsi_eh_62]
root 28048 0.0 0.0 0 0 ? S< 18:04 0:00 [scsi_tmf_62]"

Each reconnect generates two thread, which do not die.

kernel version 3.19.1

modinfo ib_srp

filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Apr 09 2015
version:        2.0.32#in-tree

Signature · 2 lines

WBR
Konstantin V. Krotov

Bart Van Assche

2015-08-10 15:30 UTC #1 lore raw

Quoted 17 lines · Konstantin Krotov

I was faced with the following problem:
Im exported to compute-node scsi disks through ib_srp from storages. And if reconnect occurs for any reason, eg
"[1201117.626498] scsi host62: ib_srp: REJ received
[1201117.626507] scsi host62: ib_srp: SRP LOGIN from
fe80:0000:0000:0000:0002:c903:0009:1796 to
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"
i find threads in output ps:
"root 28047 0.0 0.0 0 0 ? S 18:04 0:00
[scsi_eh_62]
root 28048 0.0 0.0 0 0 ? S< 18:04 0:00
[scsi_tmf_62]"

Quoted 8 lines

Each reconnect generates two thread, which do not die.
kernel version 3.19.1
modinfo ib_srp
filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Apr 09 2015
version:        2.0.32#in-tree

Hello Konstantin,

Does the above output mean that you are using the ib_srp-backport driver ? If so, please upgrade to version v2.0.34. That version includes all SRP initiator fixes that have been posted on the linux-rdma and linux-scsi mailing lists for inclusion in Linux kernel v4.3.

Bart.

Konstantin Krotov

author

2015-08-11 13:40 UTC #2 lore raw

10.08.2015 18:30, Bart Van Assche пишет:

Quoted 27 lines · Konstantin Krotov

Quoted 17 lines
I was faced with the following problem:
Im exported to compute-node scsi disks through ib_srp from storages. And if reconnect occurs for any reason, eg
"[1201117.626498] scsi host62: ib_srp: REJ received
[1201117.626507] scsi host62: ib_srp: SRP LOGIN from
fe80:0000:0000:0000:0002:c903:0009:1796 to
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"
i find threads in output ps:
"root 28047 0.0 0.0 0 0 ? S 18:04 0:00
[scsi_eh_62]
root 28048 0.0 0.0 0 0 ? S< 18:04 0:00
[scsi_tmf_62]"
Quoted 8 lines
Each reconnect generates two thread, which do not die.
kernel version 3.19.1
modinfo ib_srp
filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Apr 09 2015
version:        2.0.32#in-tree

Quoted 9 lines

Hello Konstantin,
Does the above output mean that you are using the ib_srp-backport driver ? If so, please upgrade to version v2.0.34. That version includes all SRP initiator fixes that have been posted on the linux-rdma and linux-scsi mailing lists for inclusion in Linux kernel v4.3.
Bart.

Hello, thanks for reply.

I build a package version 2.0.34, but problem is reproduced:

[  260.266374] scsi host15: ib_srp: REJ received
[  260.266383] scsi host15: ib_srp: SRP LOGIN from fe80:0000:0000:0000:0002:c903:0008:c884 to fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[  260.266588] scsi host15: ib_srp: Connection 0/8 failed
[  260.266700] scsi host15: ib_srp: Sending CM DREQ failed

and increases the number of threads scsi_tmf_ and scsi_eh_ at every reconnect..

modinfo ib_srp

filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Jul 27 2015
version:        2.0.34#in-tree
license:        Dual BSD/GPL
description:    InfiniBand SCSI RDMA Protocol initiator
author:         Roland Dreier
srcversion:     75206A3E8C5F05BED61C20B
depends: ib_core,scsi_transport_srp,ib_sa,scsi_mod,ib_cm,rdma_cm,scsi_transport_srp
vermagic:       3.19-3-amd64 SMP mod_unload modversions

modinfo scsi_transport_srp

filename:       /lib/modules/3.19-3-amd64/updates/dkms/scsi_transport_srp.ko
license:        GPL
description:    SRP Transport Attributes
author:         FUJITA Tomonori
depends:        scsi_mod
vermagic:       3.19-3-amd64 SMP mod_unload modversions

Signature · 8 lines

WBR
Konstantin V. Krotov


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bart Van Assche

2015-08-11 20:28 UTC #3 lore raw

Quoted 10 lines · Konstantin Krotov

Hello, thanks for reply.

I build a package version 2.0.34, but problem is reproduced:

[  260.266374] scsi host15: ib_srp: REJ received
[  260.266383] scsi host15: ib_srp: SRP LOGIN from
fe80:0000:0000:0000:0002:c903:0008:c884 to
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[  260.266588] scsi host15: ib_srp: Connection 0/8 failed
[  260.266700] scsi host15: ib_srp: Sending CM DREQ failed

Quoted 2 lines

and increases the number of threads scsi_tmf_ and scsi_eh_ at every reconnect..

Hello Konstantin,

I have tried to reproduce this but without success so far. Whether I install upstream kernel v4.2-rc6 or another kernel version + ib_srp-backport on the initiator system, the result is the same when I block SRP login on the target system: the ib_srp driver keeps trying to log in without creating new SCSI hosts. Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the ib_srp kernel module parameters set (cat /etc/modprobe.d/ib_srp.conf) ?

Bart.

Konstantin Krotov

author

2015-08-12 05:15 UTC #4 lore raw

11.08.2015 23:28, Bart Van Assche пишет:

Quoted 16 lines · Konstantin Krotov

Quoted 10 lines
Hello, thanks for reply.
I build a package version 2.0.34, but problem is reproduced:
[  260.266374] scsi host15: ib_srp: REJ received
[  260.266383] scsi host15: ib_srp: SRP LOGIN from
fe80:0000:0000:0000:0002:c903:0008:c884 to
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[  260.266588] scsi host15: ib_srp: Connection 0/8 failed
[  260.266700] scsi host15: ib_srp: Sending CM DREQ failed
Quoted 2 lines
and increases the number of threads scsi_tmf_ and scsi_eh_ at every reconnect..
Hello Konstantin,

Quoted 7 lines

I have tried to reproduce this but without success so far. Whether I install upstream kernel v4.2-rc6 or another kernel version + ib_srp-backport on the initiator system, the result is the same when I block SRP login on the target system: the ib_srp driver keeps trying to log in without creating new SCSI hosts.
Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the

yes, sources from upstream, 3.19.1

Quoted 1 line

ib_srp kernel module parameters set (cat /etc/modprobe.d/ib_srp.conf) ?

srp.konf missing, i load modules with the default settings:
allow_ext_sg:N
ch_count:0
cmd_sg_entries:12
dev_loss_tmo:600
enable_imm_data:Y
fast_io_fail_tmo:15
indirect_sg_entries:12
prefer_fr:N
reconnect_delay:20
register_always:N
srp_sg_tablesize:0
topspin_workarounds:1

Quoted 2 lines

Bart.

Signature · 7 lines

WBR
Konstantin V. Krotov

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bart Van Assche

2015-08-12 16:50 UTC #5 lore raw

Quoted 3 lines · Konstantin Krotov

11.08.2015 23:28, Bart Van Assche пишет:
Quoted 1 line
Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the
yes, sources from upstream, 3.19.1

Hello Konstantin,

Can you test the patch below ? That patch not only applies to the v4.2-rc6 kernel but also to the ib_srp-backport driver.

Thanks,

Bart.

Subject: [PATCH] IB/srp: Stop the scsi_eh_<n> and scsi_tmf_<n> threads if login fails

scsi_host_alloc() not only allocates memory for a SCSI host but also creates the scsi_eh_<n> kernel thread and the scsi_tmf_<n> workqueue. Stop these threads if login fails by calling scsi_host_put().

Reported-by Konstantin Krotov <[email protected]>

Signed-off-by Bart Van Assche <[email protected]>

Cc stable <[email protected]>

---
 drivers/infiniband/ulp/srp/ib_srp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

drivers/infiniband/ulp/srp/ib_srp.c +8 -0

index f962267..b5f9456 100644
@@ -2765,6 +2765,13 @@ static int srp_sdev_count(struct Scsi_Host *host)
 	return c;
 }
 
+/*
+ * Return values:
+ * < 0 upon failure. Caller is responsible for SRP target port cleanup.
+ * 0 and target->state == SRP_TARGET_REMOVED if the target port will be removed
+ *    asynchronously.
+ * 0 and target->state != SRP_TARGET_REMOVED upon success.
+ */
 static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
 {
 	struct srp_rport_identifiers ids;

@@ -3316,6 +3323,7 @@ err_disconnect:
 	}
 
 	kfree(target->ch);
+	scsi_host_put(target->scsi_host);
 	goto out;
 }

Signature · 7 lines

2.1.4


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Konstantin Krotov

author

2015-08-14 09:06 UTC #6 lore raw

12.08.2015 19:50, Bart Van Assche пишет:

Quoted 47 lines · Konstantin Krotov

Quoted 3 lines
11.08.2015 23:28, Bart Van Assche пишет:
Quoted 1 line
Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the
yes, sources from upstream, 3.19.1
Hello Konstantin,
Can you test the patch below ? That patch not only applies to the v4.2-rc6 kernel but also to the ib_srp-backport driver.
Thanks,
Bart.
Subject: [PATCH] IB/srp: Stop the scsi_eh_<n> and scsi_tmf_<n> threads if login fails
scsi_host_alloc() not only allocates memory for a SCSI host but also creates the scsi_eh_<n> kernel thread and the scsi_tmf_<n> workqueue. Stop these threads if login fails by calling scsi_host_put().
Reported-by Konstantin Krotov <[email protected]>
Signed-off-by Bart Van Assche <[email protected]>
Cc stable <[email protected]>
---
  drivers/infiniband/ulp/srp/ib_srp.c | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +7 -0
index f962267..b5f9456 100644
@@ -2765,6 +2765,13 @@ static int srp_sdev_count(struct Scsi_Host *host)
  	return c;
  }
 
+/*
+ * Return values:
+ * < 0 upon failure. Caller is responsible for SRP target port cleanup.
+ * 0 and target->state == SRP_TARGET_REMOVED if the target port will be removed
+ *    asynchronously.
+ * 0 and target->state != SRP_TARGET_REMOVED upon success.
+ */
  static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
  {
  	struct srp_rport_identifiers ids;

drivers/infiniband/ulp/srp/ib_srp.c err_disconnect: · line 3329

Quoted 7 lines

drivers/infiniband/ulp/srp/ib_srp.c +1 -0

@@ -3316,6 +3323,7 @@ err_disconnect:
  	}
 
  	kfree(target->ch);
+	scsi_host_put(target->scsi_host);
  	goto out;
  }

Hello Bart

I tested the patch, the problem is not reproduced.

Thanks.

Signature · 7 lines

WBR
Konstantin V. Krotov

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bart Van Assche

2015-08-14 16:10 UTC #7 lore raw

Quoted 1 line · Konstantin Krotov

I tested the patch, the problem is not reproduced.

Thanks for confirming that the patch fixes this issue. I will send a slightly modified version of this patch soon to the Linux RDMA maintainer.

Bart.