New: Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

Bug 79901 In review [email protected] 5 participants 14 messages Jul 10, 2014 – Jul 29, 2014 SCSI

Share & tools

[email protected]

author

2014-07-10 13:24 UTC #0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=79901

            Bug ID: 79901
           Summary: Extremely slow boot on Promise VTrak E610f due to
                    sd_mod RSOC usage
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.14.7
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Recently I've started upgrading all of my machines to kernel 3.14 (from Debian wheezy backports to be precise). Mostly there were not problems, but I've stumbled upon weird behavior on Fibre Channel servers (QLogic cards inside HP blades) using Promise VTrak E610f arrays.

As soon as SCSI subsystem tries to detect partitions a lot of SCSI errors are reported. The system stalls (but initramfs is responsive) for about 20-30 minutes (depending on number of arrays and LUNs). After that time, disk detection finishes and system continues booting as usual. Everything works perfectly afterwards.

I've spent some time fiddling with qla2xxx driver versions, SCSI scanning options and anything else I could think of. Finally, I was able to find culprit. The problem lies in sd_mod usage of scsi_report_opcode(). This function is used to determine if the disk supports WRITE SAME command. It does so by issuing REPORT SUPPORTED OPERATION CODES command. Unfortunately, it seems Promise VTrak E610f really, really does not like RSOC. As soon as RSOC is issued the array stalls for a while, then kernel tries to abort the command and finally it must reset the port. Fortunately the array starts working again after the reset. I've also verified this behavior with sg_opcodes utility.

Commit that introduced RSOC usage in sd_mod:
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=98dcc2946adbe4349ef1ef9b99873b912831edd4
Removing it fixes the issue.

I'm not sure what is the correct way to fix this as I'm not very familiar with SCSI spec. If RSOC support cannot be reliably determined then probably some kind of blacklist should be introduced.

As a workaround, I've modified qla2xxx driver to set 'no_write_same' flag. While not directly related it forces sd_mod not to issue RSOC and it is easier for me to ship Debian package with modified single driver (I'd prefer not to manage my own kernel packages).

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

Christoph Hellwig

2014-07-10 13:54 UTC #1 lore raw

Sounds like we need to skip REPORT SUPPORTED OPERATION CODES on this array.

For this it needs a new BLIST_NORSOC flag added to include/scsi/scsi_devinfo.h which sets sdev->no_report_opcodes in scsi_add_lun, and an entry for your array in the big black list array in drivers/scsi/scsi_devinfo.c. Does this sound easy enough that you could prepare a patch for it?

Janusz Dziemidowicz

2014-07-10 14:10 UTC #2 lore raw

2014-07-10 15:54 GMT+02:00 Christoph Hellwig <[email protected]>:

Quoted 8 lines

Sounds like we need to skip REPORT SUPPORTED OPERATION CODES on this array.
For this it needs a new BLIST_NORSOC flag added to include/scsi/scsi_devinfo.h which sets sdev->no_report_opcodes in scsi_add_lun, and an entry for your array in the big black list array in drivers/scsi/scsi_devinfo.c. Does this sound easy enough that you could prepare a patch for it?

Sure, I'll give it a try. It will probably take a couple of days as currently I have only one server that is available for experiments for a few hours a day.

Signature · 1 line

Janusz Dziemidowicz

[email protected]

author

2014-07-24 14:07 UTC #3 lore raw

[Bug 79901] Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #1 from Janusz Dziemidowicz <[email protected]> --- Created attachment 144101
--> https://bugzilla.kernel.org/attachment.cgi?id=144101&action=edit
Patch adding BLIST_NO_RSOC scsi scan flag

As discussed on the list, attached simple patch that blacklist RSOC on Promise VTrak E610f

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

Christoph Hellwig

2014-07-24 15:28 UTC #4 lore raw

Re: [Bug 79901] Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

Thanks, this one looks good to me. Can someone else give me a second review on the attached patch so I can merged it?

---

Quoted 1 line

From 95bc83fa0d6e9a6152cd6fce79ff87c0994e83ed Mon Sep 17 00:00:00 2001

From: Janusz Dziemidowicz <[email protected]>
Date: Thu, 24 Jul 2014 15:48:46 +0200
Subject: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f

Some devices don't like REPORT SUPPORTED OPERATION CODES and will simply timeout causing sd_mod init to take a very very long time. Introduce BLIST_NO_RSOC scsi scan flag, that stops RSOC from being issued. Add it to Promise Vtrak E610f entry in scsi scan blacklist. Fixes bug #79901 reported at
https://bugzilla.kernel.org/show_bug.cgi?id=79901

Fixes 98dcc2946adb ("SCSI: sd: Update WRITE SAME heuristics")

Signed-off-by Janusz Dziemidowicz <[email protected]>

---
 drivers/scsi/scsi_devinfo.c | 1 +
 drivers/scsi/scsi_scan.c    | 6 ++++++
 include/scsi/scsi_devinfo.h | 1 +
 3 files changed, 8 insertions(+)

drivers/scsi/scsi_devinfo.c +1 -0

index f969aca..49014a1 100644
@@ -222,6 +222,7 @@ static struct {
 	{"PIONEER", "CD-ROM DRM-602X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-604X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-624X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
+	{"Promise", "VTrak E610f", NULL, BLIST_SPARSELUN | BLIST_NO_RSOC},
 	{"Promise", "", NULL, BLIST_SPARSELUN},
 	{"QUANTUM", "XP34301", "1071", BLIST_NOTQ},
 	{"REGAL", "CDC-4X", NULL, BLIST_MAX5LUN | BLIST_SINGLELUN},

drivers/scsi/scsi_scan.c +6 -0

index e02b3aa..70919fc 100644
@@ -922,6 +922,12 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 	if (*bflags & BLIST_USE_10_BYTE_MS)
 		sdev->use_10_for_ms = 1;
 
+	/* some devices don't like REPORT SUPPORTED OPERATION CODES
+	 * and will simply timeout causing sd_mod init to take a very
+	 * very long time */
+	if (*bflags & BLIST_NO_RSOC)
+		sdev->no_report_opcodes = 1;
+
 	/* set the device running here so that slave configure
 	 * may do I/O */
 	ret = scsi_device_set_state(sdev, SDEV_RUNNING);

include/scsi/scsi_devinfo.h +1 -0

index 447d2d7..1d5ba9e 100644
@@ -32,4 +32,5 @@
 #define BLIST_ATTACH_PQ3	0x1000000 /* Scan: Attach to PQ3 devices */
 #define BLIST_NO_DIF		0x2000000 /* Disable T10 PI (DIF) */
 #define BLIST_SKIP_VPD_PAGES	0x4000000 /* Ignore SBC-3 VPD pages */
+#define BLIST_NO_RSOC		0x8000000 /* don't try REPORT SUPPORTED OPERATION CODES */
 #endif

Signature · 1 line

1.9.3

Christoph Hellwig

2014-07-29 12:53 UTC #5 lore raw

PATCH 1/1 Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

ping? �I'd really like to get a review on this one and merge it for 3.17

Quoted 71 lines · Christoph Hellwig

Thanks, this one looks good to me. Can someone else give me a second review on the attached patch so I can merged it?
---
Quoted 1 line
From 95bc83fa0d6e9a6152cd6fce79ff87c0994e83ed Mon Sep 17 00:00:00 2001
From: Janusz Dziemidowicz <[email protected]>
Date: Thu, 24 Jul 2014 15:48:46 +0200
Subject: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f
Some devices don't like REPORT SUPPORTED OPERATION CODES and will simply timeout causing sd_mod init to take a very very long time. Introduce BLIST_NO_RSOC scsi scan flag, that stops RSOC from being issued. Add it to Promise Vtrak E610f entry in scsi scan blacklist. Fixes bug #79901 reported at
https://bugzilla.kernel.org/show_bug.cgi?id=79901
Fixes 98dcc2946adb ("SCSI: sd: Update WRITE SAME heuristics")
Signed-off-by Janusz Dziemidowicz <[email protected]>
---
 drivers/scsi/scsi_devinfo.c | 1 +
 drivers/scsi/scsi_scan.c    | 6 ++++++
 include/scsi/scsi_devinfo.h | 1 +
 3 files changed, 8 insertions(+)
drivers/scsi/scsi_devinfo.c +1 -0
index f969aca..49014a1 100644
@@ -222,6 +222,7 @@ static struct {
 	{"PIONEER", "CD-ROM DRM-602X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-604X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-624X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
+	{"Promise", "VTrak E610f", NULL, BLIST_SPARSELUN | BLIST_NO_RSOC},
 	{"Promise", "", NULL, BLIST_SPARSELUN},
 	{"QUANTUM", "XP34301", "1071", BLIST_NOTQ},
 	{"REGAL", "CDC-4X", NULL, BLIST_MAX5LUN | BLIST_SINGLELUN},
drivers/scsi/scsi_scan.c +6 -0
index e02b3aa..70919fc 100644
@@ -922,6 +922,12 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 	if (*bflags & BLIST_USE_10_BYTE_MS)
 		sdev->use_10_for_ms = 1;
 
+	/* some devices don't like REPORT SUPPORTED OPERATION CODES
+	 * and will simply timeout causing sd_mod init to take a very
+	 * very long time */
+	if (*bflags & BLIST_NO_RSOC)
+		sdev->no_report_opcodes = 1;
+
 	/* set the device running here so that slave configure
 	 * may do I/O */
 	ret = scsi_device_set_state(sdev, SDEV_RUNNING);
include/scsi/scsi_devinfo.h +1 -0
index 447d2d7..1d5ba9e 100644
@@ -32,4 +32,5 @@
 #define BLIST_ATTACH_PQ3	0x1000000 /* Scan: Attach to PQ3 devices */
 #define BLIST_NO_DIF		0x2000000 /* Disable T10 PI (DIF) */
 #define BLIST_SKIP_VPD_PAGES	0x4000000 /* Ignore SBC-3 VPD pages */
+#define BLIST_NO_RSOC		0x8000000 /* don't try REPORT SUPPORTED OPERATION CODES */
 #endif
Signature · 6 lines
1.9.3

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

---end quoted text---
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hannes Reinecke

2014-07-29 12:57 UTC #6 lore raw

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

Martin K. Petersen

Reviewed-by on v1 1/1

2014-07-29 13:00 UTC #7 lore raw

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

In reply to Christoph Hellwig

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Christoph" == Christoph Hellwig <[email protected]> writes:

Christoph> ping? ÍI'd really like to get a review on this one and merge Christoph> it for 3.17

Looks good to me.

Reviewed-by Martin K. Petersen <[email protected]>

Signature · 5 lines

Martin K. Petersen	Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hannes Reinecke

2014-07-29 13:14 UTC #8 lore raw

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

In reply to Christoph Hellwig

[email protected]

author

2014-07-29 16:05 UTC #9 lore raw

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

Alan <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
            Summary|Extremely slow boot on      |[BISECTED]Extremely slow
                   |Promise VTrak E610f due to  |boot on Promise VTrak E610f
                   |sd_mod RSOC usage           |due to sd_mod RSOC usage

--- Comment #2 from Alan <[email protected]> ---
See Documentation/SubmittingPatches - we need a valid email/Signed-off-by for submissions

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

[email protected]

author

2014-07-29 19:07 UTC #10 lore raw

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #3 from Janusz Dziemidowicz <[email protected]> --- I've read that document before creating the patch. Now, I've read it again. I must say I'm at a loss. In the patch attached to this bug entry I've already included Signed-off-by line.
Can anyone point precisely what is wrong here?

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

Christoph Hellwig

2014-07-29 21:56 UTC #11 lore raw

Re: [Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

Quoted 7 lines · On Tue, Jul 29, 2014 at 07:07:51PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=79901
--- Comment #3 from Janusz Dziemidowicz <[email protected]> --- I've read that document before creating the patch. Now, I've read it again. I must say I'm at a loss. In the patch attached to this bug entry I've already included Signed-off-by line.
Can anyone point precisely what is wrong here?

Everything is fine and I already applied your patch. Thanks a lot for debugging the issue and coming up with the patch!

[email protected]

author

2014-07-29 20:26 UTC #12 lore raw

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #4 from Alan <[email protected]> ---
It needs to go via email. If you send it to [email protected] then the right things should happen (feel free to cc me as well so I can help keep an eye on it)

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

[email protected]

author

2014-07-29 20:27 UTC #13 lore raw

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

Alan <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Regression|No |Yes

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.