SCSI scanning behavior
In one of our SAN test labs where there is some storage controller error injection going on, I'm seeing some interesting behavior. We are getting into a scenario, when the target is coming back where we are going through SCSI scan for it and the Report LUNs we are issuing to it times out, so we fall back to a sequential LUN scan. When performing the sequential LUN scan, we end up adding a bunch of LUNs than we didn't previously see, 512 in fact. The target is reporting PQ=1, PDT=0 for every LUN that doesn't exist. When Report LUNs *does work*, it doesn't report these LUNs.
In net, we end up with a different result if we do a sequential LUN scan compared to a report LUNs scan.
Now, one could argue this is a defect in the SCSI target, since SPC says:
The REPORT LUNS parameter data should be returned even though the device server is not ready for other commands. The report of the logical unit inventory should be available without incurring any media access delays. If the device server is not ready with the logical unit inventory or if the inventory list is null for the requesting I_T nexus and the SELECT REPORT field set to 02h, then the device server shall provide a default logical unit inventory that contains at least LUN 0 or the REPORT LUNS well known logical unit (see 8.2). A non-empty peripheral device logical unit inventory that does not contain either LUN 0 or the REPORT LUNS well known logical unit is valid.
However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all. Are there media changer devices out there that we've seen respond like this? Even so, does it make sense to add PQ=1, PDT=0 LUNs for LUN > 0?
Thanks,
Brian
Signature · 3 lines
Brian King Power Linux I/O IBM Linux Technology Center
Quoted 13 lines · Brian King
In one of our SAN test labs where there is some storage controller error injection going on, I'm seeing some interesting behavior. We are getting into a scenario, when the target is coming back where we are going through SCSI scan for it and the Report LUNs we are issuing to it times out, so we fall back to a sequential LUN scan. When performing the sequential LUN scan, we end up adding a bunch of LUNs than we didn't previously see, 512 in fact. The target is reporting PQ=1, PDT=0 for every LUN that doesn't exist. When Report LUNs *does work*, it doesn't report these LUNs.
In net, we end up with a different result if we do a sequential LUN scan compared to a report LUNs scan.
Now, one could argue this is a defect in the SCSI target, since SPC says:
Quoted 8 lines
The REPORT LUNS parameter data should be returned even though the device server is not ready for other commands. The report of the logical unit inventory should be available without incurring any media access delays. If the device server is not ready with the logical unit inventory or if the inventory list is null for the requesting
I_Tnexus and the SELECT REPORT field set to 02h, then the device server shall provide a default logical unit inventory that contains at least LUN 0 or the REPORT LUNS well known logical unit (see 8.2). A non-empty peripheral device logical unit inventory that does not contain either LUN 0 or the REPORT LUNS well known logical unit is valid.
Hey, join the club. I've had a similar array, which were returning a default inventory during bootup. Leaving us with no chance to detect if the default inventory was the correct one or not.
I even posted a patch some time ago; if you wish I can drag it out.
Quoted 4 lines
However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all. Are there media changer devices out there that we've seen respond like this? Even so, does it make sense to add PQ=1, PDT=0 LUNs for LUN > 0?
Yes, unfortunately we need this. NetApp arrays have a habit of returning 'PQ=1' for unconnected LUN 0, even though higher LUNs are present. So we need to add devices for PQ=1, otherwise we wouldn't be able to scan them.
We _might_ be able to tweak this by ignoring devices with PQ=1 and LUN!=0; however, it might break other things.
As a net result, do avoid sequential scan if at all possible. I would rather work on getting REPORT LUNs to reliably report data.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Hannes Reinecke
Quoted 4 lines · Brian King
However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all. Are there media changer devices out there that we've seen respond like this? Even so, does it make sense to add PQ=1, PDT=0 LUNs for LUN > 0?
Yes, unfortunately we need this. NetApp arrays have a habit of returning 'PQ=1' for unconnected LUN 0, even though higher LUNs are present. So we need to add devices for PQ=1, otherwise we wouldn't be able to scan them.
We _might_ be able to tweak this by ignoring devices with PQ=1 and LUN!=0; however, it might break other things.
Hello Hannes,
The code in scsi_scan.c already skips LUNs with PQ=1 and PDT=0x1f. I'm not sure we should skip LUNS with PQ=1 and a PDT value other than 0x1f.
Thanks,
Bart.
PATCH 1/1 SCSI: Scale up REPORT_LUNS timeout on failure
In reply to Hannes ReineckeThis patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the REPORT_LUNS issued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.
This patch doubles the timeout used on the REPORT_LUNS for each retry after a timeout is seen on a REPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final, REPORT_LUNS was always successful.
---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +4 -1
@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
@@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"
@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )break ;
_
Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure
Quoted 43 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"
drivers/scsi/scsi_scan.c retry: · line 1400
Quoted 8 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )break ;
This is somewhat of a hack, but anyway:
Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure
Quoted 55 lines · Bart Van Assche
Quoted 43 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c 2015-09-02 08:49:07.272243461 -0500 renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate 2015-09-02 08:49:07.268243497 -0500 +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s struct scsi_device *sdev; struct Scsi_Host *shost = dev_to_shost(&starget->dev); int ret = 0; + int timeout = SCSI_TIMEOUT + 4 * HZ; /* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.@@ -1383,7 +1384,7 @@ retry: result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE, lun_data, length, &sshdr, - SCSI_TIMEOUT + 4 * HZ, 3, NULL); + timeout, 3, NULL); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"drivers/scsi/scsi_scan.c 2015-09-02 08:49:07.272243461 -0500 retry: · line 1400
Quoted 8 lines
drivers/scsi/scsi_scan.c 2015-09-02 08:49:07.272243461 -0500 renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate 2015-09-02 08:49:07.268243497 -0500 +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result)); if (result == 0) break; + else if (host_byte(result) == DID_TIME_OUT) + timeout = timeout * 2; else if (scsi_sense_valid(&sshdr)) { if (sshdr.sense_key != UNIT_ATTENTION) break;This is somewhat of a hack, but anyway:
Quoted 1 line
Reviewed-by Bart Van Assche <[email protected]>
Agreed.
Thanks for the review.
-Brian
Signature · 3 lines
Brian King Power Linux I/O IBM Linux Technology Center
James Bottomley
maintainerRe: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure
In reply to Brian KingQuoted 42 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"
drivers/scsi/scsi_scan.c retry: · line 1399
Quoted 7 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )
Actually, this is a bit pointless, isn't it; why retry, why not just set the initial timeout? ... I could understand if retrying and printing a message gave important or useful information, but it doesn't. How long do you actually need? ... we can just up the initial timeout to that. Currently we have a hacked 6s which looks arbitrary. Would 15s be better? Nothing really times out anyway, so everything else will still reply within the original 6s giving zero impact in the everyday case.
James
Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure
Quoted 51 lines · James Bottomley
Quoted 42 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"drivers/scsi/scsi_scan.c retry: · line 1399
Quoted 7 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )
Quoted 7 lines
Actually, this is a bit pointless, isn't it; why retry, why not just set the initial timeout? ... I could understand if retrying and printing a message gave important or useful information, but it doesn't. How long do you actually need? ... we can just up the initial timeout to that. Currently we have a hacked 6s which looks arbitrary. Would 15s be better? Nothing really times out anyway, so everything else will still reply within the original 6s giving zero impact in the everyday case.
12 seconds definitely isn't long enough, but 24 seconds seems to work, at least after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds? 15 seconds is likely to be right on the edge in this scenario.
Signature · 3 lines
Brian King Power Linux I/O IBM Linux Technology Center
James Bottomley
maintainerRe: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure
Quoted 60 lines · Brian King
Quoted 51 lines · James Bottomley
Quoted 42 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"drivers/scsi/scsi_scan.c retry: · line 1399
Quoted 7 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )Quoted 7 lines
Actually, this is a bit pointless, isn't it; why retry, why not just set the initial timeout? ... I could understand if retrying and printing a message gave important or useful information, but it doesn't. How long do you actually need? ... we can just up the initial timeout to that. Currently we have a hacked 6s which looks arbitrary. Would 15s be better? Nothing really times out anyway, so everything else will still reply within the original 6s giving zero impact in the everyday case.
Quoted 3 lines
12 seconds definitely isn't long enough, but 24 seconds seems to work, at least after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds? 15 seconds is likely to be right on the edge in this scenario.
30s is fine by me. I think the initial 2s was from the sequential inquiry scan so as not to wait too long. The extra 4s was added because that was too short for report luns on some devices; I suspect some larger arrays take a while just to gather all the data.
30s is also the traditional rq_timeout, so it may be possible to re-use this parameter. Currently it's set up in the ULD, so it's zero unless the slave_configure requested a special value. Traditionally, it's the timeout for _READ and _WRITE, not special commands, but it feels like REPORT_LUNS should follow this timeout as well and it would give you a configurable way of updating it in your driver. If we do it this way, you'd have to set it in slave_alloc, because slave_configure is too late.
James
PATCH 1/1 SCSI: Increase REPORT_LUNS timeout
Quoted 70 lines · James Bottomley
Quoted 60 lines · Brian King
Quoted 51 lines · James Bottomley
Quoted 42 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"drivers/scsi/scsi_scan.c retry: · line 1399
Quoted 7 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )Quoted 7 lines
Actually, this is a bit pointless, isn't it; why retry, why not just set the initial timeout? ... I could understand if retrying and printing a message gave important or useful information, but it doesn't. How long do you actually need? ... we can just up the initial timeout to that. Currently we have a hacked 6s which looks arbitrary. Would 15s be better? Nothing really times out anyway, so everything else will still reply within the original 6s giving zero impact in the everyday case.
Quoted 3 lines
12 seconds definitely isn't long enough, but 24 seconds seems to work, at least after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds? 15 seconds is likely to be right on the edge in this scenario.
30s is fine by me. I think the initial 2s was from the sequential inquiry scan so as not to wait too long. The extra 4s was added because that was too short for report luns on some devices; I suspect some larger arrays take a while just to gather all the data.
Quoted 8 lines
30s is also the traditional
rq_timeout, so it may be possible to re-use this parameter. Currently it's set up in the ULD, so it's zero unless theslave_configurerequested a special value. Traditionally, it's the timeout for_READand_WRITE, not special commands, but it feels likeREPORT_LUNSshould follow this timeout as well and it would give you a configurable way of updating it in your driver. If we do it this way, you'd have to set it inslave_alloc, becauseslave_configureis too late.
I think we may just need to hard code it like the patch below. Here is the current flow for setting this today:
slave_alloc
scsi scan: inquiry / report LUNsslave_configure
sd attach
Some LLDDs set a default timeout in slave_configure today, so sd.c only sets a default timeout if its not already set. It uses 30 seconds for disks and 75 seconds for optical devices. If we start setting rq_timeout earlier, then the ULD will never know when it can set it.
Additionally, in this particular scenario, its not so much a case of behavior tied to the LLDD, its more tied to the SCSI target. If there is concern about increasing the default to 30 seconds, we could use a blist attribute for this.
-Brian
8<
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the REPORT_LUNS issued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.
This patch increases the timeout used on the REPORT_LUNS to 30 seconds. This patch solves the issue of 512 non existent LUNs showing up after this event.
---
drivers/scsi/scsi_scan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_30secs +2 -1
@@ -55,6 +55,7 @@* Default timeout */ #define SCSI_TIMEOUT (2*HZ) +#define SCSI_REPORT_LUNS_TIMEOUT (30*HZ)/* * Prefix values for the SCSI id's (stored in sysfs name field)
@@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); +SCSI_REPORT_LUNS_TIMEOUT ,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"
_
Re: [PATCH] SCSI: Increase REPORT_LUNS timeout
Quoted 139 lines · Brian King
Quoted 70 lines · James Bottomley
Quoted 60 lines · Brian King
Quoted 51 lines · James Bottomley
Quoted 42 lines · Brian King
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch doubles the timeout used on the
REPORT_LUNSfor each retry after a timeout is seen on aREPORT_LUNS. This patch solves the issue of 512 non existent LUNs showing up after this event. Running the test with this patch still showed that we were regularly hitting two timeouts, but the third, and final,REPORT_LUNSwas always successful.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalatedrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-1@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct sstruct scsi_device *sdev;struct Scsi_Host *shost =dev_to_shost (&starget->dev);int ret =0 ; +int timeout =SCSI_TIMEOUT +4 *HZ ;/* * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set. @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); + timeout,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"drivers/scsi/scsi_scan.c retry: · line 1399
Quoted 7 lines
drivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate +2
-0@@ -1392,6 +1393,8 @@ retry: retries, result));if (result ==0 )break ; +else if (host_byte (result) ==DID_TIME_OUT ) + timeout = timeout *2 ;else if (scsi_sense_valid (&sshdr)) {if (sshdr.sense_key !=UNIT_ATTENTION )Quoted 7 lines
Actually, this is a bit pointless, isn't it; why retry, why not just set the initial timeout? ... I could understand if retrying and printing a message gave important or useful information, but it doesn't. How long do you actually need? ... we can just up the initial timeout to that. Currently we have a hacked 6s which looks arbitrary. Would 15s be better? Nothing really times out anyway, so everything else will still reply within the original 6s giving zero impact in the everyday case.
Quoted 3 lines
12 seconds definitely isn't long enough, but 24 seconds seems to work, at least after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds? 15 seconds is likely to be right on the edge in this scenario.
30s is fine by me. I think the initial 2s was from the sequential inquiry scan so as not to wait too long. The extra 4s was added because that was too short for report luns on some devices; I suspect some larger arrays take a while just to gather all the data.
Quoted 8 lines
30s is also the traditional
rq_timeout, so it may be possible to re-use this parameter. Currently it's set up in the ULD, so it's zero unless theslave_configurerequested a special value. Traditionally, it's the timeout for_READand_WRITE, not special commands, but it feels likeREPORT_LUNSshould follow this timeout as well and it would give you a configurable way of updating it in your driver. If we do it this way, you'd have to set it inslave_alloc, becauseslave_configureis too late.I think we may just need to hard code it like the patch below. Here is the current flow for setting this today:
slave_alloc
scsi scan: inquiry / report LUNsslave_configure
sd attachSome LLDDs set a default timeout in
slave_configuretoday, sosd.conly sets a default timeout if its not already set. It uses 30 seconds for disks and 75 seconds for optical devices. If we start settingrq_timeoutearlier, then the ULD will never know when it can set it.Additionally, in this particular scenario, its not so much a case of behavior tied to the LLDD, its more tied to the SCSI target. If there is concern about increasing the default to 30 seconds, we could use a blist attribute for this.
-Brian
8<
This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error injection test which results in paths going offline, when they came back online, the path would timeout the
REPORT_LUNSissued during the scan. This timeout situation continued until retries were expired, resulting in falling back to a sequential LUN scan. Then, since the target responds with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential LUN scan code works, we end up adding 512 LUNs for each target, when there is really only a small handful of LUNs that are actually present.This patch increases the timeout used on the
REPORT_LUNSto 30 seconds. This patch solves the issue of 512 non existent LUNs showing up after this event.Signed-off-by Brian King <[email protected]>---
drivers/scsi/scsi_scan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)diff -puN
drivers/scsi/scsi_scan.c~scsi_report_luns_30secsdrivers/scsi/scsi_scan.cdrivers/scsi/scsi_scan.c renamed from drivers/scsi/scsi_scan.c~scsi_report_luns_30secs +2
-1@@ -55,6 +55,7 @@* Default timeout */ #define SCSI_TIMEOUT (2*HZ) +#define SCSI_REPORT_LUNS_TIMEOUT (30*HZ)/* * Prefix values for the SCSI id's (stored in sysfs name field) @@ -1383,7 +1384,7 @@ retry: result =scsi_execute_req (sdev, scsi_cmd,DMA_FROM_DEVICE , lun_data, length, &sshdr, -SCSI_TIMEOUT +4 *HZ ,3 ,NULL ); +SCSI_REPORT_LUNS_TIMEOUT ,3 ,NULL ); SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev, "scsi scan: REPORT LUNS"
Quoted 3 lines
_
That's far better.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin K. Petersen
AppliedRe: [PATCH] SCSI: Increase REPORT_LUNS timeout
In reply to Brian KingQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Brian" == Brian King <[email protected]> writes:
Brian> This patch increases the timeout used on the REPORT_LUNS to 30 Brian> seconds. This patch solves the issue of 512 non existent LUNs Brian> showing up after this event.
Applied.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering