PROBLEM: special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

Hi James,

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted
Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan:
INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send:
0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12
00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180,
bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done:
0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12
00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not
Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense:
Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1
failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with
code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected

response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17 00:00:00 2001

Quoted 43 lines · Vaughan Cao

Hi James,

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan: INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send: 0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done: 0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense: Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

Quoted 4 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +1 -1

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct scsi_target *starget, int bflags,
 				lun, NULL, NULL, rescan, NULL);
 			if (res == SCSI_SCAN_NO_RESPONSE) {
 				/*
-				 * Got some results, but now none, abort.
+				 * Got some results, but now none, ignore.
 				 */

Hi Hannes,

Quoted 97 lines · Hannes Reinecke

Quoted 43 lines · Vaughan Cao

Hi James,

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan: INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send: 0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done: 0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense: Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

Quoted 4 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +1 -4

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct scsi_target *starget, int bflags,
 				lun, NULL, NULL, rescan, NULL);
 			if (res == SCSI_SCAN_NO_RESPONSE) {
 				/*
-				 * Got some results, but now none, abort.
+				 * Got some results, but now none, ignore.
 				 */
 				sdev_printk(KERN_ERR, sdev,
 					"Unexpected response"
-				        " from lun %d while scanning, scan"
-				        " aborted\n", lun);
-				break;

drivers/scsi/scsi_scan.c static int scsi_report_lun_scan(struct scsi_target *starget, int bflags, · line 1495

Quoted 5 lines

drivers/scsi/scsi_scan.c +2 -0

+					" from lun %d while scanning,"
+					" ignoring device\n", lun);
 			}
 		}
 	}

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this functionality to enable hardware virtualization without NPIV, or

masking on the target), they are likely to return -ENXIO from

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

through the same code path above.

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist (via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi lun (e.g. on remote port recovery), we see exactly above error

the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff [http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan(). Otherwise we would force the LLDD to return -ENXIO for every

explicitly added to the LLDD LUN whitelist; and this would likely

Maybe Vaughan's case needs to be distinguished in a patch.

Quoted 100 lines · Steffen Maier

Hi Hannes,

Quoted 97 lines · Hannes Reinecke

Quoted 43 lines · Vaughan Cao

Hi James,

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan: INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send: 0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done: 0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense: Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

Quoted 4 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +1 -4

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct scsi_target *starget, int bflags,
 				lun, NULL, NULL, rescan, NULL);
 			if (res == SCSI_SCAN_NO_RESPONSE) {
 				/*
-				 * Got some results, but now none, abort.
+				 * Got some results, but now none, ignore.
 				 */
 				sdev_printk(KERN_ERR, sdev,
 					"Unexpected response"
-				        " from lun %d while scanning, scan"
-				        " aborted\n", lun);
-				break;

Quoted 8 lines

drivers/scsi/scsi_scan.c static int scsi_report_lun_scan(struct scsi_target *starget, int bflags, · line 1495

Quoted 5 lines

drivers/scsi/scsi_scan.c +2 -0

+					" from lun %d while scanning,"
+					" ignoring device\n", lun);
 			}
 		}
 	}

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this functionality to enable hardware virtualization without NPIV, or

to work around suboptimal LUN

Quoted 1 line

masking on the target), they are likely to return -ENXIO from

slave_alloc(), making scsi_alloc_sdev()

Quoted 1 line

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

scsi_probe_and_add_lun() and thus going

Quoted 2 lines

through the same code path above.

Ah. Hmm. Yes, they would.

However, I personally would question this approach, as SPC states that

Quoted 3 lines

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

So by plain reading this would meant that you either should modify 'REPORT LUNS' to not show the masked LUNs, or set the pqual field to '0x10' or '0x11' for those LUNs.

Quoted 4 lines

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist (via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi lun (e.g. on remote port recovery), we see exactly above error

message for the first LUN in

Quoted 5 lines

the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff [http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan(). Otherwise we would force the LLDD to return -ENXIO for every

single LUN reported by report lun but not

Quoted 1 line

explicitly added to the LLDD LUN whitelist; and this would likely

*flood kernel messages*.

Quoted 3 lines

Maybe Vaughan's case needs to be distinguished in a patch.

Well, as mentioned initially, the real issue is that the target aborts an INQUIRY while being in 'Unavailable'. Which, according to SPC-3 (or later), is a violation of the spec.

So we _could_ just tell them to go away, but admittedly that's bad style. Which means we'll have to implement a workaround; the above was just a simple way of implementing it. If that's not working of course we'll have to do something else.

Quoted 116 lines · Hannes Reinecke

Quoted 96 lines · Steffen Maier

Quoted 95 lines · Hannes Reinecke

Quoted 41 lines · Vaughan Cao

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan: INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send: 0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done: 0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense: Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

Quoted 4 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +1 -4

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct scsi_target *starget, int bflags,
  				lun, NULL, NULL, rescan, NULL);
  			if (res == SCSI_SCAN_NO_RESPONSE) {
  				/*
-				 * Got some results, but now none, abort.
+				 * Got some results, but now none, ignore.
  				 */
  				sdev_printk(KERN_ERR, sdev,
  					"Unexpected response"
-				        " from lun %d while scanning, scan"
-				        " aborted\n", lun);
-				break;

Quoted 8 lines

drivers/scsi/scsi_scan.c static int scsi_report_lun_scan(struct scsi_target *starget, int bflags, · line 1495

Quoted 5 lines

drivers/scsi/scsi_scan.c +2 -0

+					" from lun %d while scanning,"
+					" ignoring device\n", lun);
  			}
  		}
  	}

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this functionality to enable hardware virtualization without NPIV, or

to work around suboptimal LUN

Quoted 1 line

masking on the target), they are likely to return -ENXIO from

slave_alloc(), making scsi_alloc_sdev()

Quoted 1 line

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

scsi_probe_and_add_lun() and thus going

Quoted 2 lines

through the same code path above.

Ah. Hmm. Yes, they would.

However, I personally would question this approach, as SPC states that

Quoted 7 lines

Quoted 3 lines

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

So by plain reading this would meant that you either should modify 'REPORT LUNS' to not show the masked LUNs, or set the pqual field to '0x10' or '0x11' for those LUNs.

Quoted 16 lines

Quoted 4 lines

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist (via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi lun (e.g. on remote port recovery), we see exactly above error

message for the first LUN in

Quoted 5 lines

the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff [http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan(). Otherwise we would force the LLDD to return -ENXIO for every

single LUN reported by report lun but not

Quoted 1 line

explicitly added to the LLDD LUN whitelist; and this would likely

*flood kernel messages*.

Quoted 3 lines

Maybe Vaughan's case needs to be distinguished in a patch.

Quoted 9 lines

Well, as mentioned initially, the real issue is that the target aborts an INQUIRY while being in 'Unavailable'. Which, according to SPC-3 (or later), is a violation of the spec.

So we _could_ just tell them to go away, but admittedly that's bad style. Which means we'll have to implement a workaround; the above was just a simple way of implementing it. If that's not working of course we'll have to do something else.

What about this patch:

drivers/scsi/scsi_scan.c +0 -0

index 973a121..01a7d69 100644
@@ -594,6 +594,19 @@ static int scsi_probe_lun(struct scsi_device
 *sdev, unsigne
 d char *inq_result,
                                      (sshdr.asc == 0x29)) &&
                                     (sshdr.ascq == 0))
                                         continue;

drivers/scsi/scsi_scan.c +3 -0

+                                   inq_result[0] = 3 << 5;
+                                   return 0;
+                               }
                         }

                 } else {
                         /*

(watchout, linebreaks mangled and all that).
Should be working for this particular case without interrupting normal workflow, now should it not?

Quoted 141 lines · Hannes Reinecke

Quoted 140 lines · Hannes Reinecke

Quoted 119 lines · Steffen Maier

Quoted 118 lines · Hannes Reinecke

Quoted 58 lines · Vaughan Cao

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7
is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted
Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan:
INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send:
0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB:
Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer =
0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done:
0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB:
Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key :
Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense:
Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host
busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes
done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed
with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected

response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result
of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan
process is aborted.

Quoted 5 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry
again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 4 lines

2. Since windows and solaris can continue scan, is it reasonable for
linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17
00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +2 -4

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct
 scsi_target *starget, int bflags,
                  lun, NULL, NULL, rescan, NULL);
              if (res == SCSI_SCAN_NO_RESPONSE) {
                  /*
-                 * Got some results, but now none, abort.
+                 * Got some results, but now none, ignore.
                   */
                  sdev_printk(KERN_ERR, sdev,
                      "Unexpected response"
-                        " from lun %d while scanning, scan"
-                        " aborted\n", lun);
-                break;
+                    " from lun %d while scanning,"

Quoted 8 lines

Quoted 4 lines

drivers/scsi/scsi_scan.c +1 -0

+                    " ignoring device\n", lun);
              }
          }

      }

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this
functionality to enable hardware virtualization without NPIV, or

to work around suboptimal LUN

Quoted 1 line

masking on the target), they are likely to return -ENXIO from

slave_alloc(), making scsi_alloc_sdev()

Quoted 1 line

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

scsi_probe_and_add_lun() and thus going

Quoted 2 lines

through the same code path above.

Ah. Hmm. Yes, they would.

However, I personally would question this approach, as SPC states that

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

So by plain reading this would meant that you either should modify 'REPORT LUNS' to not show the masked LUNs, or set the pqual field to '0x10' or '0x11' for those LUNs.

We need to distinguish two cases:
1) suboptimal lun masking on the target
2) hardware virtualization without NPIV

Regarding 1, one could require fixing lun masking on the target. However, some users cannot or do not want to do it very fine granular. That's why s390 also does deferred device probing ("set online" in sysfs) or even limits bus sensing (cio_ignore).

Regarding 2, fixing lun masking on the target does not help because without NPIV, the target cannot distinguish the different virtual initators since they are all behind one shared WWPN (and N-Port_ID). This forces zfcp to implement initiator based lun masking, because only the user can tell which lun to attach to which of the virtual initiators sharing the same physical port. Without that, Linux would attach all luns to all virtual initiators, i.e. share inadvertently.

Quoted 46 lines

Quoted 21 lines

Quoted 6 lines

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist
(via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi
lun (e.g. on remote port recovery), we see exactly above error

message for the first LUN in

Quoted 8 lines

the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff
[http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan().
Otherwise we would force the LLDD to return -ENXIO for every

single LUN reported by report lun but not

Quoted 1 line

explicitly added to the LLDD LUN whitelist; and this would likely

*flood kernel messages*.

Quoted 3 lines

Maybe Vaughan's case needs to be distinguished in a patch.

Quoted 9 lines

Well, as mentioned initially, the real issue is that the target aborts an INQUIRY while being in 'Unavailable'. Which, according to SPC-3 (or later), is a violation of the spec.

So we _could_ just tell them to go away, but admittedly that's bad style. Which means we'll have to implement a workaround; the above was just a simple way of implementing it. If that's not working of course we'll have to do something else.

What about this patch:

drivers/scsi/scsi_scan.c +4 -0

index 973a121..01a7d69 100644
@@ -594,6 +594,19 @@ static int scsi_probe_lun(struct scsi_device
 *sdev, unsigne
 d char *inq_result,
                                      (sshdr.asc == 0x29)) &&
                                     (sshdr.ascq == 0))
                                         continue;
+                               /*
+                                * Some buggy implementations return
+                                * 'target port in unavailable state'
+                                * even on INQUIRY.

drivers/scsi/scsi_scan.c static int scsi_probe_lun(struct scsi_device · line 608

Quoted 6 lines

drivers/scsi/scsi_scan.c +6 -0

+                                * Set peripheral qualifier 3
+                                * for these devices.
+                                */
+                               if ((sshdr.sense_key == NOT_READY) &&
+                                   ((sshdr.asc == 0x04) &&
+                                    (sshdr.ascq == 0x0C))) {

style question: lower case hex digits? 0x0c

Any reason why you put the conjunction of asc and ascq inside its own brackets instead of having all three (including sense_key) on the same level of one larger conjunction (as the code above does for UA asc 0x28/0x29 ascq 0x00)? Should be semantically equivalent, isn't it? But then again, ascq always goes with asc, so they form a kind of pair.

Quoted 10 lines

drivers/scsi/scsi_scan.c +3 -0

+                                   inq_result[0] = 3 << 5;
+                                   return 0;
+                               }
                         }

                 } else {
                         /*

(watchout, linebreaks mangled and all that).
Should be working for this particular case without interrupting normal workflow, now should it not?

The approach of distinguishing the workaround close to the response of the inquiry sounds good to me. I suppose it won't break zfcp which is good. Unfortunately, I don't know what the ramifications of PQ==3 are (the SPC-4 description sounds good, though), nor enough details about this common code to tell if e.g. the early return is OK (skipping setting sdev->scsi_level near the end of scsi_probe_lun()). But then again, without inquiry reply we cannot get the level from the response. So I think the early return is OK after all.
I guess we want to get around "if (result) return -EIO;" but also do not want to execute the parts depending on result==0.

SPC-4 says that for PQ==3 the PDT should be set to 0x1f. Do we need to fake this here as well? (I assume the target did not fill in a PDT on its own when replying with sense data.)

The clarification on the T10 reflector seems to say that Linux would then accept LUNs with PQ 3, but the target shall not have put LUs with PQ 3 into the LU inventory in the first place? Anyway, I'm not opposed to the workaround.

From 12a949d293e698960984225cdd69cd68d1bdecbc Mon Sep 17 00:00:00 2001

But seeing that this approach raises quite some issues I've attached a different patch. Vaughan, could you test with that, too? Should be functionally equivalent to the previous one. Cheers, Hannes

But seeing that this approach raises quite some issues I've attached a different patch. Vaughan, could you test with that, too? Should be functionally equivalent to the previous one. Cheers, Hannes

Quoted 3 lines · Hannes Reinecke

But seeing that this approach raises quite some issues I've attached a different patch. Vaughan, could you test with that, too? Should be functionally equivalent to the previous one. Cheers, Hannes

Hi Hannes,

We only tested the later patch which returns _TARGET_PRESENT after parsing sense, it works as expected.

About the cause of this issue, admin said he is configuring a active-active cluster mode storage. Each node has it own LUN pool and a set of rule to control which node can access the pool.
LUN7 is owned and can only be able to manipulated by the other node, but can be seen by this node for a misconfig. So it presents itself in REPORT_LUN but return NOT_READY when accessed through this node. Do you still regard this as a misbehave in response to INQUIRY?

Quoted 3 lines · Hannes Reinecke

But seeing that this approach raises quite some issues I've attached a different patch. Vaughan, could you test with that, too? Should be functionally equivalent to the previous one. Cheers, Hannes

Hi Hannes,

We only tested the later patch which returns _TARGET_PRESENT after parsing sense, it works as expected.

Quoted 207 lines · Steffen Maier

Quoted 139 lines · Hannes Reinecke

Quoted 138 lines · Hannes Reinecke

Quoted 117 lines · Steffen Maier

Quoted 116 lines · Hannes Reinecke

Quoted 58 lines · Vaughan Cao

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7
is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted
Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan:
INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send:
0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB:
Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer =
0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done:
0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB:
Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key :
Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense:
Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host
busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes
done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed
with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected

response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result
of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan
process is aborted.

Quoted 5 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry
again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 4 lines

2. Since windows and solaris can continue scan, is it reasonable for
linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

From b0e90778f012010c881f8bdc03bce63a36921b77 Mon Sep 17
00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 14 Oct 2013 13:11:22 +0200
Subject: [PATCH] scsi_scan: continue report_lun_scan after error

When scsi_probe_and_add_lun() fails in scsi_report_lun_scan() this does _not_ indicate that the entire target is done for. So continue scanning for the remaining devices.

Signed-off-by Hannes Reinecke <[email protected]>

drivers/scsi/scsi_scan.c +1 -4

index 307a811..973a121 100644
@@ -1484,13 +1484,12 @@ static int scsi_report_lun_scan(struct
 scsi_target *starget, int bflags,
                  lun, NULL, NULL, rescan, NULL);
              if (res == SCSI_SCAN_NO_RESPONSE) {
                  /*
-                 * Got some results, but now none, abort.
+                 * Got some results, but now none, ignore.
                   */
                  sdev_printk(KERN_ERR, sdev,
                      "Unexpected response"
-                        " from lun %d while scanning, scan"
-                        " aborted\n", lun);
-                break;

Quoted 8 lines

Quoted 5 lines

drivers/scsi/scsi_scan.c +2 -0

+                    " from lun %d while scanning,"
+                    " ignoring device\n", lun);
              }
          }

      }

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this
functionality to enable hardware virtualization without NPIV, or

to work around suboptimal LUN

Quoted 1 line

masking on the target), they are likely to return -ENXIO from

slave_alloc(), making scsi_alloc_sdev()

Quoted 1 line

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

scsi_probe_and_add_lun() and thus going

Quoted 2 lines

through the same code path above.

Ah. Hmm. Yes, they would.

However, I personally would question this approach, as SPC states that

Quoted 6 lines

Quoted 6 lines

Quoted 3 lines

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

So by plain reading this would meant that you either should modify 'REPORT LUNS' to not show the masked LUNs, or set the pqual field to '0x10' or '0x11' for those LUNs.

We need to distinguish two cases:
1) suboptimal lun masking on the target
2) hardware virtualization without NPIV

Regarding 1, one could require fixing lun masking on the target. However, some users cannot or do not want to do it very fine granular. That's why s390 also does deferred device probing ("set online" in sysfs) or even limits bus sensing (cio_ignore).

Regarding 2, fixing lun masking on the target does not help because without NPIV, the target cannot distinguish the different virtual initators since they are all behind one shared WWPN (and N-Port_ID). This forces zfcp to implement initiator based lun masking, because only the user can tell which lun to attach to which of the virtual initiators sharing the same physical port. Without that, Linux would attach all luns to all virtual initiators, i.e. share inadvertently.

Quoted 44 lines

Quoted 20 lines

Quoted 6 lines

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist
(via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi
lun (e.g. on remote port recovery), we see exactly above error

message for the first LUN in

Quoted 8 lines

the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff
[http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan().
Otherwise we would force the LLDD to return -ENXIO for every

single LUN reported by report lun but not

Quoted 1 line

explicitly added to the LLDD LUN whitelist; and this would likely

*flood kernel messages*.

Quoted 2 lines

Maybe Vaughan's case needs to be distinguished in a patch.

Quoted 9 lines

Well, as mentioned initially, the real issue is that the target aborts an INQUIRY while being in 'Unavailable'. Which, according to SPC-3 (or later), is a violation of the spec.

So we _could_ just tell them to go away, but admittedly that's bad style. Which means we'll have to implement a workaround; the above was just a simple way of implementing it. If that's not working of course we'll have to do something else.

What about this patch:

drivers/scsi/scsi_scan.c +3 -0

index 973a121..01a7d69 100644
@@ -594,6 +594,19 @@ static int scsi_probe_lun(struct scsi_device
 *sdev, unsigne
 d char *inq_result,
                                      (sshdr.asc == 0x29)) &&
                                     (sshdr.ascq == 0))
                                         continue;
+                               /*
+                                * Some buggy implementations return
+                                * 'target port in unavailable state'

Quoted 9 lines

drivers/scsi/scsi_scan.c static int scsi_probe_lun(struct scsi_device · line 608

Quoted 7 lines

drivers/scsi/scsi_scan.c +7 -0

+                                * even on INQUIRY.
+                                * Set peripheral qualifier 3
+                                * for these devices.
+                                */
+                               if ((sshdr.sense_key == NOT_READY) &&
+                                   ((sshdr.asc == 0x04) &&
+                                    (sshdr.ascq == 0x0C))) {

style question: lower case hex digits? 0x0c

Yeah. This is a test, after all ...

Quoted 7 lines

Any reason why you put the conjunction of asc and ascq inside its own brackets instead of having all three (including sense_key) on the same level of one larger conjunction (as the code above does for UA asc 0x28/0x29 ascq 0x00)? Should be semantically equivalent, isn't it? But then again, ascq always goes with asc, so they form a kind of pair.

No reason, Just copy&paste error from the above statement ...

Quoted 22 lines

Quoted 10 lines

drivers/scsi/scsi_scan.c +3 -0

+                                   inq_result[0] = 3 << 5;
+                                   return 0;
+                               }
                         }

                 } else {
                         /*

(watchout, linebreaks mangled and all that).
Should be working for this particular case without interrupting normal workflow, now should it not?

The approach of distinguishing the workaround close to the response of the inquiry sounds good to me. I suppose it won't break zfcp which is good. Unfortunately, I don't know what the ramifications of PQ==3 are (the SPC-4 description sounds good, though), nor enough details about this common code to tell if e.g. the early return is OK (skipping setting sdev->scsi_level near the end of scsi_probe_lun()). But then again, without inquiry reply we cannot get the level from the response. So I think the early return is OK after all.
I guess we want to get around "if (result) return -EIO;" but also do not want to execute the parts depending on result==0.

Quoted 9 lines

SPC-4 says that for PQ==3 the PDT should be set to 0x1f. Do we need to fake this here as well? (I assume the target did not fill in a PDT on its own when replying with sense data.)

The clarification on the T10 reflector seems to say that Linux would then accept LUNs with PQ 3, but the target shall not have put LUs with PQ 3 into the LU inventory in the first place? Anyway, I'm not opposed to the workaround.

Well, first and foremost this is a workaround for buggy array firmware. If any port would be in 'unavailable' the target port is still required to respond to an INQUIRY.
_Not_ doing so leaves us with no indication what's going on here.

The main reason why I chose PQ=3 here is that we'll end up ignoring this device scsi_probe_and_add_lun() later on.
Saving my coding higher up the stack.
And, seeing that the device is never actually allocated, the modifications we did for the inquiry data will be deleted anyway.

So using PQ=3 here is just a vehicle for telling the system to not create a SCSI device at this LUN, _not_ something which has some relevance to SPC.

But seeing that this approach raises quite some issues I've attached a different patch.
Vaughan, could you test with that, too? Should be functionally equivalent to the previous one.

Cheers,

Hannes

Quoted 66 lines · Steffen Maier

Hi Hannes,

Quoted 63 lines · Hannes Reinecke

Quoted 43 lines · Vaughan Cao

Hi James,

[1.] One line summary of the problem:
special sense code asc,ascq=04h,0Ch abort scsi scan in the middle

[2.] Full description of the problem/report:
For instance, storage represents 8 iscsi LUNs, however the LUN No.7 is not well configured or has something wrong.
Then messages received:
kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted Which will make LUN No.8 unavailable.
It's confirmed that Windows and Solaris systems will continue the scan and make LUN No.1,2,3,4,5,6 and 8 available.

Log snippet is as below:

Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi scan: INQUIRY pass 1 length 36
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Send: 0xffff8801e9bd4280
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: buffer = 0xffff8801f71fc180, bufflen = 36, queuecommand 0xffffffffa00b99e7
Aug 24 00:32:49 vmhodtest019 kernel: leaving scsi_dispatch_cmnd()
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Done: 0xffff8801e9bd4280 SUCCESS
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: CDB: Inquiry: 12 00 00 00 24 00
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Sense Key : Not Ready [current]
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: Add. Sense: Logical unit not accessible, target port in unavailable state
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:7: scsi host busy 1 failed 0
Aug 24 00:32:49 vmhodtest019 kernel: 0 sectors total, 36 bytes done.
Aug 24 00:32:49 vmhodtest019 kernel: scsi scan: INQUIRY failed with code 0x8000002
Aug 24 00:32:49 vmhodtest019 kernel: scsi 5:0:0:0: Unexpected response from lun 7 while scanning, scan aborted

According to scsi_report_lun_scan(), I found:
Linux use an inquiry command to probe a lun according to the result of report_lun command.
It assumes every probe cmd will get a legal result. Otherwise, it regards the whole peripheral not exist or dead.
If the return of inquiry passes its legal checking and indicates 'LUN not present', it won't break but also continue with the scan process.
In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

Quoted 4 lines

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?

Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.
But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Quoted 8 lines

Quoted 6 lines

Please try the attached patch.

Cheers,

Hannes

In LLDDs that do their own initiator based LUN masking (because the midlayer does not have this functionality to enable hardware virtualization without NPIV, or

to work around suboptimal LUN

Quoted 1 line

masking on the target), they are likely to return -ENXIO from

slave_alloc(), making scsi_alloc_sdev()

Quoted 1 line

return NULL, being converted to SCSI_SCAN_NO_RESPONSE by

scsi_probe_and_add_lun() and thus going

Quoted 2 lines

through the same code path above.

Ah. Hmm. Yes, they would.

However, I personally would question this approach, as SPC states that

Quoted 3 lines

The REPORT LUNS command (see table 284) requests the device server to return the peripheral device logical unit inventory accessible to the I_T nexus.

So by plain reading this would meant that you either should modify 'REPORT LUNS' to not show the masked LUNs,

or set the pqual field to
'0x10' or '0x11' for those LUNs.

Quoted 10 lines

E.g. zfcp does return -ENXIO if the particular LUN was not made known to the unit whitelist (via zfcp sysfs attribute unit_add).
If we attach LUN 0 (via unit_add) and trigger a target scan with SCAN_WILD_CARD for the scsi lun (e.g. on remote port recovery), we see exactly above error message for the first LUN in the response of report lun which is not explicitly attached to zfcp.
IIRC, other LLDDs such as bfa also do similar stuff [http://marc.info/?l=linux-scsi&m=134489842105383&w=2].

For those cases, I think it makes sense to abort scsi_report_lun_scan().
Otherwise we would force the LLDD to return -ENXIO for every single LUN reported by report lun but not explicitly added to the LLDD LUN whitelist; and this would likely *flood kernel messages*.

Quoted 2 lines

Maybe Vaughan's case needs to be distinguished in a patch.

Well, as mentioned initially, the real issue is that the target aborts an INQUIRY while being in 'Unavailable'. Which, according to SPC-3 (or later), is a violation of the spec.

So we _could_ just tell them to go away, but admittedly that's bad style. Which means we'll have to implement a workaround; the above was just a simple way of implementing it. If that's not working of course we'll have to do something else.

Cheers,

Hannes

In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?
Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.

Quoted 1 line

Quoted 1 line

From the above, I suppose the hardware may works very compliant with

But yeah, we probably should work around this issues. Nevertheless, please raise this issue with your array vendor.

Please try the attached patch.

Cheers,

Hannes

Quoted 14 lines · Hannes Reinecke

In the log, inquiry to LUN7 return a sense - asc,ascq=04h,0Ch (Logical unit not accessible, target port in unavailable state). And this is ignored, so scsi_probe_lun() returns -EIO and the scan process is aborted.

I have two questions:
1. Is it correct for hardware to return a sense 04h,0Ch to inquiry again, even after presenting this lun in responce to REPORT_LUN command?
Yes, this is correct. 'REPORT LUNS' is supported in 'Unavailable' state.

Quoted 3 lines

2. Since windows and solaris can continue scan, is it reasonable for linux to do the same, even for a fault-tolerance purpose?

Quoted 7 lines

Hmm. Yes, and no.

_Actually_ this is an issue with the target, as it looks as if it will return the above sense code while sending an 'INQUIRY' to the device.
SPC explicitely states that the INQUIRY command should _not_ fail for unavailable devices.

Hi all,

I found this below in spc4.

Quoted 1 line

Quoted 1 line

Quoted 1 line

5.15.2.4.4 Target port group asymmetric access states - Standby state While in the unavailable primary target port asymmetric access state, the device server shall support those of
the following commands that it supports while in the active/optimized state:
a) INQUIRY (the peripheral qualifier (see 6.6.2) shall be set to 001b);
....
For those commands that are not supported, the device server shall terminate the command with CHECK
CONDITION status, with the sense key set to NOT READY, and the additional sense code set to LOGICAL
UNIT NOT ACCESSIBLE, TARGET PORT IN UNAVAILABLE STATE.
<<<
From the above, I suppose the hardware may works very compliant with spc. The case could be:
Storage is a alua supported target. Initiator sent REPORT_LUN to target, target return all pqual=000b to it.
Then Initiator INQUIRY lun 7 which is in standby state where pqual=000b not 001b. So this INQUIRY is regarded as
'not supported', and get terminated with CHECK_CONDITION, sense key=NOT READY, asc,ascq=04h,0ch.

Could you confirm if my understanding is right or wrong?

But that notwithstanding, did you get a chance to test my patch?

Cheers,

Hannes