SCSI referrals support

LSF/MM TOPIC Hannes Reinecke 6 participants 9 messages Jan 20, 2011 – Mar 31, 2011 SCSI

Share & tools

Hannes Reinecke

author

2011-01-20 15:53 UTC #0 lore raw

Agenda topic proposal:

SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.

To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling

I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.

Cheers,

Hannes

Signature · 8 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Douglas Gilbert

2011-01-20 16:14 UTC #1 lore raw

Quoted 12 lines · Hannes Reinecke

Agenda topic proposal:
SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.
To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling

Quoted 4 lines

I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.

IMO a worrying aspect of the changes associated with SCSI referrals is that sense data can now be returned with any SCSI status (i.e. not just CHECK CONDITION). How well would the SCSI subsystem cope with that? I know that the sg driver (and probably bsg) would need changes, as would my libsgutils library used by sg3_utils.

Doug Gilbert

James Bottomley

2011-01-20 16:36 UTC #2 lore raw

Quoted 18 lines · Douglas Gilbert

Quoted 12 lines · Hannes Reinecke
Agenda topic proposal:
SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.
To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling
Quoted 4 lines
I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.

Quoted 6 lines

IMO a worrying aspect of the changes associated with SCSI referrals is that sense data can now be returned with any SCSI status (i.e. not just CHECK CONDITION). How well would the SCSI subsystem cope with that? I know that the sg driver (and probably bsg) would need changes, as would my libsgutils library used by sg3_utils.

Well, not necessarily. It scuppers plans to try to use the sense buffer more efficiently, but right at the moment, we can receive sense for any command. We only actually look at it on a check condition return, but that's easily updated.

As I read the standard, referrals sense data on successfully completed commands is only really relevant to mp implementations, so it looks like the mid-layer should ignore it anyway (as it would now) but provide a hook for the device handlers to see it if they wanted.

James

Boaz Harrosh

2011-01-23 11:25 UTC #3 lore raw

Quoted 31 lines · James Bottomley

Quoted 18 lines · Douglas Gilbert
Quoted 12 lines · Hannes Reinecke
Agenda topic proposal:
SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.
To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling
Quoted 4 lines
I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.
Quoted 6 lines
IMO a worrying aspect of the changes associated with SCSI referrals is that sense data can now be returned with any SCSI status (i.e. not just CHECK CONDITION). How well would the SCSI subsystem cope with that? I know that the sg driver (and probably bsg) would need changes, as would my libsgutils library used by sg3_utils.
Well, not necessarily. It scuppers plans to try to use the sense buffer more efficiently, but right at the moment, we can receive sense for any command. We only actually look at it on a check condition return, but that's easily updated.

Quoted 7 lines

As I read the standard, referrals sense data on successfully completed commands is only really relevant to mp implementations, so it looks like the mid-layer should ignore it anyway (as it would now) but provide a hook for the device handlers to see it if they wanted.
James

Something like:

drivers/scsi/scsi_lib.c +14 ~~-11~~

index eafeeda..454e562 100644
@@ -722,20 +722,23 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 			sense_deferred = scsi_sense_is_deferred(&sshdr);
 	}
 
+	if (req->sense) {
+		/*
+		 * SG_IO wants current and deferred errors
+		 */
+		int len = 8 + cmd->sense_buffer[7];
+
+		if (unlikely(len)) {
+			if (len > SCSI_SENSE_BUFFERSIZE)
+				len = SCSI_SENSE_BUFFERSIZE;
+			memcpy(req->sense, cmd->sense_buffer,  len);
+			req->sense_len = len;
+		}
+	}
+
 	if (req->cmd_type == REQ_TYPE_BLOCK_PC) { /* SG_IO ioctl from block level */
 		req->errors = result;
 		if (result) {
-			if (sense_valid && req->sense) {
-				/*
-				 * SG_IO wants current and deferred errors
-				 */
-				int len = 8 + cmd->sense_buffer[7];
-
-				if (len > SCSI_SENSE_BUFFERSIZE)
-					len = SCSI_SENSE_BUFFERSIZE;
-				memcpy(req->sense, cmd->sense_buffer,  len);
-				req->sense_len = len;
-			}
 			if (!sense_deferred)
 				error = -EIO;
 		}

Then an interested user can just look at req->sense_len. The reset of the stack will ignore all that because it looks for status/error-code.

I was needing something like that as well but it's to do with advance storage maintenance, that's far down the road.

Thanks
Boaz

Hannes Reinecke

author

2011-01-24 07:57 UTC #4 lore raw

Quoted 80 lines · Boaz Harrosh

Quoted 31 lines · James Bottomley
Quoted 18 lines · Douglas Gilbert
Quoted 12 lines · Hannes Reinecke
Agenda topic proposal:
SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.
To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling
Quoted 4 lines
I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.
Quoted 6 lines
IMO a worrying aspect of the changes associated with SCSI referrals is that sense data can now be returned with any SCSI status (i.e. not just CHECK CONDITION). How well would the SCSI subsystem cope with that? I know that the sg driver (and probably bsg) would need changes, as would my libsgutils library used by sg3_utils.
Well, not necessarily. It scuppers plans to try to use the sense buffer more efficiently, but right at the moment, we can receive sense for any command. We only actually look at it on a check condition return, but that's easily updated.
Quoted 7 lines
As I read the standard, referrals sense data on successfully completed commands is only really relevant to mp implementations, so it looks like the mid-layer should ignore it anyway (as it would now) but provide a hook for the device handlers to see it if they wanted.
James
Something like:
drivers/scsi/scsi_lib.c +14 ~~-11~~
index eafeeda..454e562 100644
@@ -722,20 +722,23 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 			sense_deferred = scsi_sense_is_deferred(&sshdr);
 	}
 
+	if (req->sense) {
+		/*
+		 * SG_IO wants current and deferred errors
+		 */
+		int len = 8 + cmd->sense_buffer[7];
+
+		if (unlikely(len)) {
+			if (len > SCSI_SENSE_BUFFERSIZE)
+				len = SCSI_SENSE_BUFFERSIZE;
+			memcpy(req->sense, cmd->sense_buffer,  len);
+			req->sense_len = len;
+		}
+	}
+
 	if (req->cmd_type == REQ_TYPE_BLOCK_PC) { /* SG_IO ioctl from block level */
 		req->errors = result;
 		if (result) {
-			if (sense_valid && req->sense) {
-				/*
-				 * SG_IO wants current and deferred errors
-				 */
-				int len = 8 + cmd->sense_buffer[7];
-
-				if (len > SCSI_SENSE_BUFFERSIZE)
-					len = SCSI_SENSE_BUFFERSIZE;
-				memcpy(req->sense, cmd->sense_buffer,  len);
-				req->sense_len = len;
-			}
 			if (!sense_deferred)
 				error = -EIO;
 		}

Quoted 3 lines

Then an interested user can just look at req->sense_len. The reset of the stack will ignore all that because it looks for status/error-code.

Ah, if it were so easy. Currently sense codes have two problems:

- They are limited to 96 bytes. Anything larger than that will just be discarded (or crash with your patch above :-)
- They inherit the same lifetime than the scsi command. But for any decent handling you really need to push them into some asynchronous context as you might well be within an interrupt handler here.

I'm currently working on a handling framework using relayfs (basically blktrace for SCSI Unit Attention); I can be doing a short presentation at LSF if requested.

Cheers,

Hannes

Signature · 8 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Boaz Harrosh

2011-01-24 09:17 UTC #5 lore raw

Quoted 49 lines · Hannes Reinecke

Quoted 48 lines · Boaz Harrosh
Quoted 31 lines · James Bottomley
Quoted 18 lines · Douglas Gilbert
Quoted 12 lines · Hannes Reinecke
Agenda topic proposal:
SCSI referrals support has already been discussed at last year's LSF conference. However, the solution proposed there would not support failover and would require quite a lot of changes to multipathing.
To enable failover it might be an idea to handle the LUN directly in multipathing. This would require eg:
- request splitting
- I/O alignment handling
- SCSI unit attention handling
Quoted 4 lines
I would be giving a short overview/presentation of the current state of the art, the shortcomings on the original proposal, and would like to invite a discussion on how to best support SCSI referrals.
Quoted 6 lines
IMO a worrying aspect of the changes associated with SCSI referrals is that sense data can now be returned with any SCSI status (i.e. not just CHECK CONDITION). How well would the SCSI subsystem cope with that? I know that the sg driver (and probably bsg) would need changes, as would my libsgutils library used by sg3_utils.
Well, not necessarily. It scuppers plans to try to use the sense buffer more efficiently, but right at the moment, we can receive sense for any command. We only actually look at it on a check condition return, but that's easily updated.
Quoted 7 lines
As I read the standard, referrals sense data on successfully completed commands is only really relevant to mp implementations, so it looks like the mid-layer should ignore it anyway (as it would now) but provide a hook for the device handlers to see it if they wanted.
James
Something like:
drivers/scsi/scsi_lib.c +0 -0
index eafeeda..454e562 100644
@@ -722,20 +722,23 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 			sense_deferred = scsi_sense_is_deferred(&sshdr);
 	}
 

Quoted 9 lines

drivers/scsi/scsi_lib.c void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) · line 733
Quoted 9 lines
drivers/scsi/scsi_lib.c +9 -0
+	if (req->sense) {
+		/*
+		 * SG_IO wants current and deferred errors
+		 */
+		int len = 8 + cmd->sense_buffer[7];
+
+		if (unlikely(len)) {
+			if (len > SCSI_SENSE_BUFFERSIZE)
+				len = SCSI_SENSE_BUFFERSIZE;

See here it will not crash

Quoted 23 lines

drivers/scsi/scsi_lib.c +5 ~~-11~~

+			memcpy(req->sense, cmd->sense_buffer,  len);
+			req->sense_len = len;
+		}
+	}
+
 	if (req->cmd_type == REQ_TYPE_BLOCK_PC) { /* SG_IO ioctl from block level */
 		req->errors = result;
 		if (result) {
-			if (sense_valid && req->sense) {
-				/*
-				 * SG_IO wants current and deferred errors
-				 */
-				int len = 8 + cmd->sense_buffer[7];
-
-				if (len > SCSI_SENSE_BUFFERSIZE)
-					len = SCSI_SENSE_BUFFERSIZE;
-				memcpy(req->sense, cmd->sense_buffer,  len);
-				req->sense_len = len;
-			}
 			if (!sense_deferred)
 				error = -EIO;
 		}

Quoted 7 lines

Quoted 3 lines
Then an interested user can just look at req->sense_len. The reset of the stack will ignore all that because it looks for status/error-code.
Ah, if it were so easy. Currently sense codes have two problems:
- They are limited to 96 bytes. Anything larger than that will just be discarded (or crash with your patch above :-)

It will not crash.

I have a patchset here that expands the request/scsi_cmnd sense_buffer support to the maximum 260 bytes supported by the std.

If you want I can revive it. It is a sweep of all drivers, but a small one, nothing like the accessors changes.

Quoted 4 lines

- They inherit the same lifetime than the scsi command. But for any decent handling you really need to push them into some asynchronous context as you might well be within an interrupt handler here.

No that's fine. request->sense is a user supplied buffer that extends the life of scsi_cmnd. above code just copies from a scsi-layer dma-able buffer to the request->sense buffer. interrupt-time is fine.

Quoted 4 lines

I'm currently working on a handling framework using relayfs (basically blktrace for SCSI Unit Attention); I can be doing a short presentation at LSF if requested.

That would be interesting. Thanks

Quoted 3 lines

Cheers,
Hannes

Boaz

Hannes Reinecke

author

2011-01-26 15:47 UTC #6 lore raw

Quoted 1 line · Boaz Harrosh

On 01/24/2011 10:04 AM, Hannes Reinecke wrote:

[ .. ]

Quoted 7 lines

Quoted 4 lines
Ah, if it were so easy. Currently sense codes have two problems:
- They are limited to 96 bytes. Anything larger than that will just be discarded (or crash with your patch above :-)
It will not crash.

Oh, indeed. You are right.

Quoted 6 lines

I have a patchset here that expands the request/scsi_cmnd sense_buffer support to the maximum 260 bytes supported by the std.
If you want I can revive it. It is a sweep of all drivers, but a small one, nothing like the accessors changes.

Please. Can you send it to me?
I have a prototype working right now, and expanding the sense buffer would be a good idea here.

Quoted 9 lines

Quoted 4 lines
- They inherit the same lifetime than the scsi command. But for any decent handling you really need to push them into some asynchronous context as you might well be within an interrupt handler here.
No that's fine. request->sense is a user supplied buffer that extends the life of scsi_cmnd. above code just copies from a scsi-layer dma-able buffer to the request->sense buffer. interrupt-time is fine.

Quoted 7 lines

Quoted 4 lines
I'm currently working on a handling framework using relayfs (basically blktrace for SCSI Unit Attention); I can be doing a short presentation at LSF if requested.
That would be interesting. Thanks

Ok, will then be doing it.

Cheers,

Hannes

Signature · 8 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bart Van Assche

2011-03-31 16:40 UTC #7 lore raw

In reply to Hannes Reinecke

Quoted 10 lines · Hannes Reinecke

Currently sense codes have two problems:
- They are limited to 96 bytes. Anything larger than that will just be discarded (or crash with your patch above :-)
- They inherit the same lifetime than the scsi command. But for any decent handling you really need to push them into some asynchronous context as you might well be within an interrupt handler here.

Quoted 3 lines

I'm currently working on a handling framework using relayfs (basically blktrace for SCSI Unit Attention); I can be doing a short presentation at LSF if requested.

(replying to an e-mail of two months ago)

There is another scenario where better unit attention support would help. Both the LIO-port and the SCST-port of the ibmvstgt driver suffer from the same race condition during reboot of the virtual I/O server (VIOS). Since the ibmvscsic driver reconnects to the ibmvscsis driver as soon as the ibmvscsis driver has been loaded the client reconnects before all LUNs have been configured in the VIOS system. So even if the target implementation generates an INQUIRY DATA HAS CHANGED unit attention condition for each LUN change, LUNs will have to be rescanned explicitly at the initiator side in order to get the initiator state in sync with the target state.

Also, when using iSCSI, it would beneficial if LUN changes at the target would result in an automatic update of the LUNs reported by the initiator. This is not yet the case when using a Linux iSCSI initiator because the INQUIRY DATA HAS CHANGED unit attention condition is ignored by the Linux SCSI stack.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Brian King

2011-03-31 17:56 UTC #8 lore raw

Quoted 17 lines · Bart Van Assche

Quoted 10 lines · Hannes Reinecke
Currently sense codes have two problems:
- They are limited to 96 bytes. Anything larger than that will just be discarded (or crash with your patch above :-)
- They inherit the same lifetime than the scsi command. But for any decent handling you really need to push them into some asynchronous context as you might well be within an interrupt handler here.
Quoted 3 lines
I'm currently working on a handling framework using relayfs (basically blktrace for SCSI Unit Attention); I can be doing a short presentation at LSF if requested.
(replying to an e-mail of two months ago)

Quoted 10 lines

There is another scenario where better unit attention support would help. Both the LIO-port and the SCST-port of the ibmvstgt driver suffer from the same race condition during reboot of the virtual I/O server (VIOS). Since the ibmvscsic driver reconnects to the ibmvscsis driver as soon as the ibmvscsis driver has been loaded the client reconnects before all LUNs have been configured in the VIOS system. So even if the target implementation generates an INQUIRY DATA HAS CHANGED unit attention condition for each LUN change, LUNs will have to be rescanned explicitly at the initiator side in order to get the initiator state in sync with the target state.

This sounds to me like the vscsi server drivers really need to change to only register their CRQ and not initialize their CRQ until userspace has setup all the target devices.

-Brian

Signature · 3 lines

Brian King
Linux on Power Virtualization
IBM Linux Technology Center