disk restart failure after suspend

[Repost with corrected CCs, sorry for the mess.
Problem: FireWire disk becomes inaccessible during resume because START STOP UNIT failed. http://marc.info/?t=125481515600002]

On 2009-10-16, Tino Keitel wrote at linux1394-user:

It is not a bug. IEEE 1394 rediscovery and SBP-2 reconnect can become necessary anytime (and they do become necessary at /least/ once during PM resume), in no particular order with respect to SCSI request submission. Our drivers (firewire-sbp2 mainly) need to be able to handle any order of such events.

Interesting findings.

There are two independent places of the code that could possibly be improved to fix this issue:

a.) sd's PM resume method:

1.a) sd_resume could gain this retry loop which you implemented.

1.b) sd_resume (but probably not sd_suspend) could optimistically ignore any error return from sd_start_stop_device. If the motor cannot be started immediately at resume, the SCSI core would try to start it later on when the disk is normally accessed.

My assumption here is that an error return from sd_resume causes the disk to become inaccessible (taken offline?).

2.) firewire-sbp2's bus reset handling scheme (the reconnect thing):

The originally submitted incarnation of firewire-sbp2 had very weak bus reset handling which lost contact to disks very easily. I then ported over drivers/ieee1394/sbp2.c's bus reset handling to drivers/firewire although I was not satisfied with that implementation anymore either. This scheme uses the SCSI core's host block/ unblock API to prevent queuing of new commands after firewire-sbp2 detected that a reconnect becomes necessary, until reconnect succeeded.

After reconnect, already pending requests (at most one request at the moment because we currently don't support queue depth > 1 in firewire-sbp2) will be aborted and the SCSI request completed with DID_BUS_BUSY. And this is what apparently happens in your case:
sd_resume issues START STOP UNIT, sbp2 reconnects at the earliest opportunity but alas after sd's request went out, the request is completed with busy status, and sd_resume returns an error.

Instead of this, firewire-sbp2 should rather keep requests which are present at reconnect and submit them once more. Whether this is actually feasible I don't know yet, but I have hopes. If this is possible, we can also rip out all usages of the Scsi_Host block/ unblock API in firewire-sbp2, which is a very delicate API with a high danger of deadlocks.

Stefan Richter
-=====-==--= =-=- =--=-
http://arcgraph.de/sr/

Is it possible to delay returning from the device resume routine until the rediscovery/reconnect has completed? This is more or less how the USB stack works.

This wouldn't be necessary if the transport was working before sd_resume got called.

This is probably a worthwhile idea in any case.

No. All it does is cause an error message to be printed in the system log. But it's possible that a failure lower down in the SCSI stack has this effect.

Alan Stern

Hi,

for reference, I attach the patch I used to workaround this.

Regards,
Tino

@@ -2183,6 +2185,7 @@
 	struct scsi_sense_hdr sshdr;
 	struct scsi_device *sdp = sdkp->device;
 	int res;
+	int i = 0;
 
 	if (start)
 		cmd[4] |= 1;	/* START */

@@ -2193,8 +2196,13 @@
 	if (!scsi_device_online(sdp))
 		return -ENODEV;
 
-	res = scsi_execute_req(sdp, cmd, DMA_NONE, NULL, 0, &sshdr,
-			       SD_TIMEOUT, SD_MAX_RETRIES, NULL);
+	while (res = scsi_execute_req(sdp, cmd, DMA_NONE, NULL, 0, &sshdr,
+			       SD_TIMEOUT, SD_MAX_RETRIES, NULL) && i < 10) {
+		i++;
+		sd_printk(KERN_WARNING, sdkp, "START_STOP FAILED, retrying.\n");
+		msleep(2000);
+	}
+
 	if (res) {
 		sd_printk(KERN_WARNING, sdkp, "START_STOP FAILED\n");
 		sd_print_result(sdkp, res);

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

Hmm. FireWire isn't deterministic in this regard; it's partly bus, partly network. The transport protocol SBP-2 is kind of a network protocol with remote DMA. Rediscovery and reconnect at PM resume are rather stochastic processes
- if the target went through a low power state too,
- if other nodes besides the Linux SBP-2 initiator and the SBP-2 target are on the bus,
- not to mention if those other nodes went through a low power cycle as well.

I could add .suspend and .resume methods to firewire-sbp2's struct device_driver (or just .resume if the PM core accepts that... I have to check the API), and the .resume method could contain a wait_for_completion_timeout which is unblocked when a reconnect happened. However, this could still go wrong if for some reason (e.g. see above) multiple reconnects to the target happen in a row.

So I tend to think firewire-sbp2 should learn to resubmit requests that were queued by SCSI midlayer after the SBP-2 connection broke & before reconnect happened, i.e. hide all this from SCSI midlayer rather than quitting this request with DID_BUS_BUSY.

Technically the transport does "work" at this time: It might have blocked the Scsi_Host though, or it might return "bus busy" status for one request and then block the host. But apparently that's not liked by upper layers during resume.

Anyway, I'd say it this way:

This wouldn't be necessary if the transport just hid this reconnection phase from SCSI core and everything above it.

Then we only need to rely on the reconnect (or possibly series of reconnects, see above) to finish before timeout, minus time for the actual execution of the request. That should fit comfortably into the 30 seconds SD_TIMEOUT.

I wonder what this might be.

Stefan Richter
-=====-==--= =-=- =--==
http://arcgraph.de/sr/

On Mon, Oct 19, 2009 at 22:24:43 +0200, Stefan Richter wrote:

[...]

I my case, XFS threw a lot of errors because of an incaccessible device and set the filesystem offline.

Regards,
Tino

Perhaps this happens:

- START STOP UNIT does not get through to the disk since firewire-sbp2 could not reconnect early enough.
- The kernel _does not_ actually care, unlike I thought. It logs a device resume failure like Alan mentioned, then goes about its business.
- Linux attempts to use the filesystem and issues read/ write/ whatever requests.
- Spindle motor is still off, disk returns errors.

At this moment, the target should fail these requests with a specific sense code which tells the SCSI core that another START STOP UNIT is required now. However, it is just as likely that the target firmware sends something nonsensical or nothing at all. Or maybe the SCSI core does even proceed to request START STOP UNIT but the firmware already went belly-up. Finally, SCSI core gives up and takes the disk offline.

Stefan Richter
-=====-==--= =-=- =--==
http://arcgraph.de/sr/