Dealing with flaky USB storage devices and rootfs
Hello,
We have a system where the rootfs is a partition on a USB device, and I've noticed upon a few rare cases where the USB controller loses the connection to the USB device after some uptime (days, weeks...), and the USB device reappears a very short time later.
It doesn't really matter why, I guess that USB controllers and storage devices aren't always of the best quality - let's assume that. The issue is that Linux currently makes it problematic to recover the rootfs for several reasons.
If /dev/sda1 is mounted as root and the SCSI device goes into offline mode, it is quite impossible to get out of this situation. At first, I had to disable the usermode helper that sends events to udev since it blocks on the dysfunction /dev/sda1. Afterwards, I noticed that the reappearing device is being assigned with /dev/sdb along with a new SCSI host. I guess the the VFS and block layer still keep a reference to the old scsi_disk and as a result sd doesn't free it.
So, I gave some thoughts about this, I and came up with two main solutions:
1) Improve the USB storage error handling - bind the already existing SCSI host to the USB port that has the device, e.g., if host2 got created for usb 5-3 then keep it that way for the sake of EH. /dev/sda1 should come to life when the USB device recovers, unless a few seconds have passed or some attributes (such as manufactor id or serial) have changed.
2) Block layer hack - Write a special layering block device driver that acts as a proxy to the currently functioning /dev/sd* which gets auto-detected by this block layer driver. md multipath for the 'poor', you might say..
I chose '2' for the time being. It was much easier than hacking around the USB subsystem... Yes, I know rootfs on USB is an uncommon use-case at the moment but I do like to know if there are some improvements planned in this area.
Signature · 3 lines
Dan Aloni XIV LTD, http://www.xivstorage.com da-x (at) monatomic.org, dan (at) xiv.co.il
Quoted 6 lines · Dan Aloni
Hello,
We have a system where the rootfs is a partition on a USB device, and I've noticed upon a few rare cases where the USB controller loses the connection to the USB device after some uptime (days, weeks...), and the USB device reappears a very short time later.
That failure mode is pretty uncommon. More often what happens is the connection remains intact but communication/protocol/firmware/??? errors cause the device to stop working. It never disappears but it can't be used again without unplugging or power-cycling.
Quoted 5 lines
It doesn't really matter why, I guess that USB controllers and storage devices aren't always of the best quality - let's assume that. The issue is that Linux currently makes it problematic to recover the rootfs for several reasons.
Quoted 8 lines
If
/dev/sda1is mounted as root and the SCSI device goes into offline mode, it is quite impossible to get out of this situation. At first, I had to disable the usermode helper that sends events to udev since it blocks on the dysfunction/dev/sda1. Afterwards, I noticed that the reappearing device is being assigned with/dev/sdbalong with a new SCSI host. I guess the the VFS and block layer still keep a reference to the oldscsi_diskand as a result sd doesn't free it.
Yes.
Quoted 9 lines
So, I gave some thoughts about this, I and came up with two main solutions:
1) Improve the USB storage error handling - bind the already existing SCSI host to the USB port that has the device, e.g., if host2 got created for usb 5-3 then keep it that way for the sake of EH.
/dev/sda1should come to life when the USB device recovers, unless a few seconds have passed or some attributes (such as manufactor id or serial) have changed.
The difficulty here is that this "error" is indistinguishable from normal activity -- someone simply unplugs the device and then later on another connection is made. It might be the same device as before or it might be a different one. In other words, it isn't really an error. You would solve this by relying on "a few seconds" timeout.
Quoted 9 lines
2) Block layer hack - Write a special layering block device driver that acts as a proxy to the currently functioning
/dev/sd*which gets auto-detected by this block layer driver. md multipath for the 'poor', you might say..I chose '2' for the time being. It was much easier than hacking around the USB subsystem... Yes, I know rootfs on USB is an uncommon use-case at the moment but I do like to know if there are some improvements planned in this area.
Interesting. I've been working on something very much like your 1) for some time. A version of it is now in the USB development tree. It's not meant for situations like the one you describe; instead it is intended for suspend-to-RAM or hibernation. Generally these processes involve loss of power on the USB bus, causing it to appear as though all the devices have been unplugged.
In theory the same approach could be used to recover connections even in the absence of an intervening suspend. It would be clumsy in some respects, though. For instance, when a device actually was disconnected, the USB core wouldn't be able to notify the device's driver for a few seconds. In that time lots of unwanted activity and errors could mount up.
It also goes against the USB specification. And it is potentially unsafe, in that it is possible for users to change media or make other alterations that the kernel cannot detect. The same would be true of your proposal, assuming that somebody was quick enough to unplug one device and plug in another (or swap memory cards) in the span of a few seconds.
Alan Stern
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dan Aloni
Hello,
We have a system where the rootfs is a partition on a USB device, and I've noticed upon a few rare cases where the USB controller loses the connection to the USB device after some uptime (days, weeks...), and the USB device reappears a very short time later.
Quoted 4 lines
That failure mode is pretty uncommon. More often what happens is the connection remains intact but communication/protocol/firmware/??? errors cause the device to stop working. It never disappears but it can't be used again without unplugging or power-cycling.
Yes this is also what I assume happening. i.e. more likely a USB flash disk firmware bug than a controller bug (there are lots of crappy USB flash drives out there).
Quoted 15 lines
Quoted 9 lines
So, I gave some thoughts about this, I and came up with two main solutions:
1) Improve the USB storage error handling - bind the already existing SCSI host to the USB port that has the device, e.g., if host2 got created for usb 5-3 then keep it that way for the sake of EH.
/dev/sda1should come to life when the USB device recovers, unless a few seconds have passed or some attributes (such as manufactor id or serial) have changed.The difficulty here is that this "error" is indistinguishable from normal activity -- someone simply unplugs the device and then later on another connection is made. It might be the same device as before or it might be a different one. In other words, it isn't really an error. You would solve this by relying on "a few seconds" timeout.
[...]
Quoted 7 lines
It also goes against the USB specification. And it is potentially unsafe, in that it is possible for users to change media or make other alterations that the kernel cannot detect. The same would be true of your proposal, assuming that somebody was quick enough to unplug one device and plug in another (or swap memory cards) in the span of a few seconds.
The specific use case I refer to is with a flash drive embedded inside a locked and closed chassis of a dedicated server. So, anyone repluging it must know what they are doing anyway.
Signature · 3 lines
Dan Aloni XIV LTD, http://www.xivstorage.com da-x (at) monatomic.org, dan (at) xiv.co.il
Quoted 10 lines · Dan Aloni
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dan Aloni
Hello,
We have a system where the rootfs is a partition on a USB device, and I've noticed upon a few rare cases where the USB controller loses the connection to the USB device after some uptime (days, weeks...), and the USB device reappears a very short time later.
Quoted 8 lines
Quoted 4 lines
That failure mode is pretty uncommon. More often what happens is the connection remains intact but communication/protocol/firmware/??? errors cause the device to stop working. It never disappears but it can't be used again without unplugging or power-cycling.
Yes this is also what I assume happening. i.e. more likely a USB flash disk firmware bug than a controller bug (there are lots of crappy USB flash drives out there).
The only way to tell for certain in some particular case (like yours) is to use a USB bus analyzer. However you probably don't care too much about the cause, so long as you can fix it. Have you tried using a different flash drive?
Quoted 3 lines
The specific use case I refer to is with a flash drive embedded inside a locked and closed chassis of a dedicated server. So, anyone repluging it must know what they are doing anyway.
That's fine for you. But once the code is in the kernel, it affects everybody -- including people who don't have dedicated servers with closed and locked chassis.
My personal opinion is that it's somewhat distasteful to add software workarounds for problems caused by bad hardware, unless it's really impractical to fix or replace the hardware. But to each his own...
Alan Stern