USB-to-SATA and RAID

Hi,

By chance I found this statement in the Linux RAID Wiki:

"Do NOT use [USB devices] as part of your array. They have
a timeout/disconnect mechanism which interacts very badly with the raid code." [0]

Could someone elaborate further on this? Does "interacts very badly" mean that there will be potential data loss / data corruption? Or does "very badly" just mean short, temporary performance issues / lags upon a disk failure?

Regards, Linus

[0]: https://raid.wiki.kernel.org/index.php/Choosing_your_hardware,_and_what_is_a_device%3F#USB_devices

That was me that wrote that on the wiki. However, I'm just editing stuff so I don't actually have any experience of it.

The wiki also mentions the timeout problem. As far as I can make out - and bear in mind as I said above I don't have personal experience - it seems to me from what I've picked up that USB suffers this problem in spades.

In other words, what I *think* happens, from what I've picked up, is that the USB bus times out. And when the raid comes along to use the disk, the USB bus is no longer responding ... BOOM!

I hope that's helpful, and maybe people who know more will chime in, but that's it as I understand it.

Cheers,
Wol

Hi,

I have only a little technical insight into this topic, however, I have some (terrible) personal experience with USB raid and I would point out another aspect.
Although USBs are good for occasional connections of simple drives they have quite a strong tendency to wear out. This becomes a problem especially if you plan to connect raid often and means that wires might not connect or lose connection during the transition of data. I happened to have raid connected to older USB ports and connection was pretty unstable. Multiple times unexpected disconnection due to wires losing contact caused a problem and the disconnected disk (although almost perfectly valid) had to be completely rebuilt. I thought it was not that bad but later it caused sort of desync and I was unable to connect any two of three disks so the data was lost. My only luck was that I had all the data (at least important data) backed up.

So if you're considering raid over USB I suggest you don't do that. It will probably save you a lot of troubles.

With kind regards,

2018-01-23 21:46 GMT+01:00 Wol's lists <[email protected]>:

Pokud tento mail obsahuje tautologii, pak zřejmě tautologii skutečně obsahuje.

Vojtěch Kletečka

Something else I came across recently. Cooling in most enclosures ranges from poor to non-existent. This leads to overheating drives as a matter of course, and from that to failure.

(Personally, I have two enclosures that appear to have failed, and I need to investigate to see if the drives themselves are dead. Two out of two in the space of a few months is pretty bad going.)

Cheers,
Wol

Potential data loss / data corruption.  Don't use USB.  Use eSATA instead.  The list archives of linux-raid and linux-xfs are dotted with USB disk horror stories.

Hi,

Many thanks for all the quick and helpful responses so far!

Interesting. Do you know what they could be attributed to technically? Were they usually similar to what Vojtěch had described, so connectivity issues due to the poor connectors and thin cables which can often be found with USB? And things then fell over when for a brief moment connectivity issues occured for more disks than redundant disks were configured? Or do the extra latencies with USB create an extended exposure to the write loop hole in RAID5? Or something else?

Would you guys say that ARM boards with a SATA connector internally weird to USB (for instance the Ordoid HC2[*]), and then maybe some SATA Port Multiplier are similarly affected? Or would say that those are mainly unaffected of the common issues with RAIDs via SATA over USB?

Regards, Linus

[*]: http://www.hardkernel.com/main/products/prdt_info.php?g_code=G151505170472&tab_idx=2

Here is an example of what you may see with a USB connected drive:

http://forums.debian.net/viewtopic.php?f=10&t=48056

You'll have to read up on the USB protocol to see why devices are reset.  As with the user in this post, your USB connected drive may function fine for a while, but on occasion you'll hit one of these reset conditions.  When that occurs your filesystem may survive, or you may suffer corruption, or data loss.  I for one would not rely on storage that is reliable only 99% of the time. Murphy's law dictates it will fail at the worst possible time.

Just to chime in here, I've got a five slot USB3 external enclosure running on my Debian Jessie x86_64 main file/backup server for home. The data disks arent on here, they're mirrored on SATA controllers. But I do use the USB3 disks for storing backups taken with Burp. So far, it's been really reliable for me.

But I never touch the cable between the systems, and neither system moves or gets moved around. So you *might* have better luck with USB3 than the older, slower USB2 external drives and chassis.

John

Based on my experience most motherboards only have 2 or 3 actual usb ports, and all of the others ports are off a hub connected to the real
ports. And if you put more than one high bandwidth device on the
same port then things get really ugly if both devices are being heavily used. Most add-on cards also only have 1 port, if you look
hard enough you can find cards with 2-4 real ports. I have one that
has 4 actual real ports, and worked reasonably well for several webcams, but I believe this one was the only one on the market that
had 4 ports and was reasonably cheap. In general the sharing apears
to be the major issue so it is probably kind of related to the SATA port multiplier issue since most usb ports are shared resources and really a similar design, with similar issues. I am guessing if you put one usb disk per real usb port and did not plug anything else into it it would probably work reasonably well, but to built out a setup like that you might was well just build out an esata setup.

Okay, I've written this up and summarised it in a short page on the wiki.

https://raid.wiki.kernel.org/index.php/What's_all_this_with_USB%3F

Does that seem a fair enough summary?

Cheers,
Wol

I would expect that the majority of USB disk issues can be avoided by due care and attention.

USB buses are rarely reset or re-initialised without provocation. The cause is generally flaky hardware on the bus, plugging or unplugging devices, devices that sleep waking up, sudden changes in power requirements of devices, and other physical events.

When using USB disks, I'd give the following advice:

1. Connect directly to a USB port on the computer, not via a hub. That way you don't have other devices on the same bus that could cause trouble.

2. Avoid plugging other hardware in and out of the other USB buses, and certainly avoid it on the bus with the disk. Keep the bus physically stable.

3. Avoid having high power devices on USB, like DVD burners. If the disk has an optional external power supply, use it.

4. Ensure your USB disks are cooled appropriately. Some USB disk enclosures are very tight around the disk and can overheat. Freestanding USB-to-SATA converters where you plug the bare disk in directly can be good (but beware of dust if you are leaving them for a while).

5. Use good cables and avoid plugging and unplugging things too much, to limit the risk of broken wires and worn contacts.

6. Don't rely on the USB drive - use it as an extra. They are great for taking extra backups, and to provide additional redundancy during risky operations (such as to add an asymmetric second parity drive to a raid5 set before replacing the raid5 disks with bigger devices).

7. Don't use "memory stick" USB drives for raid - these have very limited write lifetimes and are easily corrupted.

8. And for the love of $DEITY, /don't/ have USB drives connected by cables where you will trip over them and throw the drive across the room. (As Yoda says, good judgement comes from experience. Experience comes from bad judgement!)

Hi,

Thanks to everyone again! By the way, why I am considering to use USB at all is that the storage system is for a solar powered site. And therefore I have to limit power usage to roughly 10-15W...

So my thinking was to use some ARM board and low-power 2.5" disks.

Unfortunately, many of the low-power, ARM based boards seem to have no SATA. Or just a single SATA port, often internally wired to USB...

And yeah, I basically bought a Pi3 and some SATA-to-USB dongles before stumbling over this entry in the Wiki... my fault...

But I'm glad to hear that there does not seem to be an issue with USB itself, but more the circumstances.

Regarding cabling and temperature, I'm not too worried. I already 3D-printed some stackable enclosures, even with snap-in mechanisms and holes for srews, and with hopefully enough airflow capabilities (people had uploaded some nice designs at thingiverse.com).

But I'll keep an eye on what other ARM based boards with on-board SATA will come out in the future. And will keep in mind to swap the board later.

Again, many thanks!

Regards, Linus

PS: And sharing one nightmare story with USB, too: Had a disk from Toshiba connected to a SATA-to-USB adapter with no extra power supply and that was connected to an Odroid C1+ (an ARM board from Hardkernel). The Odroid was not capable of providing enough power and turned off the USB port for half a second upon noticing. So I had this nice sound of a disk trying to spin up about twice a second. Ten seconds later I caught up with what was going on and disconnected the disk. Nevertheless too late, the disk was broken then... unrecoverable bad sectors even after trying to wipe the disk. So in that case, an external USB Hub with its own power supply would have actually saved that disk, I guess.