future of raid 6

Hello!

I have one question (maybe more) regarding the raid development. Especially raid 6 and better.
I read in some articles[1] that raid 6 will become worse as the size of the disks grow but the (unrecoverable) error rate and data rate do not improve as much. (Rebuilds are likely to hit an unrecoverable error, not to mention the long time it will take to rebuild raids with +10TB per drive.) They estimated 2019 (that was written ten years ago).

Is there already something in progress for the md-raid in linux kernel?
If so what and where?
If not, why? Is it not needed yet? Is md-raid itself depreciated? What is the future for redundant mass storage?

Best regards,
Markus

[1] http://queue.acm.org/detail.cfm?id=1670144

Hard errors aren't that common (at least until a drive fails :-)

A hard error is a sector that cannot be read. A soft error is one that works fine after a reset (however you define said reset). A consumer 10TB disk can return one soft error per complete pass and still be passed "it's good, it's within spec".

(And I think a lot of hard errors are down to not looking after the data. Just as DRAM decays over nano-seconds, so do hard drives decay over time, exacerbated by nearby writes. A hard error could simply be data that's been degaussed by nearby writes. Fail to deal with that with eg scrubs, and a perfectly functional drive can trash your data :-(

IFF someone thinks it's worth it, we may add raid-6+ functionality (ie more than two parity drives), but I suspect enterprise drives will simply become more reliable as they get bigger (manufacturers will move more and more error correction technology into the firmware).

Bear in mind enterprise drives are roughly four binary orders of magnitude more reliable, so a 160TB should just about read completely without error.

As for "is md-raid deprecated", well a lot of the filesystem guys would like to do so, and I sort of agree with them - the more layers of abstraction, the worse things are. But the opposite also holds true - put raid into the filesystem itself and the complexity there can explode
- just witness the trouble higher raid has caused the btrfs folks.

In *MY* humble (very) opinion, unrecoverable errors are far less common than people think, technology is improving, and "2019" is a lot further off than two years. That said, we will probably need something more than plain raid-6 once we start using having disks in the 50-80TB range. Unless, of course, technology inside the drive improves which it probably will.

Cheers,
Wol

2017-09-03 23:03 GMT+02:00 Wols Lists <[email protected]>:

This would be great idea. Even ZFS has support for RAID-7 (3 parity disks)

URE is the same even for enterprise disks.

Rebuilds are not likely to hit an unrecoverable error at all. Rebuilds do not take a long time, and it tends to be irrelevant.

A lot of articles make rebuilds out to be some kind of mythological beast that breathes fire, spits poison, and devours your drives. It's not true.

The probability of failure during a rebuild is not higher than normal. A rebuild is as boring as can be, just a linear read (n-1 drives), linear write (1 drive). Reshapes are more interesting but it's still just normal reading and writing: there is no magic involved, there is absolutely nothing that could possibly cause undue drive failures.

Drives fail randomly, silently. The only way to verify that a drive hasn't failed yet is to read everything from start to end. If you don't run these read tests regularly, the error will go unnoticed for weeks, months, years.

That's the timeframe we're really looking at: not hours, days it actually takes for the rebuild itself to finish, but weeks (for you to order a new drive and it actually getting shipped), and months (for you to detect the error in the first place and bring yourself to order a replacement and not see how it goes first because you feel the pinch of replacement costs).

If you do not monitor and regularly test your drives to cut down this time, no amount of redundancy will ever be enough. If you never run any tests, the errors will pop up during rebuild. If you never tested and thus allowed the rebuild itself to be the first ever test in years, it's not a miracle. You can't blame the rebuild for that.

It has nothing to do with bit error rates, timeouts, same batch of drives, and all the other shit people come up with to make excuses for their lack of monitoring drives properly.

Genuine simultaneous drive failures are *very* rare.

It's still RAID-5 or something similar on a filesystem level. Simply because it works, and it's cost effective. It doesn't matter if the drive has 1000MB, 100GB, 10TB - nothing changed.

Most machines (desktops, nas, rented servers) only have 2-4 drives anyway. You need twice as many for RAID-6 to start making any kind of sense, and triple parity is even further off, like 20+ drives.

If you're currently running a three drive RAID-6 and need more redundancy, you can do it with a four drive RAID-1. Knock yourself out. Nobody does it. It's barking mad. The only problem it solves does not really exist.

And regardless of redundancy, you still need backups. If you have backups, the zero point something chance of simultaneous failure matters even less.

Regards
Andreas Klauer

No, the "drive failed and I now have URE on another drive" is not "*very rare*". Yes, it can be somewhat mitigated by frequent scrubbing.

That's why I run RAID6 and not RAID5. I have been hit by the above problem several times when running RAID5. I haven't yet had a data loss event (that I know of) because of this since I moved to RAID6.

I think "RAID6 with triple parity" would make sense for some deployment scenarios. It's not uncommon for people to have several RAID6 arrays and then have a hot spare in the chassis. This hot spare could instead actually do work by being triple redundancy in one array, instead of sitting there powered on but doing nothing.

Mikael Abrahamsson    email: [email protected]

Hi Markus,

You should be aware that some helpful people nevertheless have a magical view of technology and don't actually provide accurate information, like the above (commonly repeated) screed.

The failure event possibilities of the various raid levels can be easily analyzed if one is unafraid of math (statistics) and have a modest view of the possible events involved.

The possible events are basically true, persistent hardware failure of the drive, and potentially rewritable/fixable transient magnetic failures in data retrieval. The can combine in various ways, like so:

https://marc.info/?l=linux-raid&m=139050322510249&w=2

The odds of total hardware failure can be inferred from warranty lengths, but are generally much lower than the odds of transient flaws, based on manufacturer's specification claims for unrecoverable read errors. The math for the latter follows the Poisson distribution, like so:

https://marc.info/?l=linux-raid&m=135863964624202&w=2

Pretty bright, if you really take a look at the math. And are willing to purchase hardware that cooperates with automated maintenance instead of blowing up (the dreaded and very real timeout mismatch problem that Andreas dismisses as fantasy). Simply put, if smartctl -l scterc yields a message that Error Recovery Control is not supported, you *must* adjust the kernel's driver timeouts to avoid catastrophic array failure.

Phil