RAID1 vs RAID10 and best way to set up 6 disks

Hello,

I have 6 WD Red Pro drives, each 6TB in space. My question is, what is the best way to set these up?

The system drive (and root) are on a 500GB SSD, so these drives will only be used for /home and file storage.

Is there any caveats between RAID1 on all 6 vs RAID10?

Thanks for the help,

Mitch

Just to be safe: RAID1 in btrfs means not what RAID1 means in any other terminology about RAID.

The former has only two duplicates, the later means full mirroring of all devices.

Cheers,
Chris.

Thanks for pointing that out, so if I'm thinking correctly, with RAID1 it's just that there is a copy of the data somewhere on some other drive.

With RAID10, there's still only 1 other copy, but the entire "original" disk is mirrored to another one, right?

To be honest, I couldn't tell you for sure :-/ ... IMHO the btrfs documentation has some "issues".

mkfs.btrfs(8) says: 2 copies for RAID10, so I'd assume it's just the striped version of what btrfs - for whichever questionable reason - calls "RAID1".

Especially, when you have an odd number devices (or devices with different sizes), its not clear to me, personally, at all how far that redundancy actually goes respectively what btrfs actually does... could be that you have your 2 copies, but maybe on the same device then?

Cheers,
Chris.

As Justin mentioned, btrfs doesn't raid whole disks/devices. Instead, it works with chunks.

The "questionable reason" is simply the fact that it is, now as well as at the time the features were added, the closest existing terminology that best describes what it does. Even now, it would be difficult on the spot adequately to explain what it means for redundancy without also mentioning "RAID".

Btrfs does not raid disks/devices. It works with chunks that are allocated to devices when the previous chunk/chunk-set is full.

We're all very aware of the inherent problem of language - and have discussed various ways to address it. You will find that some on the list (but not everyone) are very careful to never call it "RAID" - but instead raid (very small difference, I know). Hugo Mills previously made headway in getting discussion and consensus of proper nomenclature. *

No, btrfs' raid1 naively guarantees that the two copies will *never* be on the same device. raid10 does the same thing - but in stripes on as many devices as possible.

The reason I say "naively" is that there is little to stop you from creating a 2-device "raid1" using two partitions on the same physical device. This is especially difficult to detect if you add abstraction layers (lvm, dm-crypt, etc). This same problem does apply to mdadm however.

Though it won't necessarily answer all questions about allocation, I strongly suggest checking out Hugo's btrfs calculator **

I hope this is helpful.

* http://comments.gmane.org/gmane.comp.file-systems.btrfs/34717 / https://www.spinics.net/lists/linux-btrfs/msg33742.html
* http://comments.gmane.org/gmane.comp.file-systems.btrfs/34792
** http://carfax.org.uk/btrfs-usage/

__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Well the RAID1 was IMHO still bad choice as it's pretty ambiguous.

A better choice would have been something simple like rep2 (rep=replicas), mirror2, or dup with either adding some additional string that it's guaranteed to be on different devices here, or one that it's not guaranteed on what's currently "DUP".

But DUP(licate) seems anyway a little bit "restricted". It's not so unlikely that some people want a level that has always exactly three copies, or one with for.
So the repN / replicaN seems good to me.

Since the standard behaviour should be to enforce replicas being on different devices I'd have said, one could have made analogous levels named e.g. "same-device-repN" (or something like that just better), with same-device-rep2, being what our current DUP is

Sure, but effectively this is quite close.
And whether it works on whole device level or chunk level doesn't change that it's pretty important to be able to have the guarantee that the different replicas are actually on different devices.

Really very very small... to non-existent. ;)

Well I'd say, for btrfs: do away with the term "RAID" at all, use e.g.:

linear = just a bunch of devices put together, no striping basically what MD's linear is
mirror (or perhaps something like clones) = each device in the fs contains a copy of everything (i.e. classic RAID1) striped = basically what RAID0 is
replicaN = N replicas of each chunk on distinct devices <whatever>-replicaN = N replicas of each chunk NOT necessarily on distinct devices
parityN = n parity chunks i.e. parity1 ~= RAID5, parity2 ~= RAID6 or perhaps better: striped-parityN or striped+parityN ??

And just mention in the manpage, which of these names comes closest to what people understand by RAID level i.

Sure... I think software should try to prevent people from doing stupid things, but not by all means ;-)
If one makes n partitions on the same device an puts a RAID on that, one probably doesn't deserve it any better ;-)

I'd guess it's probably doable to detect such stupidness for e.g. partitions and dm-crypt (because these are linearly on one device)... but for lvm/MD it really depends on the actual block allocation/layout, whether it's safe or not.
Maybe the tools could detect *if* lvm/MD is in between and just give a general warning what that could mean.

Best wishes,
Chris.

That's ridiculous. It isn't incorrect to refer to only 2 copies as raid1. You have to explicitly ask both mdadm and lvcreate for the number of copies you want, it doesn't automatically happen. The man page for mkfs.btrfs is very clear you only get two copies.

What's ambiguous is raid10 expectations with multiple device failures.

Except this isn't really how Btrfs single works. The difference between mdadm linear and Btrfs single is more different in behavior than the difference between mdadm raid1 and btrfs raid1. So you're proposing tolerating a bigger difference, while criticizing a smaller one. *shrug*

If a metaphor is going to be used for a technical thing, it would be mirrors or mirroring. Mirror would mean exactly two (the original and the mirror). See lvcreate --mirrors. Also, the lvm mirror segment type is legacy, having been replaced with raid1 (man lvcreate uses the term raid1, not RAID1 or RAID-1). So I'm not a big fan of this term.

lvcreate uses only striped, not raid0. mdadm uses only RAID0, not striped. Since striping is also employed with RAIDs 4, 5, 6, 7, it seems ambiguous even though without further qualification whether parity exists, it's considered to mean non-parity striping. The ambiguity is probably less of a problem than the contradiction that is RAID0.

This is kinda interesting. At least it's a new term so all the new rules can be stuffed into that new term and helps distinguish it from other implementations, not entirely different with how ZFS does this with their raidz.

It's not easy, is it?

It already does this. What version of btrfs-progs are you basing your criticism on that there's some inconsistency, deficiency, or ambiguity when it comes to these raid levels? The one that's unequivocally problematic alone without reading the man page is raid10. The historic understanding is that it's a stripe of mirrors, and this suggests you can lose a mirror of each stripe i.e. multiple disks and not lose data, which is not true for Btrfs raid10. But the man page makes that clear, you have 2 copies for redundancy, that's it.

On the CLI? Not worth it. If the user is that ignorant, too bad, use a GUI program to help build the storage stack from scratch. I'm really not sympathetic if a user creates a raid1 from two partitions of the same block device anymore than if it's ultimately the same physical device managed by a device mapper variant.

Anyway, I think there's a whole separate github discussion on Btrfs UI/Ux that presumably also includes terminology concerns like this.

Chris Murphy

No, if there are only two devices then not.
But obviously we're talking about how btrfs does RAID1, in which even with n>2 devices there are only 2 copies - that's incorrect.

Aha, and which option would that be?

I've said that before, but at least it allows you to use the full number of disks, so we're again back to that it's closer to the original and common meaning of RAID1 than what btrfs does.

I haven't denied that... but one shouldn't use terms that are commonly understood in a different mannor and require people to read all the small printed.
One could also have changed it's RAID0 with RAID1, and I guess people wouldn't be too delighted if the excuse was "well it's in the manpage".

What's the big difference? Would you care to explain? But I'm happy with "single" either, it just doesn't really tell that there is no striping, I mean "single" points more towards "we have no resilience but only 1 copy", whether this is striped or not.

Admittedly, I didn't like the "mirror(s)" either... I was just trying to show that different names could be used that are already a bit better.

Mhh,.. well or one makes schema names that contain all possible properties of a "RAID", something like:
replicasN-parityN-[not]striped

SINGLE would be something like "replicas1-parity0-notstriped". RAID5 would be something like "replicas0-parity1-striped".

Well first, the terminology thing is the least serious issue from my original list ;-) ... TBH I don't know why such a large discussion came out of that point.

Even though I'm not reading along all mails here, we have probably at least every month someone who wasn't aware that RAID1 is not what he assumes it to be.
And I don't think these people can be blamed for not RTFM, because IMHO this is a term commonly understood as mirror all available devices. That's how the original paper describes it, it's how Wikipedia describes it and all other sources I've ever read to the topic.

Yes, same basic problem.

Well one I have no strong opinion on that... if testing for it (or at least simple cases) would be easy, why not.
Not every situation may be as easily visible as creating a RAID1 on /dev/sda1 and /dev/sda2.
One may use LABELs, or UUIDs and accidentally catch the wrong, and in such cases a check may help.

Cheers,
Chris.

OK and I think the assertion is asinine. You reject the only neutral party's definition and distinction of RAID-1 types, and then claim on the basis of opinion that Btrf's raid1 is not merely different from traditional/classic/common understandings of RAID-1, but that they're incorrect to have called it raid1. It's just nonsense. I find your argument uncompellling.

For mdadm it's implied as a combination of -n -x and number of devices. For lvcreate it's explicit with -m. This is in the man page, so I don't understand why you're asking.

The original and common meaning defined by whom, where? You're welcome to go take it up with Wikipedia but they're using SNIA definitions for the standard RAID levels.

And I disagree because what you required is more reading by the user to understand entirely new nomenclature.

Except nothing that crazy has been done so I fail to see the point.

It's not linear. The archives detail how block groups are allocated to devices. There are rules, linearity isn't one of them.

SNIA has created such a schema.

You have the wrong understanding. The most common is exactly two disk RAID-1.

Wikipedia:

"RAID 1 consists of an exact copy (or mirror) of a set of data on two or more disks; a classic RAID 1 mirrored pair contains two disks. "

An exact copy = one copy. It does not say or imply n-way.

Chris Murphy

Go read the original standards that defined the term RAID (assuming you can find a openly accessible copy), it defines RAID1 as a mirrored _pair_ of disks. This is how every hardware RAID controller I've ever seen implements RAID1, and in fact how most software RAID implementations (including the fake raid in some motherboards) implements it with the sole exception of Linux's MD-RAID and it's direct derivatives (which includes LVM/DM based RAID, as well as BSD's GEOM framework).

Specifying more than two disks. The request is more correctly an implicit one, but the fact that it's implied by a now largely obsolete piece of software does not mean that BTRFS should have the same implications.

/me inserts reflink to the first part of my reply.

You can leave the hyperbolic theoreticals out of this, they really do detract from your argument.

On this point I actually do kind of agree with you, but Chris is also correct here, BTRFS single mode is just as different from MD linear mode as BTRFS raid1 is from MD RAID1, if not more so.

It's worth pointing out that both programmers and sysadmins are still lazy typists, so it would more likely end up being:
rep1-par0-strip0
rep0-par1-stripN (with N being the number of desired stripes).

Having a number to indicate the striping is actually useful (there are legitimate cases for not striping across everything we can, and we need some way to represent stripes that weren't allocated at full width for some reason).

Such a scheme was actually proposed back when the higher order parity patches were being discussed. Like those patches, it was decided to wait until we had basic feature completeness before trying to tackle that.

And it's also almost always someone who didn't properly read the documentation, which means they would have gotten bitten by something else eventually as well. This expectation that anyone should be able to pick up any piece of software and immediately use it without reading documentation needs to stop. Catering to such people is not a sustainable design choice for a piece of software as complex as a filesystem.

The first sentence of https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1 as of right now is exactly:
RAID 1 consists of an exact copy (or mirror) of a set of data on two or more disks; a classic RAID 1 mirrored pair contains two disks.

The 'classic RAID 1 mirrored pair' it refers to describes every RAID-1 implementation I or anyone I've asked (including people of the pre-UNIX vintage who still deal with computers) has ever seen with the sole exception of the software implementations in Linux and BSD.

To paraphrase something from the Debian IRC:
'Our job is to make sure that if the user points a gun at his foot and pulls the trigger, the bullet gets to the intended location'

There are limits to what we can protect against, and we shouldn't be preventing people from doing things that while they seem unreasonable at face value, are perfectly legitimate uses of the software. BTRFS raid1 mode on a single device falls solidly into this category for a couple of reasons:
1. Until recently, this was the only way without mixed mode to get data chunk redundancy on a single device. In many distros, this is still the only way to do so.
2. Because of how it's implemented, this actually doesn't get insanely horrible performance like trying to do the same thing with MD or LVM based RAID would.
3. This allows for forcing specific organization of data on the media, which can combined with item 1, be particularly useful (for example, some SSD's have SLC flash backing the area up through where the MFT would be on NTFS, and MLC for the rest, or some 'hybrid' disks actually just have flash for the first part of the logical device they expose, and traditional magnetic storage for the rest).

disk is mirrored to another one, right?

No, full disks are never mirrored in any configuration.

Here's how I understand Btrfs' non-parity redundancy profiles:

single: only a single instance of a file exists across the file system
dup: two instances of a file exist across the file system, and they may reside on the same physical disk (4.5.1+ required to use dup profile on multi-disk file system)
raid1: same as dup but the instances are guaranteed to be on different disks
raid0: single but can be striped between multiple disks
raid10: data is guaranteed to exist on two separate devices but if n>2 the data is load balanced between disks*

Even though my explanation is imperfect, I hopes that illustrates that Btrfs RAID is different than traditional RAID. Btrfs provides the same physical redundancy as RAID, but the implementation mechanisms are quite a bit different. This has wonderful consequences for flexibility, and it's what allowed me to run a 5x2TB RAID10 array for nearly two years and essentially allow complete allocation. The downside is that since allocations aren't enforced from start (eg. MD requiring certain number of disks and identical sizes), it's possible to get weird allocations over time, but the resolution is simple: run a balance from time to time.

different sizes), its not clear to me, personally, at all how far that redundancy actually goes respectively what btrfs actually does... could be that you have your 2 copies, but maybe on the same device then?

RAID1 (and transitively RAID10) guarantees two copies on different disks, always. Only dup allows the copies to reside on the same disk. This is guaranteed is preserved, even when n=2k+1 and mixed-capacity disks. If disks run out of available chunks to satisfy the redundancy profile, the result is ENOSPC and requires the administrator to balance the file system before new allocations can succeed. The question essentially is asking if Btrfs will spontaneously degrade into "dup" if chunks cannot be allocated on some devices. That will never happen.