List of known BTRFS Raid 5/6 Bugs?
sorry for disturb this discussion,
are there any plans/dates to fix the raid5/6 issue? Is somebody working on this issue? Cause this is for me one of the most important things for a fileserver, with a raid1 config I loose to much diskspace.
best regards
Stefan
Stefan K posted on Fri, 07 Sep 2018 15:58:36 +0200 as excerpted:
Quoted 5 lines
sorry for disturb this discussion,
are there any plans/dates to fix the raid5/6 issue? Is somebody working on this issue? Cause this is for me one of the most important things for a fileserver, with a raid1 config I loose to much diskspace.
There's a more technically complete discussion of this in at least two earlier threads you can find on the list archive, if you're interested, but here's the basics (well, extended basics...) from a btrfs-using- sysadmin perspective.
"The raid5/6 issue" can refer to at least three conceptually separate issues, with different states of solution maturity:
1) Now generally historic bugs in btrfs scrub, etc, that are fixed (thus the historic) in current kernels and tools. Unfortunately these will still affect for some time many users of longer-term stale^H^Hble distros who don't update using other sources for some time, as because the raid56 feature wasn't yet stable at the lock-in time for whatever versions they stabilized on, they're not likely to get the fixes as it's new-feature material.
If you're using a current kernel and tools, however, this issue is fixed. You can look on the wiki for the specific versions, but with the 4.18 kernel current latest stable, it or 4.17, and similar tools versions since the version numbers are synced, are the two latest release series, with the two latest release series being best supported and considered "current" on this list.
Also see...
2) General feature maturity: While raid56 mode should be /reasonably/ stable now, it remains one of the newer features and simply hasn't yet had the testing of time that tends to flush out the smaller and corner- case bugs, that more mature features such as raid1 have now had the benefit of.
There's nothing to do for this but test, report any bugs you find, and wait for the maturity that time brings.
Of course this is one of several reasons we so strongly emphasize and recommend "current" on this list, because even for reasonably stable and mature features such as raid1, btrfs itself remains new enough that they still occasionally get latent bugs found and fixed, and while /some/ of those fixes get backported to LTS kernels (with even less chance for distros to backport tools fixes), not all of them do and even when they do, current still gets the fixes first.
3) The remaining issue is the infamous parity-raid write-hole that affects all parity-raid implementations (not just btrfs) unless they take specific steps to work around the issue.
The first thing to point out here again is that it's not btrfs-specific. Between that and the fact that it *ONLY* affects parity-raid operating in degraded mode *WITH* an ungraceful-shutdown recovery situation, it could be argued not to be a btrfs issue at all, but rather one inherent to parity-raid mode and considered an acceptable risk to those choosing parity-raid because it's only a factor when operating degraded, if an ungraceful shutdown does occur.
But btrfs' COW nature along with a couple technical implementation factors (the read-modify-write cycle for incomplete stripe widths and how that risks existing metadata when new metadata is written) does amplify the risk somewhat compared to that seen with the same write-hole issue in various other parity-raid implementations that don't avoid it due to taking write-hole avoidance countermeasures.
So what can be done right now?
As it happens there is a mitigation the admin can currently take -- btrfs allows specifying data and metadata modes separately, and even where raid1 loses too much space to be used for both, it's possible to specify data as raid5/6 and metadata as raid1. While btrfs raid1 only covers loss of a single device, it doesn't have the parity-raid write-hole as it's not parity-raid, and for most use-cases at least, specifying raid1 for metadata only, while raid5 for data, should strictly limit both the risk of the parity-raid write-hole as it'll be limited to data which in most cases will be full-stripe writes and thus not subject to the problem, and the size-doubling of raid1 as it'll be limited to metadata.
Meanwhile, arguably, for a sysadmin properly following the sysadmin's first rule of backups, that the true value of data isn't defined by arbitrary claims, but by the number of backups it is considered worth the time/trouble/resources to have of that data, it's a known parity-raid risk specifically limited to the corner-case of having an ungraceful shutdown *WHILE* already operating degraded, and as such, it can be managed along with all the other known risks to the data, including admin fat-fingering, the risk that more devices will go out than the array can tolerate, the risk of general bugs affecting the filesystem or other storage-function related code, etc.
IOW, in the context of the admin's first rule of backups, no matter the issue, raid56 write hole or whatever other issue of the many possible, loss of data can *never* be a particularly big issue, because by definition, in *all* cases, what was of most value was saved, either the data if it was defined as valuable enough to have a backup, or the time/ trouble/resources that would have otherwise gone into making that backup, if the data wasn't worth it to have a backup.
(One nice thing about this rule is that it covers the loss of whatever number of backups along with the working copy just as well as it does loss of just the working copy. No matter the number of backups, the value of the data is either worth having one more backup, just in case, or it's not. Similarly, the rule covers the age of the backup and updates nicely as well, as that's just a subset of the original problem, with the value of the data in the delta between the last backup and the working copy now being the deciding factor, either the risk of losing it is worth updating the backup, or not, same rule, applied to a data subset.)
So from an admin's perspective, in practice, while not entirely stable and mature yet, and with the risk of the already-degraded crash-case corner-case that's already known to apply to parity-raid unless mitigation steps are taken, btrfs raid56 mode should now be within the acceptable risk range already well covered by the risk mitigation of following an appropriate backup policy, optionally combined with the partial write-hole-mitigation strategy of doing data as raid5/6, with metadata as raid1.
OK, but what is being done to better mitigate the parity-raid write-hole problem for the future, and when might we be able to use that mitigation?
There are a number of possible mitigation strategies, and there's actually code being written using one of them right now, tho it'll be (at least) a few kernel cycles until it's considered complete and stable enough for mainline, and as mentioned in #2 above, even after that it'll take some time to mature to reasonable stability.
The strategy being taken is partial-stripe-write logging. Full stripe writes aren't affected by the write hole and (AFAIK) won't be logged, but partial stripe writes are read-modify-write and thus write-hole succeptible, and will be logged. That means small files and modifications to existing files, the ends of large files, and much of the metadata, will be written twice, first to the log, then to the final location. In the event of a crash, on reboot and mount, anything in the log can be replayed, thus preventing the write hole.
As for the log, it'll be written using a new 3/4-way-mirroring mode, basically raid1 but mirrored more than two ways (which current btrfs raid1 is limited to, even with more than two devices in the filesystem), thus handling the loss of multiple devices.
That's actually what's being developed ATM, the 3/4-way-mirroring mode, which will be available for other uses as well.
Actually, that's what I'm personally excited about, as years ago, when I first looked into btrfs, I was running older devices in mdraid's raid1 mode, which does N-way mirroring. I liked the btrfs data checksumming and scrubbing ability, but with the older devices I didn't trust having just two-way-mirroring and wanted at least 3-way-mirroring, so back at that time I skipped btrfs and stayed with mdraid. Later I upgraded to ssds and decided btrfs-raid1's two-way-mirroring was sufficient, but when one of the ssds ended up prematurely bad and needed replaced, I would have sure felt a bit better before I got the replacement done, if I still had good two-way-mirroring even with the bad device.
So I'm still interested in 3-way-mirroring and would probably use it for some things now, were it available and "stabilish", and I'm eager to see that code merged, not for the parity-raid logging it'll also be used for, but for the reliability of 3-way-mirroring. Tho I'll probably wait at least 2-5 kernel cycles after introduction and see how it stabilizes before actually considering it stable enough to use myself, because even tho I do follow the backups policy above, just because I'm not considering the updated-data delta worth an updated backup yet, doesn't mean I want to unnecessarily risk having to redo the work since the last backup, which means choosing the newer 3-way-mirroring over the more stable and mature existing raid1 2-way-mirroring isn't going to be worth it to me until the 3-way-mirroring has at least a /few/ kernel cycles to stabilize.
And I'd recommend the same caution with the new raid5/6 logging mode built on top of that multi-way-mirroring, once it's merged as well. Don't just jump on it immediately after merge unless you're deliberately doing so to help test for bugs and get them fixed and the feature stabilized as soon as possible. Wait a few kernel cycles, follow the list to see how the feature's stability is coming, and /then/ use it, after factoring in its remaining then still new and less mature additional risk into your backup risks profile, of course.
Time? Not a dev but following the list and obviously following the new 3- way-mirroring, I'd say probably not 4.20 (5.0?) for the new mirroring modes, so 4.21/5.1 more reasonably likely (if all goes well, could be longer), probably another couple cycles (if all goes well) after that for the parity-raid logging code built on top of the new mirroring modes, so perhaps a year (~5 kernel cycles) to introduction for it. Then wait however many cycles until you think it has stabilized. Call that another year. So say about 10 kernel cycles or two years. It could be a bit less than that, say 5-7 cycles, if things go well and you take it before I'd really consider it stable enough to recommend, but given the historically much longer than predicted development and stabilization times for raid56 already, it could just as easily end up double that, 4-5 years out, too.
But raid56 logging mode for write-hole mitigation is indeed actively being worked on right now. That's what we know at this time.
And even before that, right now, raid56 mode should already be reasonably usable, especially if you do data raid5/6 and metadata raid1, as long as your backup policy and practice is equally reasonable.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
wow, holy shit, thanks for this extended answer!
Quoted 1 line
The first thing to point out here again is that it's not btrfs-specific.
so that mean that every RAID implemantation (with parity) has such Bug? I'm looking a bit, it looks like that ZFS doesn't have a write hole. And it _only_ happens when the server has a ungraceful shutdown, caused by poweroutage? So that mean if I running btrfs raid5/6 and I've no poweroutages I've no problems?
Quoted 1 line
it's possible to specify data as raid5/6 and metadata as raid1
does some have this in production? ZFS btw have 2 copies of metadata by default, maybe it would also be an option or btrfs? in this case you think 'btrfs fi balance start -mconvert=raid1 -dconvert=raid5 /path ' is safe at the moment?
Quoted 2 lines
That means small files and modifications to existing files, the ends of large files, and much of the metadata, will be written twice, first to the log, then to the final location.
that sounds that the performance will go down? So far as I can see btrfs can't beat ext4 or btrfs nor zfs and then they will made it even slower?
thanks in advanced!
best regards
Stefan
Quoted 205 lines · CEST Duncan
Stefan K posted on Fri, 07 Sep 2018 15:58:36 +0200 as excerpted:
Quoted 5 lines
sorry for disturb this discussion,
are there any plans/dates to fix the raid5/6 issue? Is somebody working on this issue? Cause this is for me one of the most important things for a fileserver, with a raid1 config I loose to much diskspace.
There's a more technically complete discussion of this in at least two earlier threads you can find on the list archive, if you're interested, but here's the basics (well, extended basics...) from a btrfs-using- sysadmin perspective.
"The raid5/6 issue" can refer to at least three conceptually separate issues, with different states of solution maturity:
1) Now generally historic bugs in btrfs scrub, etc, that are fixed (thus the historic) in current kernels and tools. Unfortunately these will still affect for some time many users of longer-term stale^H^Hble distros who don't update using other sources for some time, as because the raid56 feature wasn't yet stable at the lock-in time for whatever versions they stabilized on, they're not likely to get the fixes as it's new-feature material.
If you're using a current kernel and tools, however, this issue is fixed. You can look on the wiki for the specific versions, but with the 4.18 kernel current latest stable, it or 4.17, and similar tools versions since the version numbers are synced, are the two latest release series, with the two latest release series being best supported and considered "current" on this list.
Also see...
2) General feature maturity: While raid56 mode should be /reasonably/ stable now, it remains one of the newer features and simply hasn't yet had the testing of time that tends to flush out the smaller and corner- case bugs, that more mature features such as raid1 have now had the benefit of.
There's nothing to do for this but test, report any bugs you find, and wait for the maturity that time brings.
Of course this is one of several reasons we so strongly emphasize and recommend "current" on this list, because even for reasonably stable and mature features such as raid1, btrfs itself remains new enough that they still occasionally get latent bugs found and fixed, and while /some/ of those fixes get backported to LTS kernels (with even less chance for distros to backport tools fixes), not all of them do and even when they do, current still gets the fixes first.
3) The remaining issue is the infamous parity-raid write-hole that affects all parity-raid implementations (not just btrfs) unless they take specific steps to work around the issue.
The first thing to point out here again is that it's not btrfs-specific. Between that and the fact that it *ONLY* affects parity-raid operating in degraded mode *WITH* an ungraceful-shutdown recovery situation, it could be argued not to be a btrfs issue at all, but rather one inherent to parity-raid mode and considered an acceptable risk to those choosing parity-raid because it's only a factor when operating degraded, if an ungraceful shutdown does occur.
But btrfs' COW nature along with a couple technical implementation factors (the read-modify-write cycle for incomplete stripe widths and how that risks existing metadata when new metadata is written) does amplify the risk somewhat compared to that seen with the same write-hole issue in various other parity-raid implementations that don't avoid it due to taking write-hole avoidance countermeasures.
So what can be done right now?
As it happens there is a mitigation the admin can currently take -- btrfs allows specifying data and metadata modes separately, and even where raid1 loses too much space to be used for both, it's possible to specify data as raid5/6 and metadata as raid1. While btrfs raid1 only covers loss of a single device, it doesn't have the parity-raid write-hole as it's not parity-raid, and for most use-cases at least, specifying raid1 for metadata only, while raid5 for data, should strictly limit both the risk of the parity-raid write-hole as it'll be limited to data which in most cases will be full-stripe writes and thus not subject to the problem, and the size-doubling of raid1 as it'll be limited to metadata.
Meanwhile, arguably, for a sysadmin properly following the sysadmin's first rule of backups, that the true value of data isn't defined by arbitrary claims, but by the number of backups it is considered worth the time/trouble/resources to have of that data, it's a known parity-raid risk specifically limited to the corner-case of having an ungraceful shutdown *WHILE* already operating degraded, and as such, it can be managed along with all the other known risks to the data, including admin fat-fingering, the risk that more devices will go out than the array can tolerate, the risk of general bugs affecting the filesystem or other storage-function related code, etc.
IOW, in the context of the admin's first rule of backups, no matter the issue, raid56 write hole or whatever other issue of the many possible, loss of data can *never* be a particularly big issue, because by definition, in *all* cases, what was of most value was saved, either the data if it was defined as valuable enough to have a backup, or the time/ trouble/resources that would have otherwise gone into making that backup, if the data wasn't worth it to have a backup.
(One nice thing about this rule is that it covers the loss of whatever number of backups along with the working copy just as well as it does loss of just the working copy. No matter the number of backups, the value of the data is either worth having one more backup, just in case, or it's not. Similarly, the rule covers the age of the backup and updates nicely as well, as that's just a subset of the original problem, with the value of the data in the delta between the last backup and the working copy now being the deciding factor, either the risk of losing it is worth updating the backup, or not, same rule, applied to a data subset.)
So from an admin's perspective, in practice, while not entirely stable and mature yet, and with the risk of the already-degraded crash-case corner-case that's already known to apply to parity-raid unless mitigation steps are taken, btrfs raid56 mode should now be within the acceptable risk range already well covered by the risk mitigation of following an appropriate backup policy, optionally combined with the partial write-hole-mitigation strategy of doing data as raid5/6, with metadata as raid1.
OK, but what is being done to better mitigate the parity-raid write-hole problem for the future, and when might we be able to use that mitigation?
There are a number of possible mitigation strategies, and there's actually code being written using one of them right now, tho it'll be (at least) a few kernel cycles until it's considered complete and stable enough for mainline, and as mentioned in #2 above, even after that it'll take some time to mature to reasonable stability.
The strategy being taken is partial-stripe-write logging. Full stripe writes aren't affected by the write hole and (AFAIK) won't be logged, but partial stripe writes are read-modify-write and thus write-hole succeptible, and will be logged. That means small files and modifications to existing files, the ends of large files, and much of the metadata, will be written twice, first to the log, then to the final location. In the event of a crash, on reboot and mount, anything in the log can be replayed, thus preventing the write hole.
As for the log, it'll be written using a new 3/4-way-mirroring mode, basically raid1 but mirrored more than two ways (which current btrfs raid1 is limited to, even with more than two devices in the filesystem), thus handling the loss of multiple devices.
That's actually what's being developed ATM, the 3/4-way-mirroring mode, which will be available for other uses as well.
Actually, that's what I'm personally excited about, as years ago, when I first looked into btrfs, I was running older devices in mdraid's raid1 mode, which does N-way mirroring. I liked the btrfs data checksumming and scrubbing ability, but with the older devices I didn't trust having just two-way-mirroring and wanted at least 3-way-mirroring, so back at that time I skipped btrfs and stayed with mdraid. Later I upgraded to ssds and decided btrfs-raid1's two-way-mirroring was sufficient, but when one of the ssds ended up prematurely bad and needed replaced, I would have sure felt a bit better before I got the replacement done, if I still had good two-way-mirroring even with the bad device.
So I'm still interested in 3-way-mirroring and would probably use it for some things now, were it available and "stabilish", and I'm eager to see that code merged, not for the parity-raid logging it'll also be used for, but for the reliability of 3-way-mirroring. Tho I'll probably wait at least 2-5 kernel cycles after introduction and see how it stabilizes before actually considering it stable enough to use myself, because even tho I do follow the backups policy above, just because I'm not considering the updated-data delta worth an updated backup yet, doesn't mean I want to unnecessarily risk having to redo the work since the last backup, which means choosing the newer 3-way-mirroring over the more stable and mature existing raid1 2-way-mirroring isn't going to be worth it to me until the 3-way-mirroring has at least a /few/ kernel cycles to stabilize.
And I'd recommend the same caution with the new raid5/6 logging mode built on top of that multi-way-mirroring, once it's merged as well. Don't just jump on it immediately after merge unless you're deliberately doing so to help test for bugs and get them fixed and the feature stabilized as soon as possible. Wait a few kernel cycles, follow the list to see how the feature's stability is coming, and /then/ use it, after factoring in its remaining then still new and less mature additional risk into your backup risks profile, of course.
Time? Not a dev but following the list and obviously following the new 3- way-mirroring, I'd say probably not 4.20 (5.0?) for the new mirroring modes, so 4.21/5.1 more reasonably likely (if all goes well, could be longer), probably another couple cycles (if all goes well) after that for the parity-raid logging code built on top of the new mirroring modes, so perhaps a year (~5 kernel cycles) to introduction for it. Then wait however many cycles until you think it has stabilized. Call that another year. So say about 10 kernel cycles or two years. It could be a bit less than that, say 5-7 cycles, if things go well and you take it before I'd really consider it stable enough to recommend, but given the historically much longer than predicted development and stabilization times for raid56 already, it could just as easily end up double that, 4-5 years out, too.
But raid56 logging mode for write-hole mitigation is indeed actively being worked on right now. That's what we know at this time.
And even before that, right now, raid56 mode should already be reasonably usable, especially if you do data raid5/6 and metadata raid1, as long as your backup policy and practice is equally reasonable.
Stefan K posted on Tue, 11 Sep 2018 13:29:38 +0200 as excerpted:
Quoted 6 lines
wow, holy shit, thanks for this extended answer!
Quoted 2 lines
The first thing to point out here again is that it's not btrfs-specific.
so that mean that every RAID implemantation (with parity) has such Bug? I'm looking a bit, it looks like that ZFS doesn't have a write hole.
Every parity-raid implementation that doesn't contain specific write-hole workarounds, yes, but some already have workarounds built-in, as btrfs will after the planned code is written/tested/merged/tested-more-broadly.
https://www.google.com/search?q=parity-raid+write-hole [1]
As an example, back some years ago when I was doing raid6 on mdraid, it had the write-hole problem and I remember reading about it at the time. However, right on the first page of hits for the above search...
LWN: A journal for MD/RAID5 : https://lwn.net/Articles/665299/
Seems md/raid5's write hole was (optionally) closed in kernel 4.4 with an optional journal device... preferably a fast ssd or nvram, to avoid performance issues, and mirrored, to avoid the journal itself being a single point of failure.
For me zfs is strictly an arm's-length thing, because if Oracle wanted to they could easily resolve the licensing thing as they own the code, but they haven't, which at this point can only be deliberate, and as I result I simply don't touch it. That isn't to say I don't recommend it for those comfortable with or simply willing to overlook the licensing issues, however, because zfs remains the most mature Linux option for many of the same feature points that btrfs has, only at a lower maturity level.
But while I keep zfs at personal arm's length, from what I've picked up, I /believe/ zfs gets around the write-hole by doing strict copy-on-write combined with variable-length stripes -- unlike current btrfs, a stripe isn't always written as widely as possible, so for instance in a 20- device raid5-alike they're able to do a 3-device and possibly even 2- device "stripe", which then being entirely copy-on-write, avoids the read- modify-write cycle of modified existing data that unless mitigated creates the parity-raid write-hole.
Variable-length stripes is actually one of the possible longer-term solutions already discussed for btrfs as well, but the logging/journalling solution seems to be what they've decided to implement first, and there's other tradeoffs to it (as discussed elsewhere). Of course because as I've already explained I'm interested in the 3/4-way-mirroring option that would be used for the journal but also available to expand the current 2-way-raid1 mode to additional mirroring, this is absolutely fine with me! =:^)
Quoted 4 lines
And
it _only_ happens when the server has a ungraceful shutdown, caused by poweroutage? So that mean if I running btrfs raid5/6 and I've no poweroutages I've no problems?
Sort-of yes?
Keep in mind that power-outage isn't the /only/ way to have an ungraceful shutdown, just one of the most common. Should the kernel crash or lock up for some reason, common examples include video and occasionally network driver bugs due to the direct access to hardware and memory they get, that can trigger an "ungraceful shutdown" as well, altho with care (basically always trying ssh-ing in for a remote shutdown if possible and/ or using alt-sysrq-reisub sequences on apparent lockups) it's often possible to prevent those being /entirely/ ungraceful at the hardware level, so it's not /quite/ as bad as an abrupt power outage or perhaps even worse a brownout that doesn't kill writes entirely but can at least theoretically trigger garbage scribbling in random device blocks.
So yes, sort-of, but it'd not just power-outages.
Quoted 3 lines
Quoted 1 line
it's possible to specify data as raid5/6 and metadata as raid1
does some have this in production?
I'm sure people do. (As I said I'm a raid1 guy here, even 3-way- mirroring for some things were it possible, so no parity-raid at all for me personally.)
On btrfs, it is in fact the multi-device default and thus quite common to have data and metadata as different profiles. The multi-device default for metadata if not specified is raid1, with single profile data. So if you just specify raid5/6 data and don't specify metadata at all, you'll get exactly what was mentioned, raid5/6 data as specified, raid1 metadata as the unspecified multi-device default.
So were I to guess I'd guess that a lot of people who weren't paying attention when setting up but saying they have raid5/6, actually only have it for data, having not specified anything for metadata, so they got raid1 for it.
Quoted 2 lines
ZFS btw have 2 copies of metadata by default, maybe it would also be an option or btrfs?
It actually sounds like they do hybrid raid, then, not just pure parity- raid, but mirroring the metadata as well. That would be in accord with a couple things I'd read about zfs but hadn't quite pursued to the logical conclusion, and would be what btrfs as already available does with raid5/6 data and raid1 metadata.
Quoted 2 lines
in this case you think 'btrfs fi balance start -mconvert=raid1 -dconvert=raid5 /path ' is safe at the moment?
Provided you have backups in accordance with the "if it's more valuable than the time/trouble/resources for the backup, it's backed up" rule, and on current kernels, yes.
Quoted 6 lines
Quoted 3 lines
That means small files and modifications to existing files, the ends of large files, and much of the metadata, will be written twice, first to the log, then to the final location.
that sounds that the performance will go down? So far as I can see btrfs can't beat ext4 or btrfs nor zfs and then they will made it even slower?
That's the effect journaling[2] partial-stripe-writes will have, yes.
However, parity-raid /always/ has a write-performance tradeoff, well either that or a space/organization-tradeoff if it does less than full- width stripes, because traditional parity-raid /already/ has the read- modify-write problem for partial-stripe-width writes (and partial-width- stripe-writes for non-traditional solutions such as zfs a space layout efficiency problem), so lower small-write performance is already a tradeoff you're making choosing parity-raid in the first place, and journaling only accentuates it a bit, as the price paid for closing the write hole.
The performance issue was a big part of the reason I ended up switching from parity-raid to raid1, back in the day on mdraid. And it turned out I was /much/ happier with raid1. which had much better performance than I had thought it would (the mdraid raid1 scheduler is recognized for its high efficiency read-scheduling and for parallel-write scheduling, so write latency is only about the same as writes to a single device, while many or large reads are smart-scheduled to parallelize across all mirrors).
(The other part of the reason I switched back to raid1 on mdraid was because I had rather naively thought on parity-raid the parity would be cross-checked in the standard read path, giving me integrity checking as well. Turns out that's not the case; parity is only used for rebuilds in case of device-loss and isn't checked for normal reads, a great disappointment. That's actually why I'm so looking forward to btrfs 3- and-4-way mirroring, because btrfs already has full checksumming and routine checking on read, for data and metadata integrity, but currently only has two-way-mirroring, so if you're down a device and the copy on the remaining device is bad, you're just out of luck, whereas 3-way- mirroring would let a device be bad and still give me a backup if one of the two remaining copies ended up failing checksum verification. 4-way- mirroring would obviously add yet another copy, but 3-way is the sweet- spot for me.)
The performance is why mdraid recommends putting the journal on a faster device (or better yet mirrors, avoiding the single point of failure of a single journal device), ssd or nvram, for performance reasons, turning a slow-down into a speedup due to the write-cache. But btrfs doesn't have device-purpose-specification like that built-in yet, so it's either all devices, or use something like bcache with an ssd as the front device. (The ssd used as the bcache front device can be partitioned to allow a single ssd to cache multiple slower backing devices.)
OTOH, as stated it's only smaller less-than-stripe-width writes that will be affected. As soon as you're writing more than stripe width, as with large files for data or for metadata when copying whole subdir trees, most of it will be full-stripe-writes and thus shouldn't have to (I'm not sure how it's actually going to be implemented) be logged/journalled.
Meanwhile, at least one of the other alternatives, less-than-full-width- stripes, or writing partly empty full-width-stripes, as necessary, of course with COW so read-modify-write is entirely avoided, will likely eventually be available on btrfs as well. But they have their own tradeoffs, faster initially than the logged/journalled solution, but less efficient initial space utilization, with a clean-up balance likely periodically required to rewrite all the short-stripes (either less than full width or partially empty) to full width. So all possibilities have their tradeoffs, none are a "magic" solution that entirely does away with problems inherent to parity-raid, without tradeoffs of /some/ sort.
But zfs is already (optionally? I don't know) using these tradeoffs, and on mdraid there's options, and people often aren't even aware of the tradeoffs they're taking on those solutions, so... I suppose when it's all said and done the only people aware of the issues on btrfs are likely going to be the highly technical and case-optimizer crowds, too. Everyone else will probably just use the defaults and not even be aware of the tradeoffs they're making by doing so, as is already the case on mdraid and zfs.
---
[1] As I'm no longer running either mdraid or parity-raid, I've not followed this extremely closely, but writing this actually spurred me to google the problem and see when and how mdraid fixed it. So the links are from that. =:^)
[2] Journalling/journaling, one or two Ls? The spellcheck flags both and last I tried googling it the answer was inconclusive.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman