Btrfs is amazing! (a lack-of-bug report)

I just wanted to drop you guys a line to say that I am stunned with how excellent btrfs is. I did some testing, and the things that it did were amazing. I took a 4-disk RAID 5 and walked it all the way down to a one-disk volume and back again, mixed in devices of different sizes in different modes, balanced it in every direction, trashed data on drives without the OS knowing, and did every other form of torture I could think of, all while looping file integrity tests, and it was perfect.

The ease of use and simplicity were great. I was dreading having to administer ZFS in order to get snapshots and other features, but now I don't have to. With the exception of enterprisey features like SSD intent logs and stuff, it is hands down far better than ZFS.

Thanks for the great work!

Quoted 8 lines · Tyler Bletsch

I just wanted to drop you guys a line to say that I am stunned with how excellent btrfs is. I did some testing, and the things that it did were amazing. I took a 4-disk RAID 5 and walked it all the way down to a one-disk volume and back again, mixed in devices of different sizes in different modes, balanced it in every direction, trashed data on drives without the OS knowing, and did every other form of torture I could think of, all while looping file integrity tests, and it was perfect.

Quoted 7 lines

The ease of use and simplicity were great. I was dreading having to administer ZFS in order to get snapshots and other features, but now I don't have to. With the exception of enterprisey features like SSD intent logs and stuff, it is hands down far better than ZFS.

Thanks for the great work!

It's nice to hear success stories for once. I would suggest being careful of using BTRFS raid5 or raid6 in production, there are probably still bugs that haven't yet been discovered. Secondarily, if you can deal with slightly more setup and maintenance overhead, BTRFS works _very_ well on top of LVM (it makes online data migration much easier, and provides easy ways to do layered RAID setups).

Thanks. I will be trying raid5 in production, but "production" in this case just means my home file server, with btrfs snapshot+sync for all data and appropriate offsite non-btrfs backups for critical data. If it hoses up, I'll post a bug report.

Going to try to avoid LVM, since half the appeal of btrfs for me is getting away from the multiple duct-taped layers of indirection that I you get currently with ext4/MD/LVM setups.

Quoted 4 lines · Tyler Bletsch

Thanks. I will be trying raid5 in production, but "production" in this case just means my home file server, with btrfs snapshot+sync for all data and appropriate offsite non-btrfs backups for critical data. If it hoses up, I'll post a bug report.

So far, that's been my use case for btrfs raid6, and (barring one bug I found involving an interaction between btrfs and thinly provisioned storage that shouldn't be an issue for you if you're not using LVM thin pools) I have yet to hit any bugs, although based on the lists, I've probably been _very_ lucky in that respect. If you're willing to take a marginal performance hit (about 1-3% in my experience, which is notably less than the performance difference between MD-RAID5 and MD-RAID6), and have at least four disks, I'd suggest using btrfs's raid6 profile instead of raid5, they use exactly the same code, it's just that raid6 has one more calculation involved and provides better protection against data corruption due to the double parity.

Quoted 4 lines

Going to try to avoid LVM, since half the appeal of btrfs for me is getting away from the multiple duct-taped layers of indirection that I you get currently with ext4/MD/LVM setups.

Understandable, my main reasons for having LVM are storing virtual machine disk images and the fact that MD/DM raid is still ridiculously fast compared to btrfs raid (many of my btrfs raid volumes are themselves on top of LVM managed software raid0 or raid1 volumes).

Thanks. I'd consider raid6, but since I'll be backing up to a second btrfs raid5 array, I think I have sufficient redundancy, since equivalent to raid 5+1 on paper. I'm doing that rather than something like raid10 in a single box because I want the redundancy of a second physical server so I can failover in the event of a system-level component failure.

(And of course, "failover" means "continue being able to watch TV shows and stuff")

A question about what you said -- when you say people have hit bugs in the raid56 code, which flavor do these bugs tend to be? Are they "minding my own business and suddenly it falls over" bugs or "I tried to do something weird with btrfs and it screwed up" bugs?

Quoted 7 lines · Tyler Bletsch

Thanks. I'd consider raid6, but since I'll be backing up to a second btrfs raid5 array, I think I have sufficient redundancy, since equivalent to raid 5+1 on paper. I'm doing that rather than something like raid10 in a single box because I want the redundancy of a second physical server so I can failover in the event of a system-level component failure.

Quoted 7 lines

(And of course, "failover" means "continue being able to watch TV shows and stuff")

A question about what you said -- when you say people have hit bugs in the raid56 code, which flavor do these bugs tend to be? Are they "minding my own business and suddenly it falls over" bugs or "I tried to do something weird with btrfs and it screwed up" bugs?

More along the lines of 'I tried to do something that works fine with the other raid profiles and it kind of messed up the filesystem'. In general, you should be safe as long as you are using at least Linux 4.0 and the most recent version of btrfs-progs. It's been a while since I saw any raid56 related bugs that caused actual data loss. If you are using this on SSD's though, I would wait, there are known issues with DISCARD/TRIM not working correctly on btrfs right now (nothing involving data loss, just problems with it not properly trimming free space and therefore causing issues with wear-leveling), and it looks like the fix won't be in 4.2 as of right now.

Raid56 works fine until you have a drive with problems which really means it doesn't work because you only use parity to handle the case of a drive with problems.

Maintenance procedures such as scrubs are also a magnitude of order slower than the other raid profiles.

I would use the raid10 profile on at least one of your pools.

On Aug 20, 2015 7:03 AM, "Austin S Hemmelgarn" <[email protected] <mailto:[email protected]>> wrote:

On 2015-08-20 07:52, Austin S Hemmelgarn wrote:

On 2015-08-19 13:24, Tyler Bletsch wrote:

Thanks. I'd consider raid6, but since I'll be backing up to a second
btrfs raid5 array, I think I have sufficient redundancy, since equivalent to raid 5+1 on paper. I'm doing that rather than something
like raid10 in a single box because I want the redundancy of a second
physical server so I can failover in the event of a system-level
component failure.

(And of course, "failover" means "continue being able to watch TV shows
and stuff")

A question about what you said -- when you say people have hit bugs in
the raid56 code, which flavor do these bugs tend to be? Are they
"minding my own business and suddenly it falls over" bugs or "I tried to
do something weird with btrfs and it screwed up" bugs?

More along the lines of 'I tried to do something that works fine with
the other raid profiles and it kind of messed up the filesystem'. In
general, you should be safe as long as you are using at least Linux 4.0
and the most recent version of btrfs-progs. It's been a while since I
saw any raid56 related bugs that caused actual data loss. If you are
using this on SSD's though, I would wait, there are known issues with
DISCARD/TRIM not working correctly on btrfs right now (nothing involving
data loss, just problems with it not properly trimming free space and
therefore causing issues with wear-leveling), and it looks like the fix
won't be in 4.2 as of right now.

On second thought, you might want to wait until 4.3, I just saw this thread:
http://thread.gmane.org/gmane.comp.file-systems.btrfs/47321/focus=47325