Does btrfs "raid1" actually provide any resilience?
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
Here are the test procedure:
Testing was done using vanilla linux-3.12 (x86_64) plus btrfs-progs at commit c652e4efb8e2dd76ef1627d8cd649c6af5905902.
Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/s
Preparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2
Preparing the btrfs filesystem on the loop devices:
Quoted 6 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Quoted 8 lines
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmp
Copying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/s
Checking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfile
Unmounting before further testing:
Quoted 1 line
# umount /mnt/tmp
Now we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1
Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so... does not work.
In /var/log/messages we find:
Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed
(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowed
The message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmp
The mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output error
The testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Lutz Vieweg
The read only mount issue is by design. It is intended to make sure you know exactly what is going on before you proceed. For example, a drive may actually be fine, but may have been caused by a cable failure. In that case you would want to fix the cable problem before you break the mirror by writing to a single drive. The read only function is designed to make certain you know that you are simplex before you proceed further.
As for the rest of it, hopefully someone else here can shed more light. For sure RAID1 mode works fairly reliably (like traditional RAID1) in a non virtual setting. But it IS still experimental. I am using btrfs RAID1 on my workstation, four partitions spread over five hard drives, but I back everything up 100% multiple times daily via anacron and cron functions. I certainly wouldn't trust it just yet as it is not fully production ready. That said, I have been using it for over six months now, coming off of 3ware RAID, and I have no regrets.
Quoted 134 lines · Lutz Vieweg
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 6 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Quoted 8 lines
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Lutz Vieweg
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · George Mitchell
The read only mount issue is by design. It is intended to make sure you know exactly what is going on before you proceed.
Hmmm... but will a server be able to continue its operation (including writes) on an already mounted btrfs when a storage device in a btrfs-raid1 fails? (If not, that would contradict the idea of achieving a higher reliability.)
Quoted 2 lines
The read only function is designed to make certain you know that you are simplex before you proceed further.
Ok, but once I know - e.g. by verifying that indeed, one storage device is broken - is there any option to proceed (without redundancy) until I can replace the broken device?
Quoted 1 line
I certainly wouldn't trust it just yet as it is not fully production ready.
Sure, the server we intend to try btrfs on is one that we can restore when required, and there is a redundant server (without btrfs) that can stand in. I was just hoping for some good experiences to justify a larger field-trial.
Quoted 2 lines
That said, I have been using it for over six months now, coming off of 3ware RAID, and I have no regrets.
I guess every Linux software RAID option is an improvement when you come from those awful hardware RAID controllers, which caused us additional downtime more often than they prevented downtime.
Regards,
Lutz Vieweg
Quoted 141 lines · Lutz Vieweg
Quoted 125 lines
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 6 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Quoted 8 lines
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Quoted 9 lines
Lutz Vieweg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Lutz Vieweg
Quoted 2 lines · George Mitchell
The read only mount issue is by design. It is intended to make sure you know exactly what is going on before you proceed.
Hmmm... but will a server be able to continue its operation (including writes) on an already mounted btrfs when a storage device in a btrfs-raid1 fails? (If not, that would contradict the idea of achieving a higher reliability.)
Quoted 6 lines
Quoted 2 lines
The read only function is designed to make certain you know that you are simplex before you proceed further.
Ok, but once I know - e.g. by verifying that indeed, one storage device is broken - is there any option to proceed (without redundancy) until I can replace the broken device?
Bonus points if the raid mode is maintained during degraded operation via either dup (2 disk array) or allocating additional chunks (3+ disk array).
Quoted 164 lines
Quoted 1 line
I certainly wouldn't trust it just yet as it is not fully production ready.
Sure, the server we intend to try btrfs on is one that we can restore when required, and there is a redundant server (without btrfs) that can stand in. I was just hoping for some good experiences to justify a larger field-trial.
Quoted 2 lines
That said, I have been using it for over six months now, coming off of 3ware RAID, and I have no regrets.
I guess every Linux software RAID option is an improvement when you come from those awful hardware RAID controllers, which caused us additional downtime more often than they prevented downtime.
Regards,
Lutz Vieweg
Quoted 136 lines · Lutz Vieweg
Quoted 125 lines
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 6 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Quoted 8 lines
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Quoted 9 lines
Lutz Vieweg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Lutz Vieweg
Quoted 2 lines · George Mitchell
The read only mount issue is by design. It is intended to make sure you know exactly what is going on before you proceed.
Hmmm... but will a server be able to continue its operation (including writes) on an already mounted btrfs when a storage device in a btrfs-raid1 fails? (If not, that would contradict the idea of achieving a higher reliability.)
I am pretty sure that a drive dropping out when it is "in service" is handled differently than a drive failing to appear when the system is freshly booted. In the case of an "in service" drive, I believe there would be full transparent redundancy rw.
Quoted 8 lines
Quoted 3 lines
The read only function is designed to make certain you know that you are simplex before you proceed further.
Ok, but once I know - e.g. by verifying that indeed, one storage device is broken - is there any option to proceed (without redundancy) until I can replace the broken device?
Quoted 5 lines
Quoted 1 line
I certainly wouldn't trust it just yet as it is not fully production ready.
Sure, the server we intend to try btrfs on is one that we can restore when required, and there is a redundant server (without btrfs) that can stand in. I was just hoping for some good experiences to justify a larger field-trial.
I waited until April of this year for the same reasons, but decided it WAS ready, as long as one take precautions and doesn't bet the farm on it. Just make sure you don't try to do anything exotic with it (RAID5, etc), its really not ready for that yet. But for vanilla RAID1 it seems to work just fine. I don't really mess with snapshots and such at this point, I run a pretty spartan environment with it. It IS file system RAID so it might have a problem with something that looks like a bogus file, like a file filled with all balls for example. Additionally, as the previous poster mentioned, it is very sensitive to low free space.
Quoted 7 lines
Quoted 3 lines
That said, I have been using it for over six months now, coming off of 3ware RAID, and I have no regrets.
I guess every Linux software RAID option is an improvement when you come from those awful hardware RAID controllers, which caused us additional downtime more often than they prevented downtime.
I went to hardware RAID precisely because soft RAID sucked in my opinion. But btrfs is miles ahead of hardware RAID. There is simply no comparison.
Quoted 154 lines
Regards,
Lutz Vieweg
Quoted 141 lines · Lutz Vieweg
Quoted 134 lines
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 6 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Quoted 8 lines
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Lutz Vieweg
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Lutz Vieweg
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
I don't think that you have misunderstood btrfs. On the basis of my knowledge you are right.
With a kernel v3.11.6 I made your test and I got the following:
- 2 disks of 100M each and 1 file of 70M: I was *unable* to create the file because I got a "No space left on device". I was not surprise BTRFS behaves bad when the free space is low. However I was able to remove a disk and remount the filesystem in "degraded" mode.
- 2 disk of 3G each and 1 file of 100M: I was *able* to create the file, and to remount the filesystem in degraded mode when I deleted a disk.
Note: in any case I needed to mount the filesystem in read-only mode.
I will try also with a 3.12 kernel.
BR
G.Baroncelli
Quoted 126 lines
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 14 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups
Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1Quoted 2 lines
nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Lutz Vieweg
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?
Quoted 22 lines · Goffredo Baroncelli
Quoted 10 lines · Lutz Vieweg
Hi,
on a server that so far uses an MD RAID1 with XFS on it we wanted to try btrfs, instead.
But even the most basic check for btrfs actually providing resilience against one of the physical storage devices failing yields a "does not work" result - so I wonder whether I misunderstood that btrfs is meant to not require block-device level RAID functionality underneath.
I don't think that you have misunderstood btrfs. On the basis of my knowledge you are right.
With a kernel v3.11.6 I made your test and I got the following:
- 2 disks of 100M each and 1 file of 70M: I was *unable* to create the file because I got a "No space left on device". I was not surprise BTRFS behaves bad when the free space is low. However I was able to remove a disk and remount the filesystem in "degraded" mode.
Quoted 6 lines
- 2 disk of 3G each and 1 file of 100M: I was *able* to create the file, and to remount the filesystem in degraded mode when I deleted a disk.
Note: in any case I needed to mount the filesystem in read-only mode.
I will try also with a 3.12 kernel.
Ok, it seems to be a BUG of latest btrfs.mkfs:
If I use the standard debian "mkfs.btrfs":
ghigo@venice:/tmp$ sudo mkfs.btrfs -m raid1 -d raid1 -K /dev/loop[01] WARNING! - Btrfs v0.20-rc1 IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
SMALL VOLUME: forcing mixed metadata/data groups
Created a data/metadata chunk of size 8388608
adding device /dev/loop1 id 2
fs created label (null) on /dev/loop0
nodesize 4096 leafsize 4096 sectorsize 4096 size 202.00MB Btrfs v0.20-rc1 ghigo@venice:/tmp$ sudo mount /dev/loop1 /mnt/test ghigo@venice:/tmp$ sudo btrfs fi df /mnt/test System, RAID1: total=8.00MB, used=4.00KB System: total=4.00MB, used=0.00 Data+Metadata, RAID1: total=64.00MB, used=28.00KB Data+Metadata: total=8.00MB, used=0.00
Note the presence of the profile Data+Metadata RAID1
Instead if I use the btrfs-progs c652e4efb8e2dd7... I got
ghigo@venice:/tmp$ sudo ~ghigo/btrfs/btrfs-progs/mkfs.btrfs -m raid1 -d raid1 -K /dev/loop[01] SMALL VOLUME: forcing mixed metadata/data groups
WARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups Created a data/metadata chunk of size 8388608
adding device /dev/loop1 id 2
fs created label (null) on /dev/loop0
nodesize 4096 leafsize 4096 sectorsize 4096 size 202.00MiB Btrfs v0.20-rc1-591-gc652e4e ghigo@venice:/tmp$ sudo mount /dev/loop1 /mnt/testghigo@venice:/tmp$ sudo btrfs fi df /mnt/test System: total=4.00MB, used=4.00KB Data+Metadata: total=8.00MB, used=28.00KB
Note the absence of any RAID1 profile.
Quoted 131 lines
BR
G.BaroncelliQuoted 126 lines
Here are the test procedure:
Testing was done using vanilla linux-3.12 (
x86_64) plus btrfs-progs at commitc652e4efb8e2dd76ef1627d8cd649c6af5905902.Preparing two 100 MB image files:
Quoted 9 lines
# dd if=/dev/zero of=/tmp/img1 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s# dd if=/dev/zero of=/tmp/img2 bs=1024k count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/sPreparing two loop devices on those images to act as the underlying block devices for btrfs:
Quoted 2 lines
# losetup /dev/loop1 /tmp/img1 # losetup /dev/loop2 /tmp/img2Preparing the btrfs filesystem on the loop devices:
Quoted 14 lines
# mkfs.btrfs --data raid1 --metadata raid1 --label test /dev/loop1 /dev/loop2 SMALL VOLUME: forcing mixed metadata/data groupsWARNING! - Btrfs v0.20-rc1-591-gc652e4e IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
Performing full device TRIM (100.00MiB) ...
Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups
Created a data/metadata chunk of size 8388608
Performing full device TRIM (100.00MiB) ...
adding device/dev/loop2id 2
fs created label test on/dev/loop1Quoted 2 lines
nodesize 4096 leafsize 4096 sectorsize 4096 size 200.00MiBBtrfs v0.20-rc1-591-gc652e4e
Mounting the btfs filesystem:
Quoted 1 line
# mount -t btrfs /dev/loop1 /mnt/tmpCopying just 70MB of zeroes into a test file:
Quoted 4 lines
# dd if=/dev/zero of=/mnt/tmp/testfile bs=1024k count=70 70+0 records in 70+0 records out 73400320 bytes (73 MB) copied, 0.0657669 s, 1.1 GB/sChecking that the testfile can be read:
Quoted 2 lines
# md5sum /mnt/tmp/testfile b89fdccdd61d57b371f9611eec7d3cef /mnt/tmp/testfileUnmounting before further testing:
Quoted 1 line
# umount /mnt/tmpNow we assume that one of the two "storage devices" is broken, so we remove one of the two loop devices:
Quoted 1 line
# losetup -d /dev/loop1Trying to mount the btrfs filesystem from the one storage device that is left:
Quoted 5 lines
# mount -t btrfs -o device=/dev/loop2,degraded /dev/loop2 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so... does not work.
In
/var/log/messageswe find:Quoted 2 lines
kernel: btrfs: failed to read chunk root on loop2
kernel: btrfs:open_ctreefailed(The same happenes when adding ",ro" to the mount options.)
Ok, so if the first of two disks was broken, so is our filesystem. Isn't that what RAID1 should prevent?
We tried a different scenario, now the first disk remains but the second is broken:
Quoted 11 lines
# losetup -d /dev/loop2 # losetup /dev/loop1 /tmp/img1# mount -t btrfs -o degraded /dev/loop1 /mnt/tmp mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soIn
/var/log/messages:
kernel: Btrfs: too many missing devices, writeable mount is not allowedThe message is different, but still unsatisfactory: Not being able to write to a RAID1 because one out of two disks failed is not what one would expect - the machine should be operable just normal with a degraded RAID1.
But let's try if at least a read-only mount works:
Quoted 1 line
# mount -t btrfs -o degraded,ro /dev/loop1 /mnt/tmpThe mount command itself does work.
But then:
Quoted 2 lines
# md5sum /mnt/tmp/testfile md5sum: /mnt/tmp/testfile: Input/output errorThe testfile is not readable anymore. (At this point, no messages are to be found in dmesg/syslog - I would expect such on an input/output error.)
So the bottom line is: All the double writing that comes with RAID1 mode did not provide any usefule resilience.
I am kind of sure this is not as intended, or is it?
Regards,
Lutz Vieweg
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
Hi Anand,
after some tests and looking at the code I discovered that the current mkfs.btrfs doesn't allow any raid profile other than SINGLE for data and meta-data when the mixed metadata/data group is enabled. It seems this behaviour was introduce by a your commit [1].
mkfs.c line 1384 onwards
/*
* Set default profiles according to number of added devices.
* For mixed groups defaults are single/single.
*/
if (!mixed) {
[....]
} else {
u32 best_leafsize = max_t(u32, sysconf(_SC_PAGESIZE),
sectorsize);
metadata_profile = 0;
data_profile = 0;But in another your commit [2] it seems that you check that in case of mixed, the metadata and data profile have to be equal (implicitly allowing that they could be different than single ?).
mkfs.c line 1373 onward
if (is_vol_small(file)) {
printf("SMALL VOLUME: forcing mixed metadata/data
groups\n");
mixed = 1;
if (metadata_profile != data_profile) {
if (metadata_profile_opt || data_profile_opt) {
fprintf(stderr, "With mixed block
groups data and metadata profiles must be the same\n");
exit(1);
}
}
}So I am a bit confusing: it is allowed a raid profile different than single when the mixed is enabled ? Of course mixed and raid together doesn't make sense (or almost make very little sense) but the code of mkfs is a bit confused, and a warning should be raised when the raid profile are forced to a default different from the one selected by the user.
Thanks for the attention.
BR
G.Baroncelli
[1] btrfs-progs: avoid write to the disk before sure to create fs 71d6bd3c8d70fb682c7fd50796f587ce1f1cf6f8
.
[2] btrfs-progs: mkfs should check for small vol well before cdbc10729266c03aeb2eb812c17a3ef6c1ceae26
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
Hi G.Baroncelli, Lutz,
Thanks for the test case and heads-up on this. The code missed the check if the user has provided the option before default profile for the mixed group (due to small vol) is enforced.
I have sent out the following patch to fix it.
[PATCH] btrfs-progs: for mixed group check opt before default raid profile is enforced
Kindly let us know how it performed if you could.
Thanks,
Anand
Quoted 61 lines · Goffredo Baroncelli
Hi Anand,
after some tests and looking at the code I discovered that the current mkfs.btrfs doesn't allow any raid profile other than SINGLE for data and meta-data when the mixed metadata/data group is enabled. It seems this behaviour was introduce by a your commit [1].
mkfs.cline 1384 onwards/* * Set default profiles according to number of added devices. * For mixed groups defaults are single/single. */ if (!mixed) { [....] } else { u32 best_leafsize = max_t(u32, sysconf(_SC_PAGESIZE), sectorsize); metadata_profile = 0; data_profile = 0;But in another your commit [2] it seems that you check that in case of mixed, the metadata and data profile have to be equal (implicitly allowing that they could be different than single ?).
mkfs.cline 1373 onwardif (is_vol_small(file)) { printf("SMALL VOLUME: forcing mixed metadata/data groups\n"); mixed = 1; if (metadata_profile != data_profile) { if (metadata_profile_opt || data_profile_opt) { fprintf(stderr, "With mixed block groups data and metadata profiles must be the same\n"); exit(1); } } }So I am a bit confusing: it is allowed a raid profile different than single when the mixed is enabled ? Of course mixed and raid together doesn't make sense (or almost make very little sense) but the code of mkfs is a bit confused, and a warning should be raised when the raid profile are forced to a default different from the one selected by the user.
Thanks for the attention.
BR
G.Baroncelli[1] btrfs-progs: avoid write to the disk before sure to create fs
71d6bd3c8d70fb682c7fd50796f587ce1f1cf6f8
.
[2] btrfs-progs: mkfs should check for small vol well beforecdbc10729266c03aeb2eb812c17a3ef6c1ceae26
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
Quoted 9 lines · Anand Jain
Thanks for the test case and heads-up on this. The code missed the check if the user has provided the option before default profile for the mixed group (due to small vol) is enforced.
I have sent out the following patch to fix it.
[PATCH] btrfs-progs: for mixed group check opt before default raid profile is enforced
Kindly let us know how it performed if you could.
I just tried it: The test case I posted now works as expected.
Thanks a lot for your effort!
(The patch did not apply to the current head of
git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git without manual intervention, but was easy to fix.)
Regards,
Lutz Vieweg
PS: Will now proceed with some less basic resilience tests... ;-)
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
In reply to Anand JainQuoted 9 lines · Anand Jain
Thanks for the test case and heads-up on this. The code missed the check if the user has provided the option before default profile for the mixed group (due to small vol) is enforced.
I have sent out the following patch to fix it.
[PATCH] btrfs-progs: for mixed group check opt before default raid profile is enforced
Kindly let us know how it performed if you could.
I just tried it: The test case I posted now works as expected.
Thanks a lot for your effort!
(The patch did not apply to the current head of
git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git without manual intervention, but was easy to fix.)
Regards,
Lutz Vieweg
PS: Will now proceed with some less basic resilience tests... ;-)
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
In reply to Goffredo BaroncelliGoffredo Baroncelli posted on Thu, 14 Nov 2013 22:21:22 +0100 as excerpted:
Quoted 3 lines
after some tests and looking at the code I discovered that the current mkfs.btrfs doesn't allow any raid profile other than SINGLE for data and meta-data when the mixed metadata/data group is enabled.
That'd be a big problem for me, here, as I run a separate sub-GiB (640 MiB) btrfs filesystem /var/log, in data+metadata raid1 mode. (The mountpoint is actually /lg, with /var/log a symlink pointing at it.)
btrfs f sh /lg
Label: lg0238gcnx+35l0 uuid: c77a9eb8-9841-4c2b-925e-75d0a925dcc3
Total devices 2 FS bytes used 51.93MiB
devid 1 size 640.00MiB used 288.00MiB path /dev/sdc4
devid 2 size 640.00MiB used 288.00MiB path /dev/sda4btrfs f df /lg
System, RAID1: total=32.00MiB, used=4.00KiB Data+Metadata, RAID1: total=256.00MiB, used=51.92MiB
I've had a couple bad shutdowns, but btrfs scrub has reliably cleaned up the resulting mess on /lg due to open logfiles, and I'd be rather unhappy if that weren't possible.
Meanwhile, I also have two separate sub-GiB (256 MiB) /boot filesystems, one on each (of two) SSDs, with the grub2 on each pointing at its own /boot so a broken grub2 update won't break my ability to boot.
Those are both data+metadata dup mode:
btrfs f df /bt
System, DUP: total=8.00MiB, used=4.00KiB System, single: total=4.00MiB, used=0.00 Data+Metadata, DUP: total=114.00MiB, used=41.38MiB Data+Metadata, single: total=8.00MiB, used=0.00
You're saying data+metadata DUP wouldn't be possible here either, which would make me pretty unhappy too.
Fortunately I did those mkfs.btrfs on an earlier btrfs-tools so wasn't affected by this bug, but bug I would indeed call it, for sure!
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
Quoted 11 lines · Duncan
Goffredo Baroncelli posted on Thu, 14 Nov 2013 22:21:22 +0100 as excerpted:
Quoted 3 lines
after some tests and looking at the code I discovered that the current mkfs.btrfs doesn't allow any raid profile other than SINGLE for data and meta-data when the mixed metadata/data group is enabled.
That'd be a big problem for me, here, as I run a separate sub-GiB (640 MiB) btrfs filesystem
/var/log, in data+metadata raid1 mode. (The mountpoint is actually /lg, with/var/loga symlink pointing at it.)
[...]
Quoted 3 lines
You're saying data+metadata DUP wouldn't be possible here either, which would make me pretty unhappy too.
The problem should be in mkfs.btrfs not in the btrfs kernel code. So if the filesystem was created, there should be not problem.
Quoted 3 lines
Fortunately I did those mkfs.btrfs on an earlier btrfs-tools so wasn't affected by this bug, but bug I would indeed call it, for sure!
Anand posted a patch few hours ago (which worked for me). I think that this bug could be addressed quickly.
BR
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Re: Mixed and raid [was Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?]
Goffredo Baroncelli posted on Fri, 15 Nov 2013 08:30:49 +0100 as excerpted:
Quoted 9 lines · Duncan
Quoted 1 line
[...]
Quoted 3 lines
You're saying data+metadata DUP wouldn't be possible here either, which would make me pretty unhappy too.
The problem should be in mkfs.btrfs not in the btrfs kernel code. So if the filesystem was created, there should be not problem.
Yes. Thanks.
Quoted 5 lines
Quoted 2 lines
Fortunately I did those mkfs.btrfs on an earlier btrfs-tools so wasn't affected by this bug, but bug I would indeed call it, for sure!
Anand posted a patch few hours ago (which worked for me). I think that this bug could be addressed quickly.
His patch crossed with my reply. Quick work, indeed! =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?
In reply to Goffredo BaroncelliQuoted 8 lines · Goffredo Baroncelli
Instead if I use the btrfs-progs
c652e4efb8e2dd7... I got[snip]
Data+Metadata: total=8.00MB, used=28.00KB
Note the absence of any RAID1 profile.
What happens if the devices are large enough to avoid mandatory block group mixing? Try 100GB for each device. Is the problem reproducible?
Chris Murphy
Re: BUG: btrfsRe: Does btrfs "raid1" actually provide any resilience?
Quoted 5 lines · Chris Murphy
Quoted 3 lines · Goffredo Baroncelli
Instead if I use the btrfs-progs
c652e4efb8e2dd7... I got
Quoted 7 lines
Quoted 5 lines
[snip]
Data+Metadata: total=8.00MB, used=28.00KB
Note the absence of any RAID1 profile.
What happens if the devices are large enough to avoid mandatory block group mixing? Try 100GB for each device. Is the problem reproducible?
It seems related to the mixing (see my other email). Even looking at the code and doing some tests seems to confirm that.
Quoted 2 lines
Chris Murphy--
Goffredo
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5