Trying to mount RAID1 degraded with removed disk -> open_ctree failed
Hi,
I have setup a RAID1 using 3 devices (500G each) on separate disks. After removing one disk physically the filesystem cannot be mounted in degraded nor in recovery mode. When putting the disk back in the filesystem can be mounted without errors. I did a cold-swap (powercycle after removal/insertion of the disk).
Here are the details:
- latest kernel 3.2.1 and btrfs-tools on xubuntu 11.10
# uname -a Linux sol2 3.2.1-030201-generic #201201121644 SMP Thu Jan 12 21:45:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
# btrfs device scan
# btrfs fi show btrfs-r1b
Label: 'btrfs-r1b' uuid: 11ac3be1-62ce-4b1c-8069-63942f1354bd
Total devices 3 FS bytes used 206.60GB
devid 3 size 465.66GB used 140.00GB path /dev/sdc1
devid 2 size 465.66GB used 139.02GB path /dev/sda2
*** Some devices missing# mount -o degraded /r1b mount: /dev/sdc1: can't read superblock [ 2156.393337] device label btrfs-r1b devid 3 transid 2441 /dev/sdc1 [ 2156.407816] btrfs: allowing degraded mounts [ 2156.770745] parent transid verify failed on 448316575744 wanted 2427 found 2425 [ 2156.770762] parent transid verify failed on 448316575744 wanted 2427 found 2425 [ 2156.770769] parent transid verify failed on 448316575744 wanted 2427 found 2425 [ 2156.770774] parent transid verify failed on 448316575744 wanted 2427 found 2425 [ 2156.784722] btrfs warning page private not zero on page 448229711872 [ 2156.784842] btrfs: open_ctree failed
# mount -o recovery /r1b [ 2286.282041] device label btrfs-r1b devid 3 transid 2441 /dev/sdc1 [ 2286.282664] btrfs: enabling auto recovery [ 2286.284709] btrfs: failed to read chunk tree on sda2 [ 2286.285045] btrfs: open_ctree failed
# find-root /dev/sda2 warning devid 1 not found already Failed to read: Illegal seek warning, device 1 is missing warning, device 1 is missing warning, device 1 is missing Super think's the tree root is at 448316571648, chunk root 20983808
# btrfsck /dev/sda2 warning devid 1 not found already parent transid verify failed on 448316575744 wanted 2427 found 2425 parent transid verify failed on 448316575744 wanted 2427 found 2425 Ignoring transid failure Check tree block failed, want=448316588032, have=11529940786129179172 Check tree block failed, want=448316588032, have=11529940786129179172 Check tree block failed, want=448316588032, have=11529940786129179172 read block failed check_tree_block
What is happening? RAID1 should be mountable degraded with one missing/removed device.
-Dirk
Dirk Lutzebaeck posted on Sun, 22 Jan 2012 16:05:14 +0100 as excerpted:
Quoted 8 lines
I have setup a RAID1 using 3 devices (500G each) on separate disks. After removing one disk physically the filesystem cannot be mounted in degraded nor in recovery mode.
- latest kernel 3.2.1 and btrfs-tools on xubuntu 11.10
What is happening? RAID1 should be mountable degraded with one missing/removed device.
Note that I'm only researching btrfs for my own systems at this point and am not using it yet. However, because I *AM* researching it and already read thru most of the wiki documentation, it's fresh in mind.
Here's what the wiki says, tho of course it could be outdated:
https://btrfs.wiki.kernel.org/
Quoted 4 lines
From the multiple devices page:
Quoted 2 lines
By default, metadata will be mirrored across two devices and data will be striped across all of the devices present.
Question: Did you specify -m raid1 -d raid1 when you did the mkfs.btrfs?
While the -m raid1 would be the default given multiple devices, the -d raid1 is not. If you didn't specify -d raid1, you'll have raid0/striped data with only the metadata being raid1/mirrored, thus explaining the problem.
At least with all devices present, the following should show the raid level actually used (from the use cases page):
Quoted 6 lines
Quoted 6 lines
On a 2.6.37 or later kernel, use
btrfs fi df /mountpoint
The required support was broken accidentally in earlier kernels, but has now been fixed.
Also note since you're running a 3-device btrfs-raid-1, tho it shouldn't affect a single device dropout, from the sysadmin guide page (near the bottom of the raid and data replication section):
Quoted 3 lines
Quoted 3 lines
With RAID-1 and RAID-10, only two copies of each byte of data are written, regardless of how many block devices are actually in use on the filesystem.
IOW, unlike standard or kernel/md raid-1, that 3-device btrfs-raid-1 will **NOT** protect you if two of the three devices go bad before you've had a chance to bring in and balance to a replacement for the first bad device.
As I said I'm just now researching my own btrfs upgrade, and don't know for sure whether that's true or not, but if it is, it's a HUGE negative for me, as I'm currently running 4-way kernel/md RAID-1 on an aging set of drives, and was hoping to upgrade to btrfs raid-1 for the checksummed integrity. But given the age of the drives I really don't want to drop below dual redundancy (3 copies), and this two-copies-only (single redundancy) raid-1(-ish) no matter the number of devices, is disappointing indeed!
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman