Fixing a degraded RAID
I had a disk in my RAID0 die:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6
Label: none uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6
Total devices 3 FS bytes used 329.66GiB
devid 1 size 1.79TiB used 357.03GiB path /dev/sdd1
devid 3 size 931.51GiB used 178.00GiB path /dev/sdf
*** Some devices missingBtrfs v3.12-43-gc2081e2-dirty
So, I try to mount it with -o degraded. That fails, implying that a degraded mount needs to be read-only. So I successfully mount it read only. Then I want to add a new disk to replace the missing one:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' - Read-only file system
So I can't add a disk if it's mounted read-only, and can't mount it read-write? What am I missing? What's my way around this?
Signature · 2 lines
Michael Welsh Duggan ([email protected])
Quoted 15 lines · Michael Welsh Duggan
I had a disk in my RAID0 die:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Label: none uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Total devices 3 FS bytes used 329.66GiB devid 1 size 1.79TiB used 357.03GiB path /dev/sdd1 devid 3 size 931.51GiB used 178.00GiB path /dev/sdf *** Some devices missingBtrfs v3.12-43-gc2081e2-dirty
So, I try to mount it with -o degraded. That fails, implying that a degraded mount needs to be read-only. So I successfully mount it read only. Then I want to add a new disk to replace the missing one:
Quoted 5 lines
root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' - Read-only file systemSo I can't add a disk if it's mounted read-only, and can't mount it read-write? What am I missing? What's my way around this?
Not really because it's raid0. The minimum number of disks for a 3 disk raid0 is 3 disks. Upon losing one, the data portion of the file system is toast. I'm guessing it lets you mount it ro,degraded because the metadata and hence the file system itself, is raid1 (the default for raid0 data volumes).
The file system could determine what files aren't damaged, i.e they exist entirely on one of the remaining two drives, with no dependency on the dead one. But I don't think this code exists yet. You could possibly rsync the drive to another file system, and bad files damaged by the missing disk will be reported in dmesg while the good ones would still copy over. I haven't tried this.
Chris Murphy
Quoted 29 lines · Chris Murphy
Quoted 15 lines · Michael Welsh Duggan
I had a disk in my RAID0 die:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Label: none uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Total devices 3 FS bytes used 329.66GiB devid 1 size 1.79TiB used 357.03GiB path /dev/sdd1 devid 3 size 931.51GiB used 178.00GiB path /dev/sdf *** Some devices missingBtrfs v3.12-43-gc2081e2-dirty
So, I try to mount it with -o degraded. That fails, implying that a degraded mount needs to be read-only. So I successfully mount it read only. Then I want to add a new disk to replace the missing one:
Quoted 5 lines
root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' - Read-only file systemSo I can't add a disk if it's mounted read-only, and can't mount it read-write? What am I missing? What's my way around this?
Not really because it's raid0. The minimum number of disks for a 3 disk raid0 is 3 disks. Upon losing one, the data portion of the file system is toast. I'm guessing it lets you mount it ro,degraded because the metadata and hence the file system itself, is raid1 (the default for raid0 data volumes).
Quoted 6 lines
The file system could determine what files aren't damaged, i.e they exist entirely on one of the remaining two drives, with no dependency on the dead one. But I don't think this code exists yet. You could possibly rsync the drive to another file system, and bad files damaged by the missing disk will be reported in dmesg while the good ones would still copy over. I haven't tried this.
My mistake. I meant RAID1.
Signature · 2 lines
Michael Welsh Duggan ([email protected])
Quoted 1 line · Michael Welsh Duggan
I had a disk in my RAID0 die:
My mistake. This was a stupid typo. The drives were in a RAID 1 configuration. Both data and metadata are DUP'd.
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/ Data, RAID1: total=353.00GiB, used=328.24GiB System, single: total=32.00MiB, used=56.00KiB Metadata, RAID1: total=4.00GiB, used=1.43GiB
Quoted 18 lines
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Label: none uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Total devices 3 FS bytes used 329.66GiB devid 1 size 1.79TiB used 357.03GiB path /dev/sdd1 devid 3 size 931.51GiB used 178.00GiB path /dev/sdf *** Some devices missingBtrfs v3.12-43-gc2081e2-dirty
So, I try to mount it with -o degraded. That fails, implying that a degraded mount needs to be read-only. So I successfully mount it read only. Then I want to add a new disk to replace the missing one:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' - Read-only file systemSo I can't add a disk if it's mounted read-only, and can't mount it read-write? What am I missing? What's my way around this?
Signature · 2 lines
Michael Welsh Duggan ([email protected])
Quoted 7 lines · Michael Welsh Duggan
My mistake. This was a stupid typo. The drives were in a RAID 1 configuration. Both data and metadata are DUP'd.
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/ Data, RAID1: total=353.00GiB, used=328.24GiB System, single: total=32.00MiB, used=56.00KiB Metadata, RAID1: total=4.00GiB, used=1.43GiB
Somehow system is set to single instead of raid1. I haven't seen system and metadata be set to different profiles. Only 56KB of system data is being used, but I don't know how that's distributed across three drives when the profile is single.
Chris Murphy
Quoted 4 lines · On Fri, 18 Apr 2014 23:23:56 -0400 Michael Welsh Duggan <[email protected]> wrote:
Quoted 1 line · Michael Welsh Duggan
I had a disk in my RAID0 die:
Quoted 29 lines
My mistake. This was a stupid typo. The drives were in a RAID 1 configuration. Both data and metadata are DUP'd.
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/ Data, RAID1: total=353.00GiB, used=328.24GiB System, single: total=32.00MiB, used=56.00KiB Metadata, RAID1: total=4.00GiB, used=1.43GiBQuoted 20 lines
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Label: none uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Total devices 3 FS bytes used 329.66GiB devid 1 size 1.79TiB used 357.03GiB path /dev/sdd1 devid 3 size 931.51GiB used 178.00GiB path /dev/sdf *** Some devices missingBtrfs v3.12-43-gc2081e2-dirty
So, I try to mount it with -o degraded. That fails, implying that a degraded mount needs to be read-only. So I successfully mount it read only. Then I want to add a new disk to replace the missing one:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' - Read-only file systemSo I can't add a disk if it's mounted read-only, and can't mount it read-write? What am I missing? What's my way around this?
As Chris Murphy noted, your "System" chunk still reported as "single" is likely the issue here.
This reminds me of a bug i stumbled upon once:
https://bugzilla.kernel.org/show_bug.cgi?id=60594
Have you converted your fs from SINGLE to RAID-1 profile by running a balance?
This conversion bug should no longer occur with the patch "Btrfs: stop refusing the relocation of chunk 0" which has been merged since 3.12.
But chances are that you've converted your fs with an older kernel without that patch.
While Ilya Dryomov investigated the issue on my system, we managed to work around this by using a patched module that ignored the safety measure refusing to mount rw in this case.
I still have the patch, but i think the safest way and only other option would be to copy the data from your ro fs and recreate it from scratch.
Doing so with recent btrfs progs (v3.12 or newer) you'll benefit from the new default metadata blocksize of 16KB that should increase performance.
Xavier Bassery
Quoted 10 lines · Xavier Bassery
Quoted 6 lines · On Fri, 18 Apr 2014 23:23:56 -0400 Michael Welsh Duggan <[email protected]> wrote:
Michael Welsh Duggan <[email protected]> writes:
root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/ Data, RAID1: total=353.00GiB, used=328.24GiB System, single: total=32.00MiB, used=56.00KiB Metadata, RAID1: total=4.00GiB, used=1.43GiB
Quoted 6 lines
As Chris Murphy noted, your "System" chunk still reported as "single" is likely the issue here. This reminds me of a bug i stumbled upon
once: https://bugzilla.kernel.org/show_bug.cgi?id=60594Have you converted your fs from SINGLE to RAID-1 profile by running a balance?
Yes, I did. And I think I was still on 3.10 or 11 when this happened.
Quoted 7 lines
This conversion bug should no longer occur with the patch "Btrfs: stop refusing the relocation of chunk 0" which has been merged since 3.12. But chances are that you've converted your fs with an older kernel without that patch. While Ilya Dryomov investigated the issue on my system, we managed to work around this by using a patched module that ignored the safety measure refusing to mount rw in this case.
Quoted 5 lines
I still have the patch, but i think the safest way and only other option would be to copy the data from your ro fs and recreate it from scratch. Doing so with recent btrfs progs (v3.12 or newer) you'll benefit from the new default metadata blocksize of 16KB that should increase performance.
Right. *sigh* A pain, but needs must, and all that. Thanks for your help.
Signature · 2 lines
Michael Welsh Duggan ([email protected])