degraded BTRFS RAID 1 not mountable: open_ctree failed, unable to find block group for 0

Hello!

A degraded BTRFS RAID 1 from one 3TB SATA HDD of my former workstation is not mountable.

Debian 4.8 kernel + btrfs-tools 4.7.3.

A btrfs restore seems to work well enough, so on one hand there is no urgency. But on the other hand I want to repurpose the harddisk and I think I want to do it next weekend. So if you want me to gather some debug data, please speak up quickly. Thank you.

AFAIR I have been able to mount the filesystems in degraded mode, but this may have been on the other SATA HDD that I already wiped with shred command.

I have this:

merkaba:~> btrfs fi sh
[…]
warning, device 2 is missing
warning, device 2 is missing
warning, device 2 is missing
Label: 'debian' uuid: […]
Total devices 2 FS bytes used 20.10GiB
devid 1 size 50.00GiB used 29.03GiB path /dev/mapper/satafp1-debian
*** Some devices missing

Label: 'daten' uuid: […]
Total devices 2 FS bytes used 135.02GiB
devid 1 size 1.00TiB used 142.06GiB path /dev/mapper/satafp1-daten
*** Some devices missing

Label: 'backup' uuid: […]
Total devices 2 FS bytes used 88.38GiB
devid 1 size 1.00TiB used 93.06GiB path /dev/mapper/satafp1-backup
*** Some devices missing

But none of these filesystems seem to be mountable. Here some attempts:

merkaba:~#130> LANG=C mount -o degraded /dev/satafp1/backup /mnt/zeit
mount: wrong fs type, bad option, bad superblock on /dev/mapper/satafp1-daten,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
merkaba:~> dmesg | tail -5
[ 2945.155943] BTRFS info (device dm-13): allowing degraded mounts
[ 2945.155953] BTRFS info (device dm-13): disk space caching is enabled
[ 2945.155957] BTRFS info (device dm-13): has skinny extents
[ 2945.611236] BTRFS warning (device dm-13): missing devices (1) exceeds the limit (0), writeable mount is not allowed [ 2945.646719] BTRFS: open_ctree failed

merkaba:~> LANG=C mount -o usebackuproot /dev/satafp1/daten /mnt/zeit
mount: wrong fs type, bad option, bad superblock on /dev/mapper/satafp1-daten,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
merkaba:~#32> dmesg | tail -5
[ 5739.051433] BTRFS info (device dm-12): trying to use backup root at mount time [ 5739.051441] BTRFS info (device dm-12): disk space caching is enabled [ 5739.051444] BTRFS info (device dm-12): has skinny extents
[ 5739.103153] BTRFS error (device dm-12): failed to read chunk tree: -5 [ 5739.130304] BTRFS: open_ctree failed

merkaba:~> LANG=C mount -o degraded,usebackuproot /dev/satafp1/daten /mnt/zeit
mount: wrong fs type, bad option, bad superblock on /dev/mapper/satafp1-daten,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
merkaba:~#32> dmesg | tail -5
[ 5801.704202] BTRFS info (device dm-12): trying to use backup root at mount time
[ 5801.704206] BTRFS info (device dm-12): disk space caching is enabled
[ 5801.704208] BTRFS info (device dm-12): has skinny extents
[ 5803.928059] BTRFS warning (device dm-12): missing devices (1) exceeds the limit (0), writeable mount is not allowed [ 5804.064638] BTRFS: open_ctree failed

btrfs check reports:

merkaba:~#32> btrfs check /dev/satafp1/backup warning, device 2 is missing
Checking filesystem on /dev/satafp1/backup
UUID: 01cf0493-476f-42e8-8905-61ef205313db checking extents
checking free space cache
failed to load free space cache for block group 58003030016 failed to load free space cache for block group 60150513664 failed to load free space cache for block group 62297997312 […]
checking fs roots
^C

I aborted it at this time as I wanted to try clear_cache mount option after seeing this. I can redo this thing after btrfs restore completed.

merkaba:~> mount -o degraded,clear_cache /dev/satafp1/backup /mnt/zeit
mount: Falscher Dateisystemtyp, ungültige Optionen, der Superblock von /dev/mapper/satafp1-backup ist beschädigt, fehlende Kodierungsseite oder ein anderer Fehler

Manchmal liefert das Systemprotokoll wertvolle Informationen –
versuchen Sie dmesg | tail oder ähnlich
merkaba:~#32> dmesg | tail -6
[ 3080.120687] BTRFS info (device dm-13): allowing degraded mounts
[ 3080.120699] BTRFS info (device dm-13): force clearing of disk cache
[ 3080.120703] BTRFS info (device dm-13): disk space caching is enabled
[ 3080.120706] BTRFS info (device dm-13): has skinny extents
[ 3080.150957] BTRFS warning (device dm-13): missing devices (1) exceeds the limit (0), writeable mount is not allowed [ 3080.195941] BTRFS: open_ctree failed

merkaba:~> mount -o degraded,clear_cache,usebackuproot /dev/satafp1/backup /mnt/zeit
mount: Falscher Dateisystemtyp, ungültige Optionen, der
Superblock von /dev/mapper/satafp1-backup ist beschädigt, fehlende Kodierungsseite oder ein anderer Fehler

Manchmal liefert das Systemprotokoll wertvolle Informationen – versuchen Sie dmesg | tail oder ähnlich

merkaba:~> dmesg | tail -7
[ 3173.784713] BTRFS info (device dm-13): allowing degraded mounts
[ 3173.784728] BTRFS info (device dm-13): force clearing of disk cache
[ 3173.784737] BTRFS info (device dm-13): trying to use backup root at mount time
[ 3173.784742] BTRFS info (device dm-13): disk space caching is enabled
[ 3173.784746] BTRFS info (device dm-13): has skinny extents
[ 3173.816983] BTRFS warning (device dm-13): missing devices (1) exceeds the limit (0), writeable mount is not allowed [ 3173.865199] BTRFS: open_ctree failed

I aborted repairing after this assert:

    merkaba:~#130> btrfs check --repair /dev/satafp1/backup &| stdbuf -oL tee btrfs-check-repair-satafp1-backup.log
    enabling repair mode
    warning, device 2 is missing
    Checking filesystem on /dev/satafp1/backup
    UUID: 01cf0493-476f-42e8-8905-61ef205313db
    checking extents
    Unable to find block group for 0
    extent-tree.c:289: find_search_start: Assertion `1` failed.
    btrfs[0x43e418]
    btrfs(btrfs_reserve_extent+0x5c9)[0x4425df]
    btrfs(btrfs_alloc_free_block+0x63)[0x44297c]
    btrfs(__btrfs_cow_block+0xfc)[0x436636]
    btrfs(btrfs_cow_block+0x8b)[0x436bd8]
    btrfs[0x43ad82]
    btrfs(btrfs_commit_transaction+0xb8)[0x43c5dc]
    btrfs[0x4268b4]
    btrfs(cmd_check+0x1111)[0x427d6d]
    btrfs(main+0x12f)[0x40a341]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fb2e6bec2b1]
    btrfs(_start+0x2a)[0x40a37a]

    merkaba:~#130> btrfs --version
    btrfs-progs v4.7.3

(Honestly I think asserts like this need to be gone from btrfs-tools for good)

About this I only found this unanswered mailing list post:

btrfs-convert: Unable to find block group for 0
Date: Fri, 24 Jun 2016 11:09:27 +0200
https://www.spinics.net/lists/linux-btrfs/msg56478.html

Out of curiosity I tried:

    merkaba:~#1> btrfs rescue zero-log //dev/satafp1/daten
    warning, device 2 is missing
    Clearing log on //dev/satafp1/daten, previous log_root 0, level 0
    Unable to find block group for 0
    extent-tree.c:289: find_search_start: Assertion `1` failed.
    btrfs[0x43e418]
    btrfs(btrfs_reserve_extent+0x5c9)[0x4425df]
    btrfs(btrfs_alloc_free_block+0x63)[0x44297c]
    btrfs(__btrfs_cow_block+0xfc)[0x436636]
    btrfs(btrfs_cow_block+0x8b)[0x436bd8]
    btrfs[0x43ad82]
    btrfs(btrfs_commit_transaction+0xb8)[0x43c5dc]
    btrfs[0x42c0d4]
    btrfs(main+0x12f)[0x40a341]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fb2f16a82b1]
    btrfs(_start+0x2a)[0x40a37a]

(I didn´t expect much as this is an issue that AFAIK does not happen easily anymore, but I also thought it could not do much harm)

Superblocks themselves seem to be sane:

merkaba:~#1> btrfs rescue super-recover //dev/satafp1/daten All supers are valid, no need to recover

So "btrfs restore" it is:

merkaba:[…]> btrfs restore -mxs /dev/satafp1/daten daten-restore

This prints out a ton of:

Trying another mirror
Trying another mirror

But it actually works. Somewhat, I now just got

Trying another mirror
We seem to be looping a lot on daten-restore/[…]/virtualbox-4.1.18-dfsg/out/lib/vboxsoap.a, do you want to keep going on ? (y/N/a):

after about 35 GiB of data restored. I answered no to this one and now it is at about 53 GiB already. I just got another one of these, but also not concerning a file I actually need.

Thanks,

Martin Steigerwald  | Trainer

teamix GmbH
Südwestpark 43
90449 Nürnberg

Tel.:  +49 911 30999 55 | Fax: +49 911 30999 99
mail: [email protected] | web:  http://www.teamix.de | blog: http://blog.teamix.de

Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller

teamix Support Hotline: +49 911 30999-112
 
 *** Bitte liken Sie uns auf Facebook: facebook.com/teamix ***

I have to wonder did you read the above message? What you need at this point is simply "-o degraded,ro". But I don't see that tried anywhere down the line.

See also (or try): https://patchwork.kernel.org/patch/9419189/

With respect,
Roman

Am Mittwoch, 16. November 2016, 15:43:36 CET schrieb Roman Mamedov:

Actually I read that one, but I read more into it than what it was saying:

I read into it that BTRFS would automatically use a read only mount.

merkaba:~> mount -o degraded,ro /dev/satafp1/daten /mnt/zeit

actually really works. *Thank you*, Roman.

I do think that above kernel messages invite such a kind of interpretation tough. I took the "BTRFS: open_ctree failed" message as indicative to some structural issue with the filesystem.

So mounting work although for some reason scrubbing is aborted (I had this issue a long time ago on my laptop as well). After removing /var/lib/btrfs scrub status file for the filesystem:

merkaba:~> btrfs scrub start /mnt/zeit
scrub started on /mnt/zeit, fsid […] (pid=9054)
merkaba:~> btrfs scrub status /mnt/zeit
scrub status for […]
scrub started at Wed Nov 16 11:52:56 2016 and was aborted after
00:00:00
total bytes scrubbed: 0.00B with 0 errors

Anyway, I will now just rsync off the files.

Interestingly enough btrfs restore complained about looping over certain files… lets see whether the rsync or btrfs send/receive proceeds through.

Ciao,

Martin Steigerwald  | Trainer

teamix GmbH
Südwestpark 43
90449 Nürnberg

Tel.:  +49 911 30999 55 | Fax: +49 911 30999 99
mail: [email protected] | web:  http://www.teamix.de | blog: http://blog.teamix.de

Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller

teamix Support Hotline: +49 911 30999-112
 
 *** Bitte liken Sie uns auf Facebook: facebook.com/teamix ***

For the reason as to why the writable mount didn't work, check "btrfs fi df" for the filesystem to see if you have any "single" profile chunks on it: quite likely you did already mount it "degraded,rw" in the past *once*, after which those "single" chunks get created, and consequently it won't mount r/w anymore (without lifting the restriction on the number of missing devices as proposed).

With respect,
Roman

Am Mittwoch, 16. November 2016, 16:00:31 CET schrieb Roman Mamedov:

That exactly explains it. I very likely did a degraded mount without ro on this disk already.

Funnily enough this creates another complication:

merkaba:/mnt/zeit#1> btrfs send somesubvolume | btrfs receive /mnt/
someotherbtrfs
ERROR: subvolume /mnt/zeit/somesubvolume is not read-only

Yet:

merkaba:/mnt/zeit> btrfs property get somesubvolume ro=false
merkaba:/mnt/zeit> btrfs property set somesubvolume ro true
ERROR: failed to set flags for somesubvolume: Read-only file system

To me it seems right logic would be to allow the send to proceed in case the whole filesystem is readonly.

As there seems to be no force option to override the limitation and I do not feel like compiling my own btrfs-tools right now, I will use rsync instead.

Thanks,

Martin Steigerwald  | Trainer

teamix GmbH
Südwestpark 43
90449 Nürnberg

Tel.:  +49 911 30999 55 | Fax: +49 911 30999 99
mail: [email protected] | web:  http://www.teamix.de | blog: http://blog.teamix.de

Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller

teamix Support Hotline: +49 911 30999-112
 
 *** Bitte liken Sie uns auf Facebook: facebook.com/teamix ***

Am Mittwoch, 16. November 2016, 11:55:32 CET schrieben Sie:

I have an idea on why scrubbing may not work:

The filesystem is mounted read only and on checksum errors on one disk scrub would try to repair it with the good copy from another disk.

Yes, this is it:

merkaba:~> btrfs scrub start -r /dev/satafp1/daten
scrub started on /dev/satafp1/daten, fsid […] (pid=9375) merkaba:~> btrfs scrub status /dev/satafp1/daten
scrub status for […]
scrub started at Wed Nov 16 12:13:27 2016, running for 00:00:10 total bytes scrubbed: 45.53MiB with 0 errors

It would be helpful to receive a proper error message on this one.

Okay, seems today I learned quite something about BTRFS.

Thanks,

Martin Steigerwald  | Trainer

teamix GmbH
Südwestpark 43
90449 Nürnberg

Tel.:  +49 911 30999 55 | Fax: +49 911 30999 99
mail: [email protected] | web:  http://www.teamix.de | blog: http://blog.teamix.de

Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller

teamix Support Hotline: +49 911 30999-112
 
 *** Bitte liken Sie uns auf Facebook: facebook.com/teamix ***

Technically, the fact that a device is missing is a structural issue with the FS. Whether or not that falls under what any arbitrary person considers a structural issue or not is a different story.

General background though:
open_ctree is one of the core functions in the BTRFS code used during mounting the filesystem. Everything that calls it checks the return code and spits out 'BTRFS: open_ctree failed' if it failed. The problem is, just about everything internal (and many external things as well) to the BTRFS code that could prevent the FS from mounting happens either in open_ctree, or in a function it calls, so all that that line tells us is that the mount failed, which is less than useful in most cases. Given both the confusion you've experienced regarding this (which has happened to other people too), combined with the amount of effort I've had to put in to get the rest of the SysOps people where I work to understand that that message just means 'mount failed', I would really love to see that just be replaced with 'mount failed' in non-debug builds, preferrably with better info about _why_ things failed (the case of a degraded filesystem is pretty covered, but most other cases other than incompatible feature bits are not).

Last I knew, scrub doesn't work on degraded filesystems (in fact, by definition, it _can't_ work on a degraded array). It absolutely won't work though without the read-only flag on filesystems which are mounted read-only.

I'd expect rsync to be more likely to work than send/receive. In general, if you can read the files, rsync will work, whereas send/receive needs to read some low-level data from the FS which may not be touched when just reading files, so there are cases where rsync will work but send/receive won't.

It should, but doesn't currently. There was a thread about this a while back, but I don't think it ever resulted in anything changing.

In a case like this, I'd trust rsync more than send/receive. The following rsync switches might also be of interest:
-a: This turns on a bunch of things almost everyone wants when using rsync, similar to the same switch for cp, just with even more added in. -H: This recreates hardlinks on the receiving end.
-S: This recreates sparse files.
-A: This copies POSIX ACL's
-X: This copies extended attributes (most of them at least, there are a few that can't be arbitrarily written to).
Pre-creating the subvolumes by hand combined with using all of those will get you almost everything covered by send/receive except for sharing of extents and ctime.

Am Mittwoch, 16. November 2016, 07:57:08 CET schrieb Austin S. Hemmelgarn:

[…]

I usually use rsync -aAHXSP already :).

I was able to rsync any relevant data of the disk which is now being deleted by shred command.

Thank you,

Martin

I think the wiki should be updated to reflect that raid1 and raid10 are mostly OK. I think it's grossly misleading to consider either as green/OK when a single degraded read write mount creates single chunks that will then prevent a subsequent degraded read write mount. And also the lack of various notifications of device faultiness I think make it less than OK also. It's not in the "do not use" category but it should be in the middle ground status so users can make informed decisions.

Chris Murphy

It's worth pointing out also regarding this:
* This is handled sanely in recent kernels (the check got changed from per-fs to per-chunk, so you still have a usable FS if all the single chunks are only on devices you still have).
* This is only an issue with filesystems with exactly two disks. If a 3+ disk raid1 FS goes degraded, you still generate raid1 chunks.
* There are a couple of other cases where raid1 mode falls flat on it's face (lots of I/O errors in a short span of time with compression enabled can cause a kernel panic for example).
* raid10 has some other issues of it's own (you lose two devices, your filesystem is dead, which shouldn't be the case 100% of the time (if you lose different parts of each mirror, BTRFS _should_ be able to recover, it just doesn't do so right now)).

As far as the failed device handling issues, those are a problem with BTRFS in general, not just raid1 and raid10, so I wouldn't count those against raid1 and raid10.

Am Donnerstag, 17. November 2016, 12:05:31 CET schrieb Chris Murphy:

I agree – as error reporting I think is indead misleading. Feel free to edit it.

Ciao,

Martin

The status page should reflect the version with sane behavior, relative to the versions with not sane behavior. Otherwise people hit it unexpectedly despite the status page.

But still, the multiple device stuff really is not "OK" until there's some kind of faulty device concept and also notification for state changes from normal to faulty, faulty to normal, or even failed if that's reliably distinguishable from faulty.

Sure but raid56 are already flagged as not ready for prime time.

Chris Murphy

Everything you mentioned should be in the wiki IMHO. Knowledge is power.