trouble mounting btrfs filesystem....

DISCUSSION Scott E. Blomquist 5 participants 11 messages Aug 12, 2018 – Aug 15, 2018 Btrfs file system

Share & tools

Dmitrii Tcvetkov

2018-08-14 15:54 UTC #2 lore raw

Quoted 56 lines

Quoted 38 lines · Scott E. Blomquist
Hi All,
Early this morning there was a power glitch that affected our system.
The second enclosure went offline but the file system stayed up for a bit before rebooting and recovering the 2 missing arrays sdb1 and sdc1.
When mounting we get....
    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info
(device sda1): has skinny extents Aug 12 14:54:52 localhost
kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
[ 8666.056845] BTRFS error (device sda1): failed to read block
groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
We are here...
    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT
2018 x86_64 x86_64 x86_64 GNU/Linux
    # btrfs --version
    btrfs-progs v4.17.1
    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
            devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
            devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
Quoted 3 lines
Thanks for any help.
sb. Scott Blomquist
Hi All,
Is there any more info needed here?
I can restore from backup if needed but that will take a bit of time.
Checking around it looks like I could try...
btrfs-zero-log /dev/sda1
Or maybe ..
   btrfsck --repair /dev/sda1

Quoted 9 lines

I am just not sure here and would prefer to do the right thing.
Any help would be much appreciated.
Thanks,
sb. Scott Blomquist

I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.

"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.

[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

Scott E. Blomquist

author

2018-08-14 16:19 UTC #3 lore raw

Quoted 72 lines · Dmitrii Tcvetkov

Quoted 56 lines
Quoted 38 lines · Scott E. Blomquist
Hi All,
Early this morning there was a power glitch that affected our system.
The second enclosure went offline but the file system stayed up for a bit before rebooting and recovering the 2 missing arrays sdb1 and sdc1.
When mounting we get....
    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info
(device sda1): has skinny extents Aug 12 14:54:52 localhost
kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
[ 8666.056845] BTRFS error (device sda1): failed to read block
groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
We are here...
    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT
2018 x86_64 x86_64 x86_64 GNU/Linux
    # btrfs --version
    btrfs-progs v4.17.1
    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
            devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
            devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
Quoted 3 lines
Thanks for any help.
sb. Scott Blomquist
Hi All,
Is there any more info needed here?
I can restore from backup if needed but that will take a bit of time.
Checking around it looks like I could try...
btrfs-zero-log /dev/sda1
Or maybe ..
   btrfsck --repair /dev/sda1
Quoted 9 lines
I am just not sure here and would prefer to do the right thing.
Any help would be much appreciated.
Thanks,
sb. Scott Blomquist
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.

Quoted 7 lines

"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.
[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

Thank you. So zero-log is not the right thing...

Unfortunately when mounting ro,usebackuproot I still get the same messages...

    Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info (device sda1): trying to use backup root at mount time
    Aug 14 09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1): using free space tree
    Aug 14 09:08:15 localhost kernel: [160669.100318] BTRFS info (device sda1): has skinny extents
    Aug 14 09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 14 09:10:24 localhost kernel: [160797.887708] BTRFS error (device sda1): failed to read block groups: -5
    Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS error (device sda1): open_ctree failed

it sounds like my only option maybe 'btrfs check --repair' and that doesn't sound too hopeful.

Any other ideas?

Thanks,

sb. Scott Blomquist

Dmitrii Tcvetkov

2018-08-14 16:28 UTC #4 lore raw

Quoted 103 lines · On Tue, 14 Aug 2018 09:31:56 -0400 "Scott E. Blomquist" <[email protected]>…

Quoted 75 lines · Dmitrii Tcvetkov
Quoted 58 lines
Quoted 41 lines · Scott E. Blomquist
Hi All,
Early this morning there was a power glitch that affected our system.
The second enclosure went offline but the file system stayed up for a bit before rebooting and recovering the 2 missing arrays sdb1 and sdc1.
When mounting we get....
    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS
info (device sda1): has skinny extents Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1):
parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
[ 8665.985512] BTRFS error (device sda1): parent transid
verify failed on 177443463479296 wanted 2159304 found 2159295
Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error
(device sda1): failed to read block groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
We are here...
    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00
EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
    # btrfs --version
    btrfs-progs v4.17.1
    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB
path /dev/sda1 devid    2 size 21.83TiB used 16.61TiB
path /dev/sdb1 devid    3 size 21.83TiB used 16.61TiB
path /dev/sdc1 
Thanks for any help.
Quoted 1 line
sb. Scott Blomquist
Hi All,
Is there any more info needed here?
I can restore from backup if needed but that will take a bit of time.
Checking around it looks like I could try...
btrfs-zero-log /dev/sda1
Or maybe ..
   btrfsck --repair /dev/sda1
Quoted 9 lines
I am just not sure here and would prefer to do the right thing.
Any help would be much appreciated.
Thanks,
sb. Scott Blomquist
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.
Quoted 7 lines
"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.
[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log
Thank you. So zero-log is not the right thing...
Unfortunately when mounting ro,usebackuproot I still get the same messages...
    Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info
(device sda1): trying to use backup root at mount time Aug 14
09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1):
using free space tree Aug 14 09:08:15 localhost kernel:
[160669.100318] BTRFS info (device sda1): has skinny extents Aug 14
09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1):
parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 14 09:10:24 localhost kernel:
[160797.887708] BTRFS error (device sda1): failed to read block
groups: -5 Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS error (device sda1): open_ctree failed

Quoted 8 lines

it sounds like my only option maybe 'btrfs check --repair' and that doesn't sound too hopeful.
Any other ideas?
Thanks,
sb. Scott Blomquist

As far as I know btrfs check --repair doesn't fix transid errors. Usually devs can tell whenever "btrfs check --repair" will repair anything by looking at "btrfs check --readonly" output, but as your filesystem is quite big this will take some time.

If usebackuproot doesn't help then filesystem is beyond repair and you should try to refresh your backups with "btrfs restore" and restore from them[1].

[1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

Scott E. Blomquist

author

2018-08-14 16:38 UTC #5 lore raw

Quoted 120 lines · Dmitrii Tcvetkov

Quoted 103 lines · On Tue, 14 Aug 2018 09:31:56 -0400 "Scott E. Blomquist" <[email protected]>…
Quoted 75 lines · Dmitrii Tcvetkov
Quoted 58 lines
Quoted 41 lines · Scott E. Blomquist
Hi All,
Early this morning there was a power glitch that affected our system.
The second enclosure went offline but the file system stayed up for a bit before rebooting and recovering the 2 missing arrays sdb1 and sdc1.
When mounting we get....
    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS
info (device sda1): has skinny extents Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1):
parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
[ 8665.985512] BTRFS error (device sda1): parent transid
verify failed on 177443463479296 wanted 2159304 found 2159295
Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error
(device sda1): failed to read block groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
We are here...
    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00
EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
    # btrfs --version
    btrfs-progs v4.17.1
    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB
path /dev/sda1 devid    2 size 21.83TiB used 16.61TiB
path /dev/sdb1 devid    3 size 21.83TiB used 16.61TiB
path /dev/sdc1 
Thanks for any help.
Quoted 1 line
sb. Scott Blomquist
Hi All,
Is there any more info needed here?
I can restore from backup if needed but that will take a bit of time.
Checking around it looks like I could try...
btrfs-zero-log /dev/sda1
Or maybe ..
   btrfsck --repair /dev/sda1
Quoted 9 lines
I am just not sure here and would prefer to do the right thing.
Any help would be much appreciated.
Thanks,
sb. Scott Blomquist
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.
Quoted 7 lines
"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.
[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log
Thank you. So zero-log is not the right thing...
Unfortunately when mounting ro,usebackuproot I still get the same messages...
    Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info
(device sda1): trying to use backup root at mount time Aug 14
09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1):
using free space tree Aug 14 09:08:15 localhost kernel:
[160669.100318] BTRFS info (device sda1): has skinny extents Aug 14
09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1):
parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295 Aug 14 09:10:24 localhost kernel:
[160797.887708] BTRFS error (device sda1): failed to read block
groups: -5 Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS error (device sda1): open_ctree failed
Quoted 8 lines
it sounds like my only option maybe 'btrfs check --repair' and that doesn't sound too hopeful.
Any other ideas?
Thanks,
sb. Scott Blomquist
As far as I know btrfs check --repair doesn't fix transid errors. Usually devs can tell whenever "btrfs check --repair" will repair anything by looking at "btrfs check --readonly" output, but as your filesystem is quite big this will take some time.

Quoted 4 lines

If usebackuproot doesn't help then filesystem is beyond repair and you should try to refresh your backups with "btrfs restore" and restore from them[1].
[1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

Thanks. Yup. Looks bad...

    root@localhost:~# btrfs check --readonly /dev/sda1 
    Opening filesystem to check...
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Ignoring transid failure
    ERROR: child eb corrupted: parent bytenr=177443205808128 item=156 parent level=2 child level=0
    ERROR: cannot open file system

Looks like I will start the restore process unless there are any other ideas.

Thanks again,

sb. Scott Blomquist

Roman Mamedov

2018-08-14 23:00 UTC #6 lore raw

In reply to Dmitrii Tcvetkov

Quoted 4 lines · On Tue, 14 Aug 2018 16:41:11 +0300 Dmitrii Tcvetkov <[email protected]>…

If usebackuproot doesn't help then filesystem is beyond repair and you should try to refresh your backups with "btrfs restore" and restore from them[1].
[1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

This is really the worst unfixed Btrfs issue today. This happens a lot on unclean shutdowns or reboots, and the only advice is usually to "start over" -- even if your FS is 40 TB and the discrepancy is just by half a dozen transids. There needs to be a way in fsck to accept (likely minor!) FS damage and forcefully fixup transids to what they should be -- or even nuke the affected portions entirely.

Signature · 2 lines

With respect,
Roman

Hans van Kranenburg

2018-08-14 18:04 UTC #7 lore raw

In reply to Dmitrii Tcvetkov

Quoted 11 lines · Dmitrii Tcvetkov

Quoted 4 lines
Quoted 3 lines · Scott E. Blomquist
Hi All,
[...]
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.

Quoted 5 lines

"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.

Keep in mind that a successful mount with usebackuproot does not mean you're looking at a consistent filesystem. After each transaction commit, disk space that is no longer referenced is immediately freed for reuse.

So, even if you can mount with usebackuproot, you have to hope that none of the metadata blocks that were used back then have been overwritten already, even the ones in distant corners of trees. A full check / scrub / etc would be needed to find out.

Signature · 1 line

Hans van Kranenburg

Andrei Borzenkov

2018-08-14 19:57 UTC #8 lore raw

14.08.2018 18:16, Hans van Kranenburg пишет:

Quoted 18 lines · Dmitrii Tcvetkov

Quoted 11 lines
Quoted 4 lines
Quoted 3 lines · Scott E. Blomquist
Hi All,
[...]
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.
Quoted 5 lines
"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.

Quoted 5 lines

Keep in mind that a successful mount with usebackuproot does not mean you're looking at a consistent filesystem. After each transaction commit, disk space that is no longer referenced is immediately freed for reuse.

With all respect - in this case btrfs should not even pretend it can do "usebackuproot". What this option is for then?

Quoted 5 lines

So, even if you can mount with usebackuproot, you have to hope that none of the metadata blocks that were used back then have been overwritten already, even the ones in distant corners of trees. A full check / scrub / etc would be needed to find out.

Hans van Kranenburg

2018-08-15 02:01 UTC #9 lore raw

Quoted 19 lines · Andrei Borzenkov

14.08.2018 18:16, Hans van Kranenburg пишет:
Quoted 18 lines · Dmitrii Tcvetkov
Quoted 16 lines
Quoted 4 lines
Quoted 3 lines · Scott E. Blomquist
Hi All,
[...]
I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors. Transid errors mean that some metadata writes are missing, if they prevent you from mounting filesystem it's pretty much fatal. If btrfs could recover metadata from good copy it'd have done that.
"wanted 2159304 found 2159295" means that some metadata is stale by 9 commits. You could try to mount it with "ro,usebackuproot" mount options as readonly mount is less strict. If that works you can try "usebackuproot" without ro option. But 9 commits is probably too much and there isn't enough data to rollback so far.

Quoted 8 lines

Quoted 5 lines
Keep in mind that a successful mount with usebackuproot does not mean you're looking at a consistent filesystem. After each transaction commit, disk space that is no longer referenced is immediately freed for reuse.
With all respect - in this case btrfs should not even pretend it can do "usebackuproot". What this option is for then?

I have no idea. As the Dutch say: "schiet mij maar in de kerstboom". The commit that introduces it contains no explanation at all.

The only thing I can think of is when you're a developer and writing buggy code that writes out corrupted tree root blocks, which makes the filesystem crash in some way directly after a transaction ends while no other new writes are done again yet, combined with getting fed up of having to mkfs and put your testdata back all the time.

Quoted 4 lines

Quoted 4 lines
So, even if you can mount with usebackuproot, you have to hope that none of the metadata blocks that were used back then have been overwritten already, even the ones in distant corners of trees. A full check / scrub / etc would be needed to find out.

Signature · 1 line

Hans van Kranenburg

Hans van Kranenburg

2018-08-14 18:01 UTC #10 lore raw

In reply to Scott E. Blomquist

Quoted 25 lines · Scott E. Blomquist

Hi All,

Early this morning there was a power glitch that affected our system.

The second enclosure went offline but the file system stayed up for a bit before rebooting and recovering the 2 missing arrays sdb1 and sdc1.

When mounting we get....

    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info (device sda1): has skinny extents
    Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error (device sda1): failed to read block groups: -5
    Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed

We are here...

    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

    # btrfs --version
    btrfs-progs v4.17.1

Quoted 6 lines

    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
            devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
            devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1

What kind of devices are this? You say enclosure... is it a bunch of disks doing its own RAID, with btrfs on top?

Do you have RAID1 metadata on top of that, or single?

At least if you go the mkfs route (I read the other replies) then also find out what happened. If your storage is losing data in situations like this while it told btrfs that the data was safe, you're running a dangerous operation.

Signature · 1 line

Hans van Kranenburg