Root level mismatch after sudden shutdown
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.
- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed
- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed
- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1
Couldn't setup device tree
ERROR: cannot open file system
I think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restore
I'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.
Quoted 40 lines · BERGUE Kevin
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001]
Quoted 8 lines
[ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed
What do you get for uname -a? What are the mount options? In particular I'm wondering if this drive is subject to discards, which currently will render usebackuproots useless. And how old is the file system?
If ro,rescue=usebackuproot doesn't work, it's not likely repairable, but might still be something btrfs restore can scrape *if* you haven't tried any repairs first. Repairs write changes to disk and can make things worse and harder to recover. You pretty much have to go straight to the scrape data off the drive step before the repair step because it is lower risk and more reliable.
The central problem is some lower layer mishandled write order, most likely by dropping flush/FUA, i.e. confirming to btrfs that flush/FUA command succeeded when in fact critical metadata has not yet made it to persistent media. And there is no way for Btrfs to know this until it's too late.
Quoted 9 lines
- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1
What do you get for
btrfs rescue super -v $dev
btrfs insp dump-s -f $dev
btrfs insp dump-t -b 1869491683328 $dev
btrfs insp dump-t -b 1869863370752 $dev
btrfs insp dump-t -b 1869866401792 $dev
Quoted 14 lines
Couldn't setup device tree
ERROR: cannot open file systemI think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restore
Quoted 1 line
I'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.
--repair, but especially --init-extent-tree can make it next to impossible for 'btrfs restore' command to work. It's not obvious that trying to fix btrfs, can make it harder for the scrape tool to work.
Signature · 1 line
Chris Murphy
Quoted 48 lines
Quoted 39 lines
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001]Quoted 8 lines
[ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed
Quoted 4 lines
What do you get for uname -a? What are the mount options? In particular I'm wondering if this drive is subject to discards, which currently will render usebackuproots useless. And how old is the file system?
$ uname -a Linux Elysium 5.16.5-1-MANJARO #1 SMP PREEMPT Tue Feb 1 16:57:14 UTC 2022 x86_64 GNU/Linux
The mount options I used were probably "rw,noatime,compress=zstd,ssd,space_cache,commit=120,subvol=@" since that's what I usually use but I can't be sure since fstab is on the disk.
Again, can't know for sure but from the top of my head I'd say I created it 6 months ago? It was probably created with linux 5.10 since that would have been the latest LTS.
Quoted 8 lines
If ro,rescue=usebackuproot doesn't work, it's not likely repairable, but might still be something btrfs restore can scrape *if* you haven't tried any repairs first. Repairs write changes to disk and can make things worse and harder to recover. You pretty much have to go straight to the scrape data off the drive step before the repair step because it is lower risk and more reliable.
Quoted 5 lines
The central problem is some lower layer mishandled write order, most likely by dropping flush/FUA, i.e. confirming to btrfs that flush/FUA command succeeded when in fact critical metadata has not yet made it to persistent media. And there is no way for Btrfs to know this until it's too late.
Yeah, that was my guess too.
Quoted 10 lines
Quoted 9 lines
- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1
Quoted 3 lines
What do you get for
btrfs rescue super -v $dev
All Devices:
Device: id = 1, name = /dev/mapper/SSD-Root
Before Recovering:
[All good supers]:
device name = /dev/mapper/SSD-Root superblock bytenr = 65536
device name = /dev/mapper/SSD-Root superblock bytenr = 67108864
device name = /dev/mapper/SSD-Root superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
Quoted 1 line
btrfs insp dump-s -f $dev
see attached "dump-s" file
Quoted 1 line
btrfs insp dump-t -b 1869491683328 $dev
see attached "dump-t -b 1869491683328" file
Quoted 1 line
btrfs insp dump-t -b 1869863370752 $dev
see attached "dump-t -b 1869863370752" file
Quoted 1 line
btrfs insp dump-t -b 1869866401792 $dev
see attached "dump-t -b 1869866401792" file
Quoted 27 lines
Quoted 17 lines
Couldn't setup device tree
ERROR: cannot open file systemI think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restoreQuoted 1 line
I'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.
--repair, but especially --init-extent-tree can make it next to impossible for 'btrfs restore' command to work. It's not obvious that trying to fix btrfs, can make it harder for the scrape tool to work.
Signature · 1 line
Chris Murphy
Quoted 12 lines · BERGUE Kevin
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.
Quoted 7 lines
- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999
Transid mismatch, and it's newer tree blocks overwriting old tree blocks.
This means two possible situations:
- The SSD is lying about its flush/fua opeartions
This means, the SSD firmware doesn't really write all its data back to its NAND when it reports flush/fua operations are finished. This mostly means the SSD is not reliable for any filesystems.
Although traditional journal based filesystems may have a better chance to survive.
- Some corrupted v1 cache makes btrfs to break its CoW checks This can only happen for v1 space cache, and is pretty tricky to happen, as v1 cache has its own checksum.
And now v2 cache is the default for newly created btrfs.
- Corrupted extent tree screwing up metadata CoW
Normally it would cause transaction abort in the first place though.
- Complex storage stack added extra points to cause FLUSH/FUA problem If you're using things like LVM/LUKS/Bcache, it's more complex and any stack in the middle can cause FLUSH/FUA problem if there is some bugs.
Mind to share the following info?
- Storage stack
From hardware disk to the top level btrfs, all things like LVM/LUKS /Bcache would help us to understand the situation.
- SSD model
To see if it's some known one to have FLUSH/FUA problems.
Thanks,
Qu
Quoted 52 lines
[ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1Couldn't setup device tree
ERROR: cannot open file systemI think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restoreI'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.
Quoted 47 lines
Quoted 13 lines
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.Quoted 7 lines
- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999Transid mismatch, and it's newer tree blocks overwriting old tree blocks.
This means two possible situations:
- The SSD is lying about its flush/fua opeartions
This means, the SSD firmware doesn't really write all its data back to its NAND when it reports flush/fua operations are finished. This mostly means the SSD is not reliable for any filesystems.Although traditional journal based filesystems may have a better chance to survive.
- Some corrupted v1 cache makes btrfs to break its CoW checks This can only happen for v1 space cache, and is pretty tricky to happen, as v1 cache has its own checksum.
And now v2 cache is the default for newly created btrfs.
- Corrupted extent tree screwing up metadata CoW
Normally it would cause transaction abort in the first place though.- Complex storage stack added extra points to cause FLUSH/FUA problem If you're using things like LVM/LUKS/Bcache, it's more complex and any stack in the middle can cause FLUSH/FUA problem if there is some bugs.
Quoted 5 lines
Mind to share the following info?
- Storage stack
From hardware disk to the top level btrfs, all things like LVM/LUKS /Bcache would help us to understand the situation.
The btrfs volume is on a LVM lv which itself is on a single SATA ssd. According to SMART the disk is healthy.
Quoted 3 lines
- SSD model
To see if it's some known one to have FLUSH/FUA problems.
The SSD reports itself as an INTEL SSDSC2KG960G8.
Quoted 57 lines
Thanks,
QuQuoted 52 lines
[ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1Couldn't setup device tree
ERROR: cannot open file systemI think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere. I tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restoreI'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.
Quoted 2 lines · BERGUE Kevin
The SSD reports itself as an INTEL SSDSC2KG960G8.
Well, sample size of one, user with SSDSC2KW256G8 that has firmware LHF002C reports it was fine until 200% write endurance. It's not a known problem SSD anyway.
Signature · 1 line
Chris Murphy
Quoted 57 lines · BERGUE Kevin
Hello everyone.
After a sudden shutdown my btrfs partition seems to be corrupted. After a few hours of reading documentation and trying various repair methods I found an error message I can't find anywhere so I'm sending it your way hoping you can at least explain what the issue is. The disk was running with linux 5.16.5 at the moment of the crash, my recovery environnement is a linux 5.15.16 machine with btrfs-progs v5.16.
To retrace my steps a bit:
- I tried to mount my partition normally:
# mount /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error.- I then looked at the relevant logs from dmesg:
# dmesg [ 2118.381387] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2118.381394] BTRFS info (device dm-1): disk space caching is enabled [ 2118.381395] BTRFS info (device dm-1): has skinny extents [ 2118.384626] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384900] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2118.384905] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2118.385304] BTRFS error (device dm-1): open_ctree failed- After some reading about the "parent transid verify failed" errors I tried to mount the volume with the usebackuproot flag:
# mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/SSD-Root /mnt/broken/ mount: /mnt/broken: wrong fs type, bad option, bad superblock on /dev/mapper/SSD-Root, missing codepage or helper program, or other error. And the dmesg content: [ 2442.117867] BTRFS info (device dm-1): flagging fs with big metadata feature [ 2442.117871] BTRFS info (device dm-1): trying to use backup root at mount time [ 2442.117872] BTRFS info (device dm-1): disk space caching is enabled [ 2442.117873] BTRFS info (device dm-1): has skinny extents [ 2442.123056] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123344] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.123348] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.124743] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124939] BTRFS error (device dm-1): parent transid verify failed on 1869491683328 wanted 526959 found 526999 [ 2442.124942] BTRFS warning (device dm-1): failed to read root (objectid=4): -5 [ 2442.125196] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125201] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125500] BTRFS critical (device dm-1): corrupt leaf: block=1869863370752 slot=97 extent bytenr=920192651264 len=4096 invalid generation, have 527002 expect (0, 527001] [ 2442.125502] BTRFS error (device dm-1): block=1869863370752 read time tree block corruption detected [ 2442.125508] BTRFS warning (device dm-1): couldn't read tree root [ 2442.125806] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.125808] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126174] BTRFS critical (device dm-1): corrupt leaf: block=1869866401792 slot=117 extent bytenr=906206486528 len=4096 invalid generation, have 527003 expect (0, 527002] [ 2442.126175] BTRFS error (device dm-1): block=1869866401792 read time tree block corruption detected [ 2442.126184] BTRFS warning (device dm-1): couldn't read tree root [ 2442.126599] BTRFS error (device dm-1): open_ctree failed- I then tried a check:
# btrfs check /dev/mapper/SSD-Root Opening filesystem to check... parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 parent transid verify failed on 1869491683328 wanted 526959 found 526999 Ignoring transid failure ERROR: root [4 0] level 0 does not match 1
Quoted 6 lines
Couldn't setup device tree
ERROR: cannot open file systemI think the "root [4 0] level 0 does not match 1" error is the real cuprit but I can't seem to find any info on this message anywhere.
The real culprit is the "parent transid verify failed" event. Everything printed after the "Ignoring transid failure" message is nonsense and should be ignored, which is probably why nobody writes about it. So let's write about it...
btrfs check should not continue to process the page after a transid verify failure--the probability of success is not exactly zero, but pretty close. Any information btrfs check gets from the page will be wrong, and could cause further corruption if it somehow got past all the other validation checks.
Parent transid cannot be ignored with any useful effect. btrfs almost never writes metadata items in the same location on disk twice. Each version of the filesystem tree places its metadata in new locations, overwriting metadata from older versions of the tree. There's no preset metadata locations--if you want to find any metadata item, you have to search for it in a tree, or scan the entire disk to find it.
If a metadata page write is dropped, the effect is that future reads of the tree will get a random metadata page from an older version of the filesystem tree. To prevent errors, btrfs labels every metadata page with the transaction ID it was created in, and every pointer to a page in btrfs metadata has a matching transaction ID number. If the transid doesn't match, btrfs knows that the wrong page has been read from the disk. To protect the integrity of the filesystem, btrfs treats this event as if it was a read error, and tries to recover by reading the same page from other mirror copies of metadata. If this is successful, btrfs will replace the metadata page that failed parent transid verification.
btrfs check ignores the transid failure. Since the entire page is wrong, most fields will fail verification after the transid check is ignored (e.g. the root level check in this case). The only fields that _won't_ fail verification are fields with a small number of distinct values, like a stopped clock that is correct twice a day. The messages following "Ignoring transid failure" are garbage.
Unfortunately, if you have no intact mirror of the missing metadata page (i.e. they were all corrupted by the same write reordering bug in the storage stack), then btrfs can't access anything below the missing metadata page in the tree, nor can btrfs safely update the tree. That means the filesystem can't be mounted read-write (or will mount read-write, but become read-only immediately upon detecting the parent transit verify failure). Data loss can be severe, from zero to 100%, depending on which metadata pages are lost and how large the write cache is.
In theory it is possible to repair a parent transid verify failure by brute-force searching the disk for an older copy of the metadata items in the missing page, reinsert the items into the filesystem tree, then repair inconsistencies in the metadata. AFAIK none of the existing btrfs tools can perform this search.
If the drive firmware is the problem, then you might be able to use the drive with 'hdparm -W0' or 'nvme set-feature -f 6 -v 0 -s' or 'smartctl --set wcache,off' (you'll need to run it on every boot and resume from suspend if applicable). It's a brute-force solution, but it usually makes a drive with broken firmware usable. Another solution might be to swap the SSD for a model with stronger countermeasures against power failure.
Quoted 11 lines
I
tried a bunch of other commands including:
- btrfs rescue zero-log
- btrfs rescue chunk-recover
- btrfs check --repair
- btrfs rescue super-recover
- btrfs check --repair with the tree root found by btrfs-find-root
- btrfs check --repair --init-csum-tree --init-extent-tree
- btrfs restoreI'm aware I probably executed some commands that don't make a lot of sense in my context but all of them failed with either the "root [4 0] level 0 does not match 1" message or a more generic "could not open ctree" or equivalent.