parent transid verify failed + level verify failed
Hi!
Have a great Sunday and absolutely feel free to reply at a later time :)
After having used linux-6.7-rc1 almost rc2, git commit 791c8ab095f71327899023223940dd52257a4173 in order to test out BCacheFS, back on 6.6.1 after two attempts of non working hibernation and a strange experience with GRUB presenting its command prompt instead of a boot menu¹, I now got this funny stuff:
[ 1849.408572] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901 [ 1849.408943] BTRFS error (device dm-1: state EA): level verify failed on logical 1132724224 mirror 1 wanted 0 found 1 [ 1849.414073] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901 [ 1849.446144] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.448792] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.449515] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.451647] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.454860] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.455527] BTRFS error (device dm-1: state EA): parent transid verify failed on 2a006853-bec0-4026-94aa-8ade2e0c3674logical 1204469760 mirror 1 wanted 538478 found 538904 [ 1849.456286] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
Noticed as BTRFS for / went read only.
Kernel 6.6.1 on Devuan Ceres on ThinkPad T14 AMD Gen 1 32 GiB RAM with 2 TB Samsung 980 Pro NVME SSD.
I cannot do a scrub cause despite
WARNING: failed to write the progress status file: Read-only file system. Status recording disabled scrub started on /, fsid […] (pid=31693)
scrub does not seem to do anything.
Will now boot into GRML and recover from backup, also checking the other BTRFS filesystems.
While focus will be to get to a full working state as quickly as possible, I intend to do a full dd copy for forensic analysis.
[1] Described in more detail at the end of:
Re: Questions related to BCacheFS
Best,
Signature · 1 line
Martin
Martin Steigerwald - 19.11.23, 12:34:44 CET:
Quoted 8 lines
After having used linux-6.7-rc1 almost rc2, git commit
791c8ab095f71327899023223940dd52257a4173in order to test out BCacheFS, back on 6.6.1 after two attempts of non working hibernation and a strange experience with GRUB presenting its command prompt instead of a boot menu¹, I now got this funny stuff:[ 1849.408572] BTRFS error (device dm-1: state EA): parent transidverify failed on logical 1178386432 mirror 1 wanted 538478 found 538901
Finally recovered.
/ was not mountable anymore after booting into GRML. Made image with dd and restored from backup onto a newly created BTRFS.
/home also had errors. Did not take a chance. Scrubbed it, updated a backup with good old rsync, recreated filesystem and restored from updated backup.
Another filesystem with larger files that only sees user space writes when I write something to it, has been okay.
I have the following diagnostic data:
1) For /: dd image on external SSD with BTRFS, thus I can snapshot and/or ref-link copy it around and run various diagnostic stuff on it, btrfs check output (no repair), scrub was okay.
2) For /home: btrfs check output (no repair), scrub was okay.
3) I think I have some dmesg output from what caused the root filesystem going read only, but I think that is the output I already posted.
SMART status of 2TB Samsung 980 Pro disk okay. I do not assume that there is anything wrong with the device.
On what caused the corruption? No idea. Could be something wrong with that 6.7-almost-rc kernel, could be some corruption that happened on failed hibernation attempts, could have to do something with BCacheFS albeit I doubt it. Anyway, I am back on 6.6.1 for now and will not touch 6.7 kernel again on this laptop before I am confident that it is is safe.
About diagnostic data: Willing to post more after the weekend. Just let me know what you need. Of course I am not willing to upload complete dd image somewhere, even not with just the operating system filesystem, but I can run stuff on it and provide output. Otherwise you can have any of the btrfs check output.
Best,
Signature · 1 line
Martin
Martin Steigerwald - 19.11.23, 17:41:17 CET:
Quoted 2 lines
/ was not mountable anymore after booting into GRML. Made image with dd and restored from backup onto a newly created BTRFS.
I at least have a possible explanation.
Yesterday I experienced 6.7-almost-rc2 not hibernating correctly, but instead hanging with black screen. I rebooted into 6.6.1. Last thing I did was remove 6.7-almost-rc2 and hibernate.
In the morning I was greeted by GRUB command line prompt for a reason I do not understand yet either. I recovered from this with two GRML sessions. For that at least I unlocked LUKS and mounted / and /boot in GRML. If my memory is correct after I fixed booting up in these two GRML sessions, kernel 6.6.1 resumed from hibernation. That does make some session as in the GRML session I did not touch the swap volume. So I bet the hibernation image was not invalidated. However that would explain the corruption on / filesystem. Cause it was mounted in between and then the kernel resumed from a hibernation image with outdated in-memory data structures for BTRFS.
If my memory and analysis is correct this would easily explain the corruption on /. Still not sure what happened to /home. I think I did not touch it on attempting to repair GRUB. However I am not 100% sure about that.
Would be interesting to know whether diagnostic data would fit that explanation.
In case that is correct, then I think it would be good that in case I ever need to fix up GRUB again I also activate swap volume within GRML in the hope to invalidate the hibernation image. But I am not completely sure whether activating swap would be enough for that.
Quoted 3 lines
/home also had errors. Did not take a chance. Scrubbed it, updated a backup with good old rsync, recreated filesystem and restored from updated backup.
Best,
Signature · 1 line
Martin