Skip to contentSkip to conversation

parent transid verify failed + level verify failed

DISCUSSION Martin Steigerwald 1 participant 3 messages Nov 19, 2023 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi!

Have a great Sunday and absolutely feel free to reply at a later time :)

After having used linux-6.7-rc1 almost rc2, git commit 791c8ab095f71327899023223940dd52257a4173 in order to test out BCacheFS, back on 6.6.1 after two attempts of non working hibernation and a strange experience with GRUB presenting its command prompt instead of a boot menu¹, I now got this funny stuff:

[ 1849.408572] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901
[ 1849.408943] BTRFS error (device dm-1: state EA): level verify failed on logical 1132724224 mirror 1 wanted 0 found 1
[ 1849.414073] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901
[ 1849.446144] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.448792] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.449515] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.451647] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.454860] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.455527] BTRFS error (device dm-1: state EA): parent transid verify failed on 2a006853-bec0-4026-94aa-8ade2e0c3674logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.456286] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904

Noticed as BTRFS for / went read only.

Kernel 6.6.1 on Devuan Ceres on ThinkPad T14 AMD Gen 1 32 GiB RAM with 2 TB Samsung 980 Pro NVME SSD.

I cannot do a scrub cause despite

WARNING: failed to write the progress status file: Read-only file system. Status recording disabled scrub started on /, fsid […] (pid=31693)

scrub does not seem to do anything.

Will now boot into GRML and recover from backup, also checking the other BTRFS filesystems.

While focus will be to get to a full working state as quickly as possible, I intend to do a full dd copy for forensic analysis.

[1] Described in more detail at the end of:

Re: Questions related to BCacheFS

https://lore.kernel.org/linux-bcachefs/[email protected]/T/#m3fc22fb92f1f2d160f2b3a2387dc1f10d067fb97

Best,

Signature · 1 line
Martin
#1 lore raw

Martin Steigerwald - 19.11.23, 12:34:44 CET:

Quoted 8 lines

After having used linux-6.7-rc1 almost rc2, git commit 791c8ab095f71327899023223940dd52257a4173 in order to test out BCacheFS, back on 6.6.1 after two attempts of non working hibernation and a strange experience with GRUB presenting its command prompt instead of a boot menu¹, I now got this funny stuff:

[ 1849.408572] BTRFS error (device dm-1: state EA): parent transid

verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901

Finally recovered.

/ was not mountable anymore after booting into GRML. Made image with dd and restored from backup onto a newly created BTRFS.

/home also had errors. Did not take a chance. Scrubbed it, updated a backup with good old rsync, recreated filesystem and restored from updated backup.

Another filesystem with larger files that only sees user space writes when I write something to it, has been okay.

I have the following diagnostic data:

1) For /: dd image on external SSD with BTRFS, thus I can snapshot and/or ref-link copy it around and run various diagnostic stuff on it, btrfs check output (no repair), scrub was okay.

2) For /home: btrfs check output (no repair), scrub was okay.

3) I think I have some dmesg output from what caused the root filesystem going read only, but I think that is the output I already posted.

SMART status of 2TB Samsung 980 Pro disk okay. I do not assume that there is anything wrong with the device.

On what caused the corruption? No idea. Could be something wrong with that 6.7-almost-rc kernel, could be some corruption that happened on failed hibernation attempts, could have to do something with BCacheFS albeit I doubt it. Anyway, I am back on 6.6.1 for now and will not touch 6.7 kernel again on this laptop before I am confident that it is is safe.

About diagnostic data: Willing to post more after the weekend. Just let me know what you need. Of course I am not willing to upload complete dd image somewhere, even not with just the operating system filesystem, but I can run stuff on it and provide output. Otherwise you can have any of the btrfs check output.

Best,

Signature · 1 line
Martin
#2 lore raw

Martin Steigerwald - 19.11.23, 17:41:17 CET:

Quoted 2 lines

/ was not mountable anymore after booting into GRML. Made image with dd and restored from backup onto a newly created BTRFS.

I at least have a possible explanation.

Yesterday I experienced 6.7-almost-rc2 not hibernating correctly, but instead hanging with black screen. I rebooted into 6.6.1. Last thing I did was remove 6.7-almost-rc2 and hibernate.

In the morning I was greeted by GRUB command line prompt for a reason I do not understand yet either. I recovered from this with two GRML sessions. For that at least I unlocked LUKS and mounted / and /boot in GRML. If my memory is correct after I fixed booting up in these two GRML sessions, kernel 6.6.1 resumed from hibernation. That does make some session as in the GRML session I did not touch the swap volume. So I bet the hibernation image was not invalidated. However that would explain the corruption on / filesystem. Cause it was mounted in between and then the kernel resumed from a hibernation image with outdated in-memory data structures for BTRFS.

If my memory and analysis is correct this would easily explain the corruption on /. Still not sure what happened to /home. I think I did not touch it on attempting to repair GRUB. However I am not 100% sure about that.

Would be interesting to know whether diagnostic data would fit that explanation.

In case that is correct, then I think it would be good that in case I ever need to fix up GRUB again I also activate swap volume within GRML in the hope to invalidate the hibernation image. But I am not completely sure whether activating swap would be enough for that.

Quoted 3 lines

/home also had errors. Did not take a chance. Scrubbed it, updated a backup with good old rsync, recreated filesystem and restored from updated backup.

Best,

Signature · 1 line
Martin