Problem with a filesystem
Hi all,
I have a server running debian 7.5 and meet issues for mounting a btrfs filesystem. This fs had worked for almost one year now but crashed last sunday. I'm unable to mount the fs and each btrfs-* commands ends with something like :
Ignoring transid failure
btrfsck: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted
Ignoring transid failure
btrfs-zero-log: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted
Ignoring transid failure
btrfs: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted
So, I can't fsck, restore, repair ou erase logs on it.
I have followed what is told here [1] or there [2] but now, I'm out of ideas.
Any help is appreciated.
Thanks in advance
[1]http://ram.kossboss.com/btrfs-transid-issue-explained-fix/ [2]https://btrfs.wiki.kernel.org/index.php/Restore
Quoted 15 lines · Tovo Rabemanantsoa
Hi all,
I have a server running debian 7.5 and meet issues for mounting a btrfs filesystem. This fs had worked for almost one year now but crashed last sunday. I'm unable to mount the fs and each btrfs-* commands ends with something like :
Ignoring transid failure
btrfsck: ctree.c:1550:leaf_space_used: Assertion `!(data_len < 0)' failed. AbortedIgnoring transid failure
btrfs-zero-log: ctree.c:1550:leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted
Quoted 7 lines
Ignoring transid failure
btrfs: ctree.c:1550:leaf_space_used: Assertion `!(data_len < 0)' failed. AbortedSo, I can't fsck, restore, repair ou erase logs on it.
I have followed what is told here [1] or there [2] but now, I'm out of ideas.
Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?
Chris Murphy
Quoted 4 lines · Chris Murphy
Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?
Chris Murphy
You're right,
In the beginning, I used the btrfs-progs provided by my debian (kernel 3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git one (3.16).
More information, 'btrfs filesystem show' gives :
Label: none uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3
Total devices 1 FS bytes used 18.67TiB devid 1 size 25.34TiB used 22.47TiB path /dev/sda4
Btrfs v3.16
Quoted 11 lines · Tovo Rabemanantsoa
Quoted 4 lines · Chris Murphy
Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?
Chris Murphy
You're right,
In the beginning, I used the btrfs-progs provided by my debian (kernel 3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git one (3.16).
More information, 'btrfs filesystem show' gives :
Quoted 3 lines
Label: none uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3
Total devices 1 FS bytes used 18.67TiB devid 1 size 25.34TiB used 22.47TiB path /dev/sda4
Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.
Chris Murphy
Quoted 2 lines · Chris Murphy
Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.
Thank you,
Please find below the messages shown during a normal mount and a mount with -o ro,recovery.
======== Normal Mount ========/var/log/messages
Sep 17 11:00:24 sdeeph1 kernel: [85051.529541] device fsid 9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4 Sep 17 11:00:24 sdeeph1 kernel: [85051.529917] btrfs: disk space caching is enabled Sep 17 11:01:09 sdeeph1 kernel: [85096.286741] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:01:09 sdeeph1 kernel: [85096.286756] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:01:09 sdeeph1 kernel: [85096.286764] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:01:09 sdeeph1 kernel: [85096.286770] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:01:09 sdeeph1 kernel: [85096.286775] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:01:09 sdeeph1 kernel: [85096.408271] btrfs: open_ctree failed
/var/log/syslog adds this line :
Sep 17 11:01:09 sdeeph1 kernel: [85096.286782] Failed to read block
groups: -5
======== mount -o ro,recovery ========/var/log/messages
Sep 17 11:05:10 sdeeph1 kernel: [85337.307267] device fsid 9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4 Sep 17 11:05:10 sdeeph1 kernel: [85337.307635] btrfs: enabling auto recovery Sep 17 11:05:10 sdeeph1 kernel: [85337.307639] btrfs: disk space caching is enabled Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed
/var/log/syslog adds this line :
Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read block
groups: -5
Quoted 13 lines · Tovo Rabemanantsoa
Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify failed on 24562568384512 wanted 255444 found 255446 Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed
Quoted 3 lines
/var/log/syslogadds this line :Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read blockgroups: -5
This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a btrfs check <btrfsdev>
and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results
Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.
btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
btrfs-zero-log <btrfsdev>
Chris Murphy
Quoted 5 lines · Chris Murphy
This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a btrfs check <btrfsdev>
and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results
Quoted 7 lines
Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.
btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
btrfs-zero-log <btrfsdev>Chris Murphy
Well,
btrfs check ends also with a failed message.
In any ways, I think this FS corruption is due to a weakness of my raid card. Indeed, before it happens, the kernel panicked because of this card. The card claims that he had lost 2 disks from the pool but, just removing them and plugging them again solved the error.
My thinking is that I didn't plug the two disks on the proper sequence. So it corrupted the raid reconstruction process.
By the way, I don't have enough structure to host 26TB of an image. Moreover, as it is our backup server, we need it to get back as soon as possible.
What I'm gonna do, now, is to make a new file system instead and set up everything one more time.
Thank you so much for helping me.
Tovo