Skip to contentSkip to conversation

Problem with a filesystem

DISCUSSION Tovo Rabemanantsoa 2 participants 7 messages Sep 16, 2014 – Sep 18, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi all,
I have a server running debian 7.5 and meet issues for mounting a btrfs filesystem. This fs had worked for almost one year now but crashed last sunday. I'm unable to mount the fs and each btrfs-* commands ends with something like :
Ignoring transid failure
btrfsck: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted

Ignoring transid failure
btrfs-zero-log: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted

Ignoring transid failure
btrfs: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted

So, I can't fsck, restore, repair ou erase logs on it.
I have followed what is told here [1] or there [2] but now, I'm out of ideas.
Any help is appreciated.
Thanks in advance

[1]http://ram.kossboss.com/btrfs-transid-issue-explained-fix/ [2]https://btrfs.wiki.kernel.org/index.php/Restore

#1 lore raw
Quoted 15 lines · Tovo Rabemanantsoa

Hi all,
I have a server running debian 7.5 and meet issues for mounting a btrfs filesystem. This fs had worked for almost one year now but crashed last sunday. I'm unable to mount the fs and each btrfs-* commands ends with something like :
Ignoring transid failure
btrfsck: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted

Ignoring transid failure
btrfs-zero-log: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted

Quoted 7 lines

Ignoring transid failure
btrfs: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed. Aborted

So, I can't fsck, restore, repair ou erase logs on it.
I have followed what is told here [1] or there [2] but now, I'm out of ideas.

Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?

Chris Murphy

#2 lore raw
Quoted 4 lines · Chris Murphy

Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?

Chris Murphy

You're right,
In the beginning, I used the btrfs-progs provided by my debian (kernel 3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git one (3.16).
More information, 'btrfs filesystem show' gives :

Label: none uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3

	Total devices 1 FS bytes used 18.67TiB
	devid    1 size 25.34TiB used 22.47TiB path /dev/sda4

Btrfs v3.16

#3 lore raw
Quoted 11 lines · Tovo Rabemanantsoa
Quoted 4 lines · Chris Murphy

Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?

Chris Murphy

You're right,
In the beginning, I used the btrfs-progs provided by my debian (kernel 3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git one (3.16).
More information, 'btrfs filesystem show' gives :

Quoted 3 lines

Label: none uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3

	Total devices 1 FS bytes used 18.67TiB
	devid    1 size 25.34TiB used 22.47TiB path /dev/sda4

Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.

Chris Murphy

#4 lore raw
Quoted 2 lines · Chris Murphy

Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.

Thank you,
Please find below the messages shown during a normal mount and a mount with -o ro,recovery.

======== Normal Mount ========
/var/log/messages

Sep 17 11:00:24 sdeeph1 kernel: [85051.529541] device fsid
9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4
Sep 17 11:00:24 sdeeph1 kernel: [85051.529917] btrfs: disk space caching
is enabled
Sep 17 11:01:09 sdeeph1 kernel: [85096.286741] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286756] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286764] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286770] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286775] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.408271] btrfs: open_ctree failed

/var/log/syslog adds this line :

Sep 17 11:01:09 sdeeph1 kernel: [85096.286782] Failed to read block

groups: -5

======== mount -o ro,recovery ========
/var/log/messages

Sep 17 11:05:10 sdeeph1 kernel: [85337.307267] device fsid
9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4
Sep 17 11:05:10 sdeeph1 kernel: [85337.307635] btrfs: enabling auto recovery
Sep 17 11:05:10 sdeeph1 kernel: [85337.307639] btrfs: disk space caching
is enabled
Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed

/var/log/syslog adds this line :

Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read block

groups: -5

#5 lore raw
Quoted 13 lines · Tovo Rabemanantsoa
Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed
Quoted 3 lines

/var/log/syslog adds this line :

Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read block

groups: -5

This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a btrfs check <btrfsdev>
and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results

Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.

btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
btrfs-zero-log <btrfsdev>

Chris Murphy

#6 lore raw
Quoted 5 lines · Chris Murphy

This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a btrfs check <btrfsdev>
and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results

Quoted 7 lines

Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.

btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
btrfs-zero-log <btrfsdev>

Chris Murphy

Well,
btrfs check ends also with a failed message.
In any ways, I think this FS corruption is due to a weakness of my raid card. Indeed, before it happens, the kernel panicked because of this card. The card claims that he had lost 2 disks from the pool but, just removing them and plugging them again solved the error.
My thinking is that I didn't plug the two disks on the proper sequence. So it corrupted the raid reconstruction process.
By the way, I don't have enough structure to host 26TB of an image. Moreover, as it is our backup server, we need it to get back as soon as possible.
What I'm gonna do, now, is to make a new file system instead and set up everything one more time.
Thank you so much for helping me.

Tovo