segmentation-fault in btrfsck (git-version)
Dear all,
thanks for developing btrfsck!
Now, I'd like to contribute -as far as I can. I'm not a developer, but I do have some linux-experience.
I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under Kernel 3.0. Now it's corrupt. I had some hard resets of the machine -which might have contributed. I do have a backup of the data -at least of the important stuff. Some TV-Recordings are missing. The reason I am writing is, to support the development.
Unfortunately, btrfsck (latest git-version) crashes with a segmentation fault, when trying to repair this.
Here's the backtrace:
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
btrfsck: disk-io.c:382: __commit_transaction: Assertion `!(!eb || eb->start != start)' failed.
Program received signal SIGABRT, Aborted. 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb)
(gdb) backtrace
#0 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff7845192 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x000000000040d3ae in __commit_transaction (trans=0x62e010, root=0xb66ae0) at disk-io.c:382 #5 0x000000000040d4d8 in btrfs_commit_transaction (trans=0x62e010, root=0xb66ae0) at disk-io.c:415 #6 0x000000000040743d in main (ac=<optimized out>, av=<optimized out>) at btrfsck.c:3587
Now, here's where my debugging knowledge ends. Are you interested in debugging this further, or is it a known bug?
Regards,
Hendrik
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Quoted 37 lines · Hendrik Friedel
Dear all,
thanks for developing btrfsck!
Now, I'd like to contribute -as far as I can. I'm not a developer, but I do have some linux-experience.
I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under Kernel 3.0. Now it's corrupt. I had some hard resets of the machine -which might have contributed. I do have a backup of the data -at least of the important stuff. Some TV-Recordings are missing. The reason I am writing is, to support the development.Unfortunately, btrfsck (latest git-version) crashes with a segmentation fault, when trying to repair this.
Here's the backtrace:
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
btrfsck: disk-io.c:382:__commit_transaction: Assertion `!(!eb ||eb->start!= start)' failed.Program received signal SIGABRT, Aborted.
0x00007ffff784c425in raise () from/lib/x86_64-linux-gnu/libc.so.6(gdb)
(gdb) backtrace#0 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff7845192 in __assert_fail () from
/lib/x86_64-linux-gnu/libc.so.6
#40x000000000040d3aein__commit_transaction(trans=0x62e010, root=0xb66ae0) at disk-io.c:382
#50x000000000040d4d8inbtrfs_commit_transaction(trans=0x62e010, root=0xb66ae0) at disk-io.c:415
#60x000000000040743din main (ac=<optimized out>, av=<optimized out>) at btrfsck.c:3587
Quoted 3 lines
Now, here's where my debugging knowledge ends. Are you interested in debugging this further, or is it a known bug?
Line 382 in disk-io.c is:
BUG_ON(!eb || eb->start != start);
So, basically, btrfsck is intentionally crashing because it doesn't know how to handle this condition.
Future refinements of btrfsck will probably include proper error messages for issues that can't be handled, or perhaps even fix the error.
It might be interesting for you to try a newer kernel, and use scrub on this volume if you have the two disks RAIDed.
Hello,
thanks for letting me know.
Indeed it would be good to replace the segmentation Fault by " btrfs does not yet know how to handle this condition. Future refinements of btrfsck will probably include proper error
messages for issues that can't be handled, or perhaps even fix the
error."
Quoted 2 lines
It might be interesting for you to try a newer kernel, and use scrub on this volume if you have the two disks RAIDed.
I will try that.
Greetings,
Hendrik
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Dear Mich,
thanks for your help and suggestion:
Quoted 2 lines
It might be interesting for you to try a newer kernel, and use scrub on this volume if you have the two disks RAIDed.
I have now scrubbed the Disk:
./btrfs scrub status /mnt/other/
scrub status for a15eede9-1a92-47d8-940a-adc7cf97352d
scrub started at Sun Dec 9 13:48:57 2012 and finished after
3372 seconds
total bytes scrubbed: 1.10TB with 0 errors
That's odd, as in one folder, data is missing (I could have deleted it, but I'd be very surprised...)
Also, when I run btrfsck, I get errors:
On sdc1:
root 261 inode 64370 errors 400 root 261 inode 64373 errors 400 root 261 inode 64375 errors 400 root 261 inode 64376 errors 400 found 1203899371520 bytes used err is 1 total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608 btree space waste bytes: 212383383 file data blocks allocated: 28032005304320
referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eec
On sdb1:
root 261 inode 64373 errors 400 root 261 inode 64375 errors 400 root 261 inode 64376 errors 400 found 1203899371520 bytes used err is 1 total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608 btree space waste bytes: 212383383 file data blocks allocated: 28032005304320
referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eec
And when I try to mount one of the two raided disks, I get:
[ 1173.773861] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 1 transid 140194 /dev/sdb1 [ 1173.774695] btrfs: failed to read the system array on sdb1 [ 1173.774854] btrfs: open_ctree failed
while the other works:
[ 1177.927096] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 2 transid 140194 /dev/sdc1
Do you have hints for me?
The Kernel now is 3.3.7-030307-generic (anything more recent, I would have to compile myself, which I will do, if you suggest to)
Greetings,
Hendrik
Am 06.12.2012 20:09, schrieb Mitch Harder:
Quoted 56 lines · Hendrik Friedel
Quoted 37 lines
Dear all,
thanks for developing btrfsck!
Now, I'd like to contribute -as far as I can. I'm not a developer, but I do have some linux-experience.
I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under Kernel 3.0. Now it's corrupt. I had some hard resets of the machine -which might have contributed. I do have a backup of the data -at least of the important stuff. Some TV-Recordings are missing. The reason I am writing is, to support the development.Unfortunately, btrfsck (latest git-version) crashes with a segmentation fault, when trying to repair this.
Here's the backtrace:
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
btrfsck: disk-io.c:382:__commit_transaction: Assertion `!(!eb ||eb->start!= start)' failed.Program received signal SIGABRT, Aborted.
0x00007ffff784c425in raise () from/lib/x86_64-linux-gnu/libc.so.6(gdb)
(gdb) backtrace#0 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff7845192 in __assert_fail () from
/lib/x86_64-linux-gnu/libc.so.6
#40x000000000040d3aein__commit_transaction(trans=0x62e010, root=0xb66ae0) at disk-io.c:382
#50x000000000040d4d8inbtrfs_commit_transaction(trans=0x62e010, root=0xb66ae0) at disk-io.c:415
#60x000000000040743din main (ac=<optimized out>, av=<optimized out>) at btrfsck.c:3587Quoted 3 lines
Now, here's where my debugging knowledge ends. Are you interested in debugging this further, or is it a known bug?
Line 382 in
disk-io.cis:
BUG_ON(!eb || eb->start != start);So, basically, btrfsck is intentionally crashing because it doesn't know how to handle this condition.
Future refinements of btrfsck will probably include proper error messages for issues that can't be handled, or perhaps even fix the error.
It might be interesting for you to try a newer kernel, and use scrub on this volume if you have the two disks RAIDed.
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Quoted 56 lines · Hendrik Friedel
Dear Mich,
thanks for your help and suggestion:
Quoted 2 lines
It might be interesting for you to try a newer kernel, and use scrub on this volume if you have the two disks RAIDed.
I have now scrubbed the Disk:
./btrfs scrub status/mnt/other/
scrub status for a15eede9-1a92-47d8-940a-adc7cf97352dscrub started at Sun Dec 9 13:48:57 2012 and finished after 3372seconds
total bytes scrubbed: 1.10TB with 0 errorsThat's odd, as in one folder, data is missing (I could have deleted it, but I'd be very surprised...)
Also, when I run btrfsck, I get errors:
On sdc1:
root 261 inode 64370 errors 400
root 261 inode 64373 errors 400root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
found 1203899371520 bytes used err is 1
total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608
btree space waste bytes: 212383383
file data blocks allocated: 28032005304320
referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eecOn sdb1:
root 261 inode 64373 errors 400root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
found 1203899371520 bytes used err is 1
total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608
btree space waste bytes: 212383383
file data blocks allocated: 28032005304320
referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eecAnd when I try to mount one of the two raided disks, I get:
[ 1173.773861] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 1 transid 140194 /dev/sdb1 [ 1173.774695] btrfs: failed to read the system array on sdb1 [ 1173.774854] btrfs: open_ctree failed
Quoted 8 lines
while the other works:
[ 1177.927096] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 2transid 140194
/dev/sdc1Do you have hints for me?
The Kernel now is 3.3.7-030307-generic (anything more recent, I would have to compile myself, which I will do, if you suggest to)
Since btrfs has significant improvements and fixes in each kernel release, and since very few of these changes are backported, it is recommended to use the latest kernels available.
The "root ### inode ##### errors 400" are an indication that there is an inconsistency in the inode size. There was a patch included in the 3.1 or 3.2 kernel to address this issue
(http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
But I don't believe this patch fixed existing occurrences of this
error.
At this point, the quickest solution for you may be to rebuild and reformat this RAID assembly, and restore this data from backups.
If you don't have a backup of this data, and since your array seems to be working pretty well in a degraded state, this would be a really good time to look at a strategy of getting a backup of this data before doing many more attempts at rescue.
Hello Mitch, hello all,
Quoted 3 lines
Since btrfs has significant improvements and fixes in each kernel release, and since very few of these changes are backported, it is recommended to use the latest kernels available.
Ok, it's 3.7 now.
Quoted 6 lines
The "root ### inode ##### errors 400" are an indication that there is an inconsistency in the inode size. There was a patch included in the 3.1 or 3.2 kernel to address this issue
(http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
But I don't believe this patch fixed existing occurrences of this
error.
Apparently not. It's still there.
Quoted 2 lines
At this point, the quickest solution for you may be to rebuild and reformat this RAID assembly, and restore this data from backups.
Yepp, I did that. But in fact, some data is missing. It is not essential, but nice to have.
Quoted 4 lines
If you don't have a backup of this data, and since your array seems to be working pretty well in a degraded state, this would be a really good time to look at a strategy of getting a backup of this data before doing many more attempts at rescue.
Done. It's all save on another ext4 drive.
So, let's play ;-)
Could you please help me trying to restore the missing Data?
What I tried sofar was:
./btrfs-restore /dev/sdc1 /mnt/restore/
It worked, in a way that it restored what I already had.
What's odd aswell is, that btrfs scrub did run through without errors.
So, the missing data could have been (accidentally) deleted by me. But I don't think... nevertheless I cannot exclude.
What I know is the (original) Path of the Data.
Greetings,
Hendrik
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Quoted 46 lines · Hendrik Friedel
Hello Mitch, hello all,
Quoted 4 lines
Since btrfs has significant improvements and fixes in each kernel
release, and since very few of these changes are backported, it is recommended to use the latest kernels available.
Ok, it's 3.7 now.
Quoted 7 lines
The "root ### inode ##### errors 400" are an indication that there is an inconsistency in the inode size. There was a patch included in the 3.1 or 3.2 kernel to address this issue
(http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
But I don't believe this patch fixed existing occurrences of this
error.Apparently not. It's still there.
Quoted 2 lines
At this point, the quickest solution for you may be to rebuild and reformat this RAID assembly, and restore this data from backups.
Yepp, I did that. But in fact, some data is missing. It is not essential, but nice to have.
Quoted 4 lines
If you don't have a backup of this data, and since your array seems to be working pretty well in a degraded state, this would be a really good time to look at a strategy of getting a backup of this data before doing many more attempts at rescue.
Done. It's all save on another ext4 drive.
So, let's play ;-)
Could you please help me trying to restore the missing Data?What I tried sofar was:
./btrfs-restore/dev/sdc1/mnt/restore/
Quoted 7 lines
It worked, in a way that it restored what I already had. What's odd aswell is, that btrfs scrub did run through without errors. So, the missing data could have been (accidentally) deleted by me. But I don't think... nevertheless I cannot exclude.
What I know is the (original) Path of the Data.
You could try btrfs-debug-tree, and search for any traces of your file. However, be ready to sift through a massive amount of output.
Hi Mitch, hi all,
thanks for your hint.
I used btrfs-debug-tree now.
With -e, the output is empty. But without -e I do get a bit output file. When I search for Filenames that I am missing, I get:
grep Sting big_output_file |grep Berlin
namelen 20 datalen 0 name: Sting_Live_in_Berlin namelen 20 datalen 0 name: Sting_Live_in_Berlin inode ref index 29 namelen 20 name: Sting_Live_in_Berlin
That looks good.
That raises two questions now: Can I restore the file?
And: Can I do that for a whole Path (e.g. ./Video/)
Greetings&Thanks!
Hendrik
Am 15.12.2012 23:24, schrieb Mitch Harder:
Quoted 62 lines · Hendrik Friedel
Quoted 46 lines
Hello Mitch, hello all,
Quoted 4 lines
Since btrfs has significant improvements and fixes in each kernel
release, and since very few of these changes are backported, it is recommended to use the latest kernels available.
Ok, it's 3.7 now.
Quoted 7 lines
The "root ### inode ##### errors 400" are an indication that there is an inconsistency in the inode size. There was a patch included in the 3.1 or 3.2 kernel to address this issue
(http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
But I don't believe this patch fixed existing occurrences of this
error.Apparently not. It's still there.
Quoted 2 lines
At this point, the quickest solution for you may be to rebuild and reformat this RAID assembly, and restore this data from backups.
Yepp, I did that. But in fact, some data is missing. It is not essential, but nice to have.
Quoted 4 lines
If you don't have a backup of this data, and since your array seems to be working pretty well in a degraded state, this would be a really good time to look at a strategy of getting a backup of this data before doing many more attempts at rescue.
Done. It's all save on another ext4 drive.
So, let's play ;-)
Could you please help me trying to restore the missing Data?What I tried sofar was:
./btrfs-restore/dev/sdc1/mnt/restore/Quoted 7 lines
It worked, in a way that it restored what I already had. What's odd aswell is, that btrfs scrub did run through without errors. So, the missing data could have been (accidentally) deleted by me. But I don't think... nevertheless I cannot exclude.
What I know is the (original) Path of the Data.
You could try btrfs-debug-tree, and search for any traces of your file. However, be ready to sift through a massive amount of output. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Hello,
I re-send this message, hoping that someone can give me a hint?
Regards,
Hendrik
Am 18.12.2012 23:17, schrieb Hendrik Friedel:
Quoted 89 lines
Hi Mitch, hi all,
thanks for your hint.
I used btrfs-debug-tree now.
With -e, the output is empty. But without -e I do get a bit output file. When I search for Filenames that I am missing, I get:grep Sting
big_output_file|grep Berlin
namelen 20 datalen 0 name:Sting_Live_in_Berlinnamelen 20 datalen 0 name:Sting_Live_in_Berlininode ref index 29 namelen 20 name:Sting_Live_in_BerlinThat looks good.
That raises two questions now: Can I restore the file?
And: Can I do that for a whole Path (e.g. ./Video/)Greetings&Thanks!
HendrikAm 15.12.2012 23:24, schrieb Mitch Harder:
Quoted 65 lines
Quoted 48 lines · Hendrik Friedel
Hello Mitch, hello all,
Quoted 4 lines
Since btrfs has significant improvements and fixes in each kernel
release, and since very few of these changes are backported, it is recommended to use the latest kernels available.
Ok, it's 3.7 now.
Quoted 8 lines
The "root ### inode ##### errors 400" are an indication that there is an inconsistency in the inode size. There was a patch included in the 3.1 or 3.2 kernel to address this issue
But I don't believe this patch fixed existing occurrences of this
error.Apparently not. It's still there.
Quoted 2 lines
At this point, the quickest solution for you may be to rebuild and reformat this RAID assembly, and restore this data from backups.
Yepp, I did that. But in fact, some data is missing. It is not essential,
but nice to have.Quoted 4 lines
If you don't have a backup of this data, and since your array seems to be working pretty well in a degraded state, this would be a really good time to look at a strategy of getting a backup of this data before doing many more attempts at rescue.
Done. It's all save on another ext4 drive.
So, let's play ;-)
Could you please help me trying to restore the missing Data?What I tried sofar was:
./btrfs-restore/dev/sdc1/mnt/restore/Quoted 7 lines
It worked, in a way that it restored what I already had. What's odd aswell is, that btrfs scrub did run through without errors. So, the missing data could have been (accidentally) deleted by me. But I don't think... nevertheless I cannot exclude.
What I know is the (original) Path of the Data.
You could try btrfs-debug-tree, and search for any traces of your file. However, be ready to sift through a massive amount of output. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Hendrik Friedel Auf dem Brink 12 28844 Weyhe Mobil 0178 1874363
Quoted 7 lines · Hendrik Friedel
Hello,
I re-send this message, hoping that someone can give me a hint?
Regards,
Hendrik
Two possibilities come to mind (although there may be others).
(1) The file still exists, but it is somewhere you did not expect.
(2) Your filesystem tree has some sort of corruption.
For item (1), have you thoroughly searched the entire volume for this file with something like:
find <path/to/volume/top/level/mount> -iname 'Sting_Live_in_Berlin'
It is possible that the file exists in a snapshot or different directory then you were expecting.
If the filesystem tree is corrupted, the task becomes tricky.
Perhaps you can look at the Wiki entry for how the filesystem tree is constructed:
https://btrfs.wiki.kernel.org/index.php/Trees
Then examine the btrfs-debug-tree output around these entries, and try to determine why the tree still has entries for these files, but does not show these files nor report the problem with btrfsck.