uncorrectable errors in Raid 10
Hello,
I have done a scrub on my Btrfs Raid 10 and have 2 uncorrectable errors. In fact I cannot access 2 directories, even as root, permission is denied and all directory attributes in ls -la are ????.
Before I have run this filesystem as Raid 1 with 3 disks without any problems for more than a year. Scrubbed regularily. A month ago I added a fouth HDD and balanced to Raid10. I am not sure if I did a scub afterwards, but I usualy do. Now it found that errors. Smart status of HDDs is healthy. The 2 directories were read-only for some years and not even read in the last month.
What can I do now? Should I do a btrfs rescue? Which device, it is a raid 10? Probably my files are OK, only the directories I cannot access. How to recover the files?
Thanks in advance!
Steffen
Here is my data:
Linux bigbox 4.13.0-17-generic #20-Ubuntu SMP Mon Nov 6 10:04:08 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
btrfs-progs v4.12
Label: 'Videos' uuid: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
Total devices 4 FS bytes used 1.44TiB
devid 3 size 1.82TiB used 785.56GiB path /dev/sdc2
devid 4 size 1.82TiB used 785.56GiB path /dev/sde2
devid 5 size 1.82TiB used 785.56GiB path /dev/sdd2
devid 6 size 1.36TiB used 785.56GiB path /dev/sdf1Data, RAID10: total=1.51TiB, used=1.43TiB System, RAID10: total=128.00MiB, used=192.00KiB Metadata, RAID10: total=21.00GiB, used=17.76GiB GlobalReserve, single: total=512.00MiB, used=0.00B
% sudo btrfs scrub start -Bd /
scrub device /dev/sdc2 (id 3) canceled
scrub started at Sun Nov 19 08:43:21 2017 and was aborted after
01:58:27
total bytes scrubbed: 568.86GiB with 0 errors
scrub device /dev/sde2 (id 4) canceled
scrub started at Sun Nov 19 08:43:21 2017 and was aborted after
01:58:28
total bytes scrubbed: 702.88GiB with 1 errors
error details: verify=1
corrected errors: 0, uncorrectable errors: 1, unverified errors: 0
scrub device /dev/sdd2 (id 5) done
scrub started at Sun Nov 19 08:43:21 2017 and finished after 01:48:00 total bytes scrubbed: 737.74GiB with 1 errors
error details: verify=1
corrected errors: 0, uncorrectable errors: 1, unverified errors: 0
scrub device /dev/sdf1 (id 6) canceled
scrub started at Sun Nov 19 08:43:21 2017 and was aborted after
01:58:28
total bytes scrubbed: 506.02GiB with 0 errors
[ 4.985712] BTRFS: device label Videos devid 6 transid 772463 /dev/sdf1 [ 4.985882] BTRFS: device label Videos devid 4 transid 772463 /dev/sde2 [ 4.986541] BTRFS: device label Videos devid 5 transid 772463 /dev/sdd2 [ 4.986713] BTRFS: device label Videos devid 3 transid 772463 /dev/sdc2 [ 5.007986] BTRFS info (device sdc2): disk space caching is enabled [ 5.007988] BTRFS info (device sdc2): has skinny extents [ 5.149910] BTRFS info (device sdc2): bdev /dev/sdd2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [ 5.149916] BTRFS info (device sdc2): bdev /dev/sde2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 3 [ 25.482987] BTRFS info (device sdc2): use lzo compression [ 25.482990] BTRFS info (device sdc2): disk space caching is enabled [57830.611730] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [57830.611732] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [57830.611734] BTRFS error (device sdc2): bdev /dev/sdd2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 4 [57832.688689] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sdd2 [57870.488081] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [57870.488083] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [57870.488085] BTRFS error (device sdc2): bdev /dev/sde2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 4 [57870.500114] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sde2 [88979.005712] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [88979.005718] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [88979.005720] BTRFS error (device sdc2): bdev /dev/sdd2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 5 [88979.036670] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sdd2 [89026.609561] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [89026.609563] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [89026.609566] BTRFS error (device sdc2): bdev /dev/sde2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 5 [89026.617423] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sde2 [97765.699866] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [97765.699869] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [97765.699871] BTRFS error (device sdc2): bdev /dev/sdd2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 6 [97765.735887] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sdd2 [97789.909036] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [97789.909039] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [97789.909041] BTRFS error (device sdc2): bdev /dev/sde2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 6 [97789.918254] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sde2 [104087.660640] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [104087.660643] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [104087.660645] BTRFS error (device sdc2): bdev /dev/sdd2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 7 [104087.696248] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sdd2 [104126.980483] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [104126.980486] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sde2, sector 348423360: metadata leaf (level 0) in tree 318 [104126.980488] BTRFS error (device sdc2): bdev /dev/sde2 errs: wr 0, rd 0, flush 0, corrupt 0, gen 7 [104126.995705] BTRFS error (device sdc2): unable to fixup (regular) error at logical 17478699876352 on dev /dev/sde2
steffen@bigbox ~ % sudo btrfs device stats /
[/dev/sdc2].write_io_errs 0 [/dev/sdc2].read_io_errs 0 [/dev/sdc2].flush_io_errs 0 [/dev/sdc2].corruption_errs 0 [/dev/sdc2].generation_errs 0 [/dev/sde2].write_io_errs 0 [/dev/sde2].read_io_errs 0 [/dev/sde2].flush_io_errs 0 [/dev/sde2].corruption_errs 0 [/dev/sde2].generation_errs 7 [/dev/sdd2].write_io_errs 0 [/dev/sdd2].read_io_errs 0 [/dev/sdd2].flush_io_errs 0 [/dev/sdd2].corruption_errs 0 [/dev/sdd2].generation_errs 7 [/dev/sdf1].write_io_errs 0 [/dev/sdf1].read_io_errs 0 [/dev/sdf1].flush_io_errs 0 [/dev/sdf1].corruption_errs 0 [/dev/sdf1].generation_errs 0
Quoted 6 lines · Steffen Sindzinski
[57830.611730] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [57830.611732] BTRFS warning (device sdc2): checksum/header error at logical17478699876352 on dev
/dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318
The same leaf is corrupt in the same physical sector on two devices. I'm guessing the checksum was computed incorrectly and written twice, affecting both copies. I doubt it's a device problem. It might be useful to look through the archives specifically for checksum header error, it's kinda interesting Btrfs knows the problem is specifically there.
What do you get for:
btrfs-debut-tree -b 17478699876352 /dev/sdd2
I think the problem is isolated but you're probably best off to freshen up backups now while you can. Yes you can use restore to get data off the volume if you don't already have backups. And as for btrfs check --repair, only do that once you have backups and you're prepared to lose the file system.
Signature · 1 line
Chris Murphy
Quoted 9 lines · Steffen Sindzinski
Quoted 7 lines
[57830.611730] BTRFS warning (device sdc2): checksum/header error at logical 17478699876352 on dev /dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318 [57830.611732] BTRFS warning (device sdc2): checksum/header error at logical17478699876352 on dev
/dev/sdd2, sector 338986176: metadata leaf (level 0) in tree 318
Quoted 6 lines
The same leaf is corrupt in the same physical sector on two devices. I'm guessing the checksum was computed incorrectly and written twice, affecting both copies. I doubt it's a device problem. It might be useful to look through the archives specifically for checksum header error, it's kinda interesting Btrfs knows the problem is specifically there.
Sounds reasonable. Perhaps a memory check on the machine would be a good idea_
roy
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAJCQCtRwO5E_xN-7u22uBQ57sYpUf8qQ7O0Abz2m+K8e9+DgdA@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Steffen Sindzinski
Hello,
I did btrfs check --readonly on both disk without finding any error. To reconfirm I did a scrub again which still has found 2 uncorrectable errors.
Try --mode=lowmem option with btrfs-progs 4.3.3 or better 4.14. This is a new implementation of btrfs check and sometimes it comes up with different results. It's strange that there's only this error found by scrub and not by btrfs check which should be fully checking all metadata for sanity, and in the process it would surely hit a bad checksum.
Signature · 1 line
Chris Murphy
Hi,
I updated to btrfs-progs v4.14-5-gf09e98a3. Unfortunately after 10h Gnome desktop / Arch crashed with btrfs check --lowmem unfinished. I will run it this night again.
The result until crash was:
Checking filesystem on /dev/sdd2
UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
ERROR: extent[15923092037632, 73728] referencer count mismatch (root: 260, owner: 3631467, offset: 1019904) wanted: 4, have: 6
ERROR: extent[16078964924416, 69632] referencer count mismatch (root: 260, owner: 4086589, offset: 4296704) wanted: 5, have: 7
Steffen
Am 22.11.2017 um 17:42 schrieb Chris Murphy:
Quoted 14 lines · Steffen Sindzinski
Quoted 4 lines
Hello,
I did btrfs check --readonly on both disk without finding any error. To reconfirm I did a scrub again which still has found 2 uncorrectable errors.
Try --mode=lowmem option with btrfs-progs 4.3.3 or better 4.14. This is a new implementation of btrfs check and sometimes it comes up with different results. It's strange that there's only this error found by scrub and not by btrfs check which should be fully checking all metadata for sanity, and in the process it would surely hit a bad checksum.
Quoted 5 lines · Steffen Sindzinski
Hi,
I updated to btrfs-progs v4.14-5-gf09e98a3. Unfortunately after 10h Gnome desktop / Arch crashed with btrfs check --lowmem unfinished. I will run it this night again.
Any kernel backtrace about the crash?
IIRC user space program like btrfs check should not trigger a kernel crash.
Quoted 9 lines
The result until crash was:
Checking filesystem on
/dev/sdd2
UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
ERROR: extent[15923092037632, 73728] referencer count mismatch (root:
260, owner: 3631467, offset: 1019904) wanted: 4, have: 6
ERROR: extent[16078964924416, 69632] referencer count mismatch (root:
260, owner: 4086589, offset: 4296704) wanted: 5, have: 7
This is known bug, please use this branch instead.
https://github.com/adam900710/btrfs-progs/tree/lowmem_fix
Thanks,
Qu
Quoted 25 lines
Steffen
Am 22.11.2017 um 17:42 schrieb Chris Murphy:
Quoted 16 lines
Quoted 5 lines · Steffen Sindzinski
Hello,
I did btrfs check --readonly on both disk without finding any error. To reconfirm I did a scrub again which still has found 2 uncorrectable errors.
Try --mode=lowmem option with btrfs-progs 4.3.3 or better 4.14. This is a new implementation of btrfs check and sometimes it comes up with different results. It's strange that there's only this error found by scrub and not by btrfs check which should be fully checking all metadata for sanity, and in the process it would surely hit a bad checksum.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,
I think just some XWayland Nouveau bug:
Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: TRAP ch 22 [007d08d000 Xwayland[781]] Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: GPC0/TPC0/TEX: 80000041 Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: GPC0/TPC1/TEX: 80000041 Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: GPC0/TPC2/TEX: 80000041 Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: GPC0/TPC3/TEX: 80000041 Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: gr: GPC0/TPC4/TEX: 80000041 Nov 23 12:46:26 bigbox kernel: nouveau 0000:01:00.0: fifo: read fault at 0005860000 engine 00 [GR] client 15 [GPC0/PE_4] reason 02 [PTE] on channel 22 [007d08d000 Xwayland[781]]
Steffen
Am 23.11.2017 um 13:14 schrieb Qu Wenruo:
Quoted 55 lines · Steffen Sindzinski
Quoted 5 lines
Hi,
I updated to btrfs-progs v4.14-5-gf09e98a3. Unfortunately after 10h Gnome desktop / Arch crashed with btrfs check --lowmem unfinished. I will run it this night again.
Any kernel backtrace about the crash?
IIRC user space program like btrfs check should not trigger a kernel crash.
Quoted 9 lines
The result until crash was:
Checking filesystem on
/dev/sdd2
UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
ERROR: extent[15923092037632, 73728] referencer count mismatch (root:
260, owner: 3631467, offset: 1019904) wanted: 4, have: 6
ERROR: extent[16078964924416, 69632] referencer count mismatch (root:
260, owner: 4086589, offset: 4296704) wanted: 5, have: 7This is known bug, please use this branch instead.
https://github.com/adam900710/btrfs-progs/tree/lowmem_fixThanks,
QuQuoted 25 lines
Steffen
Am 22.11.2017 um 17:42 schrieb Chris Murphy:
Quoted 16 lines
Quoted 5 lines · Steffen Sindzinski
Hello,
I did btrfs check --readonly on both disk without finding any error. To reconfirm I did a scrub again which still has found 2 uncorrectable errors.
Try --mode=lowmem option with btrfs-progs 4.3.3 or better 4.14. This is a new implementation of btrfs check and sometimes it comes up with different results. It's strange that there's only this error found by scrub and not by btrfs check which should be fully checking all metadata for sanity, and in the process it would surely hit a bad checksum.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Steffen Sindzinski
Hello,
I did btrfs check --readonly on both disk without finding any error. To reconfirm I did a scrub again which still has found 2 uncorrectable
Still metadata corruption?
Quoted 2 lines
errors. I had to boot into Arch Linux 4.13.12-1-ARCH btrfs-progs v4.13 to run btrfs check.
Well, btrfs-progs v4.13 from Arch is out-of-data for a while. Even ignoring the latest v4.14 release, there is no v4.13.x releases for Arch.
Quoted 30 lines
Checking filesystem on
/dev/sde2
UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 1598669742080 bytes used, no error found
total csum bytes: 1540740308
total tree bytes: 19391266816
total fs tree bytes: 9844703232
total extent tree bytes: 6808731648
btree space waste bytes: 3471512438
file data blocks allocated: 2124703174656
referenced 1454353633280
sudo btrfs check --readonly /dev/sde2 2708,20s user 594,58s system 15% cpu 5:45:40,36 total
***********************************************************************
sudo btrfs check --readonly/dev/sdd2
Checking filesystem on/dev/sdd2
UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 1598669742080 bytes used, no error found
total csum bytes: 1540740308
Quoted 8 lines
total tree bytes: 19391266816
total fs tree bytes: 9844703232
total extent tree bytes: 6808731648
btree space waste bytes: 3471512438
file data blocks allocated: 2124703174656
referenced 1454353633280
sudo btrfs check --readonly /dev/sdd2 2655,23s user 626,48s system 15% cpu 5:56:16,85 total
Just as Chris mentioned, if the scrub is reporting metadata corruption, please try "btrfs check --mode=lowmem" to see if lowmem mode can detect such corruption.
Quoted 6 lines
I mentioned that I cannot access 2 directories anymore. Permission is denied, not even as root I have permission nor I can change ownership. This happens in Ubuntu 17.10, kernel 4.13.0-17-generic. It looks like this:
Quoted 10 lines
ls -la The*
The-Wire:
ls: Zugriff auf 'The-Wire/.' nicht möglich: Keine Berechtigung
ls: Zugriff auf 'The-Wire/..' nicht möglich: Keine Berechtigung <...>
insgesamt 0
d????????? ? ? ? ? ? .
d????????? ? ? ? ? ? ..
-????????? ? ? ? ? ? The Wire S1E10.m4v -????????? ? ? ? ? ? The Wire S1E11.m4v
Normally this means DIR_INDEX/DIR_ITEM missing or points to incorrect inode.
Quoted 6 lines
<...>
Oddly in Arch Linux, 4.13.12-1-ARCH, btrfs-progs v4.13, I can access this same directory, permission is set correctly.>
The debug trees for both drives you may find attached to this mail. They were done in Ubunutu.
Not really helping.
I don't know how you take the dump, but it is only a leaf of subvolume, full of EXTENT_DATA without any useful info.
Thanks,
Qu
Quoted 48 lines
For reference here the scrub report on Arch linux.
Steffen
Am 20.11.2017 um 21:26 schrieb Chris Murphy:
Quoted 39 lines
Quoted 13 lines · Steffen Sindzinski
Hi,
I did btrfs-debug-tree for this block on both devices. The result is attached to this mail.
It is really weird, same block, different drives, different sector. I have
no problem with bit rod. Btrfs worked perfectly fine with this both HDDs for
so long on Raid1. The drive sdf1 which I attached to form a Raid 10 was also
in a different Btrfs in the same machine for years flawlessly.Quoted 5 lines
I have not found any other checksum errors than the ones from this scrub.
Is there no way to just safely recreate the checksum of this particular block from the disk contents?
I'll cc Qu because I don't understand what's going on. It's the 2nd case of both copies of metadata being bad in as many days, which could just be coincidence.
I also don't understand the specific error "checksum/header error" which sounds to me like Btrfs knows the leaf is otherwise OK, but there is some kind of problem with either the leaf csum or its header. In which case I'd like to think that btrfs check --repair can fix this kind of problem.
What do you get for btrfs check without --repair?
Curiously, your scrub complains about this "checksum/header error" but btrfs-debug-tree gives no indication that leaf has any problem at all.