How to fix these btrfs errors

DISCUSSION Zach Aller 2 participants 5 messages Apr 30, 2017 – May 1, 2017 Btrfs file system

Share & tools

Zach Aller

author

2017-04-30 21:08 UTC #0 lore raw

So I have been getting a lot of errors with a btrfs filesystem. I would like to figure out how to fix them without losing data or having to redo the data. Below I have the output of some of the requested command. I have ran a btrfs check without the ‹repair option I wanted to check here first to see if it was safe to do so based off what I have read it seems to be unsafe is this true for this case or is there some other thing I should do to try and fix these errors?

uname -a
Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

./btrfs --version
btrfs-progs v4.10

./btrfs fi show
Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
Total devices 1 FS bytes used 17.04TiB
devid 1 size 21.83TiB used 17.28TiB path /dev/sda

./btrfs fi df /mnt/btrfs/
Data, single: total=17.24TiB, used=17.02TiB System, DUP: total=8.00MiB, used=1.81MiB Metadata, DUP: total=19.50GiB, used=17.97GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Here is a dmesg snippet

[    3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
devid 1 transid 72387 /dev/sda
[   12.907658] BTRFS info (device sda): disk space caching is enabled
[   12.907659] BTRFS info (device sda): has skinny extents
[   13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 217, gen 19
[20956.415076] BTRFS info (device sda): The free space cache file
(9804365955072) is invalid. skip it
[36292.358558] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[36292.358563] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 218, gen 19
[36292.364717] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[36292.368359] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[36292.368363] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[36292.368369] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 219, gen 19
[36292.372004] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[36292.375825] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[36292.375830] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[36292.375836] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 220, gen 19
[36292.379664] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[36294.785933] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[36294.785938] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[36294.785945] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 220, gen 20
[36294.811292] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[36294.815711] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[36294.815715] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[36294.815721] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 220, gen 21
[36294.819616] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[36294.823458] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[36294.823463] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[36294.823469] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 220, gen 22
[36294.827285] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[38085.152105] BTRFS error (device sda): bad tree block start 65536
5614914584576
[38085.920817] BTRFS error (device sda): bad tree block start 65536
5614914584576
[38085.934875] BTRFS error (device sda): bad tree block start 499762348032
5614914584576
[60948.070835] BTRFS info (device sda): space_info 4 has 1641807872 free,
is not full
[60948.070836] BTRFS info (device sda): space_info total=20937965568,

used=19296026624, pinned=0, reserved=0, may_use=65536, readonly=65536

[60948.070623] ------------[ cut here ]------------
[60948.070636] WARNING: CPU: 5 PID: 18318 at
/home/kernel/COD/linux/fs/btrfs/extent-tree.c:5702
btrfs_free_block_groups+0x399/0x440 [btrfs]
[60948.070637] Modules linked in: xt_nat xt_tcpudp veth binfmt_misc

ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi pcbc snd_hda_codec_realtek aesni_intel aes_x86_64 crypto_simd snd_hda_codec_generic glue_helper cryptd input_leds snd_hda_intel snd_hda_codec snd_hda_core hci_uart snd_hwdep btbcm ppdev btqca snd_pcm intel_cstate snd_timer snd parport_pc intel_rapl_perf btintel bluetooth mei_me mei intel_lpss_acpi soundcore parport acpi_als mac_hid kfifo_buf

[60948.070657]  intel_lpss shpchp industrialio acpi_pad autofs4 btrfs xor
raid6_pq hid_generic usbhid i915 i2c_algo_bit drm_kms_helper e1000e
syscopyarea sysfillrect sysimgblt fb_sys_fops ptp drm ahci pps_core hptiop
libahci pinctrl_sunrisepoint i2c_hid pinctrl_intel video hid
[60948.070667] CPU: 5 PID: 18318 Comm: umount Not tainted
4.11.0-041100rc8-generic #201704232131
[60948.070668] Hardware name: To Be Filled By O.E.M. To Be Filled By
O.E.M./Z270M Pro4, BIOS P1.10 11/14/2016
[60948.070668] Call Trace:
[60948.070671]  dump_stack+0x63/0x81
[60948.070672]  __warn+0xcb/0xf0
[60948.070673]  warn_slowpath_null+0x1d/0x20
[60948.070679]  btrfs_free_block_groups+0x399/0x440 [btrfs]
[60948.070686]  close_ctree+0x12b/0x310 [btrfs]
[60948.070690]  btrfs_put_super+0x15/0x20 [btrfs]
[60948.070692]  generic_shutdown_super+0x6f/0x100
[60948.070693]  kill_anon_super+0x12/0x20
[60948.070697]  btrfs_kill_super+0x18/0x110 [btrfs]
[60948.070698]  deactivate_locked_super+0x43/0x70
[60948.070699]  deactivate_super+0x5a/0x60
[60948.070700]  cleanup_mnt+0x3f/0x90
[60948.070701]  __cleanup_mnt+0x12/0x20
[60948.070702]  task_work_run+0x80/0xa0
[60948.070703]  exit_to_usermode_loop+0xaa/0xb0
[60948.070704]  syscall_return_slowpath+0x59/0x60
[60948.070705]  entry_SYSCALL_64_fastpath+0xab/0xad
[60948.070705] RIP: 0033:0x7f2903d44897
[60948.070706] RSP: 002b:00007fffcecd7e38 EFLAGS: 00000246 ORIG_RAX:
00000000000000a6
[60948.070706] RAX: 0000000000000000 RBX: 00000000020c6030 RCX:
00007f2903d44897
[60948.070707] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
00000000020c6210
[60948.070707] RBP: 00000000020c6210 R08: 0000000000000000 R09:
0000000000000015
[60948.070708] R10: 00000000000006b2 R11: 0000000000000246 R12:
00007f290424d83c
[60948.070708] R13: 0000000000000000 R14: 0000000000000000 R15:
00007fffcecd80c0
[60948.070709] ---[ end trace dd48a6c484b9e2c4 ]---
[60948.070709] ------------[ cut here ]------------
[60948.070715] WARNING: CPU: 5 PID: 18318 at
/home/kernel/COD/linux/fs/btrfs/extent-tree.c:5703
btrfs_free_block_groups+0x3b8/0x440 [btrfs]
[60948.070715] Modules linked in: xt_nat xt_tcpudp veth binfmt_misc

[60948.070728]  intel_lpss shpchp industrialio acpi_pad autofs4 btrfs xor
raid6_pq hid_generic usbhid i915 i2c_algo_bit drm_kms_helper e1000e
syscopyarea sysfillrect sysimgblt fb_sys_fops ptp drm ahci pps_core hptiop
libahci pinctrl_sunrisepoint i2c_hid pinctrl_intel video hid
[60948.070734] CPU: 5 PID: 18318 Comm: umount Tainted: G        W
4.11.0-041100rc8-generic #201704232131
[60948.070734] Hardware name: To Be Filled By O.E.M. To Be Filled By
O.E.M./Z270M Pro4, BIOS P1.10 11/14/2016
[60948.070734] Call Trace:
[60948.070735]  dump_stack+0x63/0x81
[60948.070736]  __warn+0xcb/0xf0
[60948.070737]  warn_slowpath_null+0x1d/0x20
[60948.070742]  btrfs_free_block_groups+0x3b8/0x440 [btrfs]
[60948.070748]  close_ctree+0x12b/0x310 [btrfs]
[60948.070753]  btrfs_put_super+0x15/0x20 [btrfs]
[60948.070754]  generic_shutdown_super+0x6f/0x100
[60948.070755]  kill_anon_super+0x12/0x20
[60948.070760]  btrfs_kill_super+0x18/0x110 [btrfs]
[60948.070761]  deactivate_locked_super+0x43/0x70
[60948.070762]  deactivate_super+0x5a/0x60
[60948.070762]  cleanup_mnt+0x3f/0x90
[60948.070763]  __cleanup_mnt+0x12/0x20
[60948.070764]  task_work_run+0x80/0xa0
[60948.070765]  exit_to_usermode_loop+0xaa/0xb0
[60948.070765]  syscall_return_slowpath+0x59/0x60
[60948.070766]  entry_SYSCALL_64_fastpath+0xab/0xad
[60948.070766] RIP: 0033:0x7f2903d44897
[60948.070767] RSP: 002b:00007fffcecd7e38 EFLAGS: 00000246 ORIG_RAX:
00000000000000a6
[60948.070768] RAX: 0000000000000000 RBX: 00000000020c6030 RCX:
00007f2903d44897
[60948.070768] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
00000000020c6210
[60948.070768] RBP: 00000000020c6210 R08: 0000000000000000 R09:
0000000000000015
[60948.070769] R10: 00000000000006b2 R11: 0000000000000246 R12:
00007f290424d83c
[60948.070769] R13: 0000000000000000 R14: 0000000000000000 R15:
00007fffcecd80c0
[60948.070770] ---[ end trace dd48a6c484b9e2c5 ]---
[60948.070775] ------------[ cut here ]------------
[60948.070780] WARNING: CPU: 5 PID: 18318 at
/home/kernel/COD/linux/fs/btrfs/extent-tree.c:9826
btrfs_free_block_groups+0x29d/0x440 [btrfs]
[60948.070780] Modules linked in: xt_nat xt_tcpudp veth binfmt_misc

[60948.070793]  intel_lpss shpchp industrialio acpi_pad autofs4 btrfs xor
raid6_pq hid_generic usbhid i915 i2c_algo_bit drm_kms_helper e1000e
syscopyarea sysfillrect sysimgblt fb_sys_fops ptp drm ahci pps_core hptiop
libahci pinctrl_sunrisepoint i2c_hid pinctrl_intel video hid
[60948.070799] CPU: 5 PID: 18318 Comm: umount Tainted: G        W
4.11.0-041100rc8-generic #201704232131
[60948.070799] Hardware name: To Be Filled By O.E.M. To Be Filled By
O.E.M./Z270M Pro4, BIOS P1.10 11/14/2016
[60948.070799] Call Trace:
[60948.070800]  dump_stack+0x63/0x81
[60948.070801]  __warn+0xcb/0xf0
[60948.070801]  warn_slowpath_null+0x1d/0x20
[60948.070807]  btrfs_free_block_groups+0x29d/0x440 [btrfs]
[60948.070813]  close_ctree+0x12b/0x310 [btrfs]
[60948.070817]  btrfs_put_super+0x15/0x20 [btrfs]
[60948.070818]  generic_shutdown_super+0x6f/0x100
[60948.070819]  kill_anon_super+0x12/0x20
[60948.070824]  btrfs_kill_super+0x18/0x110 [btrfs]
[60948.070825]  deactivate_locked_super+0x43/0x70
[60948.070826]  deactivate_super+0x5a/0x60
[60948.070826]  cleanup_mnt+0x3f/0x90
[60948.070827]  __cleanup_mnt+0x12/0x20
[60948.070828]  task_work_run+0x80/0xa0
[60948.070829]  exit_to_usermode_loop+0xaa/0xb0
[60948.070829]  syscall_return_slowpath+0x59/0x60
[60948.070830]  entry_SYSCALL_64_fastpath+0xab/0xad
[60948.070830] RIP: 0033:0x7f2903d44897
[60948.070831] RSP: 002b:00007fffcecd7e38 EFLAGS: 00000246 ORIG_RAX:
00000000000000a6
[60948.070831] RAX: 0000000000000000 RBX: 00000000020c6030 RCX:
00007f2903d44897
[60948.070832] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
00000000020c6210
[60948.070832] RBP: 00000000020c6210 R08: 0000000000000000 R09:
0000000000000015
[60948.070832] R10: 00000000000006b2 R11: 0000000000000246 R12:
00007f290424d83c
[60948.070833] R13: 0000000000000000 R14: 0000000000000000 R15:
00007fffcecd80c0
[60948.070833] ---[ end trace dd48a6c484b9e2c6 ]---
[60948.070835] BTRFS info (device sda): space_info 4 has 1641807872 free,
is not full
[60948.070836] BTRFS info (device sda): space_info total=20937965568,

used=19296026624, pinned=0, reserved=0, may_use=65536, readonly=65536

./btrfs check /dev/sda
Checking filesystem on /dev/sda
UUID: bdd89c26-038d-49fd-b895-52b8deb989cc
checking extents
checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 bytenr mismatch, want=5614914584576, have=65536
checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 bytenr mismatch, want=5614914600960, have=65536
checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED bytenr mismatch, want=5614914617344, have=196608
checksum verify failed on 15732835876864 found 0EA0AA8A wanted EA43173C checksum verify failed on 15732835876864 found 0EA0AA8A wanted EA43173C checksum verify failed on 15732835893248 found E4E3BDB6 wanted 00000000 checksum verify failed on 15732835893248 found E4E3BDB6 wanted 00000000 owner ref check failed [5614914584576 16384]
owner ref check failed [5614914600960 16384]
owner ref check failed [5614914617344 16384]
checking free space cache
checking fs roots
checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 bytenr mismatch, want=5614914584576, have=65536
checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 checksum verify failed on 5614914600960 found E0F4CA4E wanted 041777F8 bytenr mismatch, want=5614914600960, have=65536
checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED checksum verify failed on 5614914617344 found 0258CB5B wanted E6BB76ED bytenr mismatch, want=5614914617344, have=196608
checksum verify failed on 15732835876864 found 0EA0AA8A wanted EA43173C checksum verify failed on 15732835876864 found 0EA0AA8A wanted EA43173C checksum verify failed on 15732835893248 found E4E3BDB6 wanted 00000000 checksum verify failed on 15732835893248 found E4E3BDB6 wanted 00000000 checking csums
checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 checksum verify failed on 5614914584576 found A74544D2 wanted 43A6F964 bytenr mismatch, want=5614914584576, have=65536
Error going to next leaf -5
checking root refs
found 18737354526720 bytes used, no error found
total csum bytes: 18274778176
total tree bytes: 19297878016
total fs tree bytes: 388939776
total extent tree bytes: 132546560
btree space waste bytes: 421826725
file data blocks allocated: 18752764727296
referenced 18717933031424

Chris Murphy

2017-04-30 21:35 UTC #1 lore raw

Quoted 11 lines · Zach Aller

uname -a
Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
./btrfs --version
btrfs-progs v4.10
./btrfs fi show
Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
Total devices 1 FS bytes used 17.04TiB
devid 1 size 21.83TiB used 17.28TiB path /dev/sda

How old is the file system? Is this a recent problem with just 4.11rc8? Is most of the 17TB written with a particular kernel version, which?

Quoted 21 lines

Here is a dmesg snippet

[    3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
devid 1 transid 72387 /dev/sda
[   12.907658] BTRFS info (device sda): disk space caching is enabled
[   12.907659] BTRFS info (device sda): has skinny extents
[   13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 217, gen 19
[20956.415076] BTRFS info (device sda): The free space cache file
(9804365955072) is invalid. skip it
[36292.358558] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[36292.358563] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 218, gen 19
[36292.364717] BTRFS error (device sda): unable to fixup (regular) error

Quoted 1 line

at logical 5614914584576 on dev /dev/sda

Both copies of metadata are failing checksum, so it can't be fixed. It suggests there's a hardware problem (memory or storage), or maybe a new bug.

Have there been any crashes while writing to the file system? What is the storage stack configuration? 22TB for a single block device means it's built up from something else.

I'd dig around for any non-btrfs storage stack related errors in the meantime, maybe a dev will have some idea what's going on from the call traces, I'm not sure what they mean.

Signature · 1 line

Chris Murphy

Zach Aller

author

2017-04-30 21:39 UTC #2 lore raw

It is a recent filesystem the data was written with kernel 4.10, today I upgraded to 4.11rc8 to see if it helped anything which it did not.

Quoted 60 lines · On 4/30/17, 4:35 PM, "[email protected] on behalf of Chris Murphy"…

Quoted 11 lines · Zach Aller
uname -a
Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
./btrfs --version
btrfs-progs v4.10
./btrfs fi show
Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
Total devices 1 FS bytes used 17.04TiB
devid 1 size 21.83TiB used 17.28TiB path /dev/sda
How old is the file system? Is this a recent problem with just 4.11rc8? Is most of the 17TB written with a particular kernel version, which?
Quoted 23 lines
Here is a dmesg snippet
[    3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
devid 1 transid 72387 /dev/sda
[   12.907658] BTRFS info (device sda): disk space caching is enabled
[   12.907659] BTRFS info (device sda): has skinny extents
[   13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 217, gen 19
[20956.415076] BTRFS info (device sda): The free space cache file
(9804365955072) is invalid. skip it
[36292.358558] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
[36292.358563] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
[36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 218, gen 19
[36292.364717] BTRFS error (device sda): unable to fixup (regular) error
Quoted 1 line
at logical 5614914584576 on dev /dev/sda
Both copies of metadata are failing checksum, so it can't be fixed. It suggests there's a hardware problem (memory or storage), or maybe a new bug.
Have there been any crashes while writing to the file system? What is the storage stack configuration? 22TB for a single block device means it's built up from something else.
I'd dig around for any non-btrfs storage stack related errors in the meantime, maybe a dev will have some idea what's going on from the call traces, I'm not sure what they mean.
Signature · 1 line
Chris Murphy

Zach Aller

author

2017-04-30 22:20 UTC #3 lore raw

Output of one more command.
-------
./btrfs inspect-internal dump-tree /dev/sda > dump.txt parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 Ignoring transid failure
WARNING: eb corrupted: item 0 eb level 3 next level 3, skipping the rest
-------

Here is the end of dump.txt after the above command failed.
-------

checksum tree key (CSUM_TREE ROOT_ITEM 0)
node 15732050329600 level 3 items 15 free 478 generation 73879 owner 7 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33

	key (EXTENT_CSUM EXTENT_CSUM 12582912) block 15732050345984 (960208151)
gen 73879
	key (EXTENT_CSUM EXTENT_CSUM 1036259938304) block 15732062109696
(960208869) gen 73879
	key (EXTENT_CSUM EXTENT_CSUM 2055219736576) block 22061012254720
(1346497330) gen 73796
	key (EXTENT_CSUM EXTENT_CSUM 3067616186368) block 15732273971200
(960221800) gen 72465
	key (EXTENT_CSUM EXTENT_CSUM 4125595508736) block 19670583918592
(1200597163) gen 71747
	key (EXTENT_CSUM EXTENT_CSUM 5160724217856) block 22060767379456
(1346482384) gen 73047
	key (EXTENT_CSUM EXTENT_CSUM 6225035022336) block 18786519695360
(1146638165) gen 71390
	key (EXTENT_CSUM EXTENT_CSUM 7317037834240) block 22060823707648
(1346485822) gen 73791
	key (EXTENT_CSUM EXTENT_CSUM 8520881180672) block 22061229129728
(1346510567) gen 72304
	key (EXTENT_CSUM EXTENT_CSUM 9556147200000) block 21773927448576
(1328975064) gen 72764
	key (EXTENT_CSUM EXTENT_CSUM 10643353645056) block 22061034897408
(1346498712) gen 73797
	key (EXTENT_CSUM EXTENT_CSUM 11886318727168) block 21773509066752
(1328949528) gen 73741
	key (EXTENT_CSUM EXTENT_CSUM 13041959669760) block 2532038983680
(154543395) gen 71958
	key (EXTENT_CSUM EXTENT_CSUM 14315004522496) block 21773773946880
(1328965695) gen 57898
	key (EXTENT_CSUM EXTENT_CSUM 16297194516480) block 22061036748800

(1346498825) gen 72864
uuid tree key (UUID_TREE ROOT_ITEM 0)
leaf 29376512 items 0 free space 16283 generation 6 owner 9 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33 data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0) leaf 29442048 items 2 free space 16061 generation 4 owner 18446744073709551607
fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33

	item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
		inode generation 3 transid 0 size 0 nbytes 16384
		block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
		sequence 0 flags 0x0(none)
		atime 1490407706.0 (2017-03-24 21:08:26)
		ctime 1490407706.0 (2017-03-24 21:08:26)
		mtime 1490407706.0 (2017-03-24 21:08:26)
		otime 1490407706.0 (2017-03-24 21:08:26)
	item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
		inode ref index 0 namelen 2 name: ..

total bytes 24004303781888
bytes used 18737354412032
uuid bdd89c26-038d-49fd-b895-52b8deb989cc

-------

Quoted 73 lines · On 4/30/17, 4:39 PM, "[email protected] on behalf of Zach…

It is a recent filesystem the data was written with kernel 4.10, today I upgraded to 4.11rc8 to see if it helped anything which it did not.
Quoted 52 lines · On 4/30/17, 4:35 PM, "[email protected] on behalf of Chris Murphy"…
Quoted 11 lines · Zach Aller
uname -a
Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
./btrfs --version
btrfs-progs v4.10
./btrfs fi show
Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
Total devices 1 FS bytes used 17.04TiB
devid 1 size 21.83TiB used 17.28TiB path /dev/sda
How old is the file system? Is this a recent problem with just 4.11rc8? Is most of the 17TB written with a particular kernel version, which?
Quoted 24 lines
Here is a dmesg snippet
[    3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
devid 1 transid 72387 /dev/sda
[   12.907658] BTRFS info (device sda): disk space caching is enabled
[   12.907659] BTRFS info (device sda): has skinny extents
[   13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 217, gen 19
[20956.415076] BTRFS info (device sda): The free space cache file
(9804365955072) is invalid. skip it
[36292.358558] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
[36292.358563] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
[36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd
0,
flush 0, corrupt 218, gen 19
[36292.364717] BTRFS error (device sda): unable to fixup (regular)
Quoted 2 lines
error
at logical 5614914584576 on dev /dev/sda
Both copies of metadata are failing checksum, so it can't be fixed. It suggests there's a hardware problem (memory or storage), or maybe a new bug.
Quoted 10 lines
Have there been any crashes while writing to the file system? What is the storage stack configuration? 22TB for a single block device means it's built up from something else.
I'd dig around for any non-btrfs storage stack related errors in the meantime, maybe a dev will have some idea what's going on from the call traces, I'm not sure what they mean.
Signature · 1 line
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Zach Aller

author

2017-05-01 02:35 UTC #4 lore raw

One more while running a scrub dmesg logs showed this

[98761.912449] scrub_handle_errored_block: 22 callbacks suppressed
[98761.912472] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[98761.912476] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[98761.912479] btrfs_dev_stat_print_on_error: 22 callbacks suppressed
[98761.912484] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 270, gen 22
[98761.921028] scrub_handle_errored_block: 22 callbacks suppressed
[98761.921034] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[98761.925274] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[98761.925278] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[98761.925284] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 271, gen 22
[98761.929502] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[98761.933715] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[98761.933717] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[98761.933722] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 22
[98761.937951] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[98763.775452] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[98763.775457] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[98763.775463] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 23
[98763.784019] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[98763.788323] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[98763.788328] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[98763.788334] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 24
[98763.792722] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[98763.797106] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[98763.797111] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[98763.797116] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 25
[98763.801401] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[100173.104082] BTRFS error (device sda): bad tree block start 65536
5614914584576
[100173.833202] BTRFS error (device sda): bad tree block start 65536
5614914584576
[100173.845121] BTRFS error (device sda): bad tree block start

499762348032 5614914584576

And then it was aborted

scrub status for bdd89c26-038d-49fd-b895-52b8deb989cc
scrub started at Sun Apr 30 16:27:12 2017 and was aborted after 03:29:08 total bytes scrubbed: 5.79TiB with 55 errors
error details: verify=3 csum=52
corrected errors: 0, uncorrectable errors: 55, unverified errors: 0

I also tried to run this with no luck

./btrfs inspect-internal logical-resolve 5614914584576 /mnt/btrfs/
ERROR: logical ino ioctl: No such file or directory /btrfs inspect-internal logical-resolve 5614914600960 /mnt/btrfs/
ERROR: logical ino ioctl: No such file or directory

Quoted 159 lines · Zach Aller

Output of one more command.
-------
./btrfs inspect-internal dump-tree /dev/sda > dump.txt parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 Ignoring transid failure
WARNING: eb corrupted: item 0 eb level 3 next level 3, skipping the rest
-------
Here is the end of dump.txt after the above command failed.
-------
checksum tree key (CSUM_TREE ROOT_ITEM 0)
node 15732050329600 level 3 items 15 free 478 generation 73879 owner 7 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33
	key (EXTENT_CSUM EXTENT_CSUM 12582912) block 15732050345984 (960208151)
gen 73879
	key (EXTENT_CSUM EXTENT_CSUM 1036259938304) block 15732062109696
(960208869) gen 73879
	key (EXTENT_CSUM EXTENT_CSUM 2055219736576) block 22061012254720
(1346497330) gen 73796
	key (EXTENT_CSUM EXTENT_CSUM 3067616186368) block 15732273971200
(960221800) gen 72465
	key (EXTENT_CSUM EXTENT_CSUM 4125595508736) block 19670583918592
(1200597163) gen 71747
	key (EXTENT_CSUM EXTENT_CSUM 5160724217856) block 22060767379456
(1346482384) gen 73047
	key (EXTENT_CSUM EXTENT_CSUM 6225035022336) block 18786519695360
(1146638165) gen 71390
	key (EXTENT_CSUM EXTENT_CSUM 7317037834240) block 22060823707648
(1346485822) gen 73791
	key (EXTENT_CSUM EXTENT_CSUM 8520881180672) block 22061229129728
(1346510567) gen 72304
	key (EXTENT_CSUM EXTENT_CSUM 9556147200000) block 21773927448576
(1328975064) gen 72764
	key (EXTENT_CSUM EXTENT_CSUM 10643353645056) block 22061034897408
(1346498712) gen 73797
	key (EXTENT_CSUM EXTENT_CSUM 11886318727168) block 21773509066752
(1328949528) gen 73741
	key (EXTENT_CSUM EXTENT_CSUM 13041959669760) block 2532038983680
(154543395) gen 71958
	key (EXTENT_CSUM EXTENT_CSUM 14315004522496) block 21773773946880
(1328965695) gen 57898
	key (EXTENT_CSUM EXTENT_CSUM 16297194516480) block 22061036748800
(1346498825) gen 72864
uuid tree key (UUID_TREE ROOT_ITEM 0)
leaf 29376512 items 0 free space 16283 generation 6 owner 9 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33 data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0) leaf 29442048 items 2 free space 16061 generation 4 owner 18446744073709551607
fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33
	item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
		inode generation 3 transid 0 size 0 nbytes 16384
		block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
		sequence 0 flags 0x0(none)
		atime 1490407706.0 (2017-03-24 21:08:26)
		ctime 1490407706.0 (2017-03-24 21:08:26)
		mtime 1490407706.0 (2017-03-24 21:08:26)
		otime 1490407706.0 (2017-03-24 21:08:26)
	item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
		inode ref index 0 namelen 2 name: ..
total bytes 24004303781888
bytes used 18737354412032
uuid bdd89c26-038d-49fd-b895-52b8deb989cc
-------
Quoted 76 lines · On 4/30/17, 4:39 PM, "[email protected] on behalf of Zach…
It is a recent filesystem the data was written with kernel 4.10, today I upgraded to 4.11rc8 to see if it helped anything which it did not.
Quoted 55 lines · On 4/30/17, 4:35 PM, "[email protected] on behalf of Chris Murphy"…
Quoted 11 lines · Zach Aller
uname -a
Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
./btrfs --version
btrfs-progs v4.10
./btrfs fi show
Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
Total devices 1 FS bytes used 17.04TiB
devid 1 size 21.83TiB used 17.28TiB path /dev/sda
How old is the file system? Is this a recent problem with just 4.11rc8? Is most of the 17TB written with a particular kernel version, which?
Quoted 23 lines
Here is a dmesg snippet
[    3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
devid 1 transid 72387 /dev/sda
[   12.907658] BTRFS info (device sda): disk space caching is enabled
[   12.907659] BTRFS info (device sda): has skinny extents
[   13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd
0,
flush 0, corrupt 217, gen 19
[20956.415076] BTRFS info (device sda): The free space cache file
(9804365955072) is invalid. skip it
[36292.358558] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
[36292.358563] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level
0)
in tree 7
Quoted 6 lines
[36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd
0,
flush 0, corrupt 218, gen 19
[36292.364717] BTRFS error (device sda): unable to fixup (regular)
error
at logical 5614914584576 on dev /dev/sda
Both copies of metadata are failing checksum, so it can't be fixed. It suggests there's a hardware problem (memory or storage), or maybe a new bug.
Quoted 10 lines
Have there been any crashes while writing to the file system? What is the storage stack configuration? 22TB for a single block device means it's built up from something else.
I'd dig around for any non-btrfs storage stack related errors in the meantime, maybe a dev will have some idea what's going on from the call traces, I'm not sure what they mean.
Signature · 1 line
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html