Input/output error, nothing appended in dmesg
I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output error
Strangely, nothing gets appended in dmesg:
# dmesg -c #
The filesystem stores mostly remote syslog files (so, all text files, appended to).
Expected?
# btrfs fi show /var/log/remote/
Label: none uuid: 5cec93a8-7894-41f6-94a4-9d9b58216dd4
Total devices 1 FS bytes used 146.55GiB
devid 1 size 200.00GiB used 153.01GiB path /dev/xvdb# btrfs fi df /var/log/remote/ Data, single: total=149.00GiB, used=144.50GiB System, single: total=4.00MiB, used=48.00KiB Metadata, single: total=4.01GiB, used=2.05GiB GlobalReserve, single: total=512.00MiB, used=0.00B
Tomasz Chmielewski
https://lxadm.com
Quoted 6 lines · Tomasz Chmielewski
I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output error
How big is the file? We had one bug with inline files that might have caused this.
-chris
Quoted 10 lines · Chris Mason
Quoted 6 lines · Tomasz Chmielewski
I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output errorHow big is the file? We had one bug with inline files that might have caused this.
This one's tiny, 158137 bytes.
Tomasz Chmielewski
https://lxadm.com
Quoted 6 lines · Tomasz Chmielewski
Quoted 5 lines · Chris Mason
Quoted 4 lines · Tomasz Chmielewski
I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output errorHow big is the file? We had one bug with inline files that might have caused this.
This one's tiny, 158137 bytes.
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
-chris
Quoted 10 lines · Chris Mason
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output errorHow big is the file? We had one bug with inline files that might have caused this.
This one's tiny, 158137 bytes.
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
Hmmm....
# btrfs-debug-tree /dev/xvdb > /root/debug.tree parent transid verify failed on 355229302784 wanted 49943295 found 49943301 parent transid verify failed on 355229302784 wanted 49943295 found 49943301 Ignoring transid failure parent transid verify failed on 355233251328 wanted 49943299 found 49943303 parent transid verify failed on 355233251328 wanted 49943299 found 49943303 Ignoring transid failure print-tree.c:1105: btrfs_print_tree: Assertion failed. btrfs-debug-tree[0x418d99] btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6] btrfs-debug-tree(main+0x9a5)[0x432589] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45] btrfs-debug-tree[0x4070e9]
Tomasz Chmielewski
https://lxadm.com
Quoted 15 lines · Tomasz Chmielewski
Quoted 10 lines · Chris Mason
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log cat: 2016-08-02/serverX/syslog.log: Input/output errorHow big is the file? We had one bug with inline files that might have caused this.
This one's tiny, 158137 bytes.
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
Hmmm....
Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree parent transid verify failed on 355229302784 wanted 49943295 found 49943301 parent transid verify failed on 355229302784 wanted 49943295 found 49943301 Ignoring transid failure parent transid verify failed on 355233251328 wanted 49943299 found 49943303 parent transid verify failed on 355233251328 wanted 49943299 found 49943303 Ignoring transid failure print-tree.c:1105: btrfs_print_tree: Assertion failed. btrfs-debug-tree[0x418d99] btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6] btrfs-debug-tree(main+0x9a5)[0x432589] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45] btrfs-debug-tree[0x4070e9]
Looks like the FS is mounted?
-chris
Quoted 5 lines · Tomasz Chmielewski
Quoted 2 lines
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
Hmmm....
Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree parent transid verify failed on 355229302784 wanted 49943295 found 49943301 parent transid verify failed on 355229302784 wanted 49943295 found 49943301 Ignoring transid failure parent transid verify failed on 355233251328 wanted 49943299 found 49943303 parent transid verify failed on 355233251328 wanted 49943299 found 49943303 Ignoring transid failure print-tree.c:1105: btrfs_print_tree: Assertion failed. btrfs-debug-tree[0x418d99] btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6] btrfs-debug-tree(main+0x9a5)[0x432589] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45] btrfs-debug-tree[0x4070e9]
And, miracle cure O_o
# file ./2016-08-02/serverX/syslog.log ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
# echo 3 > /proc/sys/vm/drop_caches
# file 2016-08-02/serverX/syslog.log 2016-08-02/serverX/syslog.log: ASCII text, with very long lines
# cat 2016-08-02/serverX/syslog.log (...)
Tomasz Chmielewski
https://lxadm.com
Quoted 19 lines · Chris Mason
Quoted 5 lines
Quoted 2 lines
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
Hmmm....
Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree parent transid verify failed on 355229302784 wanted 49943295 found 49943301 parent transid verify failed on 355229302784 wanted 49943295 found 49943301 Ignoring transid failure parent transid verify failed on 355233251328 wanted 49943299 found 49943303 parent transid verify failed on 355233251328 wanted 49943299 found 49943303 Ignoring transid failure print-tree.c:1105: btrfs_print_tree: Assertion failed. btrfs-debug-tree[0x418d99] btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6] btrfs-debug-tree(main+0x9a5)[0x432589] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45] btrfs-debug-tree[0x4070e9]
Quoted 1 line
Looks like the FS is mounted?
It is mounted, yes. Does btrfs-debug-tree need an unmounted FS?
I'm not able to unmount it unfortunately (in sense, the system has to work).
Tomasz Chmielewski
https://lxadm.com
Quoted 30 lines · Tomasz Chmielewski
Quoted 14 lines · Tomasz Chmielewski
Quoted 2 lines
Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?
Hmmm....
# btrfs-debug-tree /dev/xvdb > /root/debug.tree parent transid verify failed on 355229302784 wanted 49943295 found 49943301 parent transid verify failed on 355229302784 wanted 49943295 found 49943301 Ignoring transid failure parent transid verify failed on 355233251328 wanted 49943299 found 49943303 parent transid verify failed on 355233251328 wanted 49943299 foundQuoted 8 lines
49943303 Ignoring transid failure print-tree.c:1105: btrfs_print_tree: Assertion failed. btrfs-debug-tree[0x418d99] btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6] btrfs-debug-tree(main+0x9a5)[0x432589] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45] btrfs-debug-tree[0x4070e9]And, miracle cure
O_o# file ./2016-08-02/serverX/syslog.log ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
Quoted 8 lines
# echo 3 > /proc/sys/vm/drop_caches# file 2016-08-02/serverX/syslog.log 2016-08-02/serverX/syslog.log: ASCII text, with very long lines# cat 2016-08-02/serverX/syslog.log (...)
If you don't already have this commit, please give it a try. Should fix things up.
commit 8dff9c85341032767d7b519217a79ea04cd676b0
Author: Chris Mason <[email protected]> Date: Sat Sep 19 11:28:25 2015 -0700
Btrfs: deal with duplciates during extent_map insertion in btrfs_get_extent
Quoted 5 lines · Tomasz Chmielewski
And, miracle cure
O_o# file ./2016-08-02/serverX/syslog.log ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
Quoted 7 lines
# echo 3 > /proc/sys/vm/drop_caches# file 2016-08-02/serverX/syslog.log 2016-08-02/serverX/syslog.log: ASCII text, with very long lines# cat 2016-08-02/serverX/syslog.log (...)
A few mins after the previous "echo 3 > /proc/sys/vm/drop_caches" (this file is around 1.5 MB and wasn't touched since 2016-06-21):
# file ./2016-06-21/serverY/nginx-dashboard-error.log ./2016-06-21/serverY/nginx-dashboard-error.log: ERROR: cannot read `./2016-06-21/serverY/nginx-dashboard-error.log' (Input/output error)
# echo 3 > /proc/sys/vm/drop_caches
# file ./2016-06-21/serverY/nginx-dashboard-error.log ./2016-06-21/serverY/nginx-dashboard-error.log: ASCII text, with very long lines
# cat ./2016-06-21/serverY/nginx-dashboard-error.log (...works OK, no corruption...)
Tomasz Chmielewski
https://lxadm.com
Quoted 10 lines · El sábado, 6 de agosto de 2016 0:45:13 (CEST) Tomasz Chmielewski escribió:
And, miracle cure
O_o# file ./2016-08-02/serverX/syslog.log ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)# echo 3 > /proc/sys/vm/drop_caches# file 2016-08-02/serverX/syslog.log 2016-08-02/serverX/syslog.log: ASCII text, with very long lines
FWIW, bugs similar to this one were reported in the past:
http://www.spinics.net/lists/linux-btrfs/msg54962.html
http://www.spinics.net/lists/linux-btrfs/msg52371.html