Skip to contentSkip to conversation

Input/output error, nothing appended in dmesg

DISCUSSION Tomasz Chmielewski 3 participants 11 messages Aug 5, 2016 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:

# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

Strangely, nothing gets appended in dmesg:

# dmesg -c
#

The filesystem stores mostly remote syslog files (so, all text files, appended to).

Expected?

# btrfs fi show /var/log/remote/
Label: none  uuid: 5cec93a8-7894-41f6-94a4-9d9b58216dd4
        Total devices 1 FS bytes used 146.55GiB
        devid    1 size 200.00GiB used 153.01GiB path /dev/xvdb
# btrfs fi df /var/log/remote/
Data, single: total=149.00GiB, used=144.50GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=4.01GiB, used=2.05GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Tomasz Chmielewski
https://lxadm.com

#1 lore raw
Quoted 6 lines · Tomasz Chmielewski

I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:

# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file? We had one bug with inline files that might have caused this.

-chris

#2 lore raw
Quoted 10 lines · Chris Mason
Quoted 6 lines · Tomasz Chmielewski

I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:

# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file? We had one bug with inline files that might have caused this.

This one's tiny, 158137 bytes.

Tomasz Chmielewski
https://lxadm.com

#3 lore raw
Quoted 6 lines · Tomasz Chmielewski
Quoted 5 lines · Chris Mason
Quoted 4 lines · Tomasz Chmielewski

I'm getting occasional (every few weeks) input/output errors on a btrfs filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 kernel:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file? We had one bug with inline files that might have caused this.

This one's tiny, 158137 bytes.

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

-chris

#4 lore raw
Quoted 10 lines · Chris Mason
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file? We had one bug with inline files that might have caused this.

This one's tiny, 158137 bytes.

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

Hmmm....

# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]

Tomasz Chmielewski
https://lxadm.com

#5 lore raw
Quoted 15 lines · Tomasz Chmielewski
Quoted 10 lines · Chris Mason
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file? We had one bug with inline files that might have caused this.

This one's tiny, 158137 bytes.

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

Hmmm....

Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]

Looks like the FS is mounted?

-chris

#6 lore raw
Quoted 19 lines · Chris Mason
Quoted 5 lines
Quoted 2 lines

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

Hmmm....

Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]
Quoted 1 line

Looks like the FS is mounted?

It is mounted, yes. Does btrfs-debug-tree need an unmounted FS?

I'm not able to unmount it unfortunately (in sense, the system has to work).

Tomasz Chmielewski
https://lxadm.com

#7 lore raw
In reply to Tomasz Chmielewski
Quoted 5 lines · Tomasz Chmielewski
Quoted 2 lines

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

Hmmm....

Quoted 13 lines
# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
parent transid verify failed on 355229302784 wanted 49943295 found 49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
parent transid verify failed on 355233251328 wanted 49943299 found 49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]

And, miracle cure O_o

# file ./2016-08-02/serverX/syslog.log
ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
# echo 3 > /proc/sys/vm/drop_caches
# file 2016-08-02/serverX/syslog.log
2016-08-02/serverX/syslog.log: ASCII text, with very long lines
# cat 2016-08-02/serverX/syslog.log
(...)

Tomasz Chmielewski
https://lxadm.com

#8 lore raw
Quoted 30 lines · Tomasz Chmielewski
Quoted 14 lines · Tomasz Chmielewski
Quoted 2 lines

Too big for the known problem though. Still, can you btrfs-debug-tree and just make sure it doesn't have inline items?

Hmmm....

# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found
49943301
parent transid verify failed on 355229302784 wanted 49943295 found
49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found
49943303
parent transid verify failed on 355233251328 wanted 49943299 found
Quoted 8 lines
49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]

And, miracle cure O_o

# file ./2016-08-02/serverX/syslog.log
ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
Quoted 8 lines
# echo 3 > /proc/sys/vm/drop_caches
# file 2016-08-02/serverX/syslog.log
2016-08-02/serverX/syslog.log: ASCII text, with very long lines
# cat 2016-08-02/serverX/syslog.log
(...)

If you don't already have this commit, please give it a try. Should fix things up.

commit 8dff9c85341032767d7b519217a79ea04cd676b0

Author: Chris Mason <[email protected]>
Date:   Sat Sep 19 11:28:25 2015 -0700

Btrfs: deal with duplciates during extent_map insertion in btrfs_get_extent

#9 lore raw
In reply to Tomasz Chmielewski
Quoted 5 lines · Tomasz Chmielewski

And, miracle cure O_o

# file ./2016-08-02/serverX/syslog.log
ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
Quoted 7 lines
# echo 3 > /proc/sys/vm/drop_caches
# file 2016-08-02/serverX/syslog.log
2016-08-02/serverX/syslog.log: ASCII text, with very long lines
# cat 2016-08-02/serverX/syslog.log
(...)

A few mins after the previous "echo 3 > /proc/sys/vm/drop_caches" (this file is around 1.5 MB and wasn't touched since 2016-06-21):

# file ./2016-06-21/serverY/nginx-dashboard-error.log
./2016-06-21/serverY/nginx-dashboard-error.log: ERROR: cannot read `./2016-06-21/serverY/nginx-dashboard-error.log' (Input/output error)
# echo 3 > /proc/sys/vm/drop_caches
# file ./2016-06-21/serverY/nginx-dashboard-error.log
./2016-06-21/serverY/nginx-dashboard-error.log: ASCII text, with very long lines
# cat ./2016-06-21/serverY/nginx-dashboard-error.log
(...works OK, no corruption...)

Tomasz Chmielewski
https://lxadm.com

#10 lore raw
In reply to Tomasz Chmielewski
Quoted 10 lines · El sábado, 6 de agosto de 2016 0:45:13 (CEST) Tomasz Chmielewski escribió:

And, miracle cure O_o

# file ./2016-08-02/serverX/syslog.log
ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output
error)
# echo 3 > /proc/sys/vm/drop_caches
# file 2016-08-02/serverX/syslog.log
2016-08-02/serverX/syslog.log: ASCII text, with very long lines

FWIW, bugs similar to this one were reported in the past:

http://www.spinics.net/lists/linux-btrfs/msg54962.html
http://www.spinics.net/lists/linux-btrfs/msg52371.html