Input/Output errors

I'm running btrfs on DM-Crypt Luks running on LVM.

Occasionally I get files that are unreadable for some period of time. Attempting to read from them results in an

Input/output error

Sometimes they'll come back on their own, and sometimes a scrub seems to help, but sometimes I just have to delete them.

Nothing shows up in dmesg when these occur, and I can't predict which files it will be, or what causes it.

It's currently happening running 4.4.1-2-ARCH, but I've seen the same thing for many previous kernel versions.

Does anyone have any suggestions?

That's weird to say the least, you should at least get *something* in dmesg.
And you are getting other error messages and btrfs kernel messages in your logs?

When whatever app you have that's trying to read them fails, I assume they also fail with cat or less?

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

I am getting other, normal btrfs messages. I'll include them at the end. When it happens I get nothing at all in dmesg/logs.

And yes, cat will fail. I can move the file to another name though, which I often do to get it out of the way.

They're mounted with:

rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

issue_discards=1 is in lvm.conf, and discard in /etc/crypttab. (I'm now reading that I probably shouldn't have it in fstab though and just run fstrim.)

I don't know if this is related yet at all, but it /seems/ more likely to happen after I delete a bunch of data. That could be a red herring though.

When the file becomes readable again it's perfectly fine. Scrub never finds any errors.

$ dmesg | grep -i btrfs
[   11.837137] Btrfs loaded
[   11.837403] BTRFS: device label root devid 1 transid 508963 /dev/dm-3
[   11.856203] BTRFS info (device dm-3): disk space caching is enabled
[   11.879366] BTRFS: detected SSD devices, enabling SSD mode
[   12.160267] BTRFS info (device dm-3): turning on discard
[   12.160272] BTRFS info (device dm-3): enabling auto defrag
[   12.160275] BTRFS info (device dm-3): enabling inode map caching
[   12.160277] BTRFS info (device dm-3): disk space caching is enabled
[   14.979093] BTRFS: device label home devid 1 transid 705779 /dev/dm-5
[   15.013978] BTRFS info (device dm-5): use ssd allocation scheme
[   15.013983] BTRFS info (device dm-5): turning on discard
[   15.013987] BTRFS info (device dm-5): enabling auto defrag
[   15.013989] BTRFS info (device dm-5): enabling inode map caching
[   15.013991] BTRFS info (device dm-5): disk space caching is enabled
[   15.100779] BTRFS error (device dm-5): could not find root 8
[   15.102889] BTRFS error (device dm-5): could not find root 8
[   15.105833] BTRFS error (device dm-3): could not find root 8
[   15.105838] BTRFS error (device dm-3): could not find root 8

It sounds like an ssd trim bug. I'd check the firmware for updates. If it's up to date, I'd drop discard mount option first and try to reproduce. Or just use the default mount options and try to reproduce, then add them back one at a time until you discover the culprit.

Also, how many files/directories are there? inode_cache isn't recommended for most use cases. And space_cache is the default so it doesn't need to be listed.

Chris Murphy

2016-02-24 05:37 keltezéssel, Chris Murphy írta:

As i wrote to the list a few weeks ago, this problem seems to be the same I have.

The difference:

- i use mount options: noatime,compress,nossd
- I don't use dm-crypt, but these machines are Xen pvms (there is a virtualization layer between btrfs and lvm)

The same:

- io error without any error or message in the dmesg.
- umount / mount always fixes the problem (for some time)

More info:

- these files are usually smalls (mysql myisam files, 10-20-50 kbyte size, without heave fragmentation)
- defrag don't help
- scrub always works (no errors) but not fix the errors
- no hw or hw read error on the block device (in any layer)
- get this problem with 4.4.1 kernel too (seems to be somewhat less frequent than before, but the problem happened with 3.18 and on)
- echo 3 > /proc/sys/vm/drop_caches sometimes fixes the problem, but not every time
- the problem is happening rarely, sometimes there are days without error
- the problem is not for specific hardware or virtual machine

I can try any debug option or patch if needed.

László Szalma

Interesting. I just had the issue happen with one of my files, tried this, and it's now readable again.

Thanks Chris,

The disk is a SAMSUNG MZMTE512HMHP-000L1 running f/w: EXT42L0Q. I didn't find any updates in my searches.

I did drop the discard option, and the inode_cache and space_cache ones as well. I'm not sure I originally got them from.

Even if it's a disk firmware bug, doesn't it seem like some layer should report something about it? Is there any extra logging I should enable in case it happens again?

As I don't have a way to reliable reproduce the error, I'll have to wait and see if this helps.