spuious I/O errors from btrfs...at the caching layer?

I am seeing a lot of spurious I/O errors that look like they come from the cache-facing side of btrfs. While running a heavy load with some extent-sharing (e.g. building 20 Linux kernels at once from source trees copied with 'cp -a --reflink=always'), some files will return spurious EIO on read. It happens often enough to prevent a Linux kernel build about 1/3 of the time.

I believe the I/O errors to be spurious because:

- there is no kernel message of any kind during the event

- scrub detects 0 errors

- device stats report 0 errors

- the drive firmware reports nothing wrong through SMART

- there seems to be no attempt to read the disk when the error is reported

- "sysctl vm.drop_caches={1,2}" makes the I/O error go away.

Files become unreadable at random, and stay unreadable indefinitely; however, any time I discover a file that gives EIO on read, I can poke vm.drop_caches and make the EIO go away. The file can then be read normally and has correct contents. The disk does not seem to be involved in the I/O error return.

This seems to happen more often when snapshots are being deleted; however, it occurs on systems with no snapshots as well (though in these cases the system had snapshots in the past).

When a file returns EIO on read, other snapshots of the same file also return EIO on read. I have not been able to test whether this affects reflink copies (clones) as well.

Observed from 3.17..3.18.3. All filesystems affected use skinny-metadata. No filesystems that are not using skinny-metadata seem to have this problem.

[...]

I ran a test overnight using 3.18.3 on a freshly formatted filesystem with no skinny-metadata.

The test consisted of creating reflink copies of a Linux kernel source tree and running kernel builds in each copy simultaneously, like this:

	# assume you have a ready-to-build kernel tree in 'linux'
	for x in $(seq 1 5); do
		cp -a --reflink linux linux-$x
	done

	# build all the kernels at once
	for x in $(seq 1 5); do
		(cd linux-$x && make -j10 2>&1 | tee make.log) &
	done

wait
# then tail all the make.logs and see how many failed due to # I/O errors

Spurious I/O errors occured with as few as two concurrent kernel builds.

The test machine has 16GB of RAM and the filesystem is also 16GB, RAID1 on two spinning disks.

It seems that the rate of spurious I/O errors varies most according to the vm.vfs_cache_pressure sysctl. At '10' the I/O errors occur so often that building a kernel is impossible. At '100' I can't reproduce even a single I/O error.

I guess this is own my fault for using non-default sysctl parameters, although I wouldn't expect any value of this sysctl to cause these symptoms... :-P

Setting that to anything less than about 75 is just asking for trouble.
What's I think is happening is that you are eating up memory with the
vfs cache (because that caches inodes and dentries, which in turn means that reflinks don't help decrease it's usage at all) faster than it can be reclaimed, and therefore have no space for the file data to be read into memory. This will probably happen with most other filesystems as well if you set vfs_cache_pressure really low.

Personally, I wouldn't recommend ever touching that particular sysctl, with the exception of increasing it some when you have really fast storage (NVMe or UFS-SCSI based SSD's for example) and a relatively small set of files that you actually access frequently.