Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels
Hi,
we are noticing some strange block device IO performance and our investigations are leading us away from the hardware and towards the kernel. This could be a simple tuning problem or a known issue, so before taking a deep dive down kernel sources and debian kernel patches we would like to rule out the simple things first.
The context is a 16 GB 32-bit intel debian workstation, using an ext4 filesystem with journalling, on a lvm SATA3 SSD disk, with relatively recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual machines. The host system (so not the virual machines) shows sporadic extremely slow write performance (around 4 megabytes per second). However, if we use the debian 3.2.0 kernel this problem does not manifest itself.
We've created a simple IO performance test script to investigate this. It is basically a smart wrapper around dd, copying data between the block device under test and a ramdisk filesystem, using flags to select the usage of cache, sync and direct io, clearing caches before the test and running the write and read tests three times to account for transient performance.
The results of these tests are unexpected: we see the expected normal write performance when using uncached, unsynced, direct IO and very slow write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!
This seems to rule out the block device and IO controller being at fault. In fact, other tests showed the same performance discrepancy with an NFS mounted filesystem and a platter disk.
We also noticed that when the performance is slow, if we shut down the KVM virtual machines the performance returns to normal.
Maybe there is something going wrong with cache/buffer handling? Thanks for your insights.
I've kept this mail intentionally free from too many technical details but I'll be happy to provide additional relevant info as required.
Regards,
Erik Cumps
Quoted 22 lines · Erik Cumps
we are noticing some strange block device IO performance and our investigations are leading us away from the hardware and towards the kernel. This could be a simple tuning problem or a known issue, so before taking a deep dive down kernel sources and debian kernel patches we would like to rule out the simple things first.
The context is a 16 GB 32-bit intel debian workstation, using an ext4 filesystem with journalling, on a lvm SATA3 SSD disk, with relatively recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual machines. The host system (so not the virual machines) shows sporadic extremely slow write performance (around 4 megabytes per second). However, if we use the debian 3.2.0 kernel this problem does not manifest itself.
We've created a simple IO performance test script to investigate this. It is basically a smart wrapper around dd, copying data between the block device under test and a ramdisk filesystem, using flags to select the usage of cache, sync and direct io, clearing caches before the test and running the write and read tests three times to account for transient performance.
Quoted 4 lines
The results of these tests are unexpected: we see the expected normal write performance when using uncached, unsynced, direct IO and very slow write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!
Actually, it is the *synchronous*, direct IO that matches the expected raw write performance of the device.
The "regular IO" test is doing roughly this:
echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file bs=1M count=100 dd if=ramdisk_file of=test_file bs=1M count=100 dd if=ramdisk_file of=test_file bs=1M count=100
The direct IO test is doing roughly this:
echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
Quoted 4 lines
This seems to rule out the block device and IO controller being at fault. In fact, other tests showed the same performance discrepancy with an NFS mounted filesystem and a platter disk.
Quoted 8 lines
We also noticed that when the performance is slow, if we shut down the KVM virtual machines the performance returns to normal.
Maybe there is something going wrong with cache/buffer handling? Thanks for your insights.
I've kept this mail intentionally free from too many technical details but I'll be happy to provide additional relevant info as required.
Regards,
Erik Cumps
On Thursday 2015-06-18 09:35, Erik Cumps wrote:
| On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <[email protected]> wrote: | > The context is a 16 GB 32-bit intel debian workstation, using an ext4 | > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively | > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual | > machines. The host system (so not the virual machines) shows sporadic | > extremely slow write performance (around 4 megabytes per second). | > However, if we use the debian 3.2.0 kernel this problem does not | > manifest itself. [...] | Actually, it is the *synchronous*, direct IO that matches the expected | raw write performance of the device. | | The "regular IO" test is doing roughly this: | | echo 3 > /proc/sys/vm/drop_caches | dd if=ramdisk_file of=test_file bs=1M count=100 | dd if=ramdisk_file of=test_file bs=1M count=100 | dd if=ramdisk_file of=test_file bs=1M count=100 | | The direct IO test is doing roughly this: | | echo 3 > /proc/sys/vm/drop_caches | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
I'm seeing this as well here on a number of new Dell Optiplex 7020 machines and one older Optiplex 780, all with 8GB RAM and running Ubuntu 14.04 in 32-bit mode.
A simple dd command shows the problem:
$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s
$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s
In my case, running:
echo 3 > /proc/sys/vm/drop_caches
will restore the normal speed for a limited time:
$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s
There is an old Ubuntu bug report describing the same issue:
https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294
Signature · 1 line
Dick
On Tuesday 2015-09-01 11:15, Dick Streefland wrote:
| I'm seeing this as well here on a number of new Dell Optiplex 7020 | machines and one older Optiplex 780, all with 8GB RAM and running | Ubuntu 14.04 in 32-bit mode.
It turned out that the dirty thresholds in /proc/vmstat became zero:
nr_dirty_threshold 0nr_dirty_background_threshold 0
A workaround is:
# sysctl vm.highmem_is_dirtyable=1
Signature · 1 line
Dick