Skip to contentSkip to conversation

Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

DISCUSSION Erik Cumps 2 participants 4 messages Jun 16, 2015 – Sep 2, 2015

lore
Share & tools
lore

Erik Cumps

author
#0 lore raw

Hi,

we are noticing some strange block device IO performance and our investigations are leading us away from the hardware and towards the kernel. This could be a simple tuning problem or a known issue, so before taking a deep dive down kernel sources and debian kernel patches we would like to rule out the simple things first.

The context is a 16 GB 32-bit intel debian workstation, using an ext4 filesystem with journalling, on a lvm SATA3 SSD disk, with relatively recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual machines. The host system (so not the virual machines) shows sporadic extremely slow write performance (around 4 megabytes per second). However, if we use the debian 3.2.0 kernel this problem does not manifest itself.

We've created a simple IO performance test script to investigate this. It is basically a smart wrapper around dd, copying data between the block device under test and a ramdisk filesystem, using flags to select the usage of cache, sync and direct io, clearing caches before the test and running the write and read tests three times to account for transient performance.

The results of these tests are unexpected: we see the expected normal write performance when using uncached, unsynced, direct IO and very slow write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!

This seems to rule out the block device and IO controller being at fault. In fact, other tests showed the same performance discrepancy with an NFS mounted filesystem and a platter disk.

We also noticed that when the performance is slow, if we shut down the KVM virtual machines the performance returns to normal.

Maybe there is something going wrong with cache/buffer handling? Thanks for your insights.

I've kept this mail intentionally free from too many technical details but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps

author
#1 lore raw
Quoted 22 lines · Erik Cumps

we are noticing some strange block device IO performance and our investigations are leading us away from the hardware and towards the kernel. This could be a simple tuning problem or a known issue, so before taking a deep dive down kernel sources and debian kernel patches we would like to rule out the simple things first.

The context is a 16 GB 32-bit intel debian workstation, using an ext4 filesystem with journalling, on a lvm SATA3 SSD disk, with relatively recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual machines. The host system (so not the virual machines) shows sporadic extremely slow write performance (around 4 megabytes per second). However, if we use the debian 3.2.0 kernel this problem does not manifest itself.

We've created a simple IO performance test script to investigate this. It is basically a smart wrapper around dd, copying data between the block device under test and a ramdisk filesystem, using flags to select the usage of cache, sync and direct io, clearing caches before the test and running the write and read tests three times to account for transient performance.

Quoted 4 lines

The results of these tests are unexpected: we see the expected normal write performance when using uncached, unsynced, direct IO and very slow write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!

Actually, it is the *synchronous*, direct IO that matches the expected raw write performance of the device.

The "regular IO" test is doing roughly this:

echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file bs=1M count=100 dd if=ramdisk_file of=test_file bs=1M count=100 dd if=ramdisk_file of=test_file bs=1M count=100

The direct IO test is doing roughly this:

echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

Quoted 4 lines

This seems to rule out the block device and IO controller being at fault. In fact, other tests showed the same performance discrepancy with an NFS mounted filesystem and a platter disk.

Quoted 8 lines

We also noticed that when the performance is slow, if we shut down the KVM virtual machines the performance returns to normal.

Maybe there is something going wrong with cache/buffer handling? Thanks for your insights.

I've kept this mail intentionally free from too many technical details but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps

#2 lore raw

On Thursday 2015-06-18 09:35, Erik Cumps wrote:

| On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <[email protected]> wrote:
| > The context is a 16 GB 32-bit intel debian workstation, using an ext4
| > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
| > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
| > machines. The host system (so not the virual machines) shows sporadic
| > extremely slow write performance (around 4 megabytes per second).
| > However, if we use the debian 3.2.0 kernel this problem does not
| > manifest itself.
[...]
| Actually, it is the *synchronous*, direct IO that matches the expected
| raw write performance of the device.
| 
| The "regular IO" test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
| 
| The direct IO test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

I'm seeing this as well here on a number of new Dell Optiplex 7020 machines and one older Optiplex 780, all with 8GB RAM and running Ubuntu 14.04 in 32-bit mode.

A simple dd command shows the problem:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s
$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s

In my case, running:

echo 3 > /proc/sys/vm/drop_caches

will restore the normal speed for a limited time:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s

There is an old Ubuntu bug report describing the same issue:

https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294

Signature · 1 line
Dick
#3 lore raw

On Tuesday 2015-09-01 11:15, Dick Streefland wrote:

| I'm seeing this as well here on a number of new Dell Optiplex 7020
| machines and one older Optiplex 780, all with 8GB RAM and running
| Ubuntu 14.04 in 32-bit mode.

It turned out that the dirty thresholds in /proc/vmstat became zero:

nr_dirty_threshold 0
nr_dirty_background_threshold 0

A workaround is:

# sysctl vm.highmem_is_dirtyable=1
Signature · 1 line
Dick