bad O_DIRECT read and write performance with small block sizes with virtio

Hi,

I've come across a problem with read and write disk IO performance when using O_DIRECT from within a kvm guest. With O_DIRECT, reads and writes are much slower with smaller block sizes. Depending on the block size used, I've seen 10 times slower.

For example, with an 8k block size, reading directly from /dev/vdb without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the backend device on the host gives 2.3 GB/s. Reading in O_DIRECT mode from a xen guest on the same hardware manages 263 MB/s.

Writing is affected in the same way, and exhibits the same behaviour with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being read, but about 14 times the number of context switches in O_DIRECT mode (4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
<driver cache='none'/>
<source dev='/dev/mapper/zero'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>

which translates to the kvm arg:

-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

I'm testing with dd:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct

As a side note, as you increase the block size read performance in O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about 150k block size). By 550k block size I'm seeing 1 GB/s reads with O_DIRECT and 770 MB/s without.

Of course I see this performance situation with real disks too, I just wanted to rule out the variables of moving metal around.

I get the same situation on Centos 5.5 and the latest RHEL 6 beta (which is kvm 0.12 and kernel 2.6.32). Hardware is a Dell i510 with 64GB RAM and 12 Intel(R) Xeon(R) L5640 2.27GHz CPUs (running only one kvm guest with 1G ram). Host disk scheduler is deadline, guest disk scheduler is noop.

Guest distro is Ubuntu Lucid, 2.6.32-22-server. I've tried with both 32bit pae and 64bit guest kernels.

Anyone got any thoughts on this?

Thanks,

John.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 8 lines · John Leach

Hi,

I've come across a problem with read and write disk IO performance when using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes are much slower with smaller block sizes.  Depending on the block size used, I've seen 10 times slower.

Quoted 7 lines

For example, with an 8k block size, reading directly from /dev/vdb without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode from a xen guest on the same hardware manages 263 MB/s.

Stefan has a few fixes for this behavior that help a lot.  One of them (avoiding memset) is already upstream but not in 0.12.x.

The other two are not done yet but should be on the ML in the next couple weeks.  They involve using ioeventfd for notification and unlocking the block queue lock while doing a kick notification.

Quoted 29 lines

Writing is affected in the same way, and exhibits the same behaviour with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being read, but about 14 times the number of context switches in O_DIRECT mode (4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
  <driver cache='none'/>
  <source dev='/dev/mapper/zero'/>
  <target dev='vdb' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>

which translates to the kvm arg:

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

Quoted 8 lines

I'm testing with dd:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct

As a side note, as you increase the block size read performance in O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about 150k block size). By 550k block size I'm seeing 1 GB/s reads with O_DIRECT and 770 MB/s without.

Quoted 17 lines · Anthony Liguori

Quoted 8 lines · John Leach

Hi,

I've come across a problem with read and write disk IO performance when using O_DIRECT from within a kvm guest. With O_DIRECT, reads and writes are much slower with smaller block sizes. Depending on the block size used, I've seen 10 times slower.

Quoted 7 lines

For example, with an 8k block size, reading directly from /dev/vdb without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the backend device on the host gives 2.3 GB/s. Reading in O_DIRECT mode from a xen guest on the same hardware manages 263 MB/s.

Quoted 6 lines

Stefan has a few fixes for this behavior that help a lot. One of them (avoiding memset) is already upstream but not in 0.12.x.

The other two are not done yet but should be on the ML in the next couple weeks. They involve using ioeventfd for notification and unlocking the block queue lock while doing a kick notification.

Thanks for mentioning those patches. The ioeventfd patch will be sent this week, I'm checking that migration works correctly and then need to check that vhost-net still works.

Quoted 23 lines

Quoted 23 lines

Writing is affected in the same way, and exhibits the same behaviour with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being read, but about 14 times the number of context switches in O_DIRECT mode (4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
<driver cache='none'/>
<source dev='/dev/mapper/zero'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</disk>

Quoted 5 lines

Quoted 5 lines

which translates to the kvm arg:

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

Quoted 9 lines

Quoted 9 lines

I'm testing with dd:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct

As a side note, as you increase the block size read performance in O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about 150k block size). By 550k block size I'm seeing 1 GB/s reads with O_DIRECT and 770 MB/s without.

Can you take QEMU out of the picture and run the same test on the host:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
vs
dd if=/dev/vdb of=/dev/null bs=8k

This isn't quite the same because QEMU will use a helper thread doing preadv. I'm not sure what syscall dd will use.

It should be close enough to determine whether QEMU and device emulation are involved at all though, or whether these differences are due to the host kernel code path down to the device mapper zero device being different for normal vs O_DIRECT.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 52 lines · Stefan Hajnoczi

Quoted 17 lines · Anthony Liguori

Quoted 8 lines · John Leach

Hi,

I've come across a problem with read and write disk IO performance when using O_DIRECT from within a kvm guest. With O_DIRECT, reads and writes are much slower with smaller block sizes. Depending on the block size used, I've seen 10 times slower.

Quoted 7 lines

For example, with an 8k block size, reading directly from /dev/vdb without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the backend device on the host gives 2.3 GB/s. Reading in O_DIRECT mode from a xen guest on the same hardware manages 263 MB/s.

Quoted 6 lines

Stefan has a few fixes for this behavior that help a lot. One of them (avoiding memset) is already upstream but not in 0.12.x.

The other two are not done yet but should be on the ML in the next couple weeks. They involve using ioeventfd for notification and unlocking the block queue lock while doing a kick notification.

Thanks for mentioning those patches. The ioeventfd patch will be sent this week, I'm checking that migration works correctly and then need to check that vhost-net still works.

Quoted 23 lines

Quoted 23 lines

Writing is affected in the same way, and exhibits the same behaviour with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being read, but about 14 times the number of context switches in O_DIRECT mode (4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
<driver cache='none'/>
<source dev='/dev/mapper/zero'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</disk>

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

which translates to the kvm arg:

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

aio=native and change the io scheduler on the host to deadline should help as well.

Quoted 12 lines · Anthony Liguori

Quoted 11 lines · John Leach

Hi,

I've come across a problem with read and write disk IO performance when using O_DIRECT from within a kvm guest. With O_DIRECT, reads and writes are much slower with smaller block sizes. Depending on the block size used, I've seen 10 times slower.

For example, with an 8k block size, reading directly from /dev/vdb without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

Quoted 7 lines

Quoted 4 lines

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the backend device on the host gives 2.3 GB/s. Reading in O_DIRECT mode from a xen guest on the same hardware manages 263 MB/s.

Stefan has a few fixes for this behavior that help a lot. One of them (avoiding memset) is already upstream but not in 0.12.x.

Quoted 4 lines

The other two are not done yet but should be on the ML in the next couple weeks. They involve using ioeventfd for notification and unlocking the block queue lock while doing a kick notification.

Thanks for mentioning those patches. The ioeventfd patch will be sent this week, I'm checking that migration works correctly and then need to check that vhost-net still works.

Quoted 29 lines

Quoted 29 lines

Writing is affected in the same way, and exhibits the same behaviour with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being read, but about 14 times the number of context switches in O_DIRECT mode (4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
<driver cache='none'/>
<source dev='/dev/mapper/zero'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</disk>

which translates to the kvm arg:

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

Quoted 8 lines

Quoted 8 lines

I'm testing with dd:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct

As a side note, as you increase the block size read performance in O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about 150k block size). By 550k block size I'm seeing 1 GB/s reads with O_DIRECT and 770 MB/s without.

Can you take QEMU out of the picture and run the same test on the host:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
vs
dd if=/dev/vdb of=/dev/null bs=8k

This isn't quite the same because QEMU will use a helper thread doing preadv. I'm not sure what syscall dd will use.

It should be close enough to determine whether QEMU and device emulation are involved at all though, or whether these differences are due to the host kernel code path down to the device mapper zero device being different for normal vs O_DIRECT.

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s

dd is just using read.

Quoted 4 lines · John Leach

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s

Quoted 5 lines

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s

dd is just using read.

What's /dev/mapper/zero? A real volume or a zero target?

Quoted 11 lines · Avi Kivity

Quoted 8 lines · John Leach

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s

dd is just using read.

What's /dev/mapper/zero? A real volume or a zero target?

zero target:

echo "0 21474836480 zero" | dmsetup create zero

The same performance penalty occurs when using real disks though, I just moved to a zero target to rule out the variables of spinning metal and raid controller caches.

Quoted 19 lines · John Leach

Quoted 6 lines · Avi Kivity

Quoted 4 lines · John Leach

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s

Quoted 7 lines

Quoted 5 lines

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s

dd is just using read.

What's /dev/mapper/zero?  A real volume or a zero target?

zero target:

echo "0 21474836480 zero" | dmsetup create zero

Quoted 3 lines

The same performance penalty occurs when using real disks though, I just moved to a zero target to rule out the variables of spinning metal and raid controller caches.

Don't, it's confusing things.  I'd expect dd to be slower with iflag=direct since the kernel can't do readahead an instead must roundtrip to the controller.  With a zero target it's faster since it doesn't have to roundtrip and instead avoids a copy.