kvm, drbd, elevator, rotational - quite an interesting co-operation
Hello.
I'm new on drbd-user@, but long-time user of kvm.
[A side note: drbd-user@ appears to be subscribers-only
list. Sad consequence of spammers activity... I doubt many knowlegeable people on kvm@ are subscribed to drbd-user@ to be able to post there. Oh well.]
Today I tried drbd for the first time, and decided to start experimenting on two virtual machines.
And immediately ran into interesting issues, described below. The way to the results wasn't easy, trust me ;)
Software used:
DRBD version 8.3.1 (8.3.2rc1 spewed numerous assertion
failures and crashed both virtual and non-virtual machines).
kernel 2.6.29.5 compiled for x86 (i686smp)
lvm kernel module from 2.6.29
kvm userspace qemu-kvm-0.10.5.
Both virtual machines were identical, using virtio_net (bridged) and virtio_blk (on raw files on ext4 fs).
When I first set drbd up, waited till it did initial resync, formatted (ext3) and mounted the filesystem on /dev/drbd1. Next I did a trivial test - copying /usr (tiny 140Mb) to the drbd filesystem.
This took........ 29 *minutes* (!!!). Which is about 80 kilobytes per sec. Reminds me ol'good dialup modem... ;)
At this point I tried many different things, including straceing kvm process on host. The thing is that both virtio_net&virtio_blk works quite well, copying that same /usr between two virtual machines over nfs takes less than a minute. And finally, after finding the key knobs, here's the results:
/sys/block/vda/queue/rotational and ../elevator on drbd *secondary* node are the knobs in question.
First of all, for some reason virtil_blk sets its 'rotational' attribute to 0. Which was a surprize to me - both to discover this flag for virtio_blk (i understand it's present on all block devices) and to see its default value being set to 0. (A side note: the flag is probably misnamed, it was meant to show how costly disk seeks are: rotational disks vs flash media where seeks are not present at all).
So, with both primary and secondary virtual disks on real hdd and with default rotational=0 and (my default) elevator=noop, the copy takes 29 minutes.
By changing the two mentioned knobs on PRIMARY I was able to reduce that to 27 minutes.
Now, with rotational=1 and elevator=noop on SECONDARY the whole thing completed in 2m40s. Which is about 10 times a difference!.
Next, with rotational=1 and elevator=cfq on SECONDARY, it completes in about 1m10s sec.
With rotational=1 and elevator=cfq on BOTH nodes it completes in 50 sec.
(for the record, elevator=cfq and rotational=0 on SECONDARY makes it complete in about 9 minutes)
I went further and placed one of the virtual disks into memory on host. And even there, while speed increased dramatically, rotational on SECONDARY (even if the disk on secondary is in host's memory) had about 10x effect. On the other hand, placing virtual disk on PRIMARY to memory changes very little.
On the host side, by straceing kvm process it may be noticed that with default rotational=0, there's no write block mergeing going on at all. All writes on host are done in 4096-byte-sized chunks (block size of ext3fs). But after setting rotational to 1 I see writes being combined -- many 32Kb writes, 64Kb writes etc.
On primary node there's almost no change in write pattern on the host, writes are being combined regardless of `rotational' flag.
Another observation on kvm side - when I/O activity starts, kvm spawns many threads to perform the I/O. This is, again, especially visible on the secondary node. It writes adjanced 4096-byte blocks in different threads, about 10..20 threads at any given time. I *guess* it's ncq/tcq-like thing implemented by kvm, but it looks utterly ineffecient - having in mind sequential block numbers.
So finally some questions.
drbd: what's the difference in write pattern on secondary and
primary nodes? Why `rotational' flag makes very big difference on secondary and no difference whatsoever on primary?
kvm: what is this 'rotational' flag, what's its meaning here?
I guess it should be controllable from the command line if all else fails. In any way it's definitely something to keep in mind. I guess more easily triggerable workload will show something similar (without drbd).
kvm: i/o threads - should there be a way to control the amount of
threads? With default workload generated by drbd on secondary node having less thread makes more sense.
kvm: it has been said that using noop elevator on guest makes sense
since host does its own elevator/reordering. But this example shows "nicely" that this isn't always the case. I wonder how "general" this example is. Will try to measure further.
Another "interesting" issue shown by drbd here: on the slow case, secondary is so much busy with the writes that it misses drbd pings and pongs so that nodes frequently considers the other peer as dead and reconnects/resyncs. This is not on a pre-stone-age machine and with not slowest disks (it's 2.5GHz AMD Athlon X2-64 CPU and ~90Mb/sec sata disk).
I didn't try other combinations yet, including non-virtio model. Because all the above took almost whole day today, especially finding the knobs :)
Comments?
Thanks!
/mjt
Quoted 3 lines · Michael Tokarev
kvm: i/o threads - should there be a way to control the amount of threads? With default workload generated by drbd on secondary node having less thread makes more sense.
+1 on this. it seems reasonable to have one thread per device, or am
i wrong?
it also bothers me because when i have a couple of moderately disk-heavy VMs, the load average numbers skyrockets. that's because each blocked thread counts as 1 on this figure, even if they're all waiting on the same device.
Quoted 4 lines
kvm: it has been said that using noop elevator on guest makes sense since host does its own elevator/reordering. But this example shows "nicely" that this isn't always the case. I wonder how "general" this example is. Will try to measure further.
on my own (quick) tests, changing the elevator on the guest has very little effect on performance; but does affect the host CPU utilization. using drbd on the guest while testing with bonnie++ increased host CPU by around 20% for each VM
Signature · 1 line
Javier
Quoted 7 lines · Javier Guerra
Quoted 3 lines · Michael Tokarev
kvm: i/o threads - should there be a way to control the amount of
threads? With default workload generated by drbd on secondary node having less thread makes more sense.+1 on this. it seems reasonable to have one thread per device, or am
i wrong?
Definitely not one thread per device. This is because even simple hard drives nowadays have quite advanced NCQ/TCQ implementations, when it is better to keep the drive queue deeper than 1, to let the drive to re-order requests as it see fit, to optimize head movements etc.
For larger devices (which are arrays of disks with maybe large battery-backed write caches) more threads makes even more sense.
Also, with this queue/reordering in mind, think about how it should look like from host vs guest perspective: ideally kvm should be able to provide a queue of depth >1 to the guest, and that a guest is "multi-threaded" (multi-processes really) by its own.
To be fair, I can't construct an example when deeper queue may be bad (not counting bad NCQ implementations).
Quoted 4 lines
it also bothers me because when i have a couple of moderately disk-heavy VMs, the load average numbers skyrockets. that's because each blocked thread counts as 1 on this figure, even if they're all waiting on the same device.
And how having large LA is bad? I mean, LA by itself is not an indicator of bad or good performance, don't you think?
Quoted 9 lines
Quoted 4 lines
kvm: it has been said that using noop elevator on guest makes sense
since host does its own elevator/reordering. But this example shows "nicely" that this isn't always the case. I wonder how "general" this example is. Will try to measure further.on my own (quick) tests, changing the elevator on the guest has very little effect on performance; but does affect the host CPU utilization. using drbd on the guest while testing with bonnie++ increased host CPU by around 20% for each VM
Increased compared what with what? Also, which virtual disk format did you use?
By doing just an strace of kvm process here it's trivial to see the
difference: switching from non-rotational to rotational makes kvm to start writing in large chunks instead of 1024-sized blocks. Which means, at least, much less context switches, which should improve performance. Yes it increases CPU usage somewhat (maybe around 20%), but it also increases I/O speed here quite significantly. Your test shows no increase in speed. Which suggests we're doing something differently.
/mjt
Quoted 8 lines · Michael Tokarev
Quoted 4 lines · Javier Guerra
it also bothers me because when i have a couple of moderately disk-heavy VMs, the load average numbers skyrockets. that's because each blocked thread counts as 1 on this figure, even if they're all waiting on the same device.
And how having large LA is bad? I mean, LA by itself is not an indicator of bad or good performance, don't you think?
it's not a real problem, of course; but it's a nuisance because some reporting tools (zabbix/nessus) use this figure to raise alarms, meaning i have to adjust it.
also, even a single-threaded high-IO process on a guest fires a lot of IO threads on the host, and other not-so-aggressive VMs suffer.
definitely using deadline scheduler on the host reduces the impact. (down to zero? i don't think so, but it's certainly manageable)
Quoted 7 lines
Quoted 4 lines
on my own (quick) tests, changing the elevator on the guest has very little effect on performance; but does affect the host CPU utilization. using drbd on the guest while testing with bonnie++ increased host CPU by around 20% for each VM
Increased compared what with what? Also, which virtual disk format did you use?
sorry, i had a typo there, i meant: using cfq vs. noop on the guest (running bonnie++, no drbd anywhere) produced around 20% more CPU load on the host, with no measurable performance advantage.
Signature · 1 line
Javier
Quoted 2 lines · Michael Tokarev
To be fair, I can't construct an example when deeper queue may be bad (not counting bad NCQ implementations).
Increased latency for new requests. Larger queue depths improve throughput at the expense of latency.
Signature · 1 line
Carson
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Michael Tokarev
Hello.
I'm new on drbd-user@, but long-time user of kvm.
[A side note: drbd-user@ appears to be subscribers-only
list. Sad consequence of spammers activity... I doubt many knowlegeable people on kvm@ are subscribed to drbd-user@ to be able to post there. Oh well.]
I'm usually near-realtime letting relevant non-subscriber posts through, so no worries.
Quoted 10 lines
Today I tried drbd for the first time, and decided to start experimenting on two virtual machines.
And immediately ran into interesting issues, described below. The way to the results wasn't easy, trust me ;)
Software used:
DRBD version 8.3.1 (8.3.2rc1 spewed numerous assertion
failures and crashed both virtual and non-virtual machines).
did that improve?
we are about to release 8.3.2 now,
so feedback on those issues would have been nice.
Quoted 76 lines
kernel 2.6.29.5 compiled for x86 (i686smp)
lvm kernel module from 2.6.29
kvm userspace qemu-kvm-0.10.5.Both virtual machines were identical, using
virtio_net(bridged) andvirtio_blk(on raw files on ext4 fs).When I first set drbd up, waited till it did initial resync, formatted (ext3) and mounted the filesystem on
/dev/drbd1. Next I did a trivial test - copying /usr (tiny 140Mb) to the drbd filesystem.This took........ 29 *minutes* (!!!). Which is about 80 kilobytes per sec. Reminds me ol'good dialup modem... ;)
At this point I tried many different things, including straceing kvm process on host. The thing is that both virtio_net&virtio_blk works quite well, copying that same /usr between two virtual machines over nfs takes less than a minute. And finally, after finding the key knobs, here's the results:
/sys/block/vda/queue/rotationaland ../elevator on drbd *secondary* node are the knobs in question.First of all, for some reason
virtil_blksets its 'rotational' attribute to 0. Which was a surprize to me - both to discover this flag forvirtio_blk(i understand it's present on all block devices) and to see its default value being set to 0. (A side note: the flag is probably misnamed, it was meant to show how costly disk seeks are: rotational disks vs flash media where seeks are not present at all).So, with both primary and secondary virtual disks on real hdd and with default rotational=0 and (my default) elevator=noop, the copy takes 29 minutes.
By changing the two mentioned knobs on PRIMARY I was able to reduce that to 27 minutes.
Now, with rotational=1 and elevator=noop on SECONDARY the whole thing completed in 2m40s. Which is about 10 times a difference!.
Next, with rotational=1 and elevator=cfq on SECONDARY, it completes in about 1m10s sec.
With rotational=1 and elevator=cfq on BOTH nodes it completes in 50 sec.
(for the record, elevator=cfq and rotational=0 on SECONDARY makes it complete in about 9 minutes)
I went further and placed one of the virtual disks into memory on host. And even there, while speed increased dramatically, rotational on SECONDARY (even if the disk on secondary is in host's memory) had about 10x effect. On the other hand, placing virtual disk on PRIMARY to memory changes very little.
On the host side, by straceing kvm process it may be noticed that with default rotational=0, there's no write block mergeing going on at all. All writes on host are done in 4096-byte-sized chunks (block size of ext3fs). But after setting rotational to 1 I see writes being combined -- many 32Kb writes, 64Kb writes etc.
On primary node there's almost no change in write pattern on the host, writes are being combined regardless of `rotational' flag.
Another observation on kvm side - when I/O activity starts, kvm spawns many threads to perform the I/O. This is, again, especially visible on the secondary node. It writes adjanced 4096-byte blocks in different threads, about 10..20 threads at any given time. I *guess* it's ncq/tcq-like thing implemented by kvm, but it looks utterly ineffecient - having in mind sequential block numbers.
Quoted 1 line
So finally some questions.
Interessting findings.
What IO scheduler is used on the HOST?
did you try to leave the guests in their default setting, (rotational=0, elevator=noop)
and use the _deadline_ scheduler in the host?
Quoted 3 lines
drbd: what's the difference in write pattern on secondary and
primary nodes? Why `rotational' flag makes very big difference on secondary and no difference whatsoever on primary?
not much.
disk IO on Primary is usually submitted in the context of the submitter (vm subsystem, filesystem or the process itself) whereas on Secondary, IO is naturally submitted just by the DRBD receiver and worker threads.
Quoted 15 lines
kvm: what is this 'rotational' flag, what's its meaning here?
I guess it should be controllable from the command line if all else fails. In any way it's definitely something to keep in mind. I guess more easily triggerable workload will show something similar (without drbd).kvm: i/o threads - should there be a way to control the amount of
threads? With default workload generated by drbd on secondary node having less thread makes more sense.kvm: it has been said that using noop elevator on guest makes sense
since host does its own elevator/reordering. But this example shows "nicely" that this isn't always the case. I wonder how "general" this example is. Will try to measure further.
Quoted 4 lines
Another "interesting" issue shown by drbd here: on the slow case, secondary is so much busy with the writes that it misses drbd pings and pongs so that nodes frequently considers the other peer as dead and reconnects/resyncs.
that should not happen, these "DRBD pings" are answered by the "drbd_asender" thread ("a" for "acknowledgement"), which is not involved in submitting/waiting for the IO, and should be mostly idle, if only few requests are completed per unit time.
there are two timeouts here in DRBD: the timeout (default 6 seconds, iirc), and ping-timeout (default 0.5 seconds, iirc).
is it possible, that 500 ms (rtt!) is just too short for the kvm briged network stack, or the drbd_asender is not scheduled in time?
Quoted 6 lines
This is not on a pre-stone-age machine and with not slowest disks (it's 2.5GHz AMD Athlon X2-64 CPU and ~90Mb/sec sata disk).
I didn't try other combinations yet, including non-virtio model. Because all the above took almost whole day today, especially finding the knobs :)
Signature · 7 lines
: Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed
Quoted 10 lines · Lars Ellenberg
Quoted 3 lines · Michael Tokarev
drbd: what's the difference in write pattern on secondary and
primary nodes? Why `rotational' flag makes very big difference on secondary and no difference whatsoever on primary?not much.
disk IO on Primary is usually submitted in the context of the submitter (vm subsystem, filesystem or the process itself) whereas on Secondary, IO is naturally submitted just by the DRBD receiver and worker threads.
just like with KVM itself, using several worker threads against a single IO device makes performance heavily dependent on a sensible elevator algorithm. ideally, there should be only one worker thread for each thread/process originating the initial write. unfortunately DRBD, being a block/level protocol, might have a hard time unraveling which writes belong to which process. maybe just merging adjacent (in block address space, not in time) write operations would keep most of the relationships.
Signature · 1 line
Javier
A message that is not in the archive (sent off-list or not yet received): 200907030806.07624.javier-796Irmz5ZkZBDgjK7y7TUQ@public.gmane.org
Quoted 12 lines · Javier Guerra
Quoted 10 lines · Lars Ellenberg
Quoted 3 lines · Michael Tokarev
drbd: what's the difference in write pattern on secondary and
primary nodes? Why `rotational' flag makes very big difference on secondary and no difference whatsoever on primary?not much.
disk IO on Primary is usually submitted in the context of the submitter (vm subsystem, filesystem or the process itself) whereas on Secondary, IO is naturally submitted just by the DRBD receiver and worker threads.
Quoted 8 lines
just like with KVM itself, using several worker threads against a single IO device makes performance heavily dependent on a sensible elevator algorithm. ideally, there should be only one worker thread for each thread/process originating the initial write. unfortunately DRBD, being a block/level protocol, might have a hard time unraveling which writes belong to which process. maybe just merging adjacent (in block address space, not in time) write operations would keep most of the relationships.
again, on the DRBD secondary, _all_ IO is naturally submitted by _one_ thread (the receiver thread), appart from just a few "internally" generated writes, which have to happen from a second thread, the (one) drbd worker thread.
no, we do not do thread pooling (yet).
the elevator of the lower level block device (in this case, the kvm virtual block device, or the host real block device) is responsible for merging requests adjacent in block address space.
you have to choose a sensible elevator there.
please do use deadline IO scheduler on the host.
please do use deadline IO scheduler on the host.
if you really think you have reason to use cfq,
just _try_ deadline anyways.
or upgrade to a 2.6.30 or 2.6.31 kernel, where cfq should finally be usable for many IO requests of one thread (receiver) depending on single synchronous IO from an other thread (drbd meta data updates from worker). but using cfq on kernel < 2.6.30 for this access pattern introduces unneccessary latency of "idle-wait" (in the range of ms!) for those meta data updates.
so, on kernel < 2.6.30,
please do use deadline IO scheduler.
ok?
;)
and if performance then still sucks,
we can talk further about why this may be.
Signature · 7 lines
: Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed
Quoted 2 lines · Lars Ellenberg
the elevator of the lower level block device (in this case, the kvm virtual block device, or the host real block device)
so, the original post (Michael) was running drbd on the KVM guests??
i thought the only sensible setup was using dbrb on the hosts, to make VMs live-migrable.
Signature · 1 line
Javier
A message that is not in the archive (sent off-list or not yet received): 90eb1dc70907031559y47c97667s7a38fcb133a42c6d-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 5 lines · Javier Guerra
Quoted 2 lines · Lars Ellenberg
the elevator of the lower level block device (in this case, the kvm virtual block device, or the host real block device)
so, the original post (Michael) was running drbd on the KVM guests??
That was my impression, from the introduction:
Quoted 5 lines
Quoted 2 lines
Today I tried drbd for the first time, and decided to start experimenting on two virtual machines.
i thought the only sensible setup was using dbrb on the hosts, to make VMs live-migrable.
Right.
But his findings are still interessting. It may help to track down all spurious timeouts and "anticipatory" idle-waits, and try to disable them, or move them out of the picture. Even ignoring DRBD for a moment, that should also be an interessting and beneficial excercise for VM block device performance in virtualization alone.
Signature · 7 lines
: Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed