I/O performance of VirtIO
Hello!
I just tested qemu-kvm-0.11.0 with the KVM module of kernel 2.6.31.1. I noticed that the I/O performance of an unattended stock Debian Lenny install dropped somehow. The test machines ran with kvm-88 and 2.6.30.x before. The difference is very noticeable (went from about 5 minutes up to 15-25 minutes). The two test machines have different CPUs (one is an Intel Core2 CPU, the other runs with an AMD Athlon 64 X2 Dual).
Is this the effect of added code regarding caching/data integrity to the VirtIO block layer or somewhere else? The qemu-system-x86_64 seems to hang a lot more in heavy I/O (showing 'D' in top/htop).
The command line is quite straight-forward:
qemu-system-x86_64 -drive file=debian.qcow2,if=virtio,boot=on -cdrom \/srv/isos/debian-502-i386-netinst.iso -smp 2 -boot d -m 512 -net nic \
-net user -usb
Installation was repeated multiple times, every time the test machines hat no other load. The effect is the same with a Windows XP guest running without VirtIO.
Best,
René.
Signature · 4 lines
)\._.,--....,'``. fL Let GNU/Linux work for you while you take a nap. /, _.. \ _\ (`._ ,. R. Pfeiffer <lynx at luchs.at> + http://web.luchs.at/ `._.-(,_..'--(,_..'`-.;.' - System administration + Consulting + Teaching - Got mail delivery problems? http://web.luchs.at/information/blockedmail.php
Quoted 9 lines · René Pfeiffer
Hello!
I just tested qemu-kvm-0.11.0 with the KVM module of kernel 2.6.31.1. I noticed that the I/O performance of an unattended stock Debian Lenny install dropped somehow. The test machines ran with kvm-88 and 2.6.30.x before. The difference is very noticeable (went from about 5 minutes up to 15-25 minutes). The two test machines have different CPUs (one is an Intel Core2 CPU, the other runs with an AMD Athlon 64 X2 Dual).
Quoted 8 lines
Is this the effect of added code regarding caching/data integrity to the VirtIO block layer or somewhere else? The qemu-system-x86_64 seems to hang a lot more in heavy I/O (showing 'D' in top/htop).
The command line is quite straight-forward:
qemu-system-x86_64 -drive file=debian.qcow2,if=virtio,boot=on -cdrom \/srv/isos/debian-502-i386-netinst.iso-smp 2 -boot d -m 512 -net nic \
-net user -usb
^^^^^^^^^
Care to try with something more real than user-level networking? You're using netinstall which - apparently - tries to use some networking d/loading components etc, and userlevel networking is known to be very very slow....
Also try the same with raw images.
I for one does not see any noticeable speed difference with tap networking (virtio or e1000 or rtl8139) and with raw disks (either virtio or ide), on either linux or windows guests (windows without virtio so far). But granted, I didn't try user-level networking, and I don't try qcow too often (however collegue of mine who uses qcow didn't complain about speed either).
/mjt
Quoted 3 lines
Installation was repeated multiple times, every time the test machines hat no other load. The effect is the same with a Windows XP guest running without VirtIO.
Quoted 14 lines · Michael Tokarev appeared and
Quoted 13 lines · René Pfeiffer
Hello!
I just tested qemu-kvm-0.11.0 with the KVM module of kernel 2.6.31.1. I noticed that the I/O performance of an unattended stock Debian Lenny install dropped somehow. The test machines ran with kvm-88 and 2.6.30.x before. The difference is very noticeable (went from about 5 minutes up to 15-25 minutes). The two test machines have different CPUs (one is an Intel Core2 CPU, the other runs with an AMD Athlon 64 X2 Dual).
Is this the effect of added code regarding caching/data integrity to the VirtIO block layer or somewhere else? The qemu-system-x86_64 seems to hang a lot more in heavy I/O (showing 'D' in top/htop).
Quoted 7 lines
Quoted 4 lines
The command line is quite straight-forward:
qemu-system-x86_64 -drive file=debian.qcow2,if=virtio,boot=on -cdrom \/srv/isos/debian-502-i386-netinst.iso-smp 2 -boot d -m 512 -net nic \
-net user -usb^^^^^^^^^
Care to try with something more real than user-level networking?
Yes, I tried that on the other machine. It made not much difference (the installation files are on local Squid proxies).
But I found that setting cache=writeback restores the old behaviour. I think the default changed to cache=writethrough.
Quoted 3 lines
You're using netinstall which - apparently - tries to use some networking d/loading components etc, and userlevel networking is known to be very very slow....
Right, I just verified the disk I/O performance with severall runs of hdparm and fresh installations using cache=none, cache=writeback and cache=writethrough settings. The network settings were the same (the test machine with the software bridge setup is down at the moment). I wanted to compare the behaviour of the I/O load.
Best,
René.
Signature · 4 lines
)\._.,--....,'``. fL Let GNU/Linux work for you while you take a nap. /, _.. \ _\ (`._ ,. R. Pfeiffer <lynx at luchs.at> + http://web.luchs.at/ `._.-(,_..'--(,_..'`-.;.' - System administration + Consulting + Teaching - Got mail delivery problems? http://web.luchs.at/information/blockedmail.php
Quoted 18 lines · Michael Tokarev
Quoted 9 lines · René Pfeiffer
Hello!
I just tested qemu-kvm-0.11.0 with the KVM module of kernel 2.6.31.1. I noticed that the I/O performance of an unattended stock Debian Lenny install dropped somehow. The test machines ran with kvm-88 and 2.6.30.x before. The difference is very noticeable (went from about 5 minutes up to 15-25 minutes). The two test machines have different CPUs (one is an Intel Core2 CPU, the other runs with an AMD Athlon 64 X2 Dual).
Quoted 8 lines
Is this the effect of added code regarding caching/data integrity to the VirtIO block layer or somewhere else? The qemu-system-x86_64 seems to hang a lot more in heavy I/O (showing 'D' in top/htop).
The command line is quite straight-forward:
qemu-system-x86_64 -drive file=debian.qcow2,if=virtio,boot=on -cdrom \/srv/isos/debian-502-i386-netinst.iso-smp 2 -boot d -m 512 -net nic \
-net user -usb
Quoted 6 lines
^^^^^^^^^
Care to try with something more real than user-level networking? You're using netinstall which - apparently - tries to use some networking d/loading components etc, and userlevel networking is known to be very very slow....
It can be particularly slow if you use in-kernel irqchips and the default NIC emulation (up to 10 times slower), some effect I always wanted to understand on a rainy day. So, when you actually want -net user, try -no-kvm-irqchip.
Jan
Quoted 5 lines · Jan Kiszka
It can be particularly slow if you use in-kernel irqchips and the default NIC emulation (up to 10 times slower), some effect I always wanted to understand on a rainy day. So, when you actually want -net user, try -no-kvm-irqchip.
This might be due to a missing SIGIO or SIGALRM; -no-kvm-irqchip generates a lot of extra signals and thus polling opportunities.
I see 300kB/sec from slirp; always assumed it was due to missing tcp features.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 22.10.2009 um 18:29 schrieb Avi Kivity <[email protected]>:
Quoted 8 lines · Jan Kiszka
Quoted 5 lines
It can be particularly slow if you use in-kernel irqchips and the default NIC emulation (up to 10 times slower), some effect I always wanted to understand on a rainy day. So, when you actually want -net user, try -no-kvm-irqchip.
This might be due to a missing SIGIO or SIGALRM; -no-kvm-irqchip generates a lot of extra signals and thus polling opportunities.
Isn't that what dedicated io threads are supposed to solve?
Alex
Quoted 10 lines · Alexander Graf
Am 22.10.2009 um 18:29 schrieb Avi Kivity <[email protected]>:
Quoted 7 lines · Jan Kiszka
Quoted 5 lines
It can be particularly slow if you use in-kernel irqchips and the default NIC emulation (up to 10 times slower), some effect I always wanted to understand on a rainy day. So, when you actually want -net user, try -no-kvm-irqchip.
Quoted 4 lines
Quoted 1 line
This might be due to a missing SIGIO or SIGALRM; -no-kvm-irqchip generates a lot of extra signals and thus polling opportunities.
Isn't that what dedicated io threads are supposed to solve?
No. Dedicated I/O threads provide parallelism. All latency needs is to have SIGIO sent on all file descriptors (or rather, in qemu-kvm with irqchip, to have all file descriptors in the poll() call).
Jan, does slirp add new connections to the select set?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Alexander Graf
Am 22.10.2009 um 18:29 schrieb Avi Kivity <[email protected]>:
Quoted 9 lines · Jan Kiszka
Quoted 5 lines
It can be particularly slow if you use in-kernel irqchips and the default NIC emulation (up to 10 times slower), some effect I always wanted to understand on a rainy day. So, when you actually want -net user, try -no-kvm-irqchip.
This might be due to a missing SIGIO or SIGALRM; -no-kvm-irqchip generates a lot of extra signals and thus polling opportunities.
Quoted 9 lines
Quoted 2 lines
Isn't that what dedicated io threads are supposed to solve?
No. Dedicated I/O threads provide parallelism. All latency needs is to have SIGIO sent on all file descriptors (or rather, in qemu-kvm with irqchip, to have all file descriptors in the
poll()call).Jan, does slirp add new connections to the select set?
It should do so in slirp_select_fill (it iterates over all TCP&UDP sockets of all instances). I think without doing this, slirp wouldn't receive a single bit at all (no activity without FD_ISSET).
Jan
Quoted 11 lines · Jan Kiszka
Quoted 7 lines
No. Dedicated I/O threads provide parallelism. All latency needs is to have SIGIO sent on all file descriptors (or rather, in qemu-kvm with irqchip, to have all file descriptors in the
poll()call).Jan, does slirp add new connections to the select set?
It should do so in
slirp_select_fill(it iterates over all TCP&UDP sockets of all instances). I think without doing this, slirp wouldn't receive a single bit at all (no activity withoutFD_ISSET).
Yes, so it seems from the code. But something is missing if you get better performance with -no-kvm-irqchip. Perhaps timers are off.
Signature · 1 line
error compiling committee.c: too many arguments to function