how to tweak kernel to get the best out of kvm?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi folks,
Problem: My kvm server (8 cores, 64 GByte RAM, amd64) can eat up all block device or file system performance, so that the kvm clients become almost unresponsive. This is _very_ bad. I would like to make sure that the kvm clients do not affect each other, and that all (including the server itself) get a fair part of computing power and memory space.
What config options would you suggest to build and run a Linux kernel optimized for running kvm clients?
Sorry for asking, but AFAICS some general guidelines for kvm are missing here. Of course I saw a lot of options in Documentation/\
kernel-parameters.txt, but unfortunately I am not a kernel hacker.
Any helpful comment would be highly appreciated.
Regards
Harri
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkuRIVYACgkQUTlbRTxpHjebXQCdHSKXYPfkwzSeyawrumELfVPu MbYAn07JoomtdVkA6YES4EgKayn6KSH6
=2mVb
-----END PGP SIGNATURE-----
Quoted 12 lines · Harald Dunkel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hi folks,
Problem: My kvm server (8 cores, 64 GByte RAM, amd64) can eat up all block device or file system performance, so that the kvm clients become almost unresponsive. This is _very_ bad. I would like to make sure that the kvm clients do not affect each other, and that all (including the server itself) get a fair part of computing power and memory space.
Please describe the issue in detail, provide output from 'vmstat' and 'top'.
Quoted 9 lines
What config options would you suggest to build and run a Linux kernel optimized for running kvm clients?
Sorry for asking, but AFAICS some general guidelines for kvm are missing here. Of course I saw a lot of options in Documentation/\
kernel-parameters.txt, but unfortunately I am not a kernel hacker.Any helpful comment would be highly appreciated.
One way to ensure guests don't affect each other is not to overcommit, that is make sure each guest gets its own cores, there is enough memory for all guests, and guests have separate disks. Of course that defeats the some of the reasons for virtualizing in the first place; but if you share resources, some compromises must be made.
If you do share resources, then Linux manages how they are shared. The scheduler will share the processors, the memory management subsystem will share memory, and the I/O scheduler will share disk bandwidth. If you see a problem in one of these areas you will need to tune the subsystem that is misbehaving.
There is also a larger effort to improve control of sharing called control groups (cgroups). You may want to read up on this as it can provide very fine grain control on resource sharing.
Signature · 1 line
error compiling committee.c: too many arguments to function
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 41 lines · Harald Dunkel
Hi Avi,
Quoted 15 lines · Avi Kivity
Quoted 13 lines · Harald Dunkel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hi folks,
Problem: My kvm server (8 cores, 64 GByte RAM, amd64) can eat up all block device or file system performance, so that the kvm clients become almost unresponsive. This is _very_ bad. I would like to make sure that the kvm clients do not affect each other, and that all (including the server itself) get a fair part of computing power and memory space.
Quoted 4 lines
Please describe the issue in detail, provide output from 'vmstat' and 'top'.
Sorry for the delay. I cannot put these services at risk, so I have setup a test environment on another host (2 quadcore Xeons, ht enabled, 32 GByte RAM, no swap, bridged networking) to reproduce the problem.
There are 8 virtual hosts, each with a single CPU, 1 GByte RAM and 4 GByte swap on a virtual disk. The virtual disks are image files in the local file system. These images are not shared.
For testing each virtual host builds the Linux kernel. In parallel I am running rsync to clone a remote virtual machine (22 GByte) to the local physical disk.
Attached you can find the requested logs. The kern.log shows the problem: The virtual CPUs get stuck (as it seems). Several virtual hosts showed this effect. One vhost was unresponsive for more than 30 minutes.
Quoted 7 lines
Surely this is a stress test, but I had a similar effect with our virtual mail server on the production system, while I was running a similar rsync session. mailhost was unresponsive for more than 2 minutes, then it was back. The other 8 virtual hosts on this system were started, but idle (AFAICT).
You have tons of iowait time, indicating an I/O bottleneck.
What filesystem are you using for the host? Are you using qcow2 or raw access? What's the qemu command line.
Perhaps your filesystem doesn't perform well on synchronous writes. For testing only, you might try cache=writeback.
Quoted 4 lines
BTW, please note that free memory goes down over time. This happens only if the rsync is running. Without rsync the free memory is stable.
That's expected. rsync fills up the guest and host pagecache, both drain free memory (the guest only until it has touched all of its memory).
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
What config options would you suggest to build and run a Linux kernel optimized for running kvm clients?
Sorry for asking, but AFAICS some general guidelines for kvm are missing here. Of course I saw a lot of options in Documentation/\
kernel-parameters.txt, but unfortunately I am not a kernel hacker.Quoted 10 lines
Quoted 3 lines
Any helpful comment would be highly appreciated.
One way to ensure guests don't affect each other is not to overcommit, that is make sure each guest gets its own cores, there is enough memory for all guests, and guests have separate disks. Of course that defeats the some of the reasons for virtualizing in the first place; but if you share resources, some compromises must be made.
Quoted 5 lines
How many virtual machines would you assume I could run on a host with 64 GByte RAM, 2 quad cores, a bonding NIC with 4*1Gbit/sec and a hardware RAID? Each vhost is supposed to get 4 GByte RAM and 1 CPU.
15 guests should fit comfortably, more with ksm running if the workloads are similar, or if you use ballooning.
Quoted 8 lines
Quoted 8 lines
If you do share resources, then Linux manages how they are shared. The scheduler will share the processors, the memory management subsystem will share memory, and the I/O scheduler will share disk bandwidth. If you see a problem in one of these areas you will need to tune the subsystem that is misbehaving.
Quoted 4 lines
Do you think that the bridge connecting the tunnel devices and the real NIC makes the problems? Is there also a subsystem managing network access?
Here the problem is likely the host filesystem and/or I/O scheduler.
The optimal layout is placing guest disks in LVM volumes, and accessing them with -drive file=...,cache=none. However, file-based access should also work.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 2 lines · Avi Kivity
15 guests should fit comfortably, more with ksm running if the workloads are similar, or if you use ballooning.
is there any simple way to get some stats to see how is ksm doing?
Signature · 1 line
Javier
Quoted 7 lines · Javier Guerra Giraldez
Quoted 3 lines · Avi Kivity
15 guests should fit comfortably, more with ksm running if the workloads are similar, or if you use ballooning.
is there any simple way to get some stats to see how is ksm doing?
See /sys/kernel/mm/ksm
Signature · 1 line
error compiling committee.c: too many arguments to function
Hi Avi,
I had missed to include some important syslog lines from the host system. See attachment.
Quoted 3 lines · Avi Kivity
You have tons of iowait time, indicating an I/O bottleneck.
Is this disk IO or network IO? The rsync session puts a high load on both, but actually I do not see how a high load on disk or block IO could make the virtual hosts unresponsive, as shown by the hosts syslog?
Quoted 3 lines
What filesystem are you using for the host? Are you using qcow2 or raw access? What's the qemu command line.
It is ext3 and qcow2. Currently I am testing with reiserfs on the host system. The system performance seems to be worse, compared with ext3.
Here is the kvm command line (as generated by libvirt):
/usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 1024 -smp 1 -name test0.0 \ -uuid 74e71149-4baf-3af0-9c99-f4e50273296f \ -monitor unix:/var/lib/libvirt/qemu/test0.0.monitor,server,nowait \ -boot c -drive if=ide,media=cdrom,bus=1,unit=0 \ -drive file=/export/storage/test0.0.img,if=virtio,boot=on \ -net nic,macaddr=00:16:36:94:7e:f3,vlan=0,model=virtio,name=net0 \ -net tap,fd=60,vlan=0,name=hostnet0 -serial pty -parallel none \ -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -balloon virtio
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
How many virtual machines would you assume I could run on a host with 64 GByte RAM, 2 quad cores, a bonding NIC with 4*1Gbit/sec and a hardware RAID? Each vhost is supposed to get 4 GByte RAM and 1 CPU.
15 guests should fit comfortably, more with ksm running if the workloads are similar, or if you use ballooning.
15 vhosts would be nice. ksm is in the kernel, but not in my qemu-kvm (yet).
Quoted 7 lines
Here the problem is likely the host filesystem and/or I/O scheduler.
The optimal layout is placing guest disks in LVM volumes, and accessing them with -drive file=...,cache=none. However, file-based access should also work.
I will try LVM tomorrow, when the test with reiserfs is completed.
Many thanx
Harri
Quoted 11 lines · Harald Dunkel
Hi Avi,
I had missed to include some important syslog lines from the host system. See attachment.
Quoted 3 lines · Avi Kivity
You have tons of iowait time, indicating an I/O bottleneck.
Is this disk IO or network IO?
disk.
Quoted 6 lines
The rsync session puts a
high load on both, but actually I do not see how a high load on disk or block IO could make the virtual hosts unresponsive, as shown by the hosts syslog?
qcow2 is still not fully asynchronous, so sometimes when it waits, a vcpu waits as well.
Quoted 10 lines
Quoted 7 lines
Here the problem is likely the host filesystem and/or I/O scheduler.
The optimal layout is placing guest disks in LVM volumes, and accessing them with -drive file=...,cache=none. However, file-based access should also work.
I will try LVM tomorrow, when the test with reiserfs is completed.
If the slowdown is indeed due to I/O, LVM (with cache=off) should eliminate it completely.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 4 lines · Avi Kivity
If the slowdown is indeed due to I/O, LVM (with cache=off) should eliminate it completely.
As promised I have installed LVM: The difference is remarkable. My test case (running 8 vhosts in parallel, each building a Linux kernel) just works. There is no blocking job (by now), all vhosts can be pinged, great.
Many thanx for your help, and for the nice software, of course.
Regards
Harri