Memory under KVM?
Hi everyone, I'm new to the list and I have a couple questions that we are wondering about here at work...
we have notice that the KVM processes on the host take much more memory than the memory we have told the VM to use.. a ruff example.. if we tell KVM to use 2 gigs for one VM it will end up showing on the host process list for that VM like 3 gigs or more...
Why do I ask this? well we need to figure out how much memory to add to our host server so we can calculate the number of VM's we can run there etc etc..
Thanks for the help
Quoted 10 lines · On Friday 11 December 2009 15:43:01 rek2 wrote:
Hi everyone, I'm new to the list and I have a couple questions that we are wondering about here at work...
we have notice that the KVM processes on the host take much more memory than the memory we have told the VM to use.. a ruff example.. if we tell KVM to use 2 gigs for one VM it will end up showing on the host process list for that VM like 3 gigs or more...
Why do I ask this? well we need to figure out how much memory to add to our host server so we can calculate the number of VM's we can run there etc etc..
Quoted 2 lines
Thanks for the help
My guests' RES size are all either right at or below what the -m parameter is set to.
So I think you have some sort of issue.
Quoted 4 lines · On 12/11/2009 11:43 PM, rek2 wrote:
Hi everyone, I'm new to the list and I have a couple questions that we are wondering about here at work...
we have notice that the KVM processes on the host take much more memory than the memory we have told the VM to use.. a ruff example.. if we tell KVM to use 2 gigs for one VM it will end up showing on the host process list for that VM like 3 gigs or more...
Why do I ask this? well we need to figure out how much memory to add to our host server so we can calculate the number of VM's we can run there etc etc..
Can you give an example? A snapshot from 'top' would do.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 11 lines · Avi Kivity
Quoted 9 lines · On 12/11/2009 11:43 PM, rek2 wrote:
Hi everyone, I'm new to the list and I have a couple questions that we are wondering about here at work...
we have notice that the KVM processes on the host take much more memory than the memory we have told the VM to use.. a ruff example.. if we tell KVM to use 2 gigs for one VM it will end up showing on the host process list for that VM like 3 gigs or more...
Why do I ask this? well we need to figure out how much memory to add to our host server so we can calculate the number of VM's we can run there etc etc..
Quoted 2 lines
Can you give an example? A snapshot from 'top' would do.
I have the opposite happen, when a VM is started, RES is usually lower than -m, which I find slightly odd. But makes sense if qemu/kvm don't actually allocate memory from the host till its requested the first time (if only it would return some of it afterwards, it would be even better).
I just fully shut down and restarted on of my vms, which is set to use 128-256 MB ram max. RES is like 72MB on start, and VIRT is 454M. RES generally gets up around 120MB ram when its doing something.
One thing I do find a little odd is one of my VMs which is allocated 512MB ram, has a VIRT of 826MB ram. I didn't realize that qemu had so many lib dependencies. Due to kvm not supporting giving memory back, besides by swapping large portions of unused guest ram, my host currently has over 1G used swap. Not particularly happy with that, but it doesn't seem to effect performance too much (except that it generally likes to swap host processes first, guest performance is decent, but host, not so much).
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 4 lines · Thomas Fjellstrom
I have the opposite happen, when a VM is started, RES is usually lower than -m, which I find slightly odd. But makes sense if qemu/kvm don't actually allocate memory from the host till its requested the first time
That is the case.
Quoted 3 lines
(if only it
would return some of it afterwards, it would be even better).
Use the balloon driver to return memory to the host.
Quoted 7 lines
I just fully shut down and restarted on of my vms, which is set to use 128-256 MB ram max. RES is like 72MB on start, and VIRT is 454M. RES generally gets up around 120MB ram when its doing something.
One thing I do find a little odd is one of my VMs which is allocated 512MB ram, has a VIRT of 826MB ram. I didn't realize that qemu had so many lib dependencies.
It's not just libraries, it's mostly glibc malloc() allocating huge pools per thread, as well as large thread stacks.
Quoted 6 lines
Due to kvm not supporting giving memory back, besides by swapping large portions of unused guest ram, my host currently has over 1G used swap. Not particularly happy with that, but it doesn't seem to effect performance too much (except that it generally likes to swap host processes first, guest performance is decent, but host, not so much).
The Linux vm prefers anonymous memory, so guests do get an advantage.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Avi Kivity
Quoted 4 lines · Thomas Fjellstrom
I have the opposite happen, when a VM is started, RES is usually lower than -m, which I find slightly odd. But makes sense if qemu/kvm don't actually allocate memory from the host till its requested the first time
Quoted 6 lines
That is the case.
Quoted 2 lines
(if only it
would return some of it afterwards, it would be even better).Use the balloon driver to return memory to the host.
Will it actually just free the memory and leave the total memory size in the VM alone? Last I checked it would just decrease the total memory size, which isn't that useful. Sometimes it needs more ram, so its given 512M ram, but most of the time can live on 100M or so.
Quoted 11 lines
Quoted 7 lines
I just fully shut down and restarted on of my vms, which is set to use 128-256 MB ram max. RES is like 72MB on start, and VIRT is 454M. RES generally gets up around 120MB ram when its doing something.
One thing I do find a little odd is one of my VMs which is allocated 512MB ram, has a VIRT of 826MB ram. I didn't realize that qemu had so many lib dependencies.
It's not just libraries, it's mostly glibc
malloc()allocating huge pools per thread, as well as large thread stacks.
Quoted 9 lines
Quoted 6 lines
Due to kvm not supporting giving memory back, besides by swapping large portions of unused guest ram, my host currently has over 1G used swap. Not particularly happy with that, but it doesn't seem to effect performance too much (except that it generally likes to swap host processes first, guest performance is decent, but host, not so much).
The Linux vm prefers anonymous memory, so guests do get an advantage.
I think the only thing I'd like to have now is automatic memory return, much like vmware server has. It doesn't change what the guest VM sees, it just flushes the unused ram back to the host.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 8 lines · Thomas Fjellstrom
Quoted 3 lines
Use the balloon driver to return memory to the host.
Will it actually just free the memory and leave the total memory size in the VM alone? Last I checked it would just decrease the total memory size, which isn't that useful. Sometimes it needs more ram, so its given 512M ram, but most of the time can live on 100M or so.
If you balloon and than balloon back the guest will be able to reallocate all this memory.
Quoted 7 lines
Quoted 3 lines
The Linux vm prefers anonymous memory, so guests do get an advantage.
I think the only thing I'd like to have now is automatic memory return, much like vmware server has. It doesn't change what the guest VM sees, it just flushes the unused ram back to the host.
Linux usually keeps very little RAM free (it's kept as cache). So there has to be some action on the part of the host to get the guest to free things. For Windows guests you can use ksm to reclaim free memory (since Windows will zero it).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 13 lines · Avi Kivity
Quoted 6 lines · Thomas Fjellstrom
Quoted 1 line
Use the balloon driver to return memory to the host.
Will it actually just free the memory and leave the total memory size in the VM alone? Last I checked it would just decrease the total memory size, which isn't that useful. Sometimes it needs more ram, so its given 512M ram, but most of the time can live on 100M or so.
If you balloon and than balloon back the guest will be able to reallocate all this memory.
Quoted 2 lines
Quoted 1 line
The Linux vm prefers anonymous memory, so guests do get an advantage.
Quoted 9 lines
Quoted 3 lines
I think the only thing I'd like to have now is automatic memory return, much like vmware server has. It doesn't change what the guest VM sees, it just flushes the unused ram back to the host.
Linux usually keeps very little RAM free (it's kept as cache). So there has to be some action on the part of the host to get the guest to free things. For Windows guests you can use ksm to reclaim free memory (since Windows will zero it).
I'm waiting for 2.6.32 to hit Debian Sid before I start playing with ksm (I don't think its in 2.6.31).
The problem is it should be automatic. The balloon driver itself or some other mechanism should be capable of noticing when it can free up a bunch of guest memory. I can't be bothered to manually sit around and monitor memory usage on my host so I can then go into virt-manager to reduce memory to each guest.
What vmware server had worked great. After some time (quite a lot usually) something would flush the cache, and return most of the unused guest ram back to the host.
Also, I don't have any windows guests atm. just 2 BSDs and 5 linux guests. I've had to do some tweaking guest side to cut down on ram (reduce apache, mysql, nginx and other services threads/forks). Something that wasn't necessary at all with vmware server.
As I said before, that last feature is pretty much the last thing that would make KVM perfect for my purposes. that is, returning guest memory without actually changing the allocation in the guest.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 7 lines · Thomas Fjellstrom
Quoted 7 lines
Linux usually keeps very little RAM free (it's kept as cache). So there has to be some action on the part of the host to get the guest to free things. For Windows guests you can use ksm to reclaim free memory (since Windows will zero it).
Quoted 9 lines
I'm waiting for 2.6.32 to hit Debian Sid before I start playing with ksm (I don't think its in 2.6.31).
The problem is it should be automatic. The balloon driver itself or some other mechanism should be capable of noticing when it can free up a bunch of guest memory. I can't be bothered to manually sit around and monitor memory usage on my host so I can then go into virt-manager to reduce memory to each guest.
That should be pretty easy though it will have an effect on guest performance.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Thomas Fjellstrom
Quoted 4 lines
Linux usually keeps very little RAM free (it's kept as cache). So there has to be some action on the part of the host to get the guest to free things. For Windows guests you can use ksm to reclaim free memory (since Windows will zero it).
I'm waiting for 2.6.32 to hit Debian Sid before I start playing with ksm (I don't think its in 2.6.31).
Quoted 9 lines
Quoted 5 lines
The problem is it should be automatic. The balloon driver itself or some other mechanism should be capable of noticing when it can free up a bunch of guest memory. I can't be bothered to manually sit around and monitor memory usage on my host so I can then go into virt-manager to reduce memory to each guest.
That should be pretty easy though it will have an effect on guest performance.
As long as its only done after an appropriately long idle period (ie: theres been X MB's free for a long time, give it back), I can't see it harming performance too much. At least not more than setting ram too low when manually (de)ballooning memory.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 7 lines · Thomas Fjellstrom
Quoted 7 lines
Quoted 7 lines
The problem is it should be automatic. The balloon driver itself or some other mechanism should be capable of noticing when it can free up a bunch of guest memory. I can't be bothered to manually sit around and monitor memory usage on my host so I can then go into virt-manager to reduce memory to each guest.
Quoted 9 lines
Quoted 4 lines
That should be pretty easy though it will have an effect on guest performance.
As long as its only done after an appropriately long idle period (ie: theres been X MB's free for a long time, give it back), I can't see it harming performance too much. At least not more than setting ram too low when manually (de)ballooning memory.
It depends on what your expectations are. If you have a lot of memory you might be surprised when you access an idle guest and have to wait for it to page itself back from disk.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 9 lines · Thomas Fjellstrom
Quoted 8 lines
Quoted 5 lines
The problem is it should be automatic. The balloon driver itself or some other mechanism should be capable of noticing when it can free up a bunch of guest memory. I can't be bothered to manually sit around and monitor memory usage on my host so I can then go into virt-manager to reduce memory to each guest.
That should be pretty easy though it will have an effect on guest performance.
Quoted 9 lines
Quoted 4 lines
As long as its only done after an appropriately long idle period (ie:
theres been X MB's free for a long time, give it back), I can't see it harming performance too much. At least not more than setting ram too low when manually (de)ballooning memory.It depends on what your expectations are. If you have a lot of memory you might be surprised when you access an idle guest and have to wait for it to page itself back from disk.
Why would it be swaping in that case? Only unused/free/cache memory should be returned to the host.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 8 lines · Thomas Fjellstrom
Quoted 6 lines
It depends on what your expectations are. If you have a lot of memory you might be surprised when you access an idle guest and have to wait for it to page itself back from disk.
Why would it be swaping in that case? Only unused/free/cache memory should be returned to the host
Right, it would return cache memory, and when you use the guest next time, it will have to refill its cache.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Thomas Fjellstrom
Quoted 3 lines
It depends on what your expectations are. If you have a lot of memory you might be surprised when you access an idle guest and have to wait for it to page itself back from disk.
Why would it be swaping in that case? Only unused/free/cache memory should be returned to the host
Unless of course you were referring to the case of manually de-ballooning memory in the guests. Yes, swapping in the guests is slow, and you should try not to set the memory limit (-m) too small for a given workload.
Having a dynamic ballooning feature that did not actually change the guests view of ram wouldn't have that problem, especially since you're not returning any memory that's in use in the guest. And since KVM already supports running with large ranges of its assigned memory not actually assigned to it, dynamic ballooning probably isn't hard to support.
The memory over-commit "rate" on my old setup was rather astonishing. A couple of my guests would eventually get as low as showing 10MB ram in use. Even the larger memory users would get down as low as 1/5th the allocated ram after sitting mostly idle for a while. But since the full assigned ram is sometimes needed, just reducing the total assignment isn't a good option.
Quoted 2 lines
Right, it would return cache memory, and when you use the guest next time, it will have to refill its cache.
Sure, but there are hours where the guests can run with minimal memory use. It would allow one to run many more guests at the same time, if you know some/many of them won't always be using all of their assigned ram.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Hi Thanks for the responses, but look:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 1 -name vm_hsci -uuid 52ed4c7c-65e4-325e-0f96-87a5be6d854c -monitor unix:/var/run/libvirt/qemu/vm_hsci.monitor,server,nowait -boot c -drive file=/var/kvm_images/vm_hsci.img,if=virtio,index=0,boot=on -net nic,macaddr=00:16:36:5b:c4:e2,vlan=0,name=nic.0 -net tap,fd=16,vlan=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -soundhw es1370
I have -m 1ith 1024 right?
but if I do a top:
1214m 364m 3360 S 0 4.5 46:55.24 kvm
so is clearly above -m
what could be the issue?
Thanks
is using 1214 at this moment sometimes it goes up a lot more..
Quoted 50 lines · Thomas Fjellstrom
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Thomas Fjellstrom
I have the opposite happen, when a VM is started, RES is usually lower than -m, which I find slightly odd. But makes sense if qemu/kvm don't actually allocate memory from the host till its requested the first time
Quoted 8 lines
That is the case.
Quoted 3 lines
(if only it
would return some of it afterwards, it would be even better).Use the balloon driver to return memory to the host.
Will it actually just free the memory and leave the total memory size in the VM alone? Last I checked it would just decrease the total memory size, which isn't that useful. Sometimes it needs more ram, so its given 512M ram, but most of the time can live on 100M or so.
Quoted 12 lines
Quoted 8 lines
I just fully shut down and restarted on of my vms, which is set to use 128-256 MB ram max. RES is like 72MB on start, and VIRT is 454M. RES generally gets up around 120MB ram when its doing something.
One thing I do find a little odd is one of my VMs which is allocated 512MB ram, has a VIRT of 826MB ram. I didn't realize that qemu had so many lib dependencies.
It's not just libraries, it's mostly glibc
malloc()allocating huge pools per thread, as well as large thread stacks.Quoted 10 lines
Quoted 7 lines
Due to kvm not supporting giving memory back, besides by swapping large portions of unused guest ram, my host currently has over 1G used swap. Not particularly happy with that, but it doesn't seem to effect performance too much (except that it generally likes to swap host processes first, guest performance is decent, but host, not so much).
The Linux vm prefers anonymous memory, so guests do get an advantage.
I think the only thing I'd like to have now is automatic memory return, much like vmware server has. It doesn't change what the guest VM sees, it just flushes the unused ram back to the host.
Quoted 10 lines · On Sun December 13 2009, rek2 wrote:
Hi Thanks for the responses, but look:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin/usr/bin/kvm-S -M pc-0.11 -m 1024 -smp 1 -namevm_hsci-uuid 52ed4c7c-65e4-325e-0f96-87a5be6d854c -monitor unix:/var/run/libvirt/qemu/vm_hsci.monitor,server,nowait -boot c -drive file=/var/kvm_images/vm_hsci.img,if=virtio,index=0,boot=on -net nic,macaddr=00:16:36:5b:c4:e2,vlan=0,name=nic.0 -net tap,fd=16,vlan=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -soundhw es1370
Quoted 7 lines
I have -m 1ith 1024 right?
but if I do a top:
1214m 364m 3360 S 0 4.5 46:55.24 kvmso is clearly above -m
what could be the issue?
Thanks
VIRT includes a lot of shared memory, so it's not a very useful number to look at when trying to gauge how much memory a process is using.
Quoted 52 lines
is using 1214 at this moment sometimes it goes up a lot more..
Quoted 37 lines · Thomas Fjellstrom
Quoted 6 lines · Avi Kivity
Quoted 4 lines · Thomas Fjellstrom
I have the opposite happen, when a VM is started, RES is usually lower than -m, which I find slightly odd. But makes sense if qemu/kvm don't actually allocate memory from the host till its requested the first time
Quoted 6 lines
That is the case.
Quoted 2 lines
(if only it
would return some of it afterwards, it would be even better).Use the balloon driver to return memory to the host.
Will it actually just free the memory and leave the total memory size in the VM alone? Last I checked it would just decrease the total memory size, which isn't that useful. Sometimes it needs more ram, so its given 512M ram, but most of the time can live on 100M or so.
Quoted 18 lines
Quoted 7 lines
I just fully shut down and restarted on of my vms, which is set to use 128-256 MB ram max. RES is like 72MB on start, and VIRT is 454M. RES generally gets up around 120MB ram when its doing something.
One thing I do find a little odd is one of my VMs which is allocated 512MB ram, has a VIRT of 826MB ram. I didn't realize that qemu had so many lib dependencies.
It's not just libraries, it's mostly glibc
malloc()allocating huge pools per thread, as well as large thread stacks.Quoted 6 lines
Due to kvm not supporting giving memory back, besides by swapping large portions of unused guest ram, my host currently has over 1G used swap. Not particularly happy with that, but it doesn't seem to effect performance too much (except that it generally likes to swap host processes first, guest performance is decent, but host, not so much).
Quoted 5 lines
Quoted 1 line
The Linux vm prefers anonymous memory, so guests do get an advantage.
I think the only thing I'd like to have now is automatic memory return, much like vmware server has. It doesn't change what the guest VM sees, it just flushes the unused ram back to the host.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 16 lines · Thomas Fjellstrom
Quoted 14 lines · On Sun December 13 2009, rek2 wrote:
Hi Thanks for the responses, but look:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin/usr/bin/kvm-S -M pc-0.11 -m 1024 -smp 1 -namevm_hsci-uuid 52ed4c7c-65e4-325e-0f96-87a5be6d854c -monitor unix:/var/run/libvirt/qemu/vm_hsci.monitor,server,nowait -boot c -drive file=/var/kvm_images/vm_hsci.img,if=virtio,index=0,boot=on -net nic,macaddr=00:16:36:5b:c4:e2,vlan=0,name=nic.0 -net tap,fd=16,vlan=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -soundhw es1370I have -m 1ith 1024 right?
but if I do a top:
1214m 364m 3360 S 0 4.5 46:55.24 kvm
Quoted 7 lines
Quoted 4 lines
so is clearly above -m
what could be the issue?
ThanksVIRT includes a lot of shared memory, so it's not a very useful number to look at when trying to gauge how much memory a process is using.
Ok, so then what stats should we look to calculate the amount of memory a server should have depending on how many guest VM's we will like to use?
Thanks again
Quoted 5 lines
Quoted 2 lines
VIRT includes a lot of shared memory, so it's not a very useful number to look at when trying to gauge how much memory a process is using.
Ok, so then what stats should we look to calculate the amount of memory a server should have depending on how many guests we will like to use?
Thanks again
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · On Mon December 14 2009, rek2 wrote:
Quoted 2 lines
VIRT includes a lot of shared memory, so it's not a very useful number to look at when trying to gauge how much memory a process is using.
Ok, so then what stats should we look to calculate the amount of memory a server should have depending on how many guests we will like to use?
Thanks again
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If you want to be extra careful, you'll want to get as much ram as you need for all the guests, and the host, and probably leave a little free for the host to use as a disk cache.
At least with the kvm setup I have, there is no way to overcommit memory, other than to use the hosts swap. Things are pretty tight memory wise for me, I'm hoping KSM in 2.6.32 will help alleviate matters some what.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 5 lines · On 12/14/2009 05:50 PM, rek2 wrote:
Quoted 4 lines
VIRT includes a lot of shared memory, so it's not a very useful number to look at when trying to gauge how much memory a process is using.
Ok, so then what stats should we look to calculate the amount of memory a server should have depending on how many guests we will like to use?
The -m parameter to qemu, plus some overhead. If you use ksm or ballooning or ssd swap you can overcommit, how much depends on your workload.
Signature · 1 line
error compiling committee.c: too many arguments to function