Huge buffer size on KVM host
Hi all,
We have noticed a strange thing in one of our setups which uses KVM for virtualization.
The size of the buffer cache turns out to be rather large, here is what 'top' on the host
has to say about it:
Mem: 12274084k total, 12202860k used, 71224k free, 2991728k buffers Swap: 8191992k total, 20232k used, 8171760k free, 44392k cached
As the host machine itself does not run any data intensive services, we assume that
this is due to the guest VM which receives/sends about 200 GB of data per day of which
about 35G have to be stored on disk (disk array connected to the host and used inside
the guest via virtio). We did not run into problems with this setup, it basically runs stable
for more than a year now (the buffer size never went past 30% of the total memory).
However, we are not sure if the huge size of the buffer cache is something one would
expect here. If this is the case, I would really love to know the reason behind it and if
it has indeed something to do with KVM/virtio. Any enlightenment here would make
my day.
Our setup:
--------------
Host:
IBM BladeCenter HS22, 12 GB RAM, Xeon QC CPU CentOS 5.5 (64-bit) running 2.6.18-194.11.1.el5 kvm-83-164.el5_5.25 (qemu 0.9.1, modules 2.6.18.194.17.1) libvirt 0.6.3
Guest:
Ubuntu 10.04 (64-bit) running 2.6.32-27
virtio for storage and network
Best regards,
Signature · 6 lines
-------------------------------------------------------------------- Martin Wawro | Digital Medics GmbH Managing Director | Otto-Hahn-Str. 15, 44227 Dortmund, Germany Tel. +49-231-9742-6622 | Fax: +49-231-9742-6623 Key: 0xB0A225BD | Registered at AG Dortmund, HRB 19360 --------------------------------------------------------------------
Quoted 29 lines · Martin Wawro
Hi all,
We have noticed a strange thing in one of our setups which uses KVM for virtualization.
The size of the buffer cache turns out to be rather large, here is what 'top' on the host
has to say about it:Mem: 12274084k total, 12202860k used, 71224k free, 2991728k buffers Swap: 8191992k total, 20232k used, 8171760k free, 44392k cachedAs the host machine itself does not run any data intensive services, we assume that
this is due to the guest VM which receives/sends about 200 GB of data per day of which
about 35G have to be stored on disk (disk array connected to the host and used inside
the guest via virtio). We did not run into problems with this setup, it basically runs stable
for more than a year now (the buffer size never went past 30% of the total memory).
However, we are not sure if the huge size of the buffer cache is something one would
expect here. If this is the case, I would really love to know the reason behind it and if
it has indeed something to do with KVM/virtio. Any enlightenment here would make
my day.
Quoted 9 lines
Our setup:
--------------Host:
IBM BladeCenter HS22, 12 GB RAM, Xeon QC CPU CentOS 5.5 (64-bit) running 2.6.18-194.11.1.el5 kvm-83-164.el5_5.25 (qemu 0.9.1, modules 2.6.18.194.17.1) libvirt 0.6.3
How is you storage set up? Files (which format?) or logical volumes? what's the cache= setting?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 5 lines
We have noticed a strange thing in one of our setups which uses KVM for virtualization.
The size of the buffer cache turns out to be rather large [,,,]How is you storage set up? Files (which format?) or logical volumes? what's the cache= setting?
We are using logical volumes and the cache is set to 'none'.
With best regards,
Signature · 6 lines
-------------------------------------------------------------------- Martin Wawro | Digital Medics GmbH Managing Director | Otto-Hahn-Str. 15, 44227 Dortmund, Germany Tel. +49-231-9742-6622 | Fax: +49-231-9742-6623 Key: 0xB0A225BD | Registered at AG Dortmund, HRB 19360 --------------------------------------------------------------------
Quoted 9 lines · Martin Wawro
Quoted 6 lines · Avi Kivity
Quoted 5 lines
We have noticed a strange thing in one of our setups which uses KVM for virtualization.
The size of the buffer cache turns out to be rather large [,,,]
Quoted 4 lines
Quoted 2 lines
How is you storage set up? Files (which format?) or logical volumes? what's the cache= setting?
We are using logical volumes and the cache is set to 'none'.
Strange, that should work without any buffering.
What the contents of
/sys/block/sda/queue/hw_sector_size
and
/sys/block/sda/queue/logical_block_size
?
Replace sda by the disk underlying your physical volume.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 2 lines
We are using logical volumes and the cache is set to 'none'.
Strange, that should work without any buffering.
What the contents of
Quoted 8 lines
/sys/block/sda/queue/hw_sector_sizeand
/sys/block/sda/queue/logical_block_size?
Hi Avi,
It seems that the kernel on that particular machine is too old, those entries are not featured. We checked on a comparable setup with a newer kernel and both entries were set to 512.
We also did have a third more thorough look on the caching. It turns out that the virt-manager does not seem to honor the caching adjusted in the GUI correctly. We disabled caching on all virtual devices for this particular VM and checking with "ps -fxal" revealed, that only one of those devices (and a rather small one too) had this set. We corrected this in the XML file directly and the buffer size currently resides at around 1.8 GB after rebooting the VM (the only virtio device not having the cache=none option set is now the (non-mounted) cdrom).
Best regards,
Signature · 6 lines
-------------------------------------------------------------------- Martin Wawro | Digital Medics GmbH Managing Director | Otto-Hahn-Str. 15, 44227 Dortmund, Germany Tel. +49-231-9742-6622 | Fax: +49-231-9742-6623 Key: 0xB0A225BD | Registered at AG Dortmund, HRB 19360 --------------------------------------------------------------------
Quoted 25 lines · Martin Wawro
Quoted 7 lines · Avi Kivity
Quoted 2 lines
We are using logical volumes and the cache is set to 'none'.
Strange, that should work without any buffering.
What the contents of
Quoted 8 lines
/sys/block/sda/queue/hw_sector_sizeand
/sys/block/sda/queue/logical_block_size?
Hi Avi,
It seems that the kernel on that particular machine is too old, those entries are not featured. We checked on a comparable setup with a newer kernel and both entries were set to 512.
Quoted 8 lines
We also did have a third more thorough look on the caching. It turns out that the virt-manager does not seem to honor the caching adjusted in the GUI correctly. We disabled caching on all virtual devices for this particular VM and checking with "ps -fxal" revealed, that only one of those devices (and a rather small one too) had this set. We corrected this in the XML file directly and the buffer size currently resides at around 1.8 GB after rebooting the VM (the only virtio device not having the cache=none option set is now the (non-mounted) cdrom).
cc += libvirt-list
Is there a reason that cdroms don't get cache=none?
Signature · 1 line
error compiling committee.c: too many arguments to function
On 08/16/2012 04:57 PM, Avi Kivity wrote:
Hi Avi,
No, there was no reason and we disabled it there too. Interestingly, the buffer
size did not go down significantly, even when manually flushing the pages using /proc/sys/vm/drop_caches (3), the buffer size did not go down. Finally,
after rebooting the host, buffer size was back to normal again and when all caches were disabled, we were also able to manually drop the buffered pages.
However, after 2 weeks or so, we encountered significant instabilities in the system.
Inside the guest OS, the load went up like crazy (way past 30) and the guest
was virtually unusable. Checking the host: buffer size, load, memory, I/O was all in
a well defined range.
When rebooting the guest OS, the problem manifested again after about 20-30mins (reproducible a couple of times). The only thing that worked to
stop this (I am afraid only for a couple of weeks or so), was rebooting the host system.
On the guest and the host there was nothing suspicious in the logs or dmesg output.
Best regards,
Martin
Quoted 19 lines
Hi Avi,
It seems that the kernel on that particular machine is too old, those entries are not featured. We checked on a comparable setup with a newer kernel and both entries were set to 512.
We also did have a third more thorough look on the caching. It turns out that the virt-manager does not seem to honor the caching adjusted in the GUI correctly. We disabled caching on all virtual devices for this particular VM and checking with "ps -fxal" revealed, that only one of those devices (and a rather small one too) had this set. We corrected this in the XML file directly and the buffer size currently resides at around 1.8 GB after rebooting the VM (the only virtio device not having the cache=none option set is now the (non-mounted) cdrom).
cc += libvirt-list
Is there a reason that cdroms don't get cache=none?
Signature · 6 lines
-------------------------------------------------------------------- Martin Wawro | Digital Medics GmbH Managing Director | Otto-Hahn-Str. 15, 44227 Dortmund, Germany Tel. +49-231-9742-6622 | Fax: +49-231-9742-6623 Key: 0xB0A225BD | Registered at AG Dortmund, HRB 19360 --------------------------------------------------------------------