Disk activity issue...
I've got this issue that I've been banging my head against a wall for a while over and I think another pair of eyes may help, if anyone have a moment. We have this new-ish KVM VM server (with the latest CentOS 5.5 updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs. It works mostly as expected except it has a very high load all the time, like 40-60, when the VMs are running. I suspect it has to do with memory management, because when all 3 VMs are online, they should consume 5GB RAM on the VM server and they only consume like 2GB, so I think the rest of the RAM is swapping or something, because the disks are spinning at 100% all the time (even when the VMs are doing nothing). Although, the VM server does not report any swapping happening. When I shut down the VMs one by one, the load drops and so does the disk activity. I don't think I set this server up with anything out of the ordinary... I've tried rebooting, but the same thing happens immediately upon reboot. Google searches, for me at least, yielded nothing useful.
Anyone heard of this kind of thing happening?
-erich
Quoted 7 lines · Erich Weiler
I've got this issue that I've been banging my head against a wall for a while over and I think another pair of eyes may help, if anyone have a moment. We have this new-ish KVM VM server (with the latest CentOS 5.5 updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs. It works mostly as expected except it has a very high load all the time, like 40-60, when the VMs are running. I suspect it has to do with memory management, because when all 3 VMs are online, they should consume 5GB RAM on the VM server and
Quoted 8 lines
they only consume like 2GB, so I think the rest of the RAM is swapping or something, because the disks are spinning at 100% all the time (even when the VMs are doing nothing). Although, the VM server does not report any swapping happening. When I shut down the VMs one by one, the load drops and so does the disk activity. I don't think I set this server up with anything out of the ordinary... I've tried rebooting, but the same thing happens immediately upon reboot. Google searches, for me at least, yielded nothing useful.
"very high load all the time, like 40-60"
Is this number the host CPU utilization or load average?
Which guest OS (and versions) are you running?
Can you paste the qemu-kvm command-line for the VMs?
Can you send a few lines of "vmstat 5" output on the host while running the 3 VMs?
Stefan
Hi Stefan,
Thanks for replying! I was able to figure it out - it was not the fault of KVM. One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one. It was just a needle in a haystack. ;)
Thanks again!
Quoted 28 lines · Stefan Hajnoczi
Quoted 7 lines · Erich Weiler
I've got this issue that I've been banging my head against a wall for a while over and I think another pair of eyes may help, if anyone have a moment. We have this new-ish KVM VM server (with the latest CentOS 5.5 updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs. It works mostly as expected except it has a very high load all the time, like 40-60, when the VMs are running. I suspect it has to do with memory management, because when all 3 VMs are online, they should consume 5GB RAM on the VM server and
Quoted 8 lines
they only consume like 2GB, so I think the rest of the RAM is swapping or something, because the disks are spinning at 100% all the time (even when the VMs are doing nothing). Although, the VM server does not report any swapping happening. When I shut down the VMs one by one, the load drops and so does the disk activity. I don't think I set this server up with anything out of the ordinary... I've tried rebooting, but the same thing happens immediately upon reboot. Google searches, for me at least, yielded nothing useful.
"very high load all the time, like 40-60"
Is this number the host CPU utilization or load average?Which guest OS (and versions) are you running?
Can you paste the qemu-kvm command-line for the VMs?
Can you send a few lines of "vmstat 5" output on the host while running the 3 VMs?
Stefan
Quoted 1 line · Erich Weiler
Thanks for replying! I was able to figure it out - it was not the fault of KVM. One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one. It was just a needle in a haystack. ;)
i use iotop to check which VM is hitting the disk
Signature · 1 line
Javier
Yeah, some other folks have suggested that too, but it looks like you need kernel version 2.6.18-199 or higher to run iotop, and CentOS 5.5 only has 2.6.18-192 or something... Guess I need to wait for the kernel to be updated there if I want to use iotop, which sounds like a very cool utility.
-erich
Quoted 7 lines · Javier Guerra Giraldez
Quoted 1 line · Erich Weiler
Thanks for replying! I was able to figure it out - it was not the fault of KVM. One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one. It was just a needle in a haystack. ;)
i use iotop to check which VM is hitting the disk
Signature · 1 line
Javier