XP guests sluggish after around 1 week of uptime

If this is the wrong place to ask this question, please let me know.

I’m running kvm-80 on ubuntu 8.04 x64 kernel 2.6.24-22-server, md raid 5 with LVM. When my host is first booted my windows XP guest runs fine, but after around a week or so of uptime, they start to run very sluggish. It takes around a minute just to open the start menu.

I'm starting to wonder if it is a problem with kvm (qemu) disk performance. The Read/Write speeds seems to be half as fast as my VMWare XP guest. But not sure why this would happen after running fine for around a week.

I have the XP guest set to use 2GB of ram and disabled the paging file but that still doesn’t help.

Any advice on troubleshooting this would be appreciated.

Thanks,

-J

Hi Jay,

My host (Intel Core 2 Duo 2.4GHz, 2GB) has successfully run windows guests for more than one month. I had to shutdown them to have kvm updated. What kind of video output do you use? I have no problem with VNC so far. Have you kept the video viewer opened for this week? If you have file servers (such as FTP) on this guest, do they respond quickly while the display is as slow as you're described?

Regards,

Florent

Hi Florent,
Thanks for the response.

I also use VNC, and i usually leave it open the whole time on the host and then just remote desktop in from different boxes (work, laptop)

I do have a windows file share i can test next time it slows down. I'll let you know what happends.

-J

How much memory do you have on the host? Is the host swapping? Can you post 'vmstat 1' output while Windows is opening the start menu?

error compiling committee.c: too many arguments to function

I have 8GB RAM on the host, and i've run htop and the swap space is not being used at all. I have a total of 4 guest (2 windows + 2 linux)

I will have to try the 'vmstat 1' command tonight when i get home from work. My work has blocked my ssh access to my home PC a few months ago so i started using proxyTunnel over https which worked fine, but somehow they figured out how to block that today :(

Maybe it's time to find a new job.

-J

Please keep the cc list when replying (use reply-all).

Provide kvm_stat output as well.

error compiling committee.c: too many arguments to function

The guest is not acting up right now since I just rebooted the host on Sunday.
Here is the current outputs of the command, and I will post them again when they start acting up:

[root@e6400:~/ 500]$vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
1 0 212 95816 992968 1775292 0 0 464 280 61 28 45 6 46
2
0 0 212 95608 992968 1775312 0 0 0 120 98 8179 8 8 84
0
0 0 212 95592 992968 1775312 0 0 0 0 18 8099 7 8 86
0
1 0 212 95716 992968 1775312 0 0 0 0 42 8626 4 7 89
0
1 0 212 95716 992968 1775312 0 0 0 56 56 8034 7 11 82
0
0 0 212 95716 992968 1775312 0 0 8 36 59 8317 7 6 86
1
1 0 212 95660 992968 1775324 0 0 0 84 61 8203 11 11 78
0
0 0 212 95304 992968 1775324 0 0 0 48 65 8003 8 7 85
0
1 0 212 95264 992968 1775324 0 0 0 0 13 7704 15 7 78
0
0 0 212 95264 992968 1775324 0 0 0 80 97 8018 15 5 80
0
2 0 212 95468 992968 1775324 0 0 0 16 35 7891 9 8 83
0
3 0 212 95380 992968 1775324 0 0 0 124 103 8179 9 8 81
1
0 0 212 95380 992968 1775324 0 0 0 40 48 8006 11 8 80
1
0 0 212 95336 992972 1775320 0 0 0 116 104 8285 9 5 85
1
0 0 212 95356 992976 1775320 0 0 0 312 173 8421 7 11 79
3
1 0 212 95380 992976 1775324 0 0 0 0 42 8233 5 9 87
0
0 0 212 95460 992976 1775324 0 0 0 72 48 8188 7 11 82
0
1 0 212 95380 992976 1775324 0 0 0 0 39 8221 8 8 84
0
0 0 212 95468 992976 1775324 0 0 0 0 17 7982 10 6 83
0
0 0 212 95460 992976 1775324 0 0 0 100 100 8421 7 6 85
2
1 0 212 95380 992976 1775324 0 0 0 32 35 8000 8 7 85
0

kvm statistics

efer_reload 15 0
exits 3315693540 22034
fpu_reload 174984854 724
halt_exits 216355599 1503
halt_wakeup 55495383 341
host_state_reload 867498430 4631
hypercalls 0 0
insn_emulation 56483585 491
insn_emulation_fail 1025 0
invlpg 7980288 5
io_exits 1796250843 11377
irq_exits 85442292 224
irq_injections 250928733 1661
irq_window 9135944 52

 kvm_request_irq              0       0
 largepages                   0       0
 mmio_exits            28133579     310
 mmu_cache_miss         1041626       0
 mmu_flooded             426300       0
 mmu_pde_zapped          704522       0
 mmu_pte_updated        2702458       0
 mmu_pte_write          3546140       0
 mmu_recycled             16553       0
 mmu_shadow_zapped      1245678       0
 mmu_unsync               11192      -1
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed              65801856      54
 pf_guest              22111366       5
 remote_tlb_flush           235       0
 request_nmi                  0       0
 signal_exits                 5       0

tlb_flush 850332604 6382

-----Original Message-----
From: Avi Kivity [mailto:[email protected]]
Sent: Tuesday, January 13, 2009 4:29 AM
To: Jay
Cc: [email protected]
Subject: Re: XP guests sluggish after around 1 week of uptime

Please keep the cc list when replying (use reply-all).

being

work.

Provide kvm_stat output as well.

error compiling committee.c: too many arguments to function

Ok, it started doing it again, and I actually just rebooted the host on Friday Morning. Here are the outputs:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
0 1 204 49520 1008796 2068728 0 0 21 43 33 67 8 7 81
4
1 1 204 49196 1008824 2069256 0 0 112 712 401 8968 5 12 33
49
0 1 204 47824 1008836 2070616 0 0 100 1716 386 10048 9 11
40 41
0 1 204 47760 1008836 2070640 0 0 4 48 73 8797 9 10 46
34
0 1 204 47588 1008836 2070640 0 0 0 24 66 9007 5 9 42
43
0 1 204 47488 1008836 2070660 0 0 4 104 80 8641 8 10 41
41
0 1 204 47464 1008836 2070668 0 0 0 32 40 8812 8 9 45
39
1 2 204 47240 1008844 2070712 0 0 4 104 71 8843 7 9 21
62
3 1 204 47388 1008844 2070720 0 0 0 16 37 8786 8 7 43
43
0 0 204 47016 1008844 2070796 0 0 4 36 69 8986 10 6 67
17
0 0 204 46940 1008844 2070796 0 0 4 84 43 8720 10 7 48
35
0 1 204 46940 1008844 2070860 0 0 0 88 70 8732 10 8 56
26
0 2 204 46892 1008844 2070860 0 0 0 16 34 8855 5 9 42
45
0 1 204 47164 1008856 2070920 0 0 4 88 64 7857 9 8 39
44
0 1 204 47164 1008860 2070924 0 0 0 12 34 6123 4 3 44
48
0 0 204 47016 1008860 2070924 0 0 4 44 75 8745 11 10 49
30
0 1 204 47064 1008860 2070976 0 0 0 88 34 8840 10 9 51
30
0 1 204 47288 1008860 2070976 0 0 4 76 59 8820 5 11 39
45
0 1 204 46968 1008868 2071044 0 0 4 24 38 6535 2 3 51
44
0 1 204 46668 1008868 2071048 0 0 4 84 57 7942 6 7 41
46
1 1 204 46592 1008868 2071176 0 0 4 84 41 8969 10 8 41
41
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
1 1 204 46792 1008868 2071176 0 0 0 8 56 9162 14 6 43
37
2 1 204 46148 1008876 2071992 0 0 92 1004 229 9344 9 7 44
39
1 1 204 45948 1008876 2072004 0 0 12 80 70 7934 25 9 26
40
2 1 204 46072 1008876 2072068 0 0 4 64 38 8642 8 7 42
42
3 1 204 45776 1008876 2072116 0 0 4 52 57 8655 8 11 42
39
0 0 204 45924 1008876 2072212 0 0 4 36 48 8632 7 7 64
22
0 1 204 45776 1008880 2072208 0 0 0 120 133 8054 10 6 40
44
0 2 204 45700 1008884 2072276 0 0 4 72 114 7845 6 8 35
52
1 1 204 52744 1008892 2065788 0 0 60 924 261 9307 8 8 30
54
3 1 204 50872 1008904 2066632 0 0 128 1996 571 10162 13 17
32 38
0 1 204 50548 1008928 2067232 0 0 332 896 343 7798 13 14 38
34
0 1 204 50688 1008928 2067340 0 0 8 92 49 7719 16 17 48
18
0 2 204 50564 1008928 2067404 0 0 4 132 185 9020 9 8 48
36
1 1 204 50456 1008928 2067480 0 0 4 36 42 9006 10 8 24
58
1 1 204 50464 1008928 2067480 0 0 0 52 64 7598 22 17 18
42
1 2 204 50284 1008932 2067476 0 0 4 76 50 8743 11 9 22
58
0 1 204 50192 1008936 2067508 0 0 8 12 71 8419 9 12 23
56
0 1 204 50324 1008936 2067512 0 0 0 124 45 7672 26 16 37
21
2 1 204 50440 1008936 2067520 0 0 8 48 83 9076 9 7 51
33
1 1 204 50564 1008936 2067524 0 0 12 24 34 8741 12 11 38
40
0 2 204 50564 1008944 2067532 0 0 0 48 62 8829 10 8 36
46

kvm statistics

efer_reload 17 0
exits 3821966006 18394
fpu_reload 142068215 725
halt_exits 278799576 1487
halt_wakeup 65717572 329
host_state_reload 842507576 4219
hypercalls 0 0
insn_emulation 62556294 206
insn_emulation_fail 1015 0
invlpg 14737923 2
io_exits 1322828696 6784
irq_exits 45319236 249
irq_injections 311333395 1610
irq_window 9206502 35

 kvm_request_irq              0       0
 largepages                   0       0
 mmio_exits            24868562      64
 mmu_cache_miss         1099292       1
 mmu_flooded             481532       0
 mmu_pde_zapped          723088       0
 mmu_pte_updated        5740122       0
 mmu_pte_write          6589738       0
 mmu_recycled             23245       0
 mmu_shadow_zapped      1347124       1
 mmu_unsync               11803       0
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed              95632964     282
 pf_guest              37388330     129
 remote_tlb_flush             7       0
 request_nmi                  0       0
 signal_exits                 5       0

tlb_flush 1533760876 7017

kvm statistics

efer_reload 17 0
exits 3822814952 23106
fpu_reload 142094651 811
halt_exits 278856080 1446
halt_wakeup 65731489 341
host_state_reload 842686149 4763
hypercalls 0 0
insn_emulation 62568224 553
insn_emulation_fail 1015 0
invlpg 14741639 123
io_exits 1323153639 7657
irq_exits 45329372 252
irq_injections 311402755 1701
irq_window 9214599 61

 kvm_request_irq              0       0
 largepages                   0       0
 mmio_exits            24871965     400
 mmu_cache_miss         1100097      10
 mmu_flooded             482025       6
 mmu_pde_zapped          723420       4
 mmu_pte_updated        5740727       8
 mmu_pte_write          6590675      12
 mmu_recycled             23287       0
 mmu_shadow_zapped      1348261      13
 mmu_unsync               11803       1
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed              95663549     732
 pf_guest              37398263     274
 remote_tlb_flush             7       0
 request_nmi                  0       0
 signal_exits                 5       0

tlb_flush 1534079607 10076

kvm statistics

efer_reload 17 0
exits 3819143501 18578
fpu_reload 141974265 694
halt_exits 278594554 1499
halt_wakeup 65669505 337
host_state_reload 841893357 4180
hypercalls 0 0
insn_emulation 62517121 218
insn_emulation_fail 1015 0
invlpg 14729583 80
io_exits 1321824794 6220
irq_exits 45282278 220
irq_injections 311098803 1608
irq_window 9193364 36

 kvm_request_irq              0       0
 largepages                   0       0
 mmio_exits            24856333      54
 mmu_cache_miss         1098433       0
 mmu_flooded             481127       0
 mmu_pde_zapped          722788       0
 mmu_pte_updated        5739239       2
 mmu_pte_write          6588555       2
 mmu_recycled             23224       0
 mmu_shadow_zapped      1346024       1
 mmu_unsync               11790      -2
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed              95573807     218
 pf_guest              37370248      35
 remote_tlb_flush             7       0
 request_nmi                  0       0
 signal_exits                 5       0

tlb_flush 1532611984 7928

Any suggestions would be appreciated, I REALLY don't want to have to go back to vmware.

Thanks in advance,

-J

-----Original Message-----
From: Avi Kivity [mailto:[email protected]]
Sent: Tuesday, January 13, 2009 4:29 AM
To: Jay
Cc: [email protected]
Subject: Re: XP guests sluggish after around 1 week of uptime

Please keep the cc list when replying (use reply-all).

being

work.

Provide kvm_stat output as well.

error compiling committee.c: too many arguments to function

There is a lot of wait time (last column, wrapped around to the first in your output). Not a lot of real I/O though. Can you describe your I/O subsystem? Are you using NFS?

error compiling committee.c: too many arguments to function

(re-add kvm@vger)

Software RAID or hardware RAID?

I think the root cause is the RAID here, not kvm.

Some suggestions:
- use the raw format directly on top of LVM. That is, use -drive file=/dev/volgroup/logvol.
- if you use the raw format, disable disk caching: -drive file=/dev/volgroup/logvol,cache=off
- try non-RAID5 storage

To migrate your volumes, use

qemu-img convert /path/to/image.qcow -O raw /dev/volgroup/logvol

Be sure to size the volume appropriately (same size as the virtual image).

S=D: waiting for disk. Low cpu usage as expected.

This seems fine if a bit low. You'll get better numbers sitting directly atop LVM.

error compiling committee.c: too many arguments to function

Yes, it is software raid, I will try moving the guest images to one of my other disks that is not RAIDed on their own LVM volume.

Does the LVM logical volume size need to be equal the guest disk size exactly?

I will let you know the results.

Thanks again for the help,

-J

It can be a larger. In fact, I recommend making it larger, because if it is smaller even by a small amount, data loss is likely.

error compiling committee.c: too many arguments to function

It's definitely some kind of Disk IO problem, I can't even copy a 700MB file on that guest without waiting a loooong time. Running bonnie on my linux guest shows considerable slow down in disk performance.

Results after a fresh Reboot of the Host:

root@wserver:~# cat bonnieresults.txt
Version 1.03b       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
wserver        1G 10617  23 12241   1 10328   2 47915  93 301061  93  5095
98
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
wserver,1G,10617,23,12241,1,10328,2,47915,93,301061,93,5094.8,98,16,+++++,++
+,                    +++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Results after noticing considerable slow down on other guest machine:

root@wserver:~# cat bonnieresults_SLOW.txt
Version 1.03b       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
wserver        1G  2252   3  3771   0  6844   1 46057  70 355858  50 752.3
7
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
wserver,1G,2252,3,3771,0,6844,1,46057,70,355858,50,752.3,7,16,+++++,+++,++++
+,                    +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

After a reboot of the host, they guest are fine again, but for how long...

Any advice or suggestions?

Thanks,

-J

-----Original Message-----
From: Avi Kivity [mailto:[email protected]]
Sent: Tuesday, January 13, 2009 4:29 AM
To: Jay
Cc: [email protected]
Subject: Re: XP guests sluggish after around 1 week of uptime

Please keep the cc list when replying (use reply-all).

being

work.

Provide kvm_stat output as well.

error compiling committee.c: too many arguments to function

What does 'top' show for the guests? I'm interested in qemu's VIRT and RES columns.

Also, 'vmstat 1' on host and guest while slow copying is happening.

error compiling committee.c: too many arguments to function