context switches and responsiveness
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi,
I've recently increased the number of guests (+20%) running on one of my hosts and found that the responsiveness suffered.
Before that, the context switches were hovering around 10K, now they're close to 30K. Or this could just be because I upgraded the host kernel and kvm: host and guests are running 2.6.29.4 + kvm-86
Is there anything that can be done to reduce this?
I thought dynticks would prevent unecessary context switching? As you can see from the dstat sample below, the host can be very quiet at times, and yet the guests are a bit sluggish. (there is no swapping going on in the guests either)
Is 30K context switches normal for a host that's mostly idle?? Here is a sample guest command line:
- -clock dynticks -usbdevice tablet -m 1024 -monitor telnet:127.0.0.1:10005,server,nowait -L ./ -kernel ./bzImage-2.6.29.4
- -append earlyprintk=serial,ttyS0,115200,keep console=ttyS0,115200 root=/dev/vda elevator=noop -nographic -drive file=./vm/root_fs,if=virtio,boot=on -net nic,vlan=14,macaddr=fe:f0:00:00:00:0e,model=virtio -net tap,vlan=14,ifname=tap14,script=no
Cheers
Antoine
- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 14 15 71 0 0 0| 0 0 | 942B 6912B| 0 0 | 20k 29k 11 17 71 0 0 0| 0 160k| 460B 818B| 0 0 | 19k 30k 9 20 71 0 0 0| 0 0 | 270B 567B| 0 0 | 20k 29k 17 16 66 1 0 0|8192B 1776k| 548B 876B| 0 0 | 20k 31k 8 17 75 0 0 0| 0 336k| 927B 1291B| 0 0 | 20k 29k 13 16 71 0 0 0| 0 1088k|1422B 8505B| 0 296k| 20k 31k 23 14 63 0 0 0| 0 0 |3918B 3058B| 0 0 | 19k 33k 11 16 71 0 0 0|8192B 1120k| 132B 534B| 0 0 | 19k 30k 16 18 66 0 0 0| 0 32k| 330B 7662B| 0 0 | 19k 30k 13 18 69 0 0 0| 0 128k| 264B 420B| 0 0 | 19k 31k 16 17 66 0 0 0| 0 240k| 474B 420B| 0 0 | 19k 30k 11 21 68 0 0 0| 0 1648k| 372B 4196B| 0 0 | 20k 32k 13 18 69 0 0 0| 0 96k| 708B 500B| 0 0 | 20k 30k 15 19 66 0 0 0| 0 256k| 66B 322B| 0 0 | 20k 30k 15 22 63 0 0 0| 0 32k| 804B 8906B| 0 0 | 20k 31k 14 19 67 0 0 0|8192B 256k| 838B 590B|4096B 0 | 19k 31k 13 16 71 0 0 0| 0 0 | 132B 534B| 0 0 | 20k 31k 16 18 62 4 0 0| 312k 976k| 396B 7872B| 156k 0 | 20k 32k 13 22 65 0 0 0| 0 352k| 573B 534B| 0 0 | 19k 31k 14 20 67 0 0 0| 0 160k| 304B 696B| 0 0 | 19k 31k 12 12 74 0 1 0| 0 512k| 514B 4268B| 0 0 | 19k 31k 10 19 72 0 0 0| 0 152k| 290B 420B| 0 0 | 19k 31k 14 18 67 2 0 0| 16k 232k| 224B 306B| 0 0 | 19k 31k 20 23 57 0 0 0| 0 80k|1898B 5544B| 0 0 | 20k 32k 14 14 72 0 0 0| 0 112k|2286B 2482B| 0 0 | 20k 32k 18 17 65 0 0 0| 0 176k|1057B 1169B| 0 0 | 20k 31k 15 13 72 0 0 0| 0 144k| 422B 7614B| 0 0 | 19k 31k 11 15 72 0 0 0| 0 944k| 448B 436B| 0 0 | 19k 31k 9 18 73 0 0 0| 0 1584k| 266B 348B| 0 0 | 19k 33k 15 17 68 0 0 0| 0 0 | 721B 7646B| 0 0 | 20k 31k 14 20 66 0 0 0| 0 0 | 356B 420B| 0 0 | 20k 31k 16 15 69 0 0 0| 0 432k| 532B 420B| 0 0 | 19k 31k 14 18 68 0 0 0| 0 144k| 628B 4058B| 0 0 | 19k 30k 10 20 71 0 0 0| 0 160k| 316B 486B| 0 0 | 20k 31k 11 17 72 0 0 0| 0 80k| 266B 462B| 0 0 | 20k 31k
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkobqSwACgkQGK2zHPGK1rumkgCfVkBuS45ouLCCPXV2xSUjeTUd 9FkAnjxl5sufeL+aYUZsuw4sHLEN7UMt
=wAwM
-----END PGP SIGNATURE-----
Quoted 8 lines · Antoine Martin
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512Hi,
I've recently increased the number of guests (+20%) running on one of my hosts and found that the responsiveness suffered.
How many guests? How many vcpus?
Quoted 6 lines
Before that, the context switches were hovering around 10K, now they're close to 30K. Or this could just be because I upgraded the host kernel and kvm: host and guests are running 2.6.29.4 + kvm-86
Is there anything that can be done to reduce this?
I thought dynticks would prevent unecessary context switching?
dynticks is the default IIRC. What type guests are you running?
If Linux, make sure the guests are dynticks as well.
Quoted 5 lines
As you can see from the dstat sample below, the host can be very quiet at times, and yet the guests are a bit sluggish. (there is no swapping going on in the guests either)
Is 30K context switches normal for a host that's mostly idle??
Depends on guest count! Please provide kvm_stat output.
Quoted 5 lines
Here is a sample guest command line:
- -clock dynticks -usbdevice tablet -m 1024 -monitor telnet:127.0.0.1:10005,server,nowait -L ./ -kernel ./bzImage-2.6.29.4
- -append earlyprintk=serial,ttyS0,115200,keep console=ttyS0,115200
Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.
Quoted 5 lines
- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 14 15 71 0 0 0| 0 0 | 942B 6912B| 0 0 | 20k 29k 11 17 71 0 0 0| 0 160k| 460B 818B| 0 0 | 19k 30k
Interrupt rate is close to context switch rate, which is good.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: context switches and responsiveness [SOLVED]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 2 lines
Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.
Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.
Quoted 4 lines
Quoted 4 lines
- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 14 15 71 0 0 0| 0 0 | 942B 6912B| 0 0 | 20k 29k 11 17 71 0 0 0| 0 160k| 460B 818B| 0 0 | 19k 30k
New (healthy) dstat with 10 guests running:
- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 2 1 92 5 0 0| 0 80k| 87k 88k| 0 0 |1222 2567 1 1 98 0 0 0| 0 0 | 89k 89k| 0 0 |1230 2618
What a difference this tablet makes!!
Antoine
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkobuFcACgkQGK2zHPGK1rtkKQCeIHwkjl+oue/ijeLKCZElguVd IRUAn3telEboBCeT75zOLZo3yDqsRp0Z
=Veq+
-----END PGP SIGNATURE-----
Re: context switches and responsiveness [SOLVED]
Quoted 11 lines · Antoine Martin
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512Quoted 3 lines
Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.
Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.
Why do you need it? vnc? Switching to a better vnc client (I use vinagre) may allow you to drop it.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: context switches and responsiveness [SOLVED]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Antoine Martin
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512Quoted 3 lines
Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.
Quoted 6 lines
Quoted 4 lines
Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.Why do you need it? vnc?
Yes.
Quoted 2 lines
Switching to a better vnc client (I use
vinagre) may allow you to drop it.
I thought I had tried them all in desperation (mouse going out of sync), I'll try again.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkobuTcACgkQGK2zHPGK1rvI7wCfd0d7DE5ZYnOzu76LjLjc5Kse YlMAn0Z5KPe9FsRKUrk15aLVW++JVOcz
=ysC7
-----END PGP SIGNATURE-----
Re: context switches and responsiveness [SOLVED]
Quoted 6 lines · Antoine Martin
Quoted 3 lines
Switching to a better vnc client (I use
vinagre) may allow you to drop it.I thought I had tried them all in desperation (mouse going out of sync), I'll try again.
Vinagre does mouse capture like SDL. It isn't perfect, but much better than the standard vnc client.
Signature · 1 line
error compiling committee.c: too many arguments to function