Skip to contentSkip to conversation

context switches and responsiveness

DISCUSSION Antoine Martin 2 participants 6 messages May 26, 2009 KVM

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I've recently increased the number of guests (+20%) running on one of my hosts and found that the responsiveness suffered.
Before that, the context switches were hovering around 10K, now they're close to 30K. Or this could just be because I upgraded the host kernel and kvm: host and guests are running 2.6.29.4 + kvm-86
Is there anything that can be done to reduce this?
I thought dynticks would prevent unecessary context switching? As you can see from the dstat sample below, the host can be very quiet at times, and yet the guests are a bit sluggish. (there is no swapping going on in the guests either)
Is 30K context switches normal for a host that's mostly idle?? Here is a sample guest command line:
- -clock dynticks -usbdevice tablet -m 1024 -monitor telnet:127.0.0.1:10005,server,nowait -L ./ -kernel ./bzImage-2.6.29.4
- -append earlyprintk=serial,ttyS0,115200,keep console=ttyS0,115200 root=/dev/vda elevator=noop -nographic -drive file=./vm/root_fs,if=virtio,boot=on -net nic,vlan=14,macaddr=fe:f0:00:00:00:0e,model=virtio -net tap,vlan=14,ifname=tap14,script=no

Cheers
Antoine

- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 14  15  71   0   0   0|   0     0 | 942B 6912B|   0     0 |  20k   29k
 11  17  71   0   0   0|   0   160k| 460B  818B|   0     0 |  19k   30k
  9  20  71   0   0   0|   0     0 | 270B  567B|   0     0 |  20k   29k
 17  16  66   1   0   0|8192B 1776k| 548B  876B|   0     0 |  20k   31k
  8  17  75   0   0   0|   0   336k| 927B 1291B|   0     0 |  20k   29k
 13  16  71   0   0   0|   0  1088k|1422B 8505B|   0   296k|  20k   31k
 23  14  63   0   0   0|   0     0 |3918B 3058B|   0     0 |  19k   33k
 11  16  71   0   0   0|8192B 1120k| 132B  534B|   0     0 |  19k   30k
 16  18  66   0   0   0|   0    32k| 330B 7662B|   0     0 |  19k   30k
 13  18  69   0   0   0|   0   128k| 264B  420B|   0     0 |  19k   31k
 16  17  66   0   0   0|   0   240k| 474B  420B|   0     0 |  19k   30k
 11  21  68   0   0   0|   0  1648k| 372B 4196B|   0     0 |  20k   32k
 13  18  69   0   0   0|   0    96k| 708B  500B|   0     0 |  20k   30k
 15  19  66   0   0   0|   0   256k|  66B  322B|   0     0 |  20k   30k
 15  22  63   0   0   0|   0    32k| 804B 8906B|   0     0 |  20k   31k
 14  19  67   0   0   0|8192B  256k| 838B  590B|4096B    0 |  19k   31k
 13  16  71   0   0   0|   0     0 | 132B  534B|   0     0 |  20k   31k
 16  18  62   4   0   0| 312k  976k| 396B 7872B| 156k    0 |  20k   32k
 13  22  65   0   0   0|   0   352k| 573B  534B|   0     0 |  19k   31k
 14  20  67   0   0   0|   0   160k| 304B  696B|   0     0 |  19k   31k
 12  12  74   0   1   0|   0   512k| 514B 4268B|   0     0 |  19k   31k
 10  19  72   0   0   0|   0   152k| 290B  420B|   0     0 |  19k   31k
 14  18  67   2   0   0|  16k  232k| 224B  306B|   0     0 |  19k   31k
 20  23  57   0   0   0|   0    80k|1898B 5544B|   0     0 |  20k   32k
 14  14  72   0   0   0|   0   112k|2286B 2482B|   0     0 |  20k   32k
 18  17  65   0   0   0|   0   176k|1057B 1169B|   0     0 |  20k   31k
 15  13  72   0   0   0|   0   144k| 422B 7614B|   0     0 |  19k   31k
 11  15  72   0   0   0|   0   944k| 448B  436B|   0     0 |  19k   31k
  9  18  73   0   0   0|   0  1584k| 266B  348B|   0     0 |  19k   33k
 15  17  68   0   0   0|   0     0 | 721B 7646B|   0     0 |  20k   31k
 14  20  66   0   0   0|   0     0 | 356B  420B|   0     0 |  20k   31k
 16  15  69   0   0   0|   0   432k| 532B  420B|   0     0 |  19k   31k
 14  18  68   0   0   0|   0   144k| 628B 4058B|   0     0 |  19k   30k
 10  20  71   0   0   0|   0   160k| 316B  486B|   0     0 |  20k   31k
 11  17  72   0   0   0|   0    80k| 266B  462B|   0     0 |  20k   31k

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREKAAYFAkobqSwACgkQGK2zHPGK1rumkgCfVkBuS45ouLCCPXV2xSUjeTUd 9FkAnjxl5sufeL+aYUZsuw4sHLEN7UMt
=wAwM
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 8 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I've recently increased the number of guests (+20%) running on one of my hosts and found that the responsiveness suffered.

How many guests? How many vcpus?

Quoted 6 lines

Before that, the context switches were hovering around 10K, now they're close to 30K. Or this could just be because I upgraded the host kernel and kvm: host and guests are running 2.6.29.4 + kvm-86
Is there anything that can be done to reduce this?
I thought dynticks would prevent unecessary context switching?

dynticks is the default IIRC. What type guests are you running?

If Linux, make sure the guests are dynticks as well.

Quoted 5 lines

As you can see from the dstat sample below, the host can be very quiet at times, and yet the guests are a bit sluggish. (there is no swapping going on in the guests either)
Is 30K context switches normal for a host that's mostly idle??

Depends on guest count! Please provide kvm_stat output.

Quoted 5 lines

Here is a sample guest command line:
- -clock dynticks -usbdevice tablet -m 1024 -monitor telnet:127.0.0.1:10005,server,nowait -L ./ -kernel ./bzImage-2.6.29.4
- -append earlyprintk=serial,ttyS0,115200,keep console=ttyS0,115200

Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.

Quoted 5 lines

- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 14  15  71   0   0   0|   0     0 | 942B 6912B|   0     0 |  20k   29k
 11  17  71   0   0   0|   0   160k| 460B  818B|   0     0 |  19k   30k

Interrupt rate is close to context switch rate, which is good.

Signature · 1 line
error compiling committee.c: too many arguments to function
#2 lore raw

Re: context switches and responsiveness [SOLVED]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 2 lines

Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.

Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.

Quoted 4 lines
Quoted 4 lines

- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 14  15  71   0   0   0|   0     0 | 942B 6912B|   0     0 |  20k   29k
 11  17  71   0   0   0|   0   160k| 460B  818B|   0     0 |  19k   30k

New (healthy) dstat with 10 guests running:
- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  2   1  92   5   0   0|   0    80k|  87k   88k|   0     0 |1222  2567
  1   1  98   0   0   0|   0     0 |  89k   89k|   0     0 |1230  2618

What a difference this tablet makes!!

Antoine
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREKAAYFAkobuFcACgkQGK2zHPGK1rtkKQCeIHwkjl+oue/ijeLKCZElguVd IRUAn3telEboBCeT75zOLZo3yDqsRp0Z
=Veq+
-----END PGP SIGNATURE-----

#3 lore raw

Re: context switches and responsiveness [SOLVED]

Quoted 11 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 3 lines

Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.

Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.

Why do you need it? vnc? Switching to a better vnc client (I use vinagre) may allow you to drop it.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw

Re: context switches and responsiveness [SOLVED]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 8 lines · Avi Kivity
Quoted 7 lines · Antoine Martin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 3 lines

Ah, -usbdevice tablet, interrupt generator from hell. Let it go and you'll see your context switch rate drop.

Quoted 6 lines
Quoted 4 lines

Indeed that solved it!
(only kept it for the one windows guest that really needs it) Many many thanks.

Why do you need it? vnc?

Yes.

Quoted 2 lines

Switching to a better vnc client (I use
vinagre) may allow you to drop it.

I thought I had tried them all in desperation (mouse going out of sync), I'll try again.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREKAAYFAkobuTcACgkQGK2zHPGK1rvI7wCfd0d7DE5ZYnOzu76LjLjc5Kse YlMAn0Z5KPe9FsRKUrk15aLVW++JVOcz
=ysC7
-----END PGP SIGNATURE-----

#5 lore raw

Re: context switches and responsiveness [SOLVED]

Quoted 6 lines · Antoine Martin
Quoted 3 lines

Switching to a better vnc client (I use
vinagre) may allow you to drop it.

I thought I had tried them all in desperation (mouse going out of sync), I'll try again.

Vinagre does mouse capture like SDL. It isn't perfect, but much better than the standard vnc client.

Signature · 1 line
error compiling committee.c: too many arguments to function