Skip to contentSkip to conversation

KVM got slow after adding more physical memory to host

DISCUSSION Nikola Ciprich 4 participants 13 messages Sep 6, 2011 – Sep 8, 2011 KVM

lore
Share & tools
lore
#0 lore raw

Hello,

I'm trying to solve quite a weird problem on one of our customers' box.
It's quad core Xeon X3430 @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to 2.6.39).
After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just one windows or linux guest causes host to be quite slow, and quest seems to be even slower. hardware virtualisation is on, KVM is used, no relevant update since the update. here's about 3 seconds of kvm_stat:

 kvm_exit                                    659513  103922
 kvm_entry                                   659505  103920
 kvm_emulate_insn                            631118   99344
 kvm_exit(IO_INSTRUCTION)                    625223   98398
 kvm_userspace_exit                          624979   98359
 kvm_pio                                     624970   98357

but according to atop, there is almost no I/O activity from guests, and running top in them shows them to really be idle...
but they eat like 60-70% of host CPU core time each..

Do You have any idea on what should I check?
thanks a lot in advance

nik

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#1 lore raw
Quoted 10 lines · Nikola Ciprich

Hello,

I'm trying to solve quite a weird problem on one of our customers' box.
It's quad core Xeon X3430 @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to 2.6.39).
After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just one windows or linux guest causes host to be quite slow, and quest seems to be even slower. hardware virtualisation is on, KVM is used, no relevant update since the update. here's about 3 seconds of kvm_stat:

Quoted 6 lines
  kvm_exit                                    659513  103922
  kvm_entry                                   659505  103920
  kvm_emulate_insn                            631118   99344
  kvm_exit(IO_INSTRUCTION)                    625223   98398
  kvm_userspace_exit                          624979   98359
  kvm_pio                                     624970   98357

How many guests in there?

Please post a trace as per http://www.linux-kvm.org/page/Tracing.

Quoted 6 lines

but according to atop, there is almost no I/O activity from guests, and running top in them shows them to really be idle...
but they eat like 60-70% of host CPU core time each..

Do You have any idea on what should I check?
thanks a lot in advance

Please post /proc/mtrr and /proc/iomem.

Signature · 1 line
error compiling committee.c: too many arguments to function
#2 lore raw

Hello Avi,
thanks for quick reply!

Quoted 1 line

How many guests in there?

two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.

Quoted 2 lines

Please post a trace as per http://www.linux-kvm.org/page/Tracing.

here it goes:
http://nelide.cz/nik/trace.dat.bz2

Quoted 2 lines

Please post /proc/mtrr and /proc/iomem.

[root@virtualbox ~]# cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable

[root@virtualbox ~]# cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009d7ff : System RAM
0009d800-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000cffff : pnp 00:0f
000d0000-000dffff : PCI Bus 0000:00
000e0000-000fffff : reserved
00100000-bf78ffff : System RAM
01000000-013917ae : Kernel code
013917af-016c88ff : Kernel data
0176a000-0182ffff : Kernel bss
bf790000-bf79dfff : RAM buffer
bf79e000-bf79ffff : reserved
bf7a0000-bf7adfff : ACPI Tables
bf7ae000-bf7cffff : ACPI Non-volatile Storage
bf7d0000-bf7dffff : reserved
bf7e0000-bf7ecfff : RAM buffer
bf7ed000-bfffffff : reserved
c0000000-dfffffff : PCI Bus 0000:00
c0000000-c01fffff : PCI Bus 0000:08
c0200000-c03fffff : PCI Bus 0000:07
c0400000-c05fffff : PCI Bus 0000:06
c0600000-c07fffff : PCI Bus 0000:06
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : reserved

    e0000000-efffffff : pnp 00:0e
f0000000-fed8ffff : PCI Bus 0000:00
  fa000000-faffffff : PCI Bus 0000:09
    fa000000-faffffff : 0000:09:03.0
  fb4f6000-fb4f60ff : 0000:00:1f.3
  fb4f8000-fb4f87ff : 0000:00:1f.2
    fb4f8000-fb4f87ff : ahci
  fb4fa000-fb4fa3ff : 0000:00:1d.0
    fb4fa000-fb4fa3ff : ehci_hcd
  fb4fc000-fb4fc3ff : 0000:00:1a.0
    fb4fc000-fb4fc3ff : ehci_hcd
  fb500000-fb5fffff : PCI Bus 0000:07
    fb5dc000-fb5dffff : 0000:07:00.0
      fb5dc000-fb5dffff : e1000e
    fb5e0000-fb5fffff : 0000:07:00.0
      fb5e0000-fb5fffff : e1000e
  fb600000-fb6fffff : PCI Bus 0000:08
    fb6dc000-fb6dffff : 0000:08:00.0
      fb6dc000-fb6dffff : e1000e
    fb6e0000-fb6fffff : 0000:08:00.0
      fb6e0000-fb6fffff : e1000e
  fb700000-fbffffff : PCI Bus 0000:09
    fb7fc000-fb7fffff : 0000:09:03.0
    fb800000-fbffffff : 0000:09:03.0
  fc000000-fcffffff : pnp 00:01
  fd000000-fdffffff : pnp 00:01
  fe000000-febfffff : pnp 00:01
  fec00000-fec003ff : IOAPIC 0
  fed00000-fed003ff : HPET 0
  fed14000-fed19fff : pnp 00:01
  fed1c000-fed1ffff : pnp 00:0b
  fed20000-fed3ffff : reserved
    fed20000-fed3ffff : pnp 00:0b
  fed40000-fed8ffff : pnp 00:0b
fee00000-fee00fff : Local APIC
  fee00000-fee00fff : reserved
    fee00000-fee00fff : pnp 00:0d

100000000-53fffffff : System RAM

n.

Quoted 4 lines
Signature · 2 lines
-- 
error compiling committee.c: too many arguments to function
Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#3 lore raw
Quoted 10 lines · Nikola Ciprich

Hello Avi,
thanks for quick reply!

Quoted 1 line

How many guests in there?

two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.

Quoted 2 lines

Please post a trace as per http://www.linux-kvm.org/page/Tracing.

here it goes:
http://nelide.cz/nik/trace.dat.bz2

404

Quoted 10 lines
Quoted 3 lines

Please post /proc/mtrr and /proc/iomem.

[root@virtualbox ~]# cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable

These seem fine - all your memory is write-back.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 1 line

404

ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.

Quoted 17 lines
Quoted 11 lines
Quoted 4 lines

Please post /proc/mtrr and /proc/iomem.

[root@virtualbox ~]# cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable

These seem fine - all your memory is write-back.

Signature · 1 line
error compiling committee.c: too many arguments to function
Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#5 lore raw
Quoted 6 lines · Nikola Ciprich
Quoted 1 line

404

ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.

        qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:     
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803170: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:   reason 
KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803189: kvm_exit:             reason 

IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

        qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:     
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803192: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:   reason 
KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803198: kvm_exit:             reason 

IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.

Gleb, is rep/outsw our pio regression or was it another instruction?

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw
Quoted 26 lines · Avi Kivity
Quoted 6 lines · Nikola Ciprich
Quoted 1 line

404

ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.

        qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:     
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803170: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:   
reason KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803189: kvm_exit:             

reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

        qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:     
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803192: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:   
reason KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803198: kvm_exit:             
Quoted 4 lines

reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.

Even better, switch it to virtio.

Signature · 1 line
error compiling committee.c: too many arguments to function
#12 lore raw
In reply to Nikola Ciprich
Quoted 6 lines · Nikola Ciprich

Hello Avi,
thanks for quick reply!

Quoted 1 line

How many guests in there?

two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.

I don't really understand why the box is unresponsive. What do top and 'vmstat 1' say?

Signature · 1 line
error compiling committee.c: too many arguments to function
#7 lore raw
In reply to Avi Kivity
Quoted 26 lines · Avi Kivity
Quoted 6 lines · Nikola Ciprich
Quoted 1 line

404

ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.

        qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803170: kvm_pio:
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:
reason KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803189: kvm_exit:

reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

        qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:
0:806edb12: rep outsw
        qemu-kvm-4618  [000] 13698.803192: kvm_pio:
pio_write at 0x1f0 size 2 count 1
        qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:
reason KVM_EXIT_IO (2)
        qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
        qemu-kvm-4618  [000] 13698.803198: kvm_exit:
Quoted 7 lines

reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.

Gleb, is rep/outsw our pio regression or was it another instruction?

It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.

--
Gleb.

#8 lore raw

OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled and how can I reenable it, I'm not experienced that much with XP :(

Quoted 2 lines

It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.

now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??

Quoted 4 lines

--
Gleb.

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#9 lore raw
Quoted 2 lines · Nikola Ciprich

OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled

It can happen due to a timeout, perhaps a temporary load spike.

Quoted 1 line

and how can I reenable it,

Google knows.

Quoted 12 lines

I'm not experienced that much with XP :(

Quoted 2 lines

It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.

now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??

I don't think it's related to the memory.

Signature · 1 line
error compiling committee.c: too many arguments to function
#10 lore raw

Re: KVM got slow after adding more physical memory to host - SOLVED

Hello guys,

thanks to both of You for Your replies. The problem is solved, exactly as Avi said, the DMA in windows got somehow disabled. So this certainly was not related to adding the memory...

anyways, note for further generations:
in windows XP, the DMA usage can be checked in
Device Manager->IDE ATA/ATAPI Controllers -> Properties->Advanced Settings Current Transfer Mode must be Multi-Word DMA2 or something similar, NOT PIO!

The way I enabled this, was to uninstall both primary and secondary controller in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!) after reboot, controllers got detected and installed again, with DMA properly enabled. Note that when controller is in PIO mode, this is really a patience test, switching to DMA took me like half an hour to complete, so slow the system was :-/

So thanks a lot for Your help once more.

have a nice day.

nik

Quoted 33 lines · Avi Kivity
Quoted 2 lines · Nikola Ciprich

OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled

It can happen due to a timeout, perhaps a temporary load spike.

Quoted 1 line

and how can I reenable it,

Google knows.

Quoted 12 lines

I'm not experienced that much with XP :(

Quoted 2 lines

It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.

now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??

I don't think it's related to the memory.

Signature · 6 lines
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#11 lore raw

Re: KVM got slow after adding more physical memory to host - SOLVED

Quoted 12 lines · Nikola Ciprich

Hello guys,

thanks to both of You for Your replies. The problem is solved, exactly as Avi said, the DMA in windows got somehow disabled. So this certainly was not related to adding the memory...

anyways, note for further generations:
in windows XP, the DMA usage can be checked in
Device Manager->IDE ATA/ATAPI Controllers -> Properties->Advanced Settings Current Transfer Mode must be Multi-Word DMA2 or something similar, NOT PIO!

Quoted 5 lines

The way I enabled this, was to uninstall both primary and secondary controller in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!) after reboot, controllers got detected and installed again, with DMA properly enabled. Note that when controller is in PIO mode, this is really a patience test, switching to DMA took me like half an hour to complete, so slow the system was :-/

You can open regedit and search for "NoIDE" - it should find one key with that name, with value "1" or "yes" - just delete it, there's no need to go that route with removing/reinstalling device drivers.

The reason why it has been disabled is - most likely - due to some timeout while handling i/o -- eg, when your host was loaded too much or were swapping - you should be able to find something about that in windows event log.

/mjt