KVM got slow after adding more physical memory to host
Hello,
I'm trying to solve quite a weird problem on one of our customers' box.
It's quad core Xeon X3430 @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to 2.6.39).
After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just one windows or linux guest causes host to be quite slow, and quest seems to be even slower. hardware virtualisation is on, KVM is used, no relevant update since the update. here's about 3 seconds of kvm_stat:
kvm_exit 659513 103922 kvm_entry 659505 103920 kvm_emulate_insn 631118 99344 kvm_exit(IO_INSTRUCTION) 625223 98398 kvm_userspace_exit 624979 98359 kvm_pio 624970 98357
but according to atop, there is almost no I/O activity from guests, and running top in them shows them to really be idle...
but they eat like 60-70% of host CPU core time each..
Do You have any idea on what should I check?
thanks a lot in advance
nik
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 10 lines · Nikola Ciprich
Hello,
I'm trying to solve quite a weird problem on one of our customers' box.
It's quad core Xeon X3430 @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to 2.6.39).
After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just one windows or linux guest causes host to be quite slow, and quest seems to be even slower. hardware virtualisation is on, KVM is used, no relevant update since the update. here's about 3 seconds ofkvm_stat:
Quoted 6 lines
kvm_exit 659513 103922 kvm_entry 659505 103920 kvm_emulate_insn 631118 99344 kvm_exit(IO_INSTRUCTION) 625223 98398 kvm_userspace_exit 624979 98359 kvm_pio 624970 98357
How many guests in there?
Please post a trace as per http://www.linux-kvm.org/page/Tracing.
Quoted 6 lines
but according to atop, there is almost no I/O activity from guests, and running top in them shows them to really be idle...
but they eat like 60-70% of host CPU core time each..Do You have any idea on what should I check?
thanks a lot in advance
Please post /proc/mtrr and /proc/iomem.
Signature · 1 line
error compiling committee.c: too many arguments to function
Hello Avi,
thanks for quick reply!
Quoted 1 line
How many guests in there?
two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.
Quoted 2 lines
Please post a trace as per http://www.linux-kvm.org/page/Tracing.
here it goes:
http://nelide.cz/nik/trace.dat.bz2
Quoted 2 lines
Please post
/proc/mtrrand/proc/iomem.
[root@virtualbox ~]# cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
[root@virtualbox ~]# cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009d7ff : System RAM
0009d800-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000cffff : pnp 00:0f
000d0000-000dffff : PCI Bus 0000:00
000e0000-000fffff : reserved
00100000-bf78ffff : System RAM
01000000-013917ae : Kernel code
013917af-016c88ff : Kernel data
0176a000-0182ffff : Kernel bss
bf790000-bf79dfff : RAM buffer
bf79e000-bf79ffff : reserved
bf7a0000-bf7adfff : ACPI Tables
bf7ae000-bf7cffff : ACPI Non-volatile Storage
bf7d0000-bf7dffff : reserved
bf7e0000-bf7ecfff : RAM buffer
bf7ed000-bfffffff : reserved
c0000000-dfffffff : PCI Bus 0000:00
c0000000-c01fffff : PCI Bus 0000:08
c0200000-c03fffff : PCI Bus 0000:07
c0400000-c05fffff : PCI Bus 0000:06
c0600000-c07fffff : PCI Bus 0000:06
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : reserved
e0000000-efffffff : pnp 00:0e
f0000000-fed8ffff : PCI Bus 0000:00
fa000000-faffffff : PCI Bus 0000:09
fa000000-faffffff : 0000:09:03.0
fb4f6000-fb4f60ff : 0000:00:1f.3
fb4f8000-fb4f87ff : 0000:00:1f.2
fb4f8000-fb4f87ff : ahci
fb4fa000-fb4fa3ff : 0000:00:1d.0
fb4fa000-fb4fa3ff : ehci_hcd
fb4fc000-fb4fc3ff : 0000:00:1a.0
fb4fc000-fb4fc3ff : ehci_hcd
fb500000-fb5fffff : PCI Bus 0000:07
fb5dc000-fb5dffff : 0000:07:00.0
fb5dc000-fb5dffff : e1000e
fb5e0000-fb5fffff : 0000:07:00.0
fb5e0000-fb5fffff : e1000e
fb600000-fb6fffff : PCI Bus 0000:08
fb6dc000-fb6dffff : 0000:08:00.0
fb6dc000-fb6dffff : e1000e
fb6e0000-fb6fffff : 0000:08:00.0
fb6e0000-fb6fffff : e1000e
fb700000-fbffffff : PCI Bus 0000:09
fb7fc000-fb7fffff : 0000:09:03.0
fb800000-fbffffff : 0000:09:03.0
fc000000-fcffffff : pnp 00:01
fd000000-fdffffff : pnp 00:01
fe000000-febfffff : pnp 00:01
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed14000-fed19fff : pnp 00:01
fed1c000-fed1ffff : pnp 00:0b
fed20000-fed3ffff : reserved
fed20000-fed3ffff : pnp 00:0b
fed40000-fed8ffff : pnp 00:0b
fee00000-fee00fff : Local APIC
fee00000-fee00fff : reserved
fee00000-fee00fff : pnp 00:0d100000000-53fffffff : System RAM
n.
Quoted 4 lines
Signature · 2 lines
-- error compiling committee.c: too many arguments to function
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 10 lines · Nikola Ciprich
Hello Avi,
thanks for quick reply!Quoted 1 line
How many guests in there?
two currently (one
x86_64centos, one winXP), with more, the box get even more unresponsive.Quoted 2 lines
Please post a trace as per http://www.linux-kvm.org/page/Tracing.
here it goes:
http://nelide.cz/nik/trace.dat.bz2
404
Quoted 10 lines
Quoted 3 lines
Please post
/proc/mtrrand/proc/iomem.[root@virtualbox ~]# cat
/proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
These seem fine - all your memory is write-back.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line
404
ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.
Quoted 17 lines
Quoted 11 lines
Quoted 4 lines
Please post
/proc/mtrrand/proc/iomem.[root@virtualbox ~]# cat
/proc/mtrr
reg00: base=0x000000000 ( 0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachableThese seem fine - all your memory is write-back.
Signature · 1 line
error compiling committee.c: too many arguments to function
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 6 lines · Nikola Ciprich
Quoted 1 line
404
ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.
qemu-kvm-4618 [000] 13698.803169: kvm_emulate_insn:
0:806edb12: rep outsw
qemu-kvm-4618 [000] 13698.803170: kvm_pio:
pio_write at 0x1f0 size 2 count 1
qemu-kvm-4618 [000] 13698.803171: kvm_userspace_exit: reason
KVM_EXIT_IO (2)
qemu-kvm-4618 [000] 13698.803186: kvm_entry: vcpu 0
qemu-kvm-4618 [000] 13698.803189: kvm_exit: reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
qemu-kvm-4618 [000] 13698.803191: kvm_emulate_insn:
0:806edb12: rep outsw
qemu-kvm-4618 [000] 13698.803192: kvm_pio:
pio_write at 0x1f0 size 2 count 1
qemu-kvm-4618 [000] 13698.803193: kvm_userspace_exit: reason
KVM_EXIT_IO (2)
qemu-kvm-4618 [000] 13698.803195: kvm_entry: vcpu 0
qemu-kvm-4618 [000] 13698.803198: kvm_exit: reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.
Gleb, is rep/outsw our pio regression or was it another instruction?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 26 lines · Avi Kivity
Quoted 6 lines · Nikola Ciprich
Quoted 1 line
404
ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.qemu-kvm-4618 [000] 13698.803169: kvm_emulate_insn: 0:806edb12: rep outsw qemu-kvm-4618 [000] 13698.803170: kvm_pio: pio_write at 0x1f0 size 2 count 1 qemu-kvm-4618 [000] 13698.803171: kvm_userspace_exit: reason KVM_EXIT_IO (2) qemu-kvm-4618 [000] 13698.803186: kvm_entry: vcpu 0 qemu-kvm-4618 [000] 13698.803189: kvm_exit:reason
IO_INSTRUCTIONrip0x806edb12info 1f00031 0qemu-kvm-4618 [000] 13698.803191: kvm_emulate_insn: 0:806edb12: rep outsw qemu-kvm-4618 [000] 13698.803192: kvm_pio: pio_write at 0x1f0 size 2 count 1 qemu-kvm-4618 [000] 13698.803193: kvm_userspace_exit: reason KVM_EXIT_IO (2) qemu-kvm-4618 [000] 13698.803195: kvm_entry: vcpu 0 qemu-kvm-4618 [000] 13698.803198: kvm_exit:
Quoted 4 lines
reason
IO_INSTRUCTIONrip0x806edb12info 1f00031 0One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.
Even better, switch it to virtio.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Nikola Ciprich
Hello Avi,
thanks for quick reply!Quoted 1 line
How many guests in there?
two currently (one
x86_64centos, one winXP), with more, the box get even more unresponsive.
I don't really understand why the box is unresponsive. What do top and 'vmstat 1' say?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 26 lines · Avi Kivity
Quoted 6 lines · Nikola Ciprich
Quoted 1 line
404
ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.qemu-kvm-4618 [000] 13698.803169: kvm_emulate_insn: 0:806edb12: rep outsw qemu-kvm-4618 [000] 13698.803170: kvm_pio: pio_write at 0x1f0 size 2 count 1 qemu-kvm-4618 [000] 13698.803171: kvm_userspace_exit: reason KVM_EXIT_IO (2) qemu-kvm-4618 [000] 13698.803186: kvm_entry: vcpu 0 qemu-kvm-4618 [000] 13698.803189: kvm_exit:reason
IO_INSTRUCTIONrip0x806edb12info 1f00031 0qemu-kvm-4618 [000] 13698.803191: kvm_emulate_insn: 0:806edb12: rep outsw qemu-kvm-4618 [000] 13698.803192: kvm_pio: pio_write at 0x1f0 size 2 count 1 qemu-kvm-4618 [000] 13698.803193: kvm_userspace_exit: reason KVM_EXIT_IO (2) qemu-kvm-4618 [000] 13698.803195: kvm_entry: vcpu 0 qemu-kvm-4618 [000] 13698.803198: kvm_exit:
Quoted 7 lines
reason
IO_INSTRUCTIONrip0x806edb12info 1f00031 0One of your guests is using IDE PIO instead of DMA. Switch it back to DMA.
Gleb, is rep/outsw our pio regression or was it another instruction?
It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.
--
Gleb.
OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled and how can I reenable it, I'm not experienced that much with XP :(
Quoted 2 lines
It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.
now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??
Quoted 4 lines
--
Gleb.
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 2 lines · Nikola Ciprich
OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled
It can happen due to a timeout, perhaps a temporary load spike.
Quoted 1 line
and how can I reenable it,
Google knows.
Quoted 12 lines
I'm not experienced that much with XP :(
Quoted 2 lines
It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.
now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??
I don't think it's related to the memory.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: KVM got slow after adding more physical memory to host - SOLVED
Hello guys,
thanks to both of You for Your replies. The problem is solved, exactly as Avi said, the DMA in windows got somehow disabled. So this certainly was not related to adding the memory...
anyways, note for further generations:
in windows XP, the DMA usage can be checked in
Device Manager->IDE ATA/ATAPI Controllers -> Properties->Advanced Settings Current Transfer Mode must be Multi-Word DMA2 or something similar, NOT PIO!
The way I enabled this, was to uninstall both primary and secondary controller in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!) after reboot, controllers got detected and installed again, with DMA properly enabled. Note that when controller is in PIO mode, this is really a patience test, switching to DMA took me like half an hour to complete, so slow the system was :-/
So thanks a lot for Your help once more.
have a nice day.
nik
Quoted 33 lines · Avi Kivity
Quoted 2 lines · Nikola Ciprich
OK, seems like Avi's right, at least the windows guest is not using DMA though the question is, why did it got disabled
It can happen due to a timeout, perhaps a temporary load spike.
Quoted 1 line
and how can I reenable it,
Google knows.
Quoted 12 lines
I'm not experienced that much with XP :(
Quoted 2 lines
It is rep/in-out. But why it became noticeable only after adding more physical memory to the host is a mystery.
now that I'm thinking about it, we had similar problem in the past, the windows guest was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly solved this issue, but maybe the DMA just got randomly enabled, and now after adding the memory (and restarting of course), it just got disabled again?
but why could this be??I don't think it's related to the memory.
Signature · 6 lines
error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Re: KVM got slow after adding more physical memory to host - SOLVED
Quoted 12 lines · Nikola Ciprich
Hello guys,
thanks to both of You for Your replies. The problem is solved, exactly as Avi said, the DMA in windows got somehow disabled. So this certainly was not related to adding the memory...
anyways, note for further generations:
in windows XP, the DMA usage can be checked in
DeviceManager->IDEATA/ATAPI Controllers ->Properties->AdvancedSettings Current Transfer Mode must be Multi-Word DMA2 or something similar, NOT PIO!
Quoted 5 lines
The way I enabled this, was to uninstall both primary and secondary controller in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!) after reboot, controllers got detected and installed again, with DMA properly enabled. Note that when controller is in PIO mode, this is really a patience test, switching to DMA took me like half an hour to complete, so slow the system was :-/
You can open regedit and search for "NoIDE" - it should find one key with that name, with value "1" or "yes" - just delete it, there's no need to go that route with removing/reinstalling device drivers.
The reason why it has been disabled is - most likely - due to some timeout while handling i/o -- eg, when your host was loaded too much or were swapping - you should be able to find something about that in windows event log.
/mjt