qemu-kvm-0.13.0, 2.6.37.1 - after migration, some of guests got stuck
Hello,
I've just migrated a bunch of guests from one node to another due to upgrading host kernel from 2.6.37 to 2.6.37.1.
After that, some of guests started consuming 100% of CPU time and their os seems to be stuck.
Looks like all the stuck guest were running 2.6.32 with kvm paravirt enabled. others running same as well as different kernels seem to be ok, including windows guests.
I guess I can keep stuck guest in this state for few hours, so if somebody would be willing to have a look at this, I'll be happy to try to help.
running kvm_stat for few seconds reports:
kvm_exit 237284 19119 kvm_entry 237183 19119 kvm_emulate_insn 123548 10088 kvm_mmio 92947 7622 kvm_exit(IO_INSTRUCTION) 90744 7395 kvm_pio 90726 7395 kvm_exit(EXCEPTION_NMI) 69993 5416 kvm_page_fault 69254 5347 kvm_apic 51528 4157
host is x86_64, guests were x86_64 too..
BR
nik
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Hello,
I'm no KVM core developer, so please take my advise with a grain of caution.
Am Sonntag 20 Februar 2011 20:32:35 schrieb Nikola Ciprich:
Quoted 8 lines
I've just migrated a bunch of guests from one node to another due to upgrading host kernel from 2.6.37 to 2.6.37.1.
After that, some of guests started consuming 100% of CPU time and their os seems to be stuck.Looks like all the stuck guest were running 2.6.32 with kvm paravirt enabled. others running same as well as different kernels seem to be ok, including windows guests.
I also sometimes experience problems with migrated (in my case: suspended) VMs, see <http://marc.info/?l=kvm&m=129484437222822&w=2>: They were either stuck in an interrupt storm or did not receive any interrupts any more (looks like some interrupts relates state gets not saved/restored properly). For me the following did help:
1. Suspend VM to disk
2. Restart VM with the -no-kvm-irqchip Option
Perhaps you could test that as well. (An alternative would be to use qemus internal debugger or gdb to remote debug the VM and detect what your VMs are doing, especially "x/20i $rip" to dump the next 20 assembler instructions) Please be advised, that -no-kvm-irqchip opens a new can of worms and should only be used to diagnose the problem. You probably should also backup the save file and use the -snapshot option to be able to repeatly test different strategies.
Sincerely
Philipp
Signature · 7 lines
Philipp Hahn Open Source Software Engineer [email protected] Univention GmbH Linux for Your Business fon: +49 421 22 232- 0 Mary-Somerville-Str.1 28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/ ** Besuchen Sie uns auf der CeBIT in Hannover ** ** Auf dem Univention Stand D36 in Halle 2 ** ** Vom 01. bis 05. März 2011 **