Migration problems

All,
I've been doing some migration testing of KVM guests, and have been running
into some problems. Let me describe the setup and what I've tried, and maybe somebody has some ideas about what might be going on here.

Setup:
2 identical Intel SDV boxes, Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz
Both machines are running F-8 kernel, 2.6.23.8-63.fc8 x86_64, with updated
KVM kernel modules from git.

     On one machine, I have a /kvm directory that holds my guest disk image;

this directory is exported via NFS and mounted as /kvm on the secondary machine.

The guest in question is also an F-8 x86_64 guest, running the same kernel as the hosts. I start it up with the following command-line:

qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup -monitor stdio

The guest starts up just fine. On the secondary machine, I use the following command-line:

qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup -monitor stdio -incoming tcp://0:4444

(i.e. exactly the same, but I add the -incoming parameter). When I try to do live migration this way, things seem like they work, and it even seems like a few instructions get executed on the destination side. However, fairly quickly I'll get "Disabling IRQ #11" on the console of the guest at the destination, and the qemu process will just spin at 100%, with no interaction possible. IRQ #11, incidentally, is the IRQ associated with the emulated rtl8139 card.

This led me to suspect the in-kernel PIC/APIC emulation, and, sure enough, if I pass -no-kvm-irqchip (on both the source and destination), things are better; I can at least migrate from the host to the destination without the "Disabling IRQ #11" message. However, if I put any sort of load on the guest while doing migration, I still get a hang-up, even with -no-kvm-irqchip.

Has anyone else seen this, or have ideas where I can start debugging it?

Thanks,
Chris Lalancette

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[copying Uri, who is also chasing migration bugs]

- check the guest kernel without an ioapic
- if that works, check the ioapic load/save paths
- I'd also suspect 3ead9ca0bd2214af63ea2ebf84573576b38e004e or 71ab66c92f1ecd3f1aabed0bfa2e356fb6bbfebc

Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Hi Chris,

I am chasing migration bug(s) too.
I think the problem lies in userspace and not in the kernel modules. It seems that for kvm-56 migration works but for kvm-57 it does not. The problem seems to be related to the qemu-cvs merge:
25a2e04e5689af63d104de2dea755f30bf643afd
If you run it with -no-kvm migration seems to be working.

The test case I'm using is savevm/loadvm.
After a loadvm my FC6 guest gets double-fault (<qemu-cmdline> -loadvm <id>). Adding '-no-kvm-irqchip' does not help.
But if I start qemu in a stopped mode (<qemu-cmdline> -S) and 'loadvm <id>' in qemu-monitor, it works.
Sometimes after lodavm guest network does work better with '-no-kvm-irqchip' I am trying to use bisection to find the problematic patch.

Thanks,
Uri.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Another migration problem that pops up with kvm 60, doing:
migrate "exec:dd of=/tmp/bla"
works in the qemu monitor however it doesn't, when you start kvm with "-monitor pty" and pipe the same command into /dev/pts/X (which is what libvirt does). Kvm segfaults immediately. I'll debug this further but maybe someone else has seen this already?
Cheers,
-- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Actually the above is true vor kvm 58, with kvm 60 the migration starts but doesn't finish, qemu becomes unresponsive and every access to /proc/<kvmpid>/cmdline hangs forever (on 2.6.23.11). Attaching gdb to the process doesn't work either. CPU is a Dual Core Opteron 2220. Things work fine if I pass -no-kvm. Any ideas?
-- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/