Live migration stops working...

Hi,

since upgrade from qemu-kvm 0.10.5 to 0.10.6 (Debian Lenny, Packages from
http://www.corpit.ru/debian/tls/kvm/, kvm module from kernel 2.6.30) live-migration does not work anymore.

After "migrate -d tcp:192.168.0.2:4444" info says "migration completed", but "incoming" kvm machine dies.

Any hint how to debug this?

Regards
Thomas

What do you mean exactly by "dies"?

error compiling committee.c: too many arguments to function

Not sure if this is related but this is what happens on my system when I try to do a live migration with kvm-88 and the latest kernel 2.6.30 (gentoo 2.6.30-gentoo-r5).

[   73.168871] general protection fault: 0000 [#1] SMP
[   73.169409] last sysfs file: /sys/kernel/uevent_seqnum
[   73.169733] Modules linked in:
[   73.169733]
[   73.169733] Pid: 1, comm: init Not tainted (2.6.30-gentoo-r5 #1)
[   73.169733] EIP: 0060:[<c0102f01>] EFLAGS: 00010246 CPU: 0
[   73.169733] EIP is at sysenter_exit+0xf/0x16
[   73.169733] EAX: 00000000 EBX: 0000000b ECX: bfbc2a80 EDX: b7f90424
[   73.169733] ESI: 00000000 EDI: bfbc2d74 EBP: 00000000 ESP: df82bfb4
[   73.169733]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   73.169733] Process init (pid: 1, ti=df82a000 task=df830000 task.ti=df82a000)
[   73.169733] Stack:
[   73.169733]  0000000b bfbc2c44 00000000 00000000 bfbc2d74 bfbc2d88 00000000 0000007b
[   73.169733]  0000007b 00000000 00000000 0000008e b7f90424 00000073 00000246 bfbc2a80
[   73.169733]  0000007b 00000000 00000000
[   73.169733] Call Trace:
[   73.169733] Code: 42 c0 89 44 24 18 fa 8d 80 00 00 00 00 8b 4d 08 f7 c1 ff fe 00 08 0f 85 56 01 00 00 8b 54 24 30 8b 4c 24 3c 31 ed 8e 64 24 24 fb <0f> 35 8d 44 20 00 90 50 fc 6a 00 0f a0 06 1e 50 55 57 56 52 51
[   73.169733] EIP: [<c0102f01>] sysenter_exit+0xf/0x16 SS:ESP 0068:df82bfb4
[   73.179556] ---[ end trace f1a39bb1992a39a4 ]---
[   73.180072] general protection fault: 0000 [#2] SMP
[   73.180547] last sysfs file: /sys/kernel/uevent_seqnum
[   73.181009] Modules linked in:
[   73.181038]
[   73.181038] Pid: 2620, comm: klogd Tainted: G      D (2.6.30-gentoo-r5 #1)
[   73.181038] EIP: 0060:[<c0102f01>] EFLAGS: 00010246 CPU: 0
[   73.181038] EIP is at sysenter_exit+0xf/0x16
[   73.181038] EAX: 00000511 EBX: 00000000 ECX: bf97b0d8 EDX: b7ef3424
[   73.181038] ESI: 0804c479 EDI: 00000001 EBP: 00000000 ESP: df3d7fb4
[   73.181038]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   73.181038] Process klogd (pid: 2620, ti=df3d6000 task=df86a5d0 task.ti=df3d6000)
[   73.181038] Stack:
[   73.181038]  00000000 0804e5e0 00000fff 0804c479 00000001 bf97b108 00000511 0000007b
[   73.181038]  0000007b 00000000 00000000 00000003 b7ef3424 00000073 00000246 bf97b0d8
[   73.181038]  0000007b 00000000 00000000
[   73.181038] Call Trace:
[   73.181038] Code: 42 c0 89 44 24 18 fa 8d 80 00 00 00 00 8b 4d 08 f7 c1 ff fe 00 08 0f 85 56 01 00 00 8b 54 24 30 8b 4c 24 3c 31 ed 8e 64 24 24 fb <0f> 35 8d 44 20 00 90 50 fc 6a 00 0f a0 06 1e 50 55 57 56 52 51
[   73.181038] EIP: [<c0102f01>] sysenter_exit+0xf/0x16 SS:ESP 0068:df3d7fb4
[   73.190938] ---[ end trace f1a39bb1992a39a5 ]---
[   73.191390] general protection fault: 0000 [#3] SMP
[   73.191865] last sysfs file: /sys/kernel/uevent_seqnum
[   73.192329] Modules linked in:
[   73.192365]
[   73.192365] Pid: 2672, comm: apache2 Tainted: G      D (2.6.30-gentoo-r5 #1)
[   73.192365] EIP: 0060:[<c0102f01>] EFLAGS: 00010246 CPU: 0
[   73.192365] EIP is at sysenter_exit+0xf/0x16
[   73.192365] EAX: 00000000 EBX: 00000000 ECX: bfb7aea0 EDX: b80b7424
[   73.192365] ESI: 00000000 EDI: bfb7aee4 EBP: 00000000 ESP: dc453fb4
[   73.192365]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   73.192365] Process apache2 (pid: 2672, ti=dc452000 task=df833390 task.ti=dc452000)
[   73.192365] Stack:
[   73.192365]  00000000 00000000 00000000 00000000 bfb7aee4 bfb7aef8 00000000 0000007b
[   73.192365]  0000007b 00000000 00000000 0000008e b80b7424 00000073 00000246 bfb7aea0
[   73.192365]  0000007b 00000000 00000000
[   73.192365] Call Trace:
[   73.192365] Code: 42 c0 89 44 24 18 fa 8d 80 00 00 00 00 8b 4d 08 f7 c1 ff fe 00 08 0f 85 56 01 00 00 8b 54 24 30 8b 4c 24 3c 31 ed 8e 64 24 24 fb <0f> 35 8d 44 20 00 90 50 fc 6a 00 0f a0 06 1e 50 55 57 56 52 51
[   73.192365] EIP: [<c0102f01>] sysenter_exit+0xf/0x16 SS:ESP 0068:dc453fb4
[   73.202214] ---[ end trace f1a39bb1992a39a6 ]---
[   73.202655] general protection fault: 0000 [#4] SMP
[   73.203137] last sysfs file: /sys/kernel/uevent_seqnum
[   73.203207] Modules linked in:
[   73.203207]
[   73.203207] Pid: 2598, comm: mysqld Tainted: G      D (2.6.30-gentoo-r5 #1)
[   73.203207] EIP: 0060:[<c0102f01>] EFLAGS: 00210246 CPU: 0
[   73.203207] EIP is at sysenter_exit+0xf/0x16
[   73.203207] EAX: 00000000 EBX: 00000000 ECX: b324832c EDX: b7f23424
[   73.203207] ESI: 00000000 EDI: b3248370 EBP: 00000000 ESP: df3c9fb4
[   73.203207]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   73.203207] Process mysqld (pid: 2598, ti=df3c8000 task=df831810 task.ti=df3c8000)
[   73.203207] Stack:
[   73.203207]  00000000 00000000 00000000 00000000 b3248370 b3248378 00000000 0000007b
[   73.203207]  0000007b 00000000 00000000 0000008e b7f23424 00000073 00200246 b324832c
[   73.203207]  0000007b 00000000 00000000
[   73.203207] Call Trace:
[   73.203207] Code: 42 c0 89 44 24 18 fa 8d 80 00 00 00 00 8b 4d 08 f7 c1 ff fe 00 08 0f 85 56 01 00 00 8b 54 24 30 8b 4c 24 3c 31 ed 8e 64 24 24 fb <0f> 35 8d 44 20 00 90 50 fc 6a 00 0f a0 06 1e 50 55 57 56 52 51
[   73.203207] EIP: [<c0102f01>] sysenter_exit+0xf/0x16 SS:ESP 0068:df3c9fb4
[   73.213415] ---[ end trace f1a39bb1992a39a7 ]---
[   73.213874] general protection fault: 0000 [#5] SMP
[   73.214366] last sysfs file: /sys/kernel/uevent_seqnum
[   73.214409] Modules linked in:
[   73.214409]
[   73.214409] Pid: 2599, comm: mysqld Tainted: G      D (2.6.30-gentoo-r5 #1)
[   73.214409] EIP: 0060:[<c0102f01>] EFLAGS: 00210246 CPU: 0
[   73.214409] EIP is at sysenter_exit+0xf/0x16
[   73.214409] EAX: 00000000 EBX: 00000000 ECX: b2a4732c EDX: b7f23424
[   73.214409] ESI: 00000000 EDI: b2a47370 EBP: 00000000 ESP: df3cbfb4
[   73.214409]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   73.214409] Process mysqld (pid: 2599, ti=df3ca000 task=df8314a0 task.ti=df3ca000)
[   73.214409] Stack:
[   73.214409]  00000000 00000000 00000000 00000000 b2a47370 b2a47378 00000000 0000007b
[   73.214409]  0000007b 00000000 00000000 0000008e b7f23424 00000073 00200246 b2a4732c
[   73.214409]  0000007b 00000000 00000000
[   73.214409] Call Trace:
[   73.214409] Code: 42 c0 89 44 24 18 fa 8d 80 00 00 00 00 8b 4d 08 f7 c1 ff fe 00 08 0f 85 56 01 00 00 8b 54 24 30 8b 4c 24 3c 31 ed 8e 64 24 24 fb <0f> 35 8d 44 20 00 90 50 fc 6a 00 0f a0 06 1e 50 55 57 56 52 51
[   73.214409] EIP: [<c0102f01>] sysenter_exit+0xf/0x16 SS:ESP 0068:df3cbfb4
[   73.224816] ---[ end trace f1a39bb1992a39a8 ]---
[   73.225781] Kernel panic - not syncing: Attempted to kill init!
[   73.226337] Pid: 1, comm: init Tainted: G      D    2.6.30-gentoo-r5 #1
[   73.226930] Call Trace:
[   73.227185]  [<c0428219>] ? panic+0x42/0xf0
[   73.227571]  [<c012ca9e>] ? do_exit+0x66e/0x670
[   73.227984]  [<c01037c6>] ? apic_timer_interrupt+0x2a/0x30
[   73.228491]  [<c04282de>] ? printk+0x17/0x1b
[   73.228884]  [<c0104a50>] ? do_general_protection+0x0/0x190
[   73.229406]  [<c0106491>] ? oops_end+0x91/0xa0
[   73.229815]  [<c042acc5>] ? error_code+0x6d/0x74
[   73.230253]  [<c0104a50>] ? do_general_protection+0x0/0x190
[   73.230757]  [<c0102f01>] ? sysenter_exit+0xf/0x16

This is the sysexit instruction.

Are you doing cross-vendor migration (AMD->Intel or vice versa)?

This is supported in kvm-88, but not in previous versions. Are you sure the kvm-88 modules are loaded on the target host, and not the modules that came with the kernel?

Try 'cat /sys/module/kvm/version'.

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Only AMD->AMD, all of them are at kvm-kmod-devel-88 All of them run the userspace with kvm-88

Migration works if I do it with a live cd, but not if I do it with a running machine.

Each guest only has 1 cpu, 2 nics (I tried both virtio and e1000),

I tried with both 2.6.29 and 2.6.30

2.6.29 was a known working with kvm-84.

Hi,

I recently stumbled upon a similar issue.
Can you try the following patch? This is not a very nice one, but it cured the problem I had.
Do you see anything in the guest? I had some messages on the console about a faulting init (at boot time, when I was booting a 32bit userland on a 64bit kernel).

Regards,
Andre.

Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632 

---
 arch/x86/kvm/svm.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

index fc14bdf..bddcde5 100644
@@ -2140,6 +2140,17 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, unsigned ecx, u64 data)
 	case MSR_VM_IGNNE:
 		pr_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data);
 		break;
+	case MSR_EFER:
+		{
+		int enable = !(data & (1<<_EFER_LME));
+		u32 *msrpm = svm->msrpm;
+		printk(KERN_INFO "%sabling SYSENTER MSR intercepts (%llx)\n",
+			enable ? "dis" : "en", data);
+		set_msr_interception(msrpm, MSR_IA32_SYSENTER_CS, enable, enable);
+		set_msr_interception(msrpm, MSR_IA32_SYSENTER_ESP, enable, enable);
+		set_msr_interception(msrpm, MSR_IA32_SYSENTER_EIP, enable, enable);
+		}
+		return kvm_set_msr_common(vcpu, ecx, data);
 	default:
 		return kvm_set_msr_common(vcpu, ecx, data);
 	}

1.6.1.3

I've tried the patch and it did not help, however I could only have it on the receiver of the migration. Would it need to be installed on the one sending the data ?

The guest kernel and userland are 32bits, the host's are 64 bits.

I do not see any message about a faulting init.

It needs to be on both. However try this better patch (attached, needs to be on both hosts).

error compiling committee.c: too many arguments to function

index e158a2f..6df3cea 100644
@@ -101,9 +101,6 @@ struct vcpu_svm {
 	unsigned long vmcb_pa;
 	struct svm_cpu_data *svm_data;
 	uint64_t asid_generation;
-	uint64_t sysenter_cs;
-	uint64_t sysenter_esp;
-	uint64_t sysenter_eip;
 
 	u64 next_rip;
 

@@ -2087,13 +2084,13 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, unsigned ecx, u64 *data)
 		break;
 #endif
 	case MSR_IA32_SYSENTER_CS:
-		*data = svm->sysenter_cs;
+		*data = svm->vmcb->save.sysenter_cs;
 		break;
 	case MSR_IA32_SYSENTER_EIP:
-		*data = svm->sysenter_eip;
+		*data = svm->vmcb->save.sysenter_eip;
 		break;
 	case MSR_IA32_SYSENTER_ESP:
-		*data = svm->sysenter_esp;
+		*data = svm->vmcb->save.sysenter_esp;
 		break;
 	/* Nobody will change the following 5 values in the VMCB so
 	   we can safely return them on rdmsr. They will always be 0

@@ -2176,13 +2173,13 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, unsigned ecx, u64 data)
 		break;
 #endif
 	case MSR_IA32_SYSENTER_CS:
-		svm->sysenter_cs = data;
+		svm->vmcb->save.sysenter_cs = data;
 		break;
 	case MSR_IA32_SYSENTER_EIP:
-		svm->sysenter_eip = data;
+		svm->vmcb->save.sysenter_eip = data;
 		break;
 	case MSR_IA32_SYSENTER_ESP:
-		svm->sysenter_esp = data;
+		svm->vmcb->save.sysenter_esp = data;
 		break;
 	case MSR_IA32_DEBUGCTLMSR:
 		if (!svm_has(SVM_FEATURE_LBRV)) {

That solved it, thanks!

I'm starting the 'waiting' kvm process like this:

/usr/bin/kvm -name testbench -vnc :5 -net nic,vlan=0,model=virtio,macaddr=00:08:15:00:09:95 -net tap,vlan=0,ifname=tap05,script=no -m 256 -drive file=/vm/testbench/testbench.raw,if=virtio,boot=on -daemonize -vga std -k de -tdf -incoming tcp:0:4444

The resulting kvm process which is waiting for the incoming migration isn't there anymore after the migration command on the 'source' host.

Please run without -daemonize and capture stdio output, monitor output, and/or core dump.

error compiling committee.c: too many arguments to function

Output from the starting shell:
Guest moved used index from 23 to 3995

Because the whole kvm process dies no vnc output can be captured. Or did you mean something else with "monitor output"?

Howto produce more debug informations (i.e. core dump)?

virtio issue. What's the guest kernel version?

The monitor where you issue the migrate command (unless you're using a management application like virt-manager or virsh).

'ulimit -c unlimited' before running qemu. But in this case you won't get a dump since qemu exited cleanly.

error compiling committee.c: too many arguments to function

2.6.30.4 (self-compiled, reduced for qemu/kvm, no modules)

There are no error messages.

For completion: not vanilla sources, debian patched kernel sources.