guest gets stuck on the migration from AMD to Intel

Hi folks,

If I migrate a virtual machine (2.6.31.6, amd64) from a host with AMD cpu to an Intel host, then the guest is terminated on the old host as expected, but it gets stuck on the new host. Every 60 seconds it prints a message on the virtual console saying

	BUG: soft lockup - CPU#0 got stuck for 61s!

If I reset the guest, then it boots (without problems, as it seems).

There is no migration problem for AMD --> AMD and Intel --> AMD. I didn't had a chance to test Intel --> Intel yet.

The virtual disk is on a common NFSv3 partition. All hosts are running 2.6.31.6 (amd64).

Can anybody reproduce this? I saw the error message several times on Google, but not together with a migration from AMD to Intel.

Any helpful comment would be highly appreciated.

Regards

Harri
===================================================================
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 67
model name : Dual-Core AMD Opteron(tm) Processor 1210
stepping : 2
cpu MHz : 1795.378
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips : 3590.75
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
:
:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
stepping : 10
cpu MHz : 2500.605
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority
bogomips : 5001.21
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
:
:

See http://bugzilla.kernel.org/show_bug.cgi?id=14687

Regards

Harri

Quite possible that the guest could not be scheduled to run for a longer time during migration. In such a case the Linux kernel will find that the cpu was stuck/locked and hence throw the call trace. These messages are not harmful(is not it?) and the guest keeps running without any problem.

Sudhir Kumar

Please set up serial console for the guest and any post any detailed messages printed there (e.g. a stacktrace).

error compiling committee.c: too many arguments to function

This is what I got on the new host:

[  677.532010] BUG: soft lockup - CPU#0 stuck for 61s! [ntpd:1665]
[  677.532010] Modules linked in: loop serio_raw snd_pcsp psmouse virtio_balloon snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd floppy ide_core ehci_hcd virtio_pci virtio_ring virtio thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  677.532010] CPU 0:
[  677.532010] Modules linked in: loop serio_raw snd_pcsp psmouse virtio_balloon snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd floppy ide_core ehci_hcd virtio_pci virtio_ring virtio thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  677.532010] Pid: 1665, comm: ntpd Not tainted 2.6.30-2-amd64 #1
[  677.532010] RIP: 0010:[<ffffffff8022726e>]  [<ffffffff8022726e>] kvm_deferred_mmu_op+0x57/0xd2
[  677.532010] RSP: 0018:ffff88003d40dc68  EFLAGS: 00000293
[  677.532010] RAX: 0000000000000000 RBX: 0000000000000016 RCX: 000000003d40dcaa
[  677.532010] RDX: 0000000000000000 RSI: 0000000000000018 RDI: ffff88003d40dcaa
[  677.532010] RBP: ffffffff802105ce R08: 0000000000000080 R09: ffffe20000d2b2b8
[  677.532010] R10: 0000000000039d69 R11: 0000000000000001 R12: 0000000000000001
[  677.532010] R13: ffff88000000e808 R14: 000000003f401980 R15: 0000000000000000
[  677.532010] FS:  00007f932e5b36f0(0000) GS:ffff880002000000(0000) knlGS:0000000000000000
[  677.532010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  677.532010] CR2: 00007f932e5bd000 CR3: 000000003cd9c000 CR4: 00000000000006e0
[  677.532010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  677.532010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  677.532010] Call Trace:
[  677.532010]  [<ffffffff80227262>] ? kvm_deferred_mmu_op+0x4b/0xd2
[  677.532010]  [<ffffffff80227339>] ? kvm_mmu_write+0x2b/0x31
[  677.532010]  [<ffffffff802a6081>] ? handle_mm_fault+0x283/0x700
[  677.532010]  [<ffffffff8022cce1>] ? do_page_fault+0x1f3/0x208
[  677.532010]  [<ffffffff804b5d55>] ? page_fault+0x25/0x30
[  677.532010]  [<ffffffff80352fed>] ? copy_user_generic_string+0x2d/0x40
[  677.532010]  [<ffffffff802d6d1a>] ? seq_read+0x300/0x380
[  677.532010]  [<ffffffff802ff052>] ? proc_reg_read+0x6f/0x8a
[  677.532010]  [<ffffffff802c15ed>] ? vfs_read+0xa6/0xff
[  677.532010]  [<ffffffff802c1702>] ? sys_read+0x45/0x6e
[  677.532010]  [<ffffffff8020fa42>] ? system_call_fastpath+0x16/0x1b
[  743.032010] BUG: soft lockup - CPU#0 stuck for 61s! [ntpd:1665]
[  743.032010] Modules linked in: loop serio_raw snd_pcsp psmouse virtio_balloon snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd floppy ide_core ehci_hcd virtio_pci virtio_ring virtio thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  743.032010] CPU 0:
[  743.032010] Modules linked in: loop serio_raw snd_pcsp psmouse virtio_balloon snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd floppy ide_core ehci_hcd virtio_pci virtio_ring virtio thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  743.032010] Pid: 1665, comm: ntpd Not tainted 2.6.30-2-amd64 #1
[  743.032010] RIP: 0010:[<ffffffff8022726e>]  [<ffffffff8022726e>] kvm_deferred_mmu_op+0x57/0xd2
[  743.032010] RSP: 0018:ffff88003d40dc68  EFLAGS: 00000293
[  743.032010] RAX: 0000000000000000 RBX: 0000000000000016 RCX: 000000003d40dcaa
[  743.032010] RDX: 0000000000000000 RSI: 0000000000000018 RDI: ffff88003d40dcaa
[  743.032010] RBP: ffffffff802105ce R08: 0000000000000080 R09: ffffe20000d2b2b8
[  743.032010] R10: 0000000000039d69 R11: 0000000000000001 R12: 0000000000000001
[  743.032010] R13: ffff88000000e808 R14: 000000003f401980 R15: 0000000000000000
[  743.032010] FS:  00007f932e5b36f0(0000) GS:ffff880002000000(0000) knlGS:0000000000000000
[  743.032010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  743.032010] CR2: 00007f932e5bd000 CR3: 000000003cd9c000 CR4: 00000000000006e0
[  743.032010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  743.032010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  743.032010] Call Trace:
[  743.032010]  [<ffffffff80227262>] ? kvm_deferred_mmu_op+0x4b/0xd2
[  743.032010]  [<ffffffff80227339>] ? kvm_mmu_write+0x2b/0x31
[  743.032010]  [<ffffffff802a6081>] ? handle_mm_fault+0x283/0x700
[  743.032010]  [<ffffffff8022cce1>] ? do_page_fault+0x1f3/0x208
[  743.032010]  [<ffffffff804b5d55>] ? page_fault+0x25/0x30
[  743.032010]  [<ffffffff80352fed>] ? copy_user_generic_string+0x2d/0x40
[  743.032010]  [<ffffffff802d6d1a>] ? seq_read+0x300/0x380
[  743.032010]  [<ffffffff802ff052>] ? proc_reg_read+0x6f/0x8a
[  743.032010]  [<ffffffff802c15ed>] ? vfs_read+0xa6/0xff
[  743.032010]  [<ffffffff802c1702>] ? sys_read+0x45/0x6e
[  743.032010]  [<ffffffff8020fa42>] ? system_call_fastpath+0x16/0x1b

:
:

Hope this helps. Please mail.

Regards

Harri

Sorry, wrong kernel. Here is the output for 2.6.31.6:

[  374.736010] BUG: soft lockup - CPU#0 stuck for 61s! [ntpd:1657]
[  374.736010] Modules linked in: ipv6 loop snd_pcm snd_timer snd soundcore snd_page_alloc virtio_balloon psmouse serio_raw pcspkr evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic ata_piix libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd virtio_pci virtio_ring virtio floppy ehci_hcd ide_core thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  374.736010] CPU 0:
[  374.736010] Modules linked in: ipv6 loop snd_pcm snd_timer snd soundcore snd_page_alloc virtio_balloon psmouse serio_raw pcspkr evdev i2c_piix4 i2c_core button processor reiserfs ide_cd_mod cdrom ata_generic ata_piix libata scsi_mod ide_pci_generic virtio_blk virtio_net piix uhci_hcd virtio_pci virtio_ring virtio floppy ehci_hcd ide_core thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  374.736010] Pid: 1657, comm: ntpd Not tainted 2.6.31.6 #1
[  374.736010] RIP: 0010:[<ffffffff8102524d>]  [<ffffffff8102524d>] kvm_deferred_mmu_op+0x58/0xd6
[  374.736010] RSP: 0018:ffff88003d8ffc68  EFLAGS: 00000293
[  374.736010] RAX: 0000000000000000 RBX: 0000000000000016 RCX: 000000003d8ffcaa
[  374.736010] RDX: 0000000000000000 RSI: 0000000000000018 RDI: ffff88003d8ffcaa
[  374.736010] RBP: ffffffff8100c5ae R08: 0000000000000080 R09: ffffea0000a8a598
[  374.736010] R10: 000000000003a0d5 R11: 0000000000000001 R12: 00000000000280da
[  374.736010] R13: 000000003d8ffe48 R14: ffff880000001700 R15: 000000000000fdf0
[  374.736010] FS:  00007fa19b21f6f0(0000) GS:ffff8800015ac000(0000) knlGS:0000000000000000
[  374.736010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  374.736010] CR2: 00007fa19b229000 CR3: 000000003dcad000 CR4: 00000000000006f0
[  374.736010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  374.736010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  374.736010] Call Trace:
[  374.736010]  [<ffffffff81025241>] ? kvm_deferred_mmu_op+0x4c/0xd6
[  374.736010]  [<ffffffff8102531b>] ? kvm_mmu_write+0x2b/0x31
[  374.736010]  [<ffffffff810b7840>] ? handle_mm_fault+0x300/0x77d
[  374.736010]  [<ffffffff8111b49f>] ? seq_release_net+0x0/0x3b
[  374.736010]  [<ffffffff81028f29>] ? do_page_fault+0x25f/0x27b
[  374.736010]  [<ffffffff812a19a5>] ? page_fault+0x25/0x30
[  374.736010]  [<ffffffff81171bfd>] ? copy_user_generic_string+0x2d/0x40
[  374.736010]  [<ffffffff810ea37c>] ? seq_read+0x300/0x380
[  374.736010]  [<ffffffff81113e9d>] ? proc_reg_read+0x6d/0x88
[  374.736010]  [<ffffffff810d3ca2>] ? vfs_read+0xaa/0x166
[  374.736010]  [<ffffffff810d3e1a>] ? sys_read+0x45/0x6e
[  374.736010]  [<ffffffff8100ba02>] ? system_call_fastpath+0x16/0x1b

:
:

Regards

Harri

Hm, pvmmu. Can you provide /proc/cpuinfo on the source (AMD) host?

Marcelo, shouldn't this be inactive after migration from AMD to Intel? Or maybe hypercall patching is screwing up?

error compiling committee.c: too many arguments to function

(resend with better cc list)

Hm, pvmmu. Can you provide /proc/cpuinfo on the source (AMD) host?

Marcelo, shouldn't this be inactive after migration from AMD to Intel? Or maybe hypercall patching is screwing up?

error compiling committee.c: too many arguments to function

Sure:

% cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 67
model name : Dual-Core AMD Opteron(tm) Processor 1210
stepping : 2
cpu MHz : 1795.804
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips : 3591.60
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 67
model name : Dual-Core AMD Opteron(tm) Processor 1210
stepping : 2
cpu MHz : 1795.804
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips : 3591.17
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

Hope this helps. Please mail.

Regards

Harri

:

Any news about this problem?

Regards

Harri

Can you use gdb to figure out which line in handle_mm_fault() is responsible? 'list *0xffffffff810b784' might do the trick.

error compiling committee.c: too many arguments to function