AMD Opteron 6276 to Intel Xeon E5645 live migration failure
Hi there,
We recently acquired some of the latter CPUs, and are attempting to add them to our existing cluster of AMD 6200/6300 KVM hosts. My life is made much easier if live migration between all hosts in the cluster works flawlessly, but I can reliably trigger a migration failure when moving a guest from the 6276 to the E5645 hosts.
The failure looks like:
KVM: injection failed, MSI lost (Operation not permitted)
KVM: entry failed, hardware error 0x80000021
If you're running a guest on an Intel machine without unrestricted mode support, the failure can be most likely due to the guest entering an invalid state for Intel VT. For example, the guest maybe running in big real mode which is not supported on less recent Intel processors.
EAX=00000000 EBX=81600000 ECX=00000000 EDX=00000000 ESI=00000003 EDI=81600000 EBP=8168f060 ESP=81601f10 EIP=8102b36c EFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The guest itself is running Linux in ordinary 64-bit mode at the point where it's migrated.
I also get this in dmesg:
kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround
I've tried kernels 3.2.54 and 3.10.26 on the source host, and 3.2.54, 3.10.26 and 3.12.9 on the destination; and qemu versions 1.5.3 and 1.7.0. Same error in all cases. I've also tried twiddling the emulate_invalid_guest_state flag, to no effect.
I can reproduce it with a guest as simple as:
/opt/qemu-1.5.3/bin/qemu-system-x86_64 -enable-kvm -nographic -monitor stdio -kernel /boot/vmlinuz-3.10.26-bigv-20 -initrd /boot/initrd.img-3.10.26-bigv-20 -net none
on the source, then migrating that to the destination.
I can reliably boot and run the guest on the E5645 host, and migrate that guest to the AMD host. Migrating the same guest back again fails with the same error.
Migrating from a later Intel machine (i7-4600U) to the earlier Intel seems to work fine.
I'd love to get this working, but I'm a little ignorant on where to begin, or even if it's possible at all. Are these CPUs just too old, or is a fixup missing in qemu (or kvm)?
/Nick
Paolo Bonzini
maintainerIl 28/02/2014 13:16, Nick Thomas ha scritto:
Quoted 3 lines
I'd love to get this working, but I'm a little ignorant on where to begin, or even if it's possible at all. Are these CPUs just too old, or is a fixup missing in qemu (or kvm)?
It's the latter (in kvm). Note that for migration to work, especially for such different models, you have to disable CPU features that aren't present in both models. In general the way to do this is to add "-cpu".
I'd start debugging with "-cpu kvm64" on both sides.
Paolo
Quoted 5 lines · Paolo Bonzini
Il 28/02/2014 13:16, Nick Thomas ha scritto:
Quoted 3 lines
I'd love to get this working, but I'm a little ignorant on where to begin, or even if it's possible at all. Are these CPUs just too old, or is a fixup missing in qemu (or kvm)?
Quoted 7 lines
It's the latter (in kvm). Note that for migration to work, especially for such different models, you have to disable CPU features that aren't present in both models. In general the way to do this is to add "-cpu".
I'd start debugging with "-cpu kvm64" on both sides.
Paolo
Sorry, I forgot to mention - most of my tests have been -cpu qemu64,-vmx,-svm. I've just re-run the minimal ones with that, and kvm64, for sanity. Same behaviour.
In the interests of debugging, I guess the next step is to attach gdb to the destination and find out what's happening around the time the error messages are pushed out?
I found an earlier thread (
http://www.spinics.net/lists/kvm/msg73478.html ) with a similar flavour that referenced a "vmxcap" tool - the output of that is here:
Basic VMX Information
Revision 15 VMCS size 1024 VMCS restricted to 32 bit addresses no Dual-monitor support yes VMCS memory type 6 INS/OUTS instruction information yes IA32_VMX_TRUE_*_CTLS support yes pin-based controls External interrupt exiting yes NMI exiting yes Virtual NMIs yes Activate VMX-preemption timer yes Process posted interrupts no primary processor-based controls Interrupt window exiting yes Use TSC offsetting yes HLT exiting yes INVLPG exiting yes MWAIT exiting yes RDPMC exiting yes RDTSC exiting yes CR3-load exiting default CR3-store exiting default CR8-load exiting yes CR8-store exiting yes Use TPR shadow yes NMI-window exiting yes MOV-DR exiting yes Unconditional I/O exiting yes Use I/O bitmaps yes Monitor trap flag yes Use MSR bitmaps yes MONITOR exiting yes PAUSE exiting yes Activate secondary control yes secondary processor-based controls Virtualize APIC accesses yes Enable EPT yes Descriptor-table exiting yes Enable RDTSCP yes Virtualize x2APIC mode yes Enable VPID yes WBINVD exiting yes Unrestricted guest yes APIC register emulation no Virtual interrupt delivery no PAUSE-loop exiting yes RDRAND exiting no Enable INVPCID no Enable VM functions no VMCS shadowing no EPT-violation #VE no VM-Exit controls Save debug controls default Host address-space size yes Load IA32_PERF_GLOBAL_CTRL yes Acknowledge interrupt on exit yes Save IA32_PAT yes Load IA32_PAT yes Save IA32_EFER yes Load IA32_EFER yes Save VMX-preemption timer value yes VM-Entry controls Load debug controls default IA-64 mode guest yes Entry to SMM yes Deactivate dual-monitor treatment yes Load IA32_PERF_GLOBAL_CTRL yes Load IA32_PAT yes Load IA32_EFER yes Miscellaneous data VMX-preemption timer scale (log2) 7 Store EFER.LMA into IA-32e mode guest control yes HLT activity state yes Shutdown activity state yes Wait-for-SIPI activity state yes IA32_SMBASE support no Number of CR3-target values 4 MSR-load/store count recommenation 0 IA32_SMM_MONITOR_CTL[2] can be set to 1 no VMWRITE to VM-exit information fields no MSEG revision identifier 0 VPID and EPT capabilities Execute-only EPT translations yes Page-walk length 4 yes Paging-structure memory type UC yes Paging-structure memory type WB yes 2MB EPT pages yes 1GB EPT pages yes INVEPT supported yes EPT accessed and dirty flags no Single-context INVEPT yes All-context INVEPT yes INVVPID supported yes Individual-address INVVPID yes Single-context INVVPID yes All-context INVVPID yes
Single-context-retaining-globals INVVPID yes
VM Functions
EPTP Switching no
Mostly Greek, but I assume one of the things labeled "no" is the source of my woes?
/Nick