Bugs in kvm guest migration
Hi,
my migration issues currently look like they are driven by at least two bugs (that's real fun). One slipped into qemu-kvm during merge of upstream, the other is either an issue of current kvm.git master or kvm-kmod's wrapping of the latter.
Problem 1
---------
Setup: qemu-kvm after 249cdd049c7976614ad46eb86409d5e1e53835cc (merge f9ea97e859dc0fce26c35f7e047a51788a3e5441 is the first usable one again, and it's affected), any kernel module version, 64-bit host & guest (Linux).
Effect: After resume of the migration target, its timer is still ticking (APIC IRQs arrive) but timed Linux tasks make no progress. Linux guest uses highres/nohz, when disabling both, the problem disappears.
Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
Effect: The migration target either locks up or reboots immediately. I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every kvm module build before that works, everything including and after 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in between crashes the host for known reasons). It's still unclear if this is an kvm-kmod wrapping issue of the user-space return notifiers. Will check once problem #1 is understood.
Any early ideas on this are welcome while I continue digging.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
Effect: The migration target either locks up or reboots immediately. I've nailed this down to84d0b66c778d881eafca2a5d0d66678211c4e861. Every kvm module build before that works, everything including and after26ede77f536d1bb369527a96c7fe7fdc8ba2f890shows the effect (everything in between crashes the host for known reasons). It's still unclear if this is an kvm-kmod wrapping issue of the user-space return notifiers. Will check once problem #1 is understood.
It is kernel-related, using kvm.git as host kernel makes no difference.
This may now mean that kvm is buggy or that it triggers some msr save/restore related issues in qemu. Digging even deeper...
Jan
Quoted 13 lines · Jan Kiszka
Quoted 11 lines · Jan Kiszka
Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host& guest.
Effect: The migration target either locks up or reboots immediately. I've nailed this down to84d0b66c778d881eafca2a5d0d66678211c4e861. Every kvm module build before that works, everything including and after26ede77f536d1bb369527a96c7fe7fdc8ba2f890shows the effect (everything in between crashes the host for known reasons). It's still unclear if this is an kvm-kmod wrapping issue of the user-space return notifiers. Will check once problem #1 is understood.
Quoted 5 lines
It is kernel-related, using kvm.git as host kernel makes no difference.
This may now mean that kvm is buggy or that it triggers some msr save/restore related issues in qemu. Digging even deeper...
Does calling drop_user_return_notifiers() (static in x86.c, will need exporting) in vmx_load_host_state() within the preempt-disable region help?
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jan Kiszka
Quoted 11 lines · Jan Kiszka
Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host& guest.
Effect: The migration target either locks up or reboots immediately. I've nailed this down to84d0b66c778d881eafca2a5d0d66678211c4e861. Every kvm module build before that works, everything including and after26ede77f536d1bb369527a96c7fe7fdc8ba2f890shows the effect (everything in between crashes the host for known reasons). It's still unclear if this is an kvm-kmod wrapping issue of the user-space return notifiers. Will check once problem #1 is understood.
Quoted 9 lines
Quoted 5 lines
It is kernel-related, using kvm.git as host kernel makes no difference.
This may now mean that kvm is buggy or that it triggers some msr save/restore related issues in qemu. Digging even deeper...
Does calling
drop_user_return_notifiers()(static inx86.c, will need exporting) invmx_load_host_state()within the preempt-disable region help?
Nope.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 17 lines · Jan Kiszka
Quoted 16 lines · Avi Kivity
Quoted 15 lines · Jan Kiszka
Quoted 11 lines · Jan Kiszka
Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host& guest.
Effect: The migration target either locks up or reboots immediately. I've nailed this down to84d0b66c778d881eafca2a5d0d66678211c4e861. Every kvm module build before that works, everything including and after26ede77f536d1bb369527a96c7fe7fdc8ba2f890shows the effect (everything in between crashes the host for known reasons). It's still unclear if this is an kvm-kmod wrapping issue of the user-space return notifiers. Will check once problem #1 is understood.It is kernel-related, using kvm.git as host kernel makes no difference.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
This may now mean that kvm is buggy or that it triggers some msr save/restore related issues in qemu. Digging even deeper...
Does calling
drop_user_return_notifiers()(static inx86.c, will need exporting) invmx_load_host_state()within the preempt-disable region help?Nope.
It was a qemu bug, resolved by "KVM: x86: Fix initial kvm_has_msr_star".
Jan