Live migration of x86-64 kvm guests broken?
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 9 lines · Ryan Harper
Quoted 8 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Quoted 6 lines
Quoted 2 lines
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Hell, too many variables. It looks like the kernel modules have some if not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, kvm-kmod stable-2.6.32 as well as master cause troubles.
What is your host kernel and/or kvm module version?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 17 lines · Jan Kiszka
Quoted 9 lines · Ryan Harper
Quoted 8 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Quoted 6 lines
Quoted 2 lines
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Quoted 5 lines
Hell, too many variables. It looks like the kernel modules have some if not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, kvm-kmod stable-2.6.32 as well as master cause troubles.
What is your host kernel and/or kvm module version?
Stock 64-bit Ubuntu Karmic:
% uname -a
Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
% modinfo kvm
filename:/lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.ko
license: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool
% modinfo kvm-intel
filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 37 lines · Ryan Harper
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Ryan Harper
Quoted 10 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Quoted 8 lines
Quoted 3 lines
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Hell, too many variables. It looks like the kernel modules have some if not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, kvm-kmod stable-2.6.32 as well as master cause troubles.
What is your host kernel and/or kvm module version?
Stock 64-bit Ubuntu Karmic:
% uname -a
Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009x86_64GNU/Linux% modinfo kvm
filename:/lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.kolicense: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool% modinfo kvm-intel
Quoted 12 lines
filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
My tests were with badblocks -svw /dev/vda
(qemu.git not qemu-kvm) I can try if you want.
Later, Juan.
Quoted 51 lines · Jan Kiszka
Quoted 37 lines · Ryan Harper
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Ryan Harper
Quoted 10 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Quoted 8 lines
Quoted 3 lines
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Hell, too many variables. It looks like the kernel modules have some if not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, kvm-kmod stable-2.6.32 as well as master cause troubles.
What is your host kernel and/or kvm module version?
Stock 64-bit Ubuntu Karmic:
% uname -a
Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009x86_64GNU/Linux% modinfo kvm
filename:/lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.kolicense: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool% modinfo kvm-intel
Quoted 12 lines
filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool
Quoted 2 lines
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Sure, can you remind me of the kvm-kmod build magic for building that branch?
Quoted 5 lines
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 50 lines · Ryan Harper
Quoted 49 lines · Jan Kiszka
Quoted 48 lines · Ryan Harper
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Ryan Harper
Quoted 10 lines · Jan Kiszka
Hi,
I'm facing stalled x86-64 guests after live migration when using kvm (share disk images). This does not happen with x86-32 guests or when disabling kvm. Both qemu and qemu-kvm git heads are affected (recent vmstate fixes applied). Running I/O load during the migration (e.g. a simple "ls -R /") seems to trigger it reliably.
Can anyone confirm / comment on this? Do we fail to restore some magic state that only long mode cares about?
Quoted 8 lines
Quoted 3 lines
Just tested upstream qemu.git (with vmstate fixes) and I could migrate RHEL 5.3 64-bit guest with your light io load over localhost. I'll try remote hosts next.
Hell, too many variables. It looks like the kernel modules have some if not the ultimate impact: kvm-mod-2.6.31.6b on my 2.6.27 host works fine, kvm-kmod stable-2.6.32 as well as master cause troubles.
What is your host kernel and/or kvm module version?
Stock 64-bit Ubuntu Karmic:
% uname -a
Linux symmetry 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009x86_64GNU/Linux% modinfo kvm
filename:/lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm.kolicense: GPL author: Qumranet srcversion: 82D6B673524596F9CF3E84C depends: vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: oos_shadow:bool% modinfo kvm-intel
filename: /lib/modules/2.6.31-15-generic/kernel/arch/x86/kvm/kvm-intel.ko license: GPL author: Qumranet srcversion: 8FEA479DFCD7F174DA7864E depends: kvm vermagic: 2.6.31-15-generic SMP mod_unload modversions parm: bypass_guest_pf:bool parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: emulate_invalid_guest_state:bool
Quoted 6 lines
Quoted 2 lines
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Sure, can you remind me of the kvm-kmod build magic for building that branch?
If you want to build from git, check Wolfgang's nice README. But it might be easier to just download the tarball from sourceforge.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 10 lines · Jan Kiszka
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Sure, can you remind me of the kvm-kmod build magic for building that branch?
If you want to build from git, check Wolfgang's nice README. But it might be easier to just download the tarball from sourceforge.
Indeed, tarball is easiest. Still working on top of qemu.git+vmstate fixes. Here's my invocation:
Source:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvm
Dest:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvm -incoming tcp:0:4444
Lemme try qemu-kvm.git
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 21 lines · Ryan Harper
Quoted 10 lines · Jan Kiszka
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Sure, can you remind me of the kvm-kmod build magic for building that branch?
If you want to build from git, check Wolfgang's nice README. But it might be easier to just download the tarball from sourceforge.
Indeed, tarball is easiest. Still working on top of qemu.git+vmstate fixes. Here's my invocation:
Source:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvm
Quoted 7 lines
Dest:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvm -incoming tcp:0:4444Lemme try qemu-kvm.git
Working there as well on top of 2.6.32-rc7 kmod modules and kvm-qemu.git tip+vmstate fixes. Any other varient?
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 24 lines · Ryan Harper
Quoted 23 lines · Ryan Harper
Quoted 7 lines · Jan Kiszka
Quoted 5 lines
Quoted 2 lines
So far, so consistent. Could you try kvm-kmod-2.6.32-rc7 on top of that?
Sure, can you remind me of the kvm-kmod build magic for building that branch?
If you want to build from git, check Wolfgang's nice README. But it might be easier to just download the tarball from sourceforge.
Indeed, tarball is easiest. Still working on top of qemu.git+vmstate fixes. Here's my invocation:
Source:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvmDest:
sudo ./x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 512 -drive file=/media/blackstar/images/rhel5-64.raw,if=ide -net nic,model=e1000 -monitor stdio -enable-kvm -incoming tcp:0:4444
Quoted 5 lines
Quoted 1 line
Lemme try qemu-kvm.git
Working there as well on top of 2.6.32-rc7 kmod modules and kvm-qemu.git tip+vmstate fixes. Any other varient?
Interesting. I think I need to try with a more recent host kernel (not sure if you want to downgrade your kernel :) ). Maybe it's a kvm-kmod wrapping issue.
Thanks so far,
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux