From: Philipp Hahn Subject: [BUG,PATCH qemu-kvm_0.14.x] dead-lock by load_vmstate() =?utf-8?q?=E2=86=92?= kvm_tpr_enable_vapic() =?utf-8?q?=E2=86=92?= on_vcpu() Date: Tue, 29 Nov 2011 18:44:16 +0100 Message-ID: <201111291844.22053.hahn@univention.de> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3254970.ntbi5pWVha"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit To: kvm Return-path: Received: from mail.univention.de ([82.198.197.8]:2080 "EHLO mail.univention.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751530Ab1K2RoZ (ORCPT ); Tue, 29 Nov 2011 12:44:25 -0500 Received: from localhost (localhost [127.0.0.1]) by slugis.knut.univention.de (Postfix) with ESMTP id 9FD48D78E56 for ; Tue, 29 Nov 2011 18:44:21 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by slugis.knut.univention.de (Postfix) with ESMTP id 8E35F5A7B8D for ; Tue, 29 Nov 2011 18:44:21 +0100 (CET) Received: from mail.univention.de ([127.0.0.1]) by localhost (slugis.knut.univention.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vlLpJZSI0GU0 for ; Tue, 29 Nov 2011 18:44:20 +0100 (CET) Received: from stave.knut.univention.de (stave.knut.univention.de [192.168.0.191]) by slugis.knut.univention.de (Postfix) with ESMTPSA id CEB31D78E56 for ; Tue, 29 Nov 2011 18:44:20 +0100 (CET) Sender: kvm-owner@vger.kernel.org List-ID: --nextPart3254970.ntbi5pWVha Content-Type: multipart/mixed; boundary="Boundary-01=_xnR1OCNH8k6GnQb" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_xnR1OCNH8k6GnQb Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, This bug/patch is only relevant for qemu-0.14.x users, since in qemu-0.15 t= he=20 main-loop is completely different. So this is more or less for reference=20 only, if others experience the same dead-lock. I have created a snapshot of an VM, which now doesn't load. Thread 1 loads the saved state and calls on_vcpu(), which notifies Thread 2= =20 before going to sleep. Thread 2 waits for "qemu_system_ready" being set, so it isn't yet waiting f= or=20 qemu_work_cond. qemu_system_ready would only later be set by Thread 1 in kvm_main_loop(). Here's the gdb backtrace: (gdb) thread apply all bt Thread 2 (Thread 0x7fe9bcd89700 (LWP 16607)): #0 0x00007fe9c5ad716c in pthread_cond_wait@@GLIBC_2.3.2 ()=20 from /lib/libpthread.so.0 #1 0x000000000043c60a in qemu_cond_wait (_env=3D0x13e0ec0) at=20 qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1107 #2 ap_main_loop (_env=3D0x13e0ec0) at qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1460 #3 0x00007fe9c5ad28ba in start_thread () from /lib/libpthread.so.0 #4 0x00007fe9c2cb002d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 1 (Thread 0x7fe9c62ea760 (LWP 16601)): #0 0x00007fe9c5ad716c in pthread_cond_wait@@GLIBC_2.3.2 ()=20 from /lib/libpthread.so.0 #1 0x0000000000438b1f in qemu_cond_wait (env=3D,=20 func=3D, data=3D) at qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1107 #2 on_vcpu (env=3D, func=3D,=20 data=3D) at qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1161 #3 0x000000000057a275 in cpu_synchronize_state (env=3D0x13e0ec0) at=20 qemu-kvm-0.14.1+dfsg/kvm.h:183 #4 get_pcr_cpu (env=3D0x13e0ec0) at qemu-kvm-0.14.1+dfsg/kvm-tpr-opt.c:213 #5 kvm_tpr_enable_vapic (env=3D0x13e0ec0) at=20 qemu-kvm-0.14.1+dfsg/kvm-tpr-opt.c:224 #6 0x00000000004370ae in kvm_arch_load_regs (env=3D0x13e0ec0, level=3D3)=20 at ../qemu-kvm-x86.c:605 #7 0x0000000000438fe2 in kvm_cpu_synchronize_post_init (env=3D0x96de44) at= =20 qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1190 #8 0x000000000040ce58 in cpu_synchronize_post_init () at=20 qemu-kvm-0.14.1+dfsg/kvm.h:197 #9 cpu_synchronize_all_post_init () at qemu-kvm-0.14.1+dfsg/cpus.c:98 #10 0x00000000004a5087 in qemu_loadvm_state (f=3D0x167fa10) at savevm.c:1826 #11 0x00000000004a5281 in load_vmstate (name=3D) at=20 savevm.c:2071 #12 0x000000000041bcbc in main (argc=3D52, argv=3D,=20 envp=3D) at qemu-kvm-0.14.1+dfsg/vl.c:3187 The VM was started by libvirtd as /usr/bin/kvm -S \ -M pc-0.14 \ -enable-kvm \ -m 512 \ -smp 1,sockets=3D1,cores=3D1,threads=3D1 \ -name winxp-1 -uuid 35dcee44-64bb-4fe0-b75c-afc50fe09c0c \ -nodefconfig -nodefaults \ -chardev=20 socket,id=3Dmonitor,path=3D/var/lib/libvirt/qemu/winxp-1.monitor,server,now= ait \ -mon chardev=3Dmonitor,mode=3Dreadline \ -rtc base=3Dlocaltime \ -boot dc \ -drive=20 file=3D/var/lib/libvirt/images/winxp-1-0.qcow2,if=3Dnone,id=3Ddrive-virtio-= disk0,boot=3Don,format=3Dqcow2=20 \ -device=20 virtio-blk-pci,bus=3Dpci.0,addr=3D0x5,drive=3Ddrive-virtio-disk0,id=3Dvirti= o-disk0 \ -drive=20 file=3D/var/lib/libvirt/images/win_xp_pro.iso,if=3Dnone,media=3Dcdrom,id=3D= drive-ide0-0-1,readonly=3Don,format=3Draw=20 \ -device ide-drive,bus=3Dide.0,unit=3D1,drive=3Ddrive-ide0-0-1,id=3Dide0-0-= 1 \ -drive=20 file=3D/var/lib/libvirt/images/kvm-windows-drivers_\(virtio-1.1.16\).vfd,if= =3Dnone,id=3Ddrive-fdc0-0-0,format=3Draw=20 \ -global isa-fdc.driveA=3Ddrive-fdc0-0-0 \ -netdev tap,fd=3D46,id=3Dhostnet0 \ -device=20 rtl8139,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:8f:53:6f,bus=3Dpci.0,add= r=3D0x3 \ -usb \ -device usb-tablet,id=3Dinput0 \ -vnc 0.0.0.0:0 -k de -vga cirrus \ -device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x4 \ -loadvm "Start der Installation XP1" =2D-=20 Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH Linux for Your Business fon: +49 421 22 232- 0 Mary-Somerville-Str.1 D-28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/ --Boundary-01=_xnR1OCNH8k6GnQb Content-Type: text/x-diff; charset="iso-8859-15"; name="44_fix-on_vcpu-deadlock.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="44_fix-on_vcpu-deadlock.patch" Bug #22877: Fix on_vcpu() deadlock an loadvm Thread 1 loads the saved state and calls on_vcpu(), which notifies Thread 2= before going to sleep. Thread 2 waits for "qemu_system_ready" being set, so it isn't yet waiting f= or qemu_work_cond. qemu_system_ready would only later be set by Thread 1 in kvm_main_loop(). (gdb) thread apply all bt Thread 2 (Thread 0x7fe9bcd89700 (LWP 16607)): #0 0x00007fe9c5ad716c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpt= hread.so.0 #1 0x000000000043c60a in qemu_cond_wait (_env=3D0x13e0ec0) at qemu-kvm-0.1= 4.1+dfsg/qemu-kvm.c:1107 #2 ap_main_loop (_env=3D0x13e0ec0) at qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1460 #3 0x00007fe9c5ad28ba in start_thread () from /lib/libpthread.so.0 #4 0x00007fe9c2cb002d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 1 (Thread 0x7fe9c62ea760 (LWP 16601)): #0 0x00007fe9c5ad716c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpt= hread.so.0 #1 0x0000000000438b1f in qemu_cond_wait (env=3D, func= =3D, data=3D) at qemu-kvm-0.14.1+= dfsg/qemu-kvm.c:1107 #2 on_vcpu (env=3D, func=3D, dat= a=3D) at qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1161 #3 0x000000000057a275 in cpu_synchronize_state (env=3D0x13e0ec0) at qemu-k= vm-0.14.1+dfsg/kvm.h:183 #4 get_pcr_cpu (env=3D0x13e0ec0) at qemu-kvm-0.14.1+dfsg/kvm-tpr-opt.c:213 #5 kvm_tpr_enable_vapic (env=3D0x13e0ec0) at qemu-kvm-0.14.1+dfsg/kvm-tpr-= opt.c:224 #6 0x00000000004370ae in kvm_arch_load_regs (env=3D0x13e0ec0, level=3D3) a= t ../qemu-kvm-x86.c:605 #7 0x0000000000438fe2 in kvm_cpu_synchronize_post_init (env=3D0x96de44) at= qemu-kvm-0.14.1+dfsg/qemu-kvm.c:1190 #8 0x000000000040ce58 in cpu_synchronize_post_init () at qemu-kvm-0.14.1+d= fsg/kvm.h:197 #9 cpu_synchronize_all_post_init () at qemu-kvm-0.14.1+dfsg/cpus.c:98 #10 0x00000000004a5087 in qemu_loadvm_state (f=3D0x167fa10) at savevm.c:1826 #11 0x00000000004a5281 in load_vmstate (name=3D) at sa= vevm.c:2071 #12 0x000000000041bcbc in main (argc=3D52, argv=3D, en= vp=3D) at qemu-kvm-0.14.1+dfsg/vl.c:3187 Skip on_vcpu() while the system is not yet fully initialized. This patch is only relevant for qemu-0.14.x, because the main event loop is completely rewritten in qemu-0.15 and the implementation using qemu_system_= * is replaced by commit fa7d1867578b6a1afc39d4ece8629a1e92baddd7. =2D-- a/qemu-kvm.c +++ b/qemu-kvm.c @@ -1174,7 +1174,7 @@ static void do_kvm_cpu_synchronize_state(void *_env) =20 void kvm_cpu_synchronize_state(CPUState *env) { =2D if (!env->kvm_vcpu_dirty) { + if (qemu_system_ready && !env->kvm_vcpu_dirty) { on_vcpu(env, do_kvm_cpu_synchronize_state, env); } } --Boundary-01=_xnR1OCNH8k6GnQb-- --nextPart3254970.ntbi5pWVha Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEABECAAYFAk7VGfEACgkQYPlgoZpUDjk3aACeJ/aJOPXd+rayWjf91OdolIr8 eeUAn1jzoHQbsqLNUIn/UjXxbwhXCOjc =NQ1u -----END PGP SIGNATURE----- --nextPart3254970.ntbi5pWVha--