qemu-kvm-1.0.1 - unable to exit if vcpu is in infinite loop
Hi,
i debugged my initial problem further and found out that the problem happens to be that the main thread is stuck in pause_all_vcpus() on reset or quit commands in the monitor if one cpu is stuck in the do-while loop kvm_cpu_exec. If I modify the condition from while (ret == 0) to while ((ret == 0) && !env->stop); it works, but is this the right fix?
"Quit" command seems to work, but on "Reset" the VM enterns pause state.
Thanks,
Peter
Quoted 10 lines · Peter Lieven
Hi,
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.
Before entering the wait loop in pause_all_vcpus, there are kicks sent to all vcpus. Now we need to find out why some of those kicks apparently don't reach the destination.
Again:
- on which host kernels does this occur, and which change may have changed it?
- with which qemu-kvm version is it reproducible, and which commit introduced or fixed it?
I failed reproducing so far.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 11 lines · Jan Kiszka
Quoted 10 lines · Peter Lieven
Hi,
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0)&& !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.
Quoted 3 lines
Before entering the wait loop in
pause_all_vcpus, there are kicks sent to all vcpus. Now we need to find out why some of those kicks apparently don't reach the destination.
can you explain shot what exactly these kicks do? does these kicks lead to leaving the kernel mode and returning to userspace?
Quoted 3 lines
Again:
- on which host kernels does this occur, and which change may have changed it?
I do not see it in 3.0.0 and have also not seen it in 2.6.38. both the mainline 64-bit ubuntu-server kernels (for natty / oneiric respectively). If I compile a more recent kvm-kmod 3.3 or 3.4 on these machines, it is no longer working.
Quoted 2 lines
- with which qemu-kvm version is it reproducible, and which commit introduced or fixed it?
qemu-kvm-1.0.1 from sourceforge. to get into the scenario it is not sufficient to boot from an empty harddisk. to reproduce i have use a live cd like ubuntu-server 12.04 and choose to boot from the first harddisk. i think the isolinux loader does not check for a valid bootsector and just executes what is found in sector 0. this leads to the mmio reads i posted and 100% cpu load (most spent in kernel). at that time the monitor/qmp is still responsible. if i sent a command that pauses all vcpus, the first cpu is looping in kvm_cpu_exec and the main thread is waiting. at that time the monitor stops responding. i have also seen this issue on very old windows 2000 servers where the system fails to power off and is just halted. maybe this is also a busy loop.
i will try to bisect this asap and let you know, maybe the above info helps you already to reproduce.
thanks,
peter
Quoted 4 lines
I failed reproducing so far.
Jan
Quoted 12 lines · Peter Lieven
Quoted 11 lines · Jan Kiszka
Quoted 10 lines · Peter Lieven
Hi,
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0)&& !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.
Quoted 5 lines
Quoted 3 lines
Before entering the wait loop in
pause_all_vcpus, there are kicks sent to all vcpus. Now we need to find out why some of those kicks apparently don't reach the destination.can you explain shot what exactly these kicks do? does these kicks lead to leaving the kernel mode and returning to userspace?
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
Quoted 8 lines
Quoted 3 lines
Again:
- on which host kernels does this occur, and which change may have changed it?I do not see it in 3.0.0 and have also not seen it in 2.6.38. both the mainline 64-bit ubuntu-server kernels (for natty / oneiric respectively).
If I compile a more recent kvm-kmod 3.3 or 3.4 on these machines, it is no longer working.
I was asking for kernel 3.3 or 3.4 without kvm-kmod.
Quoted 4 lines
Quoted 2 lines
- with which qemu-kvm version is it reproducible, and which commit introduced or fixed it?
qemu-kvm-1.0.1 from sourceforge. to get into the scenario it is not sufficient to boot from an empty harddisk. to reproduce
Please also try qemu-kvm git to see if something fixed it there.
Quoted 12 lines
i have use a live cd like ubuntu-server 12.04 and choose to boot from the first harddisk. i think the isolinux loader does not check for a valid bootsector and just executes what is found in sector 0. this leads to the mmio reads i posted and 100% cpu load (most spent in kernel). at that time the monitor/qmp is still responsible. if i sent a command that pauses all vcpus, the first cpu is looping in
kvm_cpu_execand the main thread is waiting. at that time the monitor stops responding. i have also seen this issue on very old windows 2000 servers where the system fails to power off and is just halted. maybe this is also a busy loop.
Quoted 2 lines
i will try to bisect this asap and let you know, maybe the above info helps you already to reproduce.
OK, thanks,
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Quoted 13 lines · Jan Kiszka
Quoted 12 lines · Peter Lieven
Quoted 11 lines · Jan Kiszka
Quoted 10 lines · Peter Lieven
Hi,
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0)&& !env->stop); it works, but is this the right fix?
"Quit" command seems to work, but on "Reset" the VM enterns pause state.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Before entering the wait loop in
pause_all_vcpus, there are kicks sent to all vcpus. Now we need to find out why some of those kicks apparently don't reach the destination.can you explain shot what exactly these kicks do? does these kicks lead to leaving the kernel mode and returning to userspace?
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
thanks
peter
Quoted 4 lines · Peter Lieven
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in vcpu_enter_guest(), see the check for signal_pending(). But this hasn't seen changes for quite a long while.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 28.06.2012 um 18:32 schrieb Avi Kivity:
Quoted 8 lines · Peter Lieven
Quoted 4 lines
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in
vcpu_enter_guest(), see the check forsignal_pending(). But this hasn't seen changes for quite a long while.
Thank you, i will have a look. I noticed a few patches that where submitted during the last year, maybe one of them is related:
Switch SIG_IPI to SIGUSR1
Fix signal handling of SIG_IPI when io-thread is enabled
In the first commit there is mentioned a "32-on-64-bit Linux kernel bug" is there any reference to that?
Thank you,
Peter
Quoted 5 lines
Signature · 2 lines
-- error compiling committee.c: too many arguments to function
Quoted 12 lines · Peter Lieven
Am 28.06.2012 um 18:32 schrieb Avi Kivity:
Quoted 8 lines · Peter Lieven
Quoted 4 lines
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in
vcpu_enter_guest(), see the check forsignal_pending(). But this hasn't seen changes for quite a long while.
Quoted 8 lines
Thank you, i will have a look. I noticed a few patches that where submitted during the last year, maybe one of them is related:
Switch
SIG_IPIto SIGUSR1
Fix signal handling ofSIG_IPIwhen io-thread is enabledIn the first commit there is mentioned a "32-on-64-bit Linux kernel bug" is there any reference to that?
http://web.archiveorange.com/archive/v/1XS1vwGSFLyYygwTXg1K. Are you running 32-on-64?
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 01.07.2012 um 10:19 schrieb Avi Kivity:
Quoted 19 lines · Peter Lieven
Quoted 18 lines
Am 28.06.2012 um 18:32 schrieb Avi Kivity:
Quoted 8 lines · Peter Lieven
Quoted 4 lines
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in
vcpu_enter_guest(), see the check forsignal_pending(). But this hasn't seen changes for quite a long while.Thank you, i will have a look. I noticed a few patches that where submitted during the last year, maybe one of them is related:
Switch
SIG_IPIto SIGUSR1
Fix signal handling ofSIG_IPIwhen io-thread is enabled
Quoted 6 lines
Quoted 2 lines
In the first commit there is mentioned a "32-on-64-bit Linux kernel bug" is there any reference to that?
http://web.archiveorange.com/archive/v/1XS1vwGSFLyYygwTXg1K. Are you running 32-on-64?
I think the issue occurs when running a 32-bit guest on a 64-bit system. Afaik, the isolinux loader where is see the race is 32-bit altough it is a 64-bit ubuntu lts cd image. The second case where i have seen the race is on shutdown of a Windows 2000 Server which is also 32-bit.
Peter
Quoted 6 lines
Signature · 2 lines
-- error compiling committee.c: too many arguments to function
Quoted 26 lines · Peter Lieven
Am 01.07.2012 um 10:19 schrieb Avi Kivity:
Quoted 23 lines · Peter Lieven
Quoted 20 lines
Am 28.06.2012 um 18:32 schrieb Avi Kivity:
Quoted 8 lines · Peter Lieven
Quoted 4 lines
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in
vcpu_enter_guest(), see the check forsignal_pending(). But this hasn't seen changes for quite a long while.Thank you, i will have a look. I noticed a few patches that where submitted during the last year, maybe one of them is related:
Switch
SIG_IPIto SIGUSR1
Fix signal handling ofSIG_IPIwhen io-thread is enabledIn the first commit there is mentioned a "32-on-64-bit Linux kernel bug" is there any reference to that?
Quoted 7 lines
Quoted 2 lines
http://web.archiveorange.com/archive/v/1XS1vwGSFLyYygwTXg1K. Are you running 32-on-64?
I think the issue occurs when running a 32-bit guest on a 64-bit system. Afaik, the isolinux loader where is see the race is 32-bit altough it is a 64-bit ubuntu lts cd image. The second case where i have seen the race is on shutdown of a Windows 2000 Server which is also 32-bit.
"32-on-64" particularly means using a 32-bit QEMU[-kvm] binary on a 64-bit host kernel. What does "file qemu-system-x86_64" report about yours?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 21 lines · Jan Kiszka
Quoted 20 lines · Peter Lieven
Am 01.07.2012 um 10:19 schrieb Avi Kivity:
Quoted 18 lines · Peter Lieven
Quoted 16 lines
Am 28.06.2012 um 18:32 schrieb Avi Kivity:
Quoted 6 lines · Peter Lieven
Quoted 3 lines
Quoted 2 lines
Yes. A signal is sent, and KVM returns from the guest to userspace on pending signals.
is there a description available how this process exactly works?
The kernel part is in
vcpu_enter_guest(), see the check forsignal_pending(). But this hasn't seen changes for quite a long while.Thank you, i will have a look. I noticed a few patches that where submitted during the last year, maybe one of them is related:
Switch
SIG_IPIto SIGUSR1
Fix signal handling ofSIG_IPIwhen io-thread is enabledIn the first commit there is mentioned a "32-on-64-bit Linux kernel bug" is there any reference to that?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
http://web.archiveorange.com/archive/v/1XS1vwGSFLyYygwTXg1K. Are you running 32-on-64?
I think the issue occurs when running a 32-bit guest on a 64-bit system. Afaik, the isolinux loader where is see the race is 32-bit altough it is a 64-bit ubuntu lts cd image. The second case where i have seen the race is on shutdown of a Windows 2000 Server which is also 32-bit.
"32-on-64" particularly means using a 32-bit QEMU[-kvm] binary on a 64-bit host kernel. What does "file qemu-system-x86_64" report about yours?
Its custom build on a 64-bit linux as 64-bit application. I will try to continue to find out today whats going wrong. Any help or hints appreciated ;-)
Thanks,
Peter
Quoted 2 lines
Jan
Quoted 8 lines · Peter Lieven
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.
I think I'm hitting something similar. I installed a F17 amd64 guest (3.5 kernel) but before booting entered the GRUB boot menu edit mode. The guest seemed unresponsive so I switched to the monitor, which also froze shortly afterwards. The VNC screen ended up being all black.
qemu-kvm.git/master 3e4305694fd891b69e4450e59ec4c65420907ede Linux 3.2.0-3-amd64 from Debian testing
$ qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive if=virtio,cache=none,file=f17.img,aio=native -serial stdio
(gdb) thread apply all bt
Thread 3 (Thread 0x7f8008e23700 (LWP 367)):
#0 0x00007f800f891727 in ioctl () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f80137b92c9 in kvm_vcpu_ioctl (env=env@entry=0x7f8015b49640, type=type@entry=44672)
at /home/stefanha/qemu-kvm/kvm-all.c:1619
#2 0x00007f80137b93fe in kvm_cpu_exec (env=env@entry=0x7f8015b49640)
at /home/stefanha/qemu-kvm/kvm-all.c:1506
#3 0x00007f8013766f31 in qemu_kvm_cpu_thread_fn (arg=0x7f8015b49640)
at /home/stefanha/qemu-kvm/cpus.c:756#4 0x00007f800fb4db50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#5 0x00007f800f8986dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
This vcpu is still executing guest code and I've seen it successfully dispatching I/O. The problem is it's missing the exit_request...
Thread 2 (Thread 0x7f8008622700 (LWP 368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f801372b229 in qemu_cond_wait (cond=<optimized out>,
mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113
#2 0x00007f8013766eff in qemu_kvm_wait_io_event (env=<optimized out>)
at /home/stefanha/qemu-kvm/cpus.c:724#3 qemu_kvm_cpu_thread_fn (arg=0x7f8015b67450) at /home/stefanha/qemu-kvm/cpus.c:761
#4 0x00007f800fb4db50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#5 0x00007f800f8986dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
No problems here.
Thread 1 (Thread 0x7f801347b8c0 (LWP 365)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f801372b229 in qemu_cond_wait (cond=cond@entry=0x7f801402fd80,
mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113
#2 0x00007f8013768949 in pause_all_vcpus () at
/home/stefanha/qemu-kvm/cpus.c:962
#3 0x00007f80136028c8 in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>) at /home/stefanha/qemu-kvm/vl.c:3695We're deadlocked in pause_all_vcpus(), waiting for vcpu #0 to pause. Unfortunately vcpu #0 has ->exit_request=0 although ->stop=1.
Here are the vcpus:
(gdb) p first_cpu
$6 = (struct CPUX86State *) 0x7f8015b49640
(gdb) p first_cpu->next_cpu
$7 = (struct CPUX86State *) 0x7f8015b67450
(gdb) p first_cpu->next_cpu->next_cpu
$8 = (struct CPUX86State *) 0x0
(gdb) p first_cpu->stop
$9 = 1
(gdb) p first_cpu->stopped
$10 = 0
(gdb) p first_cpu->exit_request
$11 = 0
:(
This isn't easy to reproduce. I tried entering the GRUB boot menu again and there was no deadlock.
Stefan
Quoted 80 lines · Stefan Hajnoczi
Quoted 8 lines · Peter Lieven
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.I think I'm hitting something similar. I installed a F17 amd64 guest (3.5 kernel) but before booting entered the GRUB boot menu edit mode. The guest seemed unresponsive so I switched to the monitor, which also froze shortly afterwards. The VNC screen ended up being all black.
qemu-kvm.git/master
3e4305694fd891b69e4450e59ec4c65420907edeLinux 3.2.0-3-amd64 from Debian testing$ qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive if=virtio,cache=none,file=f17.img,aio=native -serial stdio(gdb) thread apply all bt
Thread 3 (Thread
0x7f8008e23700(LWP 367)):
#00x00007f800f891727in ioctl () at../sysdeps/unix/syscall-template.S:82#10x00007f80137b92c9inkvm_vcpu_ioctl(env=env@entry=0x7f8015b49640, type=type@entry=44672)at /home/stefanha/qemu-kvm/kvm-all.c:1619 #2 0x00007f80137b93fe in kvm_cpu_exec (env=env@entry=0x7f8015b49640) at /home/stefanha/qemu-kvm/kvm-all.c:1506 #3 0x00007f8013766f31 in qemu_kvm_cpu_thread_fn (arg=0x7f8015b49640) at /home/stefanha/qemu-kvm/cpus.c:756#4
0x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()This vcpu is still executing guest code and I've seen it successfully dispatching I/O. The problem is it's missing the
exit_request...Thread 2 (Thread
0x7f8008622700(LWP 368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013766eff in qemu_kvm_wait_io_event (env=<optimized out>) at /home/stefanha/qemu-kvm/cpus.c:724#3
qemu_kvm_cpu_thread_fn(arg=0x7f8015b67450) at/home/stefanha/qemu-kvm/cpus.c:761
#40x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()No problems here.
Thread 1 (Thread
0x7f801347b8c0(LWP 365)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=cond@entry=0x7f801402fd80, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013768949 in pause_all_vcpus () at /home/stefanha/qemu-kvm/cpus.c:962 #3 0x00007f80136028c8 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/stefanha/qemu-kvm/vl.c:3695We're deadlocked in
pause_all_vcpus(), waiting for vcpu #0 to pause. Unfortunately vcpu #0 has ->exit_request=0 although ->stop=1.Here are the vcpus:
(gdb) p
first_cpu
$6 = (struct CPUX86State *)0x7f8015b49640
(gdb) pfirst_cpu->next_cpu
$7 = (struct CPUX86State *)0x7f8015b67450
(gdb) pfirst_cpu->next_cpu->next_cpu
$8 = (struct CPUX86State *)0x0
Quoted 6 lines
(gdb) p
first_cpu->stop
$9 = 1
(gdb) pfirst_cpu->stopped
$10 = 0
(gdb) pfirst_cpu->exit_request
$11 = 0
CPUState::exit_request is only set on specific synchronous events, see target-i386/kvm.c.
More interesting is CPUState::thread_kicked. If it's set, qemu_cpu_kick will skip the kicking via a signal. Maybe there is some race. Let me think about such possibilities again...
Jan
Quoted 8 lines
:(
This isn't easy to reproduce. I tried entering the GRUB boot menu again and there was no deadlock.
Stefan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 88 lines · Jan Kiszka
Quoted 80 lines · Stefan Hajnoczi
Quoted 8 lines · Peter Lieven
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.I think I'm hitting something similar. I installed a F17 amd64 guest (3.5 kernel) but before booting entered the GRUB boot menu edit mode. The guest seemed unresponsive so I switched to the monitor, which also froze shortly afterwards. The VNC screen ended up being all black.
qemu-kvm.git/master
3e4305694fd891b69e4450e59ec4c65420907edeLinux 3.2.0-3-amd64 from Debian testing$ qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive if=virtio,cache=none,file=f17.img,aio=native -serial stdio(gdb) thread apply all bt
Thread 3 (Thread
0x7f8008e23700(LWP 367)):
#00x00007f800f891727in ioctl () at../sysdeps/unix/syscall-template.S:82#10x00007f80137b92c9inkvm_vcpu_ioctl(env=env@entry=0x7f8015b49640, type=type@entry=44672)at /home/stefanha/qemu-kvm/kvm-all.c:1619 #2 0x00007f80137b93fe in kvm_cpu_exec (env=env@entry=0x7f8015b49640) at /home/stefanha/qemu-kvm/kvm-all.c:1506 #3 0x00007f8013766f31 in qemu_kvm_cpu_thread_fn (arg=0x7f8015b49640) at /home/stefanha/qemu-kvm/cpus.c:756#4
0x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()This vcpu is still executing guest code and I've seen it successfully dispatching I/O. The problem is it's missing the
exit_request...Thread 2 (Thread
0x7f8008622700(LWP 368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013766eff in qemu_kvm_wait_io_event (env=<optimized out>) at /home/stefanha/qemu-kvm/cpus.c:724#3
qemu_kvm_cpu_thread_fn(arg=0x7f8015b67450) at/home/stefanha/qemu-kvm/cpus.c:761
#40x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()No problems here.
Thread 1 (Thread
0x7f801347b8c0(LWP 365)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=cond@entry=0x7f801402fd80, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013768949 in pause_all_vcpus () at /home/stefanha/qemu-kvm/cpus.c:962 #3 0x00007f80136028c8 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/stefanha/qemu-kvm/vl.c:3695We're deadlocked in
pause_all_vcpus(), waiting for vcpu #0 to pause. Unfortunately vcpu #0 has ->exit_request=0 although ->stop=1.Here are the vcpus:
(gdb) p
first_cpu
$6 = (struct CPUX86State *)0x7f8015b49640
(gdb) pfirst_cpu->next_cpu
$7 = (struct CPUX86State *)0x7f8015b67450
(gdb) pfirst_cpu->next_cpu->next_cpu
$8 = (struct CPUX86State *)0x0Quoted 6 lines
(gdb) p
first_cpu->stop
$9 = 1
(gdb) pfirst_cpu->stopped
$10 = 0
(gdb) pfirst_cpu->exit_request
$11 = 0
Quoted 6 lines
CPUState::exit_requestis only set on specific synchronous events, seetarget-i386/kvm.c.More interesting is
CPUState::thread_kicked. If it's set,qemu_cpu_kickwill skip the kicking via a signal. Maybe there is some race. Let me think about such possibilities again...
cpus.c +3 -0
@@ -726,6 +726,9 @@ static void qemu_kvm_wait_io_event(CPUArchState *env) }qemu_kvm_eat_signals (env); +/* Ensure that checking env->stop cannot overtake signal processing so +* that we lose the latter without stopping. */ +smp_rmb ();qemu_wait_io_event_common (env); }
Can anyone imagine that such a barrier may actually be required? If it is currently possible that env->stop is evaluated before we called into sigtimedwait in qemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 101 lines · Jan Kiszka
Quoted 94 lines · Jan Kiszka
Quoted 80 lines · Stefan Hajnoczi
Quoted 8 lines · Peter Lieven
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.I think I'm hitting something similar. I installed a F17 amd64 guest (3.5 kernel) but before booting entered the GRUB boot menu edit mode. The guest seemed unresponsive so I switched to the monitor, which also froze shortly afterwards. The VNC screen ended up being all black.
qemu-kvm.git/master
3e4305694fd891b69e4450e59ec4c65420907edeLinux 3.2.0-3-amd64 from Debian testing$ qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive if=virtio,cache=none,file=f17.img,aio=native -serial stdio(gdb) thread apply all bt
Thread 3 (Thread
0x7f8008e23700(LWP 367)):
#00x00007f800f891727in ioctl () at../sysdeps/unix/syscall-template.S:82#10x00007f80137b92c9inkvm_vcpu_ioctl(env=env@entry=0x7f8015b49640, type=type@entry=44672)at /home/stefanha/qemu-kvm/kvm-all.c:1619 #2 0x00007f80137b93fe in kvm_cpu_exec (env=env@entry=0x7f8015b49640) at /home/stefanha/qemu-kvm/kvm-all.c:1506 #3 0x00007f8013766f31 in qemu_kvm_cpu_thread_fn (arg=0x7f8015b49640) at /home/stefanha/qemu-kvm/cpus.c:756#4
0x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()This vcpu is still executing guest code and I've seen it successfully dispatching I/O. The problem is it's missing the
exit_request...Thread 2 (Thread
0x7f8008622700(LWP 368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013766eff in qemu_kvm_wait_io_event (env=<optimized out>) at /home/stefanha/qemu-kvm/cpus.c:724#3
qemu_kvm_cpu_thread_fn(arg=0x7f8015b67450) at/home/stefanha/qemu-kvm/cpus.c:761
#40x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()No problems here.
Thread 1 (Thread
0x7f801347b8c0(LWP 365)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=cond@entry=0x7f801402fd80, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013768949 in pause_all_vcpus () at /home/stefanha/qemu-kvm/cpus.c:962 #3 0x00007f80136028c8 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/stefanha/qemu-kvm/vl.c:3695We're deadlocked in
pause_all_vcpus(), waiting for vcpu #0 to pause. Unfortunately vcpu #0 has ->exit_request=0 although ->stop=1.Here are the vcpus:
(gdb) p
first_cpu
$6 = (struct CPUX86State *)0x7f8015b49640
(gdb) pfirst_cpu->next_cpu
$7 = (struct CPUX86State *)0x7f8015b67450
(gdb) pfirst_cpu->next_cpu->next_cpu
$8 = (struct CPUX86State *)0x0Quoted 6 lines
(gdb) p
first_cpu->stop
$9 = 1
(gdb) pfirst_cpu->stopped
$10 = 0
(gdb) pfirst_cpu->exit_request
$11 = 0
CPUState::exit_requestis only set on specific synchronous events, seetarget-i386/kvm.c.More interesting is
CPUState::thread_kicked. If it's set,qemu_cpu_kickwill skip the kicking via a signal. Maybe there is some race. Let me think about such possibilities again...cpus.c +0
-0
cpus.c static void qemu_kvm_wait_io_event(CPUArchState *env) · line 731
Quoted 6 lines
cpus.c +3
-0@@ -726,6 +726,9 @@ static void qemu_kvm_wait_io_event(CPUArchState *env) }qemu_kvm_eat_signals (env); +/* Ensure that checking env->stop cannot overtake signal processing so +* that we lose the latter without stopping. */ +smp_rmb ();
rmb is nonsense. Should be a plain barrier() - if at all.
Quoted 7 lines
cpus.c +0
-0qemu_wait_io_event_common (env); }Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 104 lines · Jan Kiszka
Quoted 103 lines · Jan Kiszka
Quoted 94 lines · Jan Kiszka
Quoted 80 lines · Stefan Hajnoczi
Quoted 8 lines · Peter Lieven
i debugged my initial problem further and found out that the problem happens to be that
the main thread is stuck inpause_all_vcpus()on reset or quit commands in the monitor
if one cpu is stuck in the do-while loopkvm_cpu_exec. If I modify the condition from while (ret == 0)
to while ((ret == 0) && !env->stop); it works, but is this the right fix? "Quit" command seems to work, but on "Reset" the VM enterns pause state.I think I'm hitting something similar. I installed a F17 amd64 guest (3.5 kernel) but before booting entered the GRUB boot menu edit mode. The guest seemed unresponsive so I switched to the monitor, which also froze shortly afterwards. The VNC screen ended up being all black.
qemu-kvm.git/master
3e4305694fd891b69e4450e59ec4c65420907edeLinux 3.2.0-3-amd64 from Debian testing$ qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive if=virtio,cache=none,file=f17.img,aio=native -serial stdio(gdb) thread apply all bt
Thread 3 (Thread
0x7f8008e23700(LWP 367)):
#00x00007f800f891727in ioctl () at../sysdeps/unix/syscall-template.S:82#10x00007f80137b92c9inkvm_vcpu_ioctl(env=env@entry=0x7f8015b49640, type=type@entry=44672)at /home/stefanha/qemu-kvm/kvm-all.c:1619 #2 0x00007f80137b93fe in kvm_cpu_exec (env=env@entry=0x7f8015b49640) at /home/stefanha/qemu-kvm/kvm-all.c:1506 #3 0x00007f8013766f31 in qemu_kvm_cpu_thread_fn (arg=0x7f8015b49640) at /home/stefanha/qemu-kvm/cpus.c:756#4
0x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()This vcpu is still executing guest code and I've seen it successfully dispatching I/O. The problem is it's missing the
exit_request...Thread 2 (Thread
0x7f8008622700(LWP 368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013766eff in qemu_kvm_wait_io_event (env=<optimized out>) at /home/stefanha/qemu-kvm/cpus.c:724#3
qemu_kvm_cpu_thread_fn(arg=0x7f8015b67450) at/home/stefanha/qemu-kvm/cpus.c:761
#40x00007f800fb4db50instart_thread(arg=<optimized out>) at pthread_create.c:304
#50x00007f800f8986ddin clone () at../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#60x0000000000000000in ?? ()No problems here.
Thread 1 (Thread
0x7f801347b8c0(LWP 365)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f801372b229 in qemu_cond_wait (cond=cond@entry=0x7f801402fd80, mutex=mutex@entry=0x7f80144367c0) at qemu-thread-posix.c:113 #2 0x00007f8013768949 in pause_all_vcpus () at /home/stefanha/qemu-kvm/cpus.c:962 #3 0x00007f80136028c8 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/stefanha/qemu-kvm/vl.c:3695We're deadlocked in
pause_all_vcpus(), waiting for vcpu #0 to pause. Unfortunately vcpu #0 has ->exit_request=0 although ->stop=1.Here are the vcpus:
(gdb) p
first_cpu
$6 = (struct CPUX86State *)0x7f8015b49640
(gdb) pfirst_cpu->next_cpu
$7 = (struct CPUX86State *)0x7f8015b67450
(gdb) pfirst_cpu->next_cpu->next_cpu
$8 = (struct CPUX86State *)0x0Quoted 6 lines
(gdb) p
first_cpu->stop
$9 = 1
(gdb) pfirst_cpu->stopped
$10 = 0
(gdb) pfirst_cpu->exit_request
$11 = 0
CPUState::exit_requestis only set on specific synchronous events, seetarget-i386/kvm.c.More interesting is
CPUState::thread_kicked. If it's set,qemu_cpu_kickwill skip the kicking via a signal. Maybe there is some race. Let me think about such possibilities again...cpus.c +0
-0@@ -726,6 +726,9 @@ static void qemu_kvm_wait_io_event(CPUArchState *env) }
Quoted 14 lines
cpus.c static void qemu_kvm_wait_io_event(CPUArchState *env) · line 731
Quoted 4 lines
cpus.c +3
-0qemu_kvm_eat_signals (env); +/* Ensure that checking env->stop cannot overtake signal processing so +* that we lose the latter without stopping. */ +smp_rmb ();rmb is nonsense. Should be a plain
barrier()- if at all.Quoted 7 lines
cpus.c +0
-0qemu_wait_io_event_common (env); }Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).
Should not be required (TM): Both signal eating / stop checking and stop setting / signal generation happens under the BQL, thus the ordering must not make a difference here.
Don't see where we could lose a signal. Maybe due to a subtle memory corruption that sets thread_kicked to non-zero, preventing the kicking this way.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 5 lines
Quoted 5 lines
Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).Should not be required (TM): Both signal eating / stop checking and stop setting / signal generation happens under the BQL, thus the ordering must not make a difference here.
Agree.
Quoted 3 lines
Don't see where we could lose a signal. Maybe due to a subtle memory corruption that sets
thread_kickedto non-zero, preventing the kicking this way.
Cannot be ruled out, yet too much of a coincidence.
Could be a kernel bug (either in kvm or elsewhere), we've had several before in this area.
Is this reproducible?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 9 lines · Jan Kiszka
Quoted 5 lines
Quoted 5 lines
Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).Should not be required (TM): Both signal eating / stop checking and stop setting / signal generation happens under the BQL, thus the ordering must not make a difference here.
Agree.
Quoted 3 lines
Don't see where we could lose a signal. Maybe due to a subtle memory corruption that sets
thread_kickedto non-zero, preventing the kicking this way.
Quoted 6 lines
Cannot be ruled out, yet too much of a coincidence.
Could be a kernel bug (either in kvm or elsewhere), we've had several before in this area.
Is this reproducible?
Not for me. Peter only hit it very rarely, Peter obviously more easily.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Avi Kivity
Quoted 9 lines · Jan Kiszka
Quoted 5 lines
Quoted 5 lines
Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).Should not be required (TM): Both signal eating / stop checking and stop setting / signal generation happens under the BQL, thus the ordering must not make a difference here.
Agree.
Quoted 3 lines
Don't see where we could lose a signal. Maybe due to a subtle memory corruption that sets
thread_kickedto non-zero, preventing the kicking this way.
Quoted 8 lines
Quoted 6 lines
Cannot be ruled out, yet too much of a coincidence.
Could be a kernel bug (either in kvm or elsewhere), we've had several before in this area.
Is this reproducible?
Not for me. Peter only hit it very rarely, Peter obviously more easily.
I have only hit this once and was not able to reproduce it.
Stefan
Quoted 16 lines · Stefan Hajnoczi
Quoted 15 lines · Jan Kiszka
Quoted 14 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 4 lines
Quoted 4 lines
Can anyone imagine that such a barrier may actually be required? If it is currently possible that
env->stopis evaluated before we called into sigtimedwait inqemu_kvm_eat_signals, then we could actually eat the signal without properly processing its reason (stop).Should not be required (TM): Both signal eating / stop checking and stop setting / signal generation happens under the BQL, thus the ordering must not make a difference here.
Agree.
Quoted 3 lines
Don't see where we could lose a signal. Maybe due to a subtle memory corruption that sets
thread_kickedto non-zero, preventing the kicking this way.
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Cannot be ruled out, yet too much of a coincidence.
Could be a kernel bug (either in kvm or elsewhere), we've had several before in this area.
Is this reproducible?
Not for me. Peter only hit it very rarely, Peter obviously more easily.
I have only hit this once and was not able to reproduce it.
For me it was very reproducible, but my issue was fixed by:
http://www.mail-archive.com/[email protected]/msg70908.html
Never seen this since then,
Peter
Quoted 1 line
Stefan