qemu-kvm: sigsegv at exit

DISCUSSION Michael S. Tsirkin 2 participants 5 messages Oct 22, 2009 – Oct 26, 2009 KVM

Share & tools

Michael S. Tsirkin

author

2009-10-22 12:02 UTC #0 lore raw

Hi!
I'm sometimes getting segfaults when I kill qemu. This time I caught it when qemu was under gdb:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)

    at /home/mst/scm/qemu-kvm/vl.c:1009

1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004 ts->next = *pt;
1005 *pt = ts;
1006
1007 /* Rearm if necessary */
1008 if (pt == &active_timers[ts->clock->type]) {
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010 qemu_rearm_alarm_timer(alarm_timer);

1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)

(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)

    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6

(gdb)

So this probably means that we have already run quit_timers:

static void quit_timers(void)

{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}

but kvm vcpu thread is still running.

Not sure what the right fix is here: should we stop kvm after main loop has exited?

Signature · 1 line

MST

Marcelo Tosatti

2009-10-22 20:57 UTC #1 lore raw

Quoted 43 lines · Michael S. Tsirkin

Hi!
I'm sometimes getting segfaults when I kill qemu. This time I caught it when qemu was under gdb:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004 ts->next = *pt;
1005 *pt = ts;
1006
1007 /* Rearm if necessary */
1008 if (pt == &active_timers[ts->clock->type]) {
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010 qemu_rearm_alarm_timer(alarm_timer);
1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)
(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
(gdb)
So this probably means that we have already run quit_timers:
static void quit_timers(void)
{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}

Quoted 5 lines

but kvm vcpu thread is still running.
Not sure what the right fix is here: should we stop kvm after main loop has exited?

kvm_main_loop_wait(env, 0) can process the stop request (signalling iothread that vcpu is stopped, so its OK to exit) and continue to kvm_cpu_exec.

Can you please try this:

qemu-kvm.c +2 -1

index 87ece3d..141c8b1 100644
@@ -1931,7 +1931,8 @@ static int kvm_main_loop_cpu(CPUState *env)
         }
         if (run_cpu) {
             kvm_main_loop_wait(env, 0);
-            kvm_cpu_exec(env);
+            if (!is_cpu_stopped(env))
+                kvm_cpu_exec(env);
         } else {
             kvm_main_loop_wait(env, 1000);
         }

Michael S. Tsirkin

author

2009-10-25 10:13 UTC #2 lore raw

Quoted 47 lines · Marcelo Tosatti

Quoted 46 lines · Michael S. Tsirkin
Hi!
I'm sometimes getting segfaults when I kill qemu. This time I caught it when qemu was under gdb:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004 ts->next = *pt;
1005 *pt = ts;
1006
1007 /* Rearm if necessary */
1008 if (pt == &active_timers[ts->clock->type]) {
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010 qemu_rearm_alarm_timer(alarm_timer);
1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)
(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
(gdb)
So this probably means that we have already run quit_timers:
static void quit_timers(void)
{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}
but kvm vcpu thread is still running.

Quoted 8 lines

Quoted 2 lines
Not sure what the right fix is here: should we stop kvm after main loop has exited?
kvm_main_loop_wait(env, 0) can process the stop request (signalling iothread that vcpu is stopped, so its OK to exit) and continue to kvm_cpu_exec.
Can you please try this:

I applied this, and have not yet see any segfaults at exit. Not sure whether this is means anything as the crash is not 100% reproducable. Push it out to Anthony and we'll see, long term? Based on the knowledge of how to fix this,
how would you go about reproducing it?

Quoted 14 lines

qemu-kvm.c +2 -1

index 87ece3d..141c8b1 100644
@@ -1931,7 +1931,8 @@ static int kvm_main_loop_cpu(CPUState *env)
         }
         if (run_cpu) {
             kvm_main_loop_wait(env, 0);
-            kvm_cpu_exec(env);
+            if (!is_cpu_stopped(env))
+                kvm_cpu_exec(env);
         } else {
             kvm_main_loop_wait(env, 1000);
         }

Marcelo Tosatti

2009-10-26 18:44 UTC #3 lore raw

Quoted 55 lines · Marcelo Tosatti

Quoted 54 lines · Michael S. Tsirkin
Quoted 43 lines
Hi!
I'm sometimes getting segfaults when I kill qemu. This time I caught it when qemu was under gdb:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004 ts->next = *pt;
1005 *pt = ts;
1006
1007 /* Rearm if necessary */
1008 if (pt == &active_timers[ts->clock->type]) {
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010 qemu_rearm_alarm_timer(alarm_timer);
1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)
(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
(gdb)
So this probably means that we have already run quit_timers:
static void quit_timers(void)
{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}
Quoted 5 lines
but kvm vcpu thread is still running.
Not sure what the right fix is here: should we stop kvm after main loop has exited?
kvm_main_loop_wait(env, 0) can process the stop request (signalling iothread that vcpu is stopped, so its OK to exit) and continue to kvm_cpu_exec.

Quoted 7 lines

Quoted 1 line
Can you please try this:
I applied this, and have not yet see any segfaults at exit. Not sure whether this is means anything as the crash is not 100% reproducable. Push it out to Anthony and we'll see, long term? Based on the knowledge of how to fix this,
how would you go about reproducing it?

Add code to trigger the race manually, but i'm pretty sure thats it.

Thanks for testing.

Michael S. Tsirkin

author

2009-10-26 18:51 UTC #4 lore raw

Quoted 57 lines · Marcelo Tosatti

Quoted 57 lines · Marcelo Tosatti
Quoted 55 lines · Michael S. Tsirkin
Quoted 43 lines
Hi!
I'm sometimes getting segfaults when I kill qemu. This time I caught it when qemu was under gdb:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004 ts->next = *pt;
1005 *pt = ts;
1006
1007 /* Rearm if necessary */
1008 if (pt == &active_timers[ts->clock->type]) {
1009 if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010 qemu_rearm_alarm_timer(alarm_timer);
1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)
(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
(gdb)
So this probably means that we have already run quit_timers:
static void quit_timers(void)
{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}
Quoted 5 lines
but kvm vcpu thread is still running.
Not sure what the right fix is here: should we stop kvm after main loop has exited?
kvm_main_loop_wait(env, 0) can process the stop request (signalling iothread that vcpu is stopped, so its OK to exit) and continue to kvm_cpu_exec.
Can you please try this:

Quoted 7 lines

Quoted 5 lines
I applied this, and have not yet see any segfaults at exit. Not sure whether this is means anything as the crash is not 100% reproducable. Push it out to Anthony and we'll see, long term? Based on the knowledge of how to fix this,
how would you go about reproducing it?
Add code to trigger the race manually,

If you like, send a patch adding such code, I will test.

Quoted 3 lines

but i'm pretty sure thats it.
Thanks for testing.