one out of four existing kvm guest's not starting after system upgrade

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 9 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Quoted 21 lines · Jan Kiszka

Quoted 9 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Quoted 1 line

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

Quoted 15 lines · Jan Kiszka

Quoted 9 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 7 lines

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0 0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm- all.c:1101
#2 0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd- qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:740 #4 0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0 0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main- loop.c:456
#2 0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8- amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3523

Thanks :)

Quoted 61 lines · Thomas Fjellstrom

Quoted 15 lines · Jan Kiszka

Quoted 9 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 7 lines

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0 0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm- all.c:1101
#2 0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd- qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:
740 #4 0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0 0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main- loop.c:456
#2 0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8- amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:35 23

Quoted 1 line

Thanks :)

OK, then we need a kernel view on this. Can you try

http://www.linux-kvm.org/page/Tracing

?

Thanks,
Jan

Quoted 61 lines · Thomas Fjellstrom

Quoted 15 lines · Jan Kiszka

Quoted 9 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 7 lines

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright (c) 2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0 0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm- all.c:1101
#2 0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd- qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c:
740 #4 0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0 0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main- loop.c:456
#2 0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8- amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:35 23

Quoted 1 line

Thanks :)

OK, then we need a kernel view on this. Can you try

http://www.linux-kvm.org/page/Tracing

?

Thanks,
Jan

Quoted 72 lines · Jan Kiszka

Quoted 63 lines · Thomas Fjellstrom

Quoted 16 lines · Jan Kiszka

Quoted 10 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 7 lines

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
(c) 2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0 0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm- all.c:1101
#2 0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd- qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c : 740 #4 0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0 0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main- loop.c:456
#2 0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8- amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3 5 23

Quoted 1 line

Thanks :)

OK, then we need a kernel view on this. Can you try

http://www.linux-kvm.org/page/Tracing

?

Quoted 2 lines

Thanks,
Jan

It made a rather large trace file. I'm pretty sure no-one wants me to actually link (or attach) the full 1.5G file, but the last 5000 lines might be useful, so I've compressed and attached them. (but just in case, I'm lzma'ing the entire 1.5G trace data file)

Quoted 73 lines · Thomas Fjellstrom

Quoted 72 lines · Jan Kiszka

Quoted 63 lines · Thomas Fjellstrom

Quoted 16 lines · Jan Kiszka

Quoted 10 lines · Thomas Fjellstrom

I just updated my kvm host, kernel upgraded from 2.6.38 up to 3.2, and qemu+qemu-kvm updated (not sure from what to what). But after the upgrade, one of my guests will not start up. It gets stuck with 60-80% cpu use, almost no memory is allocated by qemu/kvm, and no output of any kind is seen (in the host console, or a bunch of the guest output options like curses display, stdio output, virsh console, vnc or sdl output). I normally use libvirt to manage the guests, but I've attempted to run qemu manually, and have the same problems.

Quoted 4 lines

What can cause this?

Just tested booting back into the old kernel, the one guest still won't start, while the rest do. I'm thoroughly confused.

Quoted 7 lines

You mean if you only update qemu-kvm, the problem persists, just with lower probability? In that case, we definitely need the version of your current qemu-kvm installation. Also, it would be nice to attach gdb to the stuck qemu-kvm process, issuing a "thread apply all backtrace" in that state.

Jan

Sorry I wasn't clear. If I just update qemu-kvm (And qemu with it, and not the kernel), it always just hangs on load.

current version:
QEMU emulator version 1.0 (qemu-kvm-1.0 Debian 1.0+dfsg-8), Copyright
(c) 2003-2008 Fabrice Bellard

gdb thread apply all bt:
Thread 2 (Thread 0x7fe7b810c700 (LWP 14650)):
#0 0x00007fe7c0a11957 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c53155e9 in kvm_vcpu_ioctl (env=<optimized out>, type=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm- all.c:1101
#2 0x00007fe7c5315731 in kvm_cpu_exec (env=0x7fe7c61cb350) at /build/buildd- qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/kvm-all.c:987 #3 0x00007fe7c52ecf31 in qemu_kvm_cpu_thread_fn (arg=0x7fe7c61cb350) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/cpus.c : 740 #4 0x00007fe7c0ccdb50 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007fe7c0a1890d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe7c5101900 (LWP 14648)):
#0 0x00007fe7c0a12403 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe7c525b56c in main_loop_wait (nonblocking=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/main- loop.c:456
#2 0x00007fe7c51a372f in main_loop () at /build/buildd-qemu-kvm_1.0+dfsg-8- amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:1482
#3 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/buildd-qemu-kvm_1.0+dfsg-8-amd64-ppNMqm/qemu-kvm-1.0+dfsg/vl.c:3 5 23

Quoted 1 line

Thanks :)

OK, then we need a kernel view on this. Can you try

http://www.linux-kvm.org/page/Tracing

?

Quoted 7 lines

Quoted 2 lines

Thanks,
Jan

It made a rather large trace file. I'm pretty sure no-one wants me to actually link (or attach) the full 1.5G file, but the last 5000 lines might be useful, so I've compressed and attached them. (but just in case, I'm lzma'ing the entire 1.5G trace data file)

I'm pretty much stumped on this. So I decided to try re-creating the vm through virt-manager. Its up and running now. The only to major differences I can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) parameter, and the uuid. The rest of the parameters I played with a lot trying to get it to work by starting up the vm manually from the cli. I can't really see how those two changes would do much of anything considering the other three VM's still are configured to use -M pc-0.12, and they work fine.

I'm pretty much stumped on this. So I decided to try re-creating the vm through virt-manager. Its up and running now. The only to major differences I can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) parameter, and the uuid. The rest of the parameters I played with a lot trying to get it to work by starting up the vm manually from the cli. I can't really see how those two changes would do much of anything considering the other three VM's still are configured to use -M pc-0.12, and they work fine.

Quoted 8 lines · Thomas Fjellstrom

I'm pretty much stumped on this. So I decided to try re-creating the vm through virt-manager. Its up and running now. The only to major differences I can see in the old and new config is the machine (-M pc-0.12 vs -M pc-1.0) parameter, and the uuid. The rest of the parameters I played with a lot trying to get it to work by starting up the vm manually from the cli. I can't really see how those two changes would do much of anything considering the other three VM's still are configured to use -M pc-0.12, and they work fine.

To pick up this topic again: The trace contains no clear indication what is going on. Now I'm trying to understand what works and what not. Please correct / extend as required:

- qemu-kvm-0.12 problematic-vm.img [OK]
- qemu-kvm-1.0 -M pc-0.12 problematic-vm.img [HANG]
- qemu-kvm-1.0 -M pc-1.0 problematic-vm.img [OK]

In all cases, the image is the same, never reinstalled?

BTW, what is your guest again? What is your VM configuration?

That IOCTL error messages you find in the kernel log likely relate to direct cdrom access from the qemu process. Do you pass a host drive through?

Jan