Skip to contentSkip to conversation

qemu-kvm-1.0 crashes with threaded vnc server?

DISCUSSION Peter Lieven 4 participants 11 messages Feb 9, 2012 – Jun 28, 2012 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server? I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

#1 lore raw
Quoted 12 lines · Peter Lieven

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

Quoted 9 lines

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Signature · 2 lines
Corentin Chary
http://xf.iksaif.net
#2 lore raw

Am 11.02.2012 um 09:55 schrieb Corentin Chary:

Quoted 23 lines · Peter Lieven
Quoted 12 lines

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

Quoted 9 lines

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

Quoted 8 lines

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Hi Corentin,

thanks for rebasing those patches. I remember that I have seen them the last time I noticed (about 1 year ago) that the threaded VNC is crashing. I'm on vacation this week, but I will test them next week
and let you know if I can force a crash with them applied. If not we should consider to include them asap.

Peter

Quoted 5 lines
Signature · 3 lines
-- 
Corentin Chary
http://xf.iksaif.net
#3 lore raw
Quoted 35 lines · Peter Lieven

Am 11.02.2012 um 09:55 schrieb Corentin Chary:

Quoted 23 lines · Peter Lieven
Quoted 12 lines

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

Quoted 9 lines

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

Quoted 8 lines

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 7 lines

Hi Corentin,

thanks for rebasing those patches. I remember that I have seen them the last time I noticed (about 1 year ago) that the threaded VNC is crashing. I'm on vacation this week, but I will test them next week
and let you know if I can force a crash with them applied. If not we should consider to include them asap.

Hi Peter, any news on that ?

Signature · 2 lines
Corentin Chary
http://xf.iksaif.net
#4 lore raw
Quoted 33 lines · Corentin Chary
Quoted 32 lines · Peter Lieven

Am 11.02.2012 um 09:55 schrieb Corentin Chary:

Quoted 30 lines · Peter Lieven
Quoted 12 lines

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

Quoted 9 lines

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 8 lines
Quoted 7 lines

Hi Corentin,

thanks for rebasing those patches. I remember that I have seen them the last time I noticed (about 1 year ago) that the threaded VNC is crashing. I'm on vacation this week, but I will test them next week
and let you know if I can force a crash with them applied. If not we should consider to include them asap.

Hi Peter, any news on that ?

sorry, i had much trouble debugging nasty slow windows vm problems last 2 weeks. but its still on my list. i'll keep you all posted.

peter

Quoted 2 lines
#5 lore raw
In reply to Corentin Chary
Quoted 22 lines · Corentin Chary
Quoted 12 lines · Peter Lieven

Hi,

is anyone aware if there are still problems when enabling the threaded vnc server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0 error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

Quoted 9 lines

I had no time to debug further. It seems to happen shortly after migrating, but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using libz except for VNC server.

Thanks,
Peter

Quoted 9 lines

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

I have seen no more crashes with these to patches applied. I would suggest it would be good to push them to the master repository.

Thank you,
Peter

#6 lore raw
Quoted 38 lines · Peter Lieven
Quoted 28 lines · Corentin Chary
Quoted 22 lines · Peter Lieven

Hi,

is anyone aware if there are still problems when enabling the threaded vnc
server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0
error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

I had no time to debug further. It seems to happen shortly after migrating,
but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using
libz except for VNC server.

Quoted 4 lines

Thanks,
Peter

Quoted 9 lines

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 6 lines

I have seen no more crashes with these to patches applied. I would suggest it would be good to push them to the master repository.

Thank you,
Peter

Ccing Alexander,

Signature · 2 lines
Corentin Chary
http://xf.iksaif.net
#7 lore raw
Quoted 39 lines · Corentin Chary
Quoted 38 lines · Peter Lieven
Quoted 37 lines · Corentin Chary
Quoted 22 lines · Peter Lieven

Hi,

is anyone aware if there are still problems when enabling the threaded vnc
server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0
error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

I had no time to debug further. It seems to happen shortly after migrating,
but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using
libz except for VNC server.

Quoted 4 lines

Thanks,
Peter

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 8 lines
Quoted 6 lines

I have seen no more crashes with these to patches applied. I would suggest it would be good to push them to the master repository.

Thank you,
Peter

Ccing Alexander,

Ah, cool. Corentin, I think you're right now the closest thing we have to a maintainer for VNC. Could you please just send out a pull request for those?

Alex

#8 lore raw
Quoted 39 lines · Alexander Graf
Quoted 37 lines · Corentin Chary
Quoted 36 lines · Peter Lieven
Quoted 35 lines · Corentin Chary
Quoted 21 lines · Peter Lieven

Hi,

is anyone aware if there are still problems when enabling the threaded vnc
server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0
error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+17a000]

I had no time to debug further. It seems to happen shortly after migrating,
but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using
libz except for VNC server.

Quoted 4 lines

Thanks,
Peter

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

I have seen no more crashes with these to patches applied. I would suggest it would be good to push them to the master repository.

Thank you,
Peter

Ccing Alexander,

Ah, cool. Corentin, I think you're right now the closest thing we have to a maintainer for VNC. Could you please just send out a pull request for those?

hi all,

i suspect there is still a problem with the threaded vnc server. its just a guess, but we saw a resonable number of vms hanging in the last weeks. hanging meaning the emulation is stopped and the qemu-kvm process does no longer react, not on monitor, not on vnc, not on qmp. why i suspect the threaded vnc server is that in all cases we have analyzed this happened with an open vnc session and only on nodes with the threaded vnc server enabled. it might also be the case that this happens at a resolution change. is there anything known or has someone an idea?

we are running qemu-kvm 1.0.1 with

vnc: don't mess up with iohandlers in the vnc thread

vnc: Limit r/w access to size of allocated memory

compiled in.

unfortunately, i was not yet able to reproduce this with a debugger attached.

thanks,
peter

Quoted 3 lines

Alex

#9 lore raw

Please use "gdb -p <pid>" to attach qemu when stuck and use it to dump a full backtrace. Note that I never tested qemu-kvm and the vnc code was originally not written for it, so the locking stuff may be broken if the codebase differs from upstream qemu too much.

Quoted 88 lines · Peter Lieven
Quoted 46 lines · Alexander Graf

On 13.03.2012, at 16:05, Corentin Chary wrote:

Quoted 43 lines · Peter Lieven
Quoted 43 lines · Corentin Chary
Quoted 37 lines · Peter Lieven
Quoted 22 lines

Hi,

is anyone aware if there are still problems when enabling the threaded vnc
server?
I saw some VMs crashing when using a qemu-kvm build with --enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0
error 6 in libz.so.1.2.3.3[7fec1ca75000+**16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so[7f06e09aa000+**17a000]

I had no time to debug further. It seems to happen shortly after migrating,
but thats uncertain. At least the segfault in libz seems to give a hint to VNC since I cannot image of any other part of qemu-kvm using
libz except for VNC server.

Quoted 5 lines

Thanks,
Peter

Hi Peter,

I found two patches on my git tree that I sent long ago but somehow get lost on the mailing list. I rebased the tree but did not have the time (yet) to test them.
http://git.iksaif.net/?p=qemu.**git;a=shortlog;h=refs/heads/**wip<http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip>
Feel free to try them. If QEMU segfault again, please send a full gdb backtrace / valgrind trace / way to reproduce :).
Thanks,

Quoted 1 line

I have seen no more crashes with these to patches applied. I would

suggest
it would be good to push them to the master repository.

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

Thank you,
Peter

Ccing Alexander,

Ah, cool. Corentin, I think you're right now the closest thing we have to a maintainer for VNC. Could you please just send out a pull request for those?

hi all,

i suspect there is still a problem with the threaded vnc server. its just a guess, but we saw a resonable number of vms hanging in the last weeks. hanging meaning the emulation is stopped and the qemu-kvm process does no longer react, not on monitor, not on vnc, not on qmp. why i suspect the threaded vnc server is that in all cases we have analyzed this happened with an open vnc session and only on nodes with the threaded vnc server
enabled. it might also be the case that this happens at a resolution change. is there anything known or has someone an idea?

we are running qemu-kvm 1.0.1 with

vnc: don't mess up with iohandlers in the vnc thread

vnc: Limit r/w access to size of allocated memory

compiled in.

unfortunately, i was not yet able to reproduce this with a debugger attached.

thanks,
peter

Quoted 3 lines

Alex

#10 lore raw
Quoted 3 lines · Corentin Chary

Please use "gdb -p <pid>" to attach qemu when stuck and use it to dump a full backtrace. Note that I never tested qemu-kvm and the vnc code was originally not written for it, so the locking stuff may be broken if the codebase differs from upstream qemu too much.

Hi Corentin,

it seems that this issue is not caused by the threaded vnc server. The race condition is also reproducible if the threaded vnc server is deactivated.

Sorry for the noise.

Peter

Quoted 103 lines · Le 26 juin 2012 17:02, "Peter Lieven" <[email protected] <mailto:[email protected]>> a…

On 13.03.2012 16:06, Alexander Graf wrote:

On 13.03.2012, at 16:05, Corentin Chary wrote:

On Tue, Mar 13, 2012 at 12:29 PM, Peter Lieven<[email protected] <mailto:[email protected]>> wrote:

On 11.02.2012 09:55, Corentin Chary wrote:

On Thu, Feb 9, 2012 at 7:08 PM, Peter
Lieven<[email protected] <mailto:[email protected]>> wrote:

Hi,

is anyone aware if there are still problems when enabling the threaded
vnc
server?
I saw some VMs crashing when using a qemu-kvm build with
--enable-vnc-thread.

qemu-kvm-1.0[22646]: segfault at 0 ip 00007fec1ca7ea0b sp 00007fec19d056d0
error 6 in libz.so.1.2.3.3[7fec1ca75000+16000] qemu-kvm-1.0[26056]: segfault at 7f06d8d6e010 ip 00007f06e0a30d71 sp 00007f06df035748 error 6 in libc-2.11.1.so
<http://libc-2.11.1.so>[7f06e09aa000+17a000]

I had no time to debug further. It seems to happen shortly after
migrating,
but thats uncertain. At least the segfault in libz seems to
give a hint to VNC since I cannot image of any other part of qemu-kvm
using
libz except for VNC server.

Thanks,
Peter

Hi Peter,
I found two patches on my git tree that I sent long ago but somehow
get lost on the mailing list. I rebased the tree but did not have the
time (yet) to test them.
http://git.iksaif.net/?p=qemu.git;a=shortlog;h=refs/heads/wip
Feel free to try them. If QEMU segfault again, please send a full gdb
backtrace / valgrind trace / way to reproduce :). Thanks,

I have seen no more crashes with these to patches applied. I would suggest
it would be good to push them to the master repository.

Thank you,
Peter

Ccing Alexander,

Ah, cool. Corentin, I think you're right now the closest thing we have to a maintainer for VNC. Could you please just send out a pull request for those?

hi all,

i suspect there is still a problem with the threaded vnc server. its just a guess, but we saw a resonable number of vms hanging in the last weeks. hanging meaning the emulation is stopped and the qemu-kvm process does no longer react, not on monitor, not on vnc, not on qmp.
why i suspect the threaded vnc server is that in all cases we have analyzed this happened with an open vnc session and only on nodes with the threaded vnc server
enabled. it might also be the case that this happens at a resolution change. is there anything known or has someone an idea?

we are running qemu-kvm 1.0.1 with

vnc: don't mess up with iohandlers in the vnc thread

vnc: Limit r/w access to size of allocated memory

compiled in.

unfortunately, i was not yet able to reproduce this with a debugger attached.

thanks,
peter

Alex