char-socket: Fix race condition
cover char-socket: Fix race condition
Hi,
this series fixes the crash I reported yesterday in char-socket and adds a new qemu_idle_add() function, fixing a couple of memory leaks along the way.
Regards,
Berto
Alberto Garcia (3):
main-loop: Don't leak GSources attached to a GMainContext
main-loop: Add qemu_idle_add()
char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
chardev/char-pty.c | 8 ++------ chardev/char-socket.c | 19 +++++++++++++++++-- include/qemu/main-loop.h | 8 ++++++++ io/task.c | 6 ++---- util/main-loop.c | 9 +++++++++ 5 files changed, 38 insertions(+), 12 deletions(-)
Signature · 1 line
2.11.0
PATCH 1/3 main-loop: Don't leak GSources attached to a GMainContext
After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).
--- chardev/char-pty.c | 2 +- io/task.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
chardev/char-pty.c +1 -1
@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s) {if (s->open_source) {g_source_destroy (s->open_source); -g_source_unref (s->open_source); s->open_source =NULL ; } }
@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected) qemu_chr_be_generic_open_func, chr, NULL);g_source_attach (s->open_source, chr->gcontext); +g_source_unref (s->open_source); }if (!chr->gsource) { chr->gsource = io_add_watch_poll(chr, s->ioc,
io/task.c +1 -0
@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque) idle =g_idle_source_new ();g_source_set_callback (idle, qio_task_thread_result, data,NULL );g_source_attach (idle, data->context); +g_source_unref (idle);return NULL ; }
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
Quoted 4 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so
s/are not doing this/are not releasing their references/
I'll correct this if I need to send a new version of the series, otherwise please fix when committing this patch.
Berto
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
In reply to Alberto GarciaQuoted 8 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel()is the exception here because it does remove the extra reference correctly).
Quoted 5 lines
Signed-off-by Alberto Garcia <[email protected]>--- chardev/char-pty.c | 2 +- io/task.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
In reply to Alberto GarciaQuoted 10 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel()is the exception here because it does remove the extra reference correctly).Signed-off-by Alberto Garcia <[email protected]>
You could mention this is a fix for regression from a17536c594bfed94d05667b419f747b692f5fc7f
Quoted 40 lines
--- chardev/char-pty.c | 2 +- io/task.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)chardev/char-pty.c +1
-1@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s) {if (s->open_source) {g_source_destroy (s->open_source); -g_source_unref (s->open_source); s->open_source =NULL ; } }@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected) qemu_chr_be_generic_open_func, chr, NULL);g_source_attach (s->open_source, chr->gcontext); +g_source_unref (s->open_source); }if (!chr->gsource) { chr->gsource = io_add_watch_poll(chr, s->ioc,io/task.c +1
-1@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque) idle =g_idle_source_new ();g_source_set_callback (idle, qio_task_thread_result, data,NULL );g_source_attach (idle, data->context); +g_source_unref (idle);return NULL ; } --2.11.0
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
Quoted 11 lines · Marc-André Lureau
Quoted 9 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel()is the exception here because it does remove the extra reference correctly).
Quoted 5 lines
Quoted 1 line
Signed-off-by Alberto Garcia <[email protected]>You could mention this is a fix for regression from
a17536c594bfed94d05667b419f747b692f5fc7f
and 938eb9e9c83d34d90cac6ec5c5388e7998840e4e
Quoted 44 lines
Reviewed-by Marc-André Lureau <[email protected]>Quoted 40 lines
--- chardev/char-pty.c | 2 +- io/task.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)chardev/char-pty.c +1
-1@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s) {if (s->open_source) {g_source_destroy (s->open_source); -g_source_unref (s->open_source); s->open_source =NULL ; } }@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected) qemu_chr_be_generic_open_func, chr, NULL);g_source_attach (s->open_source, chr->gcontext); +g_source_unref (s->open_source); }if (!chr->gsource) { chr->gsource = io_add_watch_poll(chr, s->ioc,io/task.c +1
-1@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque) idle =g_idle_source_new ();g_source_set_callback (idle, qio_task_thread_result, data,NULL );g_source_attach (idle, data->context); +g_source_unref (idle);return NULL ; } --2.11.0
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
Quoted 13 lines · Marc-André Lureau
Quoted 11 lines · Marc-André Lureau
Quoted 9 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel()is the exception here because it does remove the extra reference correctly).
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Alberto Garcia <[email protected]>You could mention this is a fix for regression from
a17536c594bfed94d05667b419f747b692f5fc7fand
938eb9e9c83d34d90cac6ec5c5388e7998840e4e
Ok, feel free to add this to the commit message if I don't need to resend the series.
Berto
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
Hi
Quoted 17 lines · Alberto Garcia
Quoted 15 lines · Marc-André Lureau
Quoted 13 lines · Marc-André Lureau
Quoted 10 lines · Alberto Garcia
After
g_source_attach()the GMainContext holds a reference to the GSource, so the caller does not need to keep it.
pty_chr_state()andqio_task_thread_worker()are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel()is the exception here because it does remove the extra reference correctly).Signed-off-by Alberto Garcia <[email protected]>
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
You could mention this is a fix for regression from
a17536c594bfed94d05667b419f747b692f5fc7fand
938eb9e9c83d34d90cac6ec5c5388e7998840e4eOk, feel free to add this to the commit message if I don't need to resend the series.
Well, the series conflicts with the currently queued chardev series. I think I will send the pullreq and let you rebase.
(I also have a related series pending review "[PATCH v2 0/6] chardev:
various cleanups and improvements")
Signature · 1 line
Marc-André Lureau
Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext
Quoted 2 lines · Marc-André Lureau
Well, the series conflicts with the currently queued chardev series. I think I will send the pullreq and let you rebase.
I'll wait for it then
Berto
PATCH 2/3 main-loop: Add qemu_idle_add()
In reply to Alberto GarciaThis works like g_idle_add() but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.
pty_chr_state() and qio_task_thread_worker() are modified to make use of this new function.
--- chardev/char-pty.c | 8 ++------ include/qemu/main-loop.h | 8 ++++++++ io/task.c | 7 ++----- util/main-loop.c | 9 +++++++++ 4 files changed, 21 insertions(+), 11 deletions(-)
chardev/char-pty.c +2 -6
@@ -209,13 +209,9 @@ static void pty_chr_state(Chardev *chr, int connected)pty_chr_timer_cancel (s);if (!s->connected) {g_assert (s->open_source ==NULL ); - s->open_source =g_idle_source_new (); + s->open_source =qemu_idle_add (qemu_chr_be_generic_open_func, + chr, chr->gcontext); s->connected =1 ; -g_source_set_callback (s->open_source, - qemu_chr_be_generic_open_func, - chr,NULL ); -g_source_attach (s->open_source, chr->gcontext); -g_source_unref (s->open_source); }if (!chr->gsource) { chr->gsource = io_add_watch_poll(chr, s->ioc,
include/qemu/main-loop.h +8 -0
@@ -295,6 +295,14 @@ void qemu_mutex_lock_iothread_impl(const char *file, int line);*/ void qemu_mutex_unlock_iothread (void ); +/** +* qemu_idle_add: Add an idle function to a GMainContext +* +* This function is similar to GLib's g_idle_add() but it allows +* specifying a GMainContext instead of using the global one. +*/ +GSource *qemu_idle_add (GSourceFunc func,gpointer data,GMainContext *ctx); +/* internal interfaces */ void qemu_fd_register (int fd);
io/task.c +2 -5
@@ -19,6 +19,7 @@*/ #include "qemu/osdep.h" +#include "qemu/main-loop.h" #include "io/task.h" #include "qapi/error.h" #include "qemu/thread.h"
@@ -105,7 +106,6 @@ static gboolean qio_task_thread_result(gpointer opaque)static gpointer qio_task_thread_worker (gpointer opaque) {struct QIOTaskThreadData *data = opaque; -GSource *idle;trace_qio_task_thread_run (data->task); data->worker (data->task, data->opaque);
@@ -117,10 +117,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)*/ trace_qio_task_thread_exit (data->task); - idle =g_idle_source_new (); -g_source_set_callback (idle, qio_task_thread_result, data,NULL ); -g_source_attach (idle, data->context); -g_source_unref (idle); +qemu_idle_add (qio_task_thread_result, data, data->context);return NULL ; }
util/main-loop.c +9 -0
@@ -171,6 +171,15 @@ int qemu_init_main_loop(Error **errp)return 0 ; } +GSource *qemu_idle_add (GSourceFunc func,gpointer data,GMainContext *ctx) +{ +GSource *idle =g_idle_source_new (); +g_source_set_callback (idle, func, data,NULL ); +g_source_attach (idle, ctx); +g_source_unref (idle); +return idle; +} +static int max_priority;#ifndef _WIN32
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 2/3] main-loop: Add qemu_idle_add()
Quoted 7 lines · Alberto Garcia
This works like
g_idle_add()but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.
pty_chr_state()andqio_task_thread_worker()are modified to make use of this new function.
Quoted 7 lines
Signed-off-by Alberto Garcia <[email protected]>--- chardev/char-pty.c | 8 ++------ include/qemu/main-loop.h | 8 ++++++++ io/task.c | 7 ++----- util/main-loop.c | 9 +++++++++ 4 files changed, 21 insertions(+), 11 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 2/3] main-loop: Add qemu_idle_add()
In reply to Alberto GarciaHi
Quoted 41 lines · Alberto Garcia
This works like
g_idle_add()but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.
pty_chr_state()andqio_task_thread_worker()are modified to make use of this new function.Signed-off-by Alberto Garcia <[email protected]>--- chardev/char-pty.c | 8 ++------ include/qemu/main-loop.h | 8 ++++++++ io/task.c | 7 ++----- util/main-loop.c | 9 +++++++++ 4 files changed, 21 insertions(+), 11 deletions(-)chardev/char-pty.c +2
-6@@ -209,13 +209,9 @@ static void pty_chr_state(Chardev *chr, int connected)pty_chr_timer_cancel (s);if (!s->connected) {g_assert (s->open_source ==NULL ); - s->open_source =g_idle_source_new (); + s->open_source =qemu_idle_add (qemu_chr_be_generic_open_func, + chr, chr->gcontext); s->connected =1 ; -g_source_set_callback (s->open_source, - qemu_chr_be_generic_open_func, - chr,NULL ); -g_source_attach (s->open_source, chr->gcontext); -g_source_unref (s->open_source); }if (!chr->gsource) { chr->gsource = io_add_watch_poll(chr, s->ioc,include/qemu/main-loop.h +0
-0
include/qemu/main-loop.h void qemu_mutex_lock_iothread_impl(const char *file, int line); · line 304
Quoted 10 lines
include/qemu/main-loop.h +7
-0@@ -295,6 +295,14 @@ void qemu_mutex_lock_iothread_impl(const char *file, int line);*/ void qemu_mutex_unlock_iothread (void ); +/** +* qemu_idle_add: Add an idle function to a GMainContext +* +* This function is similar to GLib's g_idle_add() but it allows +* specifying a GMainContext instead of using the global one. +*/ +GSource *qemu_idle_add (GSourceFunc func,gpointer data,GMainContext *ctx);
I think you should document that returned GSource* isn't ref/owned.
Quoted 57 lines
include/qemu/main-loop.h +1
-0+/* internal interfaces */ void qemu_fd_register (int fd);io/task.c +2
-5@@ -19,6 +19,7 @@*/ #include "qemu/osdep.h" +#include "qemu/main-loop.h" #include "io/task.h" #include "qapi/error.h" #include "qemu/thread.h" @@ -105,7 +106,6 @@ static gboolean qio_task_thread_result(gpointer opaque)static gpointer qio_task_thread_worker (gpointer opaque) {struct QIOTaskThreadData *data = opaque; -GSource *idle;trace_qio_task_thread_run (data->task); data->worker (data->task, data->opaque);@@ -117,10 +117,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)*/ trace_qio_task_thread_exit (data->task); - idle =g_idle_source_new (); -g_source_set_callback (idle, qio_task_thread_result, data,NULL ); -g_source_attach (idle, data->context); -g_source_unref (idle); +qemu_idle_add (qio_task_thread_result, data, data->context);return NULL ; }util/main-loop.c +9
-1@@ -171,6 +171,15 @@ int qemu_init_main_loop(Error **errp)return 0 ; } +GSource *qemu_idle_add (GSourceFunc func,gpointer data,GMainContext *ctx) +{ +GSource *idle =g_idle_source_new (); +g_source_set_callback (idle, func, data,NULL ); +g_source_attach (idle, ctx); +g_source_unref (idle); +return idle; +} +static int max_priority;#ifndef _WIN32 --
Quoted 2 lines
2.11.0
looks ok otherwise
PATCH 3/3 char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
In reply to Alberto GarciaThere's a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread.
Running iotest 136 in a loop triggers this problem and crashes QEMU.
(gdb) bt
#0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498
#10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885
#16 0x00005558b819140d in main_loop () at vl.c:1924
#17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665
This patch adds a lock to protect tcp_chr_disconnect() and socket_reconnect_timeout()
--- chardev/char-socket.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
chardev/char-socket.c +17 -2
@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)if (ret <0 && errno !=EAGAIN ) {if (tcp_chr_read_poll (chr) <=0 ) { +qemu_mutex_unlock (&chr->chr_write_lock);tcp_chr_disconnect (chr); +qemu_mutex_lock (&chr->chr_write_lock);return len; }/* else let the read handler finish it properly */ }
@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s) } } +static gboolean tcp_chr_be_event_closed (gpointer opaque) +{ +Chardev *chr = opaque; +qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +return FALSE; +} +/* NB may be called even if tcp_chr_connect has not been * reached, due to TLS or telnet initialization failure, * so can *not* assume s->connected == true
@@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)static void tcp_chr_disconnect (Chardev *chr) {SocketChardev *s =SOCKET_CHARDEV (chr); -bool emit_close = s->connected; +bool emit_close; + +qemu_mutex_lock (&chr->chr_write_lock); + emit_close = s->connected;tcp_chr_free_connection (chr);
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr) }update_disconnected_filename (s);if (emit_close) { -qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +qemu_idle_add (tcp_chr_be_event_closed, chr, chr->gcontext); }if (s->reconnect_time) {qemu_chr_socket_restart_timer (chr); } +qemu_mutex_unlock (&chr->chr_write_lock); }static gboolean tcp_chr_read (QIOChannel *chan,GIOCondition cond,void *opaque)
@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)Chardev *chr =CHARDEV (opaque);SocketChardev *s =SOCKET_CHARDEV (opaque); +qemu_mutex_lock (&chr->chr_write_lock);g_source_unref (s->reconnect_timer); s->reconnect_timer =NULL ; +qemu_mutex_unlock (&chr->chr_write_lock);if (chr->be_open) {return false;
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 74 lines · Alberto Garcia
There's a race condition in which the
tcp_chr_read()ioc handler can close a connection that is being written to from another thread.Running iotest 136 in a loop triggers this problem and crashes QEMU.
(gdb) bt
#0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498#10
0x00005558b808920cinmonitor_qapi_event_emit(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #110x00005558b8089307inmonitor_qapi_event_queue_no_reenter(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #120x00005558b80896c0inqapi_event_emit(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#130x00005558b855f23binqapi_event_send_shutdown(guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) atqapi/qapi-events-run-state.c:43#140x00005558b81911efinqemu_system_shutdown(cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#150x00005558b8191308inmain_loop_should_exit() at vl.c:1885
#160x00005558b819140dinmain_loop() at vl.c:1924
#170x00005558b8198c84in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665This patch adds a lock to protect
tcp_chr_disconnect()andsocket_reconnect_timeout()Signed-off-by Alberto Garcia <[email protected]>--- chardev/char-socket.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)chardev/char-socket.c +13
-1@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)if (ret <0 && errno !=EAGAIN ) {if (tcp_chr_read_poll (chr) <=0 ) { +qemu_mutex_unlock (&chr->chr_write_lock);tcp_chr_disconnect (chr); +qemu_mutex_lock (&chr->chr_write_lock);return len; }/* else let the read handler finish it properly */ }@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s) } } +static gboolean tcp_chr_be_event_closed (gpointer opaque) +{ +Chardev *chr = opaque; +qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +return FALSE; +} +/* NB may be called even if tcp_chr_connect has not been * reached, due to TLS or telnet initialization failure, * so can *not* assume s->connected == true @@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)static void tcp_chr_disconnect (Chardev *chr) {SocketChardev *s =SOCKET_CHARDEV (chr); -bool emit_close = s->connected; +bool emit_close; + +qemu_mutex_lock (&chr->chr_write_lock); + emit_close = s->connected;tcp_chr_free_connection (chr);
chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472
Quoted 5 lines
chardev/char-socket.c +1
-1@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr) }update_disconnected_filename (s);if (emit_close) { -qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +qemu_idle_add (tcp_chr_be_event_closed, chr, chr->gcontext);
I'm a little concerned that this change might trigger some unexpected interactions with the qemu_chr_wait_connected() function. I added some more testing around that code, so if it passes the new tests I have in:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05962.html
and also works with the vhostuser reconnect device, then we are probably ok.
Assuming it passes the test above:
Quoted 9 lines
chardev/char-socket.c +1
-0}if (s->reconnect_time) {qemu_chr_socket_restart_timer (chr); } +qemu_mutex_unlock (&chr->chr_write_lock); }static gboolean tcp_chr_read (QIOChannel *chan,GIOCondition cond,void *opaque)
chardev/char-socket.c static gboolean socket_reconnect_timeout(gpointer opaque) · line 997
Quoted 10 lines
chardev/char-socket.c +2
-0@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)Chardev *chr =CHARDEV (opaque);SocketChardev *s =SOCKET_CHARDEV (opaque); +qemu_mutex_lock (&chr->chr_write_lock);g_source_unref (s->reconnect_timer); s->reconnect_timer =NULL ; +qemu_mutex_unlock (&chr->chr_write_lock);if (chr->be_open) {return false;
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 12 lines · Daniel P. Berrangé
chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472
Quoted 6 lines
chardev/char-socket.c +1
-1@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr) }update_disconnected_filename (s);if (emit_close) { -qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +qemu_idle_add (tcp_chr_be_event_closed, chr, chr->gcontext);I'm a little concerned that this change might trigger some unexpected interactions with the
qemu_chr_wait_connected()function. I added some more testing around that code, so if it passes the new tests I have in:
Quoted 4 lines
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05962.html
and also works with the vhostuser reconnect device, then we are probably ok.
Seems that it doesn't ...
/char/null: OK
/char/invalid: OK
/char/ringbuf: OK
/char/mux: OK
/char/stdio: OK
/char/pipe: OK
/char/file: OK
/char/file-fifo: OK
/char/udp: OK
/char/serial: OK
/char/hotswap: OK
/char/websocket: OK
/char/socket/server/mainloop/tcp: OK
/char/socket/server/mainloop/unix: OK
/char/socket/server/wait-conn/tcp: OK
/char/socket/server/wait-conn/unix: OK
/char/socket/server/mainloop-fdpass/tcp: OK
/char/socket/server/mainloop-fdpass/unix: OK
/char/socket/server/wait-conn-fdpass/tcp: OK
/char/socket/server/wait-conn-fdpass/unix: OK
/char/socket/client/mainloop/tcp: OK
/char/socket/client/mainloop/unix: OK
/char/socket/client/wait-conn/tcp: OK
/char/socket/client/wait-conn/unix: OK
/char/socket/client/mainloop-reconnect/tcp: OK
/char/socket/client/mainloop-reconnect/unix: OK
/char/socket/client/wait-conn-reconnect/tcp: Unexpected error in qio_channel_readv_all() at /home/berto/work/qemu/qemu/io/channel.c:147:
Unexpected end-of-file before all bytes were read
Berto
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
In reply to Daniel P. BerrangéQuoted 11 lines · Daniel P. Berrangé
chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472
Quoted 6 lines
chardev/char-socket.c +1
-1@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr) }update_disconnected_filename (s);if (emit_close) { -qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +qemu_idle_add (tcp_chr_be_event_closed, chr, chr->gcontext);I'm a little concerned that this change might trigger some unexpected interactions with the
qemu_chr_wait_connected()function. I added some more testing around that code, so if it passes the new tests I have in:
Forget my previous e-mail, there was a mistake on my side. The test does pass after rebasing my series on top of that one.
Quoted 2 lines
and also works with the vhostuser reconnect device, then we are probably ok.
How do I test this?
Berto
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 3 lines · Alberto Garcia
Quoted 2 lines
and also works with the vhostuser reconnect device, then we are probably ok.
How do I test this?
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-test
Paolo
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 8 lines · Paolo Bonzini
Quoted 3 lines · Alberto Garcia
Quoted 2 lines
and also works with the vhostuser reconnect device, then we are probably ok.
How do I test this?
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-test
I see. The ones enabled by QTEST_VHOST_USER_FIXME never finish with this patch applied.
Berto
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 4 lines · Alberto Garcia
Quoted 3 lines · Paolo Bonzini
Quoted 2 lines · Alberto Garcia
Quoted 2 lines
and also works with the vhostuser reconnect device, then we are probably ok.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
How do I test this?
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-testI see. The ones enabled by
QTEST_VHOST_USER_FIXMEnever finish with this patch applied.
I can look at it next week.
Paolo
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
In reply to Alberto GarciaHi
Quoted 26 lines · Alberto Garcia
There's a race condition in which the
tcp_chr_read()ioc handler can close a connection that is being written to from another thread.Running iotest 136 in a loop triggers this problem and crashes QEMU.
(gdb) bt
#0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498#10
0x00005558b808920cinmonitor_qapi_event_emit(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #110x00005558b8089307inmonitor_qapi_event_queue_no_reenter(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #120x00005558b80896c0inqapi_event_emit(event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#130x00005558b855f23binqapi_event_send_shutdown(guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) atqapi/qapi-events-run-state.c:43#140x00005558b81911efinqemu_system_shutdown(cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#150x00005558b8191308inmain_loop_should_exit() at vl.c:1885
#160x00005558b819140dinmain_loop() at vl.c:1924
#170x00005558b8198c84in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665
Quoted 4 lines
This patch adds a lock to protect
tcp_chr_disconnect()andsocket_reconnect_timeout()Signed-off-by Alberto Garcia <[email protected]>
That looks like a subset of "[Qemu-devel] Segfaults in chardev due to races". The problem is likely to occur with other chardevs I am afraid.
Quoted 73 lines
--- chardev/char-socket.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)chardev/char-socket.c +17
-3@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)if (ret <0 && errno !=EAGAIN ) {if (tcp_chr_read_poll (chr) <=0 ) { +qemu_mutex_unlock (&chr->chr_write_lock);tcp_chr_disconnect (chr); +qemu_mutex_lock (&chr->chr_write_lock);return len; }/* else let the read handler finish it properly */ }@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s) } } +static gboolean tcp_chr_be_event_closed (gpointer opaque) +{ +Chardev *chr = opaque; +qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +return FALSE; +} +/* NB may be called even if tcp_chr_connect has not been * reached, due to TLS or telnet initialization failure, * so can *not* assume s->connected == true @@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)static void tcp_chr_disconnect (Chardev *chr) {SocketChardev *s =SOCKET_CHARDEV (chr); -bool emit_close = s->connected; +bool emit_close; + +qemu_mutex_lock (&chr->chr_write_lock); + emit_close = s->connected;tcp_chr_free_connection (chr);@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr) }update_disconnected_filename (s);if (emit_close) { -qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); +qemu_idle_add (tcp_chr_be_event_closed, chr, chr->gcontext); }if (s->reconnect_time) {qemu_chr_socket_restart_timer (chr); } +qemu_mutex_unlock (&chr->chr_write_lock); }static gboolean tcp_chr_read (QIOChannel *chan,GIOCondition cond,void *opaque)@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)Chardev *chr =CHARDEV (opaque);SocketChardev *s =SOCKET_CHARDEV (opaque); +qemu_mutex_lock (&chr->chr_write_lock);g_source_unref (s->reconnect_timer); s->reconnect_timer =NULL ; +qemu_mutex_unlock (&chr->chr_write_lock);if (chr->be_open) {return false; --2.11.0
Signature · 1 line
Marc-André Lureau
Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()
Quoted 3 lines · Marc-André Lureau
That looks like a subset of "[Qemu-devel] Segfaults in chardev due to races". The problem is likely to occur with other chardevs I am afraid.
Yes, it could possibly happen with pty at least. However, Alberto's patch is a start.
Paolo