Skip to contentSkip to conversation

char-socket: Fix race condition

PATCH Superseded Alberto Garcia 5 participants 21 messages Feb 7, 2019 – Feb 8, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover char-socket: Fix race condition

Hi,

this series fixes the crash I reported yesterday in char-socket and adds a new qemu_idle_add() function, fixing a couple of memory leaks along the way.

Regards,

Berto

Alberto Garcia (3):
main-loop: Don't leak GSources attached to a GMainContext
main-loop: Add qemu_idle_add()
char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

 chardev/char-pty.c       |  8 ++------
 chardev/char-socket.c    | 19 +++++++++++++++++--
 include/qemu/main-loop.h |  8 ++++++++
 io/task.c                |  6 ++----
 util/main-loop.c         |  9 +++++++++
 5 files changed, 38 insertions(+), 12 deletions(-)
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/3 main-loop: Don't leak GSources attached to a GMainContext

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

---
 chardev/char-pty.c | 2 +-
 io/task.c          | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
chardev/char-pty.c +1 -1
index f681d637c1..f16a5e8d59 100644
@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s)
 {
     if (s->open_source) {
         g_source_destroy(s->open_source);
-        g_source_unref(s->open_source);
         s->open_source = NULL;
     }
 }
@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected)
                                   qemu_chr_be_generic_open_func,
                                   chr, NULL);
             g_source_attach(s->open_source, chr->gcontext);
+            g_source_unref(s->open_source);
         }
         if (!chr->gsource) {
             chr->gsource = io_add_watch_poll(chr, s->ioc,
io/task.c +1 -0
index 2886a2c1bc..c8489fb790 100644
@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)
     idle = g_idle_source_new();
     g_source_set_callback(idle, qio_task_thread_result, data, NULL);
     g_source_attach(idle, data->context);
+    g_source_unref(idle);
 
     return NULL;
 }
Signature · 1 line
2.11.0
#2 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

Quoted 4 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so

s/are not doing this/are not releasing their references/

I'll correct this if I need to send a new version of the series, otherwise please fix when committing this patch.

Berto

Reviewed-by on v1 1/3
#3 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

In reply to Alberto Garcia
Quoted 8 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

Quoted 5 lines
---
 chardev/char-pty.c | 2 +-
 io/task.c          | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v1 1/3
#4 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

In reply to Alberto Garcia
Quoted 10 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

You could mention this is a fix for regression from a17536c594bfed94d05667b419f747b692f5fc7f

Quoted 40 lines
---
 chardev/char-pty.c | 2 +-
 io/task.c          | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
chardev/char-pty.c +1 -1
index f681d637c1..f16a5e8d59 100644
@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s)
 {
     if (s->open_source) {
         g_source_destroy(s->open_source);
-        g_source_unref(s->open_source);
         s->open_source = NULL;
     }
 }
@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected)
                                   qemu_chr_be_generic_open_func,
                                   chr, NULL);
             g_source_attach(s->open_source, chr->gcontext);
+            g_source_unref(s->open_source);
         }
         if (!chr->gsource) {
             chr->gsource = io_add_watch_poll(chr, s->ioc,
io/task.c +1 -1
index 2886a2c1bc..c8489fb790 100644
@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)
     idle = g_idle_source_new();
     g_source_set_callback(idle, qio_task_thread_result, data, NULL);
     g_source_attach(idle, data->context);
+    g_source_unref(idle);
 
     return NULL;
 }
--

2.11.0

#5 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

Quoted 11 lines · Marc-André Lureau
Quoted 9 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

Quoted 5 lines
Quoted 1 line

You could mention this is a fix for regression from a17536c594bfed94d05667b419f747b692f5fc7f

and 938eb9e9c83d34d90cac6ec5c5388e7998840e4e

Quoted 44 lines
Quoted 40 lines
---
 chardev/char-pty.c | 2 +-
 io/task.c          | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
chardev/char-pty.c +1 -1
index f681d637c1..f16a5e8d59 100644
@@ -60,7 +60,6 @@ static void pty_chr_open_src_cancel(PtyChardev *s)
 {
     if (s->open_source) {
         g_source_destroy(s->open_source);
-        g_source_unref(s->open_source);
         s->open_source = NULL;
     }
 }
@@ -216,6 +215,7 @@ static void pty_chr_state(Chardev *chr, int connected)
                                   qemu_chr_be_generic_open_func,
                                   chr, NULL);
             g_source_attach(s->open_source, chr->gcontext);
+            g_source_unref(s->open_source);
         }
         if (!chr->gsource) {
             chr->gsource = io_add_watch_poll(chr, s->ioc,
io/task.c +1 -1
index 2886a2c1bc..c8489fb790 100644
@@ -120,6 +120,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)
     idle = g_idle_source_new();
     g_source_set_callback(idle, qio_task_thread_result, data, NULL);
     g_source_attach(idle, data->context);
+    g_source_unref(idle);
 
     return NULL;
 }
--

2.11.0

#6 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

Quoted 13 lines · Marc-André Lureau
Quoted 11 lines · Marc-André Lureau
Quoted 9 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

You could mention this is a fix for regression from a17536c594bfed94d05667b419f747b692f5fc7f

and 938eb9e9c83d34d90cac6ec5c5388e7998840e4e

Ok, feel free to add this to the commit message if I don't need to resend the series.

Berto

#7 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

Hi

Quoted 17 lines · Alberto Garcia
Quoted 15 lines · Marc-André Lureau
Quoted 13 lines · Marc-André Lureau
Quoted 10 lines · Alberto Garcia

After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it.

pty_chr_state() and qio_task_thread_worker() are not doing this, so the GSource is being leaked in both cases (pty_chr_open_src_cancel() is the exception here because it does remove the extra reference correctly).

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

You could mention this is a fix for regression from a17536c594bfed94d05667b419f747b692f5fc7f

and 938eb9e9c83d34d90cac6ec5c5388e7998840e4e

Ok, feel free to add this to the commit message if I don't need to resend the series.

Well, the series conflicts with the currently queued chardev series. I think I will send the pullreq and let you rebase.

(I also have a related series pending review "[PATCH v2 0/6] chardev:
various cleanups and improvements")

Signature · 1 line
Marc-André Lureau
#8 lore raw

Re: [Qemu-devel] [PATCH 1/3] main-loop: Don't leak GSources attached to a GMainContext

Quoted 2 lines · Marc-André Lureau

Well, the series conflicts with the currently queued chardev series. I think I will send the pullreq and let you rebase.

I'll wait for it then

Berto

#9 lore raw

PATCH 2/3 main-loop: Add qemu_idle_add()

In reply to Alberto Garcia

This works like g_idle_add() but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.

pty_chr_state() and qio_task_thread_worker() are modified to make use of this new function.

---
 chardev/char-pty.c       | 8 ++------
 include/qemu/main-loop.h | 8 ++++++++
 io/task.c                | 7 ++-----
 util/main-loop.c         | 9 +++++++++
 4 files changed, 21 insertions(+), 11 deletions(-)
chardev/char-pty.c +2 -6
index f16a5e8d59..6562c38f39 100644
@@ -209,13 +209,9 @@ static void pty_chr_state(Chardev *chr, int connected)
         pty_chr_timer_cancel(s);
         if (!s->connected) {
             g_assert(s->open_source == NULL);
-            s->open_source = g_idle_source_new();
+            s->open_source = qemu_idle_add(qemu_chr_be_generic_open_func,
+                                           chr, chr->gcontext);
             s->connected = 1;
-            g_source_set_callback(s->open_source,
-                                  qemu_chr_be_generic_open_func,
-                                  chr, NULL);
-            g_source_attach(s->open_source, chr->gcontext);
-            g_source_unref(s->open_source);
         }
         if (!chr->gsource) {
             chr->gsource = io_add_watch_poll(chr, s->ioc,
include/qemu/main-loop.h +8 -0
index e59f9ae1e9..1865afd993 100644
@@ -295,6 +295,14 @@ void qemu_mutex_lock_iothread_impl(const char *file, int line);
  */
 void qemu_mutex_unlock_iothread(void);
 
+/**
+ * qemu_idle_add: Add an idle function to a GMainContext
+ *
+ * This function is similar to GLib's g_idle_add() but it allows
+ * specifying a GMainContext instead of using the global one.
+ */
+GSource *qemu_idle_add(GSourceFunc func, gpointer data, GMainContext *ctx);
+
 /* internal interfaces */
 
 void qemu_fd_register(int fd);
io/task.c +2 -5
index c8489fb790..bdd395b0ad 100644
@@ -19,6 +19,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/main-loop.h"
 #include "io/task.h"
 #include "qapi/error.h"
 #include "qemu/thread.h"
@@ -105,7 +106,6 @@ static gboolean qio_task_thread_result(gpointer opaque)
 static gpointer qio_task_thread_worker(gpointer opaque)
 {
     struct QIOTaskThreadData *data = opaque;
-    GSource *idle;
 
     trace_qio_task_thread_run(data->task);
     data->worker(data->task, data->opaque);
@@ -117,10 +117,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)
      */
     trace_qio_task_thread_exit(data->task);
 
-    idle = g_idle_source_new();
-    g_source_set_callback(idle, qio_task_thread_result, data, NULL);
-    g_source_attach(idle, data->context);
-    g_source_unref(idle);
+    qemu_idle_add(qio_task_thread_result, data, data->context);
 
     return NULL;
 }
util/main-loop.c +9 -0
index 443cb4cfe8..1c68e3bf5f 100644
@@ -171,6 +171,15 @@ int qemu_init_main_loop(Error **errp)
     return 0;
 }
 
+GSource *qemu_idle_add(GSourceFunc func, gpointer data, GMainContext *ctx)
+{
+    GSource *idle = g_idle_source_new();
+    g_source_set_callback(idle, func, data, NULL);
+    g_source_attach(idle, ctx);
+    g_source_unref(idle);
+    return idle;
+}
+
 static int max_priority;
 
 #ifndef _WIN32
Signature · 1 line
2.11.0
Reviewed-by on v1 2/3
#10 lore raw

Re: [Qemu-devel] [PATCH 2/3] main-loop: Add qemu_idle_add()

Quoted 7 lines · Alberto Garcia

This works like g_idle_add() but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.

pty_chr_state() and qio_task_thread_worker() are modified to make use of this new function.

Quoted 7 lines
---
 chardev/char-pty.c       | 8 ++------
 include/qemu/main-loop.h | 8 ++++++++
 io/task.c                | 7 ++-----
 util/main-loop.c         | 9 +++++++++
 4 files changed, 21 insertions(+), 11 deletions(-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v1 2/3
#11 lore raw

Re: [Qemu-devel] [PATCH 2/3] main-loop: Add qemu_idle_add()

In reply to Alberto Garcia

Hi

Quoted 41 lines · Alberto Garcia

This works like g_idle_add() but allows specifying a different GMainContext. It also returns the GSource directly instead of its ID number for convenience.

pty_chr_state() and qio_task_thread_worker() are modified to make use of this new function.

---
 chardev/char-pty.c       | 8 ++------
 include/qemu/main-loop.h | 8 ++++++++
 io/task.c                | 7 ++-----
 util/main-loop.c         | 9 +++++++++
 4 files changed, 21 insertions(+), 11 deletions(-)
chardev/char-pty.c +2 -6
index f16a5e8d59..6562c38f39 100644
@@ -209,13 +209,9 @@ static void pty_chr_state(Chardev *chr, int connected)
         pty_chr_timer_cancel(s);
         if (!s->connected) {
             g_assert(s->open_source == NULL);
-            s->open_source = g_idle_source_new();
+            s->open_source = qemu_idle_add(qemu_chr_be_generic_open_func,
+                                           chr, chr->gcontext);
             s->connected = 1;
-            g_source_set_callback(s->open_source,
-                                  qemu_chr_be_generic_open_func,
-                                  chr, NULL);
-            g_source_attach(s->open_source, chr->gcontext);
-            g_source_unref(s->open_source);
         }
         if (!chr->gsource) {
             chr->gsource = io_add_watch_poll(chr, s->ioc,
include/qemu/main-loop.h +0 -0
index e59f9ae1e9..1865afd993 100644

include/qemu/main-loop.h void qemu_mutex_lock_iothread_impl(const char *file, int line); · line 304

Quoted 10 lines
include/qemu/main-loop.h +7 -0
@@ -295,6 +295,14 @@ void qemu_mutex_lock_iothread_impl(const char *file, int line);
  */
 void qemu_mutex_unlock_iothread(void);
 
+/**
+ * qemu_idle_add: Add an idle function to a GMainContext
+ *
+ * This function is similar to GLib's g_idle_add() but it allows
+ * specifying a GMainContext instead of using the global one.
+ */
+GSource *qemu_idle_add(GSourceFunc func, gpointer data, GMainContext *ctx);

I think you should document that returned GSource* isn't ref/owned.

Quoted 57 lines
include/qemu/main-loop.h +1 -0
+
 /* internal interfaces */
 
 void qemu_fd_register(int fd);
io/task.c +2 -5
index c8489fb790..bdd395b0ad 100644
@@ -19,6 +19,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/main-loop.h"
 #include "io/task.h"
 #include "qapi/error.h"
 #include "qemu/thread.h"
@@ -105,7 +106,6 @@ static gboolean qio_task_thread_result(gpointer opaque)
 static gpointer qio_task_thread_worker(gpointer opaque)
 {
     struct QIOTaskThreadData *data = opaque;
-    GSource *idle;
 
     trace_qio_task_thread_run(data->task);
     data->worker(data->task, data->opaque);
@@ -117,10 +117,7 @@ static gpointer qio_task_thread_worker(gpointer opaque)
      */
     trace_qio_task_thread_exit(data->task);
 
-    idle = g_idle_source_new();
-    g_source_set_callback(idle, qio_task_thread_result, data, NULL);
-    g_source_attach(idle, data->context);
-    g_source_unref(idle);
+    qemu_idle_add(qio_task_thread_result, data, data->context);
 
     return NULL;
 }
util/main-loop.c +9 -1
index 443cb4cfe8..1c68e3bf5f 100644
@@ -171,6 +171,15 @@ int qemu_init_main_loop(Error **errp)
     return 0;
 }
 
+GSource *qemu_idle_add(GSourceFunc func, gpointer data, GMainContext *ctx)
+{
+    GSource *idle = g_idle_source_new();
+    g_source_set_callback(idle, func, data, NULL);
+    g_source_attach(idle, ctx);
+    g_source_unref(idle);
+    return idle;
+}
+
 static int max_priority;
 
 #ifndef _WIN32
--
Quoted 2 lines

2.11.0

looks ok otherwise

#12 lore raw

PATCH 3/3 char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

In reply to Alberto Garcia

There's a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread.

Running iotest 136 in a loop triggers this problem and crashes QEMU.

(gdb) bt

 #0  0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860
 #1  0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76
 #2  0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123
 #3  0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135
 #4  0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112
 #5  0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147
 #6  0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42
 #7  0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406
 #8  0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449
 #9  0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498

#10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885
#16 0x00005558b819140d in main_loop () at vl.c:1924
#17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665

This patch adds a lock to protect tcp_chr_disconnect() and socket_reconnect_timeout()

---
 chardev/char-socket.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
chardev/char-socket.c +17 -2
index eaa8e8b68f..5f421bdc58 100644
@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)
 
         if (ret < 0 && errno != EAGAIN) {
             if (tcp_chr_read_poll(chr) <= 0) {
+                qemu_mutex_unlock(&chr->chr_write_lock);
                 tcp_chr_disconnect(chr);
+                qemu_mutex_lock(&chr->chr_write_lock);
                 return len;
             } /* else let the read handler finish it properly */
         }
@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s)
     }
 }
 
+static gboolean tcp_chr_be_event_closed(gpointer opaque)
+{
+    Chardev *chr = opaque;
+    qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+    return FALSE;
+}
+
 /* NB may be called even if tcp_chr_connect has not been
  * reached, due to TLS or telnet initialization failure,
  * so can *not* assume s->connected == true
@@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)
 static void tcp_chr_disconnect(Chardev *chr)
 {
     SocketChardev *s = SOCKET_CHARDEV(chr);
-    bool emit_close = s->connected;
+    bool emit_close;
+
+    qemu_mutex_lock(&chr->chr_write_lock);
+    emit_close = s->connected;
 
     tcp_chr_free_connection(chr);
 
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr)
     }
     update_disconnected_filename(s);
     if (emit_close) {
-        qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+        qemu_idle_add(tcp_chr_be_event_closed, chr, chr->gcontext);
     }
     if (s->reconnect_time) {
         qemu_chr_socket_restart_timer(chr);
     }
+    qemu_mutex_unlock(&chr->chr_write_lock);
 }
 
 static gboolean tcp_chr_read(QIOChannel *chan, GIOCondition cond, void *opaque)
@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)
     Chardev *chr = CHARDEV(opaque);
     SocketChardev *s = SOCKET_CHARDEV(opaque);
 
+    qemu_mutex_lock(&chr->chr_write_lock);
     g_source_unref(s->reconnect_timer);
     s->reconnect_timer = NULL;
+    qemu_mutex_unlock(&chr->chr_write_lock);
 
     if (chr->be_open) {
         return false;
Signature · 1 line
2.11.0
Reviewed-by on v1 3/3
#13 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 74 lines · Alberto Garcia

There's a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread.

Running iotest 136 in a loop triggers this problem and crashes QEMU.

(gdb) bt

 #0  0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860
 #1  0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76
 #2  0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123
 #3  0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135
 #4  0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112
 #5  0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147
 #6  0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42
 #7  0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406
 #8  0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449
 #9  0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498

#10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885
#16 0x00005558b819140d in main_loop () at vl.c:1924
#17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665

This patch adds a lock to protect tcp_chr_disconnect() and socket_reconnect_timeout()

---
 chardev/char-socket.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
chardev/char-socket.c +13 -1
index eaa8e8b68f..5f421bdc58 100644
@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)
 
         if (ret < 0 && errno != EAGAIN) {
             if (tcp_chr_read_poll(chr) <= 0) {
+                qemu_mutex_unlock(&chr->chr_write_lock);
                 tcp_chr_disconnect(chr);
+                qemu_mutex_lock(&chr->chr_write_lock);
                 return len;
             } /* else let the read handler finish it properly */
         }
@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s)
     }
 }
 
+static gboolean tcp_chr_be_event_closed(gpointer opaque)
+{
+    Chardev *chr = opaque;
+    qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+    return FALSE;
+}
+
 /* NB may be called even if tcp_chr_connect has not been
  * reached, due to TLS or telnet initialization failure,
  * so can *not* assume s->connected == true
@@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)
 static void tcp_chr_disconnect(Chardev *chr)
 {
     SocketChardev *s = SOCKET_CHARDEV(chr);
-    bool emit_close = s->connected;
+    bool emit_close;
+
+    qemu_mutex_lock(&chr->chr_write_lock);
+    emit_close = s->connected;
 
     tcp_chr_free_connection(chr);
 

chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472

Quoted 5 lines
chardev/char-socket.c +1 -1
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr)
     }
     update_disconnected_filename(s);
     if (emit_close) {
-        qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+        qemu_idle_add(tcp_chr_be_event_closed, chr, chr->gcontext);

I'm a little concerned that this change might trigger some unexpected interactions with the qemu_chr_wait_connected() function. I added some more testing around that code, so if it passes the new tests I have in:

https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05962.html

and also works with the vhostuser reconnect device, then we are probably ok.

Assuming it passes the test above:

Quoted 9 lines
chardev/char-socket.c +1 -0
     }
     if (s->reconnect_time) {
         qemu_chr_socket_restart_timer(chr);
     }
+    qemu_mutex_unlock(&chr->chr_write_lock);
 }
 
 static gboolean tcp_chr_read(QIOChannel *chan, GIOCondition cond, void *opaque)

chardev/char-socket.c static gboolean socket_reconnect_timeout(gpointer opaque) · line 997

Quoted 10 lines
chardev/char-socket.c +2 -0
@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)
     Chardev *chr = CHARDEV(opaque);
     SocketChardev *s = SOCKET_CHARDEV(opaque);
 
+    qemu_mutex_lock(&chr->chr_write_lock);
     g_source_unref(s->reconnect_timer);
     s->reconnect_timer = NULL;
+    qemu_mutex_unlock(&chr->chr_write_lock);
 
     if (chr->be_open) {
         return false;

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 12 lines · Daniel P. Berrangé

chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472

Quoted 6 lines
chardev/char-socket.c +1 -1
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr)
     }
     update_disconnected_filename(s);
     if (emit_close) {
-        qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+        qemu_idle_add(tcp_chr_be_event_closed, chr, chr->gcontext);

I'm a little concerned that this change might trigger some unexpected interactions with the qemu_chr_wait_connected() function. I added some more testing around that code, so if it passes the new tests I have in:

Quoted 4 lines

https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05962.html

and also works with the vhostuser reconnect device, then we are probably ok.

Seems that it doesn't ...

/char/null: OK
/char/invalid: OK
/char/ringbuf: OK
/char/mux: OK
/char/stdio: OK
/char/pipe: OK
/char/file: OK
/char/file-fifo: OK
/char/udp: OK
/char/serial: OK
/char/hotswap: OK
/char/websocket: OK
/char/socket/server/mainloop/tcp: OK
/char/socket/server/mainloop/unix: OK
/char/socket/server/wait-conn/tcp: OK
/char/socket/server/wait-conn/unix: OK
/char/socket/server/mainloop-fdpass/tcp: OK
/char/socket/server/mainloop-fdpass/unix: OK
/char/socket/server/wait-conn-fdpass/tcp: OK
/char/socket/server/wait-conn-fdpass/unix: OK
/char/socket/client/mainloop/tcp: OK
/char/socket/client/mainloop/unix: OK
/char/socket/client/wait-conn/tcp: OK
/char/socket/client/wait-conn/unix: OK
/char/socket/client/mainloop-reconnect/tcp: OK
/char/socket/client/mainloop-reconnect/unix: OK
/char/socket/client/wait-conn-reconnect/tcp: Unexpected error in qio_channel_readv_all() at /home/berto/work/qemu/qemu/io/channel.c:147:
Unexpected end-of-file before all bytes were read

Berto

#15 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

In reply to Daniel P. Berrangé
Quoted 11 lines · Daniel P. Berrangé

chardev/char-socket.c static void tcp_chr_disconnect(Chardev *chr) · line 472

Quoted 6 lines
chardev/char-socket.c +1 -1
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr)
     }
     update_disconnected_filename(s);
     if (emit_close) {
-        qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+        qemu_idle_add(tcp_chr_be_event_closed, chr, chr->gcontext);

I'm a little concerned that this change might trigger some unexpected interactions with the qemu_chr_wait_connected() function. I added some more testing around that code, so if it passes the new tests I have in:

Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05962.html

Forget my previous e-mail, there was a mistake on my side. The test does pass after rebasing my series on top of that one.

Quoted 2 lines

and also works with the vhostuser reconnect device, then we are probably ok.

How do I test this?

Berto

#16 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 3 lines · Alberto Garcia
Quoted 2 lines

and also works with the vhostuser reconnect device, then we are probably ok.

How do I test this?

QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-test

Paolo

#17 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 8 lines · Paolo Bonzini
Quoted 3 lines · Alberto Garcia
Quoted 2 lines

and also works with the vhostuser reconnect device, then we are probably ok.

How do I test this?

QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-test

I see. The ones enabled by QTEST_VHOST_USER_FIXME never finish with this patch applied.

Berto

#18 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 4 lines · Alberto Garcia
Quoted 3 lines · Paolo Bonzini
Quoted 2 lines · Alberto Garcia
Quoted 2 lines

and also works with the vhostuser reconnect device, then we are probably ok.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

How do I test this?

QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 \
QTEST_VHOST_USER_FIXME=1 \
tests/vhost-user-test

I see. The ones enabled by QTEST_VHOST_USER_FIXME never finish with this patch applied.

I can look at it next week.

Paolo

#19 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

In reply to Alberto Garcia

Hi

Quoted 26 lines · Alberto Garcia

There's a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread.

Running iotest 136 in a loop triggers this problem and crashes QEMU.

(gdb) bt

 #0  0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860
 #1  0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76
 #2  0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123
 #3  0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135
 #4  0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112
 #5  0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147
 #6  0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42
 #7  0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406
 #8  0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449
 #9  0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498

#10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626
#13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837
#15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885
#16 0x00005558b819140d in main_loop () at vl.c:1924
#17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665

Quoted 4 lines

This patch adds a lock to protect tcp_chr_disconnect() and socket_reconnect_timeout()

That looks like a subset of "[Qemu-devel] Segfaults in chardev due to races". The problem is likely to occur with other chardevs I am afraid.

Quoted 73 lines
---
 chardev/char-socket.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
chardev/char-socket.c +17 -3
index eaa8e8b68f..5f421bdc58 100644
@@ -148,7 +148,9 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len)
 
         if (ret < 0 && errno != EAGAIN) {
             if (tcp_chr_read_poll(chr) <= 0) {
+                qemu_mutex_unlock(&chr->chr_write_lock);
                 tcp_chr_disconnect(chr);
+                qemu_mutex_lock(&chr->chr_write_lock);
                 return len;
             } /* else let the read handler finish it properly */
         }
@@ -440,6 +442,13 @@ static void update_disconnected_filename(SocketChardev *s)
     }
 }
 
+static gboolean tcp_chr_be_event_closed(gpointer opaque)
+{
+    Chardev *chr = opaque;
+    qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+    return FALSE;
+}
+
 /* NB may be called even if tcp_chr_connect has not been
  * reached, due to TLS or telnet initialization failure,
  * so can *not* assume s->connected == true
@@ -447,7 +456,10 @@ static void update_disconnected_filename(SocketChardev *s)
 static void tcp_chr_disconnect(Chardev *chr)
 {
     SocketChardev *s = SOCKET_CHARDEV(chr);
-    bool emit_close = s->connected;
+    bool emit_close;
+
+    qemu_mutex_lock(&chr->chr_write_lock);
+    emit_close = s->connected;
 
     tcp_chr_free_connection(chr);
 
@@ -457,11 +469,12 @@ static void tcp_chr_disconnect(Chardev *chr)
     }
     update_disconnected_filename(s);
     if (emit_close) {
-        qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
+        qemu_idle_add(tcp_chr_be_event_closed, chr, chr->gcontext);
     }
     if (s->reconnect_time) {
         qemu_chr_socket_restart_timer(chr);
     }
+    qemu_mutex_unlock(&chr->chr_write_lock);
 }
 
 static gboolean tcp_chr_read(QIOChannel *chan, GIOCondition cond, void *opaque)
@@ -975,8 +988,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque)
     Chardev *chr = CHARDEV(opaque);
     SocketChardev *s = SOCKET_CHARDEV(opaque);
 
+    qemu_mutex_lock(&chr->chr_write_lock);
     g_source_unref(s->reconnect_timer);
     s->reconnect_timer = NULL;
+    qemu_mutex_unlock(&chr->chr_write_lock);
 
     if (chr->be_open) {
         return false;
--

2.11.0

Signature · 1 line
Marc-André Lureau
#20 lore raw

Re: [Qemu-devel] [PATCH 3/3] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout()

Quoted 3 lines · Marc-André Lureau

That looks like a subset of "[Qemu-devel] Segfaults in chardev due to races". The problem is likely to occur with other chardevs I am afraid.

Yes, it could possibly happen with pty at least. However, Alberto's patch is a start.

Paolo