Skip to contentSkip to conversation

iothread: fix iothread hang when stop too soon

PATCH Applied Peter Xu 4 participants 6 messages Jan 29, 2019 – Jan 30, 2019

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

PATCH 1/1 iothread: fix iothread hang when stop too soon

Lukas reported an hard to reproduce QMP iothread hang on s390 that QEMU might hang at pthread_join() of the QMP monitor iothread before quitting:

Thread 1
#0 0x000003ffad10932c in pthread_join
#1 0x0000000109e95750 in qemu_thread_join

      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:570
  #2  0x0000000109c95a1c in iothread_stop
  #3  0x0000000109bb0874 in monitor_cleanup
  #4  0x0000000109b55042 in main

While the iothread is still in the main loop:

Thread 4

  #0  0x000003ffad0010e4 in ??
  #1  0x000003ffad553958 in g_main_context_iterate.isra.19
  #2  0x000003ffad553d90 in g_main_loop_run
  #3  0x0000000109c9585a in iothread_run
      at /home/thuth/devel/qemu/iothread.c:74
  #4  0x0000000109e94752 in qemu_thread_start
      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:502
  #5  0x000003ffad10825a in start_thread
  #6  0x000003ffad00dcf2 in thread_start

IMHO it's because there's a race between the main thread and iothread when stopping the thread in following sequence:

    main thread                       iothread
    ===========                       ==============
                                      aio_poll()
    iothread_get_g_main_context
      set iothread->worker_context
    iothread_stop
      schedule iothread_stop_bh
                                        execute iothread_stop_bh [1]
                                          set iothread->running=false
                                          (since main_loop==NULL so
                                           skip to quit main loop.
                                           Note: although main_loop is
                                           NULL but worker_context is
                                           not!)
                                      atomic_read(&iothread->worker_context) [2]
                                        create main_loop object
                                        g_main_loop_run() [3]
    pthread_join() [4]

We can see that when execute iothread_stop_bh() at [1] it's possible that main_loop is still NULL because it's only created until the first check of the worker_context later at [2]. Then the iothread will hang in the main loop [3] and it'll starve the main thread too [4].

Here the simple solution should be that we check again the "running" variable before check against worker_context.

CC Thomas Huth <[email protected]>
CC Dr. David Alan Gilbert <[email protected]>
CC Stefan Hajnoczi <[email protected]>
CC Lukáš Doktor <[email protected]>
CC Markus Armbruster <[email protected]>
CC Eric Blake <[email protected]>
CC Paolo Bonzini <[email protected]>

---

This hasn't yet been verified on the initial s390 systems, but since I can reproduce it locally with this code clip:

        IOThread *iothread = iothread_create("test", NULL);
        iothread_get_g_main_context(iothread);
        iothread_stop(iothread);

so I'm still posting this out for review first in case it was hit by other users.

---
 iothread.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
iothread.c +5 -1
index 2fb1cdf55d..e615b7ae52 100644
@@ -63,7 +63,11 @@ static void *iothread_run(void *opaque)
     while (iothread->running) {
         aio_poll(iothread->ctx, true);
 
-        if (atomic_read(&iothread->worker_context)) {
+        /*
+         * We must check the running state again in case it was
+         * changed in previous aio_poll()
+         */
+        if (iothread->running && atomic_read(&iothread->worker_context)) {
             GMainLoop *loop;
 
             g_main_context_push_thread_default(iothread->worker_context);
Signature · 1 line
2.17.1
Tested-by on v1 1/1
#1 lore raw
Quoted 87 lines · Peter Xu

Lukas reported an hard to reproduce QMP iothread hang on s390 that QEMU might hang at pthread_join() of the QMP monitor iothread before quitting:

Thread 1
#0 0x000003ffad10932c in pthread_join
#1 0x0000000109e95750 in qemu_thread_join

      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:570
  #2  0x0000000109c95a1c in iothread_stop
  #3  0x0000000109bb0874 in monitor_cleanup
  #4  0x0000000109b55042 in main

While the iothread is still in the main loop:

Thread 4

  #0  0x000003ffad0010e4 in ??
  #1  0x000003ffad553958 in g_main_context_iterate.isra.19
  #2  0x000003ffad553d90 in g_main_loop_run
  #3  0x0000000109c9585a in iothread_run
      at /home/thuth/devel/qemu/iothread.c:74
  #4  0x0000000109e94752 in qemu_thread_start
      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:502
  #5  0x000003ffad10825a in start_thread
  #6  0x000003ffad00dcf2 in thread_start

IMHO it's because there's a race between the main thread and iothread when stopping the thread in following sequence:

    main thread                       iothread
    ===========                       ==============
                                      aio_poll()
    iothread_get_g_main_context
      set iothread->worker_context
    iothread_stop
      schedule iothread_stop_bh
                                        execute iothread_stop_bh [1]
                                          set iothread->running=false
                                          (since main_loop==NULL so
                                           skip to quit main loop.
                                           Note: although main_loop is
                                           NULL but worker_context is
                                           not!)
                                      atomic_read(&iothread->worker_context) [2]
                                        create main_loop object
                                        g_main_loop_run() [3]
    pthread_join() [4]

We can see that when execute iothread_stop_bh() at [1] it's possible that main_loop is still NULL because it's only created until the first check of the worker_context later at [2]. Then the iothread will hang in the main loop [3] and it'll starve the main thread too [4].

Here the simple solution should be that we check again the "running" variable before check against worker_context.

CC Thomas Huth <[email protected]>
CC Dr. David Alan Gilbert <[email protected]>
CC Stefan Hajnoczi <[email protected]>
CC Lukáš Doktor <[email protected]>
CC Markus Armbruster <[email protected]>
CC Eric Blake <[email protected]>
CC Paolo Bonzini <[email protected]>

---

This hasn't yet been verified on the initial s390 systems, but since I can reproduce it locally with this code clip:

        IOThread *iothread = iothread_create("test", NULL);
        iothread_get_g_main_context(iothread);
        iothread_stop(iothread);

so I'm still posting this out for review first in case it was hit by other users.

---
 iothread.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
iothread.c +0 -0
index 2fb1cdf55d..e615b7ae52 100644
@@ -63,7 +63,11 @@ static void *iothread_run(void *opaque)
     while (iothread->running) {
         aio_poll(iothread->ctx, true);

iothread.c static void *iothread_run(void *opaque) · line 73

Quoted 10 lines
iothread.c +5 -1
 
-        if (atomic_read(&iothread->worker_context)) {
+        /*
+         * We must check the running state again in case it was
+         * changed in previous aio_poll()
+         */
+        if (iothread->running && atomic_read(&iothread->worker_context)) {
             GMainLoop *loop;
 
             g_main_context_push_thread_default(iothread->worker_context);

I ran this on s390x with Lukáš' reproducer for a while now, and so far I haven't seen any hangs anymore. Thus this seems to fix the issue as far as I can see, thanks!

author
#2 lore raw

On Tue, Jan 29, 2019 at 03:44:05PM +0100, Thomas Huth wrote:

[...]

Quoted 5 lines

I ran this on s390x with Lukáš' reproducer for a while now, and so far I haven't seen any hangs anymore. Thus this seems to fix the issue as far as I can see, thanks!

Thanks for the quick follow up, Thomas!

Signature · 1 line
Peter Xu
#3 lore raw
In reply to Peter Xu

Marc-André (cc'ed) recently fixed a deadlock (commit 34f1f3e06d8 "monitor: avoid potential dead-lock when cleaning up"). Looks like we got more.

Quoted 96 lines · Peter Xu

Lukas reported an hard to reproduce QMP iothread hang on s390 that QEMU might hang at pthread_join() of the QMP monitor iothread before quitting:

Thread 1
#0 0x000003ffad10932c in pthread_join
#1 0x0000000109e95750 in qemu_thread_join

      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:570
  #2  0x0000000109c95a1c in iothread_stop
  #3  0x0000000109bb0874 in monitor_cleanup
  #4  0x0000000109b55042 in main

While the iothread is still in the main loop:

Thread 4

  #0  0x000003ffad0010e4 in ??
  #1  0x000003ffad553958 in g_main_context_iterate.isra.19
  #2  0x000003ffad553d90 in g_main_loop_run
  #3  0x0000000109c9585a in iothread_run
      at /home/thuth/devel/qemu/iothread.c:74
  #4  0x0000000109e94752 in qemu_thread_start
      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:502
  #5  0x000003ffad10825a in start_thread
  #6  0x000003ffad00dcf2 in thread_start

IMHO it's because there's a race between the main thread and iothread when stopping the thread in following sequence:

    main thread                       iothread
    ===========                       ==============
                                      aio_poll()
    iothread_get_g_main_context
      set iothread->worker_context
    iothread_stop
      schedule iothread_stop_bh
                                        execute iothread_stop_bh [1]
                                          set iothread->running=false
                                          (since main_loop==NULL so
                                           skip to quit main loop.
                                           Note: although main_loop is
                                           NULL but worker_context is
                                           not!)
                                      atomic_read(&iothread->worker_context) [2]
                                        create main_loop object
                                        g_main_loop_run() [3]
    pthread_join() [4]

We can see that when execute iothread_stop_bh() at [1] it's possible that main_loop is still NULL because it's only created until the first check of the worker_context later at [2]. Then the iothread will hang in the main loop [3] and it'll starve the main thread too [4].

Here the simple solution should be that we check again the "running" variable before check against worker_context.

CC Thomas Huth <[email protected]>
CC Dr. David Alan Gilbert <[email protected]>
CC Stefan Hajnoczi <[email protected]>
CC Lukáš Doktor <[email protected]>
CC Markus Armbruster <[email protected]>
CC Eric Blake <[email protected]>
CC Paolo Bonzini <[email protected]>

---

This hasn't yet been verified on the initial s390 systems, but since I can reproduce it locally with this code clip:

        IOThread *iothread = iothread_create("test", NULL);
        iothread_get_g_main_context(iothread);
        iothread_stop(iothread);

so I'm still posting this out for review first in case it was hit by other users.

---
 iothread.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
iothread.c +5 -1
index 2fb1cdf55d..e615b7ae52 100644
@@ -63,7 +63,11 @@ static void *iothread_run(void *opaque)
     while (iothread->running) {
         aio_poll(iothread->ctx, true);
 
-        if (atomic_read(&iothread->worker_context)) {
+        /*
+         * We must check the running state again in case it was
+         * changed in previous aio_poll()
+         */
+        if (iothread->running && atomic_read(&iothread->worker_context)) {
             GMainLoop *loop;
 
             g_main_context_push_thread_default(iothread->worker_context);
author
#4 lore raw
Quoted 3 lines · Markus Armbruster

Marc-André (cc'ed) recently fixed a deadlock (commit 34f1f3e06d8 "monitor: avoid potential dead-lock when cleaning up"). Looks like we got more.

Yes, and a few tests too. Hopefully these fallouts can settle soon before the rcs. Looks like there's still plenty of time for it to be ready (softfreeze is March 19th).

Thanks,

Signature · 1 line
Peter Xu
#5 lore raw
In reply to Peter Xu
Quoted 73 lines · Peter Xu

Lukas reported an hard to reproduce QMP iothread hang on s390 that QEMU might hang at pthread_join() of the QMP monitor iothread before quitting:

Thread 1
#0 0x000003ffad10932c in pthread_join
#1 0x0000000109e95750 in qemu_thread_join

      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:570
  #2  0x0000000109c95a1c in iothread_stop
  #3  0x0000000109bb0874 in monitor_cleanup
  #4  0x0000000109b55042 in main

While the iothread is still in the main loop:

Thread 4

  #0  0x000003ffad0010e4 in ??
  #1  0x000003ffad553958 in g_main_context_iterate.isra.19
  #2  0x000003ffad553d90 in g_main_loop_run
  #3  0x0000000109c9585a in iothread_run
      at /home/thuth/devel/qemu/iothread.c:74
  #4  0x0000000109e94752 in qemu_thread_start
      at /home/thuth/devel/qemu/util/qemu-thread-posix.c:502
  #5  0x000003ffad10825a in start_thread
  #6  0x000003ffad00dcf2 in thread_start

IMHO it's because there's a race between the main thread and iothread when stopping the thread in following sequence:

    main thread                       iothread
    ===========                       ==============
                                      aio_poll()
    iothread_get_g_main_context
      set iothread->worker_context
    iothread_stop
      schedule iothread_stop_bh
                                        execute iothread_stop_bh [1]
                                          set iothread->running=false
                                          (since main_loop==NULL so
                                           skip to quit main loop.
                                           Note: although main_loop is
                                           NULL but worker_context is
                                           not!)
                                      atomic_read(&iothread->worker_context) [2]
                                        create main_loop object
                                        g_main_loop_run() [3]
    pthread_join() [4]

We can see that when execute iothread_stop_bh() at [1] it's possible that main_loop is still NULL because it's only created until the first check of the worker_context later at [2]. Then the iothread will hang in the main loop [3] and it'll starve the main thread too [4].

Here the simple solution should be that we check again the "running" variable before check against worker_context.

CC Thomas Huth <[email protected]>
CC Dr. David Alan Gilbert <[email protected]>
CC Stefan Hajnoczi <[email protected]>
CC Lukáš Doktor <[email protected]>
CC Markus Armbruster <[email protected]>
CC Eric Blake <[email protected]>
CC Paolo Bonzini <[email protected]>

---

This hasn't yet been verified on the initial s390 systems, but since I can reproduce it locally with this code clip:

        IOThread *iothread = iothread_create("test", NULL);
        iothread_get_g_main_context(iothread);
        iothread_stop(iothread);
Quoted 5 lines

so I'm still posting this out for review first in case it was hit by other users.

---
 iothread.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan