Skip to contentSkip to conversation

virtio-blk: schedule virtio_notify_config to run on main context

RFC PATCH Superseded Sergio Lopez 3 participants 6 messages Sep 12, 2019 – Sep 13, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 virtio-blk: schedule virtio_notify_config to run on main context

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +24 -1
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }
 
 static const BlockDevOps virtio_block_ops = {
Signature · 1 line
2.21.0
#1 lore raw
Quoted 10 lines · Sergio Lopez

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

Quoted 4 lines

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).

Hmm is this really the only case iothread takes qemu mutex? If any such access can deadlock, don't we need a generic solution? Maybe main thread can drop qemu mutex before taking io thread AioContext lock?

Quoted 68 lines

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +24 -1
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }
 
 static const BlockDevOps virtio_block_ops = {
Signature · 1 line
2.21.0
#2 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Sergio Lopez

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

Quoted 6 lines
Quoted 4 lines

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).

Hmm is this really the only case iothread takes qemu mutex?

Not the only one that takes the mutex, but the only one so far we found doing so upon request from a job running on the main thread (should be quite noticeable, due to the deadlock).

Quoted 3 lines

If any such access can deadlock, don't we need a generic solution? Maybe main thread can drop qemu mutex before taking io thread AioContext lock?

The mutex is acquired very early at os_host_main_loop_wait(), so I assume there may be many assumptions in multiple code paths that it has been acquired.

Quoted 68 lines
Quoted 68 lines

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +24 -1
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }
 
 static const BlockDevOps virtio_block_ops = {
Signature · 1 line
2.21.0
#3 lore raw
In reply to Michael S. Tsirkin

Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben:

Quoted 16 lines · Sergio Lopez
Quoted 10 lines

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

Quoted 4 lines

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).
Quoted 4 lines

Hmm is this really the only case iothread takes qemu mutex? If any such access can deadlock, don't we need a generic solution? Maybe main thread can drop qemu mutex before taking io thread AioContext lock?

The rule is that iothreads must not take the qemu mutex. If they do (like in this case), it's a bug.

Maybe we could actually assert this in qemu_mutex_lock_iothread()?

Quoted 49 lines
Quoted 49 lines

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +11 -0
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
Quoted 15 lines
Quoted 5 lines
hw/block/virtio-blk.c +4 -1
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,

hw/block/virtio-blk.c static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f, · line 1117

Quoted 10 lines
hw/block/virtio-blk.c +9 -0
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }

Wouldn't a simple BH suffice (aio_bh_schedule_oneshot)? I don't see why you need a coroutine when you never yield.

The reason why it deadlocks also has nothing to do with whether we are called from a coroutine or not. The important part is that we're running in an iothread.

Kevin

#4 lore raw
Quoted 86 lines · Kevin Wolf

Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben:

Quoted 16 lines · Sergio Lopez
Quoted 10 lines

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

Quoted 4 lines

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).
Quoted 4 lines

Hmm is this really the only case iothread takes qemu mutex? If any such access can deadlock, don't we need a generic solution? Maybe main thread can drop qemu mutex before taking io thread AioContext lock?

The rule is that iothreads must not take the qemu mutex. If they do (like in this case), it's a bug.

Maybe we could actually assert this in qemu_mutex_lock_iothread()?

Quoted 59 lines
Quoted 59 lines

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +20 -1
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
Quoted 8 lines
Quoted 5 lines

hw/block/virtio-blk.c static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f, · line 1117

Quoted 5 lines
hw/block/virtio-blk.c +4 -0
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }

Wouldn't a simple BH suffice (aio_bh_schedule_oneshot)? I don't see why you need a coroutine when you never yield.

You're right, that's actually simpler, haven't thought of it.

Do you see any drawbacks or should I send a non-RFC fixed version of this patch?

Quoted 5 lines

The reason why it deadlocks also has nothing to do with whether we are called from a coroutine or not. The important part is that we're running in an iothread.

Kevin

#5 lore raw

Am 13.09.2019 um 11:28 hat Sergio Lopez geschrieben:

Quoted 98 lines
Quoted 86 lines · Kevin Wolf

Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben:

Quoted 16 lines · Sergio Lopez
Quoted 10 lines

Another AioContext-related issue, and this is a tricky one.

Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes:

- main thead
* Has acquired: qemu_mutex_global.
* Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll).

Quoted 4 lines

- iothread
* Has acquired: AioContext lock.
* Is trying to acquire: qemu_mutex_global (via

    virtio_notify_config->prepare_mmio_access).
Quoted 4 lines

Hmm is this really the only case iothread takes qemu mutex? If any such access can deadlock, don't we need a generic solution? Maybe main thread can drop qemu mutex before taking io thread AioContext lock?

The rule is that iothreads must not take the qemu mutex. If they do (like in this case), it's a bug.

Maybe we could actually assert this in qemu_mutex_lock_iothread()?

Quoted 59 lines
Quoted 59 lines

With this change, virtio_blk_resize checks if it's being called from a coroutine context running on a non-main thread, and if that's the case, creates a new coroutine and schedules it to be run on the main thread.

This works, but means the actual operation is done asynchronously, perhaps opening a window in which a "device_del" operation may fit and remove the VirtIODevice before virtio_notify_config() is executed.

I *think* it shouldn't be possible, as BHs will be processed before any new QMP/monitor command, but I'm open to a different approach.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1744955

---
 hw/block/virtio-blk.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
hw/block/virtio-blk.c +20 -1
index 18851601cb..c763d071f6 100644
@@ -16,6 +16,7 @@
 #include "qemu/iov.h"
 #include "qemu/module.h"
 #include "qemu/error-report.h"
+#include "qemu/main-loop.h"
 #include "trace.h"
 #include "hw/block/block.h"
 #include "hw/qdev-properties.h"
@@ -1086,11 +1087,33 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     return 0;
 }
 
+static void coroutine_fn virtio_resize_co_entry(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+    virtio_notify_config(vdev);
+    aio_wait_kick();
+}
+
 static void virtio_blk_resize(void *opaque)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(opaque);
+    Coroutine *co;
 
-    virtio_notify_config(vdev);
+    if (qemu_in_coroutine() &&
+        qemu_get_current_aio_context() != qemu_get_aio_context()) {
+        /*
+         * virtio_notify_config() needs to acquire the global mutex,
+         * so calling it from a coroutine running on a non-main context
+         * may cause a deadlock. Instead, create a new coroutine and
+         * schedule it to be run on the main thread.
+         */
+        co = qemu_coroutine_create(virtio_resize_co_entry, vdev);
Quoted 8 lines
Quoted 5 lines

hw/block/virtio-blk.c static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f, · line 1117

Quoted 5 lines
hw/block/virtio-blk.c +4 -0
+        aio_co_schedule(qemu_get_aio_context(), co);
+    } else {
+        virtio_notify_config(vdev);
+    }
 }

Wouldn't a simple BH suffice (aio_bh_schedule_oneshot)? I don't see why you need a coroutine when you never yield.

Quoted 4 lines

You're right, that's actually simpler, haven't thought of it.

Do you see any drawbacks or should I send a non-RFC fixed version of this patch?

Sending a fixed non-RFC version sounds good to me.

Kevin