Skip to contentSkip to conversation

nbd/server: attach client channel to the export's AioContext

PATCH In review Sergio Lopez 3 participants 6 messages Sep 12, 2019 – Sep 20, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 nbd/server: attach client channel to the export's AioContext

On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched.

As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is serviced from the main_loop, potentially triggering the assertion at qio_channel_restart_[read|write].

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253

---
Changelog

v2:
- Attach the channel once after negotiation completes, avoiding duplication. (thanks Kevin Wolf).

---
 nbd/server.c | 5 +++++
 1 file changed, 5 insertions(+)
nbd/server.c +5 -0
index 28c3c8be85..31d624e146 100644
@@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp)
         return ret;
     }
 
+    /* Attach the channel to the same AioContext as the export */
+    if (client->exp && client->exp->ctx) {
+        qio_channel_attach_aio_context(client->ioc, client->exp->ctx);
+    }
+
     assert(!client->optlen);
     trace_nbd_negotiate_success();
 
Signature · 1 line
2.21.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 31 lines · Sergio Lopez

On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched.

As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is serviced from the main_loop, potentially triggering the assertion at qio_channel_restart_[read|write].

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253

---
Changelog

v2:
- Attach the channel once after negotiation completes, avoiding duplication. (thanks Kevin Wolf).

---
 nbd/server.c | 5 +++++
 1 file changed, 5 insertions(+)
nbd/server.c +0 -0
index 28c3c8be85..31d624e146 100644

nbd/server.c static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp) · line 1304

Quoted 8 lines
nbd/server.c +5 -0
@@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp)
         return ret;
     }
 
+    /* Attach the channel to the same AioContext as the export */
+    if (client->exp && client->exp->ctx) {
+        qio_channel_attach_aio_context(client->ioc, client->exp->ctx);
+    }
+

Will queue through my NBD tree.

Quoted 4 lines
nbd/server.c +0 -0
     assert(!client->optlen);
     trace_nbd_negotiate_success();
 
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#2 lore raw
In reply to Sergio Lopez
Quoted 33 lines · Sergio Lopez

On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched.

As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is serviced from the main_loop, potentially triggering the assertion at qio_channel_restart_[read|write].

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253

---
Changelog

v2:
- Attach the channel once after negotiation completes, avoiding duplication. (thanks Kevin Wolf).

---
 nbd/server.c | 5 +++++
 1 file changed, 5 insertions(+)
nbd/server.c +0 -0
index 28c3c8be85..31d624e146 100644
@@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp)
         return ret;

nbd/server.c static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp) · line 1307

Quoted 10 lines
nbd/server.c +5 -0
     }
 
+    /* Attach the channel to the same AioContext as the export */
+    if (client->exp && client->exp->ctx) {
+        qio_channel_attach_aio_context(client->ioc, client->exp->ctx);
+    }
+
     assert(!client->optlen);
     trace_nbd_negotiate_success();
 

I assume this patch has been superseded by Eric's later patches?

--js

#3 lore raw
Quoted 37 lines · John Snow
Quoted 34 lines · Sergio Lopez

On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched.

As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is serviced from the main_loop, potentially triggering the assertion at qio_channel_restart_[read|write].

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253

---
Changelog

v2:
- Attach the channel once after negotiation completes, avoiding duplication. (thanks Kevin Wolf).

---
 nbd/server.c | 5 +++++
 1 file changed, 5 insertions(+)
nbd/server.c +0 -0
index 28c3c8be85..31d624e146 100644
@@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp)
         return ret;
     }
 
Quoted 11 lines

nbd/server.c static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp) · line 1307

Quoted 9 lines
nbd/server.c +5 -0
+    /* Attach the channel to the same AioContext as the export */
+    if (client->exp && client->exp->ctx) {
+        qio_channel_attach_aio_context(client->ioc, client->exp->ctx);
+    }
+
     assert(!client->optlen);
     trace_nbd_negotiate_success();
 

I assume this patch has been superseded by Eric's later patches?

Nevermind -- my filtering got messed up slightly and I missed the followup. I see that Eric staged this.

--js

#4 lore raw
Quoted 9 lines · John Snow
Quoted 9 lines
Quoted 7 lines

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

Quoted 4 lines
Quoted 1 line

I assume this patch has been superseded by Eric's later patches?

Nevermind -- my filtering got messed up slightly and I missed the followup. I see that Eric staged this.

I actually think both patches are needed: this one covers transactions, while my later patch was on top of this to protect shutdown. But now you've made me curious; I'll see if my patch hoisted in front still solves everything, or if we really do need both.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#5 lore raw
Quoted 12 lines · Eric Blake
Quoted 11 lines · John Snow
Quoted 10 lines
Quoted 7 lines

To fix this, as soon we have the export corresponding to the client, we call qio_channel_attach_aio_context() to attach the QIOChannel context to the export's AioContext. This matches with the logic at blk_aio_attached().

I assume this patch has been superseded by Eric's later patches?

Quoted 8 lines
Quoted 2 lines

Nevermind -- my filtering got messed up slightly and I missed the followup. I see that Eric staged this.

I actually think both patches are needed: this one covers transactions, while my later patch was on top of this to protect shutdown. But now you've made me curious; I'll see if my patch hoisted in front still solves everything, or if we really do need both.

Nope, both patches are still needed. Sergio's fixes the assertion:
(qemu) qemu-kvm: io/channel.c:411: qio_channel_restart_read:
Assertion `qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)' failed.

while mine fixes:
+qemu: qemu_mutex_unlock_impl: Operation not permitted

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org