Skip to contentSkip to conversation

migration: Fix rare hang of migration_channel_read_peek()

PATCH In review Peter Xu 3 participants 3 messages Aug 12 – Aug 13

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

PATCH v3 1/1 migration: Fix rare hang of migration_channel_read_peek()

In an unlikely case, when a migration stream is attached to the destination QEMU and only send <4 bytes to the channel as magic, it's possible that migration_channel_read_peek() may spin forever.

Fix it by adding a manual sleep for partial read.

Since the path isn't attached to a coroutine, it means when partial read happens, there's yet not much we can do but hang the main thread, it will happen even for len==0 case. It means monitors can hang due to this, either partial read or no data arrived (but connection established).

Leave this for later, the hope is this is extremely rare in production.

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3889

---
 migration/channel.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
migration/channel.c +9 -2
index 1e2935f926..266ae8f776 100644
@@ -296,9 +296,16 @@ int migration_channel_read_peek(QIOChannel *ioc,
 
         if (len == buflen) {
             break;
+        } else if (len == QIO_CHANNEL_ERR_BLOCK) {
+            qio_channel_wait_cond(ioc, G_IO_IN);
+        } else {
+            /*
+             * When partially ready, we can't use qio_channel_wait_cond()
+             * because it will return immediately.  Apply a manual wait.
+             */
+            assert(!qemu_in_coroutine());
+            g_usleep(1000);
         }
-
-        qio_channel_wait_cond(ioc, G_IO_IN);
     }
 
     return 0;
Signature · 1 line
2.54.0
Reviewed-by on v3 1/1
#1 lore raw
Quoted 13 lines · Peter Xu

In an unlikely case, when a migration stream is attached to the destination QEMU and only send <4 bytes to the channel as magic, it's possible that migration_channel_read_peek() may spin forever.

Fix it by adding a manual sleep for partial read.

Since the path isn't attached to a coroutine, it means when partial read happens, there's yet not much we can do but hang the main thread, it will happen even for len==0 case. It means monitors can hang due to this, either partial read or no data arrived (but connection established).

Leave this for later, the hope is this is extremely rare in production.

Quoted 7 lines

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3889

---
 migration/channel.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
Reviewed-by on v3 1/1
#2 lore raw
In reply to Peter Xu
Quoted 13 lines · Peter Xu

In an unlikely case, when a migration stream is attached to the destination QEMU and only send <4 bytes to the channel as magic, it's possible that migration_channel_read_peek() may spin forever.

Fix it by adding a manual sleep for partial read.

Since the path isn't attached to a coroutine, it means when partial read happens, there's yet not much we can do but hang the main thread, it will happen even for len==0 case. It means monitors can hang due to this, either partial read or no data arrived (but connection established).

Leave this for later, the hope is this is extremely rare in production.

Quoted 7 lines

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3889

---
 migration/channel.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)