Skip to contentSkip to conversation

migration/rdma disconnect fixes

PATCH In review Dr. David Alan Gilbert (git) 2 participants 5 messages Sep 13, 2019 – Sep 25, 2019

lore
Share & tools
lore
#0 lore raw

cover migration/rdma disconnect fixes

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This fixes a deadlock that can occur on the source after
a failed RDMA migration and cleans up some warning messages that can appear during normal completion.

https://bugzilla.redhat.com/show_bug.cgi?id=1746787

Dr. David Alan Gilbert (2):
migration/rdma: Don't moan about disconnects at the end migration/rdma.c: Swap synchronize_rcu for call_rcu

 migration/rdma.c | 51 +++++++++++++++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 16 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/2 migration/rdma: Don't moan about disconnects at the end

From: "Dr. David Alan Gilbert" <[email protected]>

If we've already finished the migration or something has already gone wrong, don't moan about the migration stream disconnecting.

---
 migration/rdma.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
migration/rdma.c +8 -4
index 78e6b72bac..0fcf02f48e 100644
@@ -3253,10 +3253,14 @@ static void rdma_cm_poll_handler(void *opaque)
 
     if (cm_event->event == RDMA_CM_EVENT_DISCONNECTED ||
         cm_event->event == RDMA_CM_EVENT_DEVICE_REMOVAL) {
-        error_report("receive cm event, cm event is %d", cm_event->event);
-        rdma->error_state = -EPIPE;
-        if (rdma->return_path) {
-            rdma->return_path->error_state = -EPIPE;
+        if (!rdma->error_state &&
+            migration_incoming_get_current()->state !=
+              MIGRATION_STATUS_COMPLETED) {
+            error_report("receive cm event, cm event is %d", cm_event->event);
+            rdma->error_state = -EPIPE;
+            if (rdma->return_path) {
+                rdma->return_path->error_state = -EPIPE;
+            }
         }
 
         if (mis->migration_incoming_co) {
Signature · 1 line
2.21.0
#2 lore raw

PATCH 2/2 migration/rdma.c: Swap synchronize_rcu for call_rcu

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a deadlock that can occur on the migration source after a failed RDMA migration; as the source tries to cleanup it clears a pair of pointers and uses synchronize_rcu to wait; this is happening on the main thread. With the CPUs running
a CPU thread can be an rcu reader and attempt to grab the main lock (kvm_handle_io->address_space_write->flatview_write->flatview_write_continue-> prepare_mmio_access->qemu_mutex_lock_iothread_impl)

Replace the synchronize_rcu with a call_rcu to postpone the freeing.

Fixes 74637e6f08fceda98806 ("migration: implement bi-directional RDMA QIOChannel")

( https://bugzilla.redhat.com/show_bug.cgi?id=1746787 )

---
 migration/rdma.c | 39 +++++++++++++++++++++++++++------------
 1 file changed, 27 insertions(+), 12 deletions(-)
migration/rdma.c +27 -12
index 0fcf02f48e..4c74e88a37 100644
@@ -3017,11 +3017,35 @@ static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc,
     }
 }
 
+struct rdma_close_rcu {
+    struct rcu_head rcu;
+    RDMAContext *rdmain;
+    RDMAContext *rdmaout;
+};
+
+/* callback from qio_channel_rdma_close via call_rcu */
+static void qio_channel_rdma_close_rcu(struct rdma_close_rcu *rcu)
+{
+    if (rcu->rdmain) {
+        qemu_rdma_cleanup(rcu->rdmain);
+    }
+
+    if (rcu->rdmaout) {
+        qemu_rdma_cleanup(rcu->rdmaout);
+    }
+
+    g_free(rcu->rdmain);
+    g_free(rcu->rdmaout);
+    g_free(rcu);
+}
+
 static int qio_channel_rdma_close(QIOChannel *ioc,
                                   Error **errp)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
     RDMAContext *rdmain, *rdmaout;
+    struct rdma_close_rcu *rcu = g_new(struct rdma_close_rcu, 1);
+
     trace_qemu_rdma_close();
 
     rdmain = rioc->rdmain;
@@ -3034,18 +3058,9 @@ static int qio_channel_rdma_close(QIOChannel *ioc,
         atomic_rcu_set(&rioc->rdmaout, NULL);
     }
 
-    synchronize_rcu();
-
-    if (rdmain) {
-        qemu_rdma_cleanup(rdmain);
-    }
-
-    if (rdmaout) {
-        qemu_rdma_cleanup(rdmaout);
-    }
-
-    g_free(rdmain);
-    g_free(rdmaout);
+    rcu->rdmain = rdmain;
+    rcu->rdmaout = rdmaout;
+    call_rcu(rcu, qio_channel_rdma_close_rcu, rcu);
 
     return 0;
 }
Signature · 1 line
2.21.0
#4 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · * Dr. David Alan Gilbert (git) ([email protected]) wrote:

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This fixes a deadlock that can occur on the source after
a failed RDMA migration and cleans up some warning messages that can appear during normal completion.

https://bugzilla.redhat.com/show_bug.cgi?id=1746787

Queued

Quoted 11 lines

Dr. David Alan Gilbert (2):
migration/rdma: Don't moan about disconnects at the end migration/rdma.c: Swap synchronize_rcu for call_rcu

 migration/rdma.c | 51 +++++++++++++++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 16 deletions(-)
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK