Skip to contentSkip to conversation

migration/multifd: a new mechanism for send thread sync

PATCH In review Wei Yang 2 participants 10 messages Oct 26, 2019 – Dec 16, 2019

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

v2 cover migration/multifd: a new mechanism for send thread sync

Current send thread could work while the sync mechanism has some problem:

* has spuriously wakeup
* number of channels_ready will *overflow* the number of real channels

The reason is:

* if MULTIFD_FLAG_SYNC is set in the middle of send thread running, there is one more spurious wakeup
* if MULTIFD_FLAG_SYNC is set when send thread is not running, there is one more channels_ready be triggered

To solve this situation, one new mechanism is introduced to synchronize send threads. The idea is simple, a new field *sync* is introduced to indicate a synchronization is required.

---
v2: rebase on latest code

Wei Yang (6):
migration/multifd: move Params update and pages cleanup into
multifd_send_fill_packet()
migration/multifd: notify channels_ready when send thread starts migration/multifd: use sync field to synchronize send threads migration/multifd: used must not be 0 for a pending job migration/multifd: use boolean for pending_job is enough migration/multifd: there is no spurious wakeup now

 migration/ram.c | 74 +++++++++++++++++++++++++++++++------------------
 1 file changed, 47 insertions(+), 27 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v2 1/6 migration/multifd: move Params update and pages cleanup into multifd_send_fill_packet()

Fill data and update/cleanup related field in one place. Also make the code a little clean.

---
 migration/ram.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
migration/ram.c +11 -9
index 5876054195..35f147388b 100644
@@ -789,15 +789,16 @@ static void multifd_pages_clear(MultiFDPages_t *pages)
     g_free(pages);
 }
 
-static void multifd_send_fill_packet(MultiFDSendParams *p)
+static void multifd_send_fill_packet(MultiFDSendParams *p, uint32_t used)
 {
     MultiFDPacket_t *packet = p->packet;
+    uint32_t next_packet_size = used * qemu_target_page_size();
     int i;
 
     packet->flags = cpu_to_be32(p->flags);
     packet->pages_alloc = cpu_to_be32(p->pages->allocated);
     packet->pages_used = cpu_to_be32(p->pages->used);
-    packet->next_packet_size = cpu_to_be32(p->next_packet_size);
+    packet->next_packet_size = cpu_to_be32(next_packet_size);
     packet->packet_num = cpu_to_be64(p->packet_num);
 
     if (p->pages->block) {
@@ -807,6 +808,13 @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
     for (i = 0; i < p->pages->used; i++) {
         packet->offset[i] = cpu_to_be64(p->pages->offset[i]);
     }
+
+    p->next_packet_size = next_packet_size;
+    p->flags = 0;
+    p->num_packets++;
+    p->num_pages += used;
+    p->pages->used = 0;
+    p->pages->block = NULL;
 }
 
 static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
@@ -1109,13 +1117,7 @@ static void *multifd_send_thread(void *opaque)
             uint64_t packet_num = p->packet_num;
             flags = p->flags;
 
-            p->next_packet_size = used * qemu_target_page_size();
-            multifd_send_fill_packet(p);
-            p->flags = 0;
-            p->num_packets++;
-            p->num_pages += used;
-            p->pages->used = 0;
-            p->pages->block = NULL;
+            multifd_send_fill_packet(p, used);
             qemu_mutex_unlock(&p->mutex);
 
             trace_multifd_send(p->id, packet_num, used, flags,
Signature · 1 line
2.17.1
author
#3 lore raw

Re: [PATCH v2 1/6] migration/multifd: move Params update and pages cleanup into multifd_send_fill_packet()

Quoted 10 lines · Juan Quintela
Quoted 4 lines · Wei Yang

Fill data and update/cleanup related field in one place. Also make the code a little clean.

right cleanup.

Hi, Juan

Do you have other comments on the following patches?

Signature · 2 lines
Wei Yang
Help you, Help me
author
#4 lore raw

PATCH v2 2/6 migration/multifd: notify channels_ready when send thread starts

In reply to Wei Yang

multifd_send_state->channels_ready is initialized to 0. It is proper to let main thread know we are ready when thread start running.

Current implementation works since ram_save_setup() calls multifd_send_sync_main() which wake up send thread and posts channels_ready. This behavior will introduce some unpredictable situation and disturb the semaphore value.

This is a preparation patch to use another mechanism to do send thread synchronization to avoid post channels_ready in this case. So this patch posts channels_ready when send threads start running.

---
 migration/ram.c | 2 ++
 1 file changed, 2 insertions(+)
migration/ram.c +2 -0
index 35f147388b..25d477796e 100644
@@ -1107,6 +1107,8 @@ static void *multifd_send_thread(void *opaque)
     }
     /* initial packet */
     p->num_packets = 1;
+    /* let main thread know we are ready */
+    qemu_sem_post(&multifd_send_state->channels_ready);
 
     while (true) {
         qemu_sem_wait(&p->sem);
Signature · 1 line
2.17.1
author
#5 lore raw

PATCH v2 3/6 migration/multifd: use sync field to synchronize send threads

In reply to Wei Yang

Add a field in MultiFDSendParams to indicate there is a request to synchronize send threads.

By doing so, send_thread will just post sem_sync on synchronization request and channels_ready will not *overflow*.

---
 migration/ram.c | 30 ++++++++++++++++++++++++------
 1 file changed, 24 insertions(+), 6 deletions(-)
migration/ram.c +24 -6
index 25d477796e..62072b7a35 100644
@@ -641,6 +641,8 @@ typedef struct {
     QemuMutex mutex;
     /* is this channel thread running */
     bool running;
+    /* should sync this channel */
+    bool sync;
     /* should this thread finish */
     bool quit;
     /* thread has work to do */
@@ -1074,8 +1076,7 @@ static void multifd_send_sync_main(RAMState *rs)
         }
 
         p->packet_num = multifd_send_state->packet_num++;
-        p->flags |= MULTIFD_FLAG_SYNC;
-        p->pending_job++;
+        p->sync = true;
         qemu_file_update_transfer(rs->f, p->packet_len);
         ram_counters.multifd_bytes += p->packet_len;
         ram_counters.transferred += p->packet_len;
@@ -1143,10 +1144,27 @@ static void *multifd_send_thread(void *opaque)
             p->pending_job--;
             qemu_mutex_unlock(&p->mutex);
 
-            if (flags & MULTIFD_FLAG_SYNC) {
-                qemu_sem_post(&p->sem_sync);
-            }
             qemu_sem_post(&multifd_send_state->channels_ready);
+        } else if (p->sync) {
+            uint64_t packet_num = p->packet_num;
+            uint32_t flags = p->flags;
+            assert(!p->pages->used);
+
+            p->flags |= MULTIFD_FLAG_SYNC;
+            multifd_send_fill_packet(p, 0);
+            p->sync = false;
+            qemu_mutex_unlock(&p->mutex);
+
+            trace_multifd_send(p->id, packet_num, 0, flags | MULTIFD_FLAG_SYNC,
+                               p->next_packet_size);
+
+            ret = qio_channel_write_all(p->c, (void *)p->packet,
+                                        p->packet_len, &local_err);
+            if (ret != 0) {
+                break;
+            }
+
+            qemu_sem_post(&p->sem_sync);
         } else if (p->quit) {
             qemu_mutex_unlock(&p->mutex);
             break;
@@ -1221,7 +1239,7 @@ int multifd_save_setup(void)
         qemu_mutex_init(&p->mutex);
         qemu_sem_init(&p->sem, 0);
         qemu_sem_init(&p->sem_sync, 0);
-        p->quit = false;
+        p->quit = p->sync = false;
         p->pending_job = 0;
         p->id = i;
         p->pages = multifd_pages_init(page_count);
Signature · 1 line
2.17.1
author
#6 lore raw

PATCH v2 4/6 migration/multifd: used must not be 0 for a pending job

In reply to Wei Yang

After thread synchronization request is handled in another case, this means when we only get pending_job when there is used pages.

---
 migration/ram.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
migration/ram.c +5 -6
index 62072b7a35..12c270e86d 100644
@@ -1132,12 +1132,11 @@ static void *multifd_send_thread(void *opaque)
                 break;
             }
 
-            if (used) {
-                ret = qio_channel_writev_all(p->c, p->pages->iov,
-                                             used, &local_err);
-                if (ret != 0) {
-                    break;
-                }
+            assert(used);
+            ret = qio_channel_writev_all(p->c, p->pages->iov,
+                                         used, &local_err);
+            if (ret != 0) {
+                break;
             }
 
             qemu_mutex_lock(&p->mutex);
Signature · 1 line
2.17.1
author
#7 lore raw

PATCH v2 5/6 migration/multifd: use boolean for pending_job is enough

In reply to Wei Yang

After synchronization request is handled in another case, there only could be one pending_job for one send thread at most.

This is fine to use boolean to represent this behavior.

---
 migration/ram.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
migration/ram.c +4 -5
index 12c270e86d..fccdbfabc5 100644
@@ -646,7 +646,7 @@ typedef struct {
     /* should this thread finish */
     bool quit;
     /* thread has work to do */
-    int pending_job;
+    bool pending_job;
     /* array of pages to sent */
     MultiFDPages_t *pages;
     /* packet allocated len */
@@ -933,7 +933,7 @@ static int multifd_send_pages(RAMState *rs)
             return -1;
         }
         if (!p->pending_job) {
-            p->pending_job++;
+            p->pending_job = true;
             next_channel = (i + 1) % migrate_multifd_channels();
             break;
         }
@@ -1140,7 +1140,7 @@ static void *multifd_send_thread(void *opaque)
             }
 
             qemu_mutex_lock(&p->mutex);
-            p->pending_job--;
+            p->pending_job = false;
             qemu_mutex_unlock(&p->mutex);
 
             qemu_sem_post(&multifd_send_state->channels_ready);
@@ -1238,8 +1238,7 @@ int multifd_save_setup(void)
         qemu_mutex_init(&p->mutex);
         qemu_sem_init(&p->sem, 0);
         qemu_sem_init(&p->sem_sync, 0);
-        p->quit = p->sync = false;
-        p->pending_job = 0;
+        p->quit = p->sync = p->pending_job = false;
         p->id = i;
         p->pages = multifd_pages_init(page_count);
         p->packet_len = sizeof(MultiFDPacket_t)
Signature · 1 line
2.17.1
author
#8 lore raw

PATCH v2 6/6 migration/multifd: there is no spurious wakeup now

In reply to Wei Yang

The spurious wakeup is gone.

---
 migration/ram.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
migration/ram.c +2 -2
index fccdbfabc5..73ace40b1b 100644
@@ -1168,8 +1168,8 @@ static void *multifd_send_thread(void *opaque)
             qemu_mutex_unlock(&p->mutex);
             break;
         } else {
-            qemu_mutex_unlock(&p->mutex);
-            /* sometimes there are spurious wakeups */
+            /* no other case should trigger me */
+            g_assert_not_reached();
         }
     }
 
Signature · 1 line
2.17.1
author
#9 lore raw
In reply to Wei Yang

Ping for comment.

Quoted 33 lines · Wei Yang

Current send thread could work while the sync mechanism has some problem:

* has spuriously wakeup
* number of channels_ready will *overflow* the number of real channels

The reason is:

* if MULTIFD_FLAG_SYNC is set in the middle of send thread running, there is one more spurious wakeup
* if MULTIFD_FLAG_SYNC is set when send thread is not running, there is one more channels_ready be triggered

To solve this situation, one new mechanism is introduced to synchronize send threads. The idea is simple, a new field *sync* is introduced to indicate a synchronization is required.

---
v2: rebase on latest code

Wei Yang (6):
migration/multifd: move Params update and pages cleanup into
multifd_send_fill_packet()
migration/multifd: notify channels_ready when send thread starts migration/multifd: use sync field to synchronize send threads migration/multifd: used must not be 0 for a pending job migration/multifd: use boolean for pending_job is enough migration/multifd: there is no spurious wakeup now

migration/ram.c | 74 +++++++++++++++++++++++++++++++------------------

1 file changed, 47 insertions(+), 27 deletions(-)
Signature · 1 line
2.17.1
Signature · 2 lines
Wei Yang
Help you, Help me