migration/multifd: a new mechanism for send thread sync
v2 cover migration/multifd: a new mechanism for send thread sync
Current send thread could work while the sync mechanism has some problem:
* has spuriously wakeup
* number of channels_ready will *overflow* the number of real channels
The reason is:
* if MULTIFD_FLAG_SYNC is set in the middle of send thread running, there is one more spurious wakeup
* if MULTIFD_FLAG_SYNC is set when send thread is not running, there is one more channels_ready be triggered
To solve this situation, one new mechanism is introduced to synchronize send threads. The idea is simple, a new field *sync* is introduced to indicate a synchronization is required.
---
v2: rebase on latest code
Wei Yang (6):
migration/multifd: move Params update and pages cleanup into
multifd_send_fill_packet()
migration/multifd: notify channels_ready when send thread starts migration/multifd: use sync field to synchronize send threads migration/multifd: used must not be 0 for a pending job migration/multifd: use boolean for pending_job is enough migration/multifd: there is no spurious wakeup now
migration/ram.c | 74 +++++++++++++++++++++++++++++++------------------ 1 file changed, 47 insertions(+), 27 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/6 migration/multifd: move Params update and pages cleanup into multifd_send_fill_packet()
Fill data and update/cleanup related field in one place. Also make the code a little clean.
--- migration/ram.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
migration/ram.c +11 -9
@@ -789,15 +789,16 @@ static void multifd_pages_clear(MultiFDPages_t *pages)g_free (pages); } -static void multifd_send_fill_packet (MultiFDSendParams *p) +static void multifd_send_fill_packet (MultiFDSendParams *p,uint32_t used) {MultiFDPacket_t *packet = p->packet; +uint32_t next_packet_size = used *qemu_target_page_size ();int i; packet->flags =cpu_to_be32 (p->flags); packet->pages_alloc =cpu_to_be32 (p->pages->allocated); packet->pages_used =cpu_to_be32 (p->pages->used); - packet->next_packet_size =cpu_to_be32 (p->next_packet_size); + packet->next_packet_size =cpu_to_be32 (next_packet_size); packet->packet_num =cpu_to_be64 (p->packet_num);if (p->pages->block) {
@@ -807,6 +808,13 @@ static void multifd_send_fill_packet(MultiFDSendParams *p)for (i =0 ; i < p->pages->used; i++) { packet->offset[i] =cpu_to_be64 (p->pages->offset[i]); } + + p->next_packet_size = next_packet_size; + p->flags =0 ; + p->num_packets++; + p->num_pages += used; + p->pages->used =0 ; + p->pages->block =NULL ; }static int multifd_recv_unfill_packet (MultiFDRecvParams *p,Error **errp)
@@ -1109,13 +1117,7 @@ static void *multifd_send_thread(void *opaque)uint64_t packet_num = p->packet_num; flags = p->flags; - p->next_packet_size = used *qemu_target_page_size (); -multifd_send_fill_packet (p); - p->flags =0 ; - p->num_packets++; - p->num_pages += used; - p->pages->used =0 ; - p->pages->block =NULL ; +multifd_send_fill_packet (p, used);qemu_mutex_unlock (&p->mutex); trace_multifd_send(p->id, packet_num, used, flags,
Signature · 1 line
2.17.1
Re: [PATCH v2 1/6] migration/multifd: move Params update and pages cleanup into multifd_send_fill_packet()
Quoted 4 lines · Wei Yang
Fill data and update/cleanup related field in one place. Also make the code a little clean.
Signed-off-by Wei Yang <[email protected]>
right cleanup.
Re: [PATCH v2 1/6] migration/multifd: move Params update and pages cleanup into multifd_send_fill_packet()
Quoted 10 lines · Juan Quintela
Quoted 4 lines · Wei Yang
Fill data and update/cleanup related field in one place. Also make the code a little clean.
Signed-off-by Wei Yang <[email protected]>Reviewed-by Juan Quintela <[email protected]>right cleanup.
Hi, Juan
Do you have other comments on the following patches?
Signature · 2 lines
Wei Yang Help you, Help me
PATCH v2 2/6 migration/multifd: notify channels_ready when send thread starts
In reply to Wei Yangmultifd_send_state->channels_ready is initialized to 0. It is proper to let main thread know we are ready when thread start running.
Current implementation works since ram_save_setup() calls multifd_send_sync_main() which wake up send thread and posts channels_ready. This behavior will introduce some unpredictable situation and disturb the semaphore value.
This is a preparation patch to use another mechanism to do send thread synchronization to avoid post channels_ready in this case. So this patch posts channels_ready when send threads start running.
--- migration/ram.c | 2 ++ 1 file changed, 2 insertions(+)
migration/ram.c +2 -0
@@ -1107,6 +1107,8 @@ static void *multifd_send_thread(void *opaque) }/* initial packet */ p->num_packets =1 ; +/* let main thread know we are ready */ +qemu_sem_post (&multifd_send_state->channels_ready);while (true) {qemu_sem_wait (&p->sem);
Signature · 1 line
2.17.1
PATCH v2 3/6 migration/multifd: use sync field to synchronize send threads
In reply to Wei YangAdd a field in MultiFDSendParams to indicate there is a request to synchronize send threads.
By doing so, send_thread will just post sem_sync on synchronization request and channels_ready will not *overflow*.
--- migration/ram.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-)
migration/ram.c +24 -6
@@ -641,6 +641,8 @@ typedef struct {QemuMutex mutex;/* is this channel thread running */ bool running; +/* should sync this channel */ +bool sync;/* should this thread finish */ bool quit;/* thread has work to do */
@@ -1074,8 +1076,7 @@ static void multifd_send_sync_main(RAMState *rs) } p->packet_num = multifd_send_state->packet_num++; - p->flags |=MULTIFD_FLAG_SYNC ; - p->pending_job++; + p->sync = true;qemu_file_update_transfer (rs->f, p->packet_len); ram_counters.multifd_bytes += p->packet_len; ram_counters.transferred += p->packet_len;
@@ -1143,10 +1144,27 @@ static void *multifd_send_thread(void *opaque) p->pending_job--;qemu_mutex_unlock (&p->mutex); -if (flags &MULTIFD_FLAG_SYNC ) { -qemu_sem_post (&p->sem_sync); - }qemu_sem_post (&multifd_send_state->channels_ready); + }else if (p->sync) { +uint64_t packet_num = p->packet_num; +uint32_t flags = p->flags; +assert (!p->pages->used); + + p->flags |=MULTIFD_FLAG_SYNC ; +multifd_send_fill_packet (p,0 ); + p->sync = false; +qemu_mutex_unlock (&p->mutex); + +trace_multifd_send (p->id, packet_num,0 , flags |MULTIFD_FLAG_SYNC , + p->next_packet_size); + + ret =qio_channel_write_all (p->c, (void *)p->packet, + p->packet_len, &local_err); +if (ret !=0 ) { +break ; + } + +qemu_sem_post (&p->sem_sync); }else if (p->quit) {qemu_mutex_unlock (&p->mutex);break ;
@@ -1221,7 +1239,7 @@ int multifd_save_setup(void)qemu_mutex_init (&p->mutex);qemu_sem_init (&p->sem,0 );qemu_sem_init (&p->sem_sync,0 ); - p->quit = false; + p->quit = p->sync = false; p->pending_job =0 ; p->id = i; p->pages =multifd_pages_init (page_count);
Signature · 1 line
2.17.1
PATCH v2 4/6 migration/multifd: used must not be 0 for a pending job
In reply to Wei YangAfter thread synchronization request is handled in another case, this means when we only get pending_job when there is used pages.
--- migration/ram.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
migration/ram.c +5 -6
@@ -1132,12 +1132,11 @@ static void *multifd_send_thread(void *opaque)break ; } -if (used) { - ret =qio_channel_writev_all (p->c, p->pages->iov, - used, &local_err); -if (ret !=0 ) { -break ; - } +assert (used); + ret =qio_channel_writev_all (p->c, p->pages->iov, + used, &local_err); +if (ret !=0 ) { +break ; }qemu_mutex_lock (&p->mutex);
Signature · 1 line
2.17.1
PATCH v2 5/6 migration/multifd: use boolean for pending_job is enough
In reply to Wei YangAfter synchronization request is handled in another case, there only could be one pending_job for one send thread at most.
This is fine to use boolean to represent this behavior.
--- migration/ram.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
migration/ram.c +4 -5
@@ -646,7 +646,7 @@ typedef struct {/* should this thread finish */ bool quit;/* thread has work to do */ -int pending_job; +bool pending_job;/* array of pages to sent */ MultiFDPages_t *pages;/* packet allocated len */
@@ -933,7 +933,7 @@ static int multifd_send_pages(RAMState *rs)return -1 ; }if (!p->pending_job) { - p->pending_job++; + p->pending_job = true; next_channel = (i +1 ) %migrate_multifd_channels ();break ; }
@@ -1140,7 +1140,7 @@ static void *multifd_send_thread(void *opaque) }qemu_mutex_lock (&p->mutex); - p->pending_job--; + p->pending_job = false;qemu_mutex_unlock (&p->mutex);qemu_sem_post (&multifd_send_state->channels_ready);
@@ -1238,8 +1238,7 @@ int multifd_save_setup(void)qemu_mutex_init (&p->mutex);qemu_sem_init (&p->sem,0 );qemu_sem_init (&p->sem_sync,0 ); - p->quit = p->sync = false; - p->pending_job =0 ; + p->quit = p->sync = p->pending_job = false; p->id = i; p->pages =multifd_pages_init (page_count); p->packet_len =sizeof (MultiFDPacket_t)
Signature · 1 line
2.17.1
PATCH v2 6/6 migration/multifd: there is no spurious wakeup now
In reply to Wei YangThe spurious wakeup is gone.
--- migration/ram.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
migration/ram.c +2 -2
@@ -1168,8 +1168,8 @@ static void *multifd_send_thread(void *opaque)qemu_mutex_unlock (&p->mutex);break ; }else { -qemu_mutex_unlock (&p->mutex); -/* sometimes there are spurious wakeups */ +/* no other case should trigger me */ +g_assert_not_reached (); } }
Signature · 1 line
2.17.1
Ping for comment.
Quoted 33 lines · Wei Yang
Current send thread could work while the sync mechanism has some problem:
* has spuriously wakeup
* number ofchannels_readywill *overflow* the number of real channelsThe reason is:
* if
MULTIFD_FLAG_SYNCis set in the middle of send thread running, there is one more spurious wakeup
* ifMULTIFD_FLAG_SYNCis set when send thread is not running, there is one morechannels_readybe triggeredTo solve this situation, one new mechanism is introduced to synchronize send threads. The idea is simple, a new field *sync* is introduced to indicate a synchronization is required.
---
v2: rebase on latest codeWei Yang (6):
migration/multifd: move Params update and pages cleanup into
multifd_send_fill_packet()
migration/multifd: notifychannels_readywhen send thread starts migration/multifd: use sync field to synchronize send threads migration/multifd: used must not be 0 for a pending job migration/multifd: use boolean forpending_jobis enough migration/multifd: there is no spurious wakeup now
migration/ram.c| 74 +++++++++++++++++++++++++++++++------------------1 file changed, 47 insertions(+), 27 deletions(-)Signature · 1 line
2.17.1
Signature · 2 lines
Wei Yang Help you, Help me