migration: Yield coroutine when receiving MIG_CMD_POSTCOPY_LISTEN
PATCH 1/1 migration: Yield coroutine when receiving MIG_CMD_POSTCOPY_LISTEN
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source connect() the preemption channel in postcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphore postcopy_qemufile_dst_done for the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphore postcopy_qemufile_dst_done is not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle the connect() event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise,
the qemu_get_be64() in ram_load_precopy() will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.
Fix this by yielding the load coroutine when receiving MIG_CMD_POSTCOPY_LISTEN so the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)
migration/savevm.c +5 -0
@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }return loadvm_postcopy_handle_listen (mis);case MIG_CMD_POSTCOPY_RUN :
Signature · 1 line
2.39.3
Quoted 33 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphorepostcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.
Quoted 6 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>
This seems to be a regression issue caused by this commit:737840e2c6ea (migration: Use the number of transferred bytes directly)
Adding qemu_fflush back to migration_rate_exceeded() or ram_save_iterate seems to work (might not be a good fix though).
Quoted 9 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 8 lines
migration/savevm.c +5
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }
The above could be moved to loadvm_postcopy_handle_listen().
Another option is to follow the old way (i.e. pre_7_2) to do postcopy_preempt_setup in migrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.
Quoted 6 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3
Quoted 5 lines · Wei W
Quoted 4 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
Quoted 13 lines
Quoted 5 lines
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in theQuoted 8 lines
event loop. and doesn't post the semaphore
postcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).
Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.
Quoted 5 lines
Quoted 5 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.
How hard would this reproduce?
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 25 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 11 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);
Quoted 8 lines
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 6 lines
migration/savevm.c +5
-0case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().
I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1) accept() happens, and (2) receive LISTEN cmd here. What happens if the accept() request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can connect() after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.
One right fix that I can think of is moving the sem_wait(&done) into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't think sem_wait() will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..
https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_destination
I attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 5 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.
IIUC we still need that pre_7_2 stuff and keep the postponed connect() to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.
You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3
===8<===
migration/migration.c +6 -0
@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);
migration/postcopy-ram.c +20 -6
@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 )) { +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main +* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side. +*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + } + }/* * This thread needs to be created after the temp pages because * it'll fetch RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately.
@@ -1743,12 +1763,6 @@ void *postcopy_preempt_thread(void *opaque)qemu_sem_post (&mis->thread_sync_sem); -/* -* The preempt channel is established in asynchronous way. Wait -* for its completion. -*/ -qemu_sem_wait (&mis->postcopy_qemufile_dst_done); -/* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ qemu_mutex_lock (&mis->postcopy_prio_thread_mutex);while (preempt_thread_should_run (mis)) {
Signature · 5 lines
2.44.0 -- Peter Xu
Quoted 165 lines · Peter Xu
Quoted 5 lines · Wei W
Quoted 4 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
Quoted 13 lines
Quoted 5 lines
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in theQuoted 8 lines
event loop. and doesn't post the semaphore
postcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 5 lines
Quoted 5 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 25 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 11 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);Quoted 8 lines
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 6 lines
migration/savevm.c +5
-0case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1)
accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_destination
I attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 5 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3===8<===
migration/migration.c +6
-0@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);migration/postcopy-ram.c +8
-0@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 )) { +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main
migration/postcopy-ram.c int postcopy_ram_incoming_setup(MigrationIncomingState *mis) · line 1274
Quoted 10 lines
migration/postcopy-ram.c +10
-0+* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side. +*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield ();
I think the correct way to do this these days isaio_co_reschedule_self().
Anyway, what we are yielding to here? I see qemu_loadvm_state_main() called from a bunch of places, it's not clear to me where will the execution resume after yielding. Is that end up going to be migration_incoming_process()?
I don't know much about the postcopy parts, excuse my ignorance.
Quoted 20 lines
migration/postcopy-ram.c +2
-6+ } + }/* * This thread needs to be created after the temp pages because * it'll fetch RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately. @@ -1743,12 +1763,6 @@ void *postcopy_preempt_thread(void *opaque)qemu_sem_post (&mis->thread_sync_sem); -/* -* The preempt channel is established in asynchronous way. Wait -* for its completion. -*/ -qemu_sem_wait (&mis->postcopy_qemufile_dst_done); -/* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ qemu_mutex_lock (&mis->postcopy_prio_thread_mutex);while (preempt_thread_should_run (mis)) {Signature · 1 line
2.44.0
Quoted 172 lines · Fabiano Rosas
Quoted 170 lines · Peter Xu
Quoted 5 lines · Wei W
Quoted 4 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
Quoted 13 lines
Quoted 5 lines
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in theQuoted 8 lines
event loop. and doesn't post the semaphore
postcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 5 lines
Quoted 5 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 25 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 11 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);Quoted 8 lines
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 6 lines
migration/savevm.c +5
-0case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1)
accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_destination
I attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 5 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3===8<===
migration/migration.c +6
-0@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);migration/postcopy-ram.c +13
-0@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 )) { +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main +* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side.
Quoted 8 lines
migration/postcopy-ram.c int postcopy_ram_incoming_setup(MigrationIncomingState *mis) · line 1274
Quoted 5 lines
migration/postcopy-ram.c +5
-0+*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield ();I think the correct way to do this these days is
aio_co_reschedule_self().
The helper checks old v.s. new contexts, where here we want to pass in the current context. Would that be a no-op then?
Quoted 5 lines
Anyway, what we are yielding to here? I see
qemu_loadvm_state_main()called from a bunch of places, it's not clear to me where will the execution resume after yielding. Is that end up going to bemigration_incoming_process()?
In this specific case it should try to yield to the port listener that is waiting for the preempt channel, aka, socket_accept_incoming_migration(), and ultimately it'll kick off this sem, by:
socket_accept_incoming_migration ->
migration_ioc_process_incoming ->
postcopy_preempt_new_channel
Quoted 2 lines
I don't know much about the postcopy parts, excuse my ignorance.
Not a problem at all, please shoot if there's any questions either here or elsewhere. You're going to maintain it anyway as part of the migration code base. :-D
Thanks,
Signature · 1 line
Peter Xu
Quoted 191 lines · Peter Xu
Quoted 172 lines · Fabiano Rosas
Quoted 170 lines · Peter Xu
Quoted 5 lines · Wei W
Quoted 4 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
Quoted 13 lines
Quoted 5 lines
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in theQuoted 8 lines
event loop. and doesn't post the semaphore
postcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 5 lines
Quoted 5 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 25 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 11 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);Quoted 8 lines
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 6 lines
migration/savevm.c +5
-0case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1)
accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_destination
I attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 5 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3===8<===
migration/migration.c +6
-0@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);migration/postcopy-ram.c +13
-0@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 )) { +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main +* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side. Quoted 8 lines
migration/postcopy-ram.c int postcopy_ram_incoming_setup(MigrationIncomingState *mis) · line 1274
Quoted 5 lines
migration/postcopy-ram.c +5
-0+*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield ();I think the correct way to do this these days is
aio_co_reschedule_self().The helper checks old v.s. new contexts, where here we want to pass in the current context. Would that be a no-op then?
Quoted 5 lines
Anyway, what we are yielding to here? I see
qemu_loadvm_state_main()called from a bunch of places, it's not clear to me where will the execution resume after yielding. Is that end up going to bemigration_incoming_process()?
Quoted 7 lines
In this specific case it should try to yield to the port listener that is waiting for the preempt channel, aka,
socket_accept_incoming_migration(), and ultimately it'll kick off this sem, by:
socket_accept_incoming_migration->
migration_ioc_process_incoming->
postcopy_preempt_new_channel
Ok, I think I get it. So the issue is just a plain old "blocking the main loop" kind of bug. We have in ram_load_precopy:
/*
* Yield periodically to let main loop run, but an iteration of
* the main loop is expensive, so do it each some iterations
*/
if ((i & 32767) == 0 && qemu_in_coroutine()) {
aio_co_schedule(qemu_get_current_aio_context(),
qemu_coroutine_self());
qemu_coroutine_yield();
}That's similar to why I had to move multifd_send_setup() to the migration thread, we need to allow glib_pollfds_poll() to run so it dispatches the listener callbacks.
Quoted 7 lines
Quoted 3 lines
I don't know much about the postcopy parts, excuse my ignorance.
Not a problem at all, please shoot if there's any questions either here or elsewhere. You're going to maintain it anyway as part of the migration code base. :-D
/me runs
But yeah, I didn't spend enough time looking at this code yet to form a good mental picture. I only looked at the super-specific recovery cases.
Quoted 2 lines
Thanks,
Quoted 32 lines · Wei W
Quoted 5 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
Quoted 4 lines
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
Quoted 13 lines
Quoted 5 lines
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in theQuoted 8 lines
event loop. and doesn't post the semaphore
postcopy_qemufile_dst_donefor the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
We can manually make this easier to reproduce by adding the following code to make the destination busier to load the pages:
migration/ram.c +2 -0
@@ -4232,6 +4232,7 @@ static int ram_load_precopy(QEMUFile *f) {MigrationIncomingState *mis =migration_incoming_get_current ();int flags =0 , ret =0 , invalid_flags =0 , len =0 , i =0 ; +volatile unsigned long long a;if (!migrate_compress ()) { invalid_flags |=RAM_SAVE_FLAG_COMPRESS_PAGE ;
@@ -4347,6 +4348,7 @@ static int ram_load_precopy(QEMUFile *f)break ;case RAM_SAVE_FLAG_PAGE : +for (a =0 ; a <100000000 ; a++);qemu_get_buffer (f, host,TARGET_PAGE_SIZE );break ;
Quoted 46 lines
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 25 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 11 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile *f)return loadvm_postcopy_handle_advise (mis, len);Quoted 8 lines
migration/savevm.c static int loadvm_process_command(QEMUFile *f) · line 2452
Quoted 6 lines
migration/savevm.c +5
-0case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1)
accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?
Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.
I think the following code:
postcopy_start() ->
postcopy_preempt_establish_channel() -> qemu_sem_wait(&s->postcopy_qemufile_src_sem);
can guarantee that the connect() syscall is successful so the destination side receives the connect() request before it loads the LISTEN command, otherwise it won't post the sem:
postcopy_preempt_send_channel_new() ->
postcopy_preempt_send_channel_done() ->
qemu_sem_post(&s->postcopy_qemufile_src_sem);Quoted 9 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..
Quoted 4 lines
https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_destination
I attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Sadly it doesn't work, there is an unknown segfault.
Quoted 7 lines
Quoted 6 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see Peter and others' opinions.
Quoted 8 lines
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Per my very simple test, using post-copy preemption to live migrate an 8G VM:
w/o this patch: 121ms in avg in 5 tries
w/ this patch: 115ms in avg in 5 tries
So it seems the overhead introduced is not too high (maybe ignorable?).
Quoted 74 lines
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3===8<===
migration/migration.c +6
-0@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);migration/postcopy-ram.c +20
-6@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 )) { +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main +* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side. +*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + } + }/* * This thread needs to be created after the temp pages because * it'll fetch RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately. @@ -1743,12 +1763,6 @@ void *postcopy_preempt_thread(void *opaque)qemu_sem_post (&mis->thread_sync_sem); -/* -* The preempt channel is established in asynchronous way. Wait -* for its completion. -*/ -qemu_sem_wait (&mis->postcopy_qemufile_dst_done); -/* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ qemu_mutex_lock (&mis->postcopy_prio_thread_mutex);while (preempt_thread_should_run (mis)) {
Quoted 57 lines · On Tuesday, April 2, 2024 2:56 PM, Wang, Lei4 wrote:
Quoted 9 lines · Wei W
Quoted 9 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
Quoted 8 lines
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulkRAM
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphore
postcopy_qemufile_dst_doneforQuoted 21 lines
Quoted 7 lines
Quoted 7 lines
the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
We can manually make this easier to reproduce by adding the following code to make the destination busier to load the pages:
migration/ram.c index 0ad9fbba48..0b42877e1f renamed from migration/ram.c +0
-0100644
migration/ram.c +1
-0@@ -4232,6 +4232,7 @@ static int ram_load_precopy(QEMUFile *f) {MigrationIncomingState *mis =migration_incoming_get_current ();int flags =0 , ret =0 , invalid_flags =0 , len =0 , i =0 ; +volatile unsigned long long a;
migration/ram.c static int ram_load_precopy(QEMUFile *f) {
Quoted 10 lines
migration/ram.c +2
-0if (!migrate_compress ()) { invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; @@ -4347,6 +4348,7 @@ static int ram_load_precopy(QEMUFile *f) break;case RAM_SAVE_FLAG_PAGE : +for (a =0 ; a <100000000 ; a++);qemu_get_buffer (f, host,TARGET_PAGE_SIZE );break ;
Which version of QEMU are you using?
I tried with the latest upstream QEMU (e.g. v8.2.0 release, 1600b9f46b1bd), it's always reproducible without any changes (with local migration tests).
Quoted 62 lines
Quoted 23 lines
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 19 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 4 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 5 lines
Quoted 5 lines
migration/savevm.c static int
Quoted 5 lines
migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int
loadvm_process_command(QEMUFile *f)Quoted 21 lines
Quoted 10 lines
migration/savevm.c static int · line 2452
Quoted 8 lines
migration/savevm.c +5
-0return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of
(1)accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.I think the following code:
postcopy_start()->postcopy_preempt_establish_channel() -> qemu_sem_wait(&s->postcopy_qemufile_src_sem);
Quoted 8 lines
can guarantee that the
connect()syscall is successful so the destination side receives theconnect()request before it loads the LISTEN command, otherwise it won't post the sem:
postcopy_preempt_send_channel_new()->postcopy_preempt_send_channel_done() -> qemu_sem_post(&s->postcopy_qemufile_src_sem);
Yes. But as mentioned in another thread, connect() and accept() are async. So in theory accept() could still come later after the LISTEN command.
Quoted 34 lines
Quoted 9 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..Quoted 5 lines
https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration
_destinationI attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Sadly it doesn't work, there is an unknown segfault.
Quoted 6 lines
Quoted 6 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let'ssee Peter and others' opinions.
Quoted 9 lines
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Quoted 6 lines
Per my very simple test, using post-copy preemption to live migrate an 8G VM:
w/o this patch: 121ms in avg in 5 tries
w/ this patch: 115ms in avg in 5 triesSo it seems the overhead introduced is not too high (maybe ignorable?).
You could just measure the time for the added qemu_coroutine_yield() part. The time will depend on how many events happen to be there waiting for a dispatch.
Quoted 62 lines · On 4/2/2024 15:25, Wang, Wei W wrote:> On Tuesday, April 2, 2024 2:56 PM…
Quoted 62 lines
Quoted 9 lines · Wei W
Quoted 9 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
Quoted 8 lines
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulkRAM
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphore
postcopy_qemufile_dst_doneforQuoted 21 lines
Quoted 7 lines
Quoted 7 lines
the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
We can manually make this easier to reproduce by adding the following code to make the destination busier to load the pages:
migration/ram.c index 0ad9fbba48..0b42877e1f renamed from migration/ram.c +0
-0100644
migration/ram.c +2
-0@@ -4232,6 +4232,7 @@ static int ram_load_precopy(QEMUFile *f) {MigrationIncomingState *mis =migration_incoming_get_current ();int flags =0 , ret =0 , invalid_flags =0 , len =0 , i =0 ; +volatile unsigned long long a;if (!migrate_compress ()) { invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; @@ -4347,6 +4348,7 @@ static int ram_load_precopy(QEMUFile *f) break;
Quoted 9 lines
Quoted 5 lines
case
RAM_SAVE_FLAG_PAGE:+ for (a = 0; a < 100000000; a++); qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break;Which version of QEMU are you using?
I tried with the latest upstream QEMU (e.g. v8.2.0 release,1600b9f46b1bd), it's always reproducible without any changes (with local migration tests).
I'm using the latest tip:
6af9d12c88b9720f209912f6e4b01fefe5906d59
and it cannot be reproduced without the modification.
Quoted 68 lines
Quoted 68 lines
Quoted 25 lines
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 19 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 4 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 5 lines
Quoted 5 lines
migration/savevm.c static int
Quoted 5 lines
migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int
loadvm_process_command(QEMUFile *f)Quoted 21 lines
Quoted 10 lines
migration/savevm.c static int · line 2452
Quoted 8 lines
migration/savevm.c +5
-0return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of
(1)accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.I think the following code:
postcopy_start()->postcopy_preempt_establish_channel() -> qemu_sem_wait(&s->postcopy_qemufile_src_sem);can guarantee that the
connect()syscall is successful so the destination side receives theconnect()request before it loads the LISTEN command, otherwise it won't post the sem:
Quoted 7 lines
Quoted 4 lines
postcopy_preempt_send_channel_new()->postcopy_preempt_send_channel_done() -> qemu_sem_post(&s->postcopy_qemufile_src_sem);Yes. But as mentioned in another thread,
connect()andaccept()are async. So in theoryaccept()could still come later after the LISTEN command.
IIUC accept() is the callback and will be triggered by the connect() event.
The reason accept() is not called in the destination is the main loop doesn't get a chance to handle other events (connect()), so if we can guarantee connect() is before LISTEN, then when handling the LISTEN cmd, we yield to the main loop and the connect() event is there, then we can handle it by calling the accept():
qio_net_listener_channel_func
qio_channel_socket_accept
qemu_accept
accept
so it seems the case accept() comes alter after LISTEN is in our expectation?
Quoted 37 lines
Quoted 37 lines
Quoted 10 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..Quoted 5 lines
https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration
_destinationI attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Sadly it doesn't work, there is an unknown segfault.
Quoted 6 lines
Quoted 6 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let'ssee Peter and others' opinions.
Quoted 9 lines
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Per my very simple test, using post-copy preemption to live migrate an 8G VM:
Quoted 7 lines
Quoted 4 lines
w/o this patch: 121ms in avg in 5 tries
w/ this patch: 115ms in avg in 5 triesSo it seems the overhead introduced is not too high (maybe ignorable?).
You could just measure the time for the added
qemu_coroutine_yield()part. The time will depend on how many events happen to be there waiting for a dispatch.
Still less than 1ms (0.05ms).
Indeed, that may depends on how many events needs to be handled.
Quoted 74 lines · Lei
Quoted 62 lines · On 4/2/2024 15:25, Wang, Wei W wrote:> On Tuesday, April 2, 2024 2:56 PM…
Quoted 62 lines
Quoted 9 lines · Wei W
Quoted 9 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
Quoted 8 lines
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulkRAM
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphore
postcopy_qemufile_dst_doneforQuoted 21 lines
Quoted 7 lines
Quoted 7 lines
the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
We can manually make this easier to reproduce by adding the following code to make the destination busier to load the pages:
migration/ram.c index 0ad9fbba48..0b42877e1f renamed from migration/ram.c +0
-0100644
migration/ram.c +2
-0@@ -4232,6 +4232,7 @@ static int ram_load_precopy(QEMUFile *f) {MigrationIncomingState *mis =migration_incoming_get_current ();int flags =0 , ret =0 , invalid_flags =0 , len =0 , i =0 ; +volatile unsigned long long a;if (!migrate_compress ()) { invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; @@ -4347,6 +4348,7 @@ static int ram_load_precopy(QEMUFile *f) break;Quoted 9 lines
Quoted 5 lines
case
RAM_SAVE_FLAG_PAGE:+ for (a = 0; a < 100000000; a++); qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break;Which version of QEMU are you using?
I tried with the latest upstream QEMU (e.g. v8.2.0 release,1600b9f46b1bd), it's always reproducible without any changes (with local migration tests).
Quoted 5 lines
I'm using the latest tip:
6af9d12c88b9720f209912f6e4b01fefe5906d59and it cannot be reproduced without the modification.
It does look like there's some mistery in reproducability. I can't reproduce it locally with v8.2.0 tag, neither can I reproduce it with the code above.
I've requested our QE team to involve, but before that I may need help on any verifications.
Quoted 85 lines
Quoted 69 lines
Quoted 69 lines
Quoted 26 lines
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 19 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 4 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 5 lines
Quoted 5 lines
migration/savevm.c static int
Quoted 5 lines
migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int
loadvm_process_command(QEMUFile *f)Quoted 21 lines
Quoted 10 lines
migration/savevm.c static int · line 2452
Quoted 8 lines
migration/savevm.c +5
-0return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of
(1)accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.I think the following code:
postcopy_start()->postcopy_preempt_establish_channel() -> qemu_sem_wait(&s->postcopy_qemufile_src_sem);can guarantee that the
connect()syscall is successful so the destination side receives theconnect()request before it loads the LISTEN command, otherwise it won't post the sem:Quoted 7 lines
Quoted 4 lines
postcopy_preempt_send_channel_new()->postcopy_preempt_send_channel_done() -> qemu_sem_post(&s->postcopy_qemufile_src_sem);Yes. But as mentioned in another thread,
connect()andaccept()are async. So in theoryaccept()could still come later after the LISTEN command.IIUC
accept()is the callback and will be triggered by theconnect()event.The reason
accept()is not called in the destination is the main loop doesn't get a chance to handle other events (connect()), so if we can guaranteeconnect()is before LISTEN, then when handling the LISTEN cmd, we yield to the main loop and theconnect()event is there, then we can handle it by calling theaccept():
Quoted 6 lines
qio_net_listener_channel_func
qio_channel_socket_accept
qemu_accept
acceptso it seems the case
accept()comes alter after LISTEN is in our expectation?
The major thing uncertain to me is "accept() will return with a valid fd" on dest host is not guaranteed to order against anything.
For example, I won't be surprised if a kernel implementation provides an async model of "accept()" syscall, so that even if the other side returned with "connect()", the "accept()" can still fetch nothing if the async model will need a delay for the new channel to be finally delivered to the "accept()" thread context. It just sounds like tricky to rely on such thing.
What I proposed below shouldn't rely on any kernel details on how accept() could be implemented, it simply waits for the fd to be created before doing anything else (including creating the preempt thread and process packed data).
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration
_destinationI attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Sadly it doesn't work, there is an unknown segfault.
Could you paste the stack of the segfault ("(gdb) thread apply all bt")? Or help to figure out what is wrong?
Since I cannot reproduce myself, I may need your help debugging this. If you agree with what I said above and agree on such fix, please also feel free to go ahead and fix the segfault.
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let'ssee Peter and others' opinions.
Quoted 9 lines
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Per my very simple test, using post-copy preemption to live migrate an 8G VM:
w/o this patch: 121ms in avg in 5 tries
w/ this patch: 115ms in avg in 5 tries
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
So it seems the overhead introduced is not too high (maybe ignorable?).
You could just measure the time for the added
qemu_coroutine_yield()part. The time will depend on how many events happen to be there waiting for a dispatch.Still less than 1ms (0.05ms).
Indeed, that may depends on how many events needs to be handled.
That sounds fine as of now. I think it means if we can come up with a solid plan this week (without intrusive changes), we still have chance to land it in 9.0.
Thanks,
Signature · 1 line
Peter Xu
On 4/3/2024 5:39, Peter Xu wrote:>>>>>
Quoted 59 lines
Quoted 42 lines
Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of
(1)accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.I think the following code:
postcopy_start()->postcopy_preempt_establish_channel() -> qemu_sem_wait(&s->postcopy_qemufile_src_sem);can guarantee that the
connect()syscall is successful so the destination side receives theconnect()request before it loads the LISTEN command, otherwise it won't post the sem:Quoted 7 lines
Quoted 4 lines
postcopy_preempt_send_channel_new()->postcopy_preempt_send_channel_done() -> qemu_sem_post(&s->postcopy_qemufile_src_sem);Yes. But as mentioned in another thread,
connect()andaccept()are async. So in theoryaccept()could still come later after the LISTEN command.IIUC
accept()is the callback and will be triggered by theconnect()event.The reason
accept()is not called in the destination is the main loop doesn't get a chance to handle other events (connect()), so if we can guaranteeconnect()is before LISTEN, then when handling the LISTEN cmd, we yield to the main loop and theconnect()event is there, then we can handle it by calling theaccept():Quoted 6 lines
qio_net_listener_channel_func
qio_channel_socket_accept
qemu_accept
acceptso it seems the case
accept()comes alter after LISTEN is in our expectation?The major thing uncertain to me is "
accept()will return with a valid fd" on dest host is not guaranteed to order against anything.For example, I won't be surprised if a kernel implementation provides an async model of "
accept()" syscall, so that even if the other side returned with "connect()", the "accept()" can still fetch nothing if the async model will need a delay for the new channel to be finally delivered to the "accept()" thread context. It just sounds like tricky to rely on such thing.
Quoted 4 lines
What I proposed below shouldn't rely on any kernel details on how
accept()could be implemented, it simply waits for the fd to be created before doing anything else (including creating the preempt thread and process packed data).
Thanks for the detailed explanation!
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration
_destinationI attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Sadly it doesn't work, there is an unknown segfault.
Could you paste the stack of the segfault ("(gdb) thread apply all bt")? Or help to figure out what is wrong?
Since I cannot reproduce myself, I may need your help debugging this. If you agree with what I said above and agree on such fix, please also feel free to go ahead and fix the segfault.
We should change the following line from
while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 100)) {to
while (qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 100)) {After that fix, test passed and no segfault.
Given that the test shows a yield to the main loop won't introduce much overhead (<1ms), how about first yield unconditionally, then we enter the while loop to wait for several ms and yield periodically?
Quoted 7 lines · Lei
We should change the following line from
while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 100)) {to
while (qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done, 100)) {
Stupid me.. :( Thanks for figuring this out.
Quoted 6 lines
After that fix, test passed and no segfault.
Given that the test shows a yield to the main loop won't introduce much overhead (<1ms), how about first yield unconditionally, then we enter the while loop to wait for several ms and yield periodically?
Shouldn't the expectation be that this should return immediately without a wait? We're already processing LISTEN command, and on the source as you said it was much after the connect(). It won't guarantee the ordering but IIUC the majority should still have a direct hit?
What we can do though is reducing the 100ms timeout if you see that's perhaps a risk of having too large a downtime when by accident. We can even do it in a tight loop here considering downtime is important, but to provide an intermediate ground: how about 100ms -> 1ms poll?
If you agree (and also to Wei; please review this and comment if there's any!), would you write up the commit log, fully test it in whatever way you could, and resend as a formal patch (please do this before Friday if possible)? You can keep a "Suggested-by:" for me. I want to queue it for rc3 if it can catch it. It seems important if Wei can always reproduce it.
Thanks,
Signature · 1 line
Peter Xu
Quoted 25 lines · Peter Xu
Quoted 3 lines · Lei
We should change the following line from
while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,100)) {
Quoted 4 lines
to
while (qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,100)) {
Stupid me.. :( Thanks for figuring this out.
Quoted 6 lines
After that fix, test passed and no segfault.
Given that the test shows a yield to the main loop won't introduce much overhead (<1ms), how about first yield unconditionally, then we enter the while loop to wait for several ms and yield periodically?
Shouldn't the expectation be that this should return immediately without a wait? We're already processing LISTEN command, and on the source as you said it was much after the
connect(). It won't guarantee the ordering but IIUC the majority should still have a direct hit?
Quoted 4 lines
What we can do though is reducing the 100ms timeout if you see that's perhaps a risk of having too large a downtime when by accident. We can even do it in a tight loop here considering downtime is important, but to provide an intermediate ground: how about 100ms -> 1ms poll?
Would it be better to use busy wait here, instead of blocking for even 1ms here? It's likely that the preempt channel is waiting for the main thread to dispatch for accept(), but we are calling qemu_sem_timedwait here to block the main thread for 1 more ms.
Quoted 6 lines
If you agree (and also to Wei; please review this and comment if there's any!), would you write up the commit log, fully test it in whatever way you could, and resend as a formal patch (please do this before Friday if possible)? You can keep a "Suggested-by:" for me. I want to queue it for
rc3 if it can catch it. It seems important if Wei can always reproduce it.
Not sure if Lei would be able to online as the following two days are Chinese holiday. If not, I could help take over to send late tomorrow. Let's see.
Quoted 26 lines · Wei W
Quoted 25 lines · Peter Xu
Quoted 3 lines · Lei
We should change the following line from
while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,100)) {
Quoted 4 lines
to
while (qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,100)) {
Stupid me.. :( Thanks for figuring this out.
Quoted 6 lines
After that fix, test passed and no segfault.
Given that the test shows a yield to the main loop won't introduce much overhead (<1ms), how about first yield unconditionally, then we enter the while loop to wait for several ms and yield periodically?
Shouldn't the expectation be that this should return immediately without a wait? We're already processing LISTEN command, and on the source as you said it was much after the
connect(). It won't guarantee the ordering but IIUC the majority should still have a direct hit?
Quoted 8 lines
Quoted 4 lines
What we can do though is reducing the 100ms timeout if you see that's perhaps a risk of having too large a downtime when by accident. We can even do it in a tight loop here considering downtime is important, but to provide an intermediate ground: how about 100ms -> 1ms poll?
Would it be better to use busy wait here, instead of blocking for even 1ms here? It's likely that the preempt channel is waiting for the main thread to dispatch for
accept(), but we are callingqemu_sem_timedwaithere to block the main thread for 1 more ms.
I think it's about the expectation of whether we should already received that sem post. My understanding is in most cases we should directly return and avoid such wait.
Per my previous experience, 1ms is not a major issue to be added on top of downtime in corner cases like this.
We do have a lot of othre potential optimizations to reduce downtime, or I should say in the other way, that.. there can be a lot of cases where we can hit much larger downtime than expected. Consider when we don't even account downtime for device states for now, either load_state or save_state, we only count RAM but that's far from accurate.. and we do have more chances to optimize. Some are listed here, but some may not:
https://wiki.qemu.org/ToDo/LiveMigration#Optimizations
If you agree with my above "expectation" statement, I'd say we should avoid using a busy loop whenever possible in QEMU unless extremely necessary.
Quoted 11 lines
Quoted 8 lines
If you agree (and also to Wei; please review this and comment if there's any!), would you write up the commit log, fully test it in whatever way you could, and resend as a formal patch (please do this before Friday if possible)? You can keep a "Suggested-by:" for me. I want to queue it for
rc3 if it can catch it. It seems important if Wei can always reproduce it.Not sure if Lei would be able to online as the following two days are Chinese holiday. If not, I could help take over to send late tomorrow. Let's see.
Oops, I forgot that even if I was aware..
Please do so if you can do this. Thank you, Wei! (I hope you can switch some working hours later on!)
Let me know if that doesn't work; it'll be all fine.
Thanks,
Signature · 1 line
Peter Xu
Quoted 78 lines · Peter Xu
Quoted 52 lines · Wei W
Quoted 49 lines · Peter Xu
Quoted 5 lines · Lei
We should change the following line from
while (!qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
100)) {
Quoted 7 lines
to
while (qemu_sem_timedwait(&mis->postcopy_qemufile_dst_done,
100)) {
Stupid me.. :( Thanks for figuring this out.
Quoted 11 lines
After that fix, test passed and no segfault.
Given that the test shows a yield to the main loop won't introduce
much overhead (<1ms), how about first yield unconditionally, then
we enter the while loop to wait for several ms and yield periodically?
Shouldn't the expectation be that this should return immediately
without a wait? We're already processing LISTEN command, and on the
source as you said it was much after the
connect(). It won'tguarantee the ordering but IIUC the majority should still have a direct hit?
Quoted 7 lines
Quoted 7 lines
What we can do though is reducing the 100ms timeout if you see
that's perhaps a risk of having too large a downtime when by
accident. We can even do it in a tight loop here considering
downtime is important, but to provide an intermediate ground: how about
100ms -> 1ms poll?
Quoted 3 lines
Would it be better to use busy wait here, instead of blocking for even 1ms
here?
Quoted 3 lines
It's likely that the preempt channel is waiting for the main thread to
dispatch for
accept(), but we are callingqemu_sem_timedwaithere to blockthe main thread for 1 more ms.
Quoted 5 lines
I think it's about the expectation of whether we should already received that
sem post. My understanding is in most cases we should directly return and
avoid such wait.
The assumption of " should already received" might not be true.
On my machine, it seems always "not received".
Quoted 5 lines
Per my previous experience, 1ms is not a major issue to be added on top of
downtime in corner cases like this.
Yeah, in most cases, probably yes. There are some usages requiring relatively low
downtime. Remember NFV usages in early days expect the downtime to be close to
10ms. In this case, adding 1ms would mean ~10% performance regression ☹
Quoted 16 lines
We do have a lot of othre potential optimizations to reduce downtime, or I
should say in the other way, that.. there can be a lot of cases where we can hit
much larger downtime than expected. Consider when we don't even account
downtime for device states for now, either
load_stateorsave_state, we onlycount RAM but that's far from accurate.. and we do have more chances to
optimize. Some are listed here, but some may not:
Quoted 7 lines
https://wiki.qemu.org/ToDo/LiveMigration#Optimizations
If you agree with my above "expectation" statement, I'd say we should avoid
using a busy loop whenever possible in QEMU unless extremely necessary.
I just posted a version with a bit changes from your suggestion:
It still blocks by the sem in the loop, but before that it checks if the channel is
created. If not, yield to main loop. Then when reaching to sem, it is likely not
need to wait.
Another change is that I moved it to the place before we load the states (in
loadvm_handle_cmd_packaged). Because I think that's logically the place we
should ensure the channel to be ready (don’t have a good reason to add it
when handling the LISTEN cmd).
Please let me know if you would disagree anywhere.
Quoted 34 lines · Peter Xu
Quoted 11 lines · Wei W
Quoted 10 lines · On Friday, March 29, 2024 11:32 AM, Wang, Lei4 wrote:
When using the post-copy preemption feature to perform post-copy live migration, the below scenario could lead to a deadlock and the migration will never finish:
- Source
connect()the preemption channel inpostcopy_start().
- Source and the destination side TCP stack finished the 3-way handshake thus the connection is successful.
- The destination side main thread is handling the loading of the bulk RAM pages thus it doesn't start to handle the pending connection event in the event loop. and doesn't post the semaphore
postcopy_qemufile_dst_doneforQuoted 7 lines
Quoted 7 lines
the preemption thread.
- The source side sends non-iterative device states, such as the virtio states.
- The destination main thread starts to receive the virtio states, this process may lead to a page fault (e.g., virtio_load()->vring_avail_idx() may trigger a page fault since the avail ring page may not be received yet).Ouch. Yeah I think this part got overlooked when working on the preempt channel.
Quoted 10 lines
Quoted 10 lines
- The page request is sent back to the source side. Source sends the page content to the destination side preemption thread.
- Since the event is not arrived and the semaphorepostcopy_qemufile_dst_doneis not posted, the preemption thread in destination side is blocked, and cannot handle receiving the page.
- The QEMU main load thread on the destination side is stuck at the page fault, and cannot yield and handle theconnect()event for the preemption channel to unblock the preemption thread.
- The postcopy will stuck there forever since this is a deadlock.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
The key point to reproduce this bug is that the source side is sending pages at a rate faster than the destination handling, otherwise, the
qemu_get_be64()inram_load_precopy()will have a chance to yield since at that time there are no pending data in the buffer to get. This will make this bug harder to be reproduced.How hard would this reproduce?
It seems 100% reproducible on my machine (with migration src and dst on the same physical machine).
Quoted 47 lines
I'm thinking whether this should be 9.0 material or 9.1. It's pretty late for 9.0 though, but we can still discuss.
Quoted 18 lines
Quoted 7 lines
Fix this by yielding the load coroutine when receiving
MIG_CMD_POSTCOPY_LISTENso the main event loop can handle the connection event before loading the non-iterative devices state to avoid the deadlock condition.Signed-off-by Lei Wang <[email protected]>This seems to be a regression issue caused by this commit:
737840e2c6ea(migration: Use the number of transferred bytes directly)Adding
qemu_fflushback tomigration_rate_exceeded()orram_save_iterateseems to work (might not be a good fix though).Quoted 4 lines
--- migration/savevm.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 5 lines
migration/savevm.c static int loadvm_process_command(QEMUFile
Quoted 5 lines
migration/savevm.c index renamed from migration/savevm.c +0
-0
e386c5267f..8fd4dc92f2100644migration/savevm.c +0
-0@@ -2445,6 +2445,11 @@ static int loadvm_process_command(QEMUFile*f)
Quoted 10 lines
migration/savevm.c static int loadvm_process_command(QEMUFile · line 2452
Quoted 8 lines
migration/savevm.c +5
-0return loadvm_postcopy_handle_advise (mis, len);case MIG_CMD_POSTCOPY_LISTEN : +if (migrate_postcopy_preempt () &&qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + }The above could be moved to
loadvm_postcopy_handle_listen().I'm not 100% sure such thing (no matter here or moved into it, which does look cleaner) would work for us.
The problem is I still don't yet see an ordering restricted on top of (1)
accept()happens, and (2) receive LISTEN cmd here. What happens if theaccept()request is not yet received when reaching LISTEN? Or is it always guaranteed the accept(fd) will always be polled here?
Quoted 5 lines
For example, the source QEMU (no matter pre-7.2 or later) will always setup the preempt channel asynchrounously, then IIUC it can
connect()after sending the whole chunk of packed data which should include this LISTEN. I think it means it's not guaranteed this will 100% work, but maybe further reduce the possibility of the race.
When handling LISTEN, it seems guaranteed that connect() on the source has returned (synchronized by postcopy_qemufile_src_sem) , but destination side accept() might not be invoked yet (src connect() and dest accept() are async).
For this approach, we would need to improve the fix with a loop to check if accept() has been called to establish the new channel (yield if not):
if (migrate_postcopy_preempt() && qemu_in_coroutine()) {
while (!mis->postcopy_qemufile_dst) {
aio_co_schedule(qemu_get_current_aio_context(),
qemu_coroutine_self());
qemu_coroutine_yield();
}
}Quoted 20 lines
One right fix that I can think of is moving the
sem_wait(&done)into the main thread too, so we wait for the sem _before_ reading the packed data, so there's no chance of fault. However I don't thinksem_wait()will be smart enough to yield when in a coroutine.. In the long term run I think we should really make migration loadvm to do work in the thread rather than the main thread. I think it means we have one more example to be listed in this todo so that's preferred..https://wiki.qemu.org/ToDo/LiveMigration#Create_a_thread_for_migration_
destinationI attached such draft patch below, but I'm not sure it'll work. Let me know how both of you think about it.
Quoted 5 lines
Another option is to follow the old way (i.e.
pre_7_2) to dopostcopy_preempt_setupinmigrate_fd_connect. This can save the above overhead of switching to the main thread during the downtime. Seems Peter's previous patch already solved the channel disordering issue. Let's see
Quoted 5 lines
Peter and others' opinions.
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.
Sure. I saw your patch (5655aab079) ensures that the preempt channel to be created when PONG is received from the destination, which indicates that the main channel has been ready on the destination side.
Why was it decided to postpone the preempt channel's connect() for newer QEMU only? The old QEMU (before 7.2) still performs preempt channel connect() in migrate_fd_connect(), doesn't it have the disordering issue?
Quoted 40 lines
You raised a good point that this may introduce higher downtime. Did you or Lei tried to measure how large it is? If that is too high, we may need to think another solution, e.g., wait the channel connection before vm stop happens.
Thanks,
Quoted 8 lines
Quoted 7 lines
migration/savevm.c +0
-0return loadvm_postcopy_handle_listen (mis);case
MIG_CMD_POSTCOPY_RUN:
--
2.39.3===8<===
migration/migration.c index renamed from migration/migration.c +0
-0
696762bc64..bacd1328cf100644migration/migration.c +6
-0@@ -2593,6 +2593,12 @@ static int postcopy_start(MigrationState *ms, Error **errp)/* * Make sure the receiver can get incoming pages before we send the rest * of the state +* +* When preempt mode enabled, this must be done after we initiate the +* preempt channel, as destination QEMU will wait for the channel when +* processing the LISTEN request. Currently it may not matter a huge +* deal if we always create the channel asynchrously with a qio task, +* but we need to keep this in mind. */ qemu_savevm_send_postcopy_listen (fb);migration/postcopy-ram.c index renamed from migration/postcopy-ram.c +0
-0
eccff499cb..4f26a89ac9100644
migration/postcopy-ram.c int · line 1262
Quoted 9 lines
migration/postcopy-ram.c +5
-0@@ -1254,6 +1254,26 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) }if (migrate_postcopy_preempt ()) { +/* +* The preempt channel is established in asynchronous way. Wait +* for its completion. +*/ +while (!qemu_sem_timedwait (&mis->postcopy_qemufile_dst_done,100 ))
Why blocking the main thread on the semaphore for some amount of time?
I think we need to yield to the main thread as soon as possible (the accept() would be right there waiting for a dispatch). Maybe replace the sem with mis->postcopy_qemufile_dst?
Quoted 38 lines
migration/postcopy-ram.c +15
-0{ +/* +* Note that to make sure the main thread can still schedule an +* accept() request we need to proactively yield for the main +* loop to run for some duration (100ms in this case), which is +* pretty ugly. +* +* TODO: we should do this in a separate thread to load the VM +* rather than in the main thread, just like the source side. +*/ +if (qemu_in_coroutine ()) { +aio_co_schedule (qemu_get_current_aio_context (), +qemu_coroutine_self ()); +qemu_coroutine_yield (); + } + }/* * This thread needs to be created after the temp pages because
* it'll fetchRAM_CHANNEL_POSTCOPYPostcopyTmpPage immediately.migration/postcopy-ram.c +0
-7@@ -1743,12 +1763,6 @@ void *postcopy_preempt_thread(void *opaque)qemu_sem_post (&mis->thread_sync_sem); -/* -* The preempt channel is established in asynchronous way. Wait -* for its completion. -*/ -qemu_sem_wait (&mis->postcopy_qemufile_dst_done); -/* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ qemu_mutex_lock (&mis->postcopy_prio_thread_mutex);while (preempt_thread_should_run (mis)) { --2.44.0
--
Peter Xu
Quoted 11 lines · Wei W
Quoted 3 lines
IIUC we still need that
pre_7_2stuff and keep the postponedconnect()to make sure the ordering is done properly. Wei, could you elaborate the patch you mentioned? Maybe I missed some spots.Sure. I saw your patch (
5655aab079) ensures that the preempt channel to be created when PONG is received from the destination, which indicates that the main channel has been ready on the destination side.Why was it decided to postpone the preempt channel's
connect()for newer QEMU only? The old QEMU (before 7.2) still performs preempt channelconnect()inmigrate_fd_connect(), doesn't it have the disordering issue?
Yes it has, but the solution actually was not compatible with older binaries, so migration was broken back then.. See:
https://lore.kernel.org/qemu-devel/[email protected]/
My memory was that we didn't have a good solution to fix it without breaking the old protocol, so it can only be fixed on newer binaries.
Thanks,
Signature · 1 line
Peter Xu