migration/postcopy: cleanup related to postcopy
cover migration/postcopy: cleanup related to postcopy
The first one just tries to make function name more easy for reading and understanding.
The next two patches are related to PostcopyState.
Wei Yang (3):
migration/postcopy: rename postcopy_ram_enable_notify to
postcopy_ram_incoming_setup
migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup
when state is ADVISE
migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case
properly
migration/migration.c | 3 +-- migration/postcopy-ram.c | 17 +++++++++++------ migration/postcopy-ram.h | 5 +++-- migration/savevm.c | 13 +++++++------ 4 files changed, 22 insertions(+), 16 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/3 migration/postcopy: rename postcopy_ram_enable_notify to postcopy_ram_incoming_setup
Function postcopy_ram_incoming_setup and postcopy_ram_incoming_cleanup is a pair. Rename to make it clear for audience.
--- migration/postcopy-ram.c | 4 ++-- migration/postcopy-ram.h | 2 +- migration/savevm.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
migration/postcopy-ram.c +2 -2
@@ -1094,7 +1094,7 @@ retry:return NULL ; } -int postcopy_ram_enable_notify (MigrationIncomingState *mis) +int postcopy_ram_incoming_setup (MigrationIncomingState *mis) {/* Open the fd for the kernel to give us userfaults */ mis->userfault_fd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK );
@@ -1321,7 +1321,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,return -1 ; } -int postcopy_ram_enable_notify (MigrationIncomingState *mis) +int postcopy_ram_incoming_setup (MigrationIncomingState *mis) {assert (0 );return -1 ;
migration/postcopy-ram.h +1 -1
@@ -20,7 +20,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis);* Make all of RAM sensitive to accesses to areas that haven't yet been written * and wire up anything necessary to deal with it. */ -int postcopy_ram_enable_notify (MigrationIncomingState *mis); +int postcopy_ram_incoming_setup (MigrationIncomingState *mis);/* * Initialise postcopy-ram, setting the RAM to a state where we can go into
migration/savevm.c +1 -1
@@ -1865,7 +1865,7 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)* shouldn't be doing anything yet so don't actually expect requests */ if (migrate_postcopy_ram ()) { -if (postcopy_ram_enable_notify (mis)) { +if (postcopy_ram_incoming_setup (mis)) {postcopy_ram_incoming_cleanup (mis);return -1 ; }
Signature · 1 line
2.17.1
Re: [PATCH 1/3] migration/postcopy: rename postcopy_ram_enable_notify to postcopy_ram_incoming_setup
Quoted 4 lines · * Wei Yang ([email protected]) wrote:
Function
postcopy_ram_incoming_setupandpostcopy_ram_incoming_cleanupis a pair. Rename to make it clear for audience.Signed-off-by Wei Yang <[email protected]>
Yes OK.
It really initially just setup the userfault, but it also kicks off the fault thread as well so it has got a bit more hairy.
Quoted 57 lines
--- migration/postcopy-ram.c | 4 ++-- migration/postcopy-ram.h | 2 +- migration/savevm.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)migration/postcopy-ram.c +2
-2@@ -1094,7 +1094,7 @@ retry:return NULL ; } -int postcopy_ram_enable_notify (MigrationIncomingState *mis) +int postcopy_ram_incoming_setup (MigrationIncomingState *mis) {/* Open the fd for the kernel to give us userfaults */ mis->userfault_fd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK );@@ -1321,7 +1321,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,return -1 ; } -int postcopy_ram_enable_notify (MigrationIncomingState *mis) +int postcopy_ram_incoming_setup (MigrationIncomingState *mis) {assert (0 );return -1 ;migration/postcopy-ram.h +1
-1@@ -20,7 +20,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis);* Make all of RAM sensitive to accesses to areas that haven't yet been written * and wire up anything necessary to deal with it. */ -int postcopy_ram_enable_notify (MigrationIncomingState *mis); +int postcopy_ram_incoming_setup (MigrationIncomingState *mis);/* * Initialise postcopy-ram, setting the RAM to a state where we can go into migration/savevm.c +1
-1@@ -1865,7 +1865,7 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)* shouldn't be doing anything yet so don't actually expect requests */ if (migrate_postcopy_ram ()) { -if (postcopy_ram_enable_notify (mis)) { +if (postcopy_ram_incoming_setup (mis)) {postcopy_ram_incoming_cleanup (mis);return -1 ; }Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 2/3 migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE
In reply to Wei Yangpostcopy_ram_incoming_cleanup() does cleanup for postcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.
This means there is nothing to cleanup when migration is still ADVISE state.
--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)
migration/migration.c +0 -1
@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)* but managed to complete within the precopy period, we can use * the normal exit. */ -postcopy_ram_incoming_cleanup (mis); }else if (ret >=0 ) {/* * Postcopy was started, cleanup should happen at the end of the
Signature · 1 line
2.17.1
Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE
Quoted 17 lines · * Wei Yang ([email protected]) wrote:
postcopy_ram_incoming_cleanup()does cleanup forpostcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.This means there is nothing to cleanup when migration is still ADVISE state.
Signed-off-by Wei Yang <[email protected]>--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)migration/migration.c +0
-0
migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466
Quoted 7 lines
migration/migration.c +0
-1@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)* but managed to complete within the precopy period, we can use * the normal exit. */ -postcopy_ram_incoming_cleanup (mis); }else if (ret >=0 ) {/* * Postcopy was started, cleanup should happen at the end of the
I think that misses the cleanup of mlock that corresponds to the munlockall in postcopy_ram_supported_by_host - that's called very early on; I think in the advise stage.
Dave
Quoted 3 lines
Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:
postcopy_ram_incoming_cleanup()does cleanup forpostcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.This means there is nothing to cleanup when migration is still ADVISE state.
Signed-off-by Wei Yang <[email protected]>--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)
Quoted 17 lines
migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466
Quoted 12 lines
migration/migration.c +0
-1@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)* but managed to complete within the precopy period, we can use * the normal exit. */ -postcopy_ram_incoming_cleanup (mis); }else if (ret >=0 ) {/* * Postcopy was started, cleanup should happen at the end of the I think that misses the cleanup of mlock that corresponds to the munlockall in
postcopy_ram_supported_by_host- that's called very early on; I think in the advise stage.
Thanks you are right.
BTW, do we need to check enable_mlock when calling munlockall() in postcopy_ram_supported_by_host() ?
Quoted 7 lines
Dave
Quoted 3 lines
Signature · 1 line
2.17.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE
Quoted 32 lines · * Wei Yang ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:
postcopy_ram_incoming_cleanup()does cleanup forpostcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.This means there is nothing to cleanup when migration is still ADVISE state.
Signed-off-by Wei Yang <[email protected]>--- migration/migration.c | 1 - 1 file changed, 1 deletion(-)Quoted 17 lines
migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466
Quoted 12 lines
migration/migration.c +0
-1@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)* but managed to complete within the precopy period, we can use * the normal exit. */ -postcopy_ram_incoming_cleanup (mis); }else if (ret >=0 ) {/* * Postcopy was started, cleanup should happen at the end of the I think that misses the cleanup of mlock that corresponds to the munlockall in
postcopy_ram_supported_by_host- that's called very early on; I think in the advise stage.
Quoted 4 lines
Thanks you are right.
BTW, do we need to check
enable_mlockwhen callingmunlockall()inpostcopy_ram_supported_by_host()?
I don't think so; it does an extra munlock in that case when nothing should be locked anyway, no harm.
Dave
Quoted 11 lines
Quoted 7 lines
Dave
Quoted 3 lines
Signature · 1 line
2.17.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UKSignature · 2 lines
Wei Yang Help you, Help me
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 3/3 migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
In reply to Wei YangCurrently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()
ret = loadvm_process_command()
loadvm_postcopy_handle_run()
return -1;
if (ret < 0) {
if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
...
}From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.
Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.
--- migration/migration.c | 2 +- migration/postcopy-ram.c | 13 +++++++++---- migration/postcopy-ram.h | 3 ++- migration/savevm.c | 11 ++++++----- 4 files changed, 18 insertions(+), 11 deletions(-)
migration/migration.c +1 -1
@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)assert (mis->from_src_file); mis->migration_incoming_co =qemu_coroutine_self (); mis->largest_page_size =qemu_ram_pagesize_largest (); -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );migrate_set_state (&mis->state,MIGRATION_STATUS_NONE ,MIGRATION_STATUS_ACTIVE ); ret =qemu_loadvm_state (mis->from_src_file);
migration/postcopy-ram.c +9 -4
@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) } } -postcopy_state_set (POSTCOPY_INCOMING_END ); +postcopy_state_set (POSTCOPY_INCOMING_END ,NULL );if (mis->postcopy_tmp_page) {munmap (mis->postcopy_tmp_page, mis->largest_page_size);
@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)return -1 ; } -postcopy_state_set (POSTCOPY_INCOMING_DISCARD ); +postcopy_state_set (POSTCOPY_INCOMING_DISCARD ,NULL );return 0 ; }
@@ -1457,9 +1457,14 @@ PostcopyState postcopy_state_get(void) }/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state) +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state) { -return atomic_xchg (&incoming_postcopy_state, new_state); +if (!old_state) { +return atomic_xchg (&incoming_postcopy_state, new_state); + }else { +return atomic_cmpxchg (&incoming_postcopy_state, *old_state, new_state); + } }/* Register a handler for external shared memory postcopy
migration/postcopy-ram.h +2 -1
@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);PostcopyState postcopy_state_get (void );/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state); +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state);void postcopy_fault_thread_notify (MigrationIncomingState *mis);
migration/savevm.c +6 -5
@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {static int loadvm_postcopy_handle_advise (MigrationIncomingState *mis,uint16_t len) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ,NULL );uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;Error *local_err =NULL ;
@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, }if (!postcopy_ram_supported_by_host (mis)) { -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );return -1 ; }
@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)/* After this message we must be able to immediately receive postcopy data */ static int loadvm_postcopy_handle_listen (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ,NULL );trace_loadvm_postcopy_handle_listen ();Error *local_err =NULL ;
@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)/* After all discards we can start running and asking for pages */ static int loadvm_postcopy_handle_run (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING ); +PostcopyState old_ps =POSTCOPY_INCOMING_LISTENING ; +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING , &old_ps);trace_loadvm_postcopy_handle_run (); -if (ps !=POSTCOPY_INCOMING_LISTENING ) { +if (ps != old_ps) {error_report ("CMD_POSTCOPY_RUN in wrong postcopy state (%d)" , ps);return -1 ; }
Signature · 1 line
2.17.1
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 24 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.
Quoted 5 lines
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Signed-off-by Wei Yang <[email protected]>
OK, it's a shame to use a pointer there, but it works.
Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.
Dave
Quoted 121 lines
--- migration/migration.c | 2 +- migration/postcopy-ram.c | 13 +++++++++---- migration/postcopy-ram.h | 3 ++- migration/savevm.c | 11 ++++++----- 4 files changed, 18 insertions(+), 11 deletions(-)migration/migration.c +1
-1@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)assert (mis->from_src_file); mis->migration_incoming_co =qemu_coroutine_self (); mis->largest_page_size =qemu_ram_pagesize_largest (); -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );migrate_set_state (&mis->state,MIGRATION_STATUS_NONE ,MIGRATION_STATUS_ACTIVE ); ret =qemu_loadvm_state (mis->from_src_file);migration/postcopy-ram.c +9
-4@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) } } -postcopy_state_set (POSTCOPY_INCOMING_END ); +postcopy_state_set (POSTCOPY_INCOMING_END ,NULL );if (mis->postcopy_tmp_page) {munmap (mis->postcopy_tmp_page, mis->largest_page_size);@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)return -1 ; } -postcopy_state_set (POSTCOPY_INCOMING_DISCARD ); +postcopy_state_set (POSTCOPY_INCOMING_DISCARD ,NULL );return 0 ; }@@ -1457,9 +1457,14 @@ PostcopyState postcopy_state_get(void) }/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state) +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state) { -return atomic_xchg (&incoming_postcopy_state, new_state); +if (!old_state) { +return atomic_xchg (&incoming_postcopy_state, new_state); + }else { +return atomic_cmpxchg (&incoming_postcopy_state, *old_state, new_state); + } }/* Register a handler for external shared memory postcopy migration/postcopy-ram.h +2
-1@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);PostcopyState postcopy_state_get (void );/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state); +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state);void postcopy_fault_thread_notify (MigrationIncomingState *mis);migration/savevm.c +6
-5@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {static int loadvm_postcopy_handle_advise (MigrationIncomingState *mis,uint16_t len) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ,NULL );uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;Error *local_err =NULL ;@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, }if (!postcopy_ram_supported_by_host (mis)) { -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );return -1 ; }@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)/* After this message we must be able to immediately receive postcopy data */ static int loadvm_postcopy_handle_listen (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ,NULL );trace_loadvm_postcopy_handle_listen ();Error *local_err =NULL ;@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)/* After all discards we can start running and asking for pages */ static int loadvm_postcopy_handle_run (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING ); +PostcopyState old_ps =POSTCOPY_INCOMING_LISTENING ; +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING , &old_ps);trace_loadvm_postcopy_handle_run (); -if (ps !=POSTCOPY_INCOMING_LISTENING ) { +if (ps != old_ps) {error_report ("CMD_POSTCOPY_RUN in wrong postcopy state (%d)" , ps);return -1 ; }Signature · 1 line
2.17.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.
Quoted 7 lines
Quoted 5 lines
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
You mean second parameter of postcopy_state_set()?
I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?
Quoted 133 lines
Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.
Reviewed-by Dr. David Alan Gilbert <[email protected]>Dave
Quoted 121 lines
--- migration/migration.c | 2 +- migration/postcopy-ram.c | 13 +++++++++---- migration/postcopy-ram.h | 3 ++- migration/savevm.c | 11 ++++++----- 4 files changed, 18 insertions(+), 11 deletions(-)migration/migration.c +1
-1@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)assert (mis->from_src_file); mis->migration_incoming_co =qemu_coroutine_self (); mis->largest_page_size =qemu_ram_pagesize_largest (); -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );migrate_set_state (&mis->state,MIGRATION_STATUS_NONE ,MIGRATION_STATUS_ACTIVE ); ret =qemu_loadvm_state (mis->from_src_file);migration/postcopy-ram.c +9
-4@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) } } -postcopy_state_set (POSTCOPY_INCOMING_END ); +postcopy_state_set (POSTCOPY_INCOMING_END ,NULL );if (mis->postcopy_tmp_page) {munmap (mis->postcopy_tmp_page, mis->largest_page_size);@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)return -1 ; } -postcopy_state_set (POSTCOPY_INCOMING_DISCARD ); +postcopy_state_set (POSTCOPY_INCOMING_DISCARD ,NULL );return 0 ; }@@ -1457,9 +1457,14 @@ PostcopyState postcopy_state_get(void) }/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state) +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state) { -return atomic_xchg (&incoming_postcopy_state, new_state); +if (!old_state) { +return atomic_xchg (&incoming_postcopy_state, new_state); + }else { +return atomic_cmpxchg (&incoming_postcopy_state, *old_state, new_state); + } }/* Register a handler for external shared memory postcopy migration/postcopy-ram.h +2
-1@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);PostcopyState postcopy_state_get (void );/* Set the state and return the old state */ -PostcopyState postcopy_state_set (PostcopyState new_state); +PostcopyState postcopy_state_set (PostcopyState new_state, +const PostcopyState *old_state);void postcopy_fault_thread_notify (MigrationIncomingState *mis);migration/savevm.c +6
-5@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {static int loadvm_postcopy_handle_advise (MigrationIncomingState *mis,uint16_t len) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_ADVISE ,NULL );uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;Error *local_err =NULL ;@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, }if (!postcopy_ram_supported_by_host (mis)) { -postcopy_state_set (POSTCOPY_INCOMING_NONE ); +postcopy_state_set (POSTCOPY_INCOMING_NONE ,NULL );return -1 ; }@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)/* After this message we must be able to immediately receive postcopy data */ static int loadvm_postcopy_handle_listen (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ); +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_LISTENING ,NULL );trace_loadvm_postcopy_handle_listen ();Error *local_err =NULL ;@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)/* After all discards we can start running and asking for pages */ static int loadvm_postcopy_handle_run (MigrationIncomingState *mis) { -PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING ); +PostcopyState old_ps =POSTCOPY_INCOMING_LISTENING ; +PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING , &old_ps);trace_loadvm_postcopy_handle_run (); -if (ps !=POSTCOPY_INCOMING_LISTENING ) { +if (ps != old_ps) {error_report ("CMD_POSTCOPY_RUN in wrong postcopy state (%d)" , ps);return -1 ; }Signature · 1 line
2.17.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 30 lines · Wei Yang
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
You mean second parameter of
postcopy_state_set()?I don't have a better idea. Or we introduce a new state
POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?
Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?
Quoted 5 lines
Quoted 5 lines
Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.
It is true that we might hang on some real errors, but IMHO it might be no where better to quit QEMU if we're in postcopy...
(What I'm thinking in mind here is that sometimes even if postcopy
failed we might still have a chance to recover a full VM by dumping both src/dst of the during-postcopy VM instances and manually merge them by black magic, in very extreme cases)
Regards,
Signature · 1 line
Peter Xu
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 35 lines · Peter Xu
Quoted 34 lines · Wei Yang
Quoted 32 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.Quoted 5 lines
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
Quoted 8 lines
Quoted 4 lines
You mean second parameter of
postcopy_state_set()?I don't have a better idea. Or we introduce a new state
POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?Maybe simply fix
loadvm_postcopy_handle_run()to set the state after thePOSTCOPY_INCOMING_LISTENINGcheck?
Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?
Sounds like another option.
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 41 lines · Wei Yang
Quoted 40 lines · Peter Xu
Quoted 38 lines · Wei Yang
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.Quoted 7 lines
Quoted 5 lines
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
You mean second parameter of
postcopy_state_set()?I don't have a better idea. Or we introduce a new state
POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?
Quoted 7 lines
Quoted 3 lines
Maybe simply fix
loadvm_postcopy_handle_run()to set the state after thePOSTCOPY_INCOMING_LISTENINGcheck?Set state back to ps if ps is not
POSTCOPY_INCOMING_LISTENING?Sounds like another option.
Even simpler?
ps = postcopy_state_get();
if (ps != INCOMING)
return -1;
postcopy_state_set(RUNNING);Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 48 lines · Peter Xu
Quoted 47 lines · Wei Yang
Quoted 35 lines · Peter Xu
Quoted 34 lines · Wei Yang
Quoted 32 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.Quoted 5 lines
This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
Quoted 8 lines
Quoted 4 lines
You mean second parameter of
postcopy_state_set()?I don't have a better idea. Or we introduce a new state
POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?Maybe simply fix
loadvm_postcopy_handle_run()to set the state after thePOSTCOPY_INCOMING_LISTENINGcheck?Set state back to ps if ps is not
POSTCOPY_INCOMING_LISTENING?
Quoted 9 lines
Quoted 1 line
Sounds like another option.
Even simpler?
ps =
postcopy_state_get();
if (ps != INCOMING)
return -1;
postcopy_state_set(RUNNING);
Looks good to me.
Dave,
Do you feel good with it?
Quoted 4 lines
Thanks,
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 47 lines · * Wei Yang ([email protected]) wrote:
Quoted 46 lines · Peter Xu
Quoted 45 lines · Wei Yang
Quoted 43 lines · Peter Xu
Quoted 30 lines · Wei Yang
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Wei Yang ([email protected]) wrote:
Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.
First let's look at the code flow:
qemu_loadvm_state_main()ret = loadvm_process_command() loadvm_postcopy_handle_run() return -1; if (ret < 0) { if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING) ... }From above snippet, the corner case is
loadvm_postcopy_handle_run()always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.Then ret is checked in
qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an
old_stateparameter inpostcopy_state_set(). New state only would be set when current state equals toold_state.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Wei Yang <[email protected]>OK, it's a shame to use a pointer there, but it works.
You mean second parameter of
postcopy_state_set()?I don't have a better idea. Or we introduce a new state
POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?Maybe simply fix
loadvm_postcopy_handle_run()to set the state after thePOSTCOPY_INCOMING_LISTENINGcheck?
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Set state back to ps if ps is not
POSTCOPY_INCOMING_LISTENING?Sounds like another option.
Even simpler?
ps =
postcopy_state_get();
if (ps != INCOMING)
^^ LISTENING
Quoted 9 lines
Quoted 3 lines
return -1;
postcopy_state_set(RUNNING);Looks good to me.
Dave,
Do you feel good with it?
Yes, I think so; it's simpler.
Dave
Quoted 8 lines
Quoted 4 lines
Thanks,
Signature · 1 line
Peter XuSignature · 2 lines
Wei Yang Help you, Help me
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Maybe simply fix
loadvm_postcopy_handle_run()to set the state after thePOSTCOPY_INCOMING_LISTENINGcheck?Set state back to ps if ps is not
POSTCOPY_INCOMING_LISTENING?
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Sounds like another option.
Even simpler?
ps =
postcopy_state_get();
if (ps != INCOMING)^^ LISTENING
oops
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
return -1;
postcopy_state_set(RUNNING);
Quoted 8 lines
Quoted 5 lines
Looks good to me.
Dave,
Do you feel good with it?
Yes, I think so; it's simpler.
I will prepare v2.
Quoted 12 lines
Dave
Quoted 8 lines
Quoted 4 lines
Thanks,
Signature · 1 line
Peter XuSignature · 2 lines
Wei Yang Help you, Help me--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 2 lines
Wei Yang Help you, Help me