Skip to contentSkip to conversation

migration/postcopy: cleanup related to postcopy

PATCH Superseded Wei Yang 3 participants 16 messages Oct 1, 2019 – Oct 10, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

cover migration/postcopy: cleanup related to postcopy

The first one just tries to make function name more easy for reading and understanding.

The next two patches are related to PostcopyState.

Wei Yang (3):
migration/postcopy: rename postcopy_ram_enable_notify to
postcopy_ram_incoming_setup
migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup
when state is ADVISE
migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case
properly

 migration/migration.c    |  3 +--
 migration/postcopy-ram.c | 17 +++++++++++------
 migration/postcopy-ram.h |  5 +++--
 migration/savevm.c       | 13 +++++++------
 4 files changed, 22 insertions(+), 16 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/3 migration/postcopy: rename postcopy_ram_enable_notify to postcopy_ram_incoming_setup

Function postcopy_ram_incoming_setup and postcopy_ram_incoming_cleanup is a pair. Rename to make it clear for audience.

---
 migration/postcopy-ram.c | 4 ++--
 migration/postcopy-ram.h | 2 +-
 migration/savevm.c       | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
migration/postcopy-ram.c +2 -2
index 1f63e65ed7..b24c4a10c2 100644
@@ -1094,7 +1094,7 @@ retry:
     return NULL;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
     /* Open the fd for the kernel to give us userfaults */
     mis->userfault_fd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
@@ -1321,7 +1321,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,
     return -1;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
     assert(0);
     return -1;
migration/postcopy-ram.h +1 -1
index 9c8bd2bae0..d2668cc820 100644
@@ -20,7 +20,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis);
  * Make all of RAM sensitive to accesses to areas that haven't yet been written
  * and wire up anything necessary to deal with it.
  */
-int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis);
 
 /*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
migration/savevm.c +1 -1
index adad938f57..f3292eb003 100644
@@ -1865,7 +1865,7 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
      * shouldn't be doing anything yet so don't actually expect requests
      */
     if (migrate_postcopy_ram()) {
-        if (postcopy_ram_enable_notify(mis)) {
+        if (postcopy_ram_incoming_setup(mis)) {
             postcopy_ram_incoming_cleanup(mis);
             return -1;
         }
Signature · 1 line
2.17.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] migration/postcopy: rename postcopy_ram_enable_notify to postcopy_ram_incoming_setup

Quoted 4 lines · * Wei Yang ([email protected]) wrote:

Function postcopy_ram_incoming_setup and postcopy_ram_incoming_cleanup is a pair. Rename to make it clear for audience.

Yes OK.
It really initially just setup the userfault, but it also kicks off the fault thread as well so it has got a bit more hairy.

Quoted 57 lines
---
 migration/postcopy-ram.c | 4 ++--
 migration/postcopy-ram.h | 2 +-
 migration/savevm.c       | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
migration/postcopy-ram.c +2 -2
index 1f63e65ed7..b24c4a10c2 100644
@@ -1094,7 +1094,7 @@ retry:
     return NULL;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
     /* Open the fd for the kernel to give us userfaults */
     mis->userfault_fd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
@@ -1321,7 +1321,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,
     return -1;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
     assert(0);
     return -1;
migration/postcopy-ram.h +1 -1
index 9c8bd2bae0..d2668cc820 100644
@@ -20,7 +20,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis);
  * Make all of RAM sensitive to accesses to areas that haven't yet been written
  * and wire up anything necessary to deal with it.
  */
-int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis);
 
 /*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
migration/savevm.c +1 -1
index adad938f57..f3292eb003 100644
@@ -1865,7 +1865,7 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
      * shouldn't be doing anything yet so don't actually expect requests
      */
     if (migrate_postcopy_ram()) {
-        if (postcopy_ram_enable_notify(mis)) {
+        if (postcopy_ram_incoming_setup(mis)) {
             postcopy_ram_incoming_cleanup(mis);
             return -1;
         }
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#3 lore raw

PATCH 2/3 migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE

In reply to Wei Yang

postcopy_ram_incoming_cleanup() does cleanup for postcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.

This means there is nothing to cleanup when migration is still ADVISE state.

---
 migration/migration.c | 1 -
 1 file changed, 1 deletion(-)
migration/migration.c +0 -1
index 5f7e4d15e9..34d5e66f06 100644
@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)
              * but managed to complete within the precopy period, we can use
              * the normal exit.
              */
-            postcopy_ram_incoming_cleanup(mis);
         } else if (ret >= 0) {
             /*
              * Postcopy was started, cleanup should happen at the end of the
Signature · 1 line
2.17.1
#4 lore raw

Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE

Quoted 17 lines · * Wei Yang ([email protected]) wrote:

postcopy_ram_incoming_cleanup() does cleanup for postcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.

This means there is nothing to cleanup when migration is still ADVISE state.

---
 migration/migration.c | 1 -
 1 file changed, 1 deletion(-)
migration/migration.c +0 -0
index 5f7e4d15e9..34d5e66f06 100644

migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466

Quoted 7 lines
migration/migration.c +0 -1
@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)
              * but managed to complete within the precopy period, we can use
              * the normal exit.
              */
-            postcopy_ram_incoming_cleanup(mis);
         } else if (ret >= 0) {
             /*
              * Postcopy was started, cleanup should happen at the end of the

I think that misses the cleanup of mlock that corresponds to the munlockall in postcopy_ram_supported_by_host - that's called very early on; I think in the advise stage.

Dave

Quoted 3 lines
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#5 lore raw

Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE

Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:

postcopy_ram_incoming_cleanup() does cleanup for postcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.

This means there is nothing to cleanup when migration is still ADVISE state.

---
 migration/migration.c | 1 -
 1 file changed, 1 deletion(-)
Quoted 17 lines

migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466

Quoted 12 lines
migration/migration.c +0 -1
index 5f7e4d15e9..34d5e66f06 100644
@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)
              * but managed to complete within the precopy period, we can use
              * the normal exit.
              */
-            postcopy_ram_incoming_cleanup(mis);
         } else if (ret >= 0) {
             /*
              * Postcopy was started, cleanup should happen at the end of the

I think that misses the cleanup of mlock that corresponds to the munlockall in postcopy_ram_supported_by_host - that's called very early on; I think in the advise stage.

Thanks you are right.

BTW, do we need to check enable_mlock when calling munlockall() in postcopy_ram_supported_by_host() ?

Quoted 7 lines

Dave

Quoted 3 lines
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Signature · 2 lines
Wei Yang
Help you, Help me
#6 lore raw

Re: [PATCH 2/3] migration/postcopy: not necessary to do postcopy_ram_incoming_cleanup when state is ADVISE

Quoted 32 lines · * Wei Yang ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:

postcopy_ram_incoming_cleanup() does cleanup for postcopy_ram_incoming_setup(), while the setup happens only after migration enters LISTEN state.

This means there is nothing to cleanup when migration is still ADVISE state.

---
 migration/migration.c | 1 -
 1 file changed, 1 deletion(-)
Quoted 17 lines

migration/migration.c static void process_incoming_migration_co(void *opaque) · line 466

Quoted 12 lines
migration/migration.c +0 -1
index 5f7e4d15e9..34d5e66f06 100644
@@ -461,7 +461,6 @@ static void process_incoming_migration_co(void *opaque)
              * but managed to complete within the precopy period, we can use
              * the normal exit.
              */
-            postcopy_ram_incoming_cleanup(mis);
         } else if (ret >= 0) {
             /*
              * Postcopy was started, cleanup should happen at the end of the

I think that misses the cleanup of mlock that corresponds to the munlockall in postcopy_ram_supported_by_host - that's called very early on; I think in the advise stage.

Quoted 4 lines

Thanks you are right.

BTW, do we need to check enable_mlock when calling munlockall() in postcopy_ram_supported_by_host() ?

I don't think so; it does an extra munlock in that case when nothing should be locked anyway, no harm.

Dave

Quoted 11 lines
Quoted 7 lines

Dave

Quoted 3 lines
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Signature · 2 lines
Wei Yang
Help you, Help me

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#7 lore raw

PATCH 3/3 migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

In reply to Wei Yang

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

---
 migration/migration.c    |  2 +-
 migration/postcopy-ram.c | 13 +++++++++----
 migration/postcopy-ram.h |  3 ++-
 migration/savevm.c       | 11 ++++++-----
 4 files changed, 18 insertions(+), 11 deletions(-)
migration/migration.c +1 -1
index 34d5e66f06..369cf3826e 100644
@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)
     assert(mis->from_src_file);
     mis->migration_incoming_co = qemu_coroutine_self();
     mis->largest_page_size = qemu_ram_pagesize_largest();
-    postcopy_state_set(POSTCOPY_INCOMING_NONE);
+    postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
     migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
                       MIGRATION_STATUS_ACTIVE);
     ret = qemu_loadvm_state(mis->from_src_file);
migration/postcopy-ram.c +9 -4
index b24c4a10c2..8f741d636d 100644
@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
         }
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_END);
+    postcopy_state_set(POSTCOPY_INCOMING_END, NULL);
 
     if (mis->postcopy_tmp_page) {
         munmap(mis->postcopy_tmp_page, mis->largest_page_size);
@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
         return -1;
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_DISCARD);
+    postcopy_state_set(POSTCOPY_INCOMING_DISCARD, NULL);
 
     return 0;
 }
@@ -1457,9 +1457,14 @@ PostcopyState  postcopy_state_get(void)
 }
 
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state)
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state)
 {
-    return atomic_xchg(&incoming_postcopy_state, new_state);
+    if (!old_state) {
+        return atomic_xchg(&incoming_postcopy_state, new_state);
+    } else {
+        return atomic_cmpxchg(&incoming_postcopy_state, *old_state, new_state);
+    }
 }
 
 /* Register a handler for external shared memory postcopy
migration/postcopy-ram.h +2 -1
index d2668cc820..e3dde32155 100644
@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);
 
 PostcopyState postcopy_state_get(void);
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state);
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state);
 
 void postcopy_fault_thread_notify(MigrationIncomingState *mis);
 
migration/savevm.c +6 -5
index f3292eb003..45474d9c95 100644
@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {
 static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
                                          uint16_t len)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE, NULL);
     uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;
     Error *local_err = NULL;
 
@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
     }
 
     if (!postcopy_ram_supported_by_host(mis)) {
-        postcopy_state_set(POSTCOPY_INCOMING_NONE);
+        postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
         return -1;
     }
 
@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)
 /* After this message we must be able to immediately receive postcopy data */
 static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING, NULL);
     trace_loadvm_postcopy_handle_listen();
     Error *local_err = NULL;
 
@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
 /* After all discards we can start running and asking for pages */
 static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+    PostcopyState old_ps = POSTCOPY_INCOMING_LISTENING;
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING, &old_ps);
 
     trace_loadvm_postcopy_handle_run();
-    if (ps != POSTCOPY_INCOMING_LISTENING) {
+    if (ps != old_ps) {
         error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
         return -1;
     }
Signature · 1 line
2.17.1
Reviewed-by on v1 3/3
#8 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 24 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

Quoted 5 lines

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

OK, it's a shame to use a pointer there, but it works.
Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.

Dave

Quoted 121 lines
---
 migration/migration.c    |  2 +-
 migration/postcopy-ram.c | 13 +++++++++----
 migration/postcopy-ram.h |  3 ++-
 migration/savevm.c       | 11 ++++++-----
 4 files changed, 18 insertions(+), 11 deletions(-)
migration/migration.c +1 -1
index 34d5e66f06..369cf3826e 100644
@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)
     assert(mis->from_src_file);
     mis->migration_incoming_co = qemu_coroutine_self();
     mis->largest_page_size = qemu_ram_pagesize_largest();
-    postcopy_state_set(POSTCOPY_INCOMING_NONE);
+    postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
     migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
                       MIGRATION_STATUS_ACTIVE);
     ret = qemu_loadvm_state(mis->from_src_file);
migration/postcopy-ram.c +9 -4
index b24c4a10c2..8f741d636d 100644
@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
         }
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_END);
+    postcopy_state_set(POSTCOPY_INCOMING_END, NULL);
 
     if (mis->postcopy_tmp_page) {
         munmap(mis->postcopy_tmp_page, mis->largest_page_size);
@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
         return -1;
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_DISCARD);
+    postcopy_state_set(POSTCOPY_INCOMING_DISCARD, NULL);
 
     return 0;
 }
@@ -1457,9 +1457,14 @@ PostcopyState  postcopy_state_get(void)
 }
 
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state)
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state)
 {
-    return atomic_xchg(&incoming_postcopy_state, new_state);
+    if (!old_state) {
+        return atomic_xchg(&incoming_postcopy_state, new_state);
+    } else {
+        return atomic_cmpxchg(&incoming_postcopy_state, *old_state, new_state);
+    }
 }
 
 /* Register a handler for external shared memory postcopy
migration/postcopy-ram.h +2 -1
index d2668cc820..e3dde32155 100644
@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);
 
 PostcopyState postcopy_state_get(void);
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state);
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state);
 
 void postcopy_fault_thread_notify(MigrationIncomingState *mis);
 
migration/savevm.c +6 -5
index f3292eb003..45474d9c95 100644
@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {
 static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
                                          uint16_t len)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE, NULL);
     uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;
     Error *local_err = NULL;
 
@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
     }
 
     if (!postcopy_ram_supported_by_host(mis)) {
-        postcopy_state_set(POSTCOPY_INCOMING_NONE);
+        postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
         return -1;
     }
 
@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)
 /* After this message we must be able to immediately receive postcopy data */
 static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING, NULL);
     trace_loadvm_postcopy_handle_listen();
     Error *local_err = NULL;
 
@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
 /* After all discards we can start running and asking for pages */
 static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+    PostcopyState old_ps = POSTCOPY_INCOMING_LISTENING;
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING, &old_ps);
 
     trace_loadvm_postcopy_handle_run();
-    if (ps != POSTCOPY_INCOMING_LISTENING) {
+    if (ps != old_ps) {
         error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
         return -1;
     }
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#9 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

Quoted 7 lines
Quoted 5 lines

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

OK, it's a shame to use a pointer there, but it works.

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Quoted 133 lines

Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.

Dave

Quoted 121 lines
---
 migration/migration.c    |  2 +-
 migration/postcopy-ram.c | 13 +++++++++----
 migration/postcopy-ram.h |  3 ++-
 migration/savevm.c       | 11 ++++++-----
 4 files changed, 18 insertions(+), 11 deletions(-)
migration/migration.c +1 -1
index 34d5e66f06..369cf3826e 100644
@@ -447,7 +447,7 @@ static void process_incoming_migration_co(void *opaque)
     assert(mis->from_src_file);
     mis->migration_incoming_co = qemu_coroutine_self();
     mis->largest_page_size = qemu_ram_pagesize_largest();
-    postcopy_state_set(POSTCOPY_INCOMING_NONE);
+    postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
     migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
                       MIGRATION_STATUS_ACTIVE);
     ret = qemu_loadvm_state(mis->from_src_file);
migration/postcopy-ram.c +9 -4
index b24c4a10c2..8f741d636d 100644
@@ -577,7 +577,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
         }
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_END);
+    postcopy_state_set(POSTCOPY_INCOMING_END, NULL);
 
     if (mis->postcopy_tmp_page) {
         munmap(mis->postcopy_tmp_page, mis->largest_page_size);
@@ -626,7 +626,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
         return -1;
     }
 
-    postcopy_state_set(POSTCOPY_INCOMING_DISCARD);
+    postcopy_state_set(POSTCOPY_INCOMING_DISCARD, NULL);
 
     return 0;
 }
@@ -1457,9 +1457,14 @@ PostcopyState  postcopy_state_get(void)
 }
 
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state)
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state)
 {
-    return atomic_xchg(&incoming_postcopy_state, new_state);
+    if (!old_state) {
+        return atomic_xchg(&incoming_postcopy_state, new_state);
+    } else {
+        return atomic_cmpxchg(&incoming_postcopy_state, *old_state, new_state);
+    }
 }
 
 /* Register a handler for external shared memory postcopy
migration/postcopy-ram.h +2 -1
index d2668cc820..e3dde32155 100644
@@ -109,7 +109,8 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mis);
 
 PostcopyState postcopy_state_get(void);
 /* Set the state and return the old state */
-PostcopyState postcopy_state_set(PostcopyState new_state);
+PostcopyState postcopy_state_set(PostcopyState new_state,
+                                 const PostcopyState *old_state);
 
 void postcopy_fault_thread_notify(MigrationIncomingState *mis);
 
migration/savevm.c +6 -5
index f3292eb003..45474d9c95 100644
@@ -1599,7 +1599,7 @@ enum LoadVMExitCodes {
 static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
                                          uint16_t len)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_ADVISE, NULL);
     uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps;
     Error *local_err = NULL;
 
@@ -1628,7 +1628,7 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
     }
 
     if (!postcopy_ram_supported_by_host(mis)) {
-        postcopy_state_set(POSTCOPY_INCOMING_NONE);
+        postcopy_state_set(POSTCOPY_INCOMING_NONE, NULL);
         return -1;
     }
 
@@ -1841,7 +1841,7 @@ static void *postcopy_ram_listen_thread(void *opaque)
 /* After this message we must be able to immediately receive postcopy data */
 static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING);
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_LISTENING, NULL);
     trace_loadvm_postcopy_handle_listen();
     Error *local_err = NULL;
 
@@ -1929,10 +1929,11 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
 /* After all discards we can start running and asking for pages */
 static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+    PostcopyState old_ps = POSTCOPY_INCOMING_LISTENING;
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING, &old_ps);
 
     trace_loadvm_postcopy_handle_run();
-    if (ps != POSTCOPY_INCOMING_LISTENING) {
+    if (ps != old_ps) {
         error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
         return -1;
     }
Signature · 1 line
2.17.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Signature · 2 lines
Wei Yang
Help you, Help me
#10 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 30 lines · Wei Yang
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

OK, it's a shame to use a pointer there, but it works.

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Quoted 5 lines
Quoted 5 lines

Note, something else; using '-1' as the return value and checking for it is something we do a lot; but in this case it's an example of an error we could never recover from so it never makes sense to try and recover. We should probably look at different types of error.

It is true that we might hang on some real errors, but IMHO it might be no where better to quit QEMU if we're in postcopy...

(What I'm thinking in mind here is that sometimes even if postcopy
failed we might still have a chance to recover a full VM by dumping both src/dst of the during-postcopy VM instances and manually merge them by black magic, in very extreme cases)

Regards,

Signature · 1 line
Peter Xu
author
#11 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 35 lines · Peter Xu
Quoted 34 lines · Wei Yang
Quoted 32 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

Quoted 5 lines

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

OK, it's a shame to use a pointer there, but it works.

Quoted 8 lines
Quoted 4 lines

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?

Sounds like another option.

Signature · 2 lines
Wei Yang
Help you, Help me
#12 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 41 lines · Wei Yang
Quoted 40 lines · Peter Xu
Quoted 38 lines · Wei Yang
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

Quoted 7 lines
Quoted 5 lines

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

OK, it's a shame to use a pointer there, but it works.

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Quoted 7 lines
Quoted 3 lines

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?

Sounds like another option.

Even simpler?

  ps = postcopy_state_get();
  if (ps != INCOMING)
    return -1;
  postcopy_state_set(RUNNING);

Thanks,

Signature · 1 line
Peter Xu
author
#13 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 48 lines · Peter Xu
Quoted 47 lines · Wei Yang
Quoted 35 lines · Peter Xu
Quoted 34 lines · Wei Yang
Quoted 32 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

Quoted 5 lines

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

OK, it's a shame to use a pointer there, but it works.

Quoted 8 lines
Quoted 4 lines

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?

Quoted 9 lines
Quoted 1 line

Sounds like another option.

Even simpler?

ps = postcopy_state_get();
if (ps != INCOMING)
return -1;
postcopy_state_set(RUNNING);

Looks good to me.

Dave,

Do you feel good with it?

Quoted 4 lines

Thanks,

Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me
#14 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 47 lines · * Wei Yang ([email protected]) wrote:
Quoted 46 lines · Peter Xu
Quoted 45 lines · Wei Yang
Quoted 43 lines · Peter Xu
Quoted 30 lines · Wei Yang
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Wei Yang ([email protected]) wrote:

Currently, we set PostcopyState blindly to RUNNING, even we found the previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()

    ret = loadvm_process_command()
        loadvm_postcopy_handle_run()
            return -1;
    if (ret < 0) {
        if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
            ...
    }

From above snippet, the corner case is loadvm_postcopy_handle_run() always sets state to RUNNING. And then it checks the previous state. If the previous state is not LISTENING, it will return -1. But at this moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1 PostcopyState is checked. Current logic would pause postcopy and retry if PostcopyState is RUNNING. This is not what we expect, because postcopy is not active yet.

This patch makes sure state is set to RUNNING only previous state is LISTENING by introducing an old_state parameter in postcopy_state_set(). New state only would be set when current state equals to old_state.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

OK, it's a shame to use a pointer there, but it works.

You mean second parameter of postcopy_state_set()?

I don't have a better idea. Or we introduce a new state POSTCOPY_INCOMING_NOCHECK. Do you feel better with this?

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?

Sounds like another option.

Even simpler?

ps = postcopy_state_get();
if (ps != INCOMING)

^^ LISTENING

Quoted 9 lines
Quoted 3 lines
   return -1;

postcopy_state_set(RUNNING);

Looks good to me.

Dave,

Do you feel good with it?

Yes, I think so; it's simpler.

Dave

Quoted 8 lines
Quoted 4 lines

Thanks,

Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#15 lore raw

Re: [PATCH 3/3] migration/postcopy: handle POSTCOPY_INCOMING_RUNNING corner case properly

Quoted 7 lines · Dr. David Alan Gilbert
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Maybe simply fix loadvm_postcopy_handle_run() to set the state after the POSTCOPY_INCOMING_LISTENING check?

Set state back to ps if ps is not POSTCOPY_INCOMING_LISTENING?

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Sounds like another option.

Even simpler?

ps = postcopy_state_get();
if (ps != INCOMING)

^^ LISTENING

oops

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
   return -1;

postcopy_state_set(RUNNING);

Quoted 8 lines
Quoted 5 lines

Looks good to me.

Dave,

Do you feel good with it?

Yes, I think so; it's simpler.

I will prepare v2.

Quoted 12 lines

Dave

Quoted 8 lines
Quoted 4 lines

Thanks,

Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Signature · 2 lines
Wei Yang
Help you, Help me