Skip to contentSkip to conversation

Auto exit source QEMU process after a successful migration

PATCH In review Kunkun Jiang 3 participants 8 messages Jul 5, 2021 – Jul 14, 2021

lore
Share & tools
lore
#0 lore raw

cover Auto exit source QEMU process after a successful migration

Hi all,

This serial include patches as below:
Patch 1:
- add a new shutdown cause 'migration-completed', which used for automatically exit of source QEMU process after a successful migration

Patch 2:
- add a new migration capability 'auto-quit' to control whether to automatically exit source QEMU process after a successful migration

Kunkun Jiang (2):
qapi/run-state: Add a new shutdown cause 'migration-completed' qapi/migration: Add a new migration capability 'auto-quit'

 migration/migration.c | 13 +++++++++++++
 migration/migration.h |  1 +
 qapi/migration.json   |  6 +++++-
 qapi/run-state.json   |  4 +++-
 4 files changed, 22 insertions(+), 2 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/2 qapi/run-state: Add a new shutdown cause 'migration-completed'

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

---
 migration/migration.c | 1 +
 qapi/run-state.json   | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
     case MIGRATION_STATUS_COMPLETED:
         migration_calculate_complete(s);
         runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
         break;
 
     case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
 #                   ignores --no-reboot. This is useful for sanitizing
 #                   hypercalls on s390 that are used during kexec/kdump/boot
 #
+# @migration-completed: Reaction to the successful migration
+#
 ##
 { 'enum': 'ShutdownCause',
   # Beware, shutdown_caused_by_guest() depends on enumeration order
   'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
             'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
 
 ##
 # @StatusInfo:
Signature · 1 line
2.23.0
#2 lore raw

Re: [PATCH 1/2] qapi/run-state: Add a new shutdown cause 'migration-completed'

Quoted 5 lines · Kunkun Jiang

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

IIUC, 'STATUS_COMPLETED' state is entered on the source host once it has finished sending all VM state, and thus does not guarantee that the target host has successfully received and loaded all VM state.

Typically a mgmt app will need to directly confirm that the target host QEMU has succesfully started running, before it will tell the source QEMU to quit.

So, AFAICT, this automatic exit after STATUS_COMPLETED is not safe and could lead to total loss of the running VM in error scenarios.

Quoted 43 lines
---
 migration/migration.c | 1 +
 qapi/run-state.json   | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
     case MIGRATION_STATUS_COMPLETED:
         migration_calculate_complete(s);
         runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
         break;
 
     case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
 #                   ignores --no-reboot. This is useful for sanitizing
 #                   hypercalls on s390 that are used during kexec/kdump/boot
 #
+# @migration-completed: Reaction to the successful migration
+#
 ##
 { 'enum': 'ShutdownCause',
   # Beware, shutdown_caused_by_guest() depends on enumeration order
   'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
             'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
 
 ##
 # @StatusInfo:
Signature · 1 line
2.23.0

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw

Re: [PATCH 1/2] qapi/run-state: Add a new shutdown cause 'migration-completed'

Hi Daniel,

Quoted 10 lines · Daniel P. Berrangé
Quoted 5 lines · Kunkun Jiang

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

IIUC, 'STATUS_COMPLETED' state is entered on the source host once it has finished sending all VM state, and thus does not guarantee that the target host has successfully received and loaded all VM state.

Thanks for your reply.

If the target host doesn't successfully receive and load all VM state, we can send { "execute": "cont" } to resume the soruce in time to ensure that VM will not lost?

Quoted 3 lines

Typically a mgmt app will need to directly confirm that the target host QEMU has succesfully started running, before it will tell the source QEMU to quit.

'a mgmt app', such as libvirt?

Thanks,
Kunkun Jiang

Quoted 50 lines

So, AFAICT, this automatic exit after STATUS_COMPLETED is not safe and could lead to total loss of the running VM in error scenarios.

Quoted 42 lines
---
  migration/migration.c | 1 +
  qapi/run-state.json   | 4 +++-
  2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
      case MIGRATION_STATUS_COMPLETED:
          migration_calculate_complete(s);
          runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
          break;
 
      case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
  #                   ignores --no-reboot. This is useful for sanitizing
  #                   hypercalls on s390 that are used during kexec/kdump/boot
  #
+# @migration-completed: Reaction to the successful migration
+#
  ##
  { 'enum': 'ShutdownCause',
    # Beware, shutdown_caused_by_guest() depends on enumeration order
    'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
              'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
 
  ##
  # @StatusInfo:
Signature · 1 line
2.23.0

Regards,
Daniel

#4 lore raw

Re: [PATCH 1/2] qapi/run-state: Add a new shutdown cause 'migration-completed'

Quoted 13 lines · * Kunkun Jiang ([email protected]) wrote:

Hi Daniel,

Quoted 10 lines · Daniel P. Berrangé
Quoted 5 lines · Kunkun Jiang

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

IIUC, 'STATUS_COMPLETED' state is entered on the source host once it has finished sending all VM state, and thus does not guarantee that the target host has successfully received and loaded all VM state.

Quoted 5 lines

Thanks for your reply.

If the target host doesn't successfully receive and load all VM state, we can send { "execute": "cont" } to resume the soruce in time to ensure that VM will not lost?

Yes, that's pretty common at the moment; the failed migration can happen at lots of different points:
a) The last part of the actual migration stream/loading the devices
- that's pretty easy, since the destination hasn't actually got the full migration stream.

b) If the migration itself completes, but then the management system then tries to reconfigure the networking/storage on the destination, and something goes wrong in that, then it can roll that back and cont on the source.

So, it's a pretty common type of failure/recovery - the management application has to be a bit careful not to do anything destructive until as late as possible, so it knows it can switch back.

Quoted 4 lines
Quoted 3 lines

Typically a mgmt app will need to directly confirm that the target host QEMU has succesfully started running, before it will tell the source QEMU to quit.

'a mgmt app', such as libvirt?

Yes, it's currently libvirt that does that; but any of the control things could (it's just libvirt has been going long enough so it knows about lots and lots of nasty cases of migration failure, and recovering properly).

Can you explain why did you want to get the source to automatically quit? In a real setup where does it help?

Dave

Quoted 52 lines

Thanks,
Kunkun Jiang

Quoted 48 lines

So, AFAICT, this automatic exit after STATUS_COMPLETED is not safe and could lead to total loss of the running VM in error scenarios.

Quoted 40 lines
---
  migration/migration.c | 1 +
  qapi/run-state.json   | 4 +++-
  2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
      case MIGRATION_STATUS_COMPLETED:
          migration_calculate_complete(s);
          runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
          break;
      case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
  #                   ignores --no-reboot. This is useful for sanitizing
  #                   hypercalls on s390 that are used during kexec/kdump/boot
  #
+# @migration-completed: Reaction to the successful migration
+#
  ##
  { 'enum': 'ShutdownCause',
    # Beware, shutdown_caused_by_guest() depends on enumeration order
    'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
              'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
  ##
  # @StatusInfo:
Signature · 1 line
2.23.0

Regards,
Daniel

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#5 lore raw

Re: [PATCH 1/2] qapi/run-state: Add a new shutdown cause 'migration-completed'

Quoted 25 lines · Dr. David Alan Gilbert
Quoted 13 lines · * Kunkun Jiang ([email protected]) wrote:

Hi Daniel,

Quoted 10 lines · Daniel P. Berrangé
Quoted 5 lines · Kunkun Jiang

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

IIUC, 'STATUS_COMPLETED' state is entered on the source host once it has finished sending all VM state, and thus does not guarantee that the target host has successfully received and loaded all VM state.

Quoted 5 lines

Thanks for your reply.

If the target host doesn't successfully receive and load all VM state, we can send { "execute": "cont" } to resume the soruce in time to ensure that VM will not lost?

Yes, that's pretty common at the moment; the failed migration can happen at lots of different points:
a) The last part of the actual migration stream/loading the devices
- that's pretty easy, since the destination hasn't actually got the full migration stream.

Quoted 8 lines

b) If the migration itself completes, but then the management system then tries to reconfigure the networking/storage on the destination, and something goes wrong in that, then it can roll that back and cont on the source.

So, it's a pretty common type of failure/recovery - the management application has to be a bit careful not to do anything destructive until as late as possible, so it knows it can switch back.

Okay, I see.

Quoted 11 lines
Quoted 4 lines
Quoted 3 lines

Typically a mgmt app will need to directly confirm that the target host QEMU has succesfully started running, before it will tell the source QEMU to quit.

'a mgmt app', such as libvirt?

Yes, it's currently libvirt that does that; but any of the control things could (it's just libvirt has been going long enough so it knows about lots and lots of nasty cases of migration failure, and recovering properly).

Can you explain why did you want to get the source to automatically quit? In a real setup where does it help?

Sorry, my thoughts on live migration scenarios are not comprehensive enough.

Thanks,
Kunkun Jiang

Quoted 54 lines

Dave

Quoted 51 lines

Thanks,
Kunkun Jiang

Quoted 48 lines

So, AFAICT, this automatic exit after STATUS_COMPLETED is not safe and could lead to total loss of the running VM in error scenarios.

Quoted 40 lines
---
   migration/migration.c | 1 +
   qapi/run-state.json   | 4 +++-
   2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
       case MIGRATION_STATUS_COMPLETED:
           migration_calculate_complete(s);
           runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
           break;
       case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
   #                   ignores --no-reboot. This is useful for sanitizing
   #                   hypercalls on s390 that are used during kexec/kdump/boot
   #
+# @migration-completed: Reaction to the successful migration
+#
   ##
   { 'enum': 'ShutdownCause',
     # Beware, shutdown_caused_by_guest() depends on enumeration order
     'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
               'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
   ##
   # @StatusInfo:
Signature · 1 line
2.23.0

Regards,
Daniel

#6 lore raw

Re: [PATCH 1/2] qapi/run-state: Add a new shutdown cause 'migration-completed'

Quoted 39 lines · * Kunkun Jiang ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 13 lines · * Kunkun Jiang ([email protected]) wrote:

Hi Daniel,

Quoted 10 lines · Daniel P. Berrangé
Quoted 5 lines · Kunkun Jiang

In the current version, the source QEMU process does not automatic exit after a successful migration. Additional action is required, such as sending { "execute": "quit" } or ctrl+c. For simplify, add a new shutdown cause 'migration-completed' to exit the source QEMU process after a successful migration.

IIUC, 'STATUS_COMPLETED' state is entered on the source host once it has finished sending all VM state, and thus does not guarantee that the target host has successfully received and loaded all VM state.

Quoted 5 lines

Thanks for your reply.

If the target host doesn't successfully receive and load all VM state, we can send { "execute": "cont" } to resume the soruce in time to ensure that VM will not lost?

Yes, that's pretty common at the moment; the failed migration can happen at lots of different points:
a) The last part of the actual migration stream/loading the devices
- that's pretty easy, since the destination hasn't actually got the full migration stream.

Quoted 8 lines

b) If the migration itself completes, but then the management system then tries to reconfigure the networking/storage on the destination, and something goes wrong in that, then it can roll that back and cont on the source.

So, it's a pretty common type of failure/recovery - the management application has to be a bit careful not to do anything destructive until as late as possible, so it knows it can switch back.

Okay, I see.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Typically a mgmt app will need to directly confirm that the target host QEMU has succesfully started running, before it will tell the source QEMU to quit.

'a mgmt app', such as libvirt?

Quoted 8 lines
Quoted 7 lines

Yes, it's currently libvirt that does that; but any of the control things could (it's just libvirt has been going long enough so it knows about lots and lots of nasty cases of migration failure, and recovering properly).

Can you explain why did you want to get the source to automatically quit? In a real setup where does it help?

Sorry, my thoughts on live migration scenarios are not comprehensive enough.

That's OK; it takes a little while to understand all of the recovery and error cases; people *really* want to recover from a failed migration; so we try and be very careful about not throwing away the source.

Dave

Quoted 57 lines

Thanks,
Kunkun Jiang

Quoted 54 lines

Dave

Quoted 51 lines

Thanks,
Kunkun Jiang

Quoted 48 lines

So, AFAICT, this automatic exit after STATUS_COMPLETED is not safe and could lead to total loss of the running VM in error scenarios.

Quoted 40 lines
---
   migration/migration.c | 1 +
   qapi/run-state.json   | 4 +++-
   2 files changed, 4 insertions(+), 1 deletion(-)
migration/migration.c +1 -0
index 4228635d18..16782c93c2 100644
@@ -3539,6 +3539,7 @@ static void migration_iteration_finish(MigrationState *s)
       case MIGRATION_STATUS_COMPLETED:
           migration_calculate_complete(s);
           runstate_set(RUN_STATE_POSTMIGRATE);
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
           break;
       case MIGRATION_STATUS_ACTIVE:
qapi/run-state.json +3 -1
index 43d66d700f..66aaef4e2b 100644
@@ -86,12 +86,14 @@
   #                   ignores --no-reboot. This is useful for sanitizing
   #                   hypercalls on s390 that are used during kexec/kdump/boot
   #
+# @migration-completed: Reaction to the successful migration
+#
   ##
   { 'enum': 'ShutdownCause',
     # Beware, shutdown_caused_by_guest() depends on enumeration order
     'data': [ 'none', 'host-error', 'host-qmp-quit', 'host-qmp-system-reset',
               'host-signal', 'host-ui', 'guest-shutdown', 'guest-reset',
-            'guest-panic', 'subsystem-reset'] }
+            'guest-panic', 'subsystem-reset', 'migration-completed'] }
   ##
   # @StatusInfo:
Signature · 1 line
2.23.0

Regards,
Daniel

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#7 lore raw

PATCH 2/2 qapi/migration: Add a new migration capability 'auto-quit'

In reply to Kunkun Jiang

For compatibility, a new migration capability 'auto-quit' is added to control the exit of source QEMU after a successful migration.

---
 migration/migration.c | 14 +++++++++++++-
 migration/migration.h |  1 +
 qapi/migration.json   |  6 +++++-
 3 files changed, 19 insertions(+), 2 deletions(-)
migration/migration.c +13 -1
index 16782c93c2..82ad6d35b2 100644
@@ -2567,6 +2567,15 @@ bool migrate_background_snapshot(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT];
 }
 
+bool migrate_auto_quit(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_QUIT];
+}
+
 /* migration thread support */
 /*
  * Something bad happened to the RP stream, mark an error
@@ -3539,7 +3548,10 @@ static void migration_iteration_finish(MigrationState *s)
     case MIGRATION_STATUS_COMPLETED:
         migration_calculate_complete(s);
         runstate_set(RUN_STATE_POSTMIGRATE);
-        qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
+
+        if (migrate_auto_quit()) {
+            qemu_system_shutdown_request(SHUTDOWN_CAUSE_MIGRATION_COMPLETED);
+        }
         break;
 
     case MIGRATION_STATUS_ACTIVE:
migration/migration.h +1 -0
index 2ebb740dfa..a72b178d35 100644
@@ -349,6 +349,7 @@ int migrate_decompress_threads(void);
 bool migrate_use_events(void);
 bool migrate_postcopy_blocktime(void);
 bool migrate_background_snapshot(void);
+bool migrate_auto_quit(void);
 
 /* Sending on the return path - generic and then for each message type */
 void migrate_send_rp_shut(MigrationIncomingState *mis,
qapi/migration.json +5 -1
index 1124a2dda8..26c1a52c56 100644
@@ -452,6 +452,9 @@
 #                       procedure starts. The VM RAM is saved with running VM.
 #                       (since 6.0)
 #
+# @auto-quit: If enabled, QEMU process will exit after a successful migration.
+#             (since 6.1)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
@@ -459,7 +462,8 @@
            'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
            'block', 'return-path', 'pause-before-switchover', 'multifd',
            'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
-           'x-ignore-shared', 'validate-uuid', 'background-snapshot'] }
+           'x-ignore-shared', 'validate-uuid', 'background-snapshot',
+           'auto-quit'] }
 
 ##
 # @MigrationCapabilityStatus:
Signature · 1 line
2.23.0