Skip to contentSkip to conversation

migration: Don't allow migration if vm is in POSTMIGRATE state

PATCH Superseded Tuguoyi 2 participants 3 messages Dec 7, 2020 – Dec 8, 2020

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Tuguoyi

author
#0 lore raw

PATCH v2 1/1 migration: Don't allow migration if vm is in POSTMIGRATE state

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
    at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
    at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

As Vladimir suggested, this patch just make migration job error-out with a message in migrate_fd_connect() if the vm is in RUN_STATE_POSTMIGRATE state.

---
 migration/migration.c | 7 +++++++
 1 file changed, 7 insertions(+)
migration/migration.c +7 -0
index 87a9b59..4091678 100644
@@ -3622,6 +3622,13 @@ void migrate_fd_connect(MigrationState *s, Error *error_in)
         return;
     }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_report("Can't migrate the vm that is in POSTMIGRATE state");
+        migrate_set_state(&s->state, s->state, MIGRATION_STATUS_FAILED);
+        migrate_fd_cleanup(s);
+        return;
+    }
+
     if (resume) {
         /* This is a resumed migration */
         rate_limit = s->parameters.max_postcopy_bandwidth /
Signature · 3 lines
2.7.4

[Patch v1]: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05950.html
#1 lore raw
Quoted 45 lines · Tuguoyi

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
     at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
     at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

As Vladimir suggested, this patch just make migration job error-out with a message in migrate_fd_connect() if the vm is in RUN_STATE_POSTMIGRATE state.

---
  migration/migration.c | 7 +++++++
  1 file changed, 7 insertions(+)
migration/migration.c +0 -0
index 87a9b59..4091678 100644
@@ -3622,6 +3622,13 @@ void migrate_fd_connect(MigrationState *s, Error *error_in)
          return;
      }
 

migration/migration.c void migrate_fd_connect(MigrationState *s, Error *error_in) · line 3634

Quoted 10 lines
migration/migration.c +7 -0
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_report("Can't migrate the vm that is in POSTMIGRATE state");
+        migrate_set_state(&s->state, s->state, MIGRATION_STATUS_FAILED);
+        migrate_fd_cleanup(s);
+        return;
+    }
+
      if (resume) {
          /* This is a resumed migration */
          rate_limit = s->parameters.max_postcopy_bandwidth /

I think, correct place for the check migrate_prepare, as it is called for any kind of migration, not only fd_. And in it we have already check for wrong state:

      if (runstate_check(RUN_STATE_INMIGRATE)) {
          error_setg(errp, "Guest is waiting for an incoming migration");
          return false;
      }

and no additional state change and cleanup is needed.

Signature · 2 lines
Best regards,
Vladimir
author
#2 lore raw
Quoted 79 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines · Tuguoyi

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh

snapshot-create-as'

Quoted 7 lines

- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from

/lib/x86_64-linux-gnu/libc.so.6

Quoted 1 line

#4 0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400)

at /build/qemu-5.0/block.c:5724

Quoted 1 line

#5 0x000055ca8dece967 in bdrv_inactivate_all () at

/build//qemu-5.0/block.c:5792

Quoted 1 line

#6 0x000055ca8de5539d in

qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)

Quoted 2 lines
     at /build/qemu-5.0/migration/savevm.c:1401

#7 qemu_savevm_state_complete_precopy (f=0x55ca907044b0,

iterable_only=iterable_only@entry=false,
inactivate_disks=inactivate_disks@entry=true)

Quoted 2 lines
     at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:2941

Quoted 1 line

#9 migration_iteration_run (s=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:3295

Quoted 1 line

#10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:3459

Quoted 1 line

#11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at

/build/qemu-5.0/util/qemu-thread-posix.c:519

Quoted 1 line

#12 0x00007fbf95c5f184 in start_thread () from

/lib/x86_64-linux-gnu/libpthread.so.0

Quoted 3 lines

#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set

BDRV_O_INACTIVE

Quoted 6 lines

flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

As Vladimir suggested, this patch just make migration job error-out with a message in migrate_fd_connect() if the vm is in

RUN_STATE_POSTMIGRATE state.

migration/migration.c void migrate_fd_connect(MigrationState *s,

Quoted 11 lines
---
  migration/migration.c | 7 +++++++
  1 file changed, 7 insertions(+)
migration/migration.c +0 -0
index 87a9b59..4091678 100644
@@ -3622,6 +3622,13 @@ void migrate_fd_connect(MigrationState *s,

Error *error_in)

migration/migration.c void migrate_fd_connect(MigrationState *s, Error *error_in)

Quoted 5 lines
migration/migration.c +2 -0
          return;
      }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_report("Can't migrate the vm that is in POSTMIGRATE

state");

migration/migration.c void migrate_fd_connect(MigrationState *s, · line 3627

Quoted 1 line
migration/migration.c +1 -0
+        migrate_set_state(&s->state, s->state,

MIGRATION_STATUS_FAILED);

migration/migration.c void migrate_fd_connect(MigrationState *s, Error *error_in) · line 3634

Quoted 8 lines
migration/migration.c +4 -0
+        migrate_fd_cleanup(s);
+        return;
+    }
+
      if (resume) {
          /* This is a resumed migration */
          rate_limit = s->parameters.max_postcopy_bandwidth /

I think, correct place for the check migrate_prepare, as it is called for any kind of migration, not only fd_. And in it we have already check for wrong state:

Quoted 6 lines
       if (runstate_check(RUN_STATE_INMIGRATE)) {
           error_setg(errp, "Guest is waiting for an incoming migration");
           return false;
       }

and no additional state change and cleanup is needed.

Thank you for your advise, I'll send another patch.

Quoted 3 lines

--
Best regards,
Vladimir