Skip to contentSkip to conversation

migration: Don't allow migration if vm is in POSTMIGRATE

PATCH In review Tuguoyi 4 participants 5 messages Dec 8, 2020 – Dec 17, 2020

lore
Share & tools
lore

Tuguoyi

author
#0 lore raw

PATCH v3 1/1 migration: Don't allow migration if vm is in POSTMIGRATE

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
    at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
    at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

As Vladimir suggested, this patch makes migrate_prepare check the state of vm and return error if it is in RUN_STATE_POSTMIGRATE state.

---
 migration/migration.c | 6 ++++++
 1 file changed, 6 insertions(+)
migration/migration.c +6 -0
index 87a9b59..5e33962 100644
@@ -2115,6 +2115,12 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
         return false;
     }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_setg(errp, "Can't migrate the vm that was paused due to "
+                   "previous migration");
+        return false;
+    }
+
     if (migration_is_blocked(errp)) {
         return false;
     }
Signature · 4 lines
2.7.4

[Patch v2]: https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg01318.html
[Patch v1]: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05950.html
Reviewed-by on v3 1/1
#1 lore raw
Quoted 28 lines · Tuguoyi

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
     at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
     at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

Quoted 4 lines

As Vladimir suggested, this patch makes migrate_prepare check the state of vm and return error if it is in RUN_STATE_POSTMIGRATE state.

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v3 1/1
#2 lore raw
In reply to Tuguoyi
Quoted 28 lines

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
    at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
    at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

Quoted 4 lines

As Vladimir suggested, this patch makes migrate_prepare check the state of vm and return error if it is in RUN_STATE_POSTMIGRATE state.

Similar issue is reported by Li Zhang(+CC) with almost same patch[3] to fix this.

[3] https://marc.info/?l=qemu-devel&m=160749859831357&w=2

Quoted 26 lines
---
 migration/migration.c | 6 ++++++
 1 file changed, 6 insertions(+)
migration/migration.c +6 -1
index 87a9b59..5e33962 100644
@@ -2115,6 +2115,12 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
         return false;
     }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_setg(errp, "Can't migrate the vm that was paused due to "
+                   "previous migration");
+        return false;
+    }
+
     if (migration_is_blocked(errp)) {
         return false;
     }
--

2.7.4

[Patch v2]: https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg01318.html
[Patch v1]: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05950.html

author
#3 lore raw

Ping.
It seems no one handle this patch.

Quoted 93 lines

-----Original Message-----
From: Pankaj Gupta [mailto:[email protected]]
Sent: Wednesday, December 09, 2020 10:21 PM
To: tuguoyi (Cloud) <[email protected]>
Cc: Juan Quintela <[email protected]>; Dr. David Alan Gilbert <[email protected]>; [email protected]; [email protected]; Li Zhang <[email protected]>
Subject: Re: [PATCH v3] migration: Don't allow migration if vm is in POSTMIGRATE

Quoted 3 lines

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh

snapshot-create-as'

Quoted 7 lines

- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from

/lib/x86_64-linux-gnu/libc.so.6

Quoted 1 line

#4 0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400)

at /build/qemu-5.0/block.c:5724

Quoted 1 line

#5 0x000055ca8dece967 in bdrv_inactivate_all () at

/build//qemu-5.0/block.c:5792

Quoted 1 line

#6 0x000055ca8de5539d in

qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)

Quoted 2 lines
    at /build/qemu-5.0/migration/savevm.c:1401

#7 qemu_savevm_state_complete_precopy (f=0x55ca907044b0,

iterable_only=iterable_only@entry=false,
inactivate_disks=inactivate_disks@entry=true)

Quoted 2 lines
    at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:2941

Quoted 1 line

#9 migration_iteration_run (s=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:3295

Quoted 1 line

#10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at

/build/qemu-5.0/migration/migration.c:3459

Quoted 1 line

#11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at

/build/qemu-5.0/util/qemu-thread-posix.c:519

Quoted 1 line

#12 0x00007fbf95c5f184 in start_thread () from

/lib/x86_64-linux-gnu/libpthread.so.0

Quoted 3 lines

#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set

BDRV_O_INACTIVE

Quoted 5 lines

flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

As Vladimir suggested, this patch makes migrate_prepare check the state of

vm and

Quoted 3 lines

return error if it is in RUN_STATE_POSTMIGRATE state.

Similar issue is reported by Li Zhang(+CC) with almost same patch[3] to fix this.

[3] https://marc.info/?l=qemu-devel&m=160749859831357&w=2

migration/migration.c static bool migrate_prepare(MigrationState *s,

Quoted 9 lines
---
 migration/migration.c | 6 ++++++
 1 file changed, 6 insertions(+)
migration/migration.c +0 -0
index 87a9b59..5e33962 100644
@@ -2115,6 +2115,12 @@ static bool migrate_prepare(MigrationState *s,

bool blk, bool blk_inc,

Quoted 15 lines
migration/migration.c +6 -1
         return false;
     }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_setg(errp, "Can't migrate the vm that was paused due to "
+                   "previous migration");
+        return false;
+    }
+
     if (migration_is_blocked(errp)) {
         return false;
     }
--

2.7.4

Quoted 1 line

[Patch v2]:

https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg01318.html

Quoted 1 line

[Patch v1]:

https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05950.html

#4 lore raw
In reply to Tuguoyi
Quoted 28 lines · * Tuguoyi ([email protected]) wrote:

The following steps will cause qemu assertion failure:
- pause vm by executing 'virsh suspend'
- create external snapshot of memory and disk using 'virsh snapshot-create-as'
- doing the above operation again will cause qemu crash

The backtrace looks like:

#0  0x00007fbf958c5c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbf958c9028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbf958bebf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fbf958beca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055ca8decd39d in bdrv_inactivate_recurse (bs=0x55ca90c80400) at /build/qemu-5.0/block.c:5724
#5  0x000055ca8dece967 in bdrv_inactivate_all () at /build//qemu-5.0/block.c:5792
#6  0x000055ca8de5539d in qemu_savevm_state_complete_precopy_non_iterable (inactivate_disks=true, in_postcopy=false, f=0x55ca907044b0)
    at /build/qemu-5.0/migration/savevm.c:1401
#7  qemu_savevm_state_complete_precopy (f=0x55ca907044b0, iterable_only=iterable_only@entry=false, inactivate_disks=inactivate_disks@entry=true)
    at /build/qemu-5.0/migration/savevm.c:1453

#8 0x000055ca8de4f581 in migration_completion (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:2941 #9 migration_iteration_run (s=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3295 #10 migration_thread (opaque=opaque@entry=0x55ca8f64d9f0) at /build/qemu-5.0/migration/migration.c:3459 #11 0x000055ca8dfc6716 in qemu_thread_start (args=<optimized out>) at /build/qemu-5.0/util/qemu-thread-posix.c:519 #12 0x00007fbf95c5f184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fbf9598cbed in clone () from /lib/x86_64-linux-gnu/libc.so.6

When the first migration completes, bs->open_flags will set BDRV_O_INACTIVE flag by bdrv_inactivate_all(), and during the second migration the bdrv_inactivate_recurse assert that the bs->open_flags is already BDRV_O_INACTIVE enabled which cause crash.

Quoted 4 lines

As Vladimir suggested, this patch makes migrate_prepare check the state of vm and return error if it is in RUN_STATE_POSTMIGRATE state.

Yes, we've had this problem for a long long time; there are a bunch of other similar conditions; the real answer is to figure out some command for explicitly handing back control of the block devices (without actually restarting the CPUs).

However, this is a reasonable patch to cover the common case.

Queued

Quoted 26 lines
---
 migration/migration.c | 6 ++++++
 1 file changed, 6 insertions(+)
migration/migration.c +6 -0
index 87a9b59..5e33962 100644
@@ -2115,6 +2115,12 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
         return false;
     }
 
+    if (runstate_check(RUN_STATE_POSTMIGRATE)) {
+        error_setg(errp, "Can't migrate the vm that was paused due to "
+                   "previous migration");
+        return false;
+    }
+
     if (migration_is_blocked(errp)) {
         return false;
     }
Signature · 4 lines
2.7.4

[Patch v2]: https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg01318.html
[Patch v1]: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05950.html
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK