Skip to contentSkip to conversation

fix qmp-cont vs migration-finish race-crash

PATCH Applied Vladimir Sementsov-Ogievskiy 5 participants 9 messages Jan 24, 2019 – May 15, 2019

lore
Share & tools
lore
#0 lore raw

cover fix qmp-cont vs migration-finish race-crash

Hi.

It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.html

In thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.html

So, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.

The problem of this fix is that we finally have a bit weird interface:

User gets event MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".

But it is better than crash, anyway.

Vladimir Sementsov-Ogievskiy (2):
qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

 qmp.c                  | 3 +++
 tests/qemu-iotests/169 | 7 ++++++-
 2 files changed, 9 insertions(+), 1 deletion(-)
Signature · 1 line
2.18.0
#1 lore raw

PATCH 1/2 qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE

qmp_cont in RUN_STATE_FINISH_MIGRATE may lead to moving vm to RUN_STATE_RUNNING, before actual migration finish. So, when migration thread will try to go to RUN_STATE_POSTMIGRATE, assuming transition RUN_STATE_FINISH_MIGRATE->RUN_STATE_POSTMIGRATE, it will crash, as current state is RUN_STATE_RUNNING, and transition RUN_STATE_RUNNING->RUN_STATE_POSTMIGRATE is forbidden.

---
 qmp.c | 3 +++
 1 file changed, 3 insertions(+)
qmp.c +3 -0
index 4c819dd8cf..c2ecf1d804 100644
@@ -156,6 +156,9 @@ void qmp_cont(Error **errp)
         return;
     } else if (runstate_check(RUN_STATE_SUSPENDED)) {
         return;
+    } else if (runstate_check(RUN_STATE_FINISH_MIGRATE)) {
+        error_setg(errp, "Migration is not finalized yet");
+        return;
     }
 
     for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
Signature · 1 line
2.18.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE

Quoted 18 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:

qmp_cont in RUN_STATE_FINISH_MIGRATE may lead to moving vm to RUN_STATE_RUNNING, before actual migration finish. So, when migration thread will try to go to RUN_STATE_POSTMIGRATE, assuming transition RUN_STATE_FINISH_MIGRATE->RUN_STATE_POSTMIGRATE, it will crash, as current state is RUN_STATE_RUNNING, and transition RUN_STATE_RUNNING->RUN_STATE_POSTMIGRATE is forbidden.

---
 qmp.c | 3 +++
 1 file changed, 3 insertions(+)
qmp.c +0 -0
index 4c819dd8cf..c2ecf1d804 100644

qmp.c void qmp_cont(Error **errp) · line 162

Quoted 7 lines
qmp.c +3 -0
@@ -156,6 +156,9 @@ void qmp_cont(Error **errp)
         return;
     } else if (runstate_check(RUN_STATE_SUSPENDED)) {
         return;
+    } else if (runstate_check(RUN_STATE_FINISH_MIGRATE)) {
+        error_setg(errp, "Migration is not finalized yet");
+        return;
     }

Yeh, a bit of a hack, but I think that's as best as we can do for now; we can't ban all cont's during migration.

Note, it's still racy since you could call this partway through the completion before it's set to finish.

Quoted 4 lines
qmp.c +0 -0
     for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
Signature · 1 line
2.18.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#3 lore raw

PATCH 2/2 iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

In reply to Vladimir Sementsov-Ogievskiy

qmp_cont fails if vm in RUN_STATE_FINISH_MIGRATE, so let's wait for final RUN_STATE_POSTMIGRATE. Also, while being here, check qmp_cont result.

---
 tests/qemu-iotests/169 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tests/qemu-iotests/169 +6 -1
index 527aebd0cb..7e06cc1145 100755
@@ -102,12 +102,17 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             event = self.vm_a.event_wait('MIGRATION')
             if event['data']['status'] == 'completed':
                 break
+        while True:
+            result = self.vm_a.qmp('query-status')
+            if (result['return']['status'] == 'postmigrate'):
+                break
 
         # test that bitmap is still here
         removed = (not migrate_bitmaps) and persistent
         self.check_bitmap(self.vm_a, False if removed else sha256)
 
-        self.vm_a.qmp('cont')
+        result = self.vm_a.qmp('cont')
+        self.assert_qmp(result, 'return', {})
 
         # test that bitmap is still here after invalidation
         self.check_bitmap(self.vm_a, sha256)
Signature · 1 line
2.18.0
#4 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

Quoted 15 lines · Vladimir Sementsov-Ogievskiy

qmp_cont fails if vm in RUN_STATE_FINISH_MIGRATE, so let's wait for final RUN_STATE_POSTMIGRATE. Also, while being here, check qmp_cont result.

---
  tests/qemu-iotests/169 | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
tests/qemu-iotests/169 +0 -0
index 527aebd0cb..7e06cc1145 100755

tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 108

Quoted 7 lines
tests/qemu-iotests/169 +4 -0
@@ -102,12 +102,17 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
              event = self.vm_a.event_wait('MIGRATION')
              if event['data']['status'] == 'completed':
                  break
+        while True:
+            result = self.vm_a.qmp('query-status')
+            if (result['return']['status'] == 'postmigrate'):
+                break

possibly, not bad to add

+ time.sleep(0.1)

or 0.5, or something. (Note, that time module is already imported to the test)

Quoted 12 lines
tests/qemu-iotests/169 +2 -1
          # test that bitmap is still here
          removed = (not migrate_bitmaps) and persistent
          self.check_bitmap(self.vm_a, False if removed else sha256)
 
-        self.vm_a.qmp('cont')
+        result = self.vm_a.qmp('cont')
+        self.assert_qmp(result, 'return', {})
 
          # test that bitmap is still here after invalidation
          self.check_bitmap(self.vm_a, sha256)
 
Signature · 2 lines
Best regards,
Vladimir
Tested-by on v1 2/2
#5 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

In reply to Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Vladimir Sementsov-Ogievskiy

qmp_cont fails if vm in RUN_STATE_FINISH_MIGRATE, so let's wait for final RUN_STATE_POSTMIGRATE. Also, while being here, check qmp_cont result.

---
 tests/qemu-iotests/169 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

Thanks!

#6 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Vladimir Sementsov-Ogievskiy

Hi.

It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.html

In thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.html

So, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.

The problem of this fix is that we finally have a bit weird interface:

Quoted 8 lines

User gets event MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".

But it is better than crash, anyway.

Vladimir Sementsov-Ogievskiy (2):
qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

What happened to these two patches? As far as I can see, they've never got applied? Has another fix for 169 included instead?

Thomas

#7 lore raw
Quoted 19 lines · * Thomas Huth ([email protected]) wrote:
Quoted 18 lines · Vladimir Sementsov-Ogievskiy

Hi.

It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.html

In thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.html

So, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.

The problem of this fix is that we finally have a bit weird interface:

User gets event MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".

Quoted 8 lines
Quoted 5 lines

But it is better than crash, anyway.

Vladimir Sementsov-Ogievskiy (2):
qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE

What happened to these two patches? As far as I can see, they've never got applied? Has another fix for 169 included instead?

Hmm not sure; they've slipped between the maintainers if not. 1/2 is a qmp.c so while it fixes a migration corner case it's probably armbru that should take it.

Dave

Quoted 1 line

Thomas

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Applied
#8 lore raw
In reply to Vladimir Sementsov-Ogievskiy

Am 24.01.2019 um 13:25 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 13 lines

Hi.

It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.html

In thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.html

So, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.

Quoted 6 lines

The problem of this fix is that we finally have a bit weird interface:

User gets event MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".

But it is better than crash, anyway.

Thanks, applied to the block branch.

Kevin