fix qmp-cont vs migration-finish race-crash
cover fix qmp-cont vs migration-finish race-crash
Hi.
It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.html
In thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.html
So, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.
The problem of this fix is that we finally have a bit weird interface:
User gets event MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".
But it is better than crash, anyway.
Vladimir Sementsov-Ogievskiy (2):
qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE
qmp.c | 3 +++ tests/qemu-iotests/169 | 7 ++++++- 2 files changed, 9 insertions(+), 1 deletion(-)
Signature · 1 line
2.18.0
PATCH 1/2 qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
qmp_cont in RUN_STATE_FINISH_MIGRATE may lead to moving vm to RUN_STATE_RUNNING, before actual migration finish. So, when migration thread will try to go to RUN_STATE_POSTMIGRATE, assuming transition RUN_STATE_FINISH_MIGRATE->RUN_STATE_POSTMIGRATE, it will crash, as current state is RUN_STATE_RUNNING, and transition RUN_STATE_RUNNING->RUN_STATE_POSTMIGRATE is forbidden.
--- qmp.c | 3 +++ 1 file changed, 3 insertions(+)
qmp.c +3 -0
@@ -156,6 +156,9 @@ void qmp_cont(Error **errp)return ; }else if (runstate_check (RUN_STATE_SUSPENDED )) {return ; + }else if (runstate_check (RUN_STATE_FINISH_MIGRATE )) { +error_setg (errp,"Migration is not finalized yet" ); +return ; }for (blk =blk_next (NULL ); blk; blk =blk_next (blk)) {
Signature · 1 line
2.18.0
Re: [Qemu-devel] [PATCH 1/2] qmp: forbid qmp_cont in RUN_STATE_FINISH_MIGRATE
Quoted 18 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:
qmp_continRUN_STATE_FINISH_MIGRATEmay lead to moving vm toRUN_STATE_RUNNING, before actual migration finish. So, when migration thread will try to go toRUN_STATE_POSTMIGRATE, assuming transitionRUN_STATE_FINISH_MIGRATE->RUN_STATE_POSTMIGRATE, it will crash, as current state isRUN_STATE_RUNNING, and transitionRUN_STATE_RUNNING->RUN_STATE_POSTMIGRATEis forbidden.Reported-by Max Reitz <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qmp.c | 3 +++ 1 file changed, 3 insertions(+)qmp.c +0
-0
qmp.c void qmp_cont(Error **errp) · line 162
Quoted 7 lines
qmp.c +3
-0@@ -156,6 +156,9 @@ void qmp_cont(Error **errp)return ; }else if (runstate_check (RUN_STATE_SUSPENDED )) {return ; + }else if (runstate_check (RUN_STATE_FINISH_MIGRATE )) { +error_setg (errp,"Migration is not finalized yet" ); +return ; }
Yeh, a bit of a hack, but I think that's as best as we can do for now; we can't ban all cont's during migration.
Note, it's still racy since you could call this partway through the completion before it's set to finish.
Quoted 4 lines
qmp.c +0
-0for (blk =blk_next (NULL ); blk; blk =blk_next (blk)) {Signature · 1 line
2.18.0
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 2/2 iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE
In reply to Vladimir Sementsov-Ogievskiyqmp_cont fails if vm in RUN_STATE_FINISH_MIGRATE, so let's wait for final RUN_STATE_POSTMIGRATE. Also, while being here, check qmp_cont result.
--- tests/qemu-iotests/169 | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
tests/qemu-iotests/169 +6 -1
@@ -102,12 +102,17 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): event = self.vm_a.event_wait('MIGRATION') if event['data']['status'] == 'completed': break + while True: + result = self.vm_a.qmp('query-status') + if (result['return']['status'] == 'postmigrate'): + break # test that bitmap is still here removed = (not migrate_bitmaps) and persistent self.check_bitmap(self.vm_a, False if removed else sha256) - self.vm_a.qmp('cont') + result = self.vm_a.qmp('cont') + self.assert_qmp(result, 'return', {}) # test that bitmap is still here after invalidation self.check_bitmap(self.vm_a, sha256)
Signature · 1 line
2.18.0
Re: [Qemu-devel] [PATCH 2/2] iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
qmp_contfails if vm inRUN_STATE_FINISH_MIGRATE, so let's wait for finalRUN_STATE_POSTMIGRATE. Also, while being here, checkqmp_contresult.Reported-by Max Reitz <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/169 | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)tests/qemu-iotests/169 +0
-0
tests/qemu-iotests/169 class TestDirtyBitmapMigration(iotests.QMPTestCase): · line 108
Quoted 7 lines
tests/qemu-iotests/169 +4
-0@@ -102,12 +102,17 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): event = self.vm_a.event_wait('MIGRATION') if event['data']['status'] == 'completed': break + while True: + result = self.vm_a.qmp('query-status') + if (result['return']['status'] == 'postmigrate'): + break
possibly, not bad to add
+ time.sleep(0.1)
or 0.5, or something. (Note, that time module is already imported to the test)
Quoted 12 lines
tests/qemu-iotests/169 +2
-1# test that bitmap is still here removed = (not migrate_bitmaps) and persistent self.check_bitmap(self.vm_a, False if removed else sha256) - self.vm_a.qmp('cont') + result = self.vm_a.qmp('cont') + self.assert_qmp(result, 'return', {}) # test that bitmap is still here after invalidation self.check_bitmap(self.vm_a, sha256)
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 2/2] iotest: fix 169: do not run qmp_cont in RUN_STATE_FINISH_MIGRATE
In reply to Vladimir Sementsov-OgievskiyQuoted 9 lines · Vladimir Sementsov-Ogievskiy
qmp_contfails if vm inRUN_STATE_FINISH_MIGRATE, so let's wait for finalRUN_STATE_POSTMIGRATE. Also, while being here, checkqmp_contresult.Reported-by Max Reitz <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/169 | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Thanks!
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Hi.
It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.htmlIn thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.htmlSo, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.
The problem of this fix is that we finally have a bit weird interface:
Quoted 8 lines
User gets event
MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".But it is better than crash, anyway.
Vladimir Sementsov-Ogievskiy (2):
qmp: forbidqmp_continRUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not runqmp_continRUN_STATE_FINISH_MIGRATE
What happened to these two patches? As far as I can see, they've never got applied? Has another fix for 169 included instead?
Thomas
Quoted 19 lines · * Thomas Huth ([email protected]) wrote:
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Hi.
It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.htmlIn thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.htmlSo, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.
The problem of this fix is that we finally have a bit weird interface:
User gets event
MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".
Quoted 8 lines
Quoted 5 lines
But it is better than crash, anyway.
Vladimir Sementsov-Ogievskiy (2):
qmp: forbidqmp_continRUN_STATE_FINISH_MIGRATE
iotest: fix 169: do not runqmp_continRUN_STATE_FINISH_MIGRATEWhat happened to these two patches? As far as I can see, they've never got applied? Has another fix for 169 included instead?
Hmm not sure; they've slipped between the maintainers if not. 1/2 is a qmp.c so while it fixes a migration corner case it's probably armbru that should take it.
Dave
Quoted 1 line
Thomas
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Kevin Wolf
AppliedAm 24.01.2019 um 13:25 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 13 lines
Hi.
It's a simple fix for problems reported in "Aborts in iotest 169" by Max:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05907.htmlIn thread Kevin described that a problem itself is bigger and needs more effort:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06136.htmlSo, we may continue discussion in "Aborts in iotest 169", and in parallel apply these patches at least as a temporary fix.
Quoted 6 lines
The problem of this fix is that we finally have a bit weird interface:
User gets event
MIGRATION_COMPLETED, and after it he can get error message "Migration is not finalized yet".But it is better than crash, anyway.
Thanks, applied to the block branch.
Kevin