iotests: 194: wait migration completion on target too
PATCH 1/1 iotests: 194: wait migration completion on target too
It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.
We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.
Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.
Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.
--- tests/qemu-iotests/194 | 10 ++++++++++ tests/qemu-iotests/194.out | 5 +++++ 2 files changed, 15 insertions(+)
tests/qemu-iotests/194 +10 -0
@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \ iotests.log(dest_vm.qmp('nbd-server-stop')) break + iotests.log('Wait migration completion on target...') + migr_events = (('MIGRATION', {'data': {'status': 'completed'}}), + ('MIGRATION', {'data': {'status': 'failed'}})) + event = dest_vm.events_wait(migr_events) + iotests.log(event, filters=[iotests.filter_qmp_event]) + + iotests.log('Check bitmaps on source:') iotests.log(source_vm.qmp('query-block')['return'][0]['dirty-bitmaps']) + + iotests.log('Check bitmaps on target:') + iotests.log(dest_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])
tests/qemu-iotests/194.out +5 -0
@@ -21,4 +21,9 @@ Gracefully ending the `drive-mirror` job on source... {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Stopping the NBD server on destination... {"return": {}} +Wait migration completion on target... +{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +Check bitmaps on source: +[{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}] +Check bitmaps on target: [{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]
Signature · 1 line
2.21.0
Quoted 8 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:
It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in
bdrv_release_dirty_bitmap_locked()on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.
Quoted 7 lines
Note that in this test
dest_vm.shutdown()is called implicitly, as vms used as context-providers, see__exit__()method of QEMUMachine class.Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit
ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.
Without knowing the iotests framework, the actual problem sounds right; I just fixed a similar bug in the acceptance test migration code. Every migration test seems to make the same mistake!
Dave
Quoted 44 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3bReported-by Thomas Huth <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/194 | 10 ++++++++++ tests/qemu-iotests/194.out | 5 +++++ 2 files changed, 15 insertions(+)tests/qemu-iotests/194 +10
-0@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \ iotests.log(dest_vm.qmp('nbd-server-stop')) break + iotests.log('Wait migration completion on target...') + migr_events = (('MIGRATION', {'data': {'status': 'completed'}}), + ('MIGRATION', {'data': {'status': 'failed'}})) + event = dest_vm.events_wait(migr_events) + iotests.log(event, filters=[iotests.filter_qmp_event]) + + iotests.log('Check bitmaps on source:') iotests.log(source_vm.qmp('query-block')['return'][0]['dirty-bitmaps']) + + iotests.log('Check bitmaps on target:') + iotests.log(dest_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])tests/qemu-iotests/194.out +5
-0@@ -21,4 +21,9 @@ Gracefully ending the `drive-mirror` job on source... {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} Stopping the NBD server on destination... {"return": {}} +Wait migration completion on target... +{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +Check bitmaps on source: +[{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}] +Check bitmaps on target: [{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]Signature · 1 line
2.21.0
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in
bdrv_release_dirty_bitmap_locked()on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.
Note that in this test
dest_vm.shutdown()is called implicitly, as vms used as context-providers, see__exit__()method of QEMUMachine class.Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit
ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.
Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3bReported-by Thomas Huth <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/194 | 10 ++++++++++ tests/qemu-iotests/194.out | 5 +++++ 2 files changed, 15 insertions(+)
Thanks, this fixes the issue in the gitlab CI:
https://gitlab.com/huth/qemu/-/jobs/580902477#L3780
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in
bdrv_release_dirty_bitmap_locked()on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.
Note that in this test
dest_vm.shutdown()is called implicitly, as vms used as context-providers, see__exit__()method of QEMUMachine class.Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit
ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.
Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3bReported-by Thomas Huth <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/194 | 10 ++++++++++ tests/qemu-iotests/194.out | 5 +++++ 2 files changed, 15 insertions(+)
Will queue through my NBD tree.
tests/qemu-iotests/194 with iotests.FilePath('source.img') as source_img_path, \ · line 90
Quoted 10 lines
tests/qemu-iotests/194 +1
-0@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \ iotests.log(dest_vm.qmp('nbd-server-stop')) break + iotests.log('Wait migration completion on target...')
Do you mind if I s/Wait/Wait for/ here and in the .out?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Quoted 30 lines · Eric Blake
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in
bdrv_release_dirty_bitmap_locked()on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.
Note that in this test
dest_vm.shutdown()is called implicitly, as vms used as context-providers, see__exit__()method of QEMUMachine class.Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit
ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3bReported-by Thomas Huth <[email protected]>Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
tests/qemu-iotests/194 | 10 ++++++++++
tests/qemu-iotests/194.out | 5 +++++2 files changed, 15 insertions(+)Reviewed-by Eric Blake <[email protected]>Will queue through my NBD tree.
Quoted 1 line
Quoted 11 lines
Quoted 8 lines
tests/qemu-iotests/194 +0
-0@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \ iotests.log(dest_vm.qmp('nbd-server-stop'))break
+iotests.log('Wait migration completion on target...')Do you mind if I s/Wait/Wait for/ here and in the .out?
No objections of course
Signature · 2 lines
Best regards, Vladimir