Skip to contentSkip to conversation

iotests: 194: wait migration completion on target too

PATCH In review Vladimir Sementsov-Ogievskiy 4 participants 5 messages Jun 4, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iotests: 194: wait migration completion on target too

It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.

We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.

Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.

Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.

Fixes ae00aa2398476824f0eca80461da215e7cdc1c3b
---
 tests/qemu-iotests/194     | 10 ++++++++++
 tests/qemu-iotests/194.out |  5 +++++
 2 files changed, 15 insertions(+)
tests/qemu-iotests/194 +10 -0
index 3fad7c6c1a..6dc2bc94d7 100755
@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \
             iotests.log(dest_vm.qmp('nbd-server-stop'))
             break
 
+    iotests.log('Wait migration completion on target...')
+    migr_events = (('MIGRATION', {'data': {'status': 'completed'}}),
+                   ('MIGRATION', {'data': {'status': 'failed'}}))
+    event = dest_vm.events_wait(migr_events)
+    iotests.log(event, filters=[iotests.filter_qmp_event])
+
+    iotests.log('Check bitmaps on source:')
     iotests.log(source_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])
+
+    iotests.log('Check bitmaps on target:')
+    iotests.log(dest_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])
tests/qemu-iotests/194.out +5 -0
index dd60dcc14f..f70cf7610e 100644
@@ -21,4 +21,9 @@ Gracefully ending the `drive-mirror` job on source...
 {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Stopping the NBD server on destination...
 {"return": {}}
+Wait migration completion on target...
+{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+Check bitmaps on source:
+[{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]
+Check bitmaps on target:
 [{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]
Signature · 1 line
2.21.0
#1 lore raw
Quoted 8 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:

It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.

We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.

Quoted 7 lines

Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.

Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.

Without knowing the iotests framework, the actual problem sounds right; I just fixed a similar bug in the acceptance test migration code. Every migration test seems to make the same mistake!

Dave

Quoted 44 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3b
---
 tests/qemu-iotests/194     | 10 ++++++++++
 tests/qemu-iotests/194.out |  5 +++++
 2 files changed, 15 insertions(+)
tests/qemu-iotests/194 +10 -0
index 3fad7c6c1a..6dc2bc94d7 100755
@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \
             iotests.log(dest_vm.qmp('nbd-server-stop'))
             break
 
+    iotests.log('Wait migration completion on target...')
+    migr_events = (('MIGRATION', {'data': {'status': 'completed'}}),
+                   ('MIGRATION', {'data': {'status': 'failed'}}))
+    event = dest_vm.events_wait(migr_events)
+    iotests.log(event, filters=[iotests.filter_qmp_event])
+
+    iotests.log('Check bitmaps on source:')
     iotests.log(source_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])
+
+    iotests.log('Check bitmaps on target:')
+    iotests.log(dest_vm.qmp('query-block')['return'][0]['dirty-bitmaps'])
tests/qemu-iotests/194.out +5 -0
index dd60dcc14f..f70cf7610e 100644
@@ -21,4 +21,9 @@ Gracefully ending the `drive-mirror` job on source...
 {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Stopping the NBD server on destination...
 {"return": {}}
+Wait migration completion on target...
+{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+Check bitmaps on source:
+[{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]
+Check bitmaps on target:
 [{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "persistent": false, "recording": true, "status": "active"}]
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Tested-by on v1 1/1
#2 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 16 lines · Vladimir Sementsov-Ogievskiy

It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.

We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.

Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.

Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.

Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3b
---
 tests/qemu-iotests/194     | 10 ++++++++++
 tests/qemu-iotests/194.out |  5 +++++
 2 files changed, 15 insertions(+)

Thanks, this fixes the issue in the gitlab CI:

https://gitlab.com/huth/qemu/-/jobs/580902477#L3780

Reviewed-by on v1 1/1
#3 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 16 lines · Vladimir Sementsov-Ogievskiy

It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.

We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.

Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.

Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.

Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3b
---
  tests/qemu-iotests/194     | 10 ++++++++++
  tests/qemu-iotests/194.out |  5 +++++
  2 files changed, 15 insertions(+)

Will queue through my NBD tree.

tests/qemu-iotests/194 with iotests.FilePath('source.img') as source_img_path, \ · line 90

Quoted 10 lines
tests/qemu-iotests/194 +1 -0
index 3fad7c6c1a..6dc2bc94d7 100755
@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \
              iotests.log(dest_vm.qmp('nbd-server-stop'))
              break
 
+    iotests.log('Wait migration completion on target...')

Do you mind if I s/Wait/Wait for/ here and in the .out?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#4 lore raw
Quoted 30 lines · Eric Blake
Quoted 16 lines · Vladimir Sementsov-Ogievskiy

It is possible, that shutdown on target occurs earlier than migration finish. In this case we crash in bdrv_release_dirty_bitmap_locked() on assertion "assert(!bdrv_dirty_bitmap_busy(bitmap));" as we do have busy bitmap, as bitmap migration is ongoing.

We'll fix bitmap migration to gracefully cancel on early shutdown soon. Now let's fix iotest 194 to wait migration completion before shutdown.

Note that in this test dest_vm.shutdown() is called implicitly, as vms used as context-providers, see __exit__() method of QEMUMachine class.

Actually, not waiting migration finish is a wrong thing, but the test started to crash after commit ae00aa239847682
"iotests: 194: test also migration of dirty bitmap", which added dirty bitmaps here. So, Fixes: tag won't hurt.

Quoted 7 lines
Fixes ae00aa2398476824f0eca80461da215e7cdc1c3b

---
  tests/qemu-iotests/194     | 10 ++++++++++
  tests/qemu-iotests/194.out |  5 +++++

  2 files changed, 15 insertions(+)

Will queue through my NBD tree.

Quoted 1 line
Quoted 11 lines
Quoted 8 lines
tests/qemu-iotests/194 +0 -0
index 3fad7c6c1a..6dc2bc94d7 100755
@@ -87,4 +87,14 @@ with iotests.FilePath('source.img') as source_img_path, \
               iotests.log(dest_vm.qmp('nbd-server-stop'))

              break
+    iotests.log('Wait migration completion on target...')

Do you mind if I s/Wait/Wait for/ here and in the .out?

No objections of course

Signature · 2 lines
Best regards,
Vladimir