backup: Copy only dirty areas
cover backup: Copy only dirty areas
Hi,
In a discussion with Vladimir today, we noticed that the backup job currently is pretty broken when using copy offloading. I don’t know about you, but my local filesystem (XFS) supports copy offloading, so the job uses it automatically. That means, that backup is broken and has been broken for a year on my local FS.
The last working version was 2.12, so this isn’t a regression in 4.1. We could thus justify moving it to 4.2. But I think this should really go into 4.1, because this is not really an edge case and as far as I know users cannot do anything to prevent the backup job from performing copy offloading if the system and all involved block drivers support it. I just wonder why nobody has noticed...
Max Reitz (2):
backup: Copy only dirty areas
iotests: Test backup job with two guest writes
block/backup.c | 13 +++++++++++-- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 3 files changed, 47 insertions(+), 4 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/2 backup: Copy only dirty areas
The backup job must only copy areas that the copy_bitmap reports as dirty. This is always the case when using traditional non-offloading backup, because it copies each cluster separately. When offloading the copy operation, we sometimes copy more than one cluster at a time, but we only check whether the first one is dirty.
Therefore, whenever copy offloading is possible, the backup job currently produces wrong output when the guest writes to an area of which an inner part has already been backed up, because that inner part will be re-copied.
--- block/backup.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
block/backup.c +11 -2
@@ -202,22 +202,31 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,cow_request_begin (&cow_request, job, start, end);while (start < end) { +int64_t dirty_end; +if (!hbitmap_get (job->copy_bitmap, start)) {trace_backup_do_cow_skip (job, start); start += job->cluster_size;continue ;/* already copied */ } + dirty_end =hbitmap_next_zero (job->copy_bitmap, start, (end - start)); +if (dirty_end <0 ) { + dirty_end = end; + } +trace_backup_do_cow_process (job, start);if (job->use_copy_range) { - ret =backup_cow_with_offload (job, start, end, is_write_notifier); + ret =backup_cow_with_offload (job, start, dirty_end, + is_write_notifier);if (ret <0 ) { job->use_copy_range = false; } }if (!job->use_copy_range) { - ret =backup_cow_with_bounce_buffer (job, start, end, is_write_notifier, + ret =backup_cow_with_bounce_buffer (job, start, dirty_end, + is_write_notifier, error_is_read, &bounce_buffer); }if (ret <0 ) {
Signature · 1 line
2.21.0
Quoted 27 lines · Max Reitz
The backup job must only copy areas that the
copy_bitmapreports as dirty. This is always the case when using traditional non-offloading backup, because it copies each cluster separately. When offloading the copy operation, we sometimes copy more than one cluster at a time, but we only check whether the first one is dirty.Therefore, whenever copy offloading is possible, the backup job currently produces wrong output when the guest writes to an area of which an inner part has already been backed up, because that inner part will be re-copied.
Fixes 9ded4a0114968e98b41494fc035ba14f84cdf700Signed-off-by Max Reitz <[email protected]>--- block/backup.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)block/backup.c +2
-0@@ -202,22 +202,31 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,cow_request_begin (&cow_request, job, start, end);while (start < end) { +int64_t dirty_end; +
block/backup.c static int coroutine_fn backup_do_cow(BackupBlockJob *job, · line 216
Quoted 10 lines
block/backup.c +4
-0if (!hbitmap_get (job->copy_bitmap, start)) {trace_backup_do_cow_skip (job, start); start += job->cluster_size;continue ;/* already copied */ } + dirty_end =hbitmap_next_zero (job->copy_bitmap, start, (end - start)); +if (dirty_end <0 ) { + dirty_end = end; + }
1. hbitmap_next_dirty_area is better I think
2. we can do it only in use_copy_range case, as backup_cow_with_bounce_buffer copies only one cluster anyway
But this all should be refactored anyway, so:
Quoted 19 lines
block/backup.c +5
-2+trace_backup_do_cow_process (job, start);if (job->use_copy_range) { - ret =backup_cow_with_offload (job, start, end, is_write_notifier); + ret =backup_cow_with_offload (job, start, dirty_end, + is_write_notifier);if (ret <0 ) { job->use_copy_range = false; } }if (!job->use_copy_range) { - ret =backup_cow_with_bounce_buffer (job, start, end, is_write_notifier, + ret =backup_cow_with_bounce_buffer (job, start, dirty_end, + is_write_notifier, error_is_read, &bounce_buffer); }if (ret <0 ) {
Signature · 2 lines
Best regards, Vladimir
PATCH 2/2 iotests: Test backup job with two guest writes
In reply to Max ReitzPerform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.
--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)
tests/qemu-iotests/056 +34 -0
@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()
@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:
@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])
tests/qemu-iotests/056.out +2 -2
@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OK
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 82 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +34
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +0
-0
tests/qemu-iotests/056.out line 5
Quoted 7 lines
tests/qemu-iotests/056.out +2
-2@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OK
Failed for me:
-..........
+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F...
+======================================================================
+FAIL: test_overlapping_writes (__main__.BackupTest)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+ File "056", line 212, in test_overlapping_writes
+ 'target image does not match reference image')
+AssertionError: False is not true : target image does not match reference image
+
----------------------------------------------------------------------
Ran 10 tests
-OK
+FAILED (failures=1)
So, with applied
@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase):
res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0)
self.assert_qmp(res, 'return', {})
self.qmp_backup_wait('drive0')+ self.vm.shutdown()
self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img),
'target image does not match reference image')
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 99 lines · Vladimir Sementsov-Ogievskiy
Quoted 82 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +34
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +0
-0tests/qemu-iotests/056.out line 5
Quoted 7 lines
tests/qemu-iotests/056.out +2
-2@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OKFailed for me:
-..........(quoted diff) +6
-0+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F... +====================================================================== +FAIL: test_overlapping_writes (__main__.BackupTest) +----------------------------------------------------------------------
Review
Quoted 10 lines
(quoted diff) +6
-1+Traceback (most recent call last): + File "056", line 212, in test_overlapping_writes + 'target image does not match reference image') +AssertionError: False is not true : target image does not match reference image + ---------------------------------------------------------------------- Ran 10 tests -OK +FAILED (failures=1)
Hm. I hoped seeing BLOCK_JOB_COMPLETED would be enough.
Review class BackupTest(iotests.QMPTestCase): · line 213
Quoted 10 lines
So, with applied
(quoted diff) +1
-0@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(res, 'return', {}) self.qmp_backup_wait('drive0') + self.vm.shutdown() self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), 'target image does not match reference image')
I’d personally prefer auto_dismiss=False and then block-job-dismiss. Although I can’t give a reason why.
Quoted 2 lines
In any case, thanks!
Max
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 108 lines · Max Reitz
Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 82 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +34
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +0
-0tests/qemu-iotests/056.out line 5
Quoted 7 lines
tests/qemu-iotests/056.out +2
-2@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OKFailed for me:
-..........(quoted diff) +11
-0+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F... +====================================================================== +FAIL: test_overlapping_writes (__main__.BackupTest) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "056", line 212, in test_overlapping_writes + 'target image does not match reference image') +AssertionError: False is not true : target image does not match reference image + ---------------------------------------------------------------------- Ran 10 tests
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1-OK +FAILED (failures=1)Hm. I hoped seeing
BLOCK_JOB_COMPLETEDwould be enough.
The problem is higher: "Failed to get shared "write" lock". Because of this iotests.compare_images can't work. So, because of locking, we need to shutdown qemu before starting qemu-img. And it's done so in test_complete_top() (I just looked at it as it's the only other user of compare_images in 056)
Quoted 22 lines
Review class BackupTest(iotests.QMPTestCase): · line 213
Quoted 11 lines
So, with applied
(quoted diff) +1
-0@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(res, 'return', {}) self.qmp_backup_wait('drive0') + self.vm.shutdown() self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), 'target image does not match reference image')I’d personally prefer auto_dismiss=False and then block-job-dismiss. Although I can’t give a reason why.
Quoted 2 lines
In any case, thanks!
Max
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 114 lines · Vladimir Sementsov-Ogievskiy
Quoted 108 lines · Max Reitz
Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 82 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +34
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +0
-0tests/qemu-iotests/056.out line 5
Quoted 7 lines
tests/qemu-iotests/056.out +2
-2@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OKFailed for me:
-..........(quoted diff) +11
-0+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F... +====================================================================== +FAIL: test_overlapping_writes (__main__.BackupTest) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "056", line 212, in test_overlapping_writes + 'target image does not match reference image') +AssertionError: False is not true : target image does not match reference image + ---------------------------------------------------------------------- Ran 10 testsQuoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1-OK +FAILED (failures=1)Hm. I hoped seeing
BLOCK_JOB_COMPLETEDwould be enough.
Quoted 4 lines
The problem is higher: "Failed to get shared "write" lock". Because of this iotests.compare_images can't work. So, because of locking, we need to shutdown qemu before starting qemu-img. And it's done so in
test_complete_top()(I just looked at it as it's the only other user ofcompare_imagesin 056)
The destination image shouldn’t be in use by qemu after the block job is done, though. Therefore, there shouldn’t be a lock issue. That’s what I meant.
Max
Quoted 23 lines
Quoted 21 lines
Review class BackupTest(iotests.QMPTestCase): · line 213
Quoted 10 lines
So, with applied
(quoted diff) +1
-0@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(res, 'return', {}) self.qmp_backup_wait('drive0') + self.vm.shutdown() self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), 'target image does not match reference image')I’d personally prefer auto_dismiss=False and then block-job-dismiss. Although I can’t give a reason why.
Quoted 2 lines
In any case, thanks!
Max
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 115 lines · Max Reitz
Quoted 114 lines · Vladimir Sementsov-Ogievskiy
Quoted 112 lines · Max Reitz
Quoted 99 lines · Vladimir Sementsov-Ogievskiy
Quoted 82 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +34
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +0
-0tests/qemu-iotests/056.out line 5
Quoted 7 lines
tests/qemu-iotests/056.out +2
-2@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OKFailed for me:
-..........(quoted diff) +6
-0+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F... +====================================================================== +FAIL: test_overlapping_writes (__main__.BackupTest) +----------------------------------------------------------------------Review
Quoted 10 lines
(quoted diff) +6
-1+Traceback (most recent call last): + File "056", line 212, in test_overlapping_writes + 'target image does not match reference image') +AssertionError: False is not true : target image does not match reference image + ---------------------------------------------------------------------- Ran 10 tests -OK +FAILED (failures=1)Hm. I hoped seeing
BLOCK_JOB_COMPLETEDwould be enough.
Quoted 9 lines
Quoted 4 lines
The problem is higher: "Failed to get shared "write" lock". Because of this iotests.compare_images can't work. So, because of locking, we need to shutdown qemu before starting qemu-img. And it's done so in
test_complete_top()(I just looked at it as it's the only other user ofcompare_imagesin 056)The destination image shouldn’t be in use by qemu after the block job is done, though. Therefore, there shouldn’t be a lock issue. That’s what I meant.
Ah, understand. Hmm, then it's strange that I see that error..
Clearly, in job_finalize_single(), job_clean() is called first, which should call backup_clean to unref target, then job_event_completed() is called...
Quoted 26 lines
Quoted 24 lines
Quoted 22 lines
Review class BackupTest(iotests.QMPTestCase): · line 213
Quoted 11 lines
So, with applied
(quoted diff) +1
-0@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(res, 'return', {}) self.qmp_backup_wait('drive0') + self.vm.shutdown() self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), 'target image does not match reference image')I’d personally prefer auto_dismiss=False and then block-job-dismiss. Although I can’t give a reason why.
Quoted 2 lines
In any case, thanks!
Max
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes
Quoted 122 lines · Vladimir Sementsov-Ogievskiy
Quoted 111 lines · Max Reitz
Quoted 110 lines · Vladimir Sementsov-Ogievskiy
Quoted 108 lines · Max Reitz
Quoted 103 lines · Vladimir Sementsov-Ogievskiy
Quoted 83 lines · Max Reitz
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The output differs from the reference output (what the source was before the guest writes).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.Signed-off-by Max Reitz <[email protected]>---
tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++--2 files changed, 36 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +2
-0@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img)def
hmp_io_writes(self, drive, patterns):
for pattern in patterns:tests/qemu-iotests/056 +0
-0@@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror)return False
+ deftest_overlapping_writes(self):
+ # Write something to back up
+self.hmp_io_writes('drive0', [('42', '0M', '2M')])tests/qemu-iotests/056 +29
-0+ + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') +def
test_dismiss_false(self):
res =self.vm.qmp('query-block-jobs')
self.assert_qmp(res, 'return', [])tests/qemu-iotests/056.out +1
-1@@ -1,5 +1,5 @@ -......... +..........----------------------------------------------------------------------
tests/qemu-iotests/056.out +1
-1-Ran 9 tests +Ran 10 testsQuoted 2 lines
OK
Failed for me:
-..........(quoted diff) +11
-0+qemu-img: Could not open '/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2': Failed to get shared "write" lock +Is another process using the image [/work/src/qemu/master/tests/qemu-iotests/scratch/dest.qcow2]? +......F... +====================================================================== +FAIL: test_overlapping_writes (__main__.BackupTest) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "056", line 212, in test_overlapping_writes + 'target image does not match reference image') +AssertionError: False is not true : target image does not match reference image +----------------------------------------------------------------------
Ran 10 testsReview
Quoted 2 lines
(quoted diff) +1
-1-OK +FAILED (failures=1)Hm. I hoped seeing
BLOCK_JOB_COMPLETEDwould be enough.Quoted 9 lines
Quoted 4 lines
The problem is higher: "Failed to get shared "write" lock". Because of this iotests.compare_images can't work. So, because of locking, we need to shutdown qemu before starting qemu-img. And it's done so in
test_complete_top()(I just looked at it as it's the only other user ofcompare_imagesin 056)The destination image shouldn’t be in use by qemu after the block job is done, though. Therefore, there shouldn’t be a lock issue. That’s what I meant.
Quoted 6 lines
Ah, understand. Hmm, then it's strange that I see that error..
Clearly, in
job_finalize_single(),job_clean()is called first, which should callbackup_cleanto unref target, thenjob_event_completed()is called...
But BlockJob.nodes are removed at very-end from block_job_free, so it seems to be another reference to target, which is still alive when event is sent.
Quoted 29 lines
Quoted 27 lines
Quoted 25 lines
Quoted 23 lines
Review class BackupTest(iotests.QMPTestCase): · line 213
Quoted 12 lines
So, with applied
(quoted diff) +1
-0@@ -207,6 +207,7 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(res, 'return', {}) self.qmp_backup_wait('drive0') + self.vm.shutdown() self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), 'target image does not match reference image')I’d personally prefer auto_dismiss=False and then block-job-dismiss. Although I can’t give a reason why.
Quoted 2 lines
In any case, thanks!
Max
Signature · 2 lines
Best regards, Vladimir
Quoted 8 lines · Max Reitz
Hi,
In a discussion with Vladimir today, we noticed that the backup job currently is pretty broken when using copy offloading. I don’t know about you, but my local filesystem (XFS) supports copy offloading, so the job uses it automatically. That means, that backup is broken and has been broken for a year on my local FS.
Quoted 7 lines
The last working version was 2.12, so this isn’t a regression in 4.1. We could thus justify moving it to 4.2. But I think this should really go into 4.1, because this is not really an edge case and as far as I know users cannot do anything to prevent the backup job from performing copy offloading if the system and all involved block drivers support it. I just wonder why nobody has noticed...
Agree. And if this goes into 4.1, I suggest to add [PATCH 2/3] block/backup: disable copy_range for compressed backup
as for now compressed backup just don't compress anything on FS with copy offloading supported.
Quoted 10 lines
Max Reitz (2):
backup: Copy only dirty areas
iotests: Test backup job with two guest writesblock/backup.c | 13 +++++++++++-- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 3 files changed, 47 insertions(+), 4 deletions(-)
Signature · 2 lines
Best regards, Vladimir