Skip to contentSkip to conversation

mirror: Make sure that source and target size match

PATCH In review Kevin Wolf 3 participants 20 messages May 11, 2020 – May 13, 2020 Block layer

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

v2 cover mirror: Make sure that source and target size match

Same thing as the recent fix for backup, except that mirror already forbids resizing during the job. So what remains is checking that the sizes match at the start of the job.

v2:
- Added patch 1 to fix a test that used different source/target size

Kevin Wolf (4):
iotests/109: Don't mirror with mismatched size iotests/229: Use blkdebug to inject an error
mirror: Make sure that source and target size match
iotests: Mirror with different source/target size

 block/mirror.c                   | 21 +++++----
 tests/qemu-iotests/041           | 45 +++++++++++++++++++
 tests/qemu-iotests/041.out       |  4 +-
 tests/qemu-iotests/109           | 10 ++---
 tests/qemu-iotests/109.out       | 74 +++++++++++++-------------------
 tests/qemu-iotests/229           | 15 +++++--
 tests/qemu-iotests/229.out       |  6 +--
 tests/qemu-iotests/common.filter |  5 +++
 8 files changed, 114 insertions(+), 66 deletions(-)
Signature · 1 line
2.25.3
author
#1 lore raw

PATCH v2 1/4 iotests/109: Don't mirror with mismatched size

This patch makes the raw image the same size as the file in a different format that is mirrored as raw to it to avoid errors when mirror starts to enforce that source and target are the same size.

We check only that the first 512 bytes are zeroed (instead of 64k) because some image formats create image files that are smaller than 64k, so trying to read 64k would result in I/O errors. Apart from this, 512 is more appropriate anyway because the raw format driver protects specifically the first 512 bytes.

---
 tests/qemu-iotests/109           | 10 ++---
 tests/qemu-iotests/109.out       | 74 +++++++++++++-------------------
 tests/qemu-iotests/common.filter |  5 +++
 3 files changed, 41 insertions(+), 48 deletions(-)
tests/qemu-iotests/109 +5 -5
index 5bc2e9b001..3ffeaf3c55 100755
@@ -77,14 +77,14 @@ for fmt in qcow qcow2 qed vdi vmdk vpc; do
     echo "=== Writing a $fmt header into raw ==="
     echo
 
-    _make_test_img 64M
     TEST_IMG="$TEST_IMG.src" IMGFMT=$fmt _make_test_img 64M
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size
 
     # This first test should fail: The image format was probed, we may not
     # write an image header at the start of the image
     run_qemu "$TEST_IMG" "$TEST_IMG.src" "" "BLOCK_JOB_ERROR" |
         _filter_block_job_len
-    $QEMU_IO -c 'read -P 0 0 64k' "$TEST_IMG" | _filter_qemu_io
+    $QEMU_IO -c 'read -P 0 0 512' "$TEST_IMG" | _filter_qemu_io
 
 
     # When raw was explicitly specified, the same must succeed
@@ -103,12 +103,12 @@ for sample_img in empty.bochs iotest-dirtylog-10G-4M.vhdx parallels-v1 \
 
     # Can't use _use_sample_img because that isn't designed to be used multiple
     # times and it overwrites $TEST_IMG (both breaks cleanup)
-    _make_test_img 64M
     bzcat "$SAMPLE_IMG_DIR/$sample_img.bz2" > "$TEST_IMG.src"
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size
 
     run_qemu "$TEST_IMG" "$TEST_IMG.src" "" "BLOCK_JOB_ERROR" |
         _filter_block_job_offset | _filter_block_job_len
-    $QEMU_IO -c 'read -P 0 0 64k' "$TEST_IMG" | _filter_qemu_io
+    $QEMU_IO -c 'read -P 0 0 512' "$TEST_IMG" | _filter_qemu_io
 
     run_qemu "$TEST_IMG" "$TEST_IMG.src" "'format': 'raw'," "BLOCK_JOB_READY"
     $QEMU_IMG compare -f raw -F raw "$TEST_IMG" "$TEST_IMG.src"
@@ -119,8 +119,8 @@ echo "=== Write legitimate MBR into raw ==="
 echo
 
 for sample_img in grub_mbr.raw; do
-    _make_test_img 64M
     bzcat "$SAMPLE_IMG_DIR/$sample_img.bz2" > "$TEST_IMG.src"
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size
 
     run_qemu "$TEST_IMG" "$TEST_IMG.src" "" "BLOCK_JOB_READY"
     $QEMU_IMG compare -f raw -F raw "$TEST_IMG" "$TEST_IMG.src"
tests/qemu-iotests/109.out +31 -43

257 lines not on this page · raw message

tests/qemu-iotests/common.filter +5 -0
index 3f8ee3e5f7..e4dcf5e6e7 100644
@@ -155,6 +155,11 @@ _filter_img_create()
         -e "s# force_size=\\(on\\|off\\)##g"
 }
 
+_filter_img_create_size()
+{
+    $SED -e "s# size=[0-9]\\+# size=SIZE#g"
+}
+
 _filter_img_info()
 {
     if [[ "$1" == "--format-specific" ]]; then
Signature · 1 line
2.25.3
#2 lore raw

Re: [PATCH v2 1/4] iotests/109: Don't mirror with mismatched size

Quoted 22 lines · Kevin Wolf

This patch makes the raw image the same size as the file in a different format that is mirrored as raw to it to avoid errors when mirror starts to enforce that source and target are the same size.

We check only that the first 512 bytes are zeroed (instead of 64k) because some image formats create image files that are smaller than 64k, so trying to read 64k would result in I/O errors. Apart from this, 512 is more appropriate anyway because the raw format driver protects specifically the first 512 bytes.

---
 tests/qemu-iotests/109           | 10 ++---
 tests/qemu-iotests/109.out       | 74 +++++++++++++-------------------
 tests/qemu-iotests/common.filter |  5 +++
 3 files changed, 41 insertions(+), 48 deletions(-)
tests/qemu-iotests/109 +0 -0
index 5bc2e9b001..3ffeaf3c55 100755

tests/qemu-iotests/109 for fmt in qcow qcow2 qed vdi vmdk vpc; do · line 81

Quoted 6 lines
tests/qemu-iotests/109 +1 -1
@@ -77,14 +77,14 @@ for fmt in qcow qcow2 qed vdi vmdk vpc; do
     echo "=== Writing a $fmt header into raw ==="
     echo
 
-    _make_test_img 64M
     TEST_IMG="$TEST_IMG.src" IMGFMT=$fmt _make_test_img 64M
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size

Why du and not the file length (stat -c '%s')?

Max

author
#3 lore raw

Re: [PATCH v2 1/4] iotests/109: Don't mirror with mismatched size

Am 11.05.2020 um 17:08 hat Max Reitz geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 25 lines

This patch makes the raw image the same size as the file in a different format that is mirrored as raw to it to avoid errors when mirror starts to enforce that source and target are the same size.

We check only that the first 512 bytes are zeroed (instead of 64k) because some image formats create image files that are smaller than 64k, so trying to read 64k would result in I/O errors. Apart from this, 512 is more appropriate anyway because the raw format driver protects specifically the first 512 bytes.

---
 tests/qemu-iotests/109           | 10 ++---
 tests/qemu-iotests/109.out       | 74 +++++++++++++-------------------
 tests/qemu-iotests/common.filter |  5 +++
 3 files changed, 41 insertions(+), 48 deletions(-)
tests/qemu-iotests/109 +0 -0
index 5bc2e9b001..3ffeaf3c55 100755
@@ -77,14 +77,14 @@ for fmt in qcow qcow2 qed vdi vmdk vpc; do
     echo "=== Writing a $fmt header into raw ==="
     echo
 
Quoted 5 lines

tests/qemu-iotests/109 for fmt in qcow qcow2 qed vdi vmdk vpc; do · line 81

Quoted 3 lines
tests/qemu-iotests/109 +1 -1
-    _make_test_img 64M
     TEST_IMG="$TEST_IMG.src" IMGFMT=$fmt _make_test_img 64M
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size

Why du and not the file length (stat -c '%s')?

Because the test from which I copied had 'du' and the internet claimed that 'stat -c' isn't portable. Now I see that we do use it in other test cases, so I guess it would have been fine, too. Is there a good reason why 'stat' would be better?

Kevin

Reviewed-by
#4 lore raw

Re: [PATCH v2 1/4] iotests/109: Don't mirror with mismatched size

Quoted 33 lines · Kevin Wolf

Am 11.05.2020 um 17:08 hat Max Reitz geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 25 lines

This patch makes the raw image the same size as the file in a different format that is mirrored as raw to it to avoid errors when mirror starts to enforce that source and target are the same size.

We check only that the first 512 bytes are zeroed (instead of 64k) because some image formats create image files that are smaller than 64k, so trying to read 64k would result in I/O errors. Apart from this, 512 is more appropriate anyway because the raw format driver protects specifically the first 512 bytes.

---
 tests/qemu-iotests/109           | 10 ++---
 tests/qemu-iotests/109.out       | 74 +++++++++++++-------------------
 tests/qemu-iotests/common.filter |  5 +++
 3 files changed, 41 insertions(+), 48 deletions(-)
tests/qemu-iotests/109 +0 -0
index 5bc2e9b001..3ffeaf3c55 100755
@@ -77,14 +77,14 @@ for fmt in qcow qcow2 qed vdi vmdk vpc; do
     echo "=== Writing a $fmt header into raw ==="
     echo
 
Quoted 5 lines

tests/qemu-iotests/109 for fmt in qcow qcow2 qed vdi vmdk vpc; do · line 81

Quoted 3 lines
tests/qemu-iotests/109 +1 -1
-    _make_test_img 64M
     TEST_IMG="$TEST_IMG.src" IMGFMT=$fmt _make_test_img 64M
+    _make_test_img $(du -b "$TEST_IMG.src" | cut -f1) | _filter_img_create_size

Why du and not the file length (stat -c '%s')?

Quoted 4 lines

Because the test from which I copied had 'du' and the internet claimed that 'stat -c' isn't portable. Now I see that we do use it in other test cases, so I guess it would have been fine, too. Is there a good reason why 'stat' would be better?

Oh, I didn’t know that du -b reports the file length. Well, then that works, too. (I’ve never seen du used for anything but getting the, well, disk usage.)

(I figured -b would just report the size in bytes.)

Then:

Reviewed-by on v2 1/4
#5 lore raw

Re: [PATCH v2 1/4] iotests/109: Don't mirror with mismatched size

In reply to Kevin Wolf
Quoted 11 lines · Kevin Wolf

This patch makes the raw image the same size as the file in a different format that is mirrored as raw to it to avoid errors when mirror starts to enforce that source and target are the same size.

We check only that the first 512 bytes are zeroed (instead of 64k) because some image formats create image files that are smaller than 64k, so trying to read 64k would result in I/O errors. Apart from this, 512 is more appropriate anyway because the raw format driver protects specifically the first 512 bytes.

Signature · 2 lines
Best regards,
Vladimir
author
#6 lore raw

PATCH v2 2/4 iotests/229: Use blkdebug to inject an error

In reply to Kevin Wolf

229 relies on the mirror running into an I/O error when the target is smaller than the source. After changing mirror to catch this condition while starting the job, this test case won't get a job that is paused for an I/O error any more. Use blkdebug instead to inject an error.

Message-Id: <[email protected]>

---
 tests/qemu-iotests/229     | 15 +++++++++++----
 tests/qemu-iotests/229.out |  6 +++---
 2 files changed, 14 insertions(+), 7 deletions(-)
tests/qemu-iotests/229 +11 -4
index 866168b236..a5d4e5d4f2 100755
@@ -33,6 +33,7 @@ _cleanup()
     _cleanup_test_img
     _rm_test_img "$TEST_IMG"
     _rm_test_img "$DEST_IMG"
+    rm -f "$TEST_DIR/blkdebug.conf"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
@@ -49,11 +50,10 @@ _supported_os Linux
 
 DEST_IMG="$TEST_DIR/d.$IMGFMT"
 TEST_IMG="$TEST_DIR/b.$IMGFMT"
+BLKDEBUG_CONF="$TEST_DIR/blkdebug.conf"
 
 _make_test_img 2M
-
-# destination for mirror will be too small, causing error
-TEST_IMG=$DEST_IMG _make_test_img 1M
+TEST_IMG=$DEST_IMG _make_test_img 2M
 
 $QEMU_IO -c 'write 0 2M' "$TEST_IMG" | _filter_qemu_io
 
@@ -67,11 +67,18 @@ echo
 echo '=== Starting drive-mirror, causing error & stop  ==='
 echo
 
+cat > "$BLKDEBUG_CONF" <<EOF
+[inject-error]
+event = "write_aio"
+errno = "5"
+once = "on"
+EOF
+
 _send_qemu_cmd $QEMU_HANDLE \
     "{'execute': 'drive-mirror',
                  'arguments': {'device': 'testdisk',
                                'format': '$IMGFMT',
-                               'target': '$DEST_IMG',
+                               'target': 'blkdebug:$BLKDEBUG_CONF:$DEST_IMG',
                                'sync':   'full',
                                'mode':   'existing',
                                'on-source-error': 'stop',
tests/qemu-iotests/229.out +3 -3
index 22350d75d7..fdcd249642 100644
@@ -1,6 +1,6 @@
 QA output created by 229
 Formatting 'TEST_DIR/b.IMGFMT', fmt=IMGFMT size=2097152
-Formatting 'TEST_DIR/d.IMGFMT', fmt=IMGFMT size=1048576
+Formatting 'TEST_DIR/d.IMGFMT', fmt=IMGFMT size=2097152
 wrote 2097152/2097152 bytes at offset 0
 2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {'execute': 'qmp_capabilities'}
@@ -8,7 +8,7 @@ wrote 2097152/2097152 bytes at offset 0
 
 === Starting drive-mirror, causing error & stop  ===
 
-{'execute': 'drive-mirror', 'arguments': {'device': 'testdisk', 'format': 'IMGFMT', 'target': 'TEST_DIR/d.IMGFMT', 'sync': 'full', 'mode': 'existing', 'on-source-error': 'stop', 'on-target-error': 'stop' }}
+{'execute': 'drive-mirror', 'arguments': {'device': 'testdisk', 'format': 'IMGFMT', 'target': 'blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/d.IMGFMT', 'sync': 'full', 'mode': 'existing', 'on-source-error': 'stop', 'on-target-error': 'stop' }}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "testdisk"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "testdisk"}}
 {"return": {}}
@@ -21,5 +21,5 @@ wrote 2097152/2097152 bytes at offset 0
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "testdisk"}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "testdisk"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "testdisk", "len": 2097152, "offset": 1048576, "speed": 0, "type": "mirror"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "testdisk", "len": 2097152, "offset": 2097152, "speed": 0, "type": "mirror"}}
 *** done
Signature · 1 line
2.25.3
Reviewed-by on v2 2/4
#7 lore raw

Re: [PATCH v2 2/4] iotests/229: Use blkdebug to inject an error

Quoted 9 lines · Kevin Wolf

229 relies on the mirror running into an I/O error when the target is smaller than the source. After changing mirror to catch this condition while starting the job, this test case won't get a job that is paused for an I/O error any more. Use blkdebug instead to inject an error.

Message-Id: <[email protected]>

Did you do this on purpose, or did you send the patch mail this way accidentally (with Message-ID tag and double S-o-b)?

(Because I’ve never seen anyone else do it)

Quoted 4 lines
---
 tests/qemu-iotests/229     | 15 +++++++++++----
 tests/qemu-iotests/229.out |  6 +++---
 2 files changed, 14 insertions(+), 7 deletions(-)
author
#8 lore raw

Re: [PATCH v2 2/4] iotests/229: Use blkdebug to inject an error

Am 11.05.2020 um 17:18 hat Max Reitz geschrieben:

Quoted 6 lines · Kevin Wolf
Quoted 5 lines

229 relies on the mirror running into an I/O error when the target is smaller than the source. After changing mirror to catch this condition while starting the job, this test case won't get a job that is paused for an I/O error any more. Use blkdebug instead to inject an error.

Quoted 7 lines
Quoted 4 lines

Message-Id: <[email protected]>

Did you do this on purpose, or did you send the patch mail this way accidentally (with Message-ID tag and double S-o-b)?

It's an artifact on applying my own patch from the list with Message-Id tags (as we're asked to do), rebasing my development branch (which got rid of the original version of the commit message) and then noticing in the last minute that it doesn't pass iotests.

Kevin

Reviewed-by on v2 2/4
#9 lore raw

Re: [PATCH v2 2/4] iotests/229: Use blkdebug to inject an error

In reply to Kevin Wolf
Quoted 9 lines · Kevin Wolf

229 relies on the mirror running into an I/O error when the target is smaller than the source. After changing mirror to catch this condition while starting the job, this test case won't get a job that is paused for an I/O error any more. Use blkdebug instead to inject an error.

Message-Id:<[email protected]>

Signature · 2 lines
Best regards,
Vladimir
author
#10 lore raw

PATCH v2 3/4 mirror: Make sure that source and target size match

In reply to Kevin Wolf

If the target is shorter than the source, mirror would copy data until it reaches the end of the target and then fail with an I/O error when trying to write past the end.

If the target is longer than the source, the mirror job would complete successfully, but the target wouldn't actually be an accurate copy of the source image (it would contain some additional garbage at the end).

Fix this by checking that both images have the same size when the job starts.

Message-Id: <[email protected]>

---
 block/mirror.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
block/mirror.c +12 -9
index aca95c9bc9..201ffa26f9 100644
@@ -872,6 +872,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
     BlockDriverState *target_bs = blk_bs(s->target);
     bool need_drain = true;
     int64_t length;
+    int64_t target_length;
     BlockDriverInfo bdi;
     char backing_filename[2]; /* we only need 2 characters because we are only
                                  checking for a NULL string */
@@ -887,24 +888,26 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
         goto immediate_exit;
     }
 
+    target_length = blk_getlength(s->target);
+    if (target_length < 0) {
+        ret = target_length;
+        goto immediate_exit;
+    }
+
     /* Active commit must resize the base image if its size differs from the
      * active layer. */
     if (s->base == blk_bs(s->target)) {
-        int64_t base_length;
-
-        base_length = blk_getlength(s->target);
-        if (base_length < 0) {
-            ret = base_length;
-            goto immediate_exit;
-        }
-
-        if (s->bdev_length > base_length) {
+        if (s->bdev_length > target_length) {
             ret = blk_truncate(s->target, s->bdev_length, false,
                                PREALLOC_MODE_OFF, 0, NULL);
             if (ret < 0) {
                 goto immediate_exit;
             }
         }
+    } else if (s->bdev_length != target_length) {
+        error_setg(errp, "Source and target image have different sizes");
+        ret = -EINVAL;
+        goto immediate_exit;
     }
 
     if (s->bdev_length == 0) {
Signature · 1 line
2.25.3
Reviewed-by on v2 3/4
#11 lore raw
Quoted 11 lines · Kevin Wolf

If the target is shorter than the source, mirror would copy data until it reaches the end of the target and then fail with an I/O error when trying to write past the end.

If the target is longer than the source, the mirror job would complete successfully, but the target wouldn't actually be an accurate copy of the source image (it would contain some additional garbage at the end).

Fix this by checking that both images have the same size when the job starts.

Quoted 7 lines

Message-Id: <[email protected]>

---
 block/mirror.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
#12 lore raw
In reply to Kevin Wolf
Quoted 50 lines · Kevin Wolf

If the target is shorter than the source, mirror would copy data until it reaches the end of the target and then fail with an I/O error when trying to write past the end.

If the target is longer than the source, the mirror job would complete successfully, but the target wouldn't actually be an accurate copy of the source image (it would contain some additional garbage at the end).

Fix this by checking that both images have the same size when the job starts.

Message-Id: <[email protected]>

---
  block/mirror.c | 21 ++++++++++++---------
  1 file changed, 12 insertions(+), 9 deletions(-)
block/mirror.c +7 -6
index aca95c9bc9..201ffa26f9 100644
@@ -872,6 +872,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
      BlockDriverState *target_bs = blk_bs(s->target);
      bool need_drain = true;
      int64_t length;
+    int64_t target_length;
      BlockDriverInfo bdi;
      char backing_filename[2]; /* we only need 2 characters because we are only
                                   checking for a NULL string */
@@ -887,24 +888,26 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
          goto immediate_exit;
      }
 
+    target_length = blk_getlength(s->target);
+    if (target_length < 0) {
+        ret = target_length;
+        goto immediate_exit;
+    }
+
      /* Active commit must resize the base image if its size differs from the
       * active layer. */
      if (s->base == blk_bs(s->target)) {
-        int64_t base_length;
-
-        base_length = blk_getlength(s->target);
-        if (base_length < 0) {
-            ret = base_length;
-            goto immediate_exit;

block/mirror.c static int coroutine_fn mirror_run(Job *job, Error **errp) · line 906

Quoted 10 lines
block/mirror.c +1 -3
-        }
-
-        if (s->bdev_length > base_length) {
+        if (s->bdev_length > target_length) {
              ret = blk_truncate(s->target, s->bdev_length, false,
                                 PREALLOC_MODE_OFF, 0, NULL);
              if (ret < 0) {
                  goto immediate_exit;
              }
          }

Hmm, interesting, if base is larger, is our behavior correct? Blockdev becomes larger after commit and old data becomes available? I think we should zero the tail after old EOF or shrink the target..

block/mirror.c static int coroutine_fn mirror_run(Job *job, Error **errp) · line 909

Quoted 3 lines
block/mirror.c +3 -0
+    } else if (s->bdev_length != target_length) {
+        error_setg(errp, "Source and target image have different sizes");
+        ret = -EINVAL;

Seems, the only case, when mirror_run() sets errp. And, therefore, the only correct one..

Quoted 5 lines
block/mirror.c +1 -0
+        goto immediate_exit;
      }
 
      if (s->bdev_length == 0) {
Signature · 2 lines
Best regards,
Vladimir
#13 lore raw
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines

+    } else if (s->bdev_length != target_length) {
+        error_setg(errp, "Source and target image have different sizes"); +        ret = -EINVAL;

Seems, the only case, when mirror_run() sets errp. And, therefore, the only correct one..

the only one failure case I mean, of course.

Signature · 2 lines
Best regards,
Vladimir
author
#14 lore raw
In reply to Vladimir Sementsov-Ogievskiy

Am 12.05.2020 um 19:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 56 lines
Quoted 55 lines · Kevin Wolf

If the target is shorter than the source, mirror would copy data until it reaches the end of the target and then fail with an I/O error when trying to write past the end.

If the target is longer than the source, the mirror job would complete successfully, but the target wouldn't actually be an accurate copy of the source image (it would contain some additional garbage at the end).

Fix this by checking that both images have the same size when the job starts.

Message-Id: <[email protected]>

---
  block/mirror.c | 21 ++++++++++++---------
  1 file changed, 12 insertions(+), 9 deletions(-)
block/mirror.c +8 -9
index aca95c9bc9..201ffa26f9 100644
@@ -872,6 +872,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
      BlockDriverState *target_bs = blk_bs(s->target);
      bool need_drain = true;
      int64_t length;
+    int64_t target_length;
      BlockDriverInfo bdi;
      char backing_filename[2]; /* we only need 2 characters because we are only
                                   checking for a NULL string */
@@ -887,24 +888,26 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
          goto immediate_exit;
      }
+    target_length = blk_getlength(s->target);
+    if (target_length < 0) {
+        ret = target_length;
+        goto immediate_exit;
+    }
+
      /* Active commit must resize the base image if its size differs from the
       * active layer. */
      if (s->base == blk_bs(s->target)) {
-        int64_t base_length;
-
-        base_length = blk_getlength(s->target);
-        if (base_length < 0) {
-            ret = base_length;
-            goto immediate_exit;
-        }
-
-        if (s->bdev_length > base_length) {
+        if (s->bdev_length > target_length) {
              ret = blk_truncate(s->target, s->bdev_length, false,
                                 PREALLOC_MODE_OFF, 0, NULL);
Quoted 8 lines

block/mirror.c static int coroutine_fn mirror_run(Job *job, Error **errp) · line 906

Quoted 4 lines
block/mirror.c +0 -0
              if (ret < 0) {
                  goto immediate_exit;
              }
          }

Hmm, interesting, if base is larger, is our behavior correct? Blockdev becomes larger after commit and old data becomes available? I think we should zero the tail after old EOF or shrink the target..

Yes, I agree, we should shrink it. But active commit is a different case than what I'm fixing in this patch, so I left it unmodified. We'll probably need a third series for commit after backup and mirror.

Quoted 6 lines

block/mirror.c static int coroutine_fn mirror_run(Job *job, Error **errp) · line 909

Quoted 3 lines
block/mirror.c +3 -0
+    } else if (s->bdev_length != target_length) {
+        error_setg(errp, "Source and target image have different sizes");
+        ret = -EINVAL;

Seems, the only case, when mirror_run() sets errp. And, therefore, the only correct one..

job_update_rc() takes care to fill job->err (with strerror()) if it hasn't been set yet, so the other places aren't strictly wrong, but probably provide suboptimal error messages.

Kevin

#15 lore raw
Quoted 74 lines · Kevin Wolf

Am 12.05.2020 um 19:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 56 lines
Quoted 55 lines · Kevin Wolf

If the target is shorter than the source, mirror would copy data until it reaches the end of the target and then fail with an I/O error when trying to write past the end.

If the target is longer than the source, the mirror job would complete successfully, but the target wouldn't actually be an accurate copy of the source image (it would contain some additional garbage at the end).

Fix this by checking that both images have the same size when the job starts.

Message-Id: <[email protected]>

---
   block/mirror.c | 21 ++++++++++++---------
   1 file changed, 12 insertions(+), 9 deletions(-)
block/mirror.c +8 -9
index aca95c9bc9..201ffa26f9 100644
@@ -872,6 +872,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
       BlockDriverState *target_bs = blk_bs(s->target);
       bool need_drain = true;
       int64_t length;
+    int64_t target_length;
       BlockDriverInfo bdi;
       char backing_filename[2]; /* we only need 2 characters because we are only
                                    checking for a NULL string */
@@ -887,24 +888,26 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
           goto immediate_exit;
       }
+    target_length = blk_getlength(s->target);
+    if (target_length < 0) {
+        ret = target_length;
+        goto immediate_exit;
+    }
+
       /* Active commit must resize the base image if its size differs from the
        * active layer. */
       if (s->base == blk_bs(s->target)) {
-        int64_t base_length;
-
-        base_length = blk_getlength(s->target);
-        if (base_length < 0) {
-            ret = base_length;
-            goto immediate_exit;
-        }
-
-        if (s->bdev_length > base_length) {
+        if (s->bdev_length > target_length) {
               ret = blk_truncate(s->target, s->bdev_length, false,
                                  PREALLOC_MODE_OFF, 0, NULL);
Quoted 8 lines

block/mirror.c static int coroutine_fn mirror_run(Job *job, Error **errp) · line 906

Quoted 4 lines
block/mirror.c +0 -0
               if (ret < 0) {
                   goto immediate_exit;
               }
           }

Hmm, interesting, if base is larger, is our behavior correct? Blockdev becomes larger after commit and old data becomes available? I think we should zero the tail after old EOF or shrink the target..

Yes, I agree, we should shrink it. But active commit is a different case than what I'm fixing in this patch, so I left it unmodified. We'll probably need a third series for commit after backup and mirror.

Quoted 4 lines
Quoted 3 lines
block/mirror.c +3 -0
+    } else if (s->bdev_length != target_length) {
+        error_setg(errp, "Source and target image have different sizes");
+        ret = -EINVAL;
Quoted 7 lines
Quoted 2 lines

Seems, the only case, when mirror_run() sets errp. And, therefore, the only correct one..

job_update_rc() takes care to fill job->err (with strerror()) if it hasn't been set yet, so the other places aren't strictly wrong, but probably provide suboptimal error messages.

Hmm. but I failed to find, where job->err is reported except for job_query_single(), which doesn't call job_update_rc().

block_job_event_completed() doesn't use job->err, but instead create a message using strerror(-job->job.ret).

Interesting also, that job_finish_sync may return error, not setting errp.. Still except for tests, it should influence only run_block_job() from qemu-img, which itself doesn't care too much about setting errp on failure, so it's broken anyway.

OK, seems this all is not very related to the series:

Signature · 2 lines
Best regards,
Vladimir
author
#16 lore raw

PATCH v2 4/4 iotests: Mirror with different source/target size

In reply to Kevin Wolf

This tests that the mirror job catches situations where the target node has a different size than the source node. It must also forbid resize operations when the job is already running.

Message-Id: <[email protected]>

---
 tests/qemu-iotests/041     | 45 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/041.out |  4 ++--
 2 files changed, 47 insertions(+), 2 deletions(-)
tests/qemu-iotests/041 +45 -0
index 1812dd8479..601c756117 100755
@@ -240,6 +240,49 @@ class TestSingleBlockdev(TestSingleDrive):
                              target=self.qmp_target)
         self.assert_qmp(result, 'error/class', 'GenericError')
 
+    def do_test_resize(self, device, node):
+        def pre_finalize():
+            if device:
+                result = self.vm.qmp('block_resize', device=device, size=65536)
+                self.assert_qmp(result, 'error/class', 'GenericError')
+
+            result = self.vm.qmp('block_resize', node_name=node, size=65536)
+            self.assert_qmp(result, 'error/class', 'GenericError')
+
+        result = self.vm.qmp(self.qmp_cmd, job_id='job0', device='drive0',
+                             sync='full', target=self.qmp_target,
+                             auto_finalize=False, auto_dismiss=False)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.run_job('job0', auto_finalize=False,
+                                 pre_finalize=pre_finalize)
+        self.assertEqual(result, None)
+
+    def test_source_resize(self):
+        self.do_test_resize('drive0', 'top')
+
+    def test_target_resize(self):
+        self.do_test_resize(None, self.qmp_target)
+
+    def do_test_target_size(self, size):
+        result = self.vm.qmp('block_resize', node_name=self.qmp_target,
+                             size=size)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp(self.qmp_cmd, job_id='job0',
+                             device='drive0', sync='full', auto_dismiss=False,
+                             target=self.qmp_target)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.run_job('job0')
+        self.assertEqual(result, 'Source and target image have different sizes')
+
+    def test_small_target(self):
+        self.do_test_target_size(self.image_len // 2)
+
+    def test_large_target(self):
+        self.do_test_target_size(self.image_len * 2)
+
     test_large_cluster = None
     test_image_not_found = None
     test_small_buffer2 = None
@@ -251,6 +294,8 @@ class TestSingleDriveZeroLength(TestSingleDrive):
 
 class TestSingleBlockdevZeroLength(TestSingleBlockdev):
     image_len = 0
+    test_small_target = None
+    test_large_target = None
 
 class TestSingleDriveUnalignedLength(TestSingleDrive):
     image_len = 1025 * 1024
tests/qemu-iotests/041.out +2 -2
index 877b76fd31..53abe11d73 100644
@@ -1,5 +1,5 @@
-..............................................................................................
+........................................................................................................
 ----------------------------------------------------------------------
-Ran 94 tests
+Ran 104 tests
 
 OK
Signature · 1 line
2.25.3
Reviewed-by on v2 4/4
#17 lore raw

Re: [PATCH v2 4/4] iotests: Mirror with different source/target size

Quoted 4 lines · Kevin Wolf

This tests that the mirror job catches situations where the target node has a different size than the source node. It must also forbid resize operations when the job is already running.

Quoted 8 lines

Message-Id: <[email protected]>

---
 tests/qemu-iotests/041     | 45 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/041.out |  4 ++--
 2 files changed, 47 insertions(+), 2 deletions(-)
Reviewed-by on v2 4/4
#18 lore raw

Re: [PATCH v2 4/4] iotests: Mirror with different source/target size

In reply to Kevin Wolf
Quoted 8 lines · Kevin Wolf

This tests that the mirror job catches situations where the target node has a different size than the source node. It must also forbid resize operations when the job is already running.

Message-Id: <[email protected]>

tests/qemu-iotests/041 class TestSingleBlockdev(TestSingleDrive):

Quoted 10 lines
---
  tests/qemu-iotests/041     | 45 ++++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/041.out |  4 ++--
  2 files changed, 47 insertions(+), 2 deletions(-)
tests/qemu-iotests/041 +0 -0
index 1812dd8479..601c756117 100755
@@ -240,6 +240,49 @@ class TestSingleBlockdev(TestSingleDrive):

Hmm, probably resize tests would be good in the TestSingleDrive, to cover drive-mirror too. Still, there shouldn't be any difference.. So, I don't think it worth doing.

Quoted 5 lines
tests/qemu-iotests/041 +2 -0
                               target=self.qmp_target)
          self.assert_qmp(result, 'error/class', 'GenericError')
 
+    def do_test_resize(self, device, node):
+        def pre_finalize():

[..]

tests/qemu-iotests/041 class TestSingleBlockdev(TestSingleDrive): · line 278

Quoted 12 lines
tests/qemu-iotests/041 +12 -0
+    def do_test_target_size(self, size):
+        result = self.vm.qmp('block_resize', node_name=self.qmp_target,
+                             size=size)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp(self.qmp_cmd, job_id='job0',
+                             device='drive0', sync='full', auto_dismiss=False,
+                             target=self.qmp_target)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.run_job('job0')
+        self.assertEqual(result, 'Source and target image have different sizes')

Hmm, and this proves, that we are not very good with handling early errors. A lot better would be to fail earlier, on blockdev-mirror command. And, as shown by your previous series, backup works better in this case. But again, it's for another series.

Signature · 2 lines
Best regards,
Vladimir
author
#19 lore raw

Re: [PATCH v2 4/4] iotests: Mirror with different source/target size

Am 13.05.2020 um 13:17 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 17 lines
Quoted 8 lines · Kevin Wolf

This tests that the mirror job catches situations where the target node has a different size than the source node. It must also forbid resize operations when the job is already running.

Message-Id: <[email protected]>

Quoted 5 lines
---
  tests/qemu-iotests/041     | 45 ++++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/041.out |  4 ++--
  2 files changed, 47 insertions(+), 2 deletions(-)
Quoted 10 lines

tests/qemu-iotests/041 class TestSingleBlockdev(TestSingleDrive):

Quoted 5 lines
tests/qemu-iotests/041 +0 -0
index 1812dd8479..601c756117 100755
@@ -240,6 +240,49 @@ class TestSingleBlockdev(TestSingleDrive):

Hmm, probably resize tests would be good in the TestSingleDrive, to cover drive-mirror too. Still, there shouldn't be any difference.. So, I don't think it worth doing.

At first, I tried to do a single implementation that works for both, but it wasn't easily possible. I'm not sure any more if it was the only problem, but at least drive-mirror targets don't have a known node-name, so we would have to guess it from query-named-block-nodes. Not impossible, but maybe not worth it...

Quoted 16 lines
Quoted 4 lines
tests/qemu-iotests/041 +2 -0
                               target=self.qmp_target)
          self.assert_qmp(result, 'error/class', 'GenericError')
+    def do_test_resize(self, device, node):
+        def pre_finalize():

[..]

Quoted 9 lines
tests/qemu-iotests/041 +9 -0
+    def do_test_target_size(self, size):
+        result = self.vm.qmp('block_resize', node_name=self.qmp_target,
+                             size=size)
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp(self.qmp_cmd, job_id='job0',
+                             device='drive0', sync='full', auto_dismiss=False,
+                             target=self.qmp_target)
+        self.assert_qmp(result, 'return', {})
Quoted 8 lines

tests/qemu-iotests/041 class TestSingleBlockdev(TestSingleDrive): · line 278

Quoted 3 lines
tests/qemu-iotests/041 +3 -0
+
+        result = self.vm.run_job('job0')
+        self.assertEqual(result, 'Source and target image have different sizes')

Hmm, and this proves, that we are not very good with handling early errors. A lot better would be to fail earlier, on blockdev-mirror command. And, as shown by your previous series, backup works better in this case. But again, it's for another series.

At least it shows that errp does make it to the client. :-)

I'm not sure which way is currently better. QMP handlers are still synchronous code that blocks the rest of QEMU, so doing things only inside the coroutine does have some benefits. Maybe with coroutine QMP handlers this will become less of a concern, though.

Kevin