Skip to contentSkip to conversation

qemu-img: Add salvaging mode to convert

PATCH Applied Max Reitz 2 participants 11 messages May 7, 2019 – Jun 13, 2019 Block layer

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v4 cover qemu-img: Add salvaging mode to convert

Hi,

This series adds a --salvage option to qemu-img convert. With this, qemu-img will not abort when it encounters an I/O error. Instead, it tries to narrow it down and will treat the affected sectors as being completely 0 (and print a warning).

Testing this is not so easy, because while real I/O errors during read operations should be treated as described above, errors encountered during bdrv_block_status() should just be ignored and the affected sectors should be considered allocated. But blkdebug does not yet have a way to intercept this, and:

(1) Just adding a new block-status event would be silly, because I don't want an event, I want it to fail on a certain kind of operation, on a certain sector range, independently of any events, so why can't we just do that? See patch 4.

(2) If we just make blkdebug intercept .bdrv_co_block_status() like all other kinds of operations, at least iotest 041 fails, which does exactly that silly thing: It uses the read_aio event to wait for any read. But it turns out that there may be a bdrv_*block_status() call in between, so suddenly the wrong operation yields an error. As I said, the real fault here is that it does not really make sense to pray that the operation you want to fail is the one that is immediately executed after some event that you hope will trigger that operation.
See patch 3.

So patch 3 allows blkdebug users to select which kind of I/O operation they actually want to make fail, and patch 4 allows them to not use any event, but to have a rule active all the time.

Together, we can then enable error injection for block-status in patch 5 and make use of event=none iotype=block-status in patch 6.

v4:
- Patch 2: Rebase conflict (265a7e54efad246f4bde63f77aee6311f436f5f1)
- Patch 3: Fix enum prefix to be BLKDEBUG_IO_TYPE [Vladimir]
- Patch 5: The same
- Patch 6:
- 249 -> 251
- s%#!/bin/bash%#!/usr/bin/env bash% [Thomas]
- s/2018/2019/ [Vladimir]
- Drop here=$PWD [Vladimir]
- Clarified a comment (I hope) [Vladimir]

git-backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/7:[----] [--] 'qemu-img: Move quiet into ImgConvertState' 002/7:[0006] [FC] 'qemu-img: Add salvaging mode to convert' 003/7:[0028] [FC] 'blkdebug: Add @iotype error option' 004/7:[----] [--] 'blkdebug: Add "none" event'
005/7:[0002] [FC] 'blkdebug: Inject errors on .bdrv_co_block_status()' 006/7:[0015] [FC] 'iotests: Test qemu-img convert --salvage' 007/7:[----] [--] 'iotests: Test qemu-img convert -C --salvage'

Max Reitz (7):
qemu-img: Move quiet into ImgConvertState
qemu-img: Add salvaging mode to convert
blkdebug: Add @iotype error option
blkdebug: Add "none" event
blkdebug: Inject errors on .bdrv_co_block_status()
iotests: Test qemu-img convert --salvage
iotests: Test qemu-img convert -C --salvage

 qapi/block-core.json       |  33 +++++++-
 block/blkdebug.c           |  60 +++++++++++---
 qemu-img.c                 | 103 ++++++++++++++++-------
 qemu-img-cmds.hx           |   4 +-
 qemu-img.texi              |   4 +
 tests/qemu-iotests/082     |   1 +
 tests/qemu-iotests/082.out |   3 +
 tests/qemu-iotests/251     | 162 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out |  43 ++++++++++
 tests/qemu-iotests/group   |   1 +
 10 files changed, 375 insertions(+), 39 deletions(-)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v4 1/7 qemu-img: Move quiet into ImgConvertState

Move img_convert()'s quiet flag into the ImgConvertState so it is accessible by nested functions. -q dictates that it suppresses anything but errors, so if those functions want to emit warnings, they need to query this flag first. (There currently are no such warnings, but there will be as of the next patch.)

---
 qemu-img.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
qemu-img.c +7 -6
index e6ad5978e0..f606675933 100644
@@ -1578,6 +1578,7 @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool quiet;
     int min_sparse;
     int alignment;
     size_t cluster_sectors;
@@ -2009,7 +2010,7 @@ static int img_convert(int argc, char **argv)
     QDict *open_opts = NULL;
     char *options = NULL;
     Error *local_err = NULL;
-    bool writethrough, src_writethrough, quiet = false, image_opts = false,
+    bool writethrough, src_writethrough, image_opts = false,
          skip_create = false, progress = false, tgt_image_opts = false;
     int64_t ret = -EINVAL;
     bool force_share = false;
@@ -2117,7 +2118,7 @@ static int img_convert(int argc, char **argv)
             src_cache = optarg;
             break;
         case 'q':
-            quiet = true;
+            s.quiet = true;
             break;
         case 'n':
             skip_create = true;
@@ -2206,7 +2207,7 @@ static int img_convert(int argc, char **argv)
     }
 
     /* Initialize before goto out */
-    if (quiet) {
+    if (s.quiet) {
         progress = false;
     }
     qemu_progress_init(progress, 1.0);
@@ -2217,7 +2218,7 @@ static int img_convert(int argc, char **argv)
 
     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
-                               fmt, src_flags, src_writethrough, quiet,
+                               fmt, src_flags, src_writethrough, s.quiet,
                                force_share);
         if (!s.src[bs_i]) {
             ret = -1;
@@ -2380,7 +2381,7 @@ static int img_convert(int argc, char **argv)
 
     if (skip_create) {
         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
-                            flags, writethrough, quiet, false);
+                            flags, writethrough, s.quiet, false);
     } else {
         /* TODO ultimately we should allow --target-image-opts
          * to be used even when -n is not given.
@@ -2388,7 +2389,7 @@ static int img_convert(int argc, char **argv)
          * to allow filenames in option syntax
          */
         s.target = img_open_file(out_filename, open_opts, out_fmt,
-                                 flags, writethrough, quiet, false);
+                                 flags, writethrough, s.quiet, false);
         open_opts = NULL; /* blk_new_open will have freed it */
     }
     if (!s.target) {
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v4 2/7 qemu-img: Add salvaging mode to convert

In reply to Max Reitz

This adds a salvaging mode (--salvage) to qemu-img convert which ignores read errors and treats the respective areas as containing only zeroes. This can be used for instance to at least partially recover the data from terminally corrupted qcow2 images.

---
 qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
 qemu-img-cmds.hx |  4 +--
 qemu-img.texi    |  4 +++
 3 files changed, 75 insertions(+), 23 deletions(-)
qemu-img.c +69 -21
index f606675933..9ffda08458 100644
@@ -66,6 +66,7 @@ enum {
     OPTION_SIZE = 264,
     OPTION_PREALLOCATION = 265,
     OPTION_SHRINK = 266,
+    OPTION_SALVAGE = 267,
 };
 
 typedef enum OutputFormat {
@@ -1578,6 +1579,7 @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool salvage;
     bool quiet;
     int min_sparse;
     int alignment;
@@ -1625,25 +1627,44 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     }
 
     if (s->sector_next_status <= sector_num) {
-        int64_t count = n * BDRV_SECTOR_SIZE;
+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
+        int64_t count;
 
-        if (s->target_has_backing) {
+        do {
+            count = n * BDRV_SECTOR_SIZE;
+
+            if (s->target_has_backing) {
+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
+                                        count, &count, NULL, NULL);
+            } else {
+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
+                                              offset, count, &count, NULL,
+                                              NULL);
+            }
+
+            if (ret < 0) {
+                if (s->salvage) {
+                    if (n == 1) {
+                        if (!s->quiet) {
+                            warn_report("error while reading block status at "
+                                        "offset %" PRIu64 ": %s", offset,
+                                        strerror(-ret));
+                        }
+                        /* Just try to read the data, then */
+                        ret = BDRV_BLOCK_DATA;
+                        count = BDRV_SECTOR_SIZE;
+                    } else {
+                        /* Retry on a shorter range */
+                        n = DIV_ROUND_UP(n, 4);
+                    }
+                } else {
+                    error_report("error while reading block status at offset "
+                                 "%" PRIu64 ": %s", offset, strerror(-ret));
+                    return ret;
+                }
+            }
+        } while (ret < 0);
 
-            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
-                                    (sector_num - src_cur_offset) *
-                                    BDRV_SECTOR_SIZE,
-                                    count, &count, NULL, NULL);
-        } else {
-            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                          (sector_num - src_cur_offset) *
-                                          BDRV_SECTOR_SIZE,
-                                          count, &count, NULL, NULL);
-        }
-        if (ret < 0) {
-            error_report("error while reading block status of sector %" PRId64
-                         ": %s", sector_num, strerror(-ret));
-            return ret;
-        }
         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
 
         if (ret & BDRV_BLOCK_ZERO) {
@@ -1680,6 +1701,7 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
 static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
                                         int nb_sectors, uint8_t *buf)
 {
+    uint64_t single_read_until = 0;
     int n, ret;
 
     assert(nb_sectors <= s->buf_sectors);
@@ -1687,6 +1709,7 @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         BlockBackend *blk;
         int src_cur;
         int64_t bs_sectors, src_cur_offset;
+        uint64_t offset;
 
         /* In the case of compression with multiple source files, we can get a
          * nb_sectors that spreads into the next part. So we must be able to
@@ -1695,13 +1718,29 @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         blk = s->src[src_cur];
         bs_sectors = s->src_sectors[src_cur];
 
+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
+
         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
+        if (single_read_until > offset) {
+            n = 1;
+        }
 
-        ret = blk_co_pread(
-                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
-                n << BDRV_SECTOR_BITS, buf, 0);
+        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
         if (ret < 0) {
-            return ret;
+            if (s->salvage) {
+                if (n > 1) {
+                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
+                    continue;
+                } else {
+                    if (!s->quiet) {
+                        warn_report("error while reading offset %" PRIu64
+                                    ": %s", offset, strerror(-ret));
+                    }
+                    memset(buf, 0, BDRV_SECTOR_SIZE);
+                }
+            } else {
+                return ret;
+            }
         }
 
         sector_num += n;
@@ -2032,6 +2071,7 @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {"force-share", no_argument, 0, 'U'},
             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
+            {"salvage", no_argument, 0, OPTION_SALVAGE},
             {0, 0, 0, 0}
         };
         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -2149,6 +2189,9 @@ static int img_convert(int argc, char **argv)
         case OPTION_IMAGE_OPTS:
             image_opts = true;
             break;
+        case OPTION_SALVAGE:
+            s.salvage = true;
+            break;
         case OPTION_TARGET_IMAGE_OPTS:
             tgt_image_opts = true;
             break;
@@ -2175,6 +2218,11 @@ static int img_convert(int argc, char **argv)
         goto fail_getopt;
     }
 
+    if (s.copy_range && s.salvage) {
+        error_report("Cannot use copy offloading in salvaging mode");
+        goto fail_getopt;
+    }
+
     if (tgt_image_opts && !skip_create) {
         error_report("--target-image-opts requires use of -n flag");
         goto fail_getopt;
qemu-img-cmds.hx +2 -2
index 4b47f7495d..1c93e6d185 100644
@@ -44,9 +44,9 @@ STEXI
 ETEXI
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("create", img_create,
qemu-img.texi +4 -0
index 724f244ba1..ba11a2c665 100644
@@ -175,6 +175,10 @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
 but will not automatically sparsify zero sectors, and may result in a fully
 allocated target image depending on the host support for getting allocation
 information.
+@item --salvage
+Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
+will still be printed.  Areas that cannot be read from the source will be
+treated as containing only zeroes.
 @end table
 
 Parameters to dd subcommand:
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH v4 3/7 blkdebug: Add @iotype error option

In reply to Max Reitz

This new error option allows users of blkdebug to inject errors only on certain kinds of I/O operations. Users usually want to make a very specific operation fail, not just any; but right now they simply hope that the event that triggers the error injection is followed up with that very operation. That may not be true, however, because the block layer is changing (including blkdebug, which may increase the number of types of I/O operations on which to inject errors).

The new option's default has been chosen to keep backwards compatibility.

Note that similar to the internal representation, we could choose to expose this option as a list of I/O types. But there is no practical use for this, because as described above, users usually know exactly which kind of operation they want to make fail, so there is no need to specify multiple I/O types at once. In addition, exposing this option as a list would require non-trivial changes to qemu_opts_absorb_qdict().

---
 qapi/block-core.json | 26 +++++++++++++++++++++++
 block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
 2 files changed, 67 insertions(+), 9 deletions(-)
qapi/block-core.json +26 -0
index 7ccbfff9d0..34d261721b 100644
@@ -3235,6 +3235,26 @@
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
             'cor_write'] }
 
+##
+# @BlkdebugIOType:
+#
+# Kinds of I/O that blkdebug can inject errors in.
+#
+# @read: .bdrv_co_preadv()
+#
+# @write: .bdrv_co_pwritev()
+#
+# @write-zeroes: .bdrv_co_pwrite_zeroes()
+#
+# @discard: .bdrv_co_pdiscard()
+#
+# @flush: .bdrv_co_flush_to_disk()
+#
+# Since: 4.1
+##
+{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+
 ##
 # @BlkdebugInjectErrorOptions:
 #
@@ -3245,6 +3265,11 @@
 # @state:       the state identifier blkdebug needs to be in to
 #               actually trigger the event; defaults to "any"
 #
+# @iotype:      the type of I/O operations on which this error should
+#               be injected; defaults to "all read, write,
+#               write-zeroes, discard, and flush operations"
+#               (since: 4.1)
+#
 # @errno:       error identifier (errno) to be returned; defaults to
 #               EIO
 #
@@ -3262,6 +3287,7 @@
 { 'struct': 'BlkdebugInjectErrorOptions',
   'data': { 'event': 'BlkdebugEvent',
             '*state': 'int',
+            '*iotype': 'BlkdebugIOType',
             '*errno': 'int',
             '*sector': 'int',
             '*once': 'bool',
block/blkdebug.c +41 -9

122 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#4 lore raw

PATCH v4 4/7 blkdebug: Add "none" event

In reply to Max Reitz

Together with @iotypes and @sector, this can be used to trap e.g. the first read or write access to a certain sector without having to know what happens internally in the block layer, i.e. which "real" events happen right before such an access.

---
 qapi/block-core.json | 4 +++-
 block/blkdebug.c     | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)
qapi/block-core.json +3 -1
index 34d261721b..54c927f1ad 100644
@@ -3215,6 +3215,8 @@
 #
 # @cor_write: a write due to copy-on-read (since 2.11)
 #
+# @none: triggers once at creation of the blkdebug node (since 4.1)
+#
 # Since: 2.9
 ##
 { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
@@ -3233,7 +3235,7 @@
             'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
             'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
-            'cor_write'] }
+            'cor_write', 'none' ] }
 
 ##
 # @BlkdebugIOType:
block/blkdebug.c +2 -0
index 3f3ec11230..1663ed25af 100644
@@ -491,6 +491,8 @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
         goto out;
     }
 
+    bdrv_debug_event(bs, BLKDBG_NONE);
+
     ret = 0;
 out:
     if (ret < 0) {
Signature · 1 line
2.20.1
author
#5 lore raw

PATCH v4 5/7 blkdebug: Inject errors on .bdrv_co_block_status()

In reply to Max Reitz
---
 qapi/block-core.json | 5 ++++-
 block/blkdebug.c     | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)
qapi/block-core.json +4 -1
index 54c927f1ad..028e4d6e02 100644
@@ -3252,10 +3252,13 @@
 #
 # @flush: .bdrv_co_flush_to_disk()
 #
+# @block-status: .bdrv_co_block_status()
+#
 # Since: 4.1
 ##
 { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
-  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
+            'block-status' ] }
 
 ##
 # @BlkdebugInjectErrorOptions:
block/blkdebug.c +8 -0
index 1663ed25af..5ae96c52b0 100644
@@ -670,7 +670,15 @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
                                                  int64_t *map,
                                                  BlockDriverState **file)
 {
+    int err;
+
     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
+
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
+    if (err) {
+        return err;
+    }
+
     return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
                                           pnum, map, file);
 }
Signature · 1 line
2.20.1
author
#6 lore raw

PATCH v4 6/7 iotests: Test qemu-img convert --salvage

In reply to Max Reitz

This test converts a simple image to another, but blkdebug injects block_status and read faults at some offsets. The resulting image should be the same as the input image, except that sectors that could not be read have to be 0.

---
 tests/qemu-iotests/251     | 162 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out |  43 ++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 206 insertions(+)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out
tests/qemu-iotests/251 new file +162 -0
new file mode 100755
index 0000000000..508d69769f
@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+                     $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# The second element is shared with $status_fail_offsets on purpose.
+# Starting with the third element, we test what happens when a
+# continuous range of sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+                   $((33 * 1024 * 1024 + 512))
+                   $(seq $((34 * 1024 * 1024)) $sector_size \
+                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+                  'image': {
+                      'driver': '$IMGFMT',
+                      'file': {
+                          'driver': 'file',
+                          'filename': '$TEST_IMG.orig'
+                      }
+                  },
+                  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'block-status',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'read',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+
+echo
+
+
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/status_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $3
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/read_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+    | _filter_offsets '%i' "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output image.
+# The offsets where reading failed altogether should be 0.  Make them
+# 0 in the input image, too, so we can compare both images.
+for ofs in $read_fail_offsets
+do
+    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
+        | _filter_qemu_io \
+        | _filter_offsets '%i' '' "$read_fail_offsets"
+done
+
+echo
+
+# These should be equal now.
+$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/251.out new file +43 -0

43 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 7ac9a5ea4a..dde113a552 100644
@@ -249,3 +249,4 @@
 247 rw auto quick
 248 rw auto quick
 249 rw auto quick
+251 auto quick
Signature · 1 line
2.20.1
Tested-by Reviewed-by on v4 6/7
#7 lore raw

Re: [Qemu-devel] [PATCH v4 6/7] iotests: Test qemu-img convert --salvage

Quoted 120 lines · Max Reitz

This test converts a simple image to another, but blkdebug injects block_status and read faults at some offsets. The resulting image should be the same as the input image, except that sectors that could not be read have to be 0.

---
  tests/qemu-iotests/251     | 162 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/251.out |  43 ++++++++++
  tests/qemu-iotests/group   |   1 +
  3 files changed, 206 insertions(+)

create mode 100755 tests/qemu-iotests/251
create mode 100644 tests/qemu-iotests/251.out

tests/qemu-iotests/251 new file +100 -0
new file mode 100755
index 0000000000..508d69769f
@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+                     $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# The second element is shared with $status_fail_offsets on purpose.
+# Starting with the third element, we test what happens when a
+# continuous range of sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+                   $((33 * 1024 * 1024 + 512))
+                   $(seq $((34 * 1024 * 1024)) $sector_size \
+                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+                  'image': {
+                      'driver': '$IMGFMT',
+                      'file': {
+                          'driver': 'file',
+                          'filename': '$TEST_IMG.orig'
+                      }
+                  },
+                  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'block-status',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'read',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+

tests/qemu-iotests/251 line 110

Quoted 10 lines
tests/qemu-iotests/251 new file +10 -0
+echo
+
+
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/status_fail_offset_$index/"

Why not just $ofs ? I've tested, it works for me as well

tests/qemu-iotests/251 line 117

Quoted 7 lines
tests/qemu-iotests/251 new file +7 -0
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $3
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/read_fail_offset_$index/"

and here.

Quoted 99 lines
tests/qemu-iotests/251 new file +45 -0
+        index=$((index + 1))
+    done
+
+    sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+    | _filter_offsets '%i' "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output image.
+# The offsets where reading failed altogether should be 0.  Make them
+# 0 in the input image, too, so we can compare both images.
+for ofs in $read_fail_offsets
+do
+    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
+        | _filter_qemu_io \
+        | _filter_offsets '%i' '' "$read_fail_offsets"
+done
+
+echo
+
+# These should be equal now.
+$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/251.out new file +43 -0
new file mode 100644
index 0000000000..75b8796aad
@@ -0,0 +1,43 @@
+QA output created by 251
+Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
+wrote 67108864/67108864 bytes at offset 0
+64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
+
+wrote 512/512 bytes at offset read_fail_offset_0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_1
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_2
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_3
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_4
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_5
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_6
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_7
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_8
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_9
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+Images are identical.
+*** done
tests/qemu-iotests/group +0 -0
index 7ac9a5ea4a..dde113a552 100644

tests/qemu-iotests/group line 252

Quoted 4 lines
tests/qemu-iotests/group +1 -0
@@ -249,3 +249,4 @@
  247 rw auto quick
  248 rw auto quick
  249 rw auto quick
+251 auto quick

why not rw? (actually, I don't know what means rw group...)

With simplified _filter_offsets (also, if drop $1 parameter, parameters may be changed to be $1 and $2) (or without, if there is a reason, but I don't see it):

Signature · 2 lines
Best regards,
Vladimir
author
#8 lore raw

Re: [Qemu-devel] [PATCH v4 6/7] iotests: Test qemu-img convert --salvage

Quoted 21 lines · Vladimir Sementsov-Ogievskiy
Quoted 20 lines · Max Reitz

This test converts a simple image to another, but blkdebug injects block_status and read faults at some offsets. The resulting image should be the same as the input image, except that sectors that could not be read have to be 0.

---
  tests/qemu-iotests/251     | 162 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/251.out |  43 ++++++++++
  tests/qemu-iotests/group   |   1 +
  3 files changed, 206 insertions(+)

create mode 100755 tests/qemu-iotests/251
create mode 100644 tests/qemu-iotests/251.out

tests/qemu-iotests/251 new file +0 -0
new file mode 100755
index 0000000000..508d69769f
@@ -0,0 +1,162 @@

[...]

Quoted 18 lines

tests/qemu-iotests/251 line 110

Quoted 7 lines
tests/qemu-iotests/251 new file +7 -0
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/status_fail_offset_$index/"

Why not just $ofs ? I've tested, it works for me as well

tests/qemu-iotests/251 line 117

Quoted 7 lines
tests/qemu-iotests/251 new file +7 -0
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $3
+    do
+        filters+=" -e s/$(printf "$1" $ofs)/read_fail_offset_$index/"
Quoted 1 line

and here.

Oops. Those are artifacts from when I had to use hexadecimal offsets because the output was in hex. (So the $1 parameter could be either %i or %x.) Will fix.

[...]

Quoted 11 lines

tests/qemu-iotests/group line 252

Quoted 9 lines
tests/qemu-iotests/group +1 -0
index 7ac9a5ea4a..dde113a552 100644
@@ -249,3 +249,4 @@
  247 rw auto quick
  248 rw auto quick
  249 rw auto quick
+251 auto quick

why not rw? (actually, I don't know what means rw group...)

Neither do I. :-)

So I don’t mind (or care, actually).

Quoted 4 lines

With simplified _filter_offsets (also, if drop $1 parameter, parameters may be changed to be $1 and $2) (or without, if there is a reason, but I don't see it):

Thanks!

Max

author
#9 lore raw

PATCH v4 7/7 iotests: Test qemu-img convert -C --salvage

In reply to Max Reitz

We do not support this combination (yet), so this should yield an error message.

---
 tests/qemu-iotests/082     | 1 +
 tests/qemu-iotests/082.out | 3 +++
 2 files changed, 4 insertions(+)
tests/qemu-iotests/082 +1 -0
index 278511dba4..804610c7eb 100755
@@ -163,6 +163,7 @@ echo === convert: -C and other options ===
 run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 
 echo
 echo === amend: Options specified more than once ===
tests/qemu-iotests/082.out +3 -0
index 7e25706813..58de358b38 100644
@@ -567,6 +567,9 @@ qemu-img: Cannot enable copy offloading when -S is used
 Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
 qemu-img: Cannot enable copy offloading when -c is used
 
+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+qemu-img: Cannot use copy offloading in salvaging mode
+
 === amend: Options specified more than once ===
 
 Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
Signature · 1 line
2.20.1
author Applied
#10 lore raw
In reply to Max Reitz
Quoted 29 lines · Max Reitz

Hi,

This series adds a --salvage option to qemu-img convert. With this, qemu-img will not abort when it encounters an I/O error. Instead, it tries to narrow it down and will treat the affected sectors as being completely 0 (and print a warning).

Testing this is not so easy, because while real I/O errors during read operations should be treated as described above, errors encountered during bdrv_block_status() should just be ignored and the affected sectors should be considered allocated. But blkdebug does not yet have a way to intercept this, and:

(1) Just adding a new block-status event would be silly, because I don't want an event, I want it to fail on a certain kind of operation, on a certain sector range, independently of any events, so why can't we just do that? See patch 4.

(2) If we just make blkdebug intercept .bdrv_co_block_status() like all other kinds of operations, at least iotest 041 fails, which does exactly that silly thing: It uses the read_aio event to wait for any read. But it turns out that there may be a bdrv_*block_status() call in between, so suddenly the wrong operation yields an error. As I said, the real fault here is that it does not really make sense to pray that the operation you want to fail is the one that is immediately executed after some event that you hope will trigger that operation.
See patch 3.

Quoted 6 lines

So patch 3 allows blkdebug users to select which kind of I/O operation they actually want to make fail, and patch 4 allows them to not use any event, but to have a rule active all the time.

Together, we can then enable error injection for block-status in patch 5 and make use of event=none iotype=block-status in patch 6.

Applied the series to my block branch, and fixed _filter_offsets in patch 6 as suggested by Vladimir.

Max