Skip to contentSkip to conversation

qemu-img: Make unallocated part of backing chain obvious in map

PATCH In review Eric Blake 3 participants 8 messages Jul 1, 2021 – Jul 8, 2021

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v3 cover qemu-img: Make unallocated part of backing chain obvious in map

v2 was here [1]
in v3:
- pick up iotest change from [2] (rest of that series is abandoned)
- don't change semantics of "depth"
- make new witness unconditional, rather than optional
- spell new witness "present":true/false

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg02446.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg03006.html

Eric Blake (2):
iotests: Improve and rename test 309 to nbd-qemu-allocation
qemu-img: Make unallocated part of backing chain obvious in map

 docs/tools/qemu-img.rst                       |   3 +
 qapi/block-core.json                          |   7 +-
 qemu-img.c                                    |   7 +-
 tests/qemu-iotests/122.out                    |  84 ++++----
 tests/qemu-iotests/154.out                    | 190 +++++++++---------
 tests/qemu-iotests/179.out                    | 133 ++++++++----
 tests/qemu-iotests/223.out                    |  56 +++---
 tests/qemu-iotests/244.out                    |  23 ++-
 tests/qemu-iotests/252.out                    |  10 +-
 tests/qemu-iotests/274.out                    |  48 ++---
 tests/qemu-iotests/291.out                    |  24 +--
 tests/qemu-iotests/309.out                    |  22 --
 .../{309 => tests/nbd-qemu-allocation}        |   5 +-
 .../tests/nbd-qemu-allocation.out             |  32 +++
 14 files changed, 358 insertions(+), 286 deletions(-)
 delete mode 100644 tests/qemu-iotests/309.out
 rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%)
 create mode 100644 tests/qemu-iotests/tests/nbd-qemu-allocation.out
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v3 1/2 iotests: Improve and rename test 309 to nbd-qemu-allocation

Enhance the test to inspect what qemu-nbd is advertising during handshake, and rename it now that we support useful iotest names.

---
 .../qemu-iotests/{309 => tests/nbd-qemu-allocation}  |  5 ++++-
 .../{309.out => tests/nbd-qemu-allocation.out}       | 12 +++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)
 rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%)
 rename tests/qemu-iotests/{309.out => tests/nbd-qemu-allocation.out} (81%)
tests/qemu-iotests/tests/nbd-qemu-allocation renamed from tests/qemu-iotests/309 +4 -1
similarity index 95%
rename from tests/qemu-iotests/309
rename to tests/qemu-iotests/tests/nbd-qemu-allocation
index b90b279994c9..4ee73db8033b 100755
@@ -3,7 +3,7 @@
 #
 # Test qemu-nbd -A
 #
-# Copyright (C) 2018-2020 Red Hat, Inc.
+# Copyright (C) 2018-2021 Red Hat, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -32,6 +32,7 @@ _cleanup()
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
 # get standard environment, filters and checks
+cd ..
 . ./common.rc
 . ./common.filter
 . ./common.nbd
@@ -57,6 +58,8 @@ echo
 $QEMU_IMG map --output=json -f qcow2 "$TEST_IMG"
 IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
 nbd_server_start_unix_socket -r -f qcow2 -A "$TEST_IMG"
+# Inspect what the server is exposing
+$QEMU_NBD --list -k $nbd_unix_socket
 # Normal -f raw NBD block status loses access to allocation information
 $QEMU_IMG map --output=json --image-opts \
     "$IMG" | _filter_qemu_img_map
tests/qemu-iotests/tests/nbd-qemu-allocation.out renamed from tests/qemu-iotests/309.out +11 -1
similarity index 81%
rename from tests/qemu-iotests/309.out
rename to tests/qemu-iotests/tests/nbd-qemu-allocation.out
index db75bb6b0df9..c51022b2a38d 100644
@@ -1,4 +1,4 @@
-QA output created by 309
+QA output created by nbd-qemu-allocation
 
 === Initial image setup ===
 
@@ -14,6 +14,16 @@ wrote 2097152/2097152 bytes at offset 1048576
 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680},
 { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680},
 { "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}]
+exports available: 1
+ export: ''
+  size:  4194304
+  flags: 0x58f ( readonly flush fua df multi cache )
+  min block: 1
+  opt block: 4096
+  max block: 33554432
+  available meta contexts: 2
+   base:allocation
+   qemu:allocation-depth
 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH v3 2/2 qemu-img: Make unallocated part of backing chain obvious in map

In reply to Eric Blake

The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).

The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.

An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.

---
 docs/tools/qemu-img.rst                       |   3 +
 qapi/block-core.json                          |   7 +-
 qemu-img.c                                    |   7 +-
 tests/qemu-iotests/122.out                    |  84 ++++----
 tests/qemu-iotests/154.out                    | 190 +++++++++---------
 tests/qemu-iotests/179.out                    | 133 ++++++++----
 tests/qemu-iotests/223.out                    |  56 +++---
 tests/qemu-iotests/244.out                    |  23 ++-
 tests/qemu-iotests/252.out                    |  10 +-
 tests/qemu-iotests/274.out                    |  48 ++---
 tests/qemu-iotests/291.out                    |  24 +--
 .../tests/nbd-qemu-allocation.out             |  16 +-
 12 files changed, 330 insertions(+), 271 deletions(-)
docs/tools/qemu-img.rst +3 -0
index cfe11478791f..d6300f7ee03d 100644
@@ -597,6 +597,9 @@ Command description:
     if false, the sectors are either unallocated or stored as optimized
     all-zero clusters);
   - whether the data is known to read as zero (boolean field ``zero``);
+  - whether the data is actually present (boolean field ``present``);
+    if false, rebasing the backing chain onto a deeper file would pick
+    up data from the deeper file;
   - in order to make the output shorter, the target file is expressed as
     a ``depth``; for example, a depth of 2 refers to the backing file
     of the backing file of *FILENAME*.
qapi/block-core.json +5 -2
index a54f37dbef06..912a7d9265e5 100644
@@ -261,6 +261,9 @@
 #         images in the chain)) before reaching one for which the
 #         range is allocated
 #
+# @present: true if this layer provides the data, false if adding a backing
+#           layer could impact this region (since 6.1)
+#
 # @offset: if present, the image file stores the data for this range
 #          in raw format at the given (host) offset
 #
@@ -271,8 +274,8 @@
 ##
 { 'struct': 'MapEntry',
   'data': {'start': 'int', 'length': 'int', 'data': 'bool',
-           'zero': 'bool', 'depth': 'int', '*offset': 'int',
-           '*filename': 'str' } }
+           'zero': 'bool', 'depth': 'int', 'present': 'bool',
+           '*offset': 'int', '*filename': 'str' } }
 
 ##
 # @BlockdevCacheInfo:
qemu-img.c +5 -2
index 7956a8996512..8090facc5087 100644
@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,
         break;
     case OFORMAT_JSON:
         printf("{ \"start\": %"PRId64", \"length\": %"PRId64","
-               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
-               e->start, e->length, e->depth,
+               " \"depth\": %"PRId64", \"present\": %s, \"zero\": %s,"
+               "\"data\": %s", e->start, e->length, e->depth,
+               e->present ? "true" : "false",
                e->zero ? "true" : "false",
                e->data ? "true" : "false");
         if (e->has_offset) {
@@ -3047,6 +3048,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         .offset = map,
         .has_offset = has_offset,
         .depth = depth,
+        .present = !!(ret & BDRV_BLOCK_ALLOCATED),
         .has_filename = filename,
         .filename = filename,
     };
@@ -3062,6 +3064,7 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)
     if (curr->zero != next->zero ||
         curr->data != next->data ||
         curr->depth != next->depth ||
+        curr->present != next->present ||
         curr->has_filename != next->has_filename ||
         curr->has_offset != next->has_offset) {
         return false;
tests/qemu-iotests/122.out +42 -42

158 lines not on this page · raw message

tests/qemu-iotests/154.out +97 -93

330 lines not on this page · raw message

tests/qemu-iotests/179.out +89 -44

169 lines not on this page · raw message

tests/qemu-iotests/223.out +28 -28

78 lines not on this page · raw message

tests/qemu-iotests/244.out +12 -11
index 99f56ac18c48..49121c556086 100644
@@ -57,11 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728
 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 No errors were found on the image.
 
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
-{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
-{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304},
-{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}]
+[{ "start": 0, "length": 1048576, "depth": 0, "present": false, "zero": true,"data": false},
+{ "start": 1048576, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": 1048576},
+{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false},
+{ "start": 4194304, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false, "offset": 4194304},
+{ "start": 5242880, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false},
+{ "start": 6291456, "length": 60817408, "depth": 0, "present": false, "zero": true,"data": false}]
 
 read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -93,10 +94,10 @@ wrote 3145728/3145728 bytes at offset 3145728
 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 No errors were found on the image.
 
-[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 0},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
-{ "start": 4194304, "length": 2097152, "depth": 0, "zero": true, "data": false, "offset": 4194304},
-{ "start": 6291456, "length": 60817408, "depth": 0, "zero": false, "data": true, "offset": 6291456}]
+[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": 0},
+{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false},
+{ "start": 4194304, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false, "offset": 4194304},
+{ "start": 6291456, "length": 60817408, "depth": 0, "present": true, "zero": false,"data": true, "offset": 6291456}]
 
 read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -122,8 +123,8 @@ read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 Offset          Length          Mapped to       File
 0               0x100000        0               TEST_DIR/t.qcow2.data
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0},
-{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}]
+[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": 0},
+{ "start": 1048576, "length": 66060288, "depth": 0, "present": false, "zero": true,"data": false}]
 
 === Copy offloading ===
 
tests/qemu-iotests/252.out +5 -5
index 12dce889f823..4476e9559c31 100644
@@ -23,8 +23,8 @@ read 131072/131072 bytes at offset 131072
 read 131072/131072 bytes at offset 262144
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
-[{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 262144, "length": 131072, "depth": 0, "zero": true, "data": false}]
+[{ "start": 0, "length": 262144, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 262144, "length": 131072, "depth": 0, "present": false, "zero": true,"data": false}]
 
 read 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -33,7 +33,7 @@ read 131072/131072 bytes at offset 131072
 read 131072/131072 bytes at offset 262144
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
-[{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false},
-{ "start": 327680, "length": 65536, "depth": 1, "zero": true, "data": false}]
+[{ "start": 0, "length": 262144, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 262144, "length": 65536, "depth": 0, "present": true, "zero": true,"data": false},
+{ "start": 327680, "length": 65536, "depth": 1, "present": false, "zero": true,"data": false}]
 *** done
tests/qemu-iotests/274.out +24 -24

102 lines not on this page · raw message

tests/qemu-iotests/291.out +12 -12
index 23411c0ff4d9..f3b409c29bfe 100644
@@ -103,16 +103,16 @@ Format specific information:
 
 === Check bitmap contents ===
 
-[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false},
-{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false},
-{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
-{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
-{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 3145728, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 3145728, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false},
+{ "start": 4194304, "length": 6291456, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 1048576, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false},
+{ "start": 2097152, "length": 8388608, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 2097152, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false},
+{ "start": 3145728, "length": 7340032, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 2097152, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false},
+{ "start": 3145728, "length": 7340032, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}]
 *** done
tests/qemu-iotests/tests/nbd-qemu-allocation.out +8 -8
index c51022b2a38d..c91cbc2b9585 100644
@@ -11,9 +11,9 @@ wrote 2097152/2097152 bytes at offset 1048576
 
 === Check allocation over NBD ===
 
-[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680},
-{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680},
-{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}]
+[{ "start": 0, "length": 1048576, "depth": 1, "present": true, "zero": false,"data": true, "offset": 327680},
+{ "start": 1048576, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": 327680},
+{ "start": 3145728, "length": 1048576, "depth": 1, "present": false, "zero": true,"data": false}]
 exports available: 1
  export: ''
   size:  4194304
@@ -24,9 +24,9 @@ exports available: 1
   available meta contexts: 2
    base:allocation
    qemu:allocation-depth
-[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false},
-{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 3145728, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET},
+{ "start": 3145728, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": true, "offset": OFFSET},
+{ "start": 1048576, "length": 2097152, "depth": 0, "present": false, "zero": false,"data": false},
+{ "start": 3145728, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}]
 *** done
Signature · 1 line
2.31.1
Reviewed-by on v3 2/2
#3 lore raw
Quoted 55 lines · Eric Blake

The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).

The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.

An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.

---
  docs/tools/qemu-img.rst                       |   3 +
  qapi/block-core.json                          |   7 +-
  qemu-img.c                                    |   7 +-
  tests/qemu-iotests/122.out                    |  84 ++++----
  tests/qemu-iotests/154.out                    | 190 +++++++++---------
  tests/qemu-iotests/179.out                    | 133 ++++++++----
  tests/qemu-iotests/223.out                    |  56 +++---
  tests/qemu-iotests/244.out                    |  23 ++-
  tests/qemu-iotests/252.out                    |  10 +-
  tests/qemu-iotests/274.out                    |  48 ++---
  tests/qemu-iotests/291.out                    |  24 +--
  .../tests/nbd-qemu-allocation.out             |  16 +-
  12 files changed, 330 insertions(+), 271 deletions(-)
docs/tools/qemu-img.rst +0 -0
index cfe11478791f..d6300f7ee03d 100644

docs/tools/qemu-img.rst Command description: · line 602

Quoted 6 lines
docs/tools/qemu-img.rst +3 -0
@@ -597,6 +597,9 @@ Command description:
      if false, the sectors are either unallocated or stored as optimized
      all-zero clusters);
    - whether the data is known to read as zero (boolean field ``zero``);
+  - whether the data is actually present (boolean field ``present``);
+    if false, rebasing the backing chain onto a deeper file would pick
+    up data from the deeper file;

Preexisting, but rather strange style of documentation, when described option doesn't go first in the paragraph..

Quoted 33 lines
docs/tools/qemu-img.rst +0 -0
    - in order to make the output shorter, the target file is expressed as
      a ``depth``; for example, a depth of 2 refers to the backing file
      of the backing file of *FILENAME*.
qapi/block-core.json +5 -2
index a54f37dbef06..912a7d9265e5 100644
@@ -261,6 +261,9 @@
  #         images in the chain)) before reaching one for which the
  #         range is allocated
  #
+# @present: true if this layer provides the data, false if adding a backing
+#           layer could impact this region (since 6.1)
+#
  # @offset: if present, the image file stores the data for this range
  #          in raw format at the given (host) offset
  #
@@ -271,8 +274,8 @@
  ##
  { 'struct': 'MapEntry',
    'data': {'start': 'int', 'length': 'int', 'data': 'bool',
-           'zero': 'bool', 'depth': 'int', '*offset': 'int',
-           '*filename': 'str' } }
+           'zero': 'bool', 'depth': 'int', 'present': 'bool',
+           '*offset': 'int', '*filename': 'str' } }
 
  ##
  # @BlockdevCacheInfo:
qemu-img.c +0 -0
index 7956a8996512..8090facc5087 100644

qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985

Quoted 8 lines
qemu-img.c +3 -2
@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,
          break;
      case OFORMAT_JSON:
          printf("{ \"start\": %"PRId64", \"length\": %"PRId64","
-               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
-               e->start, e->length, e->depth,
+               " \"depth\": %"PRId64", \"present\": %s, \"zero\": %s,"
+               "\"data\": %s", e->start, e->length, e->depth,
+               e->present ? "true" : "false",

Didn't you want to put present at the end? Still, this shouldn't be significant. And it make sense to keep present, zero and data together.

You missied a whitespace after '"zero": %s,', which is obvious from further test diff hunks.

With it fixed:

Quoted 24 lines
qemu-img.c +2 -0
                 e->zero ? "true" : "false",
                 e->data ? "true" : "false");
          if (e->has_offset) {
@@ -3047,6 +3048,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
          .offset = map,
          .has_offset = has_offset,
          .depth = depth,
+        .present = !!(ret & BDRV_BLOCK_ALLOCATED),
          .has_filename = filename,
          .filename = filename,
      };
@@ -3062,6 +3064,7 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)
      if (curr->zero != next->zero ||
          curr->data != next->data ||
          curr->depth != next->depth ||
+        curr->present != next->present ||
          curr->has_filename != next->has_filename ||
          curr->has_offset != next->has_offset) {
          return false;
tests/qemu-iotests/122.out +0 -0
index 3a3e121d579d..83dfcf75281f 100644
@@ -67,12 +67,12 @@ read 65536/65536 bytes at offset 4194304

[..]

Signature · 2 lines
Best regards,
Vladimir
author
#4 lore raw

On Sat, Jul 03, 2021 at 10:25:28AM +0300, Vladimir Sementsov-Ogievskiy wrote:
...

Quoted 9 lines
Quoted 8 lines

An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.

Quoted 10 lines

docs/tools/qemu-img.rst Command description: · line 602

Quoted 8 lines

+++ b/docs/tools/qemu-img.rst

docs/tools/qemu-img.rst +3 -0
@@ -597,6 +597,9 @@ Command description:
      if false, the sectors are either unallocated or stored as optimized
      all-zero clusters);
    - whether the data is known to read as zero (boolean field ``zero``);
+  - whether the data is actually present (boolean field ``present``);
+    if false, rebasing the backing chain onto a deeper file would pick
+    up data from the deeper file;

Preexisting, but rather strange style of documentation, when described option doesn't go first in the paragraph..

Yeah. I'll send a followup email with a rewording of those paragraphs for consideration.

Quoted 4 lines
Quoted 4 lines
qemu-img.c +1 -0
+++ b/qemu-img.c
@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,
          break;
      case OFORMAT_JSON:
Quoted 8 lines

qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985

Quoted 6 lines
qemu-img.c +3 -2
          printf("{ \"start\": %"PRId64", \"length\": %"PRId64","
-               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
-               e->start, e->length, e->depth,
+               " \"depth\": %"PRId64", \"present\": %s, \"zero\": %s,"
+               "\"data\": %s", e->start, e->length, e->depth,
+               e->present ? "true" : "false",

Didn't you want to put present at the end? Still, this shouldn't be significant. And it make sense to keep present, zero and data together.

I wanted it before anything optional, which "offset" is, so it already can't be at the end. If I understood Nir correctly, it was more important to always be present (it's easy to write a parser that searches for terms in the same position, and tolerates a missing term from an older verseion, but harder to parse a term that might or might not be present).

Quoted 5 lines

You missied a whitespace after '"zero": %s,', which is obvious from further test diff hunks.

With it fixed:

Thanks for catching that. I've updated that, and will queue through my NBD tree.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#5 lore raw

[PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs

In reply to Eric Blake

Reword the paragraphs to list the JSON key first, rather than in the middle of prose.

---
 docs/tools/qemu-img.rst | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
docs/tools/qemu-img.rst +10 -10
index d6300f7ee03d..1d8470eada0e 100644
@@ -593,16 +593,16 @@ Command description:
   the ``start``, ``length``, ``offset`` fields;
   it will also include other more specific information:
 
-  - whether the sectors contain actual data or not (boolean field ``data``;
-    if false, the sectors are either unallocated or stored as optimized
-    all-zero clusters);
-  - whether the data is known to read as zero (boolean field ``zero``);
-  - whether the data is actually present (boolean field ``present``);
-    if false, rebasing the backing chain onto a deeper file would pick
-    up data from the deeper file;
-  - in order to make the output shorter, the target file is expressed as
-    a ``depth``; for example, a depth of 2 refers to the backing file
-    of the backing file of *FILENAME*.
+  - boolean field ``data``: true if the sectors contain actual data,
+    false if the sectors are either unallocated or stored as optimized
+    all-zero clusters
+  - boolean field ``zero``: true if the data is known to read as zero
+  - boolean field ``present``: true if the data belongs to the backing
+    chain, false if rebasing the backing chain onto a deeper file
+    would pick up data from the deeper file;
+  - integer field ``depth``: the depth within the backing chain at
+    which the data was resolved; for example, a depth of 2 refers to
+    the backing file of the backing file of *FILENAME*.
 
   In JSON format, the ``offset`` field is optional; it is absent in
   cases where ``human`` format would omit the entry or exit with an error.
Signature · 1 line
2.31.1
Reviewed-by
#6 lore raw

Re: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs

Quoted 42 lines · Eric Blake

Reword the paragraphs to list the JSON key first, rather than in the middle of prose.

---
 docs/tools/qemu-img.rst | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
docs/tools/qemu-img.rst +10 -11
index d6300f7ee03d..1d8470eada0e 100644
@@ -593,16 +593,16 @@ Command description:
   the ``start``, ``length``, ``offset`` fields;
   it will also include other more specific information:
 
-  - whether the sectors contain actual data or not (boolean field ``data``;
-    if false, the sectors are either unallocated or stored as optimized
-    all-zero clusters);
-  - whether the data is known to read as zero (boolean field ``zero``);
-  - whether the data is actually present (boolean field ``present``);
-    if false, rebasing the backing chain onto a deeper file would pick
-    up data from the deeper file;
-  - in order to make the output shorter, the target file is expressed as
-    a ``depth``; for example, a depth of 2 refers to the backing file
-    of the backing file of *FILENAME*.
+  - boolean field ``data``: true if the sectors contain actual data,
+    false if the sectors are either unallocated or stored as optimized
+    all-zero clusters
+  - boolean field ``zero``: true if the data is known to read as zero
+  - boolean field ``present``: true if the data belongs to the backing
+    chain, false if rebasing the backing chain onto a deeper file
+    would pick up data from the deeper file;
+  - integer field ``depth``: the depth within the backing chain at
+    which the data was resolved; for example, a depth of 2 refers to
+    the backing file of the backing file of *FILENAME*.
 
   In JSON format, the ``offset`` field is optional; it is absent in
   cases where ``human`` format would omit the entry or exit with an error.
--
Quoted 1 line

2.31.1

Would be nice if this could be generated from the json schema instead of repeating the type and description of the fields, but this is a nice improvement.