qemu-img: Make unallocated part of backing chain obvious in map
v3 cover qemu-img: Make unallocated part of backing chain obvious in map
v2 was here [1]
in v3:
- pick up iotest change from [2] (rest of that series is abandoned)
- don't change semantics of "depth"
- make new witness unconditional, rather than optional
- spell new witness "present":true/false
[1] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg02446.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg03006.html
Eric Blake (2):
iotests: Improve and rename test 309 to nbd-qemu-allocation
qemu-img: Make unallocated part of backing chain obvious in map
docs/tools/qemu-img.rst | 3 +
qapi/block-core.json | 7 +-
qemu-img.c | 7 +-
tests/qemu-iotests/122.out | 84 ++++----
tests/qemu-iotests/154.out | 190 +++++++++---------
tests/qemu-iotests/179.out | 133 ++++++++----
tests/qemu-iotests/223.out | 56 +++---
tests/qemu-iotests/244.out | 23 ++-
tests/qemu-iotests/252.out | 10 +-
tests/qemu-iotests/274.out | 48 ++---
tests/qemu-iotests/291.out | 24 +--
tests/qemu-iotests/309.out | 22 --
.../{309 => tests/nbd-qemu-allocation} | 5 +-
.../tests/nbd-qemu-allocation.out | 32 +++
14 files changed, 358 insertions(+), 286 deletions(-)
delete mode 100644 tests/qemu-iotests/309.out
rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%)
create mode 100644 tests/qemu-iotests/tests/nbd-qemu-allocation.outSignature · 1 line
2.31.1
PATCH v3 1/2 iotests: Improve and rename test 309 to nbd-qemu-allocation
Enhance the test to inspect what qemu-nbd is advertising during handshake, and rename it now that we support useful iotest names.
---
.../qemu-iotests/{309 => tests/nbd-qemu-allocation} | 5 ++++-
.../{309.out => tests/nbd-qemu-allocation.out} | 12 +++++++++++-
2 files changed, 15 insertions(+), 2 deletions(-)
rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%)
rename tests/qemu-iotests/{309.out => tests/nbd-qemu-allocation.out} (81%)tests/qemu-iotests/tests/nbd-qemu-allocation renamed from tests/qemu-iotests/309 +4 -1
@@ -3,7 +3,7 @@ # # Test qemu-nbd -A # -# Copyright (C) 2018-2020 Red Hat, Inc. +# Copyright (C) 2018-2021 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by
@@ -32,6 +32,7 @@ _cleanup() trap "_cleanup; exit \$status" 0 1 2 3 15 # get standard environment, filters and checks +cd .. . ./common.rc . ./common.filter . ./common.nbd
@@ -57,6 +58,8 @@ echo $QEMU_IMG map --output=json -f qcow2 "$TEST_IMG" IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket" nbd_server_start_unix_socket -r -f qcow2 -A "$TEST_IMG" +# Inspect what the server is exposing +$QEMU_NBD --list -k $nbd_unix_socket # Normal -f raw NBD block status loses access to allocation information $QEMU_IMG map --output=json --image-opts \ "$IMG" | _filter_qemu_img_map
tests/qemu-iotests/tests/nbd-qemu-allocation.out renamed from tests/qemu-iotests/309.out +11 -1
@@ -1,4 +1,4 @@ -QA output created by 309 +QA output created by nbd-qemu-allocation === Initial image setup ===
@@ -14,6 +14,16 @@ wrote 2097152/2097152 bytes at offset 1048576 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}, { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}] +exports available: 1 + export: '' + size: 4194304 + flags: 0x58f ( readonly flush fua df multi cache ) + min block: 1 + opt block: 4096 + max block: 33554432 + available meta contexts: 2 + base:allocation + qemu:allocation-depth [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
Signature · 1 line
2.31.1
PATCH v3 2/2 qemu-img: Make unallocated part of backing chain obvious in map
In reply to Eric BlakeThe recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.
--- docs/tools/qemu-img.rst | 3 + qapi/block-core.json | 7 +- qemu-img.c | 7 +- tests/qemu-iotests/122.out | 84 ++++---- tests/qemu-iotests/154.out | 190 +++++++++--------- tests/qemu-iotests/179.out | 133 ++++++++---- tests/qemu-iotests/223.out | 56 +++--- tests/qemu-iotests/244.out | 23 ++- tests/qemu-iotests/252.out | 10 +- tests/qemu-iotests/274.out | 48 ++--- tests/qemu-iotests/291.out | 24 +-- .../tests/nbd-qemu-allocation.out | 16 +- 12 files changed, 330 insertions(+), 271 deletions(-)
docs/tools/qemu-img.rst +3 -0
@@ -597,6 +597,9 @@ Command description: if false, the sectors are either unallocated or stored as optimized all-zero clusters); - whether the data is known to read as zero (boolean field ``zero``); + - whether the data is actually present (boolean field ``present``); + if false, rebasing the backing chain onto a deeper file would pick + up data from the deeper file; - in order to make the output shorter, the target file is expressed as a ``depth``; for example, a depth of 2 refers to the backing file of the backing file of *FILENAME*.
qapi/block-core.json +5 -2
@@ -261,6 +261,9 @@ # images in the chain)) before reaching one for which the # range is allocated # +# @present: true if this layer provides the data, false if adding a backing +# layer could impact this region (since 6.1) +# # @offset: if present, the image file stores the data for this range # in raw format at the given (host) offset #
@@ -271,8 +274,8 @@ ## { 'struct': 'MapEntry', 'data': {'start': 'int', 'length': 'int', 'data': 'bool', - 'zero': 'bool', 'depth': 'int', '*offset': 'int', - '*filename': 'str' } } + 'zero': 'bool', 'depth': 'int', 'present': 'bool', + '*offset': 'int', '*filename': 'str' } } ## # @BlockdevCacheInfo:
qemu-img.c +5 -2
@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,break ;case OFORMAT_JSON :printf ("{ \"start\": %" PRId64", \"length\": %" PRId64"," -" \"depth\": %" PRId64", \"zero\": %s, \"data\": %s" , - e->start, e->length, e->depth, +" \"depth\": %" PRId64", \"present\": %s, \"zero\": %s," +"\"data\": %s" , e->start, e->length, e->depth, + e->present ?"true" :"false" , e->zero ?"true" :"false" , e->data ?"true" :"false" );if (e->has_offset) {
@@ -3047,6 +3048,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, .offset = map, .has_offset = has_offset, .depth = depth, + .present = !!(ret & BDRV_BLOCK_ALLOCATED), .has_filename = filename, .filename = filename, };
@@ -3062,6 +3064,7 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)if (curr->zero != next->zero || curr->data != next->data || curr->depth != next->depth || + curr->present != next->present || curr->has_filename != next->has_filename || curr->has_offset != next->has_offset) {return false;
tests/qemu-iotests/122.out +42 -42
158 lines not on this page · raw message
tests/qemu-iotests/154.out +97 -93
330 lines not on this page · raw message
tests/qemu-iotests/179.out +89 -44
169 lines not on this page · raw message
tests/qemu-iotests/223.out +28 -28
78 lines not on this page · raw message
tests/qemu-iotests/244.out +12 -11
@@ -57,11 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, -{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, -{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}] +[{ "start": 0, "length": 1048576, "depth": 0, "present": false, "zero": true,"data": false}, +{ "start": 1048576, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": 1048576}, +{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false}, +{ "start": 4194304, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false, "offset": 4194304}, +{ "start": 5242880, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false}, +{ "start": 6291456, "length": 60817408, "depth": 0, "present": false, "zero": true,"data": false}] read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -93,10 +94,10 @@ wrote 3145728/3145728 bytes at offset 3145728 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. -[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 0}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, -{ "start": 4194304, "length": 2097152, "depth": 0, "zero": true, "data": false, "offset": 4194304}, -{ "start": 6291456, "length": 60817408, "depth": 0, "zero": false, "data": true, "offset": 6291456}] +[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": 0}, +{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false}, +{ "start": 4194304, "length": 2097152, "depth": 0, "present": true, "zero": true,"data": false, "offset": 4194304}, +{ "start": 6291456, "length": 60817408, "depth": 0, "present": true, "zero": false,"data": true, "offset": 6291456}] read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -122,8 +123,8 @@ read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Offset Length Mapped to File 0 0x100000 0 TEST_DIR/t.qcow2.data -[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0}, -{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}] +[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": 0}, +{ "start": 1048576, "length": 66060288, "depth": 0, "present": false, "zero": true,"data": false}] === Copy offloading ===
tests/qemu-iotests/252.out +5 -5
@@ -23,8 +23,8 @@ read 131072/131072 bytes at offset 131072 read 131072/131072 bytes at offset 262144 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 262144, "length": 131072, "depth": 0, "zero": true, "data": false}] +[{ "start": 0, "length": 262144, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 262144, "length": 131072, "depth": 0, "present": false, "zero": true,"data": false}] read 131072/131072 bytes at offset 0 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -33,7 +33,7 @@ read 131072/131072 bytes at offset 131072 read 131072/131072 bytes at offset 262144 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -[{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false}, -{ "start": 327680, "length": 65536, "depth": 1, "zero": true, "data": false}] +[{ "start": 0, "length": 262144, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 262144, "length": 65536, "depth": 0, "present": true, "zero": true,"data": false}, +{ "start": 327680, "length": 65536, "depth": 1, "present": false, "zero": true,"data": false}] *** done
tests/qemu-iotests/274.out +24 -24
102 lines not on this page · raw message
tests/qemu-iotests/291.out +12 -12
@@ -103,16 +103,16 @@ Format specific information: === Check bitmap contents === -[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false}, -{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false}, -{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, -{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, -{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 3145728, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 3145728, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false}, +{ "start": 4194304, "length": 6291456, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 1048576, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false}, +{ "start": 2097152, "length": 8388608, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}] +[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 2097152, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false}, +{ "start": 3145728, "length": 7340032, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}] +[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 2097152, "length": 1048576, "depth": 0, "present": false, "zero": false,"data": false}, +{ "start": 3145728, "length": 7340032, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}] *** done
tests/qemu-iotests/tests/nbd-qemu-allocation.out +8 -8
@@ -11,9 +11,9 @@ wrote 2097152/2097152 bytes at offset 1048576 === Check allocation over NBD === -[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}, -{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}, -{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}] +[{ "start": 0, "length": 1048576, "depth": 1, "present": true, "zero": false,"data": true, "offset": 327680}, +{ "start": 1048576, "length": 2097152, "depth": 0, "present": true, "zero": false,"data": true, "offset": 327680}, +{ "start": 3145728, "length": 1048576, "depth": 1, "present": false, "zero": true,"data": false}] exports available: 1 export: '' size: 4194304
@@ -24,9 +24,9 @@ exports available: 1 available meta contexts: 2 base:allocation qemu:allocation-depth -[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false}, -{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 3145728, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}, +{ "start": 3145728, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": false, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "present": true, "zero": true,"data": true, "offset": OFFSET}, +{ "start": 1048576, "length": 2097152, "depth": 0, "present": false, "zero": false,"data": false}, +{ "start": 3145728, "length": 1048576, "depth": 0, "present": true, "zero": false,"data": true, "offset": OFFSET}] *** done
Signature · 1 line
2.31.1
Quoted 55 lines · Eric Blake
The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED(which defers to a [missing] backing file) and aQCOW2_CLUSTER_ZERO_PLAINcluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish aQCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).The task of reconstructing a qcow2 chain was made harder in commit
0da9856851(nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted inBDRV_BLOCK_ALLOCATED(showing up asNBD_STATE_ZEROin isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 3 + qapi/block-core.json | 7 +- qemu-img.c | 7 +- tests/qemu-iotests/122.out | 84 ++++---- tests/qemu-iotests/154.out | 190 +++++++++--------- tests/qemu-iotests/179.out | 133 ++++++++---- tests/qemu-iotests/223.out | 56 +++--- tests/qemu-iotests/244.out | 23 ++- tests/qemu-iotests/252.out | 10 +- tests/qemu-iotests/274.out | 48 ++--- tests/qemu-iotests/291.out | 24 +-- .../tests/nbd-qemu-allocation.out | 16 +- 12 files changed, 330 insertions(+), 271 deletions(-)docs/tools/qemu-img.rst +0
-0
docs/tools/qemu-img.rst Command description: · line 602
Quoted 6 lines
docs/tools/qemu-img.rst +3
-0@@ -597,6 +597,9 @@ Command description: if false, the sectors are either unallocated or stored as optimized all-zero clusters); - whether the data is known to read as zero (boolean field ``zero``); + - whether the data is actually present (boolean field ``present``); + if false, rebasing the backing chain onto a deeper file would pick + up data from the deeper file;
Preexisting, but rather strange style of documentation, when described option doesn't go first in the paragraph..
Quoted 33 lines
docs/tools/qemu-img.rst +0
-0- in order to make the output shorter, the target file is expressed as a ``depth``; for example, a depth of 2 refers to the backing file of the backing file of *FILENAME*.qapi/block-core.json +5
-2@@ -261,6 +261,9 @@ # images in the chain)) before reaching one for which the # range is allocated # +# @present: true if this layer provides the data, false if adding a backing +# layer could impact this region (since 6.1) +# # @offset: if present, the image file stores the data for this range # in raw format at the given (host) offset #@@ -271,8 +274,8 @@ ## { 'struct': 'MapEntry', 'data': {'start': 'int', 'length': 'int', 'data': 'bool', - 'zero': 'bool', 'depth': 'int', '*offset': 'int', - '*filename': 'str' } } + 'zero': 'bool', 'depth': 'int', 'present': 'bool', + '*offset': 'int', '*filename': 'str' } } ## # @BlockdevCacheInfo:qemu-img.c +0
-0
qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985
Quoted 8 lines
qemu-img.c +3
-2@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,break ;case OFORMAT_JSON : printf("{ \"start\": %"PRId64", \"length\": %"PRId64"," - " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s", - e->start, e->length, e->depth, + " \"depth\": %"PRId64", \"present\": %s, \"zero\": %s," + "\"data\": %s", e->start, e->length, e->depth, + e->present ? "true" : "false",
Didn't you want to put present at the end? Still, this shouldn't be significant. And it make sense to keep present, zero and data together.
You missied a whitespace after '"zero": %s,', which is obvious from further test diff hunks.
With it fixed:
Quoted 24 lines
qemu-img.c +2
-0e->zero ?"true" :"false" , e->data ? "true" : "false");if (e->has_offset) {@@ -3047,6 +3048,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, .offset = map, .has_offset = has_offset, .depth = depth, + .present = !!(ret & BDRV_BLOCK_ALLOCATED), .has_filename = filename, .filename = filename, };@@ -3062,6 +3064,7 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)if (curr->zero != next->zero || curr->data != next->data || curr->depth != next->depth || + curr->present != next->present || curr->has_filename != next->has_filename || curr->has_offset != next->has_offset) {return false;tests/qemu-iotests/122.out +0
-0@@ -67,12 +67,12 @@ read 65536/65536 bytes at offset 4194304
[..]
Signature · 2 lines
Best regards, Vladimir
On Sat, Jul 03, 2021 at 10:25:28AM +0300, Vladimir Sementsov-Ogievskiy wrote:
...
Quoted 9 lines
Quoted 8 lines
An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy.
Signed-off-by Eric Blake <[email protected]>
Quoted 10 lines
docs/tools/qemu-img.rst Command description: · line 602
Quoted 8 lines
+++
b/docs/tools/qemu-img.rstdocs/tools/qemu-img.rst +3
-0@@ -597,6 +597,9 @@ Command description: if false, the sectors are either unallocated or stored as optimized all-zero clusters); - whether the data is known to read as zero (boolean field ``zero``); + - whether the data is actually present (boolean field ``present``); + if false, rebasing the backing chain onto a deeper file would pick + up data from the deeper file;Preexisting, but rather strange style of documentation, when described option doesn't go first in the paragraph..
Yeah. I'll send a followup email with a rewording of those paragraphs for consideration.
Quoted 4 lines
Quoted 4 lines
qemu-img.c +1
-0+++ b/qemu-img.c@@ -2980,8 +2980,9 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,break ;case OFORMAT_JSON :
Quoted 8 lines
qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985
Quoted 6 lines
qemu-img.c +3
-2printf("{ \"start\": %"PRId64", \"length\": %"PRId64"," - " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s", - e->start, e->length, e->depth, + " \"depth\": %"PRId64", \"present\": %s, \"zero\": %s," + "\"data\": %s", e->start, e->length, e->depth, + e->present ? "true" : "false",Didn't you want to put present at the end? Still, this shouldn't be significant. And it make sense to keep present, zero and data together.
I wanted it before anything optional, which "offset" is, so it already can't be at the end. If I understood Nir correctly, it was more important to always be present (it's easy to write a parser that searches for terms in the same position, and tolerates a missing term from an older verseion, but harder to parse a term that might or might not be present).
Quoted 5 lines
You missied a whitespace after '"zero": %s,', which is obvious from further test diff hunks.
With it fixed:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Thanks for catching that. I've updated that, and will queue through my NBD tree.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
[PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs
In reply to Eric BlakeReword the paragraphs to list the JSON key first, rather than in the middle of prose.
--- docs/tools/qemu-img.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
docs/tools/qemu-img.rst +10 -10
@@ -593,16 +593,16 @@ Command description: the ``start``, ``length``, ``offset`` fields; it will also include other more specific information: - - whether the sectors contain actual data or not (boolean field ``data``; - if false, the sectors are either unallocated or stored as optimized - all-zero clusters); - - whether the data is known to read as zero (boolean field ``zero``); - - whether the data is actually present (boolean field ``present``); - if false, rebasing the backing chain onto a deeper file would pick - up data from the deeper file; - - in order to make the output shorter, the target file is expressed as - a ``depth``; for example, a depth of 2 refers to the backing file - of the backing file of *FILENAME*. + - boolean field ``data``: true if the sectors contain actual data, + false if the sectors are either unallocated or stored as optimized + all-zero clusters + - boolean field ``zero``: true if the data is known to read as zero + - boolean field ``present``: true if the data belongs to the backing + chain, false if rebasing the backing chain onto a deeper file + would pick up data from the deeper file; + - integer field ``depth``: the depth within the backing chain at + which the data was resolved; for example, a depth of 2 refers to + the backing file of the backing file of *FILENAME*. In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.
Signature · 1 line
2.31.1
Nir Soffer
Reviewed-byRe: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs
Quoted 42 lines · Eric Blake
Reword the paragraphs to list the JSON key first, rather than in the middle of prose.
Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)docs/tools/qemu-img.rst +10
-11@@ -593,16 +593,16 @@ Command description: the ``start``, ``length``, ``offset`` fields; it will also include other more specific information: - - whether the sectors contain actual data or not (boolean field ``data``; - if false, the sectors are either unallocated or stored as optimized - all-zero clusters); - - whether the data is known to read as zero (boolean field ``zero``); - - whether the data is actually present (boolean field ``present``); - if false, rebasing the backing chain onto a deeper file would pick - up data from the deeper file; - - in order to make the output shorter, the target file is expressed as - a ``depth``; for example, a depth of 2 refers to the backing file - of the backing file of *FILENAME*. + - boolean field ``data``: true if the sectors contain actual data, + false if the sectors are either unallocated or stored as optimized + all-zero clusters + - boolean field ``zero``: true if the data is known to read as zero + - boolean field ``present``: true if the data belongs to the backing + chain, false if rebasing the backing chain onto a deeper file + would pick up data from the deeper file; + - integer field ``depth``: the depth within the backing chain at + which the data was resolved; for example, a depth of 2 refers to + the backing file of the backing file of *FILENAME*. In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error. --
Quoted 1 line
2.31.1
Would be nice if this could be generated from the json schema instead of repeating the type and description of the fields, but this is a nice improvement.
Vladimir Sementsov-Ogievskiy
Reviewed-byRe: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs
In reply to Eric BlakeQuoted 5 lines · Eric Blake
Reword the paragraphs to list the JSON key first, rather than in the middle of prose.
Suggested-by Vladimir Sementsov-Ogievskiy<[email protected]>Signed-off-by Eric Blake<[email protected]>
Signature · 2 lines
Best regards, Vladimir