qemu-img: Use "depth":-1 to make backing probes obvious
PATCH 1/1 qemu-img: Use "depth":-1 to make backing probes obvious
The recently-added NBD context qemu:allocation-depth makes an obvious case for why it is important to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' visually distinguish between clusters that have a local allocation from those that are found nowhere in the chain, by adding "depth":-1 as the new witness of data that could not be tied to a specific backing image. Several iotests are impacted, but glancing through the changes shows that it is an improvement in that it shows more accurate details.
Note that the documentation is specifically worded to allow qemu-img to report "depth":-1 both in the scenario where the last file in the backing chain still defers the cluster (corresponding to BDRV_BLOCK_ALLOCATED not being set anywhere in the chain), and in the scenario where qemu is unable to determine which backing chain element (if any) provides the data. The latter case does not exist now, but I'm considering an upcoming patch to add a BDRV_BLOCK_BACKING that would let a specific driver (such as NBD) inform the block layer that it is known that a cluster comes from a backing layer, but where there is insufficient data to determine which layer.
As a quick demonstration:
# Create a qcow2 image with a raw backing file:
$ qemu-img create base.raw $((4*64*1024))
$ qemu-img create -f qcow2 -b base.raw -F raw top.qcow2
# Write to first 3 clusters of base:
$ qemu-io -f raw -c "w -P 65 0 64k" -c "w -P 66 64k 64k" \
-c "w -P 67 128k 64k" base.raw
# Write to second and third clusters of top, hiding base:
$ qemu-io -f qcow2 -c "w -P 69 64k 64k" -c "w -z 128k 64k" top.qcow2
# Examine the full backing chain
$ qemu-img map --output=json -f qcow2 top.qcow2
[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": true, "offset": 0},
{ "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680},
{ "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 196608, "length": 65536, "depth": 1, "zero": true, "data": false, "offset": 196608}]
# Repeat, but with the backing chain clamped. Pre-patch:
$ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \
"backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}'
[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680},
{ "start": 131072, "length": 131072, "depth": 0, "zero": true, "data": false}]
# Repeat, but post-patch:
$ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \
"backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}'
[{ "start": 0, "length": 65536, "depth": -1, "zero": true, "data": false},
{ "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680},
{ "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 196608, "length": 65536, "depth": -1, "zero": true, "data": false}]Note that pre-patch, it was impossible to determine which portions of the qcow2 file override the backing file because the "depth":0 regions were combined, so even though qemu internally can tell the difference between sclusters 2 and 3, the command line user could not. But post-patch, the "depth":-1 markings match the "depth":1 markings when the backing chain is intact, and it becomes obvious which clusters are important.
---
If desired, I can send a followup patch that omits "depth":-1 output from qemu-img output instead (that's another couple lines tweak to the sources, plus re-touching all of the same affected tests).
docs/tools/qemu-img.rst | 7 ++- qemu-img.c | 3 ++ tests/qemu-iotests/122.out | 34 ++++++------- tests/qemu-iotests/154.out | 100 +++++++++++++++++++------------------ tests/qemu-iotests/179.out | 67 +++++++++++++++++++++---- tests/qemu-iotests/223.out | 24 ++++----- tests/qemu-iotests/244.out | 7 +-- tests/qemu-iotests/252.out | 4 +- tests/qemu-iotests/274.out | 16 +++--- tests/qemu-iotests/291.out | 8 +-- tests/qemu-iotests/309.out | 4 +- 11 files changed, 165 insertions(+), 109 deletions(-)
docs/tools/qemu-img.rst +5 -2
@@ -598,8 +598,11 @@ Command description: all-zero clusters); - whether the data is known to read as zero (boolean field ``zero``); - in order to make the output shorter, the target file is expressed as - a ``depth``; for example, a depth of 2 refers to the backing file - of the backing file of *FILENAME*. + a ``depth``; a depth of 0 means the data comes from the given + file, a positive depth means the data comes from a specific file + in the backing chain, and a depth of -1 merely implies that the + data is not local, but no information was available on which + backing file (if any) provides the data. In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.
qemu-img.c +3 -0
@@ -3035,6 +3035,9 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,bdrv_refresh_filename (file); filename = file->filename; } +if (!(ret &BDRV_BLOCK_ALLOCATED )) { + depth =-1 ; + } *e = (MapEntry ) { .start = offset,
tests/qemu-iotests/122.out +17 -17
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304 read 65536/65536 bytes at offset 8388608 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 65536, "length": 4128768, "depth": -1, "zero": true, "data": false}, { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 4259840, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 4259840, "length": 4128768, "depth": -1, "zero": true, "data": false}, { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 8454144, "length": 4128768, "depth": 0, "zero": true, "data": false}] +{ "start": 8454144, "length": 4128768, "depth": -1, "zero": true, "data": false}] read 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 65536/65536 bytes at offset 4194304
@@ -95,11 +95,11 @@ wrote 1024/1024 bytes at offset 1046528 wrote 1024/1024 bytes at offset 0 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 65536, "depth": 0, "zero": true, "data": false}, +{ "start": 65536, "length": 65536, "depth": -1, "zero": true, "data": false}, { "start": 131072, "length": 196608, "depth": 0, "zero": false, "data": true}, -{ "start": 327680, "length": 655360, "depth": 0, "zero": true, "data": false}, +{ "start": 327680, "length": 655360, "depth": -1, "zero": true, "data": false}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 1048576, "length": 1046528, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 1046528, "depth": -1, "zero": true, "data": false}] read 16384/16384 bytes at offset 0 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 16384/16384 bytes at offset 16384
@@ -195,31 +195,31 @@ wrote 1024/1024 bytes at offset 17408 convert -S 4k [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 4096, "length": 4096, "depth": 0, "zero": true, "data": false}, +{ "start": 4096, "length": 4096, "depth": -1, "zero": true, "data": false}, { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 12288, "length": 4096, "depth": 0, "zero": true, "data": false}, +{ "start": 12288, "length": 4096, "depth": -1, "zero": true, "data": false}, { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 20480, "length": 67088384, "depth": 0, "zero": true, "data": false}] +{ "start": 20480, "length": 67088384, "depth": -1, "zero": true, "data": false}] convert -c -S 4k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": -1, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": -1, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": -1, "zero": true, "data": false}] convert -S 8k [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 24576, "length": 67084288, "depth": 0, "zero": true, "data": false}] +{ "start": 24576, "length": 67084288, "depth": -1, "zero": true, "data": false}] convert -c -S 8k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": -1, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": -1, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": -1, "zero": true, "data": false}] === -n to a non-zero image ===
@@ -235,7 +235,7 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 [{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +[{ "start": 0, "length": 67108864, "depth": -1, "zero": true, "data": false}] === -n to an empty image with a backing file ===
tests/qemu-iotests/154.out +52 -48
279 lines not on this page · raw message
tests/qemu-iotests/179.out +56 -11
127 lines not on this page · raw message
tests/qemu-iotests/223.out +12 -12
@@ -103,16 +103,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": -1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": -1, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] === End qemu NBD server ===
@@ -204,16 +204,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": -1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": -1, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] === End qemu NBD server ===
@@ -238,12 +238,12 @@ read 2097152/2097152 bytes at offset 2097152 === Use qemu-nbd as server === -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": -1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": -1, "zero": false, "data": false}, { "start": 1024, "length": 11321, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 12345, "length": 2084807, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": -1, "zero": false, "data": false}] *** done
tests/qemu-iotests/244.out +4 -3
@@ -57,11 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false}, +[{ "start": 0, "length": 1048576, "depth": -1, "zero": true, "data": false}, { "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, { "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, { "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, -{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}] +{ "start": 5242880, "length": 1048576, "depth": 0, "zero": true, "data": false}, +{ "start": 6291456, "length": 60817408, "depth": -1, "zero": true, "data": false}] read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -123,7 +124,7 @@ read 1048576/1048576 bytes at offset 0 Offset Length Mapped to File 0 0x100000 0 TEST_DIR/t.qcow2.data [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0}, -{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 66060288, "depth": -1, "zero": true, "data": false}] === Copy offloading ===
tests/qemu-iotests/252.out +2 -2
@@ -24,7 +24,7 @@ read 131072/131072 bytes at offset 262144 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 262144, "length": 131072, "depth": 0, "zero": true, "data": false}] +{ "start": 262144, "length": 131072, "depth": -1, "zero": true, "data": false}] read 131072/131072 bytes at offset 0 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -35,5 +35,5 @@ read 131072/131072 bytes at offset 262144 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false}, -{ "start": 327680, "length": 65536, "depth": 1, "zero": true, "data": false}] +{ "start": 327680, "length": 65536, "depth": -1, "zero": true, "data": false}] *** done
tests/qemu-iotests/274.out +8 -8
@@ -37,7 +37,7 @@ Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base [{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 1048576, "depth": -1, "zero": true, "data": false}] Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base
@@ -220,7 +220,7 @@ read 65536/65536 bytes at offset 5368709120 1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0) 7 GiB (0x1c0000000) bytes allocated at offset 1 GiB (0x40000000) -[{ "start": 0, "length": 1073741824, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 1073741824, "depth": -1, "zero": true, "data": false}, { "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}] === preallocation=metadata ===
@@ -239,7 +239,7 @@ read 65536/65536 bytes at offset 33285996544 30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0) 3 GiB (0xc0000000) bytes allocated at offset 30 GiB (0x780000000) -[{ "start": 0, "length": 32212254720, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 32212254720, "depth": -1, "zero": true, "data": false}, { "start": 32212254720, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 327680}, { "start": 32749125632, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 537264128}, { "start": 33285996544, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 1074200576},
@@ -263,7 +263,7 @@ read 65536/65536 bytes at offset 9437184 5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0) 10 MiB (0xa00000) bytes allocated at offset 5 MiB (0x500000) -[{ "start": 0, "length": 5242880, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 5242880, "depth": -1, "zero": true, "data": false}, { "start": 5242880, "length": 10485760, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=full ===
@@ -282,7 +282,7 @@ read 65536/65536 bytes at offset 11534336 8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0) 4 MiB (0x400000) bytes allocated at offset 8 MiB (0x800000) -[{ "start": 0, "length": 8388608, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 8388608, "depth": -1, "zero": true, "data": false}, { "start": 8388608, "length": 4194304, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=off ===
@@ -301,7 +301,7 @@ read 65536/65536 bytes at offset 259072 192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0) 320 KiB (0x50000) bytes allocated at offset 192 KiB (0x30000) -[{ "start": 0, "length": 196608, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 196608, "depth": -1, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
@@ -321,7 +321,7 @@ read 65536/65536 bytes at offset 344064 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 256 KiB (0x40000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": -1, "zero": true, "data": false}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}] === preallocation=off ===
@@ -340,6 +340,6 @@ read 65536/65536 bytes at offset 446464 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 244 KiB (0x3d000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": -1, "zero": true, "data": false}, { "start": 262144, "length": 249856, "depth": 0, "zero": true, "data": false}]
tests/qemu-iotests/291.out +4 -4
@@ -104,15 +104,15 @@ Format specific information: === Check bitmap contents === [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 3145728, "length": 1048576, "depth": -1, "zero": false, "data": false}, { "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 1048576, "length": 1048576, "depth": -1, "zero": false, "data": false}, { "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": -1, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": -1, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
tests/qemu-iotests/309.out +2 -2
@@ -13,10 +13,10 @@ wrote 2097152/2097152 bytes at offset 1048576 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}, { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}, -{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}] +{ "start": 3145728, "length": 1048576, "depth": -1, "zero": true, "data": false}] [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false}, +{ "start": 1048576, "length": 2097152, "depth": -1, "zero": false, "data": false}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
Signature · 1 line
2.31.1
Quoted 87 lines · Eric Blake
The recently-added NBD context qemu:allocation-depth makes an obvious case for why it is important to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED(which defers to a [missing] backing file) and aQCOW2_CLUSTER_ZERO_PLAINcluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish aQCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).The task of reconstructing a qcow2 chain was made harder in commit
0da9856851(nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted inBDRV_BLOCK_ALLOCATED(showing up asNBD_STATE_ZEROin isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.An obvious solution is to make 'qemu-img map --output=json' visually distinguish between clusters that have a local allocation from those that are found nowhere in the chain, by adding "depth":-1 as the new witness of data that could not be tied to a specific backing image. Several iotests are impacted, but glancing through the changes shows that it is an improvement in that it shows more accurate details.
Note that the documentation is specifically worded to allow qemu-img to report "depth":-1 both in the scenario where the last file in the backing chain still defers the cluster (corresponding to
BDRV_BLOCK_ALLOCATEDnot being set anywhere in the chain), and in the scenario where qemu is unable to determine which backing chain element (if any) provides the data. The latter case does not exist now, but I'm considering an upcoming patch to add aBDRV_BLOCK_BACKINGthat would let a specific driver (such as NBD) inform the block layer that it is known that a cluster comes from a backing layer, but where there is insufficient data to determine which layer.As a quick demonstration:
# Create a qcow2 image with a raw backing file: $ qemu-img create base.raw $((4*64*1024)) $ qemu-img create -f qcow2 -b base.raw -F raw top.qcow2 # Write to first 3 clusters of base: $ qemu-io -f raw -c "w -P 65 0 64k" -c "w -P 66 64k 64k" \ -c "w -P 67 128k 64k" base.raw # Write to second and third clusters of top, hiding base: $ qemu-io -f qcow2 -c "w -P 69 64k 64k" -c "w -z 128k 64k" top.qcow2 # Examine the full backing chain $ qemu-img map --output=json -f qcow2 top.qcow2 [{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": true, "offset": 0}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": 1, "zero": true, "data": false, "offset": 196608}] # Repeat, but with the backing chain clamped. Pre-patch: $ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \ "backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}' [{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 131072, "depth": 0, "zero": true, "data": false}] # Repeat, but post-patch: $ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \ "backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}' [{ "start": 0, "length": 65536, "depth": -1, "zero": true, "data": false}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": -1, "zero": true, "data": false}]Note that pre-patch, it was impossible to determine which portions of the qcow2 file override the backing file because the "depth":0 regions were combined, so even though qemu internally can tell the difference between sclusters 2 and 3, the command line user could not. But post-patch, the "depth":-1 markings match the "depth":1 markings when the backing chain is intact, and it becomes obvious which clusters are important.
Quoted 1 line
Signed-off-by Eric Blake <[email protected]>
I'm not sure do we need a kind of new option or deprecation period for such incompatible change..
Personally I'm OK with it and like the idea:
Quoted 5 lines
---
If desired, I can send a followup patch that omits "depth":-1 output from qemu-img output instead (that's another couple lines tweak to the sources, plus re-touching all of the same affected tests).
[..]
Quoted 9 lines
docs/tools/qemu-img.rst +1
-0+ backing file (if any) provides the data. In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.qemu-img.c +0
-0
qemu-img.c static int get_block_status(BlockDriverState *bs, int64_t offset, · line 3040
Quoted 6 lines
qemu-img.c +3
-0@@ -3035,6 +3035,9 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,bdrv_refresh_filename (file); filename = file->filename; } +if (!(ret &BDRV_BLOCK_ALLOCATED )) { + depth =-1 ; + }
may be more obvious would be set depth = -1 in "if (bs == NULL) {" where ret becomes 0 for this case.
Quoted 19 lines
qemu-img.c +0
-0*e = (MapEntry ) { .start = offset,tests/qemu-iotests/122.out +3
-3@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304 read 65536/65536 bytes at offset 8388608 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 65536, "length": 4128768, "depth": -1, "zero": true, "data": false}, { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 4259840, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 4259840, "length": 4128768, "depth": -1, "zero": true, "data": false}, { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 8454144, "length": 4128768, "depth": 0, "zero": true, "data": false}] +{ "start": 8454144, "length": 4128768, "depth": -1, "zero": true, "data": false}]
[..]
Signature · 2 lines
Best regards, Vladimir
Quoted 88 lines · Vladimir Sementsov-Ogievskiy
Quoted 87 lines · Eric Blake
The recently-added NBD context qemu:allocation-depth makes an obvious case for why it is important to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED(which defers to a [missing] backing file) and aQCOW2_CLUSTER_ZERO_PLAINcluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish aQCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).The task of reconstructing a qcow2 chain was made harder in commit
0da9856851(nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted inBDRV_BLOCK_ALLOCATED(showing up asNBD_STATE_ZEROin isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.An obvious solution is to make 'qemu-img map --output=json' visually distinguish between clusters that have a local allocation from those that are found nowhere in the chain, by adding "depth":-1 as the new witness of data that could not be tied to a specific backing image. Several iotests are impacted, but glancing through the changes shows that it is an improvement in that it shows more accurate details.
Note that the documentation is specifically worded to allow qemu-img to report "depth":-1 both in the scenario where the last file in the backing chain still defers the cluster (corresponding to
BDRV_BLOCK_ALLOCATEDnot being set anywhere in the chain), and in the scenario where qemu is unable to determine which backing chain element (if any) provides the data. The latter case does not exist now, but I'm considering an upcoming patch to add aBDRV_BLOCK_BACKINGthat would let a specific driver (such as NBD) inform the block layer that it is known that a cluster comes from a backing layer, but where there is insufficient data to determine which layer.As a quick demonstration:
# Create a qcow2 image with a raw backing file:
$ qemu-img create base.raw $((4*64*1024)) $ qemu-img create -f qcow2 -b base.raw -F raw top.qcow2# Write to first 3 clusters of base:
$ qemu-io -f raw -c "w -P 65 0 64k" -c "w -P 66 64k 64k" \ -c "w -P 67 128k 64k" base.raw# Write to second and third clusters of top, hiding base:
$ qemu-io -f qcow2 -c "w -P 69 64k 64k" -c "w -z 128k 64k" top.qcow2# Examine the full backing chain
$ qemu-img map --output=json -f qcow2 top.qcow2 [{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": true, "offset": 0}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": 1, "zero": true, "data": false, "offset": 196608}]# Repeat, but with the backing chain clamped. Pre-patch:
$ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \ "backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}' [{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 131072, "depth": 0, "zero": true, "data": false}]# Repeat, but post-patch:
$ qemu-img map --output=json -f qcow2 json:'{"driver":"qcow2", \ "backing":null, "file":{"driver":"file", "filename":"top.qcow2"}}' [{ "start": 0, "length": 65536, "depth": -1, "zero": true, "data": false}, { "start": 65536, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 131072, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": -1, "zero": true, "data": false}]Note that pre-patch, it was impossible to determine which portions of the qcow2 file override the backing file because the "depth":0 regions were combined, so even though qemu internally can tell the difference between sclusters 2 and 3, the command line user could not. But post-patch, the "depth":-1 markings match the "depth":1 markings when the backing chain is intact, and it becomes obvious which clusters are important.
Quoted 3 lines
Quoted 1 line
Signed-off-by Eric Blake <[email protected]>I'm not sure do we need a kind of new option or deprecation period for such incompatible change..
I see Kevin worries about same thing..
So, probably better is just add a new optional boolean field, that shows that data is "nowhere in the chain". Such change is a lot more compatible.
Quoted 3 lines
Personally I'm OK with it and like the idea:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Signature · 2 lines
Best regards, Vladimir