qemu-img: Make unallocated part of backing chain obvious in map
PATCH v2 1/1 qemu-img: Make unallocated part of backing chain obvious in map
The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and prior to this patch has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).
The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.
An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.
---
Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)
Use N+1 instead of -1 for unallocated [Kevin]
docs/tools/qemu-img.rst | 3 +- qemu-img.c | 3 ++ tests/qemu-iotests/122.out | 34 ++++++------- tests/qemu-iotests/154.out | 100 +++++++++++++++++++------------------ tests/qemu-iotests/179.out | 67 +++++++++++++++++++++---- tests/qemu-iotests/223.out | 24 ++++----- tests/qemu-iotests/244.out | 7 +-- tests/qemu-iotests/252.out | 4 +- tests/qemu-iotests/274.out | 16 +++--- tests/qemu-iotests/291.out | 8 +-- tests/qemu-iotests/309.out | 4 +- 11 files changed, 162 insertions(+), 108 deletions(-)
docs/tools/qemu-img.rst +2 -1
@@ -599,7 +599,8 @@ Command description: - whether the data is known to read as zero (boolean field ``zero``); - in order to make the output shorter, the target file is expressed as a ``depth``; for example, a depth of 2 refers to the backing file - of the backing file of *FILENAME*. + of the backing file of *FILENAME*. Depth will be one larger than + the chain length if no file in the chain provides the data. In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.
qemu-img.c +3 -0
@@ -3035,6 +3035,9 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,bdrv_refresh_filename (file); filename = file->filename; } +if (!(ret &BDRV_BLOCK_ALLOCATED )) { + depth++; + } *e = (MapEntry ) { .start = offset,
tests/qemu-iotests/122.out +17 -17
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304 read 65536/65536 bytes at offset 8388608 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false}, { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 4259840, "length": 4128768, "depth": 0, "zero": true, "data": false}, +{ "start": 4259840, "length": 4128768, "depth": 1, "zero": true, "data": false}, { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 8454144, "length": 4128768, "depth": 0, "zero": true, "data": false}] +{ "start": 8454144, "length": 4128768, "depth": 1, "zero": true, "data": false}] read 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 65536/65536 bytes at offset 4194304
@@ -95,11 +95,11 @@ wrote 1024/1024 bytes at offset 1046528 wrote 1024/1024 bytes at offset 0 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 65536, "depth": 0, "zero": true, "data": false}, +{ "start": 65536, "length": 65536, "depth": 1, "zero": true, "data": false}, { "start": 131072, "length": 196608, "depth": 0, "zero": false, "data": true}, -{ "start": 327680, "length": 655360, "depth": 0, "zero": true, "data": false}, +{ "start": 327680, "length": 655360, "depth": 1, "zero": true, "data": false}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 1048576, "length": 1046528, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 1046528, "depth": 1, "zero": true, "data": false}] read 16384/16384 bytes at offset 0 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 16384/16384 bytes at offset 16384
@@ -195,31 +195,31 @@ wrote 1024/1024 bytes at offset 17408 convert -S 4k [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 4096, "length": 4096, "depth": 0, "zero": true, "data": false}, +{ "start": 4096, "length": 4096, "depth": 1, "zero": true, "data": false}, { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 12288, "length": 4096, "depth": 0, "zero": true, "data": false}, +{ "start": 12288, "length": 4096, "depth": 1, "zero": true, "data": false}, { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 20480, "length": 67088384, "depth": 0, "zero": true, "data": false}] +{ "start": 20480, "length": 67088384, "depth": 1, "zero": true, "data": false}] convert -c -S 4k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}] convert -S 8k [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 24576, "length": 67084288, "depth": 0, "zero": true, "data": false}] +{ "start": 24576, "length": 67084288, "depth": 1, "zero": true, "data": false}] convert -c -S 8k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}] === -n to a non-zero image ===
@@ -235,7 +235,7 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 [{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] +[{ "start": 0, "length": 67108864, "depth": 1, "zero": true, "data": false}] === -n to an empty image with a backing file ===
tests/qemu-iotests/154.out +52 -48
279 lines not on this page · raw message
tests/qemu-iotests/179.out +56 -11
127 lines not on this page · raw message
tests/qemu-iotests/223.out +12 -12
@@ -103,16 +103,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] === End qemu NBD server ===
@@ -204,16 +204,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] === End qemu NBD server ===
@@ -238,12 +238,12 @@ read 2097152/2097152 bytes at offset 2097152 === Use qemu-nbd as server === -[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, { "start": 1024, "length": 11321, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 12345, "length": 2084807, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] *** done
tests/qemu-iotests/244.out +4 -3
@@ -57,11 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false}, +[{ "start": 0, "length": 1048576, "depth": 1, "zero": true, "data": false}, { "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, { "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, { "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, -{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}] +{ "start": 5242880, "length": 1048576, "depth": 0, "zero": true, "data": false}, +{ "start": 6291456, "length": 60817408, "depth": 1, "zero": true, "data": false}] read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -123,7 +124,7 @@ read 1048576/1048576 bytes at offset 0 Offset Length Mapped to File 0 0x100000 0 TEST_DIR/t.qcow2.data [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0}, -{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 66060288, "depth": 1, "zero": true, "data": false}] === Copy offloading ===
tests/qemu-iotests/252.out +2 -2
@@ -24,7 +24,7 @@ read 131072/131072 bytes at offset 262144 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 262144, "length": 131072, "depth": 0, "zero": true, "data": false}] +{ "start": 262144, "length": 131072, "depth": 1, "zero": true, "data": false}] read 131072/131072 bytes at offset 0 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -35,5 +35,5 @@ read 131072/131072 bytes at offset 262144 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false}, -{ "start": 327680, "length": 65536, "depth": 1, "zero": true, "data": false}] +{ "start": 327680, "length": 65536, "depth": 2, "zero": true, "data": false}] *** done
tests/qemu-iotests/274.out +8 -8
@@ -37,7 +37,7 @@ Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base [{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}] +{ "start": 1048576, "length": 1048576, "depth": 1, "zero": true, "data": false}] Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base
@@ -220,7 +220,7 @@ read 65536/65536 bytes at offset 5368709120 1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0) 7 GiB (0x1c0000000) bytes allocated at offset 1 GiB (0x40000000) -[{ "start": 0, "length": 1073741824, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 1073741824, "depth": 2, "zero": true, "data": false}, { "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}] === preallocation=metadata ===
@@ -239,7 +239,7 @@ read 65536/65536 bytes at offset 33285996544 30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0) 3 GiB (0xc0000000) bytes allocated at offset 30 GiB (0x780000000) -[{ "start": 0, "length": 32212254720, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 32212254720, "depth": 2, "zero": true, "data": false}, { "start": 32212254720, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 327680}, { "start": 32749125632, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 537264128}, { "start": 33285996544, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 1074200576},
@@ -263,7 +263,7 @@ read 65536/65536 bytes at offset 9437184 5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0) 10 MiB (0xa00000) bytes allocated at offset 5 MiB (0x500000) -[{ "start": 0, "length": 5242880, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 5242880, "depth": 2, "zero": true, "data": false}, { "start": 5242880, "length": 10485760, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=full ===
@@ -282,7 +282,7 @@ read 65536/65536 bytes at offset 11534336 8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0) 4 MiB (0x400000) bytes allocated at offset 8 MiB (0x800000) -[{ "start": 0, "length": 8388608, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 8388608, "depth": 2, "zero": true, "data": false}, { "start": 8388608, "length": 4194304, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=off ===
@@ -301,7 +301,7 @@ read 65536/65536 bytes at offset 259072 192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0) 320 KiB (0x50000) bytes allocated at offset 192 KiB (0x30000) -[{ "start": 0, "length": 196608, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 196608, "depth": 2, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
@@ -321,7 +321,7 @@ read 65536/65536 bytes at offset 344064 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 256 KiB (0x40000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}] === preallocation=off ===
@@ -340,6 +340,6 @@ read 65536/65536 bytes at offset 446464 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 244 KiB (0x3d000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false}, { "start": 262144, "length": 249856, "depth": 0, "zero": true, "data": false}]
tests/qemu-iotests/291.out +4 -4
@@ -104,15 +104,15 @@ Format specific information: === Check bitmap contents === [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 3145728, "length": 1048576, "depth": 1, "zero": false, "data": false}, { "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 1048576, "length": 1048576, "depth": 1, "zero": false, "data": false}, { "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
tests/qemu-iotests/309.out +2 -2
@@ -13,10 +13,10 @@ wrote 2097152/2097152 bytes at offset 1048576 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}, { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}, -{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}] +{ "start": 3145728, "length": 1048576, "depth": 2, "zero": true, "data": false}] [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false}, +{ "start": 1048576, "length": 2097152, "depth": 1, "zero": false, "data": false}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
Signature · 1 line
2.31.1
Quoted 41 lines · Eric Blake
The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and prior to this patch has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED(which defers to a [missing] backing file) and aQCOW2_CLUSTER_ZERO_PLAINcluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish aQCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).The task of reconstructing a qcow2 chain was made harder in commit
0da9856851(nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted inBDRV_BLOCK_ALLOCATED(showing up asNBD_STATE_ZEROin isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.
Quoted 8 lines
Signed-off-by Eric Blake <[email protected]>---
Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)
Use N+1 instead of -1 for unallocated [Kevin]
Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2 UNALLOCATED_ZERO cluster.
So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\
Signature · 2 lines
Best regards, Vladimir
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 18 lines
An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.
Signed-off-by Eric Blake <[email protected]>---
Quoted 7 lines
Quoted 5 lines
Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)
Use N+1 instead of -1 for unallocated [Kevin]
Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2
UNALLOCATED_ZEROcluster.
Using just 'qemu-img map --output-json', you only see depth numbers. You also have to use 'qemu-img info --backing-chain' to see what file those depth numbers correspond to, at which point it becomes obvious whether "depth":4 meant unallocated (because the chain was length 3) or allocated at depth 4 (because the chain was length 4 or longer). But that's no worse than pre-patch, where you had to use qemu-img info --backing-chain to learn which file a particular "depth" maps to.
Quoted 2 lines
So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\
It should be fairly easy to optimize the case of a backing chain where EVERY listed cluster at the final depth was "data":false,"zero":true to omit that file after all.
And in oVirt's case, Nir pointed out that we have one more tool at our disposal in recreating a backing chain: if you use json:{"driver":"qcow2", "backing":null, ...} as your image file, you don't have to worry about arbitrary files in the backing chain, only about recreating the top-most layer of a chain. And in that case, it becomes very obvious that "depth":0 is something you must recreate, and "depth":1 would be a non-existent backing file because you just passed "backing":null.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Quoted 43 lines · Eric Blake
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 18 lines
An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.
Signed-off-by Eric Blake <[email protected]>---
Quoted 7 lines
Quoted 5 lines
Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)
Use N+1 instead of -1 for unallocated [Kevin]
Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2
UNALLOCATED_ZEROcluster.Using just 'qemu-img map --output-json', you only see depth numbers. You also have to use 'qemu-img info --backing-chain' to see what file those depth numbers correspond to, at which point it becomes obvious whether "depth":4 meant unallocated (because the chain was length 3) or allocated at depth 4 (because the chain was length 4 or longer). But that's no worse than pre-patch, where you had to use qemu-img info --backing-chain to learn which file a particular "depth" maps to.
Quoted 2 lines
So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\
It should be fairly easy to optimize the case of a backing chain where EVERY listed cluster at the final depth was "data":false,"zero":true to omit that file after all.
Quoted 8 lines
And in oVirt's case, Nir pointed out that we have one more tool at our disposal in recreating a backing chain: if you use json:{"driver":"qcow2", "backing":null, ...} as your image file, you don't have to worry about arbitrary files in the backing chain, only about recreating the top-most layer of a chain. And in that case, it becomes very obvious that "depth":0 is something you must recreate, and "depth":1 would be a non-existent backing file because you just passed "backing":null.
Note that oVirt does not use qemu-img map, we use qemu-nbd to get image extents, since it is used only in context we already connect to qemu-nbd server or run qemu-nbd.
Management tools already know the image format (they should avoid doing format probing anyway), and using a json uri allows single command to get the needed info when you inspect a single layer.
But this change introduces a risk that some program using qemu-img map will interrupt the result in the wrong way, assuming that there is N+1 layer.
I think adding a new flag for absent extents is better. It cannot break any user and it is easier to understand and use.
Nir
[PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
In reply to Eric BlakeTo save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
docs/tools/qemu-img.rst | 3 ++ qapi/block-core.json | 7 ++- qemu-img.c | 15 +++++- tests/qemu-iotests/122.out | 34 +++++++------- tests/qemu-iotests/154.out | 96 +++++++++++++++++++------------------- tests/qemu-iotests/179.out | 66 +++++++++++++------------- tests/qemu-iotests/223.out | 24 +++++----- tests/qemu-iotests/244.out | 6 +-- tests/qemu-iotests/252.out | 4 +- tests/qemu-iotests/274.out | 16 +++---- tests/qemu-iotests/291.out | 8 ++-- tests/qemu-iotests/309.out | 4 +- 12 files changed, 150 insertions(+), 133 deletions(-)
docs/tools/qemu-img.rst +3 -0
@@ -601,6 +601,9 @@ Command description: a ``depth``; for example, a depth of 2 refers to the backing file of the backing file of *FILENAME*. Depth will be one larger than the chain length if no file in the chain provides the data. + - an optional ``backing`` field is present with value true if no + file in the backing chain provides the data (making it easier to + identify when ``depth`` exceeds the chain length). In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.
qapi/block-core.json +5 -2
@@ -264,6 +264,9 @@ # @offset: if present, the image file stores the data for this range # in raw format at the given (host) offset # +# @backing: if present, the range is not allocated within the backing +# chain (since 6.1) +# # @filename: filename that is referred to by @offset # # Since: 2.6
@@ -271,8 +274,8 @@ ## { 'struct': 'MapEntry', 'data': {'start': 'int', 'length': 'int', 'data': 'bool', - 'zero': 'bool', 'depth': 'int', '*offset': 'int', - '*filename': 'str' } } + 'zero': 'bool', 'depth': 'int', '*backing': 'bool', + '*offset': 'int', '*filename': 'str' } } ## # @BlockdevCacheInfo:
qemu-img.c +13 -2
@@ -2977,8 +2977,13 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,break ;case OFORMAT_JSON :printf ("{ \"start\": %" PRId64", \"length\": %" PRId64"," -" \"depth\": %" PRId64", \"zero\": %s, \"data\": %s" , - e->start, e->length, e->depth, +" \"depth\": %" PRId64, e->start, e->length, e->depth); +if (e->has_backing) { +/* Backing should only be set at the end of the chain */ +assert (e->backing && e->depth >0 ); +printf (", \"backing\": true" ); + } +printf (", \"zero\": %s, \"data\": %s" , e->zero ?"true" :"false" , e->data ?"true" :"false" );if (e->has_offset) {
@@ -2999,6 +3004,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, {int ret;int depth; +bool backing = false;BlockDriverState *file;bool has_offset;int64_t map;
@@ -3037,6 +3043,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, }if (!(ret &BDRV_BLOCK_ALLOCATED )) { depth++; + backing = true; } *e = (MapEntry ) {
@@ -3047,6 +3054,8 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, .offset = map, .has_offset = has_offset, .depth = depth, + .has_backing = backing, + .backing = backing, .has_filename = filename, .filename = filename, };
@@ -3072,6 +3081,8 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)if (curr->has_offset && curr->offset + curr->length != next->offset) {return false; } +/* backing should only ever be set for identical depth */ +assert (curr->backing == next->backing);return true; }
tests/qemu-iotests/122.out +17 -17
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304 read 65536/65536 bytes at offset 8388608 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false}, +{ "start": 65536, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 4259840, "length": 4128768, "depth": 1, "zero": true, "data": false}, +{ "start": 4259840, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 8454144, "length": 4128768, "depth": 1, "zero": true, "data": false}] +{ "start": 8454144, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false}] read 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 65536/65536 bytes at offset 4194304
@@ -95,11 +95,11 @@ wrote 1024/1024 bytes at offset 1046528 wrote 1024/1024 bytes at offset 0 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 65536, "depth": 1, "zero": true, "data": false}, +{ "start": 65536, "length": 65536, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 131072, "length": 196608, "depth": 0, "zero": false, "data": true}, -{ "start": 327680, "length": 655360, "depth": 1, "zero": true, "data": false}, +{ "start": 327680, "length": 655360, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 1048576, "length": 1046528, "depth": 1, "zero": true, "data": false}] +{ "start": 1048576, "length": 1046528, "depth": 1, "backing": true, "zero": true, "data": false}] read 16384/16384 bytes at offset 0 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 16384/16384 bytes at offset 16384
@@ -195,31 +195,31 @@ wrote 1024/1024 bytes at offset 17408 convert -S 4k [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 4096, "length": 4096, "depth": 1, "zero": true, "data": false}, +{ "start": 4096, "length": 4096, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 12288, "length": 4096, "depth": 1, "zero": true, "data": false}, +{ "start": 12288, "length": 4096, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 20480, "length": 67088384, "depth": 1, "zero": true, "data": false}] +{ "start": 20480, "length": 67088384, "depth": 1, "backing": true, "zero": true, "data": false}] convert -c -S 4k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": 1, "backing": true, "zero": true, "data": false}] convert -S 8k [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 24576, "length": 67084288, "depth": 1, "zero": true, "data": false}] +{ "start": 24576, "length": 67084288, "depth": 1, "backing": true, "zero": true, "data": false}] convert -c -S 8k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false}, +{ "start": 1024, "length": 7168, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false}, +{ "start": 9216, "length": 8192, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true}, -{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}] +{ "start": 18432, "length": 67090432, "depth": 1, "backing": true, "zero": true, "data": false}] === -n to a non-zero image ===
@@ -235,7 +235,7 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 [{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}] Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -[{ "start": 0, "length": 67108864, "depth": 1, "zero": true, "data": false}] +[{ "start": 0, "length": 67108864, "depth": 1, "backing": true, "zero": true, "data": false}] === -n to an empty image with a backing file ===
tests/qemu-iotests/154.out +48 -48
279 lines not on this page · raw message
tests/qemu-iotests/179.out +33 -33
148 lines not on this page · raw message
tests/qemu-iotests/223.out +12 -12
@@ -103,16 +103,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] === End qemu NBD server ===
@@ -204,16 +204,16 @@ read 2097152/2097152 bytes at offset 2097152 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] -[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] === Contrast to small granularity dirty-bitmap === [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] === End qemu NBD server ===
@@ -238,12 +238,12 @@ read 2097152/2097152 bytes at offset 2097152 === Use qemu-nbd as server === -[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false}, +[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false}, +{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 1024, "length": 11321, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 12345, "length": 2084807, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}] +{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}] *** done
tests/qemu-iotests/244.out +3 -3
@@ -57,12 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. -[{ "start": 0, "length": 1048576, "depth": 1, "zero": true, "data": false}, +[{ "start": 0, "length": 1048576, "depth": 1, "backing": true, "zero": true, "data": false}, { "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, { "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, { "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, { "start": 5242880, "length": 1048576, "depth": 0, "zero": true, "data": false}, -{ "start": 6291456, "length": 60817408, "depth": 1, "zero": true, "data": false}] +{ "start": 6291456, "length": 60817408, "depth": 1, "backing": true, "zero": true, "data": false}] read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -124,7 +124,7 @@ read 1048576/1048576 bytes at offset 0 Offset Length Mapped to File 0 0x100000 0 TEST_DIR/t.qcow2.data [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0}, -{ "start": 1048576, "length": 66060288, "depth": 1, "zero": true, "data": false}] +{ "start": 1048576, "length": 66060288, "depth": 1, "backing": true, "zero": true, "data": false}] === Copy offloading ===
tests/qemu-iotests/252.out +2 -2
@@ -24,7 +24,7 @@ read 131072/131072 bytes at offset 262144 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 262144, "length": 131072, "depth": 1, "zero": true, "data": false}] +{ "start": 262144, "length": 131072, "depth": 1, "backing": true, "zero": true, "data": false}] read 131072/131072 bytes at offset 0 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -35,5 +35,5 @@ read 131072/131072 bytes at offset 262144 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false}, -{ "start": 327680, "length": 65536, "depth": 2, "zero": true, "data": false}] +{ "start": 327680, "length": 65536, "depth": 2, "backing": true, "zero": true, "data": false}] *** done
tests/qemu-iotests/274.out +8 -8
@@ -37,7 +37,7 @@ Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base [{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, -{ "start": 1048576, "length": 1048576, "depth": 1, "zero": true, "data": false}] +{ "start": 1048576, "length": 1048576, "depth": 1, "backing": true, "zero": true, "data": false}] Offset Length Mapped to File 0 0x100000 0x50000 TEST_DIR/PID-base
@@ -220,7 +220,7 @@ read 65536/65536 bytes at offset 5368709120 1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0) 7 GiB (0x1c0000000) bytes allocated at offset 1 GiB (0x40000000) -[{ "start": 0, "length": 1073741824, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 1073741824, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}] === preallocation=metadata ===
@@ -239,7 +239,7 @@ read 65536/65536 bytes at offset 33285996544 30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0) 3 GiB (0xc0000000) bytes allocated at offset 30 GiB (0x780000000) -[{ "start": 0, "length": 32212254720, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 32212254720, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 32212254720, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 327680}, { "start": 32749125632, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 537264128}, { "start": 33285996544, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 1074200576},
@@ -263,7 +263,7 @@ read 65536/65536 bytes at offset 9437184 5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0) 10 MiB (0xa00000) bytes allocated at offset 5 MiB (0x500000) -[{ "start": 0, "length": 5242880, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 5242880, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 5242880, "length": 10485760, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=full ===
@@ -282,7 +282,7 @@ read 65536/65536 bytes at offset 11534336 8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0) 4 MiB (0x400000) bytes allocated at offset 8 MiB (0x800000) -[{ "start": 0, "length": 8388608, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 8388608, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 8388608, "length": 4194304, "depth": 0, "zero": false, "data": true, "offset": 327680}] === preallocation=off ===
@@ -301,7 +301,7 @@ read 65536/65536 bytes at offset 259072 192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0) 320 KiB (0x50000) bytes allocated at offset 192 KiB (0x30000) -[{ "start": 0, "length": 196608, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 196608, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 196608, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
@@ -321,7 +321,7 @@ read 65536/65536 bytes at offset 344064 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 256 KiB (0x40000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}] === preallocation=off ===
@@ -340,6 +340,6 @@ read 65536/65536 bytes at offset 446464 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0) 244 KiB (0x3d000) bytes allocated at offset 256 KiB (0x40000) -[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false}, +[{ "start": 0, "length": 262144, "depth": 2, "backing": true, "zero": true, "data": false}, { "start": 262144, "length": 249856, "depth": 0, "zero": true, "data": false}]
tests/qemu-iotests/291.out +4 -4
@@ -104,15 +104,15 @@ Format specific information: === Check bitmap contents === [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 3145728, "length": 1048576, "depth": 1, "zero": false, "data": false}, +{ "start": 3145728, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 1048576, "depth": 1, "zero": false, "data": false}, +{ "start": 1048576, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false}, +{ "start": 2097152, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
tests/qemu-iotests/309.out +2 -2
@@ -13,10 +13,10 @@ wrote 2097152/2097152 bytes at offset 1048576 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}, { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}, -{ "start": 3145728, "length": 1048576, "depth": 2, "zero": true, "data": false}] +{ "start": 3145728, "length": 1048576, "depth": 2, "backing": true, "zero": true, "data": false}] [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET}, -{ "start": 1048576, "length": 2097152, "depth": 1, "zero": false, "data": false}, +{ "start": 1048576, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}, { "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] *** done
Signature · 1 line
2.31.1
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 5 lines · Eric Blake
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake<[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
If change something, this one patch seems safer. Still, Nir said he don't use qemu-img map, so probably we don't need to modify qemu-img at all? Even our iotests change shows that this change may be incompatible with at least tests..
I'm not against the patch and don't have strict opinion.
And what I really think, is that qemu-img is outdated thing and we'd better develop QMP interface, which can be used with qemu binary or with qemu-storage-daemon.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 5 lines · Eric Blake
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake<[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
Quoted 5 lines
If change something, this one patch seems safer. Still, Nir said he don't use qemu-img map, so probably we don't need to modify qemu-img at all? Even our iotests change shows that this change may be incompatible with at least tests..
I'm not against the patch and don't have strict opinion.
And what I really think, is that qemu-img is outdated thing and we'd better develop QMP interface, which can be used with qemu binary or with qemu-storage-daemon.
I don't think qemu-storage-daemon can replace qemu-img. Having an easy to use command
line tool is important. Using qmp with qemu-storage-daemon sounds like a better option
for programs that want ultimate control.
Adding only "backing: true" seems a safe change that should not break existing users
and make qemu-img map better.
The tests are broken because they compare strings instead of parsing the json. A program
parsing qemu-img json output will not be broken by adding a new key.
Nir
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
In reply to Eric BlakeAm 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 21 lines · Kevin Wolf
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 5 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Looks hard to document:
# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)
And is not consistent with "offset". It would work better as:
# @present: if present, the range is allocated within the backing
# chain (since 6.1)
Or:
# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)
This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56d
But I'm fine with "backing", It is consistent with BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).
We use "backing" internally to be consistent with future qemu-img.
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:
Quoted 28 lines · Kevin Wolf
Quoted 21 lines
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 5 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Quoted 4 lines
Looks hard to document:
# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)
I'm not sure why you would document it with a double negative.
Quoted 4 lines
And is not consistent with "offset". It would work better as:
# @present: if present, the range is allocated within the backing
# chain (since 6.1)
Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".
Quoted 4 lines
Or:
# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)
This is possible, too, but generally positive flags are preferable to negative ones, and the internal one is already positive.
Quoted 7 lines
This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56dBut I'm fine with "backing", It is consistent with
BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).We use "backing" internally to be consistent with future qemu-img.
I just realised that I actually misunderstood "backing" to mean the opposite of what it is in this patch!
It really means "the data comes from some imaginary additional backing file that doesn't exist in the backing chain", while I understood it as "something in the (real) backing chain contains the data".
"present" or "absent" should be much less prone to such misunderstandings.
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 37 lines · Kevin Wolf
Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:
Quoted 28 lines · Kevin Wolf
Quoted 21 lines
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 5 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Quoted 4 lines
Looks hard to document:
# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)I'm not sure why you would document it with a double negative.
Quoted 7 lines
Quoted 4 lines
And is not consistent with "offset". It would work better as:
# @present: if present, the range is allocated within the backing
# chain (since 6.1)Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".
This is fine, but it means that this flag will present in all ranges, instead of
only in unallocated ranges (what this patch is doing).
Quoted 29 lines
Quoted 5 lines
Or:
# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)This is possible, too, but generally positive flags are preferable to negative ones, and the internal one is already positive.
Quoted 7 lines
This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56dBut I'm fine with "backing", It is consistent with
BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).We use "backing" internally to be consistent with future qemu-img.
I just realised that I actually misunderstood "backing" to mean the opposite of what it is in this patch!
It really means "the data comes from some imaginary additional backing file that doesn't exist in the backing chain", while I understood it as "something in the (real) backing chain contains the data".
"present" or "absent" should be much less prone to such misunderstandings.
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben:
Quoted 40 lines · Kevin Wolf
Quoted 39 lines
Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:
Quoted 28 lines · Kevin Wolf
Quoted 21 lines
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 5 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Quoted 4 lines
Looks hard to document:
# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)I'm not sure why you would document it with a double negative.
Quoted 2 lines
And is not consistent with "offset". It would work better as:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
# @present: if present, the range is allocated within the backing
# chain (since 6.1)Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
If we didn't care about this, I would still define the actual value, but also document a default.
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 48 lines · Kevin Wolf
Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben:
Quoted 46 lines · Kevin Wolf
Quoted 44 lines
Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:
Quoted 28 lines · Kevin Wolf
Quoted 21 lines
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 5 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
Quoted 4 lines
Looks hard to document:
# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)I'm not sure why you would document it with a double negative.
Quoted 4 lines
And is not consistent with "offset". It would work better as:
# @present: if present, the range is allocated within the backing
# chain (since 6.1)Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".
Quoted 6 lines
Quoted 2 lines
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
Good point, this is the best option. The disadvantage is a bigger output but if you use json you don't care about the size of the output.
Quoted 5 lines
If we didn't care about this, I would still define the actual value, but also document a default.
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
In reply to Kevin WolfQuoted 9 lines · Kevin Wolf
Quoted 2 lines
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
If we didn't care about this, I would still define the actual value, but also document a default.
So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 11 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
If we didn't care about this, I would still define the actual value, but also document a default.
Quoted 8 lines
So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.
I didn't follow the discussion carefully, but that sounds good to me.
What's the decision about patch 1?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 21 lines
Quoted 11 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
If we didn't care about this, I would still define the actual value, but also document a default.
Quoted 8 lines
So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.
Quoted 1 line
I didn't follow the discussion carefully, but that sounds good to me.
To me, too.
Quoted 1 line
What's the decision about patch 1?
I think we won't need patch 1 (and the potential backwards compatibility problems it would introduce) when we have this one.
Kevin
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
Quoted 18 lines · Kevin Wolf
Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 16 lines
Quoted 15 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines
This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).
An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.
If we didn't care about this, I would still define the actual value, but also document a default.
So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output.
Since the "present" key is always present, it does not need to be at the end.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.
Quoted 8 lines
Quoted 1 line
I didn't follow the discussion carefully, but that sounds good to me.
To me, too.
Quoted 1 line
What's the decision about patch 1?
I think we won't need patch 1 (and the potential backwards compatibility problems it would introduce) when we have this one.
Yes, looks good and patch 1 is not needed.
Nir
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
In reply to Kevin WolfQuoted 20 lines · Kevin Wolf
Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:
Quoted 5 lines
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Quoted 7 lines
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.
Quoted 6 lines
However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?
IMHO, it does cover. If we have qcow2 image with unallocated clusters, but unspecified backing file it's good to know that these unallocated clusters are not simply ZERO, but actually point to backing file, which is just absent now (and therefore read returns zeros). User may start qemu and specify backing file by options, or set backing file in the image, etc. So, the information does make sense anyway.
I think it would be good to start saying about backing chains explicitly, and not hide them under "allocated" concept which has different meanings.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments
In reply to Eric BlakeQuoted 73 lines · Eric Blake
To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.
Signed-off-by Eric Blake <[email protected]>---
Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?
docs/tools/qemu-img.rst | 3 ++ qapi/block-core.json | 7 ++- qemu-img.c | 15 +++++- tests/qemu-iotests/122.out | 34 +++++++------- tests/qemu-iotests/154.out | 96 +++++++++++++++++++------------------- tests/qemu-iotests/179.out | 66 +++++++++++++------------- tests/qemu-iotests/223.out | 24 +++++----- tests/qemu-iotests/244.out | 6 +-- tests/qemu-iotests/252.out | 4 +- tests/qemu-iotests/274.out | 16 +++---- tests/qemu-iotests/291.out | 8 ++-- tests/qemu-iotests/309.out | 4 +- 12 files changed, 150 insertions(+), 133 deletions(-)docs/tools/qemu-img.rst +3
-0@@ -601,6 +601,9 @@ Command description: a ``depth``; for example, a depth of 2 refers to the backing file of the backing file of *FILENAME*. Depth will be one larger than the chain length if no file in the chain provides the data. + - an optional ``backing`` field is present with value true if no + file in the backing chain provides the data (making it easier to + identify when ``depth`` exceeds the chain length). In JSON format, the ``offset`` field is optional; it is absent in cases where ``human`` format would omit the entry or exit with an error.qapi/block-core.json +5
-2@@ -264,6 +264,9 @@ # @offset: if present, the image file stores the data for this range # in raw format at the given (host) offset # +# @backing: if present, the range is not allocated within the backing +# chain (since 6.1) +# # @filename: filename that is referred to by @offset # # Since: 2.6@@ -271,8 +274,8 @@ ## { 'struct': 'MapEntry', 'data': {'start': 'int', 'length': 'int', 'data': 'bool', - 'zero': 'bool', 'depth': 'int', '*offset': 'int', - '*filename': 'str' } } + 'zero': 'bool', 'depth': 'int', '*backing': 'bool', + '*offset': 'int', '*filename': 'str' } } ## # @BlockdevCacheInfo:qemu-img.c +0
-0@@ -2977,8 +2977,13 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,break ;
qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985
Quoted 10 lines
qemu-img.c +6
-2case OFORMAT_JSON :printf ("{ \"start\": %" PRId64", \"length\": %" PRId64"," - " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s", - e->start, e->length, e->depth, +" \"depth\": %" PRId64, e->start, e->length, e->depth); +if (e->has_backing) { +/* Backing should only be set at the end of the chain */ +assert (e->backing && e->depth >0 ); +printf (", \"backing\": true" ); + }
It will be easier to inspect the output if common fields come before optional fields.
Quoted 4 lines
qemu-img.c +1
-0+printf (", \"zero\": %s, \"data\": %s" , e->zero ?"true" :"false" , e->data ?"true" :"false" );if (e->has_offset) {
...
tests/qemu-iotests/122.out read 65536/65536 bytes at offset 4194304 · line 71
Quoted 10 lines
tests/qemu-iotests/122.out +1
-1@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304 read 65536/65536 bytes at offset 8388608 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, -{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false}, +{ "start": 65536, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false},
So this output would be:
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
{ "start": 65536, "length": 4128768, "depth": 1, "zero": true,"data": false, "backing": true},