Skip to contentSkip to conversation

qemu-img: Make unallocated part of backing chain obvious in map

PATCH Superseded Eric Blake 4 participants 19 messages Jun 11, 2021 – Jun 29, 2021

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

PATCH v2 1/1 qemu-img: Make unallocated part of backing chain obvious in map

The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and prior to this patch has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).

The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.

An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.

---

Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)

Use N+1 instead of -1 for unallocated [Kevin]

 docs/tools/qemu-img.rst    |   3 +-
 qemu-img.c                 |   3 ++
 tests/qemu-iotests/122.out |  34 ++++++-------
 tests/qemu-iotests/154.out | 100 +++++++++++++++++++------------------
 tests/qemu-iotests/179.out |  67 +++++++++++++++++++++----
 tests/qemu-iotests/223.out |  24 ++++-----
 tests/qemu-iotests/244.out |   7 +--
 tests/qemu-iotests/252.out |   4 +-
 tests/qemu-iotests/274.out |  16 +++---
 tests/qemu-iotests/291.out |   8 +--
 tests/qemu-iotests/309.out |   4 +-
 11 files changed, 162 insertions(+), 108 deletions(-)
docs/tools/qemu-img.rst +2 -1
index cfe11478791f..c155b1bf3cc8 100644
@@ -599,7 +599,8 @@ Command description:
   - whether the data is known to read as zero (boolean field ``zero``);
   - in order to make the output shorter, the target file is expressed as
     a ``depth``; for example, a depth of 2 refers to the backing file
-    of the backing file of *FILENAME*.
+    of the backing file of *FILENAME*.  Depth will be one larger than
+    the chain length if no file in the chain provides the data.
 
   In JSON format, the ``offset`` field is optional; it is absent in
   cases where ``human`` format would omit the entry or exit with an error.
qemu-img.c +3 -0
index a5993682aad4..33a5cd012b8b 100644
@@ -3035,6 +3035,9 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         bdrv_refresh_filename(file);
         filename = file->filename;
     }
+    if (!(ret & BDRV_BLOCK_ALLOCATED)) {
+        depth++;
+    }
 
     *e = (MapEntry) {
         .start = offset,
tests/qemu-iotests/122.out +17 -17
index 3a3e121d579d..779dab4847f0 100644
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304
 read 65536/65536 bytes at offset 8388608
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 65536, "length": 4128768, "depth": 0, "zero": true, "data": false},
+{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false},
 { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 4259840, "length": 4128768, "depth": 0, "zero": true, "data": false},
+{ "start": 4259840, "length": 4128768, "depth": 1, "zero": true, "data": false},
 { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 8454144, "length": 4128768, "depth": 0, "zero": true, "data": false}]
+{ "start": 8454144, "length": 4128768, "depth": 1, "zero": true, "data": false}]
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 65536/65536 bytes at offset 4194304
@@ -95,11 +95,11 @@ wrote 1024/1024 bytes at offset 1046528
 wrote 1024/1024 bytes at offset 0
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 65536, "length": 65536, "depth": 0, "zero": true, "data": false},
+{ "start": 65536, "length": 65536, "depth": 1, "zero": true, "data": false},
 { "start": 131072, "length": 196608, "depth": 0, "zero": false, "data": true},
-{ "start": 327680, "length": 655360, "depth": 0, "zero": true, "data": false},
+{ "start": 327680, "length": 655360, "depth": 1, "zero": true, "data": false},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 1048576, "length": 1046528, "depth": 0, "zero": true, "data": false}]
+{ "start": 1048576, "length": 1046528, "depth": 1, "zero": true, "data": false}]
 read 16384/16384 bytes at offset 0
 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 16384/16384 bytes at offset 16384
@@ -195,31 +195,31 @@ wrote 1024/1024 bytes at offset 17408
 
 convert -S 4k
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 4096, "length": 4096, "depth": 0, "zero": true, "data": false},
+{ "start": 4096, "length": 4096, "depth": 1, "zero": true, "data": false},
 { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 12288, "length": 4096, "depth": 0, "zero": true, "data": false},
+{ "start": 12288, "length": 4096, "depth": 1, "zero": true, "data": false},
 { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 20480, "length": 67088384, "depth": 0, "zero": true, "data": false}]
+{ "start": 20480, "length": 67088384, "depth": 1, "zero": true, "data": false}]
 
 convert -c -S 4k
 [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false},
+{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false},
 { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false},
+{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false},
 { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}]
+{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}]
 
 convert -S 8k
 [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 24576, "length": 67084288, "depth": 0, "zero": true, "data": false}]
+{ "start": 24576, "length": 67084288, "depth": 1, "zero": true, "data": false}]
 
 convert -c -S 8k
 [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 1024, "length": 7168, "depth": 0, "zero": true, "data": false},
+{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false},
 { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 9216, "length": 8192, "depth": 0, "zero": true, "data": false},
+{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false},
 { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 18432, "length": 67090432, "depth": 0, "zero": true, "data": false}]
+{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}]
 
 === -n to a non-zero image ===
 
@@ -235,7 +235,7 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 [{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}]
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-[{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}]
+[{ "start": 0, "length": 67108864, "depth": 1, "zero": true, "data": false}]
 
 === -n to an empty image with a backing file ===
 
tests/qemu-iotests/154.out +52 -48

279 lines not on this page · raw message

tests/qemu-iotests/179.out +56 -11

127 lines not on this page · raw message

tests/qemu-iotests/223.out +12 -12
index 083b62d05386..101d1e89a8db 100644
@@ -103,16 +103,16 @@ read 2097152/2097152 bytes at offset 2097152
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 
 === Contrast to small granularity dirty-bitmap ===
 
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
 { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 
 === End qemu NBD server ===
 
@@ -204,16 +204,16 @@ read 2097152/2097152 bytes at offset 2097152
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 
 === Contrast to small granularity dirty-bitmap ===
 
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
 { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 
 === End qemu NBD server ===
 
@@ -238,12 +238,12 @@ read 2097152/2097152 bytes at offset 2097152
 
 === Use qemu-nbd as server ===
 
-[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 0, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
 { "start": 1024, "length": 11321, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 12345, "length": 2084807, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 0, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
 *** done
tests/qemu-iotests/244.out +4 -3
index 99f56ac18c48..b934f2d9b792 100644
@@ -57,11 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728
 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 No errors were found on the image.
 
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
+[{ "start": 0, "length": 1048576, "depth": 1, "zero": true, "data": false},
 { "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576},
 { "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
 { "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304},
-{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}]
+{ "start": 5242880, "length": 1048576, "depth": 0, "zero": true, "data": false},
+{ "start": 6291456, "length": 60817408, "depth": 1, "zero": true, "data": false}]
 
 read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -123,7 +124,7 @@ read 1048576/1048576 bytes at offset 0
 Offset          Length          Mapped to       File
 0               0x100000        0               TEST_DIR/t.qcow2.data
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0},
-{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}]
+{ "start": 1048576, "length": 66060288, "depth": 1, "zero": true, "data": false}]
 
 === Copy offloading ===
 
tests/qemu-iotests/252.out +2 -2
index 12dce889f823..b057b825739e 100644
@@ -24,7 +24,7 @@ read 131072/131072 bytes at offset 262144
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 262144, "length": 131072, "depth": 0, "zero": true, "data": false}]
+{ "start": 262144, "length": 131072, "depth": 1, "zero": true, "data": false}]
 
 read 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -35,5 +35,5 @@ read 131072/131072 bytes at offset 262144
 
 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false},
-{ "start": 327680, "length": 65536, "depth": 1, "zero": true, "data": false}]
+{ "start": 327680, "length": 65536, "depth": 2, "zero": true, "data": false}]
 *** done
tests/qemu-iotests/274.out +8 -8
index cfe17a865924..9a4de18c4cf2 100644
@@ -37,7 +37,7 @@ Offset          Length          Mapped to       File
 0               0x100000        0x50000         TEST_DIR/PID-base
 
 [{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680},
-{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}]
+{ "start": 1048576, "length": 1048576, "depth": 1, "zero": true, "data": false}]
 
 Offset          Length          Mapped to       File
 0               0x100000        0x50000         TEST_DIR/PID-base
@@ -220,7 +220,7 @@ read 65536/65536 bytes at offset 5368709120
 1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0)
 7 GiB (0x1c0000000) bytes     allocated at offset 1 GiB (0x40000000)
 
-[{ "start": 0, "length": 1073741824, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 1073741824, "depth": 2, "zero": true, "data": false},
 { "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}]
 
 === preallocation=metadata ===
@@ -239,7 +239,7 @@ read 65536/65536 bytes at offset 33285996544
 30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0)
 3 GiB (0xc0000000) bytes     allocated at offset 30 GiB (0x780000000)
 
-[{ "start": 0, "length": 32212254720, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 32212254720, "depth": 2, "zero": true, "data": false},
 { "start": 32212254720, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 327680},
 { "start": 32749125632, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 537264128},
 { "start": 33285996544, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 1074200576},
@@ -263,7 +263,7 @@ read 65536/65536 bytes at offset 9437184
 5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0)
 10 MiB (0xa00000) bytes     allocated at offset 5 MiB (0x500000)
 
-[{ "start": 0, "length": 5242880, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 5242880, "depth": 2, "zero": true, "data": false},
 { "start": 5242880, "length": 10485760, "depth": 0, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=full ===
@@ -282,7 +282,7 @@ read 65536/65536 bytes at offset 11534336
 8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0)
 4 MiB (0x400000) bytes     allocated at offset 8 MiB (0x800000)
 
-[{ "start": 0, "length": 8388608, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 8388608, "depth": 2, "zero": true, "data": false},
 { "start": 8388608, "length": 4194304, "depth": 0, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=off ===
@@ -301,7 +301,7 @@ read 65536/65536 bytes at offset 259072
 192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0)
 320 KiB (0x50000) bytes     allocated at offset 192 KiB (0x30000)
 
-[{ "start": 0, "length": 196608, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 196608, "depth": 2, "zero": true, "data": false},
 { "start": 196608, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680},
 { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
 
@@ -321,7 +321,7 @@ read 65536/65536 bytes at offset 344064
 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
 256 KiB (0x40000) bytes     allocated at offset 256 KiB (0x40000)
 
-[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false},
 { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
 
 === preallocation=off ===
@@ -340,6 +340,6 @@ read 65536/65536 bytes at offset 446464
 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
 244 KiB (0x3d000) bytes     allocated at offset 256 KiB (0x40000)
 
-[{ "start": 0, "length": 262144, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false},
 { "start": 262144, "length": 249856, "depth": 0, "zero": true, "data": false}]
 
tests/qemu-iotests/291.out +4 -4
index 23411c0ff4d9..5b997eb8d744 100644
@@ -104,15 +104,15 @@ Format specific information:
 === Check bitmap contents ===
 
 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 3145728, "length": 1048576, "depth": 1, "zero": false, "data": false},
 { "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 1048576, "length": 1048576, "depth": 1, "zero": false, "data": false},
 { "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
tests/qemu-iotests/309.out +2 -2
index db75bb6b0df9..7020bd33d7c1 100644
@@ -13,10 +13,10 @@ wrote 2097152/2097152 bytes at offset 1048576
 
 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680},
 { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680},
-{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}]
+{ "start": 3145728, "length": 1048576, "depth": 2, "zero": true, "data": false}]
 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false},
+{ "start": 1048576, "length": 2097152, "depth": 1, "zero": false, "data": false},
 { "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
Signature · 1 line
2.31.1
#1 lore raw
Quoted 41 lines · Eric Blake

The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But qemu-img map --output=json predates that addition, and prior to this patch has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).

The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD.

An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.

Quoted 8 lines

---

Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)

Use N+1 instead of -1 for unallocated [Kevin]

Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2 UNALLOCATED_ZERO cluster.

So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\

Signature · 2 lines
Best regards,
Vladimir
author
#2 lore raw
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 18 lines

An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.

---

Quoted 7 lines
Quoted 5 lines

Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)

Use N+1 instead of -1 for unallocated [Kevin]

Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2 UNALLOCATED_ZERO cluster.

Using just 'qemu-img map --output-json', you only see depth numbers. You also have to use 'qemu-img info --backing-chain' to see what file those depth numbers correspond to, at which point it becomes obvious whether "depth":4 meant unallocated (because the chain was length 3) or allocated at depth 4 (because the chain was length 4 or longer). But that's no worse than pre-patch, where you had to use qemu-img info --backing-chain to learn which file a particular "depth" maps to.

Quoted 2 lines

So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\

It should be fairly easy to optimize the case of a backing chain where EVERY listed cluster at the final depth was "data":false,"zero":true to omit that file after all.

And in oVirt's case, Nir pointed out that we have one more tool at our disposal in recreating a backing chain: if you use json:{"driver":"qcow2", "backing":null, ...} as your image file, you don't have to worry about arbitrary files in the backing chain, only about recreating the top-most layer of a chain. And in that case, it becomes very obvious that "depth":0 is something you must recreate, and "depth":1 would be a non-existent backing file because you just passed "backing":null.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#3 lore raw
Quoted 43 lines · Eric Blake
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 18 lines

An obvious solution is to make 'qemu-img map --output=json' distinguish between clusters that have a local allocation from those that are found nowhere in the chain. We already have a one-off mismatch between qemu-img map and NBD qemu:allocation-depth (the former chose 0, and the latter 1 for the local layer), so exposing the latter's choice of 0 for unallocated in the entire chain would mean using using "depth":-1 in the former, but a negative depth may confuse existing tools. But there is an easy out: for any chain of length N, we can simply represent an unallocated cluster as "depth":N+1. This does have a slight risk of confusing any tool that might try to dereference NULL when finding the backing image for the last file in the backing chain, but that risk sseems worth the more precise output. The iotests have several examples where this distinction demonstrates the additional accuracy.

---

Quoted 7 lines
Quoted 5 lines

Replaces v1: [email protected] (qemu-img: Use "depth":-1 to make backing probes obvious)

Use N+1 instead of -1 for unallocated [Kevin]

Bit in contrast with -1, or with separate boolean flag, you lose the possibility to distinguish case when we have 3 layers and the cluster is absent in all of them, and the case when we have 4 layers and the cluster is absent in top 3 but in 4 it is qcow2 UNALLOCATED_ZERO cluster.

Using just 'qemu-img map --output-json', you only see depth numbers. You also have to use 'qemu-img info --backing-chain' to see what file those depth numbers correspond to, at which point it becomes obvious whether "depth":4 meant unallocated (because the chain was length 3) or allocated at depth 4 (because the chain was length 4 or longer). But that's no worse than pre-patch, where you had to use qemu-img info --backing-chain to learn which file a particular "depth" maps to.

Quoted 2 lines

So, if someone use this API to reconstruct the chain, then for original 3 empty layers he will create 3 empty layers and 4rd additional ZERO layer. And such reconstructed chain would not be equal to original chain (as if we take these two chains and add additional backing file as a new bottom layer, effect would be different).. I'm not sure is it a problem in the task you are solving :\

It should be fairly easy to optimize the case of a backing chain where EVERY listed cluster at the final depth was "data":false,"zero":true to omit that file after all.

Quoted 8 lines

And in oVirt's case, Nir pointed out that we have one more tool at our disposal in recreating a backing chain: if you use json:{"driver":"qcow2", "backing":null, ...} as your image file, you don't have to worry about arbitrary files in the backing chain, only about recreating the top-most layer of a chain. And in that case, it becomes very obvious that "depth":0 is something you must recreate, and "depth":1 would be a non-existent backing file because you just passed "backing":null.

Note that oVirt does not use qemu-img map, we use qemu-nbd to get image extents, since it is used only in context we already connect to qemu-nbd server or run qemu-nbd.

Management tools already know the image format (they should avoid doing format probing anyway), and using a json uri allows single command to get the needed info when you inspect a single layer.

But this change introduces a risk that some program using qemu-img map will interrupt the result in the wrong way, assuming that there is N+1 layer.

I think adding a new flag for absent extents is better. It cannot break any user and it is easier to understand and use.

Nir

author
#4 lore raw

[PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

In reply to Eric Blake

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

 docs/tools/qemu-img.rst    |  3 ++
 qapi/block-core.json       |  7 ++-
 qemu-img.c                 | 15 +++++-
 tests/qemu-iotests/122.out | 34 +++++++-------
 tests/qemu-iotests/154.out | 96 +++++++++++++++++++-------------------
 tests/qemu-iotests/179.out | 66 +++++++++++++-------------
 tests/qemu-iotests/223.out | 24 +++++-----
 tests/qemu-iotests/244.out |  6 +--
 tests/qemu-iotests/252.out |  4 +-
 tests/qemu-iotests/274.out | 16 +++----
 tests/qemu-iotests/291.out |  8 ++--
 tests/qemu-iotests/309.out |  4 +-
 12 files changed, 150 insertions(+), 133 deletions(-)
docs/tools/qemu-img.rst +3 -0
index c155b1bf3cc8..fbc623b645c3 100644
@@ -601,6 +601,9 @@ Command description:
     a ``depth``; for example, a depth of 2 refers to the backing file
     of the backing file of *FILENAME*.  Depth will be one larger than
     the chain length if no file in the chain provides the data.
+  - an optional ``backing`` field is present with value true if no
+    file in the backing chain provides the data (making it easier to
+    identify when ``depth`` exceeds the chain length).
 
   In JSON format, the ``offset`` field is optional; it is absent in
   cases where ``human`` format would omit the entry or exit with an error.
qapi/block-core.json +5 -2
index 2ea294129e08..cebe12ba16a0 100644
@@ -264,6 +264,9 @@
 # @offset: if present, the image file stores the data for this range
 #          in raw format at the given (host) offset
 #
+# @backing: if present, the range is not allocated within the backing
+#           chain (since 6.1)
+#
 # @filename: filename that is referred to by @offset
 #
 # Since: 2.6
@@ -271,8 +274,8 @@
 ##
 { 'struct': 'MapEntry',
   'data': {'start': 'int', 'length': 'int', 'data': 'bool',
-           'zero': 'bool', 'depth': 'int', '*offset': 'int',
-           '*filename': 'str' } }
+           'zero': 'bool', 'depth': 'int', '*backing': 'bool',
+           '*offset': 'int', '*filename': 'str' } }
 
 ##
 # @BlockdevCacheInfo:
qemu-img.c +13 -2
index 33a5cd012b8b..4d357f534803 100644
@@ -2977,8 +2977,13 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,
         break;
     case OFORMAT_JSON:
         printf("{ \"start\": %"PRId64", \"length\": %"PRId64","
-               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
-               e->start, e->length, e->depth,
+               " \"depth\": %"PRId64, e->start, e->length, e->depth);
+        if (e->has_backing) {
+            /* Backing should only be set at the end of the chain */
+            assert(e->backing && e->depth > 0);
+            printf(", \"backing\": true");
+        }
+        printf(", \"zero\": %s, \"data\": %s",
                e->zero ? "true" : "false",
                e->data ? "true" : "false");
         if (e->has_offset) {
@@ -2999,6 +3004,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 {
     int ret;
     int depth;
+    bool backing = false;
     BlockDriverState *file;
     bool has_offset;
     int64_t map;
@@ -3037,6 +3043,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
     }
     if (!(ret & BDRV_BLOCK_ALLOCATED)) {
         depth++;
+        backing = true;
     }
 
     *e = (MapEntry) {
@@ -3047,6 +3054,8 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         .offset = map,
         .has_offset = has_offset,
         .depth = depth,
+        .has_backing = backing,
+        .backing = backing,
         .has_filename = filename,
         .filename = filename,
     };
@@ -3072,6 +3081,8 @@ static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)
     if (curr->has_offset && curr->offset + curr->length != next->offset) {
         return false;
     }
+    /* backing should only ever be set for identical depth */
+    assert(curr->backing == next->backing);
     return true;
 }
 
tests/qemu-iotests/122.out +17 -17
index 779dab4847f0..c5aa2c9866f1 100644
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304
 read 65536/65536 bytes at offset 8388608
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false},
+{ "start": 65536, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 4194304, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 4259840, "length": 4128768, "depth": 1, "zero": true, "data": false},
+{ "start": 4259840, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 8388608, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 8454144, "length": 4128768, "depth": 1, "zero": true, "data": false}]
+{ "start": 8454144, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false}]
 read 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 65536/65536 bytes at offset 4194304
@@ -95,11 +95,11 @@ wrote 1024/1024 bytes at offset 1046528
 wrote 1024/1024 bytes at offset 0
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 65536, "length": 65536, "depth": 1, "zero": true, "data": false},
+{ "start": 65536, "length": 65536, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 131072, "length": 196608, "depth": 0, "zero": false, "data": true},
-{ "start": 327680, "length": 655360, "depth": 1, "zero": true, "data": false},
+{ "start": 327680, "length": 655360, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 1048576, "length": 1046528, "depth": 1, "zero": true, "data": false}]
+{ "start": 1048576, "length": 1046528, "depth": 1, "backing": true, "zero": true, "data": false}]
 read 16384/16384 bytes at offset 0
 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 16384/16384 bytes at offset 16384
@@ -195,31 +195,31 @@ wrote 1024/1024 bytes at offset 17408
 
 convert -S 4k
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 4096, "length": 4096, "depth": 1, "zero": true, "data": false},
+{ "start": 4096, "length": 4096, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 12288, "length": 4096, "depth": 1, "zero": true, "data": false},
+{ "start": 12288, "length": 4096, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 20480, "length": 67088384, "depth": 1, "zero": true, "data": false}]
+{ "start": 20480, "length": 67088384, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 convert -c -S 4k
 [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false},
+{ "start": 1024, "length": 7168, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false},
+{ "start": 9216, "length": 8192, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}]
+{ "start": 18432, "length": 67090432, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 convert -S 8k
 [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 24576, "length": 67084288, "depth": 1, "zero": true, "data": false}]
+{ "start": 24576, "length": 67084288, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 convert -c -S 8k
 [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 1024, "length": 7168, "depth": 1, "zero": true, "data": false},
+{ "start": 1024, "length": 7168, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 8192, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 9216, "length": 8192, "depth": 1, "zero": true, "data": false},
+{ "start": 9216, "length": 8192, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 17408, "length": 1024, "depth": 0, "zero": false, "data": true},
-{ "start": 18432, "length": 67090432, "depth": 1, "zero": true, "data": false}]
+{ "start": 18432, "length": 67090432, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 === -n to a non-zero image ===
 
@@ -235,7 +235,7 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 [{ "start": 0, "length": 67108864, "depth": 0, "zero": true, "data": false}]
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-[{ "start": 0, "length": 67108864, "depth": 1, "zero": true, "data": false}]
+[{ "start": 0, "length": 67108864, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 === -n to an empty image with a backing file ===
 
tests/qemu-iotests/154.out +48 -48

279 lines not on this page · raw message

tests/qemu-iotests/179.out +33 -33

148 lines not on this page · raw message

tests/qemu-iotests/223.out +12 -12
index 101d1e89a8db..f3e3516b4908 100644
@@ -103,16 +103,16 @@ read 2097152/2097152 bytes at offset 2097152
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 
 === Contrast to small granularity dirty-bitmap ===
 
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 
 === End qemu NBD server ===
 
@@ -204,16 +204,16 @@ read 2097152/2097152 bytes at offset 2097152
 [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 1048576, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 
 === Contrast to small granularity dirty-bitmap ===
 
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 1024, "length": 2096128, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 
 === End qemu NBD server ===
 
@@ -238,12 +238,12 @@ read 2097152/2097152 bytes at offset 2097152
 
 === Use qemu-nbd as server ===
 
-[{ "start": 0, "length": 65536, "depth": 1, "zero": false, "data": false},
+[{ "start": 0, "length": 65536, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 65536, "length": 2031616, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 [{ "start": 0, "length": 512, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 512, "length": 512, "depth": 1, "zero": false, "data": false},
+{ "start": 512, "length": 512, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 1024, "length": 11321, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 12345, "length": 2084807, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 2097152, "depth": 1, "zero": false, "data": false}]
+{ "start": 2097152, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false}]
 *** done
tests/qemu-iotests/244.out +3 -3
index b934f2d9b792..d46f89ae4e95 100644
@@ -57,12 +57,12 @@ wrote 3145728/3145728 bytes at offset 3145728
 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 No errors were found on the image.
 
-[{ "start": 0, "length": 1048576, "depth": 1, "zero": true, "data": false},
+[{ "start": 0, "length": 1048576, "depth": 1, "backing": true, "zero": true, "data": false},
 { "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576},
 { "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
 { "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304},
 { "start": 5242880, "length": 1048576, "depth": 0, "zero": true, "data": false},
-{ "start": 6291456, "length": 60817408, "depth": 1, "zero": true, "data": false}]
+{ "start": 6291456, "length": 60817408, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -124,7 +124,7 @@ read 1048576/1048576 bytes at offset 0
 Offset          Length          Mapped to       File
 0               0x100000        0               TEST_DIR/t.qcow2.data
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0},
-{ "start": 1048576, "length": 66060288, "depth": 1, "zero": true, "data": false}]
+{ "start": 1048576, "length": 66060288, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 === Copy offloading ===
 
tests/qemu-iotests/252.out +2 -2
index b057b825739e..c9a56c78aae3 100644
@@ -24,7 +24,7 @@ read 131072/131072 bytes at offset 262144
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 262144, "length": 131072, "depth": 1, "zero": true, "data": false}]
+{ "start": 262144, "length": 131072, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 read 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -35,5 +35,5 @@ read 131072/131072 bytes at offset 262144
 
 [{ "start": 0, "length": 262144, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 262144, "length": 65536, "depth": 0, "zero": true, "data": false},
-{ "start": 327680, "length": 65536, "depth": 2, "zero": true, "data": false}]
+{ "start": 327680, "length": 65536, "depth": 2, "backing": true, "zero": true, "data": false}]
 *** done
tests/qemu-iotests/274.out +8 -8
index 9a4de18c4cf2..1c8138306fde 100644
@@ -37,7 +37,7 @@ Offset          Length          Mapped to       File
 0               0x100000        0x50000         TEST_DIR/PID-base
 
 [{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680},
-{ "start": 1048576, "length": 1048576, "depth": 1, "zero": true, "data": false}]
+{ "start": 1048576, "length": 1048576, "depth": 1, "backing": true, "zero": true, "data": false}]
 
 Offset          Length          Mapped to       File
 0               0x100000        0x50000         TEST_DIR/PID-base
@@ -220,7 +220,7 @@ read 65536/65536 bytes at offset 5368709120
 1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0)
 7 GiB (0x1c0000000) bytes     allocated at offset 1 GiB (0x40000000)
 
-[{ "start": 0, "length": 1073741824, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 1073741824, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}]
 
 === preallocation=metadata ===
@@ -239,7 +239,7 @@ read 65536/65536 bytes at offset 33285996544
 30 GiB (0x780000000) bytes not allocated at offset 0 bytes (0x0)
 3 GiB (0xc0000000) bytes     allocated at offset 30 GiB (0x780000000)
 
-[{ "start": 0, "length": 32212254720, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 32212254720, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 32212254720, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 327680},
 { "start": 32749125632, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 537264128},
 { "start": 33285996544, "length": 536870912, "depth": 0, "zero": true, "data": false, "offset": 1074200576},
@@ -263,7 +263,7 @@ read 65536/65536 bytes at offset 9437184
 5 MiB (0x500000) bytes not allocated at offset 0 bytes (0x0)
 10 MiB (0xa00000) bytes     allocated at offset 5 MiB (0x500000)
 
-[{ "start": 0, "length": 5242880, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 5242880, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 5242880, "length": 10485760, "depth": 0, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=full ===
@@ -282,7 +282,7 @@ read 65536/65536 bytes at offset 11534336
 8 MiB (0x800000) bytes not allocated at offset 0 bytes (0x0)
 4 MiB (0x400000) bytes     allocated at offset 8 MiB (0x800000)
 
-[{ "start": 0, "length": 8388608, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 8388608, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 8388608, "length": 4194304, "depth": 0, "zero": false, "data": true, "offset": 327680}]
 
 === preallocation=off ===
@@ -301,7 +301,7 @@ read 65536/65536 bytes at offset 259072
 192 KiB (0x30000) bytes not allocated at offset 0 bytes (0x0)
 320 KiB (0x50000) bytes     allocated at offset 192 KiB (0x30000)
 
-[{ "start": 0, "length": 196608, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 196608, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 196608, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 327680},
 { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
 
@@ -321,7 +321,7 @@ read 65536/65536 bytes at offset 344064
 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
 256 KiB (0x40000) bytes     allocated at offset 256 KiB (0x40000)
 
-[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 262144, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 262144, "length": 262144, "depth": 0, "zero": true, "data": false}]
 
 === preallocation=off ===
@@ -340,6 +340,6 @@ read 65536/65536 bytes at offset 446464
 256 KiB (0x40000) bytes not allocated at offset 0 bytes (0x0)
 244 KiB (0x3d000) bytes     allocated at offset 256 KiB (0x40000)
 
-[{ "start": 0, "length": 262144, "depth": 2, "zero": true, "data": false},
+[{ "start": 0, "length": 262144, "depth": 2, "backing": true, "zero": true, "data": false},
 { "start": 262144, "length": 249856, "depth": 0, "zero": true, "data": false}]
 
tests/qemu-iotests/291.out +4 -4
index 5b997eb8d744..841286fb2607 100644
@@ -104,15 +104,15 @@ Format specific information:
 === Check bitmap contents ===
 
 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 3145728, "length": 1048576, "depth": 1, "zero": false, "data": false},
+{ "start": 3145728, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 1048576, "depth": 1, "zero": false, "data": false},
+{ "start": 1048576, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false},
+{ "start": 2097152, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
-{ "start": 2097152, "length": 1048576, "depth": 1, "zero": false, "data": false},
+{ "start": 2097152, "length": 1048576, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
tests/qemu-iotests/309.out +2 -2
index 7020bd33d7c1..d3de6a8fed13 100644
@@ -13,10 +13,10 @@ wrote 2097152/2097152 bytes at offset 1048576
 
 [{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680},
 { "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680},
-{ "start": 3145728, "length": 1048576, "depth": 2, "zero": true, "data": false}]
+{ "start": 3145728, "length": 1048576, "depth": 2, "backing": true, "zero": true, "data": false}]
 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
-{ "start": 1048576, "length": 2097152, "depth": 1, "zero": false, "data": false},
+{ "start": 1048576, "length": 2097152, "depth": 1, "backing": true, "zero": false, "data": false},
 { "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
Signature · 1 line
2.31.1
#5 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 5 lines · Eric Blake

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

If change something, this one patch seems safer. Still, Nir said he don't use qemu-img map, so probably we don't need to modify qemu-img at all? Even our iotests change shows that this change may be incompatible with at least tests..

I'm not against the patch and don't have strict opinion.

And what I really think, is that qemu-img is outdated thing and we'd better develop QMP interface, which can be used with qemu binary or with qemu-storage-daemon.

Signature · 2 lines
Best regards,
Vladimir
#6 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 5 lines · Eric Blake

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

Quoted 5 lines

If change something, this one patch seems safer. Still, Nir said he don't use qemu-img map, so probably we don't need to modify qemu-img at all? Even our iotests change shows that this change may be incompatible with at least tests..

I'm not against the patch and don't have strict opinion.

And what I really think, is that qemu-img is outdated thing and we'd better develop QMP interface, which can be used with qemu binary or with qemu-storage-daemon.

I don't think qemu-storage-daemon can replace qemu-img. Having an easy to use command
line tool is important. Using qmp with qemu-storage-daemon sounds like a better option
for programs that want ultimate control.

Adding only "backing: true" seems a safe change that should not break existing users
and make qemu-img map better.

The tests are broken because they compare strings instead of parsing the json. A program
parsing qemu-img json output will not be broken by adding a new key.

Nir

#7 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

In reply to Eric Blake

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Kevin

#8 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 21 lines · Kevin Wolf

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 5 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Looks hard to document:

# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)

And is not consistent with "offset". It would work better as:

# @present: if present, the range is allocated within the backing
# chain (since 6.1)

Or:

# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)

This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56d

But I'm fine with "backing", It is consistent with BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).

We use "backing" internally to be consistent with future qemu-img.

#9 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:

Quoted 28 lines · Kevin Wolf
Quoted 21 lines

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 5 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Quoted 4 lines

Looks hard to document:

# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)

I'm not sure why you would document it with a double negative.

Quoted 4 lines

And is not consistent with "offset". It would work better as:

# @present: if present, the range is allocated within the backing
# chain (since 6.1)

Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".

Quoted 4 lines

Or:

# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)

This is possible, too, but generally positive flags are preferable to negative ones, and the internal one is already positive.

Quoted 7 lines

This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56d

But I'm fine with "backing", It is consistent with BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).

We use "backing" internally to be consistent with future qemu-img.

I just realised that I actually misunderstood "backing" to mean the opposite of what it is in this patch!

It really means "the data comes from some imaginary additional backing file that doesn't exist in the backing chain", while I understood it as "something in the (real) backing chain contains the data".

"present" or "absent" should be much less prone to such misunderstandings.

Kevin

#10 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 37 lines · Kevin Wolf

Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:

Quoted 28 lines · Kevin Wolf
Quoted 21 lines

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 5 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Quoted 4 lines

Looks hard to document:

# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)

I'm not sure why you would document it with a double negative.

Quoted 7 lines
Quoted 4 lines

And is not consistent with "offset". It would work better as:

# @present: if present, the range is allocated within the backing
# chain (since 6.1)

Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".

This is fine, but it means that this flag will present in all ranges, instead of
only in unallocated ranges (what this patch is doing).

Quoted 29 lines
Quoted 5 lines

Or:

# @absent: if present, the range is not allocated within the backing
# chain (since 6.1)

This is possible, too, but generally positive flags are preferable to negative ones, and the internal one is already positive.

Quoted 7 lines

This is used by libnbd now:
https://github.com/libguestfs/libnbd/commit/1d01d2ac4f6443b160b7d81119d555e1aaedb56d

But I'm fine with "backing", It is consistent with BLK_BACKING_FILE, meaning this area exposes data from a backing file (if one exists).

We use "backing" internally to be consistent with future qemu-img.

I just realised that I actually misunderstood "backing" to mean the opposite of what it is in this patch!

It really means "the data comes from some imaginary additional backing file that doesn't exist in the backing chain", while I understood it as "something in the (real) backing chain contains the data".

"present" or "absent" should be much less prone to such misunderstandings.

Kevin

#11 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben:

Quoted 40 lines · Kevin Wolf
Quoted 39 lines

Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:

Quoted 28 lines · Kevin Wolf
Quoted 21 lines

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 5 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Quoted 4 lines

Looks hard to document:

# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)

I'm not sure why you would document it with a double negative.

Quoted 2 lines

And is not consistent with "offset". It would work better as:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

# @present: if present, the range is allocated within the backing
# chain (since 6.1)

Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

If we didn't care about this, I would still define the actual value, but also document a default.

Kevin

#12 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 48 lines · Kevin Wolf

Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben:

Quoted 46 lines · Kevin Wolf
Quoted 44 lines

Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben:

Quoted 28 lines · Kevin Wolf
Quoted 21 lines

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 5 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

Quoted 4 lines

Looks hard to document:

# @present: if present and false, the range is not allocated within the
# backing chain (since 6.1)

I'm not sure why you would document it with a double negative.

Quoted 4 lines

And is not consistent with "offset". It would work better as:

# @present: if present, the range is allocated within the backing
# chain (since 6.1)

Completely ignoring the value? I would have documented it like this, but with "if true..." instead of "if present...".

Quoted 6 lines
Quoted 2 lines

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

Good point, this is the best option. The disadvantage is a bigger output but if you use json you don't care about the size of the output.

Quoted 5 lines

If we didn't care about this, I would still define the actual value, but also document a default.

Kevin

author
#13 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

In reply to Kevin Wolf
Quoted 9 lines · Kevin Wolf
Quoted 2 lines

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

If we didn't care about this, I would still define the actual value, but also document a default.

So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#14 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 11 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

If we didn't care about this, I would still define the actual value, but also document a default.

Quoted 8 lines

So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.

I didn't follow the discussion carefully, but that sounds good to me.

What's the decision about patch 1?

Signature · 2 lines
Best regards,
Vladimir
#15 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 21 lines
Quoted 11 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

If we didn't care about this, I would still define the actual value, but also document a default.

Quoted 8 lines

So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output. It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.

Quoted 1 line

I didn't follow the discussion carefully, but that sounds good to me.

To me, too.

Quoted 1 line

What's the decision about patch 1?

I think we won't need patch 1 (and the potential backwards compatibility problems it would introduce) when we have this one.

Kevin

#16 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

Quoted 18 lines · Kevin Wolf

Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 16 lines
Quoted 15 lines · Eric Blake
Quoted 9 lines · Kevin Wolf
Quoted 2 lines

This is fine, but it means that this flag will present in all ranges, instead of only in unallocated ranges (what this patch is doing).

An argument for always having the flag would be that it's probably useful for a tool to know whether a given block is actually absent or whether it's just running an old qemu-img.

If we didn't care about this, I would still define the actual value, but also document a default.

So to summarize, it looks like my v3 will have the best chance of approval if I go with always outputting the new field (instead of only on one of its two boolean values), and put it at the end of the JSON output.

Since the "present" key is always present, it does not need to be at the end.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

It also looks like we have consensus on spelling the new field "present":true for data found in the backing chain, and "present":false for places where we would defer to another file if a backing file is later added.

Quoted 8 lines
Quoted 1 line

I didn't follow the discussion carefully, but that sounds good to me.

To me, too.

Quoted 1 line

What's the decision about patch 1?

I think we won't need patch 1 (and the potential backwards compatibility problems it would introduce) when we have this one.

Yes, looks good and patch 1 is not needed.

Nir

#17 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

In reply to Kevin Wolf
Quoted 20 lines · Kevin Wolf

Am 11.06.2021 um 21:03 hat Eric Blake geschrieben:

Quoted 5 lines

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

Quoted 7 lines

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

I think the additional flag is better because it's guaranteed to be backwards compatible, and because you don't need to know the number of layers to infer whether a cluster was allocated in the whole backing chain. And by exposing ALLOCATED we definitely give access to the whole information that exists in QEMU.

Quoted 6 lines

However, to continue with the bike shedding: I won't insist on "allocated" even if that is what the flag is called internally and consistency is usually helpful, but "backing" is misleading, too, because intuitively it doesn't cover the top layer or standalone images without a backing file. How about something like "present"?

IMHO, it does cover. If we have qcow2 image with unallocated clusters, but unspecified backing file it's good to know that these unallocated clusters are not simply ZERO, but actually point to backing file, which is just absent now (and therefore read returns zeros). User may start qemu and specify backing file by options, or set backing file in the image, etc. So, the information does make sense anyway.

I think it would be good to start saying about backing chains explicitly, and not hide them under "allocated" concept which has different meanings.

Signature · 2 lines
Best regards,
Vladimir
#18 lore raw

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

In reply to Eric Blake
Quoted 73 lines · Eric Blake

To save the user from having to check 'qemu-img info --backing-chain' or other followup command to determine which "depth":n goes beyond the chain, add a boolean field "backing" that is set only for unallocated portions of the disk.

---

Touches the same iotest output as 1/1. If we decide that switching to "depth":n+1 is too risky, and that the mere addition of "backing":true while keeping "depth":n is good enough, then we'd have just one patch, instead of this double churn. Preferences?

 docs/tools/qemu-img.rst    |  3 ++
 qapi/block-core.json       |  7 ++-
 qemu-img.c                 | 15 +++++-
 tests/qemu-iotests/122.out | 34 +++++++-------
 tests/qemu-iotests/154.out | 96 +++++++++++++++++++-------------------
 tests/qemu-iotests/179.out | 66 +++++++++++++-------------
 tests/qemu-iotests/223.out | 24 +++++-----
 tests/qemu-iotests/244.out |  6 +--
 tests/qemu-iotests/252.out |  4 +-
 tests/qemu-iotests/274.out | 16 +++----
 tests/qemu-iotests/291.out |  8 ++--
 tests/qemu-iotests/309.out |  4 +-
 12 files changed, 150 insertions(+), 133 deletions(-)
docs/tools/qemu-img.rst +3 -0
index c155b1bf3cc8..fbc623b645c3 100644
@@ -601,6 +601,9 @@ Command description:
     a ``depth``; for example, a depth of 2 refers to the backing file
     of the backing file of *FILENAME*.  Depth will be one larger than
     the chain length if no file in the chain provides the data.
+  - an optional ``backing`` field is present with value true if no
+    file in the backing chain provides the data (making it easier to
+    identify when ``depth`` exceeds the chain length).
 
   In JSON format, the ``offset`` field is optional; it is absent in
   cases where ``human`` format would omit the entry or exit with an error.
qapi/block-core.json +5 -2
index 2ea294129e08..cebe12ba16a0 100644
@@ -264,6 +264,9 @@
 # @offset: if present, the image file stores the data for this range
 #          in raw format at the given (host) offset
 #
+# @backing: if present, the range is not allocated within the backing
+#           chain (since 6.1)
+#
 # @filename: filename that is referred to by @offset
 #
 # Since: 2.6
@@ -271,8 +274,8 @@
 ##
 { 'struct': 'MapEntry',
   'data': {'start': 'int', 'length': 'int', 'data': 'bool',
-           'zero': 'bool', 'depth': 'int', '*offset': 'int',
-           '*filename': 'str' } }
+           'zero': 'bool', 'depth': 'int', '*backing': 'bool',
+           '*offset': 'int', '*filename': 'str' } }
 
 ##
 # @BlockdevCacheInfo:
qemu-img.c +0 -0
index 33a5cd012b8b..4d357f534803 100644
@@ -2977,8 +2977,13 @@ static int dump_map_entry(OutputFormat output_format, MapEntry *e,
         break;

qemu-img.c static int dump_map_entry(OutputFormat output_format, MapEntry *e, · line 2985

Quoted 10 lines
qemu-img.c +6 -2
     case OFORMAT_JSON:
         printf("{ \"start\": %"PRId64", \"length\": %"PRId64","
-               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
-               e->start, e->length, e->depth,
+               " \"depth\": %"PRId64, e->start, e->length, e->depth);
+        if (e->has_backing) {
+            /* Backing should only be set at the end of the chain */
+            assert(e->backing && e->depth > 0);
+            printf(", \"backing\": true");
+        }

It will be easier to inspect the output if common fields come before optional fields.

Quoted 4 lines
qemu-img.c +1 -0
+        printf(", \"zero\": %s, \"data\": %s",
                e->zero ? "true" : "false",
                e->data ? "true" : "false");
         if (e->has_offset) {

...

tests/qemu-iotests/122.out read 65536/65536 bytes at offset 4194304 · line 71

Quoted 10 lines
tests/qemu-iotests/122.out +1 -1
index 779dab4847f0..c5aa2c9866f1 100644
@@ -68,11 +68,11 @@ read 65536/65536 bytes at offset 4194304
 read 65536/65536 bytes at offset 8388608
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
-{ "start": 65536, "length": 4128768, "depth": 1, "zero": true, "data": false},
+{ "start": 65536, "length": 4128768, "depth": 1, "backing": true, "zero": true, "data": false},

So this output would be:

    [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
     { "start": 65536, "length": 4128768, "depth": 1, "zero": true,

"data": false, "backing": true},