block/null: Add read-pattern
cover block/null: Add read-pattern
The null driver is very usefull for testing and benchmarking, but it can not emulate an image full of zeroes or an image full of non-zero bytes. Such images are needed for testing computing a blkhash via qemu-nbd or qemu-storage-daemon.
This change adds read-pattern option allowing emulution of image full of zeroes and image for of non-zero bytes.
I used this for testing
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html
Nir Soffer (2):
block/null: Report DATA if not reading zeroes
block/null: Add read-pattern option
block/null.c | 21 ++++++++++++++++++--- qapi/block-core.json | 9 ++++++++- 2 files changed, 26 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.5 (Apple Git-154)
PATCH 1/2 block/null: Report DATA if not reading zeroes
If read-zeroes is not set, we did not report BDRV_BLOCK_DATA or BDRV_BLOCK_ZERO. This is not consistent with other drivers and can confuse users or other programs:
% qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}"
[{ "start": 0, "length": 1073741824, "depth": 0, "present": false, "zero": false, "data": false, "compressed": false}]
% qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" &
% nbdinfo --map nbd://127.0.0.1
0 1073741824 1 holeWith this change we report DATA in this case:
% ./qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}"
[{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero": false, "data": true, "compressed": false, "offset": 0}]
% ./qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" &
% nbdinfo --map nbd://127.0.0.1
0 1073741824 0 data--- block/null.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
block/null.c +1 -3
@@ -239,9 +239,7 @@ static int coroutine_fn null_co_block_status(BlockDriverState *bs, *map = offset; *file = bs; -if (s->read_zeroes) { - ret |=BDRV_BLOCK_ZERO ; - } + ret |= s->read_zeroes ?BDRV_BLOCK_ZERO :BDRV_BLOCK_DATA ;return ret; }
Signature · 1 line
2.39.5 (Apple Git-154)
PATCH 2/2 block/null: Add read-pattern option
In reply to Nir SofferWhen the read-zeroes is set, reads produce zeroes, and block status return BDRV_BLOCK_ZERO, emulating a sparse image.
If we don't set read-zeros we report BDRV_BLOCK_DATA, but image data is undefined; posix_memalign, _aligned_malloc, valloc, or memalign do not promise to zero allocated memory.
When computing a blkhash of an image via qemu-nbd, we want to test 3 cases:
1. Sparse image: skip reading the entire image based on block status result, and use a pre-computed zero block hash.
2. Image full of zeroes: read the entire image, detect block full of zeroes and skip block hash computation.
3. Image full of data: read the entire image and compute a hash of all blocks.
This change adds read-pattern option. If the option is set, reads produce the specified pattern. With this option we can emulate an image full of zeroes or full of non-zeroes.
Specifying both read-zeroes and read-pattern != 0 is not useful since read-zeroes implies a sparse image. In this case read-zeroes wins and we ignore the pattern. Maybe we need to make the options mutual exclusive.
The following examples shows how the new option can be used with blksum (or nbdcopy --blkhash) to compute a blkhash of an image using the null-co driver.
Sparse image - the very fast path:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/sparse.sock \
"json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-zeroes': true}}" &
% time blksum 'nbd+unix:///?socket=/tmp/sparse.sock'
300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/sparse.sock
blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 0.05s user 0.01s system 92% cpu 0.061 totalImage full of zeros - same hash, 268 times slower:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/zero.sock \
"json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': 0}}" &
% time blksum 'nbd+unix:///?socket=/tmp/zero.sock'
300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/zero.sock
blksum 'nbd+unix:///?socket=/tmp/zero.sock' 7.45s user 22.57s system 183% cpu 16.347 totalImage full of data - difference hash, heavy cpu usage:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/data.sock \
"json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': -1}}" &
% time blksum 'nbd+unix:///?socket=/tmp/data.sock'
2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock
blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414 totalTested on top of
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html.
--- block/null.c | 17 +++++++++++++++++ qapi/block-core.json | 9 ++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
block/null.c +17 -0
@@ -22,11 +22,14 @@#define NULL_OPT_LATENCY "latency-ns"#define NULL_OPT_ZEROES "read-zeroes" +#define NULL_OPT_PATTERN "read-pattern"typedef struct {int64_t length;int64_t latency_ns;bool read_zeroes; +bool has_read_pattern; +int read_pattern; }BDRVNullState ;static QemuOptsList runtime_opts = {
@@ -49,6 +52,11 @@ static QemuOptsList runtime_opts = { .type = QEMU_OPT_BOOL, .help = "return zeroes when read", }, + { + .name = NULL_OPT_PATTERN, + .type = QEMU_OPT_NUMBER, + .help = "return pattern when read", + }, {/* end of list */ } }, };
@@ -95,6 +103,10 @@ static int null_open(BlockDriverState *bs, QDict *options, int flags, ret = -EINVAL ; } s->read_zeroes =qemu_opt_get_bool (opts,NULL_OPT_ZEROES , false); + s->has_read_pattern =qemu_opt_find (opts,NULL_OPT_PATTERN ) !=NULL ; +if (s->has_read_pattern) { + s->read_pattern =qemu_opt_get_number (opts,NULL_OPT_PATTERN ,0 ); + }qemu_opts_del (opts); bs->supported_write_flags =BDRV_REQ_FUA ;return ret;
@@ -125,6 +137,8 @@ static coroutine_fn int null_co_preadv(BlockDriverState *bs,if (s->read_zeroes) {qemu_iovec_memset (qiov,0 ,0 , bytes); + }else if (s->has_read_pattern) { +qemu_iovec_memset (qiov,0 , s->read_pattern, bytes); }return null_co_common (bs);
@@ -199,6 +213,8 @@ static BlockAIOCB *null_aio_preadv(BlockDriverState *bs,if (s->read_zeroes) {qemu_iovec_memset (qiov,0 ,0 , bytes); + }else if (s->has_read_pattern) { +qemu_iovec_memset (qiov,0 , s->read_pattern, bytes); }return null_aio_common (bs, cb, opaque);
@@ -272,6 +288,7 @@ null_co_get_allocated_file_size(BlockDriverState *bs)static const char *const null_strong_runtime_opts[] = {BLOCK_OPT_SIZE ,NULL_OPT_ZEROES , +NULL_OPT_PATTERN ,NULL };
qapi/block-core.json +8 -1
@@ -3297,10 +3297,17 @@ # false, the buffer is left unchanged. # (default: false; since: 4.1) # +# @read-pattern: if set, reads from the device produce the specified +# pattern; if unset, the buffer is left unchanged. +# (since: 10.1) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsNull', - 'data': { '*size': 'int', '*latency-ns': 'uint64', '*read-zeroes': 'bool' } } + 'data': { '*size': 'int', + '*latency-ns': 'uint64', + '*read-zeroes': 'bool', + '*read-pattern': 'int' } } ## # @BlockdevOptionsNVMe:
Signature · 1 line
2.39.5 (Apple Git-154)
Re: [PATCH 1/2] block/null: Report DATA if not reading zeroes
In reply to Nir SofferQuoted 12 lines · Nir Soffer
If read-zeroes is not set, we did not report
BDRV_BLOCK_DATAorBDRV_BLOCK_ZERO. This is not consistent with other drivers and can confuse users or other programs:% qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" [{ "start": 0, "length": 1073741824, "depth": 0, "present": false, "zero": false, "data": false, "compressed": false}] % qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" & % nbdinfo --map nbd://127.0.0.1 0 1073741824 1 hole
Quoted 4 lines
With this change we report DATA in this case:
% ./qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" [{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero": false, "data": true, "compressed": false, "offset": 0}]
Do we really want to represent the region as present? I think we're okay (we guarantee that reads from the region will return sensible contents), and your next patch to allow control over what data is seen makes it even better.
Quoted 16 lines
% ./qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" & % nbdinfo --map nbd://127.0.0.1 0 1073741824 0 dataSigned-off-by Nir Soffer <[email protected]>--- block/null.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)block/null.c +0
-0
block/null.c static int coroutine_fn null_co_block_status(BlockDriverState *bs, · line 243
Quoted 8 lines
block/null.c +1
-3@@ -239,9 +239,7 @@ static int coroutine_fn null_co_block_status(BlockDriverState *bs, *map = offset; *file = bs; -if (s->read_zeroes) { - ret |=BDRV_BLOCK_ZERO ; - } + ret |= s->read_zeroes ?BDRV_BLOCK_ZERO :BDRV_BLOCK_DATA ;return ret;
According to include/block/block-common.h,
* BDRV_BLOCK_DATA: allocation for data at offset is tied to this layer
* BDRV_BLOCK_ZERO: offset reads as zero
* BDRV_BLOCK_OFFSET_VALID: an associated offset exists for accessing raw data
Or read differently, BDRV_BLOCK_DATA says an area is not sparse. This patch is changing the null driver to report the entire image as either sparse and zero, or not sparse and not zero.
For testing purposes, I think that is reasonable enough; it matches that file-posix.c always returns one or the other, when using only lseek() to probe things. NBD has a bit finer-grained control (it has two bits, one for sparseness which is effectively !BDRV_BLOCK_DATA, and one for zero; where you can have something that is sparse but does not read as zero, such as SCSI that lacks coherent uninitialized read). In fact, you could even have DATA|ZERO if we know something is allocated AND that it reads as zero (which will be the case in your next patch if the user explicitly asks for a read pattern of 0).
But for all of that thinking, in the end your patch makes sense to me.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Re: [PATCH 2/2] block/null: Add read-pattern option
In reply to Nir SofferQuoted 17 lines · Nir Soffer
When the
read-zeroesis set, reads produce zeroes, and block status returnBDRV_BLOCK_ZERO, emulating a sparse image.If we don't set
read-zeroswe reportBDRV_BLOCK_DATA, but image data is undefined;posix_memalign,_aligned_malloc, valloc, or memalign do not promise to zero allocated memory.When computing a blkhash of an image via qemu-nbd, we want to test 3 cases:
1. Sparse image: skip reading the entire image based on block status result, and use a pre-computed zero block hash.
2. Image full of zeroes: read the entire image, detect block full of zeroes and skip block hash computation.
3. Image full of data: read the entire image and compute a hash of all blocks.
Quoted 8 lines
This change adds
read-patternoption. If the option is set, reads produce the specified pattern. With this option we can emulate an image full of zeroes or full of non-zeroes.Specifying both
read-zeroesandread-pattern != 0is not useful sinceread-zeroesimplies a sparse image. In this caseread-zeroeswins and we ignore the pattern. Maybe we need to make the options mutual exclusive.
I would lean towards an error. It's easier to remove an error later if we find it was too strict, than to be lax up front and then regret it down the road when we wish we had been more strict.
Quoted 32 lines
The following examples shows how the new option can be used with blksum (or nbdcopy --blkhash) to compute a blkhash of an image using the null-co driver.
Sparse image - the very fast path:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/sparse.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-zeroes': true}}" & % time blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/sparse.sock blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 0.05s user 0.01s system 92% cpu 0.061 totalImage full of zeros - same hash, 268 times slower:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/zero.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': 0}}" & % time blksum 'nbd+unix:///?socket=/tmp/zero.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/zero.sock blksum 'nbd+unix:///?socket=/tmp/zero.sock' 7.45s user 22.57s system 183% cpu 16.347 totalImage full of data - difference hash, heavy cpu usage:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/data.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': -1}}" & % time blksum 'nbd+unix:///?socket=/tmp/data.sock' 2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414 total
Quoted 8 lines
Tested on top of
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html.Signed-off-by Nir Soffer <[email protected]>--- block/null.c | 17 +++++++++++++++++ qapi/block-core.json | 9 ++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
Should block status return ZERO|DATA when read-pattern=0 is present?
qapi/block-core.json line 3310
Quoted 12 lines
qapi/block-core.json +8
-1+# @read-pattern: if set, reads from the device produce the specified +# pattern; if unset, the buffer is left unchanged. +# (since: 10.1) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsNull', - 'data': { '*size': 'int', '*latency-ns': 'uint64', '*read-zeroes': 'bool' } } + 'data': { '*size': 'int', + '*latency-ns': 'uint64', + '*read-zeroes': 'bool', + '*read-pattern': 'int' } }
Should this be 'uint8' instead of 'int', so that we aren't silently truncating spurious upper bits when passing it to memset()?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Re: [PATCH 2/2] block/null: Add read-pattern option
Quoted 22 lines · Eric Blake
Quoted 21 lines · Nir Soffer
When the
read-zeroesis set, reads produce zeroes, and block status returnBDRV_BLOCK_ZERO, emulating a sparse image.If we don't set
read-zeroswe reportBDRV_BLOCK_DATA, but image data is undefined;posix_memalign,_aligned_malloc, valloc, or memalign do not promise to zero allocated memory.When computing a blkhash of an image via qemu-nbd, we want to test 3 cases:
1. Sparse image: skip reading the entire image based on block status result, and use a pre-computed zero block hash.
2. Image full of zeroes: read the entire image, detect block full of zeroes and skip block hash computation.
3. Image full of data: read the entire image and compute a hash of all blocks.This change adds
read-patternoption. If the option is set, reads produce the specified pattern. With this option we can emulate an image full of zeroes or full of non-zeroes.
Quoted 8 lines
Quoted 4 lines
Specifying both
read-zeroesandread-pattern != 0is not useful sinceread-zeroesimplies a sparse image. In this caseread-zeroeswins and we ignore the pattern. Maybe we need to make the options mutual exclusive.I would lean towards an error. It's easier to remove an error later if we find it was too strict, than to be lax up front and then regret it down the road when we wish we had been more strict.
Seconded. Silently "fixing" the user's nonsensical instructions is commonly a bad idea.
Quoted 34 lines
Quoted 34 lines
The following examples shows how the new option can be used with blksum (or nbdcopy --blkhash) to compute a blkhash of an image using the null-co driver.
Sparse image - the very fast path:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/sparse.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-zeroes': true}}" & % time blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/sparse.sock blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 0.05s user 0.01s system 92% cpu 0.061 totalImage full of zeros - same hash, 268 times slower:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/zero.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': 0}}" & % time blksum 'nbd+unix:///?socket=/tmp/zero.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/zero.sock blksum 'nbd+unix:///?socket=/tmp/zero.sock' 7.45s user 22.57s system 183% cpu 16.347 totalImage full of data - difference hash, heavy cpu usage:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/data.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': -1}}" & % time blksum 'nbd+unix:///?socket=/tmp/data.sock' 2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414 totalTested on top of
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html.
Quoted 8 lines
Quoted 5 lines
Signed-off-by Nir Soffer <[email protected]>--- block/null.c | 17 +++++++++++++++++ qapi/block-core.json | 9 ++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)Should block status return ZERO|DATA when read-pattern=0 is present?
diff --git a/qapi/block-core.json b/qapi/block-core.json index b1937780e1..7d576cccbb 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3297,10 +3297,17 @@
# @read-zeroes: if true, reads from the device produce zeroes; if
# false, the buffer is left unchanged.
# (default: false; since: 4.1)
The commit message explains "read-zeroes": true behaves like a sparse image. The existing doc comment does not. I suspect it should.
#
Quoted 7 lines
Quoted 7 lines
qapi/block-core.json +4
-0+# @read-pattern: if set, reads from the device produce the specified +# pattern; if unset, the buffer is left unchanged. +# (since: 10.1) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsNull',
Quoted 8 lines
qapi/block-core.json line 3310
Quoted 5 lines
qapi/block-core.json +4
-1- 'data': { '*size': 'int', '*latency-ns': 'uint64', '*read-zeroes': 'bool' } } + 'data': { '*size': 'int', + '*latency-ns': 'uint64', + '*read-zeroes': 'bool', + '*read-pattern': 'int' } }Should this be 'uint8' instead of 'int', so that we aren't silently truncating spurious upper bits when passing it to
memset()?
Yes, please.
Without this, the doc comment does not sufficiently specify the contents of the image. "The specified pattern" could be read as
* Four bytes given by the 32 bit value of @read-pattern in big endian
* or in little endian
* or in host endian
In fact, it's none of the above, it's the least significant byte.
Please try to clarify the doc comment in addition to narrowing the type.
Re: [PATCH 1/2] block/null: Report DATA if not reading zeroes
In reply to Eric BlakeQuoted 19 lines · Eric Blake
Quoted 14 lines · Nir Soffer
If read-zeroes is not set, we did not report
BDRV_BLOCK_DATAorBDRV_BLOCK_ZERO. This is not consistent with other drivers and can confuse users or other programs:% qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" [{ "start": 0, "length": 1073741824, "depth": 0, "present": false, "zero": false, "data": false, "compressed": false}] % qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" & % nbdinfo --map nbd://127.0.0.1 0 1073741824 1 holeWith this change we report DATA in this case:
Quoted 7 lines
Quoted 2 lines
% ./qemu-img map --output json "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}"
[{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero": false, "data": true, "compressed": false, "offset": 0}]Do we really want to represent the region as present? I think we're okay (we guarantee that reads from the region will return sensible contents), and your next patch to allow control over what data is seen makes it even better.
I kept the existing behavior, but I think present=false is relevant only if you have a backing file (like qcow2), so for null-co it makes sense to behave like file driver.
Quoted 37 lines
Quoted 18 lines
% ./qemu-nbd "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" & % nbdinfo --map nbd://127.0.0.1 0 1073741824 0 dataSigned-off-by Nir Soffer <[email protected]>---
block/null.c| 4 +---1 file changed, 1 insertion(+), 3 deletions(-)block/null.c +0
-0block/null.c static int coroutine_fn null_co_block_status(BlockDriverState *bs, · line 243
Quoted 8 lines
block/null.c +1
-3@@ -239,9 +239,7 @@ static int coroutine_fn null_co_block_status(BlockDriverState *bs, *map = offset; *file = bs; -if (s->read_zeroes) { - ret |=BDRV_BLOCK_ZERO ; - } + ret |= s->read_zeroes ?BDRV_BLOCK_ZERO :BDRV_BLOCK_DATA ;return ret;According to
include/block/block-common.h,*
BDRV_BLOCK_DATA: allocation for data at offset is tied to this layer
*BDRV_BLOCK_ZERO: offset reads as zero
*BDRV_BLOCK_OFFSET_VALID: an associated offset exists for accessing raw dataOr read differently,
BDRV_BLOCK_DATAsays an area is not sparse. This patch is changing the null driver to report the entire image as either sparse and zero, or not sparse and not zero.
Quoted 9 lines
For testing purposes, I think that is reasonable enough; it matches that
file-posix.calways returns one or the other, when using onlylseek()to probe things. NBD has a bit finer-grained control (it has two bits, one for sparseness which is effectively !BDRV_BLOCK_DATA, and one for zero; where you can have something that is sparse but does not read as zero, such as SCSI that lacks coherent uninitialized read). In fact, you could even have DATA|ZERO if we know something is allocated AND that it reads as zero (which will be the case in your next patch if the user explicitly asks for a read pattern of 0).
If we want this we can remove read_zeros, since read_pattern support reading zeros, and have:
*read_pattern: uint8 (default unset)
*block_status: “data"|”zero” (default data?)
*allocated: bool (default true)
We can replace reads-zeros=on with read-pattern=0 in tests and docs, but it may break other users using null-co.
Maybe keep read_zeros for backward compatibility, and make it conflict with the new options?
Quoted 9 lines
But for all of that thinking, in the end your patch makes sense to me.
Reviewed-by Eric Blake <[email protected]>Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Re: [PATCH 2/2] block/null: Add read-pattern option
In reply to Eric BlakeQuoted 26 lines · Eric Blake
Quoted 21 lines · Nir Soffer
When the
read-zeroesis set, reads produce zeroes, and block status returnBDRV_BLOCK_ZERO, emulating a sparse image.If we don't set
read-zeroswe reportBDRV_BLOCK_DATA, but image data is undefined;posix_memalign,_aligned_malloc, valloc, or memalign do not promise to zero allocated memory.When computing a blkhash of an image via qemu-nbd, we want to test 3 cases:
1. Sparse image: skip reading the entire image based on block status result, and use a pre-computed zero block hash.
2. Image full of zeroes: read the entire image, detect block full of zeroes and skip block hash computation.
3. Image full of data: read the entire image and compute a hash of all blocks.This change adds
read-patternoption. If the option is set, reads produce the specified pattern. With this option we can emulate an image full of zeroes or full of non-zeroes.
Quoted 8 lines
Quoted 4 lines
Specifying both
read-zeroesandread-pattern != 0is not useful sinceread-zeroesimplies a sparse image. In this caseread-zeroeswins and we ignore the pattern. Maybe we need to make the options mutual exclusive.I would lean towards an error. It's easier to remove an error later if we find it was too strict, than to be lax up front and then regret it down the road when we wish we had been more strict.
Good point
Quoted 36 lines
Quoted 36 lines
The following examples shows how the new option can be used with blksum (or nbdcopy --blkhash) to compute a blkhash of an image using the null-co driver.
Sparse image - the very fast path:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/sparse.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-zeroes': true}}" &% time blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/sparse.sock blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 0.05s user 0.01s system 92% cpu 0.061 totalImage full of zeros - same hash, 268 times slower:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/zero.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': 0}}" &% time blksum 'nbd+unix:///?socket=/tmp/zero.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/zero.sock blksum 'nbd+unix:///?socket=/tmp/zero.sock' 7.45s user 22.57s system 183% cpu 16.347 totalImage full of data - difference hash, heavy cpu usage:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/data.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': -1}}" &% time blksum 'nbd+unix:///?socket=/tmp/data.sock' 2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414 totalTested on top of
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html.
Quoted 7 lines
Quoted 5 lines
Signed-off-by Nir Soffer <[email protected]>---
block/null.c| 17 +++++++++++++++++qapi/block-core.json| 9 ++++++++-2 files changed, 25 insertions(+), 1 deletion(-)Should block status return ZERO|DATA when read-pattern=0 is present?
This will make it useless for testing image full of zeros which is the reason I added it. qemu-img convert, nbdcopy, and blksum look only at the zero bit and skip reading the extent.
Re: [PATCH 2/2] block/null: Add read-pattern option
In reply to Markus ArmbrusterQuoted 33 lines · Markus Armbruster
Quoted 27 lines · Eric Blake
Quoted 25 lines · Nir Soffer
When the
read-zeroesis set, reads produce zeroes, and block status returnBDRV_BLOCK_ZERO, emulating a sparse image.If we don't set
read-zeroswe reportBDRV_BLOCK_DATA, but image data is undefined;posix_memalign,_aligned_malloc, valloc, or memalign do not promise to zero allocated memory.When computing a blkhash of an image via qemu-nbd, we want to test 3 cases:
1. Sparse image: skip reading the entire image based on block status result, and use a pre-computed zero block hash.
2. Image full of zeroes: read the entire image, detect block full of zeroes and skip block hash computation.
3. Image full of data: read the entire image and compute a hash of all blocks.This change adds
read-patternoption. If the option is set, reads produce the specified pattern. With this option we can emulate an image full of zeroes or full of non-zeroes.Specifying both
read-zeroesandread-pattern != 0is not useful sinceread-zeroesimplies a sparse image. In this caseread-zeroeswins and we ignore the pattern. Maybe we need to make the options mutual exclusive.
Quoted 6 lines
Quoted 3 lines
I would lean towards an error. It's easier to remove an error later if we find it was too strict, than to be lax up front and then regret it down the road when we wish we had been more strict.
Seconded. Silently "fixing" the user's nonsensical instructions is commonly a bad idea.
Thanks, making the options mutually exclusive.
Quoted 79 lines
Quoted 35 lines
Quoted 35 lines
The following examples shows how the new option can be used with blksum (or nbdcopy --blkhash) to compute a blkhash of an image using the null-co driver.
Sparse image - the very fast path:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/sparse.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-zeroes': true}}" &% time blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/sparse.sock blksum 'nbd+unix:///?socket=/tmp/sparse.sock' 0.05s user 0.01s system 92% cpu 0.061 totalImage full of zeros - same hash, 268 times slower:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/zero.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': 0}}" &% time blksum 'nbd+unix:///?socket=/tmp/zero.sock' 300ad1efddb063822fea65ae3174cd35320939d4d0b050613628c6e1e876f8f6 nbd+unix:///?socket=/tmp/zero.sock blksum 'nbd+unix:///?socket=/tmp/zero.sock' 7.45s user 22.57s system 183% cpu 16.347 totalImage full of data - difference hash, heavy cpu usage:
% ./qemu-nbd -r -t -e 0 -f raw -k /tmp/data.sock \ "json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '100g', 'read-pattern': -1}}" &% time blksum 'nbd+unix:///?socket=/tmp/data.sock' 2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414 totalTested on top of
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html.Quoted 8 lines
Quoted 5 lines
Signed-off-by Nir Soffer <[email protected]>---
block/null.c| 17 +++++++++++++++++qapi/block-core.json| 9 ++++++++-2 files changed, 25 insertions(+), 1 deletion(-)Should block status return ZERO|DATA when read-pattern=0 is present?
diff --git
a/qapi/block-core.jsonb/qapi/block-core.jsonindexb1937780e1..7d576cccbb100644
---a/qapi/block-core.json
+++b/qapi/block-core.json
@@ -3297,10 +3297,17 @@
# @read-zeroes: if true, reads from the device produce zeroes; if
# false, the buffer is left unchanged.
# (default: false; since: 4.1)The commit message explains "read-zeroes": true behaves like a sparse image. The existing doc comment does not. I suspect it should.
#Quoted 7 lines
Quoted 7 lines
qapi/block-core.json +4
-0+# @read-pattern: if set, reads from the device produce the specified +# pattern; if unset, the buffer is left unchanged. +# (since: 10.1) +## Since: 2.9
##
{ 'struct': 'BlockdevOptionsNull',Quoted 8 lines
qapi/block-core.json line 3310
Quoted 5 lines
qapi/block-core.json +4
-1- 'data': { '*size': 'int', '*latency-ns': 'uint64', '*read-zeroes': 'bool' } } + 'data': { '*size': 'int', + '*latency-ns': 'uint64', + '*read-zeroes': 'bool', + '*read-pattern': 'int' } }Should this be 'uint8' instead of 'int', so that we aren't silently truncating spurious upper bits when passing it to
memset()?Yes, please.
Without this, the doc comment does not sufficiently specify the contents of the image. "The specified pattern" could be read as
* Four bytes given by the 32 bit value of @read-pattern in big endian
Quoted 7 lines
* or in little endian
* or in host endian
In fact, it's none of the above, it's the least significant byte.
Please try to clarify the doc comment in addition to narrowing the type.
Fixing in v2.