quorum: Implement bdrv_co_block_status()
PATCH v3 2/2 quorum: Implement bdrv_co_pwrite_zeroes()
In reply to Alberto GarciaThis simply calls bdrv_co_pwrite_zeroes() in all children
--- block/quorum.c | 18 ++++++++++++++++-- tests/qemu-iotests/312 | 7 +++++++ tests/qemu-iotests/312.out | 4 ++++ 3 files changed, 27 insertions(+), 2 deletions(-)
block/quorum.c +16 -2
@@ -692,8 +692,13 @@ static void write_quorum_entry(void *opaque)QuorumChildRequest *sacb = &acb->qcrs[i]; sacb->bs = s->children[i]->bs; - sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, - acb->qiov, acb->flags); +if (acb->flags &BDRV_REQ_ZERO_WRITE ) { + sacb->ret =bdrv_co_pwrite_zeroes (s->children[i], acb->offset, + acb->bytes, acb->flags); + }else { + sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, + acb->qiov, acb->flags); + }if (sacb->ret ==0 ) { acb->success_count++; }else {
@@ -739,6 +744,14 @@ static int quorum_co_pwritev(BlockDriverState *bs, uint64_t offset,return ret; } +static int quorum_co_pwrite_zeroes (BlockDriverState *bs,int64_t offset, +int bytes,BdrvRequestFlags flags) + +{ +return quorum_co_pwritev (bs, offset, bytes,NULL , + flags |BDRV_REQ_ZERO_WRITE ); +} +static int64_t quorum_getlength (BlockDriverState *bs) {BDRVQuorumState *s = bs->opaque;
@@ -1251,6 +1264,7 @@ static BlockDriver bdrv_quorum = { .bdrv_co_preadv = quorum_co_preadv, .bdrv_co_pwritev = quorum_co_pwritev, + .bdrv_co_pwrite_zeroes = quorum_co_pwrite_zeroes, .bdrv_add_child = quorum_add_child, .bdrv_del_child = quorum_del_child,
tests/qemu-iotests/312 +7 -0
@@ -114,6 +114,13 @@ $QEMU_IO -c "write -P 0 $((0x200000)) $((0x10000))" "$TEST_IMG.0" | _filter_qemu $QEMU_IO -c "write -z $((0x200000)) $((0x30000))" "$TEST_IMG.1" | _filter_qemu_io $QEMU_IO -c "write -P 0 $((0x200000)) $((0x20000))" "$TEST_IMG.2" | _filter_qemu_io +# Test 5: write data to a region and then zeroize it, doing it +# directly on the quorum device instead of the individual images. +# This has no effect on the end result but proves that the quorum driver +# supports 'write -z'. +$QEMU_IO -c "open -o $quorum" -c "write $((0x250000)) $((0x10000))" | _filter_qemu_io +$QEMU_IO -c "open -o $quorum" -c "write -z $((0x250000)) $((0x10000))" | _filter_qemu_io + echo echo '### Launch the drive-mirror job' echo
tests/qemu-iotests/312.out +4 -0
@@ -39,6 +39,10 @@ wrote 196608/196608 bytes at offset 2097152 192 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 131072/131072 bytes at offset 2097152 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 65536/65536 bytes at offset 2424832 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 65536/65536 bytes at offset 2424832 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ### Launch the drive-mirror job
Signature · 1 line
2.20.1
PATCH v3 1/2 quorum: Implement bdrv_co_block_status()
In reply to Alberto GarciaThe quorum driver does not implement bdrv_co_block_status() and because of that it always reports to contain data even if all its children are known to be empty.
One consequence of this is that if we for example create a quorum with a size of 10GB and we mirror it to a new image the operation will write 10GB of actual zeroes to the destination image wasting a lot of time and disk space.
Since a quorum has an arbitrary number of children of potentially different formats there is no way to report all possible allocation status flags in a way that makes sense, so this implementation only reports when a given region is known to contain zeroes (BDRV_BLOCK_ZERO) or not (BDRV_BLOCK_DATA).
If all children agree that a region contains zeroes then we can return BDRV_BLOCK_ZERO using the smallest size reported by the children (because all agree that a region of at least that size contains zeroes).
If at least one child disagrees we have to return BDRV_BLOCK_DATA. In this case we use the largest of the sizes reported by the children that didn't return BDRV_BLOCK_ZERO (because we know that there won't be an agreement for at least that size).
--- block/quorum.c | 52 +++++++++++++ tests/qemu-iotests/312 | 148 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/312.out | 67 +++++++++++++++++ tests/qemu-iotests/group | 1 + 4 files changed, 268 insertions(+) create mode 100755 tests/qemu-iotests/312 create mode 100644 tests/qemu-iotests/312.out
block/quorum.c +52 -0
@@ -18,6 +18,7 @@#include "qemu/module.h" #include "qemu/option.h" #include "block/block_int.h" +#include "block/coroutines.h" #include "block/qdict.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h"
@@ -1174,6 +1175,56 @@ static void quorum_child_perm(BlockDriverState *bs, BdrvChild *c, | DEFAULT_PERM_UNCHANGED; } +/* +* Each one of the children can report different status flags even +* when they contain the same data, so what this function does is +* return BDRV_BLOCK_ZERO if *all* children agree that a certain +* region contains zeroes, and BDRV_BLOCK_DATA otherwise. +*/ +static int coroutine_fn quorum_co_block_status(BlockDriverState *bs, +bool want_zero, +int64_t offset,int64_t count, +int64_t *pnum,int64_t *map, +BlockDriverState **file) +{ +BDRVQuorumState *s = bs->opaque; +int i, ret; +int64_t pnum_zero = count; +int64_t pnum_data =0 ; + +for (i =0 ; i < s->num_children; i++) { +int64_t bytes; + ret =bdrv_co_common_block_status_above (s->children[i]->bs,NULL , false, + want_zero, offset, count, + &bytes,NULL ,NULL ,NULL ); +if (ret <0 ) { +quorum_report_bad (QUORUM_OP_TYPE_READ , offset, count, + s->children[i]->bs->node_name, ret); + pnum_data = count; +break ; + } +/* +* Even if all children agree about whether there are zeroes +* or not at @offset they might disagree on the size, so use +* the smallest when reporting BDRV_BLOCK_ZERO and the largest +* when reporting BDRV_BLOCK_DATA. +*/ +if (ret &BDRV_BLOCK_ZERO ) { + pnum_zero =MIN (pnum_zero, bytes); + }else { + pnum_data =MAX (pnum_data, bytes); + } + } + +if (pnum_data) { + *pnum = pnum_data; +return BDRV_BLOCK_DATA ; + }else { + *pnum = pnum_zero; +return BDRV_BLOCK_ZERO ; + } +} +static const char *const quorum_strong_runtime_opts[] = {QUORUM_OPT_VOTE_THRESHOLD ,QUORUM_OPT_BLKVERIFY ,
@@ -1192,6 +1243,7 @@ static BlockDriver bdrv_quorum = { .bdrv_close = quorum_close, .bdrv_gather_child_options = quorum_gather_child_options, .bdrv_dirname = quorum_dirname, + .bdrv_co_block_status = quorum_co_block_status, .bdrv_co_flush_to_disk = quorum_co_flush,
tests/qemu-iotests/312 new file +148 -0
148 lines not on this page · raw message
tests/qemu-iotests/312.out new file +67 -0
67 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -316,3 +316,4 @@ 305 rw quick 307 rw quick export 309 rw auto quick +312 rw auto quick
Signature · 1 line
2.20.1
v3 cover quorum: Implement bdrv_co_block_status()
Hi,
I realized that if one of the children returns an error during co_block_status() then we should not pass the error immediately to the caller, because that's a situation that Quorum should be able to handle.
The new version of the patch takes the simpler approach of falling back to returning BDRV_BLOCK_DATA, as if Quorum did not implement bdrv_co_block_status(). This will force the caller to try to read the actual data and the normal Quorum voting and error handling process will be used.
Berto
v3:
- Fall back to BDRV_BLOCK_DATA if a child returns an error.
v2: https://lists.gnu.org/archive/html/qemu-block/2020-11/msg00259.html
- Implement bdrv_co_pwrite_zeroes() for quorum
v1: https://lists.gnu.org/archive/html/qemu-block/2020-11/msg00163.html
Alberto Garcia (2):
quorum: Implement bdrv_co_block_status()
quorum: Implement bdrv_co_pwrite_zeroes()
block/quorum.c | 70 ++++++++++++++++- tests/qemu-iotests/312 | 155 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/312.out | 71 +++++++++++++++++ tests/qemu-iotests/group | 1 + 4 files changed, 295 insertions(+), 2 deletions(-) create mode 100755 tests/qemu-iotests/312 create mode 100644 tests/qemu-iotests/312.out
Signature · 1 line
2.20.1
Quoted 27 lines · Alberto Garcia
The quorum driver does not implement
bdrv_co_block_status()and because of that it always reports to contain data even if all its children are known to be empty.One consequence of this is that if we for example create a quorum with a size of 10GB and we mirror it to a new image the operation will write 10GB of actual zeroes to the destination image wasting a lot of time and disk space.
Since a quorum has an arbitrary number of children of potentially different formats there is no way to report all possible allocation status flags in a way that makes sense, so this implementation only reports when a given region is known to contain zeroes (
BDRV_BLOCK_ZERO) or not (BDRV_BLOCK_DATA).If all children agree that a region contains zeroes then we can return
BDRV_BLOCK_ZEROusing the smallest size reported by the children (because all agree that a region of at least that size contains zeroes).If at least one child disagrees we have to return
BDRV_BLOCK_DATA. In this case we use the largest of the sizes reported by the children that didn't returnBDRV_BLOCK_ZERO(because we know that there won't be an agreement for at least that size).Signed-off-by Alberto Garcia <[email protected]>Tested-by Tao Xu <[email protected]>
Quoted 8 lines
--- block/quorum.c | 52 +++++++++++++ tests/qemu-iotests/312 | 148 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/312.out | 67 +++++++++++++++++ tests/qemu-iotests/group | 1 + 4 files changed, 268 insertions(+)create mode 100755 tests/qemu-iotests/312
create mode 100644 tests/qemu-iotests/312.out
Re: [PATCH v3 2/2] quorum: Implement bdrv_co_pwrite_zeroes()
In reply to Alberto GarciaQuoted 8 lines · Alberto Garcia
This simply calls
bdrv_co_pwrite_zeroes()in all childrenSigned-off-by Alberto Garcia <[email protected]>--- block/quorum.c | 18 ++++++++++++++++-- tests/qemu-iotests/312 | 7 +++++++ tests/qemu-iotests/312.out | 4 ++++ 3 files changed, 27 insertions(+), 2 deletions(-)
Should we set supported_zero_flags to something? I think we can at least set BDRV_REQ_NO_FALLBACK. We could also try BDRV_REQ_FUA.
Quoted 7 lines
block/quorum.c +0
-0@@ -692,8 +692,13 @@ static void write_quorum_entry(void *opaque)QuorumChildRequest *sacb = &acb->qcrs[i];
block/quorum.c static void write_quorum_entry(void *opaque) · line 701
Quoted 10 lines
block/quorum.c +7
-2sacb->bs = s->children[i]->bs; - sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, - acb->qiov, acb->flags); +if (acb->flags &BDRV_REQ_ZERO_WRITE ) { + sacb->ret =bdrv_co_pwrite_zeroes (s->children[i], acb->offset, + acb->bytes, acb->flags); + }else { + sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, + acb->qiov, acb->flags); + }
Seems unnecessary (bdrv_co_pwritev() can handle BDRV_REQ_ZERO_WRITE), but perhaps it’s good to be explicit.
Quoted 3 lines
block/quorum.c +0
-0if (sacb->ret ==0 ) { acb->success_count++; }else {
[...]
Quoted 5 lines
tests/qemu-iotests/312 +0
-0
tests/qemu-iotests/312 $QEMU_IO -c "write -P 0 $((0x200000)) $((0x10000))" "$TEST_IMG.0" | _filter_qemu · line 123
Quoted 10 lines
tests/qemu-iotests/312 +7
-0@@ -114,6 +114,13 @@ $QEMU_IO -c "write -P 0 $((0x200000)) $((0x10000))" "$TEST_IMG.0" | _filter_qemu $QEMU_IO -c "write -z $((0x200000)) $((0x30000))" "$TEST_IMG.1" | _filter_qemu_io $QEMU_IO -c "write -P 0 $((0x200000)) $((0x20000))" "$TEST_IMG.2" | _filter_qemu_io +# Test 5: write data to a region and then zeroize it, doing it +# directly on the quorum device instead of the individual images. +# This has no effect on the end result but proves that the quorum driver +# supports 'write -z'. +$QEMU_IO -c "open -o $quorum" -c "write $((0x250000)) $((0x10000))" | _filter_qemu_io +$QEMU_IO -c "open -o $quorum" -c "write -z $((0x250000)) $((0x10000))" | _filter_qemu_io +
My gut would have preferred a test where the data region is larger than the zeroed region (so we can see that the first write has done something), but who cares about my gut.
I don’t mind not setting supported_zero_flags enough to warrant withholding a
But I’ll give you some time to reply before I’d take this patch to block-next. (That is, unless Kevin takes it during my two-week PTO...)
Quoted 3 lines
tests/qemu-iotests/312 +0
-0echo echo '### Launch the drive-mirror job' echo
Re: [PATCH v3 2/2] quorum: Implement bdrv_co_pwrite_zeroes()
Quoted 10 lines · Max Reitz
Quoted 8 lines · Alberto Garcia
This simply calls
bdrv_co_pwrite_zeroes()in all childrenSigned-off-by Alberto Garcia <[email protected]>--- block/quorum.c | 18 ++++++++++++++++-- tests/qemu-iotests/312 | 7 +++++++ tests/qemu-iotests/312.out | 4 ++++ 3 files changed, 27 insertions(+), 2 deletions(-)
Quoted 2 lines
Should we set
supported_zero_flagsto something? I think we can at least setBDRV_REQ_NO_FALLBACK. We could also tryBDRV_REQ_FUA.
We could set all supported_zero_flags as long as all children support them, right?
Quoted 10 lines
block/quorum.c static void write_quorum_entry(void *opaque) · line 701
Quoted 7 lines
block/quorum.c +7
-0+if (acb->flags &BDRV_REQ_ZERO_WRITE ) { + sacb->ret =bdrv_co_pwrite_zeroes (s->children[i], acb->offset, + acb->bytes, acb->flags); + }else { + sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, + acb->qiov, acb->flags); + }Seems unnecessary (
bdrv_co_pwritev()can handleBDRV_REQ_ZERO_WRITE), but perhaps it’s good to be explicit.
pwrite_zeroes() does this additionaly:
if (!(child->bs->open_flags & BDRV_O_UNMAP)) {
flags &= ~BDRV_REQ_MAY_UNMAP;
}Berto
Re: [PATCH v3 2/2] quorum: Implement bdrv_co_pwrite_zeroes()
Quoted 11 lines · Alberto Garcia
Quoted 10 lines · Max Reitz
Quoted 8 lines · Alberto Garcia
This simply calls
bdrv_co_pwrite_zeroes()in all childrenSigned-off-by Alberto Garcia <[email protected]>--- block/quorum.c | 18 ++++++++++++++++-- tests/qemu-iotests/312 | 7 +++++++ tests/qemu-iotests/312.out | 4 ++++ 3 files changed, 27 insertions(+), 2 deletions(-)
Quoted 5 lines
Quoted 2 lines
Should we set
supported_zero_flagsto something? I think we can at least setBDRV_REQ_NO_FALLBACK. We could also tryBDRV_REQ_FUA.We could set all
supported_zero_flagsas long as all children support them, right?
Sure, I was just thinking that we could set these regardless of whether the children support them, because (on zero-writes) the block layer will figure out for us whether the child nodes support them. O:)
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
block/quorum.c +7
-0+if (acb->flags &BDRV_REQ_ZERO_WRITE ) { + sacb->ret =bdrv_co_pwrite_zeroes (s->children[i], acb->offset, + acb->bytes, acb->flags); + }else { + sacb->ret =bdrv_co_pwritev (s->children[i], acb->offset, acb->bytes, + acb->qiov, acb->flags); + }
Quoted 8 lines
Quoted 2 lines
Seems unnecessary (
bdrv_co_pwritev()can handleBDRV_REQ_ZERO_WRITE), but perhaps it’s good to be explicit.
pwrite_zeroes()does this additionaly:if (!(child->bs->open_flags & BDRV_O_UNMAP)) { flags &= ~BDRV_REQ_MAY_UNMAP; }
Interesting. Technically, Quorum doesn’t support that flag (in supported_zero_flags O:))), so it shouldn’t appear, but, er, well then.
Max
Re: [PATCH v3 2/2] quorum: Implement bdrv_co_pwrite_zeroes()
Quoted 6 lines · Max Reitz
Quoted 2 lines
We could set all
supported_zero_flagsas long as all children support them, right?Sure, I was just thinking that we could set these regardless of whether the children support them, because (on zero-writes) the block layer will figure out for us whether the child nodes support them. O:)
But it can happen that one child supports e.g. BDRV_REQ_NO_FALLBACK but the rest don't. In this case I think the block layer should return -ENOTSUP earlier without writing to the child(ren) that do support that flag.
So Quorum's supported_zero_flags would be the logical and of all of its children's flags, right?
I'm unsure about BDRV_REQ_WRITE_UNCHANGED, many filters set that on top of the other flags, but when would a BDS not support this flag?
Quoted 9 lines
Quoted 5 lines
pwrite_zeroes()does this additionaly:if (!(child->bs->open_flags & BDRV_O_UNMAP)) { flags &= ~BDRV_REQ_MAY_UNMAP; }Interesting. Technically, Quorum doesn’t support that flag (in
supported_zero_flagsO:))), so it shouldn’t appear, but, er, well then.
It would with the change that I'm proposing above.
Berto
Re: [PATCH v3 2/2] quorum: Implement bdrv_co_pwrite_zeroes()
Quoted 5 lines · Alberto Garcia
Quoted 3 lines · Max Reitz
Quoted 2 lines
We could set all
supported_zero_flagsas long as all children support them, right?
Quoted 8 lines
Quoted 3 lines
Sure, I was just thinking that we could set these regardless of whether the children support them, because (on zero-writes) the block layer will figure out for us whether the child nodes support them. O:)
But it can happen that one child supports e.g.
BDRV_REQ_NO_FALLBACKbut the rest don't. In this case I think the block layer should return-ENOTSUPearlier without writing to the child(ren) that do support that flag.
That’s true.
Quoted 2 lines
So Quorum's
supported_zero_flagswould be the logical and of all of its children's flags, right?
Yes. (And so it would need to be recalculated whenever a child is added or removed.)
Quoted 2 lines
I'm unsure about
BDRV_REQ_WRITE_UNCHANGED, many filters set that on top of the other flags, but when would a BDS not support this flag?
The WRITE_UNCHANGED flag is basically just a hint for the permission system that for such a write, the WRITE_UNCHANGED permission is sufficient. So drivers that can pass through *all* WRITE_UNCHANGED requests as they are (i.e., filter drivers) can support this write/zero flag and then pass that flag on. This way, they are safe not to take the WRITE permission on their child unless their parent has taken the WRITE permission on them.
(Otherwise, they’d also have to take the WRITE permission if the parent only holds the WRITE_UNCHANGED permission.)
Supporting this flag doesn’t make sense for drivers that won’t be able to pass it on all the time. For example, qcow2 generally cannot pass it on, because it still needs to perform WRITE_UNCHANGED requests as normal writes. (WRITE_UNCHANGED comes from copy-on-read; the data will read the same, hence the _UNCHANGED, but it still needs to be allocated on the receiving format layer.)
(Technically, qcow2 could figure out whether the requests it generates from a WRITE_UNCHANGED request still result in unchanging writes in the protocol layer, but there is no reason to. It needs the WRITE permission anyway, because there are going to be some non-unchanging writes it has to perform. And if it holds the WRITE permission, there is no need to bother with the WRITE_UNCHANGED flag.)
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
pwrite_zeroes()does this additionaly:if (!(child->bs->open_flags & BDRV_O_UNMAP)) { flags &= ~BDRV_REQ_MAY_UNMAP; }Interesting. Technically, Quorum doesn’t support that flag (in
supported_zero_flagsO:))), so it shouldn’t appear, but, er, well then.It would with the change that I'm proposing above.
Yes, I know. Hence the “O:)”. O:)
Max