qcow2: Fix data corruption on XFS
cover qcow2: Fix data corruption on XFS
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore, bdrv_co_get_self_request() has been moved to block/io.c.
Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in handle_alloc_space() that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.
Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear that c8bb23cbdbe causes fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases, c8bb23cbdbe improves performance or we wouldn’t have it.
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Thus, I believe we should revert the commit for now (and most importantly for 4.1.1). We can think about reintroducing it for 5.0, but that would require more extensive benchmarks on a variety of systems, and we must see how subclusters change the picture.
I would have liked to do benchmarks myself before making this decision, but as far as I’m informed, patches for 4.1.1 are to be collected on Monday, so we need to be quick.
git-backport-diff against the RFC:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/4:[down] 'Revert "qcow2: skip writing zero buffers to empty COW areas"' 002/4:[----] [-C] 'block: Make wait/mark serialising requests public' 003/4:[down] 'block: Add bdrv_co_get_self_request()' 004/4:[0036] [FC] 'block/file-posix: Let post-EOF fallocate serialize'
Max Reitz (4):
Revert "qcow2: skip writing zero buffers to empty COW areas"
block: Make wait/mark serialising requests public
block: Add bdrv_co_get_self_request() block/file-posix: Let post-EOF fallocate serialize
qapi/block-core.json | 4 +- block/qcow2.h | 6 --- include/block/block_int.h | 4 ++ block/file-posix.c | 38 +++++++++++++++++ block/io.c | 42 +++++++++++++------ block/qcow2-cluster.c | 2 +- block/qcow2.c | 86 -------------------------------------- block/trace-events | 1 - tests/qemu-iotests/060 | 7 +--- tests/qemu-iotests/060.out | 5 +-- 10 files changed, 76 insertions(+), 119 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/4 Revert "qcow2: skip writing zero buffers to empty COW areas"
This reverts commit c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.
This commit causes fundamental performance problems on XFS (because fallocate() stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.
We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
If we want to reintroduce something similar to c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.
--- qapi/block-core.json | 4 +- block/qcow2.h | 6 --- block/qcow2-cluster.c | 2 +- block/qcow2.c | 86 -------------------------------------- block/trace-events | 1 - tests/qemu-iotests/060 | 7 +--- tests/qemu-iotests/060.out | 5 +-- 7 files changed, 4 insertions(+), 107 deletions(-)
qapi/block-core.json +1 -3
@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1) # # Since: 2.9
@@ -3326,7 +3324,7 @@ 'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev', 'pwritev_zero', 'pwritev_done', 'empty_image_prepare', 'l1_shrink_write_table', 'l1_shrink_free_l2_clusters', - 'cor_write', 'cluster_alloc_space', 'none'] } + 'cor_write', 'none'] } ## # @BlkdebugIOType:
block/qcow2.h +0 -6
@@ -418,12 +418,6 @@ typedef struct QCowL2Meta*/ Qcow2COWRegion cow_end; -/* -* Indicates that COW regions are already handled and do not require -* any more processing. -*/ -bool skip_cow; -/** * The I/O vector with the data from the actual guest write request. * If non-NULL, this is meant to be merged together with the data
block/qcow2-cluster.c +1 -1
@@ -809,7 +809,7 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta *m)assert (start->nb_bytes + end->nb_bytes <=UINT_MAX - data_bytes);assert (start->offset + start->nb_bytes <= end->offset); -if ((start->nb_bytes ==0 && end->nb_bytes ==0 ) || m->skip_cow) { +if (start->nb_bytes ==0 && end->nb_bytes ==0 ) {return 0 ; }
block/qcow2.c +0 -86
104 lines not on this page · raw message
block/trace-events +0 -1
@@ -72,7 +72,6 @@ qcow2_writev_done_part(void *co, int cur_bytes) "co %p cur_bytes %d" qcow2_writev_data(void *co, uint64_t offset) "co %p offset 0x%" PRIx64 qcow2_pwrite_zeroes_start_req(void *co, int64_t offset, int count) "co %p offset 0x%" PRIx64 " count %d" qcow2_pwrite_zeroes(void *co, int64_t offset, int count) "co %p offset 0x%" PRIx64 " count %d" -qcow2_skip_cow(void *co, uint64_t offset, int nb_clusters) "co %p offset 0x%" PRIx64 " nb_clusters %d" # qcow2-cluster.c qcow2_alloc_clusters_offset(void *co, uint64_t offset, int bytes) "co %p offset 0x%" PRIx64 " bytes %d"
tests/qemu-iotests/060 +1 -6
@@ -150,15 +150,10 @@ $QEMU_IO -c "$OPEN_RO" -c "read -P 1 0 512" | _filter_qemu_io echo echo "=== Testing overlap while COW is in flight ===" echo -BACKING_IMG=$TEST_IMG.base -TEST_IMG=$BACKING_IMG _make_test_img 1G - -$QEMU_IO -c 'write 0k 64k' "$BACKING_IMG" | _filter_qemu_io - # compat=0.10 is required in order to make the following discard actually # unallocate the sector rather than make it a zero sector - we want COW, after # all. -IMGOPTS='compat=0.10' _make_test_img -b "$BACKING_IMG" 1G +IMGOPTS='compat=0.10' _make_test_img 1G # Write two clusters, the second one enforces creation of an L2 table after # the first data cluster. $QEMU_IO -c 'write 0k 64k' -c 'write 512M 64k' "$TEST_IMG" | _filter_qemu_io
tests/qemu-iotests/060.out +1 -4
@@ -97,10 +97,7 @@ read 512/512 bytes at offset 0 === Testing overlap while COW is in flight === -Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1073741824 -wrote 65536/65536 bytes at offset 0 -64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 backing_file=TEST_DIR/t.IMGFMT.base +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 65536/65536 bytes at offset 536870912
Signature · 1 line
2.21.0
Re: [PATCH for-4.2 1/4] Revert "qcow2: skip writing zero buffers to empty COW areas"
Quoted 10 lines · Max Reitz
This reverts commit
c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.This commit causes fundamental performance problems on XFS (because
fallocate()stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
Quoted 6 lines
If we want to reintroduce something similar to
c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.Signed-off-by Max Reitz <[email protected]>
It's sad, but OK:
Quoted 223 lines
--- qapi/block-core.json | 4 +- block/qcow2.h | 6 --- block/qcow2-cluster.c | 2 +- block/qcow2.c | 86 -------------------------------------- block/trace-events | 1 - tests/qemu-iotests/060 | 7 +--- tests/qemu-iotests/060.out | 5 +-- 7 files changed, 4 insertions(+), 107 deletions(-)qapi/block-core.json +1
-3@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1) # # Since: 2.9@@ -3326,7 +3324,7 @@ 'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev', 'pwritev_zero', 'pwritev_done', 'empty_image_prepare', 'l1_shrink_write_table', 'l1_shrink_free_l2_clusters', - 'cor_write', 'cluster_alloc_space', 'none'] } + 'cor_write', 'none'] } ## # @BlkdebugIOType:block/qcow2.h +0
-6@@ -418,12 +418,6 @@ typedef struct QCowL2Meta*/ Qcow2COWRegion cow_end; -/* -* Indicates that COW regions are already handled and do not require -* any more processing. -*/ -bool skip_cow; -/** * The I/O vector with the data from the actual guest write request. * If non-NULL, this is meant to be merged together with the data block/qcow2-cluster.c +1
-1@@ -809,7 +809,7 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta *m)assert (start->nb_bytes + end->nb_bytes <=UINT_MAX - data_bytes);assert (start->offset + start->nb_bytes <= end->offset); -if ((start->nb_bytes ==0 && end->nb_bytes ==0 ) || m->skip_cow) { +if (start->nb_bytes ==0 && end->nb_bytes ==0 ) {return 0 ; }block/qcow2.c +0
-86@@ -2274,11 +2274,6 @@ static bool merge_cow(uint64_t offset, unsigned bytes,continue ; } -/* If COW regions are handled already, skip this too */ -if (m->skip_cow) { -continue ; - } -/* The data (middle) region must be immediately after the * start region */ if (l2meta_cow_start (m) + m->cow_start.nb_bytes != offset) {@@ -2305,81 +2300,6 @@ static bool merge_cow(uint64_t offset, unsigned bytes,return false; } -static bool is_unallocated (BlockDriverState *bs,int64_t offset,int64_t bytes) -{ -int64_t nr; -return !bytes || - (!bdrv_is_allocated_above (bs,NULL , false, offset, bytes, &nr) && - nr == bytes); -} - -static bool is_zero_cow (BlockDriverState *bs,QCowL2Meta *m) -{ -/* -* This check is designed for optimization shortcut so it must be -* efficient. -* Instead of is_zero(), use is_unallocated() as it is faster (but not -* as accurate and can result in false negatives). -*/ -return is_unallocated (bs, m->offset + m->cow_start.offset, - m->cow_start.nb_bytes) && -is_unallocated (bs, m->offset + m->cow_end.offset, - m->cow_end.nb_bytes); -} - -static int handle_alloc_space (BlockDriverState *bs,QCowL2Meta *l2meta) -{ -BDRVQcow2State *s = bs->opaque; -QCowL2Meta *m; - -if (!(s->data_file->bs->supported_zero_flags &BDRV_REQ_NO_FALLBACK )) { -return 0 ; - } - -if (bs->encrypted) { -return 0 ; - } - -for (m = l2meta; m !=NULL ; m = m->next) { -int ret; - -if (!m->cow_start.nb_bytes && !m->cow_end.nb_bytes) { -continue ; - } - -if (!is_zero_cow (bs, m)) { -continue ; - } - -/* -* instead of writing zero COW buffers, -* efficiently zero out the whole clusters -*/ - - ret =qcow2_pre_write_overlap_check (bs,0 , m->alloc_offset, - m->nb_clusters * s->cluster_size, - true); -if (ret <0 ) { -return ret; - } - -BLKDBG_EVENT (bs->file,BLKDBG_CLUSTER_ALLOC_SPACE ); - ret =bdrv_co_pwrite_zeroes (s->data_file, m->alloc_offset, - m->nb_clusters * s->cluster_size, -BDRV_REQ_NO_FALLBACK ); -if (ret <0 ) { -if (ret != -ENOTSUP && ret != -EAGAIN ) { -return ret; - } -continue ; - } - -trace_qcow2_skip_cow (qemu_coroutine_self (), m->offset, m->nb_clusters); - m->skip_cow = true; - } -return 0 ; -} -/* * qcow2_co_pwritev_task * Called with s->lock unlocked @@ -2421,12 +2341,6 @@ static coroutine_fn int qcow2_co_pwritev_task(BlockDriverState *bs, qiov_offset =0 ; } -/* Try to efficiently initialize the physical space with zeroes */ - ret =handle_alloc_space (bs, l2meta); -if (ret <0 ) { - goto out_unlocked; - } -/* * If we need to do COW, check if it's possible to merge the * writing of the guest data together with that of the COW regions. block/trace-events +0
-1@@ -72,7 +72,6 @@ qcow2_writev_done_part(void *co, int cur_bytes) "co %p cur_bytes %d" qcow2_writev_data(void *co, uint64_t offset) "co %p offset 0x%" PRIx64 qcow2_pwrite_zeroes_start_req(void *co, int64_t offset, int count) "co %p offset 0x%" PRIx64 " count %d" qcow2_pwrite_zeroes(void *co, int64_t offset, int count) "co %p offset 0x%" PRIx64 " count %d" -qcow2_skip_cow(void *co, uint64_t offset, int nb_clusters) "co %p offset 0x%" PRIx64 " nb_clusters %d" # qcow2-cluster.c qcow2_alloc_clusters_offset(void *co, uint64_t offset, int bytes) "co %p offset 0x%" PRIx64 " bytes %d"tests/qemu-iotests/060 +1
-6@@ -150,15 +150,10 @@ $QEMU_IO -c "$OPEN_RO" -c "read -P 1 0 512" | _filter_qemu_io echo echo "=== Testing overlap while COW is in flight ===" echo -BACKING_IMG=$TEST_IMG.base -TEST_IMG=$BACKING_IMG _make_test_img 1G - -$QEMU_IO -c 'write 0k 64k' "$BACKING_IMG" | _filter_qemu_io - # compat=0.10 is required in order to make the following discard actually # unallocate the sector rather than make it a zero sector - we want COW, after # all. -IMGOPTS='compat=0.10' _make_test_img -b "$BACKING_IMG" 1G +IMGOPTS='compat=0.10' _make_test_img 1G # Write two clusters, the second one enforces creation of an L2 table after # the first data cluster. $QEMU_IO -c 'write 0k 64k' -c 'write 512M 64k' "$TEST_IMG" | _filter_qemu_iotests/qemu-iotests/060.out +1
-4@@ -97,10 +97,7 @@ read 512/512 bytes at offset 0 === Testing overlap while COW is in flight === -Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1073741824 -wrote 65536/65536 bytes at offset 0 -64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 backing_file=TEST_DIR/t.IMGFMT.base +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 65536/65536 bytes at offset 536870912
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 1/4] Revert "qcow2: skip writing zero buffers to empty COW areas"
In reply to Max ReitzQuoted 20 lines · Max Reitz
This reverts commit
c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.This commit causes fundamental performance problems on XFS (because
fallocate()stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
If we want to reintroduce something similar to
c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.Signed-off-by Max Reitz <[email protected]>---
+++
b/qapi/block-core.json
qapi/block-core.json line 3307
Quoted 6 lines
qapi/block-core.json +0
-2@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1)
Deleting released qapi is not backwards-compatible. However, given that the only known user of this interface is debug testing via iotests, I'm not too concerned that we would be impacting any external users.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH for-4.2 1/4] Revert "qcow2: skip writing zero buffers to empty COW areas"
Quoted 19 lines · Eric Blake
Quoted 17 lines · Max Reitz
This reverts commit
c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.This commit causes fundamental performance problems on XFS (because
fallocate()stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
If we want to reintroduce something similar to
c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.Signed-off-by Max Reitz <[email protected]>---
Quoted 11 lines
qapi/block-core.json line 3308
Quoted 9 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +0
-2@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1)Deleting released qapi is not backwards-compatible.
Right. :-/
I’ll just not make changes to the QAPI schema. It doesn’t hurt to leave this in.
Max
Quoted 3 lines
However, given that
the only known user of this interface is debug testing via iotests, I'm not too concerned that we would be impacting any external users.
Re: [PATCH for-4.2 1/4] Revert "qcow2: skip writing zero buffers to empty COW areas"
Am 01.11.2019 um 15:01 hat Max Reitz geschrieben:
Quoted 32 lines · Eric Blake
Quoted 19 lines · Max Reitz
Quoted 17 lines
This reverts commit
c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.This commit causes fundamental performance problems on XFS (because
fallocate()stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
If we want to reintroduce something similar to
c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.Signed-off-by Max Reitz <[email protected]>---
Quoted 11 lines
qapi/block-core.json line 3308
Quoted 9 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +0
-2@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1)Deleting released qapi is not backwards-compatible.
Quoted 4 lines
Right. :-/
I’ll just not make changes to the QAPI schema. It doesn’t hurt to leave this in.
Why would it be incompatible to drop an enum value that is only ever used in output and that QEMU doesn't generate?
Kevin
Re: [PATCH for-4.2 1/4] Revert "qcow2: skip writing zero buffers to empty COW areas"
Quoted 33 lines · Kevin Wolf
Am 01.11.2019 um 15:01 hat Max Reitz geschrieben:
Quoted 32 lines · Eric Blake
Quoted 30 lines · Max Reitz
Quoted 20 lines
This reverts commit
c8bb23cbdbe32f5c326365e0a82e1b0e68cdcd8a.This commit causes fundamental performance problems on XFS (because
fallocate()stalls the AIO pipeline), and as such it is not clear that we should unconditionally enable this behavior.We expect subclusters to alleviate the performance penalty of small writes to newly allocated clusters, so when we get them, the originally intended performance gain may actually no longer be significant.
If we want to reintroduce something similar to
c8bb23cbdbe, it will require extensive benchmarking on various systems with subclusters enabled.Signed-off-by Max Reitz <[email protected]>---
+++
b/qapi/block-core.jsonqapi/block-core.json line 3308
Quoted 7 lines
qapi/block-core.json +0
-2@@ -3304,8 +3304,6 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # -# @cluster_alloc_space: an allocation of file space for a cluster (since 4.1) -# # @none: triggers once at creation of the blkdebug node (since 4.1)Deleting released qapi is not backwards-compatible.
Quoted 7 lines
Quoted 4 lines
Right. :-/
I’ll just not make changes to the QAPI schema. It doesn’t hurt to leave this in.
Why would it be incompatible to drop an enum value that is only ever used in output and that QEMU doesn't generate?
This isn’t output at all, it’s input for blockdev-add for blkdebug nodes.
Max
PATCH 2/4 block: Make wait/mark serialising requests public
In reply to Max ReitzMake both bdrv_mark_request_serialising() and bdrv_wait_serialising_requests() public so they can be used from block drivers.
--- include/block/block_int.h | 3 +++ block/io.c | 24 ++++++++++++------------ 2 files changed, 15 insertions(+), 12 deletions(-)
include/block/block_int.h +3 -0
@@ -999,6 +999,9 @@ extern unsigned int bdrv_drain_all_count;void bdrv_apply_subtree_drain (BdrvChild *child,BlockDriverState *new_parent);void bdrv_unapply_subtree_drain (BdrvChild *child,BlockDriverState *old_parent); +bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self); +void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align); +int get_tmp_filename (char *filename,int size);BlockDriver *bdrv_probe_all (const uint8_t *buf,int buf_size,const char *filename);
block/io.c +12 -12
@@ -715,7 +715,7 @@ static void tracked_request_begin(BdrvTrackedRequest *req,qemu_co_mutex_unlock (&bs->reqs_lock); } -static void mark_request_serialising (BdrvTrackedRequest *req,uint64_t align) +void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align) {int64_t overlap_offset = req->offset & ~(align -1 );uint64_t overlap_bytes =ROUND_UP (req->offset + req->bytes, align)
@@ -805,7 +805,7 @@ void bdrv_dec_in_flight(BlockDriverState *bs)bdrv_wakeup (bs); } -static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self) +bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self) {BlockDriverState *bs = self->bs;BdrvTrackedRequest *req;
@@ -1437,14 +1437,14 @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child,* with each other for the same cluster. For example, in copy-on-read * it ensures that the CoR read and write operations are atomic and * guest writes cannot interleave between them. */ -mark_request_serialising (req,bdrv_get_cluster_size (bs)); +bdrv_mark_request_serialising (req,bdrv_get_cluster_size (bs)); }/* BDRV_REQ_SERIALISING is only for write operation */ assert (!(flags &BDRV_REQ_SERIALISING ));if (!(flags &BDRV_REQ_NO_SERIALISING )) { -wait_serialising_requests (req); +bdrv_wait_serialising_requests (req); }if (flags &BDRV_REQ_COPY_ON_READ ) {
@@ -1841,10 +1841,10 @@ bdrv_co_write_req_prepare(BdrvChild *child, int64_t offset, uint64_t bytes,assert (!(flags & ~BDRV_REQ_MASK ));if (flags &BDRV_REQ_SERIALISING ) { -mark_request_serialising (req,bdrv_get_cluster_size (bs)); +bdrv_mark_request_serialising (req,bdrv_get_cluster_size (bs)); } - waited =wait_serialising_requests (req); + waited =bdrv_wait_serialising_requests (req);assert (!waited || !req->serialising ||is_request_serialising_and_aligned (req));
@@ -2008,8 +2008,8 @@ static int coroutine_fn bdrv_co_do_zero_pwritev(BdrvChild *child, padding =bdrv_init_padding (bs, offset, bytes, &pad);if (padding) { -mark_request_serialising (req, align); -wait_serialising_requests (req); +bdrv_mark_request_serialising (req, align); +bdrv_wait_serialising_requests (req);bdrv_padding_rmw_read (child, req, &pad, true);
@@ -2111,8 +2111,8 @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child, }if (bdrv_pad_request (bs, &qiov, &qiov_offset, &offset, &bytes, &pad)) { -mark_request_serialising (&req, align); -wait_serialising_requests (&req); +bdrv_mark_request_serialising (&req, align); +bdrv_wait_serialising_requests (&req);bdrv_padding_rmw_read (child, &req, &pad, false); }
@@ -3205,7 +3205,7 @@ static int coroutine_fn bdrv_co_copy_range_internal(/* BDRV_REQ_SERIALISING is only for write operation */ assert (!(read_flags &BDRV_REQ_SERIALISING ));if (!(read_flags &BDRV_REQ_NO_SERIALISING )) { -wait_serialising_requests (&req); +bdrv_wait_serialising_requests (&req); } ret = src->bs->drv->bdrv_co_copy_range_from(src->bs,
@@ -3336,7 +3336,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,* new area, we need to make sure that no write requests are made to it * concurrently or they might be overwritten by preallocation. */ if (new_bytes) { -mark_request_serialising (&req,1 ); +bdrv_mark_request_serialising (&req,1 ); }if (bs->read_only) {error_setg (errp,"Image is read-only" );
Signature · 1 line
2.21.0
PATCH 3/4 block: Add bdrv_co_get_self_request()
In reply to Max Reitz--- include/block/block_int.h | 1 + block/io.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
include/block/block_int.h +1 -0
@@ -1001,6 +1001,7 @@ void bdrv_unapply_subtree_drain(BdrvChild *child, BlockDriverState *old_parent); bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self);void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align); +BdrvTrackedRequest *bdrv_co_get_self_request (BlockDriverState *bs);int get_tmp_filename (char *filename,int size); BlockDriver *bdrv_probe_all(const uint8_t *buf, int buf_size,
block/io.c +18 -0
@@ -742,6 +742,24 @@ static bool is_request_serialising_and_aligned(BdrvTrackedRequest *req) (req->bytes == req->overlap_bytes); } +/** +* Return the tracked request on @bs for the current coroutine, or +* NULL if there is none. +*/ +BdrvTrackedRequest *bdrv_co_get_self_request (BlockDriverState *bs) +{ +BdrvTrackedRequest *req; +Coroutine *self =qemu_coroutine_self (); + + QLIST_FOREACH(req, &bs->tracked_requests, list) { +if (req->co == self) { +return req; + } + } + +return NULL ; +} +/** * Round a region to cluster boundaries */
Signature · 1 line
2.21.0
PATCH 4/4 block/file-posix: Let post-EOF fallocate serialize
In reply to Max ReitzThe XFS kernel driver has a bug that may cause data loss when using fallocate() in an I/O path, i.e. writing zeroes. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX in practice).
--- block/file-posix.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
block/file-posix.c +38 -0
@@ -2721,6 +2721,44 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes,RawPosixAIOData acb;ThreadPoolFunc *handler; +#ifdef CONFIG_FALLOCATE +if (offset + bytes > bs->total_sectors *BDRV_SECTOR_SIZE ) { +BdrvTrackedRequest *req; +uint64_t end; + +/* +* This is a workaround for a bug in the Linux XFS driver, +* where writes submitted through the AIO interface will be +* discarded if they happen beyond a concurrently running +* fallocate() that increases the file length (i.e., both the +* write and the fallocate() happen beyond the EOF). +* +* To work around it, we extend the tracked request for this +* zero write until INT64_MAX (effectively infinity), and mark +* it as serializing. +* +* We have to enable this workaround for all filesystems and +* AIO modes (not just XFS with aio=native), because for +* remote filesystems we do not know the host configuration. +* However, this should have no effect as long as no block +* driver submits zero writes beyond the EOF. +*/ + + req =bdrv_co_get_self_request (bs); +assert (req); +assert (req->type ==BDRV_TRACKED_WRITE ); +assert (req->offset <= offset); +assert (req->offset + req->bytes >= offset + bytes); + + end =INT64_MAX & -(uint64_t )bs->bl.request_alignment; + req->bytes = end - req->offset; + req->overlap_bytes = req->bytes; + +bdrv_mark_request_serialising (req, bs->bl.request_alignment); +bdrv_wait_serialising_requests (req); + } +#endif + acb = (RawPosixAIOData ) { .bs = bs, .aio_fildes = s->fd,
Signature · 1 line
2.21.0
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.
Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
$ qemu-img bench -t none -n -w -S 65536 test.qcow2
on XFS takes like 6 seconds on master, and like 50 to 80 seconds with c8bb23cbdbe reverted. So now on to guest tests...
(Well, and the question is still where the performance regression on ppc64 comes from.)
Max
Quoted 43 lines
Thus, I believe we should revert the commit for now (and most importantly for 4.1.1). We can think about reintroducing it for 5.0, but that would require more extensive benchmarks on a variety of systems, and we must see how subclusters change the picture.
I would have liked to do benchmarks myself before making this decision, but as far as I’m informed, patches for 4.1.1 are to be collected on Monday, so we need to be quick.
git-backport-diff against the RFC:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively001/4:[down] 'Revert "qcow2: skip writing zero buffers to empty COW areas"' 002/4:[----] [-C] 'block: Make wait/mark serialising requests public' 003/4:[down] 'block: Add
bdrv_co_get_self_request()' 004/4:[0036] [FC] 'block/file-posix: Let post-EOF fallocate serialize'Max Reitz (4):
Revert "qcow2: skip writing zero buffers to empty COW areas"
block: Make wait/mark serialising requests public
block: Addbdrv_co_get_self_request()block/file-posix: Let post-EOF fallocate serializeqapi/block-core.json | 4 +- block/qcow2.h | 6 --- include/block/block_int.h | 4 ++ block/file-posix.c | 38 +++++++++++++++++ block/io.c | 42 +++++++++++++------ block/qcow2-cluster.c | 2 +- block/qcow2.c | 86 -------------------------------------- block/trace-events | 1 - tests/qemu-iotests/060 | 7 +--- tests/qemu-iotests/060.out | 5 +-- 10 files changed, 76 insertions(+), 119 deletions(-)
Quoted 37 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
Quoted 6 lines
The obvious
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...
Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
Quoted 51 lines
(Well, and the question is still where the performance regression on ppc64 comes from.)
Max
Quoted 43 lines
Thus, I believe we should revert the commit for now (and most importantly for 4.1.1). We can think about reintroducing it for 5.0, but that would require more extensive benchmarks on a variety of systems, and we must see how subclusters change the picture.
I would have liked to do benchmarks myself before making this decision, but as far as I’m informed, patches for 4.1.1 are to be collected on Monday, so we need to be quick.
git-backport-diff against the RFC:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively001/4:[down] 'Revert "qcow2: skip writing zero buffers to empty COW areas"' 002/4:[----] [-C] 'block: Make wait/mark serialising requests public' 003/4:[down] 'block: Add
bdrv_co_get_self_request()' 004/4:[0036] [FC] 'block/file-posix: Let post-EOF fallocate serialize'Max Reitz (4):
Revert "qcow2: skip writing zero buffers to empty COW areas"
block: Make wait/mark serialising requests public
block: Addbdrv_co_get_self_request()block/file-posix: Let post-EOF fallocate serializeqapi/block-core.json | 4 +- block/qcow2.h | 6 --- include/block/block_int.h | 4 ++ block/file-posix.c | 38 +++++++++++++++++ block/io.c | 42 +++++++++++++------ block/qcow2-cluster.c | 2 +- block/qcow2.c | 86 -------------------------------------- block/trace-events | 1 - tests/qemu-iotests/060 | 7 +--- tests/qemu-iotests/060.out | 5 +-- 10 files changed, 76 insertions(+), 119 deletions(-)
Signature · 2 lines
Best regards, Vladimir
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
(Reverting c8bb23cbdbe makes it like 1 to 2 % faster.)
Max
Quoted 49 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
Quoted 7 lines
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Quoted 5 lines
(Reverting
c8bb23cbdbemakes it like 1 to 2 % faster.)Max
Signature · 2 lines
Best regards, Vladimir
Quoted 58 lines · Vladimir Sementsov-Ogievskiy
Quoted 49 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
Quoted 7 lines
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
(The thing is that Laurent and our QE has seen significant real-world performance regression on pcc64.)
Max
Quoted 59 lines · Max Reitz
Quoted 58 lines · Vladimir Sementsov-Ogievskiy
Quoted 56 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 7 lines
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
I’ve changed the above fio invocation to use --rw=randwrite and added --fallocate=none. The performance went down, but it went down both with and without c8bb23cbdbe.
So on my XFS system (XFS on luks on SSD), I see:
- with c8bb23cbdbe: 26.0 - 27.9 MB/s
- without c8bb23cbdbe: 25.6 - 27 MB/s
On my ext4 system (native on SSD), I see:
- with: 39.4 - 41.5 MB/s
- without: 39.4 - 42.0 MB/s
So basically no difference for XFS, and really no difference for ext4. (I ran these tests with 2 MB clusters.)
Max
Quoted 76 lines · Max Reitz
Quoted 59 lines · Max Reitz
Quoted 58 lines · Vladimir Sementsov-Ogievskiy
Quoted 56 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 7 lines
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
I’ve changed the above fio invocation to use --rw=randwrite and added --fallocate=none. The performance went down, but it went down both with and without
c8bb23cbdbe.So on my XFS system (XFS on luks on SSD), I see:
- withc8bb23cbdbe: 26.0 - 27.9 MB/s
- withoutc8bb23cbdbe: 25.6 - 27 MB/s
Quoted 7 lines
On my ext4 system (native on SSD), I see:
- with: 39.4 - 41.5 MB/s
- without: 39.4 - 42.0 MB/sSo basically no difference for XFS, and really no difference for ext4. (I ran these tests with 2 MB clusters.)
Hmm. I don't know. For me it seems obvious that zeroing 2M cluster is slow, and this is proved by simple tests with qemu-img bench, that fallocate is faster than zeroing most of the cluster.
So, if some guest test doesn't show the difference, this means that "small write into new cluster" is effectively rare case in this test.. And this doesn't prove that it's always rare and insignificant.
I don't sure that we have a real-world example that proves necessity of this optimization, or was there some original bug about low-performance which was fixed by this optimization.. Den, Anton, do we have something about it?
Signature · 2 lines
Best regards, Vladimir
Quoted 82 lines · Vladimir Sementsov-Ogievskiy
Quoted 70 lines · Max Reitz
Quoted 55 lines · Max Reitz
Quoted 54 lines · Vladimir Sementsov-Ogievskiy
Quoted 53 lines · Max Reitz
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Quoted 36 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
Quoted 8 lines
Quoted 6 lines
The obvious
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 6 lines
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
I’ve changed the above fio invocation to use --rw=randwrite and added --fallocate=none. The performance went down, but it went down both with and without
c8bb23cbdbe.So on my XFS system (XFS on luks on SSD), I see:
- withc8bb23cbdbe: 26.0 - 27.9 MB/s
- withoutc8bb23cbdbe: 25.6 - 27 MB/sQuoted 7 lines
On my ext4 system (native on SSD), I see:
- with: 39.4 - 41.5 MB/s
- without: 39.4 - 42.0 MB/sSo basically no difference for XFS, and really no difference for ext4. (I ran these tests with 2 MB clusters.)
Hmm. I don't know. For me it seems obvious that zeroing 2M cluster is slow, and this is proved by simple tests with qemu-img bench, that fallocate is faster than zeroing most of the cluster.
Quoted 8 lines
So, if some guest test doesn't show the difference, this means that "small write into new cluster" is effectively rare case in this test.. And this doesn't prove that it's always rare and insignificant.
I don't sure that we have a real-world example that proves necessity of this optimization, or was there some original bug about low-performance which was fixed by this optimization.. Den, Anton, do we have something about it?
sorry, I have missed the beginning of the thread.
Which driver is used for virtual disk - cached or non-cached IO is used in QEMU? We use non-cached by default and this could make a difference significantly.
Max,
can you pls share your domain.xml of the guest config and fio file for guest. I will recheck to be 120% sure.
Thank you in advance,
Den
Quoted 91 lines · Denis Lunev
Quoted 82 lines · Vladimir Sementsov-Ogievskiy
Quoted 70 lines · Max Reitz
Quoted 55 lines · Max Reitz
Quoted 54 lines · Vladimir Sementsov-Ogievskiy
Quoted 53 lines · Max Reitz
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Quoted 36 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
Quoted 8 lines
Quoted 6 lines
The obvious
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 6 lines
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
I’ve changed the above fio invocation to use --rw=randwrite and added --fallocate=none. The performance went down, but it went down both with and without
c8bb23cbdbe.So on my XFS system (XFS on luks on SSD), I see:
- withc8bb23cbdbe: 26.0 - 27.9 MB/s
- withoutc8bb23cbdbe: 25.6 - 27 MB/sQuoted 7 lines
On my ext4 system (native on SSD), I see:
- with: 39.4 - 41.5 MB/s
- without: 39.4 - 42.0 MB/sSo basically no difference for XFS, and really no difference for ext4. (I ran these tests with 2 MB clusters.)
Hmm. I don't know. For me it seems obvious that zeroing 2M cluster is slow, and this is proved by simple tests with qemu-img bench, that fallocate is faster than zeroing most of the cluster.
Quoted 8 lines
So, if some guest test doesn't show the difference, this means that "small write into new cluster" is effectively rare case in this test.. And this doesn't prove that it's always rare and insignificant.
I don't sure that we have a real-world example that proves necessity of this optimization, or was there some original bug about low-performance which was fixed by this optimization.. Den, Anton, do we have something about it?
Quoted 5 lines
sorry, I have missed the beginning of the thread.
Which driver is used for virtual disk - cached or non-cached IO is used in QEMU? We use non-cached by default and this could make a difference significantly.
I’m using no cache, the above tests were done with aio=native; I’ve sent another response with aio=threads numbers.
Quoted 4 lines
Max,
can you pls share your domain.xml of the guest config and fio file for guest. I will recheck to be 120% sure.
I’m running qemu directly as follows:
x86_64-softmmu/qemu-system-x86_64 \
-serial stdio \
-cdrom ~/tmp/arch.iso \
-m 4096 \
-enable-kvm \
-drive \
if=none,id=t,format=qcow2,file=test/test.qcow2,cache=none,aio=native \
-device virtio-scsi \
-device scsi-hd,drive=t \
-net user \
-net nic,model=rtl8139 \
-smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \
-cpu SandyBridge \
-nodefaults \
-nographicThe full FIO command line is:
fio --rw=randwrite --bs=4k --iodepth=8 --runtime=1m --direct=1 \
--filename=/mnt/foo --name=job1 --ioengine=libaio --thread \
--group_reporting --numjobs=16 --size=2G --time_based \
--output=/tmp/fio_result --fallocate=noneMax
Quoted 76 lines · Max Reitz
Quoted 59 lines · Max Reitz
Quoted 58 lines · Vladimir Sementsov-Ogievskiy
Quoted 56 lines · Max Reitz
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 39 lines · Max Reitz
Quoted 24 lines · Max Reitz
Hi,
This series builds on the previous RFC. The workaround is now applied unconditionally of AIO mode and filesystem because we don’t know those things for remote filesystems. Furthermore,
bdrv_co_get_self_request()has been moved toblock/io.c.Applying the workaround unconditionally is fine from a performance standpoint, because it should actually be dead code, thanks to patch 1 (the elephant in the room). As far as I know, there is no other block driver but qcow2 in
handle_alloc_space()that would submit zero writes as part of normal I/O so it can occur concurrently to other write requests. It still makes sense to take the workaround for file-posix because we can’t really prevent that any other block driver will submit zero writes as part of normal I/O in the future.Anyway, let’s get to the elephant.
From input by XFS developers
(https://bugzilla.redhat.com/show_bug.cgi?id=1765547#c7) it seems clear thatc8bb23cbdbecauses fundamental performance problems on XFS with aio=native that cannot be fixed. In other cases,c8bb23cbdbeimproves performance or we wouldn’t have it.Quoted 6 lines
In general, avoiding performance regressions is more important than improving performance, unless the regressions are just a minor corner case or insignificant when compared to the improvement. The XFS regression is no minor corner case, and it isn’t insignificant. Laurent Vivier has found performance to decrease by as much as 88 % (on ppc64le, fio in a guest with 4k blocks, iodepth=8: 1662 kB/s from 13.9 MB/s).
Ah, crap.
I wanted to send this series as early today as possible to get as much feedback as possible, so I’ve only started doing benchmarks now.
The obvious
Quoted 7 lines
Quoted 4 lines
$ qemu-img bench -t none -n -w -S 65536 test.qcow2on XFS takes like 6 seconds on master, and like 50 to 80 seconds with
c8bb23cbdbereverted. So now on to guest tests...Aha, that's very interesting) What about aio-native which should be slowed down? Could it be tested like this?
That is aio=native (-n).
But so far I don’t see any significant difference in guest tests (i.e., fio --rw=write --bs=4k --iodepth=8 --runtime=1m --direct=1 --ioengine=libaio --thread --numjobs=16 --size=2G --time_based), neither with 64 kB nor with 2 MB clusters. (But only on XFS, I’ll have to see about ext4 still.)
Quoted 7 lines
Quoted 3 lines
hmm, this possibly mostly tests writes to already allocated clusters. Has fio an option to behave like qemu-img bench with -S 65536, i.e. write once into each cluster?
Maybe, but is that a realistic depiction of whether this change is worth it? That is why I’m doing the guest test, to see whether it actually has much impact on the guest.
I’ve changed the above fio invocation to use --rw=randwrite and added --fallocate=none. The performance went down, but it went down both with and without
c8bb23cbdbe.So on my XFS system (XFS on luks on SSD), I see:
- withc8bb23cbdbe: 26.0 - 27.9 MB/s
- withoutc8bb23cbdbe: 25.6 - 27 MB/s
Quoted 6 lines
On my ext4 system (native on SSD), I see:
- with: 39.4 - 41.5 MB/s
- without: 39.4 - 42.0 MB/sSo basically no difference for XFS, and really no difference for ext4. (I ran these tests with 2 MB clusters.)
For 64 kB clusters, I have on XFS:
- with: 30.3 - 31.3 MB/s
- without: 27.4 - 27.9 MB/s
On ext4:
- with: 34.9 - 36.4 MB/s
- without: 33.8 - 38 MB/s
For good measure, 2 MB clusters with aio=threads (but still O_DIRECT) on XFS:
- with: 25.7 MB/s - 27.0 MB/s
- without: 24.6 MB/s - 26.7 MB/s
On ext4:
- with: 16.7 MB/s - 19.3 MB/s
- without: 16.3 MB/s - 18.4 MB/s
(Note that the difference between ext4 and XFS is probably just because these are two different systems with different SSDs.)
So there is little difference, if any significant at all (those were all just three test runs). There does seem to be a bit of a drift for aio=threads, but then again it’s also just much slower than aio=native for ext4.
(Also note that I ran this on an unfixed kernel. Maybe the XFS fix will slow things down, but I haven’t tested that yet.)
So unless there are realistic guest benchmarks for ext4 that say we should keep the patch, I’m going to queue the revert for now (“now” = 4.1.1 and 4.2.0).
Max
On 01.11.19 14:30, Max Reitz wrote:
[...]
Quoted 3 lines
So unless there are realistic guest benchmarks for ext4 that say we should keep the patch, I’m going to queue the revert for now (“now” = 4.1.1 and 4.2.0).
I found one case where the performance is significantly improved byc8bb23cbdbe: In the cases so far I had XFS in the guest, now I used ext4, and with aio=native (on the ext4 host with 2 MB clusters), the performance goes from 63.9 - 65.0 MB/s to 75.7 - 76.4 MB/s (so +18%).
The difference is smaller for 64 kB clusters, but still there at +13%. That’s probably the more important fact, because these are the default settings, and this is probably about what would happen with 2 MB clusters + subclusters.
(Patch 4 in this series doesn’t decrease the performance.)
This is a tough decision for me because from some people tell me “Let’s just revert it, there are problems with it and we don’t quite know what good it does in practice”, and others say “We have (not really practical) benchmarks that show it does something good for our specific case”. And all that while those two groups never seem to talk to each other directly.
So I suppose I’m going to have to make a decision. I now know a case where c8bb23cbdbe is actually beneficial. I myself have never seen c8bb23cbdbe decrease performance, but I know Laurent has seen a drastic performance degradation, and he’s used it to bisect the XFS problem to that commit, so it’s really real. But I haven’t seen it, and as far as I know it really only happens on ppc64.
In light of this, I would prefer to revert c8bb23cbdbe for 4.1.1, and keep it for 4.2.0. But I don’t know whether we can do that, all I know is that I’m not going to find out in time for 4.1.1.
If we keep c8bb23cbdbe in any way, we need patches 2 through 4, that much is clear.
I believe we can think about the performance problem after 4.2.0. I would love to benchmark c8bb23cbdbe on a fixed kernel, but there just isn’t time for that anymore.
I’m not a fan of keeping c8bb23cbdbe behind a configure switch. If it’s beneficial, it should be there for everyone.
OK. Some may see this as a wrong decision, but someone needs to make one now, so here goes: ext4 is the default Linux FS for most distributions. As far as I can tell from my own benchmarks, c8bb23cbdbe brings a significant performance improvement for qcow2 images with the default configuration on this default filesystem with aio=native and doesn’t change much in any other case.
What happens on ppc64 is a problem, but that’s a RHEL problem because it’s specific to XFS (and also ppc64). It also won’t be a regression on 4.2 compared to 4.1.
Dave’s argument was good that fallocate() and AIO cannot mix (at least on XFS), but I couldn’t see any impact of that in my benchmarks (maybe my benchmarks were just wrong).
So I think for upstream it’ll be best if I send a v2 which doesn’t touch handle_alloc_space(), and instead just consists of patches 2 through 4.
(And CC it all to stable.)
I think we still need to keep track of the XFS/ppc64 issue and do more benchmarks especially with the fixed XFS driver.
tl;dr:
The main arguments for reverting c8bb23cbdbe were (AFAIU):
- a general uneasy feeling about it
- theoretical arguments that it must be bad on XFS
- actual problems on ppc64/XFS
- “what good does it do in practice?”
- that subclusters would make it obsolete anyway
What I could see is:
- no impact on XFS in practice
- significant practical benefit on ext4
- subclusters probably wouldn’t make it obsolete, because I can still see a +13% improvement for 64 kB clusters (2 MB clusters + subclusters gives you 64 kB subclusters)
In addition, it needs to be considered that ext4 is the default FS for most Linux distributions.
As such, I personally am not convinced of reverting this patch. Let’s keep it, have patches 2 through 4 for 4.1.1 and 4.2.0, and think about what to do for ppc64/XFS later.
Max