qcow2: Fix data corruption on XFS
v2 cover qcow2: Fix data corruption on XFS
Hi,
As I reasoned here:
https://lists.nongnu.org/archive/html/qemu-block/2019-11/msg00027.html
I’m no longer convinced of reverting c8bb23cbdbe. I could see a significant performance improvement from it on ext4 with aio=native in a guest that does random 4k writes, and as such I feel it would be a regression to revert it for 4.2.
To work around the XFS corruption, we still need the other three patches from the series, of course. We cannot restrict the workaround to just XFS, because maybe the file is on a remote filesystem and then we don’t know anything about the host configuration.
The performance impact should still be minimal because this just serializes post-EOF write-zeroes and data writes, and that just doesn’t happen very often, even with handle_alloc_space() in qcow2.
I would love to have more time to make a decision, but there simply isn’t any. Patches for 4.1.1 are to be collected on Monday, AFAIU.
v2:
- Dropped patch 1
- Forgot a “coroutine_fn” in patch 2 (it isn’t really important, qemu_coroutine_self() works in non-coroutine functions, too, but calling bdrv_(co_)get_self_request() from a non-coroutine just doesn’t make any sense)
- Patch 3:
- Reverted the commit message to the RFC state to reflect that this is specifically because of handle_alloc_space()
- Dropped the two lines that said there’d be no performance impact at all because no driver would submit zero writes beyond the EOF (because qcow2 now still does)
git-backport-diff against v1:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/3:[----] [--] 'block: Make wait/mark serialising requests public' 002/3:[0004] [FC] 'block: Add bdrv_co_get_self_request()' 003/3:[0002] [FC] 'block/file-posix: Let post-EOF fallocate serialize'
Max Reitz (3):
block: Make wait/mark serialising requests public
block: Add bdrv_co_get_self_request() block/file-posix: Let post-EOF fallocate serialize
include/block/block_int.h | 4 ++++ block/file-posix.c | 36 +++++++++++++++++++++++++++++++++ block/io.c | 42 ++++++++++++++++++++++++++++----------- 3 files changed, 70 insertions(+), 12 deletions(-)
Signature · 1 line
2.21.0
PATCH v2 1/3 block: Make wait/mark serialising requests public
Make both bdrv_mark_request_serialising() and bdrv_wait_serialising_requests() public so they can be used from block drivers.
--- include/block/block_int.h | 3 +++ block/io.c | 24 ++++++++++++------------ 2 files changed, 15 insertions(+), 12 deletions(-)
include/block/block_int.h +3 -0
@@ -999,6 +999,9 @@ extern unsigned int bdrv_drain_all_count;void bdrv_apply_subtree_drain (BdrvChild *child,BlockDriverState *new_parent);void bdrv_unapply_subtree_drain (BdrvChild *child,BlockDriverState *old_parent); +bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self); +void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align); +int get_tmp_filename (char *filename,int size);BlockDriver *bdrv_probe_all (const uint8_t *buf,int buf_size,const char *filename);
block/io.c +12 -12
@@ -715,7 +715,7 @@ static void tracked_request_begin(BdrvTrackedRequest *req,qemu_co_mutex_unlock (&bs->reqs_lock); } -static void mark_request_serialising (BdrvTrackedRequest *req,uint64_t align) +void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align) {int64_t overlap_offset = req->offset & ~(align -1 );uint64_t overlap_bytes =ROUND_UP (req->offset + req->bytes, align)
@@ -805,7 +805,7 @@ void bdrv_dec_in_flight(BlockDriverState *bs)bdrv_wakeup (bs); } -static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self) +bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self) {BlockDriverState *bs = self->bs;BdrvTrackedRequest *req;
@@ -1437,14 +1437,14 @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child,* with each other for the same cluster. For example, in copy-on-read * it ensures that the CoR read and write operations are atomic and * guest writes cannot interleave between them. */ -mark_request_serialising (req,bdrv_get_cluster_size (bs)); +bdrv_mark_request_serialising (req,bdrv_get_cluster_size (bs)); }/* BDRV_REQ_SERIALISING is only for write operation */ assert (!(flags &BDRV_REQ_SERIALISING ));if (!(flags &BDRV_REQ_NO_SERIALISING )) { -wait_serialising_requests (req); +bdrv_wait_serialising_requests (req); }if (flags &BDRV_REQ_COPY_ON_READ ) {
@@ -1841,10 +1841,10 @@ bdrv_co_write_req_prepare(BdrvChild *child, int64_t offset, uint64_t bytes,assert (!(flags & ~BDRV_REQ_MASK ));if (flags &BDRV_REQ_SERIALISING ) { -mark_request_serialising (req,bdrv_get_cluster_size (bs)); +bdrv_mark_request_serialising (req,bdrv_get_cluster_size (bs)); } - waited =wait_serialising_requests (req); + waited =bdrv_wait_serialising_requests (req);assert (!waited || !req->serialising ||is_request_serialising_and_aligned (req));
@@ -2008,8 +2008,8 @@ static int coroutine_fn bdrv_co_do_zero_pwritev(BdrvChild *child, padding =bdrv_init_padding (bs, offset, bytes, &pad);if (padding) { -mark_request_serialising (req, align); -wait_serialising_requests (req); +bdrv_mark_request_serialising (req, align); +bdrv_wait_serialising_requests (req);bdrv_padding_rmw_read (child, req, &pad, true);
@@ -2111,8 +2111,8 @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child, }if (bdrv_pad_request (bs, &qiov, &qiov_offset, &offset, &bytes, &pad)) { -mark_request_serialising (&req, align); -wait_serialising_requests (&req); +bdrv_mark_request_serialising (&req, align); +bdrv_wait_serialising_requests (&req);bdrv_padding_rmw_read (child, &req, &pad, false); }
@@ -3205,7 +3205,7 @@ static int coroutine_fn bdrv_co_copy_range_internal(/* BDRV_REQ_SERIALISING is only for write operation */ assert (!(read_flags &BDRV_REQ_SERIALISING ));if (!(read_flags &BDRV_REQ_NO_SERIALISING )) { -wait_serialising_requests (&req); +bdrv_wait_serialising_requests (&req); } ret = src->bs->drv->bdrv_co_copy_range_from(src->bs,
@@ -3336,7 +3336,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,* new area, we need to make sure that no write requests are made to it * concurrently or they might be overwritten by preallocation. */ if (new_bytes) { -mark_request_serialising (&req,1 ); +bdrv_mark_request_serialising (&req,1 ); }if (bs->read_only) {error_setg (errp,"Image is read-only" );
Signature · 1 line
2.21.0
PATCH v2 2/3 block: Add bdrv_co_get_self_request()
In reply to Max Reitz--- include/block/block_int.h | 1 + block/io.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
include/block/block_int.h +1 -0
@@ -1001,6 +1001,7 @@ void bdrv_unapply_subtree_drain(BdrvChild *child, BlockDriverState *old_parent); bool coroutine_fn bdrv_wait_serialising_requests(BdrvTrackedRequest *self);void bdrv_mark_request_serialising (BdrvTrackedRequest *req,uint64_t align); +BdrvTrackedRequest *coroutine_fn bdrv_co_get_self_request(BlockDriverState *bs);int get_tmp_filename (char *filename,int size); BlockDriver *bdrv_probe_all(const uint8_t *buf, int buf_size,
block/io.c +18 -0
@@ -742,6 +742,24 @@ static bool is_request_serialising_and_aligned(BdrvTrackedRequest *req) (req->bytes == req->overlap_bytes); } +/** +* Return the tracked request on @bs for the current coroutine, or +* NULL if there is none. +*/ +BdrvTrackedRequest *coroutine_fn bdrv_co_get_self_request(BlockDriverState *bs) +{ +BdrvTrackedRequest *req; +Coroutine *self =qemu_coroutine_self (); + + QLIST_FOREACH(req, &bs->tracked_requests, list) { +if (req->co == self) { +return req; + } + } + +return NULL ; +} +/** * Round a region to cluster boundaries */
Signature · 1 line
2.21.0
PATCH v2 3/3 block/file-posix: Let post-EOF fallocate serialize
In reply to Max ReitzThe XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX in practice).
--- block/file-posix.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
block/file-posix.c +36 -0
@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes,RawPosixAIOData acb;ThreadPoolFunc *handler; +#ifdef CONFIG_FALLOCATE +if (offset + bytes > bs->total_sectors *BDRV_SECTOR_SIZE ) { +BdrvTrackedRequest *req; +uint64_t end; + +/* +* This is a workaround for a bug in the Linux XFS driver, +* where writes submitted through the AIO interface will be +* discarded if they happen beyond a concurrently running +* fallocate() that increases the file length (i.e., both the +* write and the fallocate() happen beyond the EOF). +* +* To work around it, we extend the tracked request for this +* zero write until INT64_MAX (effectively infinity), and mark +* it as serializing. +* +* We have to enable this workaround for all filesystems and +* AIO modes (not just XFS with aio=native), because for +* remote filesystems we do not know the host configuration. +*/ + + req =bdrv_co_get_self_request (bs); +assert (req); +assert (req->type ==BDRV_TRACKED_WRITE ); +assert (req->offset <= offset); +assert (req->offset + req->bytes >= offset + bytes); + + end =INT64_MAX & -(uint64_t )bs->bl.request_alignment; + req->bytes = end - req->offset; + req->overlap_bytes = req->bytes; + +bdrv_mark_request_serialising (req, bs->bl.request_alignment); +bdrv_wait_serialising_requests (req); + } +#endif + acb = (RawPosixAIOData ) { .bs = bs, .aio_fildes = s->fd,
Signature · 1 line
2.21.0
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit
c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAXin practice).
This has been fixed in the kernel a while ago. I don't think it makes sense to work around it in qemu.
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit
c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAXin practice).This has been fixed in the kernel a while ago. I don't think it makes sense to work around it in qemu.
Has it? It was my understanding that this is fixed by
https://git.kernel.org/pub/scm/fs/xfs/xfs-linux.git/commit/?h=for-next&id=249bd9087a5264d2b8a974081870e2e27671b4dcwhich
has been merged only very recently and is on track to be part of Linux 5.5, as far as I understand.
Max
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 7 lines · Max Reitz
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit
c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAXin practice).
Quoted 5 lines
Quoted 2 lines
This has been fixed in the kernel a while ago. I don't think it makes sense to work around it in qemu.
Has it? It was my understanding that this is fixed by
https://git.kernel.org/pub/scm/fs/xfs/xfs-linux.git/commit/?h=for-next&id=249bd9087a5264d2b8a974081870e2e27671b4dcwhich
Sorry, broke the link. Let me try again:
Max
Quoted 5 lines
has been merged only very recently and is on track to be part of Linux 5.5, as far as I understand.
Max
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
In reply to Max Reitz01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Quoted 35 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit
c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAXin practice).Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)block/file-posix.c +16
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes,RawPosixAIOData acb;ThreadPoolFunc *handler; +#ifdef CONFIG_FALLOCATE +if (offset + bytes > bs->total_sectors *BDRV_SECTOR_SIZE ) { +BdrvTrackedRequest *req; +uint64_t end; + +/* +* This is a workaround for a bug in the Linux XFS driver, +* where writes submitted through the AIO interface will be +* discarded if they happen beyond a concurrently running +* fallocate() that increases the file length (i.e., both the +* write and the fallocate() happen beyond the EOF). +* +* To work around it, we extend the tracked request for this +* zero write until INT64_MAX (effectively infinity), and mark +* it as serializing. +*
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2749
Quoted 10 lines
block/file-posix.c +10
-0+* We have to enable this workaround for all filesystems and +* AIO modes (not just XFS with aio=native), because for +* remote filesystems we do not know the host configuration. +*/ + + req =bdrv_co_get_self_request (bs); +assert (req); +assert (req->type ==BDRV_TRACKED_WRITE ); +assert (req->offset <= offset); +assert (req->offset + req->bytes >= offset + bytes);
Why these assertions? TrackedRequest offset and bytes fields correspond to the original request. When request is being expanded to satisfy request_alignment, these fields are not updated. So, maybe, we should assert overlap_offset and overlap_bytes?
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2752
Quoted 3 lines
block/file-posix.c +3
-0+ + end =INT64_MAX & -(uint64_t )bs->bl.request_alignment; + req->bytes = end - req->offset;
And I doubt that we should update req->bytes. We never updated it in other places, it corresponds to original request. It's enough to update overlap_bytes to achieve corresponding serialising.
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2755
Quoted 3 lines
block/file-posix.c +3
-0+ req->overlap_bytes = req->bytes; + +bdrv_mark_request_serialising (req, bs->bl.request_alignment);
Not sure, how much should we care about request_alignment here, I think, it's enough to just set req->overlap_bytes = INT64_MAX - req->overlap_offest, but it doesn't really matter.
Quoted 8 lines
block/file-posix.c +4
-0+bdrv_wait_serialising_requests (req); + } +#endif + acb = (RawPosixAIOData ) { .bs = bs, .aio_fildes = s->fd,
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Uh, well. Reasonable, but I hope you don’t mind me having no longer having this patch fresh on my mind.
Quoted 39 lines
Quoted 39 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (
INT64_MAXin practice).Signed-off-by Max Reitz <[email protected]>---
block/file-posix.c| 36 ++++++++++++++++++++++++++++++++++++1 file changed, 36 insertions(+)block/file-posix.c +0
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, RawPosixAIOData acb; ThreadPoolFunc *handler;+#ifdef
CONFIG_FALLOCATE
+ if (offset + bytes >bs->total_sectors*BDRV_SECTOR_SIZE) { + BdrvTrackedRequest *req;
+uint64_tend;
+
+ /*
+ * This is a workaround for a bug in the Linux XFS driver, + * where writes submitted through the AIO interface will be + * discarded if they happen beyond a concurrently running + *fallocate()that increases the file length (i.e., both the + * write and thefallocate()happen beyond the EOF). + *
+ * To work around it, we extend the tracked request for this + * zero write untilINT64_MAX(effectively infinity), and mark + * it as serializing.
+ *
+ * We have to enable this workaround for all filesystems and + * AIO modes (not just XFS with aio=native), because for + * remote filesystems we do not know the host configuration. + */
Quoted 8 lines
Quoted 6 lines
+
+ req =bdrv_co_get_self_request(bs);
+ assert(req);
+assert(req->type == BDRV_TRACKED_WRITE); +assert(req->offset <= offset);
+assert(req->offset + req->bytes >= offset + bytes);Why these assertions?
Mostly to see that bdrv_co_get_self_request() (introduced by the same series) actually got the right request. (I suppose.)
Quoted 3 lines
TrackedRequest offset and bytes fields correspond
to the original request. When request is being expanded to satisfyrequest_alignment, these fields are not updated.
Well, shrunk in this case, but OK.
Quoted 1 line
So, maybe, we should assert
overlap_offsetandoverlap_bytes?
Maybe, but would that have any benefit? Especially after this patch having been in qemu for over half a year?
(Also, intuitively off the top of my head I don’t see how it would make more sense to check overlap_offset and overlap_bytes, if all the assertions are for is to see that we got the right request. overlap_offset and overlap_bytes may still not exactly match @offset or @bytes, respectively.)
Your suggestion makes it sound a bit like you have a different purpose in mind what these assertions might be useful for...?
Quoted 7 lines
Quoted 3 lines
+
+ end =INT64_MAX& -(uint64_t)bs->bl.request_alignment; +req->bytes= end -req->offset;And I doubt that we should update
req->bytes. We never updated it in other places, it corresponds to original request. It's enough to updateoverlap_bytesto achieve corresponding serialising.
Does it hurt? If so, would you send a patch?
I assume you reply to this patch instead of writing a patch because you have the same feeling of “It probably doesn’t really matter, so let’s have a discussion first”.
My stance is: I don’t think it matters and this whole piece of code is a hack that shouldn’t exist, obviously. So I don’t really care how it fits into all of our other code.
I would like to say I wouldn’t mind a patch to drop the req->bytes assignment, but OTOH it would mean I’d have to review it and verify that it’s indeed sufficient to set overlap_bytes.
If it’s in any way inconvenient for you that req->bytes is adjusted, then of course please send one.
Quoted 7 lines
Quoted 3 lines
+
req->overlap_bytes=req->bytes;
+
+bdrv_mark_request_serialising(req, bs->bl.request_alignment);Not sure, how much should we care about
request_alignmenthere, I think, it's enough to just setreq->overlap_bytes=INT64_MAX-req->overlap_offest, but it doesn't really matter.
As long as req->bytes is adjusted, we have to care, or the overlap_bytes calculation in bdrv_mark_request_serialising will overflow.
Well, one could argue that it doesn’t matter because the MAX() will still do the right thing, but overflowing is never nice.
(Of course, it probably doesn’t matter at all if we just wouldn’t touch req->bytes.)
Max
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 70 lines · Max Reitz
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Uh, well. Reasonable, but I hope you don’t mind me having no longer having this patch fresh on my mind.
Quoted 39 lines
Quoted 39 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (
INT64_MAXin practice).Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)block/file-posix.c +19
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, RawPosixAIOData acb; ThreadPoolFunc *handler; +#ifdef CONFIG_FALLOCATE + if (offset + bytes > bs->total_sectors * BDRV_SECTOR_SIZE) { + BdrvTrackedRequest *req; + uint64_t end; + + /* +* This is a workaround for a bug in the Linux XFS driver, +* where writes submitted through the AIO interface will be +* discarded if they happen beyond a concurrently running +* fallocate() that increases the file length (i.e., both the +* write and the fallocate() happen beyond the EOF). +* +* To work around it, we extend the tracked request for this +* zero write until INT64_MAX (effectively infinity), and mark +* it as serializing. +* +* We have to enable this workaround for all filesystems and +* AIO modes (not just XFS with aio=native), because for +* remote filesystems we do not know the host configuration. +*/ Quoted 8 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2749
Quoted 6 lines
block/file-posix.c +6
-0+ + req = bdrv_co_get_self_request(bs); + assert(req); + assert(req->type == BDRV_TRACKED_WRITE); + assert(req->offset <= offset); + assert(req->offset + req->bytes >= offset + bytes);Why these assertions?
Mostly to see that
bdrv_co_get_self_request()(introduced by the same series) actually got the right request. (I suppose.)Quoted 3 lines
TrackedRequest offset and bytes fields correspond
to the original request. When request is being expanded to satisfyrequest_alignment, these fields are not updated.Well, shrunk in this case, but OK.
Quoted 1 line
So, maybe, we should assert
overlap_offsetandoverlap_bytes?Maybe, but would that have any benefit? Especially after this patch having been in qemu for over half a year?
Quoted 8 lines
(Also, intuitively off the top of my head I don’t see how it would make more sense to check
overlap_offsetandoverlap_bytes, if all the assertions are for is to see that we got the right request.overlap_offsetandoverlap_bytesmay still not exactly match @offset or @bytes, respectively.)Your suggestion makes it sound a bit like you have a different purpose in mind what these assertions might be useful for...?
No I just think it may have false-positives, when actual request is larger than original. So offset may be < req->offset and req->offset + req->bytes may be less than offset + bytes. And we will crash. I should make a reproducer to prove it, but it seems possible.
Quoted 9 lines
Quoted 8 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2752
Quoted 4 lines
block/file-posix.c +3
-0+ + end = INT64_MAX & -(uint64_t)bs->bl.request_alignment; + req->bytes = end - req->offset;And I doubt that we should update
req->bytes. We never updated it in other places, it corresponds to original request. It's enough to updateoverlap_bytesto achieve corresponding serialising.
Quoted 5 lines
Does it hurt? If so, would you send a patch?
I assume you reply to this patch instead of writing a patch because you have the same feeling of “It probably doesn’t really matter, so let’s have a discussion first”.
1. yes, and
2. I probably don't see the full picture around tracked requests
Quoted 20 lines
My stance is: I don’t think it matters and this whole piece of code is a hack that shouldn’t exist, obviously. So I don’t really care how it fits into all of our other code.
I would like to say I wouldn’t mind a patch to drop the
req->bytesassignment, but OTOH it would mean I’d have to review it and verify that it’s indeed sufficient to setoverlap_bytes.If it’s in any way inconvenient for you that
req->bytesis adjusted, then of course please send one.Quoted 7 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2755
Quoted 3 lines
block/file-posix.c +3
-0+ req->overlap_bytes = req->bytes; + + bdrv_mark_request_serialising(req, bs->bl.request_alignment);Not sure, how much should we care about
request_alignmenthere, I think, it's enough to just setreq->overlap_bytes=INT64_MAX-req->overlap_offest, but it doesn't really matter.
Quoted 5 lines
As long as
req->bytesis adjusted, we have to care, or theoverlap_bytescalculation inbdrv_mark_request_serialisingwill overflow.Well, one could argue that it doesn’t matter because the
MAX()will still do the right thing, but overflowing is never nice.
Hmm I think, if reduce it to just INT64_MAX, we should pass 1 as align to bdrv_mark_request_serialising.
Quoted 4 lines
(Of course, it probably doesn’t matter at all if we just wouldn’t touch
req->bytes.)
OK, thanks for the answer, I'll prepare a patch.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 79 lines · Vladimir Sementsov-Ogievskiy
Quoted 78 lines · Max Reitz
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Uh, well. Reasonable, but I hope you don’t mind me having no longer having this patch fresh on my mind.
Quoted 41 lines
Quoted 41 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2
images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX
in practice).Signed-off-by Max Reitz <[email protected]>---
block/file-posix.c| 36 ++++++++++++++++++++++++++++++++++++1 file changed, 36 insertions(+)block/file-posix.c +0
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, RawPosixAIOData acb; ThreadPoolFunc *handler;+#ifdef
CONFIG_FALLOCATE
+ if (offset + bytes >bs->total_sectors*BDRV_SECTOR_SIZE) { + BdrvTrackedRequest *req;
+uint64_tend;
+
+ /*
+ * This is a workaround for a bug in the Linux XFS driver, + * where writes submitted through the AIO interface will be + * discarded if they happen beyond a concurrently running + *fallocate()that increases the file length (i.e., both the + * write and thefallocate()happen beyond the EOF). + *
+ * To work around it, we extend the tracked request for this + * zero write untilINT64_MAX(effectively infinity), and mark + * it as serializing.
+ *
+ * We have to enable this workaround for all filesystems and + * AIO modes (not just XFS with aio=native), because for + * remote filesystems we do not know the host configuration. + */Quoted 8 lines
Quoted 6 lines
+
+ req =bdrv_co_get_self_request(bs);
+ assert(req);
+assert(req->type == BDRV_TRACKED_WRITE); +assert(req->offset <= offset);
+assert(req->offset + req->bytes >= offset + bytes);Why these assertions?
Mostly to see that
bdrv_co_get_self_request()(introduced by the same series) actually got the right request. (I suppose.)Quoted 3 lines
TrackedRequest offset and bytes fields correspond
to the original request. When request is being expanded to satisfyrequest_alignment, these fields are not updated.Well, shrunk in this case, but OK.
Quoted 1 line
So, maybe, we should assert
overlap_offsetandoverlap_bytes?Maybe, but would that have any benefit? Especially after this patch having been in qemu for over half a year?
(Also, intuitively off the top of my head I don’t see how it would make more sense to check
overlap_offsetandoverlap_bytes, if all the assertions are for is to see that we got the right request.overlap_offsetandoverlap_bytesmay still not exactly match @offset or @bytes, respectively.)
Quoted 5 lines
Quoted 2 lines
Your suggestion makes it sound a bit like you have a different purpose in mind what these assertions might be useful for...?
No I just think it may have false-positives, when actual request is larger than original.
Seems like a bug. Why would we zero more than originally requested?
Quoted 4 lines
So offset may be <
req->offsetandreq->offset+req->bytesmay be
less than offset + bytes. And we will crash. I should make a reproducer to prove it, but it seems possible.
I’m definitely curious.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
+
+ end =INT64_MAX& -(uint64_t)bs->bl.request_alignment; +req->bytes= end -req->offset;And I doubt that we should update
req->bytes. We never updated it in other places, it corresponds to original request. It's enough to updateoverlap_bytesto achieve corresponding serialising.
Quoted 8 lines
Quoted 5 lines
Does it hurt? If so, would you send a patch?
I assume you reply to this patch instead of writing a patch because you have the same feeling of “It probably doesn’t really matter, so let’s have a discussion first”.
1. yes, and
2. I probably don't see the full picture around tracked requests
Neither do I, that’s for sure.
Quoted 19 lines
Quoted 19 lines
My stance is: I don’t think it matters and this whole piece of code is a hack that shouldn’t exist, obviously. So I don’t really care how it fits into all of our other code.
I would like to say I wouldn’t mind a patch to drop the
req->bytesassignment, but OTOH it would mean I’d have to review it and verify that it’s indeed sufficient to setoverlap_bytes.If it’s in any way inconvenient for you that
req->bytesis adjusted, then of course please send one.Quoted 7 lines
Quoted 3 lines
+
req->overlap_bytes=req->bytes;
+
+bdrv_mark_request_serialising(req, bs->bl.request_alignment);Not sure, how much should we care about
request_alignmenthere, I think, it's enough to just setreq->overlap_bytes=INT64_MAX-req->overlap_offest, but it doesn't really matter.
Quoted 8 lines
Quoted 5 lines
As long as
req->bytesis adjusted, we have to care, or theoverlap_bytescalculation inbdrv_mark_request_serialisingwill overflow.Well, one could argue that it doesn’t matter because the
MAX()will still do the right thing, but overflowing is never nice.Hmm I think, if reduce it to just
INT64_MAX, we should pass 1 as align tobdrv_mark_request_serialising.
True.
Quoted 5 lines
Quoted 3 lines
(Of course, it probably doesn’t matter at all if we just wouldn’t touch
req->bytes.)OK, thanks for the answer, I'll prepare a patch.
OK? I’m not sure where the benefit is (apart from the perhaps failing assertions). So it still looks to me like putting too much energy into a hack.
(I think the original reason I set both req->bytes and req->overlap_bytes was actually because I just wanted to be sure, and didn’t want to have to look too hard whether either would be sufficient.)
((Please also note that I can’t guarantee I will review your patch in a timely manner, for one thing because I can already rarely give that promise (as you are probably painfully aware...); and now there’s also two weeks of mail on top for me to wade through after PTO. So if there’s no reason to change anything apart from saving two LoC, well. Failing assertions are a different matter altogether, though, of course.))
Max
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 80 lines · Max Reitz
Quoted 79 lines · Vladimir Sementsov-Ogievskiy
Quoted 78 lines · Max Reitz
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Uh, well. Reasonable, but I hope you don’t mind me having no longer having this patch fresh on my mind.
Quoted 41 lines
Quoted 41 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2
images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX
in practice).Signed-off-by Max Reitz <[email protected]>--- block/file-posix.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)block/file-posix.c +19
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, RawPosixAIOData acb; ThreadPoolFunc *handler; +#ifdef CONFIG_FALLOCATE + if (offset + bytes > bs->total_sectors * BDRV_SECTOR_SIZE) { + BdrvTrackedRequest *req; + uint64_t end; + + /* +* This is a workaround for a bug in the Linux XFS driver, +* where writes submitted through the AIO interface will be +* discarded if they happen beyond a concurrently running +* fallocate() that increases the file length (i.e., both the +* write and the fallocate() happen beyond the EOF). +* +* To work around it, we extend the tracked request for this +* zero write until INT64_MAX (effectively infinity), and mark +* it as serializing. +* +* We have to enable this workaround for all filesystems and +* AIO modes (not just XFS with aio=native), because for +* remote filesystems we do not know the host configuration. +*/ Quoted 8 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2749
Quoted 6 lines
block/file-posix.c +6
-0+ + req = bdrv_co_get_self_request(bs); + assert(req); + assert(req->type == BDRV_TRACKED_WRITE); + assert(req->offset <= offset); + assert(req->offset + req->bytes >= offset + bytes);Why these assertions?
Mostly to see that
bdrv_co_get_self_request()(introduced by the same series) actually got the right request. (I suppose.)Quoted 3 lines
TrackedRequest offset and bytes fields correspond
to the original request. When request is being expanded to satisfyrequest_alignment, these fields are not updated.Well, shrunk in this case, but OK.
Quoted 1 line
So, maybe, we should assert
overlap_offsetandoverlap_bytes?Maybe, but would that have any benefit? Especially after this patch having been in qemu for over half a year?
(Also, intuitively off the top of my head I don’t see how it would make more sense to check
overlap_offsetandoverlap_bytes, if all the assertions are for is to see that we got the right request.overlap_offsetandoverlap_bytesmay still not exactly match @offset or @bytes, respectively.)
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Your suggestion makes it sound a bit like you have a different purpose in mind what these assertions might be useful for...?
No I just think it may have false-positives, when actual request is larger than original.
Seems like a bug. Why would we zero more than originally requested?
Hmm, you are right, seems it's not possible. We may expand the request to do COW, but it will never produce write-zero request larger than original one.
(I really tried to reproduce to understand it :)
Quoted 80 lines
Quoted 5 lines
So offset may be <
req->offsetandreq->offset+req->bytesmay be
less than offset + bytes. And we will crash. I should make a reproducer to prove it, but it seems possible.I’m definitely curious.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2752
Quoted 3 lines
block/file-posix.c +3
-0+ + end = INT64_MAX & -(uint64_t)bs->bl.request_alignment; + req->bytes = end - req->offset;And I doubt that we should update
req->bytes. We never updated it in other places, it corresponds to original request. It's enough to updateoverlap_bytesto achieve corresponding serialising.Quoted 8 lines
Quoted 5 lines
Does it hurt? If so, would you send a patch?
I assume you reply to this patch instead of writing a patch because you have the same feeling of “It probably doesn’t really matter, so let’s have a discussion first”.
1. yes, and
2. I probably don't see the full picture around tracked requestsNeither do I, that’s for sure.
Quoted 19 lines
Quoted 19 lines
My stance is: I don’t think it matters and this whole piece of code is a hack that shouldn’t exist, obviously. So I don’t really care how it fits into all of our other code.
I would like to say I wouldn’t mind a patch to drop the
req->bytesassignment, but OTOH it would mean I’d have to review it and verify that it’s indeed sufficient to setoverlap_bytes.If it’s in any way inconvenient for you that
req->bytesis adjusted, then of course please send one.Quoted 7 lines
block/file-posix.c raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, · line 2755
Quoted 3 lines
block/file-posix.c +3
-0+ req->overlap_bytes = req->bytes; + + bdrv_mark_request_serialising(req, bs->bl.request_alignment);Not sure, how much should we care about
request_alignmenthere, I think, it's enough to just setreq->overlap_bytes=INT64_MAX-req->overlap_offest, but it doesn't really matter.Quoted 8 lines
Quoted 5 lines
As long as
req->bytesis adjusted, we have to care, or theoverlap_bytescalculation inbdrv_mark_request_serialisingwill overflow.Well, one could argue that it doesn’t matter because the
MAX()will still do the right thing, but overflowing is never nice.Hmm I think, if reduce it to just
INT64_MAX, we should pass 1 as align tobdrv_mark_request_serialising.True.
Quoted 5 lines
Quoted 3 lines
(Of course, it probably doesn’t matter at all if we just wouldn’t touch
req->bytes.)OK, thanks for the answer, I'll prepare a patch.
OK? I’m not sure where the benefit is (apart from the perhaps failing assertions). So it still looks to me like putting too much energy into a hack.
(I think the original reason I set both
req->bytesandreq->overlap_byteswas actually because I just wanted to be sure, and didn’t want to have to look too hard whether either would be sufficient.)((Please also note that I can’t guarantee I will review your patch in a timely manner, for one thing because I can already rarely give that promise (as you are probably painfully aware...); and now there’s also two weeks of mail on top for me to wade through after PTO. So if there’s no reason to change anything apart from saving two LoC, well. Failing assertions are a different matter altogether, though, of course.))
Max
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 84 lines · Vladimir Sementsov-Ogievskiy
Quoted 72 lines · Max Reitz
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
01.11.2019 18:25, Max Reitz wrote:
Sorry for being late, I have some comments
Uh, well. Reasonable, but I hope you don’t mind me having no longer having this patch fresh on my mind.
Quoted 41 lines
Quoted 41 lines
The XFS kernel driver has a bug that may cause data corruption for qcow2
images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX
in practice).Signed-off-by Max Reitz <[email protected]>---
block/file-posix.c| 36 ++++++++++++++++++++++++++++++++++++1 file changed, 36 insertions(+)block/file-posix.c +0
-0@@ -2721,6 +2721,42 @@ raw_do_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int bytes, RawPosixAIOData acb; ThreadPoolFunc *handler;+#ifdef
CONFIG_FALLOCATE
+ if (offset + bytes >bs->total_sectors*BDRV_SECTOR_SIZE) { + BdrvTrackedRequest *req;
+uint64_tend;
+
+ /*
+ * This is a workaround for a bug in the Linux XFS driver, + * where writes submitted through the AIO interface will be + * discarded if they happen beyond a concurrently running + *fallocate()that increases the file length (i.e., both the + * write and thefallocate()happen beyond the EOF). + *
+ * To work around it, we extend the tracked request for this + * zero write untilINT64_MAX(effectively infinity), and mark + * it as serializing.
+ *
+ * We have to enable this workaround for all filesystems and + * AIO modes (not just XFS with aio=native), because for + * remote filesystems we do not know the host configuration. + */Quoted 8 lines
Quoted 6 lines
+
+ req =bdrv_co_get_self_request(bs);
+ assert(req);
+assert(req->type == BDRV_TRACKED_WRITE); +assert(req->offset <= offset);
+assert(req->offset + req->bytes >= offset + bytes);Why these assertions?
Mostly to see that
bdrv_co_get_self_request()(introduced by the same series) actually got the right request. (I suppose.)Quoted 3 lines
TrackedRequest offset and bytes fields correspond
to the original request. When request is being expanded to satisfyrequest_alignment, these fields are not updated.Well, shrunk in this case, but OK.
Quoted 1 line
So, maybe, we should assert
overlap_offsetandoverlap_bytes?Maybe, but would that have any benefit? Especially after this patch having been in qemu for over half a year?
Quoted 8 lines
(Also, intuitively off the top of my head I don’t see how it would make more sense to check
overlap_offsetandoverlap_bytes, if all the assertions are for is to see that we got the right request.overlap_offsetandoverlap_bytesmay still not exactly match @offset or @bytes, respectively.)Your suggestion makes it sound a bit like you have a different purpose in mind what these assertions might be useful for...?
No I just think it may have false-positives, when actual request is larger than original.
Quoted 74 lines
Quoted 1 line
Seems like a bug. Why would we zero more than originally requested?
Hmm, you are right, seems it's not possible. We may expand the request to do COW, but it will never produce write-zero request larger than original one.
(I really tried to reproduce to understand it :)
Quoted 67 lines
Quoted 5 lines
So offset may be <
req->offsetandreq->offset+req->bytesmay be
less than offset + bytes. And we will crash. I should make a reproducer to prove it, but it seems possible.I’m definitely curious.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
+
+ end =INT64_MAX& -(uint64_t)bs->bl.request_alignment; +req->bytes= end -req->offset;And I doubt that we should update
req->bytes. We never updated it in other places, it corresponds to original request. It's enough to updateoverlap_bytesto achieve corresponding serialising.Quoted 8 lines
Quoted 5 lines
Does it hurt? If so, would you send a patch?
I assume you reply to this patch instead of writing a patch because you have the same feeling of “It probably doesn’t really matter, so let’s have a discussion first”.
1. yes, and
2. I probably don't see the full picture around tracked requestsNeither do I, that’s for sure.
Quoted 19 lines
Quoted 19 lines
My stance is: I don’t think it matters and this whole piece of code is a hack that shouldn’t exist, obviously. So I don’t really care how it fits into all of our other code.
I would like to say I wouldn’t mind a patch to drop the
req->bytesassignment, but OTOH it would mean I’d have to review it and verify that it’s indeed sufficient to setoverlap_bytes.If it’s in any way inconvenient for you that
req->bytesis adjusted, then of course please send one.Quoted 7 lines
Quoted 3 lines
+
req->overlap_bytes=req->bytes;
+
+bdrv_mark_request_serialising(req, bs->bl.request_alignment);Not sure, how much should we care about
request_alignmenthere, I think, it's enough to just setreq->overlap_bytes=INT64_MAX-req->overlap_offest, but it doesn't really matter.Quoted 8 lines
Quoted 5 lines
As long as
req->bytesis adjusted, we have to care, or theoverlap_bytescalculation inbdrv_mark_request_serialisingwill overflow.Well, one could argue that it doesn’t matter because the
MAX()will still do the right thing, but overflowing is never nice.Hmm I think, if reduce it to just
INT64_MAX, we should pass 1 as align tobdrv_mark_request_serialising.True.
Quoted 5 lines
Quoted 3 lines
(Of course, it probably doesn’t matter at all if we just wouldn’t touch
req->bytes.)OK, thanks for the answer, I'll prepare a patch.
OK? I’m not sure where the benefit is (apart from the perhaps failing assertions). So it still looks to me like putting too much energy into a hack.
Yes, now when I see that assertions should not fail I don't want to care more neither send a patch :)
Quoted 15 lines
Quoted 13 lines
(I think the original reason I set both
req->bytesandreq->overlap_byteswas actually because I just wanted to be sure, and didn’t want to have to look too hard whether either would be sufficient.)((Please also note that I can’t guarantee I will review your patch in a timely manner, for one thing because I can already rarely give that promise (as you are probably painfully aware...); and now there’s also two weeks of mail on top for me to wade through after PTO. So if there’s no reason to change anything apart from saving two LoC, well. Failing assertions are a different matter altogether, though, of course.))
Max
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
In reply to Max ReitzQuoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit
c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAXin practice).
Hi! I'm doing some investigation, and here is an interesting result:
Consider the following test:
img=/ssd/x.qcow2; ./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -w $img
Bisecting results changes between 2.12 and 5.1, I found the following:
2.12: ~20s
....
c8bb23cbdbe32 "qcow2: skip writing zero buffers to empty COW areas" -> becomes ~12s [1]
....
292d06b925b27 "block/file-posix: Let post-EOF fallocate serialize" -> becomes ~9s [2]
....
v5.1 ~9s
And [1] is obvious, it is the main purpose of c8bb23cbdbe32. But [2] is a surprise for me.. Any ideas?
===
just to check: staying at c8bb23cbdbe32 I revert c8bb23cbdbe32 and get again ~19.7s. So [2] doesn't substitute [1].
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 28 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (
INT64_MAXin practice).Hi! I'm doing some investigation, and here is an interesting result:
Consider the following test:
img=/ssd/x.qcow2; ./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -w $img
Bisecting results changes between 2.12 and 5.1, I found the following:
2.12: ~20s
....
c8bb23cbdbe32"qcow2: skip writing zero buffers to empty COW areas" -> becomes ~12s [1]....
292d06b925b27"block/file-posix: Let post-EOF fallocate serialize" -> becomes ~9s [2]....
Quoted 9 lines
v5.1 ~9s
And [1] is obvious, it is the main purpose of
c8bb23cbdbe32. But [2] is a surprise for me.. Any ideas?===
just to check: staying at
c8bb23cbdbe32I revertc8bb23cbdbe32and get again ~19.7s. So [2] doesn't substitute [1].
Note, it's all ext4.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 32 lines · Vladimir Sementsov-Ogievskiy
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (
INT64_MAXin practice).Hi! I'm doing some investigation, and here is an interesting result:
Consider the following test:
img=/ssd/x.qcow2; ./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -w $img
Bisecting results changes between 2.12 and 5.1, I found the following:
2.12: ~20s
....
c8bb23cbdbe32"qcow2: skip writing zero buffers to empty COW areas" -> becomes ~12s [1]....
292d06b925b27"block/file-posix: Let post-EOF fallocate serialize" -> becomes ~9s [2]....
v5.1 ~9s
Quoted 9 lines
Quoted 6 lines
And [1] is obvious, it is the main purpose of
c8bb23cbdbe32. But [2] is a surprise for me.. Any ideas?===
just to check: staying at
c8bb23cbdbe32I revertc8bb23cbdbe32and get again ~19.7s. So [2] doesn't substitute [1].Note, it's all ext4.
Let's go further:
If I add -n, to use aio native:
./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -n -w $img;
Then this patch doesn't have such effect..
But if consider the test without an offset on hdd, the effect is very significant:
./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 45000 -d 64 -f qcow2 -s 16k -t none -n -w $img;
v2.12 ~10.4s
c8bb23cbdbe32^ ~10.6sc8bb23cbdbe32 qcow2: skip writing zero buffers to empty COW areas: ~16.7s : degradation!!! 292d06b925b27^ ~16.4s292d06b925 block/file-posix: Let post-EOF fallocate serialize: ~7.6s: great improvement !
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 32 lines · Vladimir Sementsov-Ogievskiy
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines · Max Reitz
The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (
INT64_MAXin practice).Hi! I'm doing some investigation, and here is an interesting result:
Consider the following test:
img=/ssd/x.qcow2; ./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -w $img
Bisecting results changes between 2.12 and 5.1, I found the following:
2.12: ~20s
....
c8bb23cbdbe32"qcow2: skip writing zero buffers to empty COW areas" -> becomes ~12s [1]....
292d06b925b27"block/file-posix: Let post-EOF fallocate serialize" -> becomes ~9s [2]....
v5.1 ~9s
Quoted 9 lines
Quoted 6 lines
And [1] is obvious, it is the main purpose of
c8bb23cbdbe32. But [2] is a surprise for me.. Any ideas?===
just to check: staying at
c8bb23cbdbe32I revertc8bb23cbdbe32and get again ~19.7s. So [2] doesn't substitute [1].Note, it's all ext4.
Let's go further:
If I add -n, to use aio native:
./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 50000 -d 64 -f qcow2 -o 1k -s 64k -t none -n -w $img;Then this patch doesn't have such effect..
But if consider the test without an offset on hdd, the effect is very significant:
Quoted 8 lines
./qemu-img create -f qcow2 $img 16G; ./qemu-img bench -c 45000 -d 64 -f qcow2 -s 16k -t none -n -w $img;
v2.12 ~10.4s
c8bb23cbdbe32^ ~10.6sc8bb23cbdbe32qcow2: skip writing zero buffers to empty COW areas: ~16.7s : degradation!!! 292d06b925b27^ ~16.4s292d06b925block/file-posix: Let post-EOF fallocate serialize: ~7.6s: great improvement !
Hmm, a bit more accurate results (use separate partition on non-root hard drive). I post just numbers I get from iterations, not calculating the average.
v2.12.0 9.227 8.999 9.059 8.793
c8bb23cbdbe32^ 9.264 8.996 8.969c8bb23cbdbe32 20.694 16.050 15.898 15.918 16.136
292d06b925b27^ 15.509 15.343 16.549292d06b925 8.368 8.529 8.342 7.503 8.361 8.353 8.416 13.043 12.594 7.898 7.907 11.914
292d06b925 + don't-handle-alloc 18.730 9.321 9.016 9.037 18.286 18.268
(don't handle alloc is don't call qcow2_handle_alloc function which is actual revert of c8bb23cbdbe32)
very interesting..
292d06b925^ + don't-handle-alloc 9.200 9.285 9.297 9.296 9.194 9.332 9.155
Signature · 2 lines
Best regards, Vladimir
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
TEST check-qtest-x86_64: tests/ahci-test
TEST check-unit: tests/test-aio-multithread
test-aio-multithread: /tmp/qemu-test/src/util/async.c:283: aio_ctx_finalize: Assertion `!qemu_lockcnt_count(&ctx->list_lock)' failed.
ERROR - too few tests run (expected 6, got 2)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
TEST iotest-qcow2: 013
TEST iotest-qcow2: 017
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=bf88cbb2b1b4497f8f4fc2b674903b08', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-vhhub41q/src/docker-src.2019-11-01-11.37.23.29941:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=bf88cbb2b1b4497f8f4fc2b674903b08
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-vhhub41q/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 11m36.434s
user 0m9.145s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Quoted 16 lines · On 01.11.19 16:48, [email protected] wrote:
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
Quoted 6 lines
TEST check-qtest-x86_64: tests/ahci-test
TEST check-unit: tests/test-aio-multithread
test-aio-multithread:/tmp/qemu-test/src/util/async.c:283:aio_ctx_finalize: Assertion `!qemu_lockcnt_count(&ctx->list_lock)' failed. ERROR - too few tests run (expected 6, got 2)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
This doesn’t seem related to this series and I can’t reproduce it locally, so I’ll just ignore it.
Max
Quoted 14 lines · Max Reitz
Hi,
As I reasoned here:
https://lists.nongnu.org/archive/html/qemu-block/2019-11/msg00027.html
I’m no longer convinced of revertingc8bb23cbdbe. I could see a significant performance improvement from it on ext4 with aio=native in a guest that does random 4k writes, and as such I feel it would be a regression to revert it for 4.2.To work around the XFS corruption, we still need the other three patches from the series, of course. We cannot restrict the workaround to just XFS, because maybe the file is on a remote filesystem and then we don’t know anything about the host configuration.
Quoted 7 lines
The performance impact should still be minimal because this just serializes post-EOF write-zeroes and data writes, and that just doesn’t happen very often, even with
handle_alloc_space()in qcow2.I would love to have more time to make a decision, but there simply isn’t any. Patches for 4.1.1 are to be collected on Monday, AFAIU.
I would have liked some reviews on this series (so I waited over the weekend, even though I didn’t expect any), but now I’ve applied it anyway (and sent a pull request). I understand it was difficult last week because of KVM Forum.
AFAIU we need it today for stable, and there didn’t seem to be any opposition on these patches here, just on the revert of c8bb23cbdbe.
I welcome you to still review the series and then shout “STOP” at the pull request if you find it necessary.
Max
Quoted 55 lines · Max Reitz
Hi,
As I reasoned here:
https://lists.nongnu.org/archive/html/qemu-block/2019-11/msg00027.html
I’m no longer convinced of revertingc8bb23cbdbe. I could see a significant performance improvement from it on ext4 with aio=native in a guest that does random 4k writes, and as such I feel it would be a regression to revert it for 4.2.To work around the XFS corruption, we still need the other three patches from the series, of course. We cannot restrict the workaround to just XFS, because maybe the file is on a remote filesystem and then we don’t know anything about the host configuration.
The performance impact should still be minimal because this just serializes post-EOF write-zeroes and data writes, and that just doesn’t happen very often, even with
handle_alloc_space()in qcow2.I would love to have more time to make a decision, but there simply isn’t any. Patches for 4.1.1 are to be collected on Monday, AFAIU.
v2:
- Dropped patch 1
- Forgot a “coroutine_fn” in patch 2 (it isn’t really important,qemu_coroutine_self()works in non-coroutine functions, too, but calling bdrv_(co_)get_self_request() from a non-coroutine just doesn’t make any sense)
- Patch 3:
- Reverted the commit message to the RFC state to reflect that this is specifically because ofhandle_alloc_space()
- Dropped the two lines that said there’d be no performance impact at all because no driver would submit zero writes beyond the EOF (because qcow2 now still does)git-backport-diff against v1:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively001/3:[----] [--] 'block: Make wait/mark serialising requests public' 002/3:[0004] [FC] 'block: Add
bdrv_co_get_self_request()' 003/3:[0002] [FC] 'block/file-posix: Let post-EOF fallocate serialize'Max Reitz (3):
block: Make wait/mark serialising requests public
block: Addbdrv_co_get_self_request()block/file-posix: Let post-EOF fallocate serialize
Quoted 8 lines
include/block/block_int.h | 4 ++++ block/file-posix.c | 36 +++++++++++++++++++++++++++++++++ block/io.c | 42 ++++++++++++++++++++++++++++----------- 3 files changed, 70 insertions(+), 12 deletions(-)Signature · 1 line
2.21.0