qcow2: Fix image corruption bug in 4.1
cover qcow2: Fix image corruption bug in 4.1
This series fixes an image corruption bug that was introduced in commit 69f47505e ('block: avoid recursive block_status call if possible'), first contained in the QEMU 4.1.0 release.
This bug was reported by Michael Weiser on Launchpad:
https://bugs.launchpad.net/qemu/+bug/1846427
Kevin Wolf (3):
coroutine: Add qemu_co_mutex_assert_locked()
qcow2: Assert that qcow2_cache_get() callers hold s->lock
qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
include/qemu/coroutine.h | 7 +++++++ block/qcow2-cache.c | 5 +++++ block/qcow2-refcount.c | 2 ++ block/qcow2.c | 3 ++- 4 files changed, 16 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
PATCH 1/3 coroutine: Add qemu_co_mutex_assert_locked()
Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.
--- include/qemu/coroutine.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/qemu/coroutine.h +7 -0
@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);*/ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex); +/** +* Assert that the current coroutine holds @mutex. +*/ +static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex) +{ +assert (mutex->locked && mutex->holder ==qemu_coroutine_self ()); +}/** * CoQueues are a mechanism to queue coroutines in order to continue executing
Signature · 1 line
2.20.1
Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()
Quoted 18 lines · Kevin Wolf
Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.
Signed-off-by Kevin Wolf <[email protected]>--- include/qemu/coroutine.h | 7 +++++++ 1 file changed, 7 insertions(+)include/qemu/coroutine.h +0
-0@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);*/ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179
Quoted 10 lines
include/qemu/coroutine.h +7
-0+/** +* Assert that the current coroutine holds @mutex. +*/ +static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex) +{ +assert (mutex->locked && mutex->holder ==qemu_coroutine_self ()); +}/** * CoQueues are a mechanism to queue coroutines in order to continue executing
I think that we should use atomic_read(&mutex->locked) and require barriers working with holder.
Den
Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()
Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben:
Quoted 27 lines · Kevin Wolf
Quoted 26 lines
Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.
Signed-off-by Kevin Wolf <[email protected]>--- include/qemu/coroutine.h | 7 +++++++ 1 file changed, 7 insertions(+)include/qemu/coroutine.h +7
-0@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);*/ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex); +/** +* Assert that the current coroutine holds @mutex. +*/ +static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex) +{ +assert (mutex->locked && mutex->holder ==qemu_coroutine_self ()); +}
Quoted 4 lines
include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179
Quoted 2 lines
include/qemu/coroutine.h +0
-0/** * CoQueues are a mechanism to queue coroutines in order to continue executing I think that we should use
atomic_read(&mutex->locked)and require barriers working with holder.
Hm, this would only be necessary for the case that the assertion doesn't hold true. I'll do the atomic_read() because it's easy enough, but I don't think we need or want barriers here. If another thread modifies this concurrently, the condition is false either way.
Kevin
Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()
Quoted 32 lines · Kevin Wolf
Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben:
Quoted 27 lines · Kevin Wolf
Quoted 26 lines
Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.
Signed-off-by Kevin Wolf <[email protected]>--- include/qemu/coroutine.h | 7 +++++++ 1 file changed, 7 insertions(+)include/qemu/coroutine.h +7
-0@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);*/ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex); +/** +* Assert that the current coroutine holds @mutex. +*/ +static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex) +{ +assert (mutex->locked && mutex->holder ==qemu_coroutine_self ()); +}Quoted 4 lines
include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179
Quoted 2 lines
include/qemu/coroutine.h +0
-0/** * CoQueues are a mechanism to queue coroutines in order to continue executing I think that we should use
atomic_read(&mutex->locked)and require barriers working with holder.
Quoted 7 lines
Hm, this would only be necessary for the case that the assertion doesn't hold true. I'll do the
atomic_read()because it's easy enough, but I don't think we need or want barriers here. If another thread modifies this concurrently, the condition is false either way.Kevin
It looks like you are correct. We will see either NULL or something other if we are in the process of lock taking. Does this worth a comment? :)
Den
PATCH 2/3 qcow2: Assert that qcow2_cache_get() callers hold s->lock
In reply to Kevin Wolfqcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.
Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.
--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)
block/qcow2-cache.c +5 -0
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }trace_qcow2_cache_get (qemu_coroutine_self (), c == s->l2_table_cache, offset, read_from_disk);
@@ -386,6 +389,8 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, } } +assert (c->entries[i].ref ==0 ); +assert (c->entries[i].offset ==0 ); c->entries[i].offset = offset;/* And return the right table */
Signature · 1 line
2.20.1
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Am 23.10.2019 um 17:26 hat Kevin Wolf geschrieben:
Quoted 10 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>
Oops, this one was a bit too optimistic. :-)
I'm still running tests to see if any other code paths trigger the assertion, but image creation calls this without the lock held (which is harmless because nobody else knows about the image so there won't be concurrent requests). The following patch is needed additionally to make image creation work with the new assertion.
Kevin
block/qcow2.c +6 -1
@@ -3213,6 +3213,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)BlockBackend *blk =NULL ;BlockDriverState *bs =NULL ;BlockDriverState *data_bs =NULL ; +BDRVQcow2State *s;QCowHeader *header;size_t cluster_size;int version;
@@ -3424,7 +3425,12 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) goto out; } + s =blk_bs (blk)->opaque; + +qemu_co_mutex_lock (&s->lock); ret =qcow2_alloc_clusters (blk_bs (blk),3 * cluster_size); +qemu_co_mutex_unlock (&s->lock); +if (ret <0 ) {error_setg_errno (errp, -ret,"Could not allocate clusters for qcow2 " "header and refcount table" );
@@ -3437,7 +3443,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)/* Set the external data file if necessary */ if (data_bs) { -BDRVQcow2State *s =blk_bs (blk)->opaque; s->image_data_file =g_strdup (data_bs->filename); }
Michael Weiser
Tested-byRe: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Hi Kevin,
Quoted 8 lines · Kevin Wolf
Quoted 8 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.
Quoted 5 lines
I'm still running tests to see if any other code paths trigger the assertion, but image creation calls this without the lock held (which is harmless because nobody else knows about the image so there won't be concurrent requests). The following patch is needed additionally to make image creation work with the new assertion.
I can confirm that with all four patches corruption does no longer occur as of commit 69f47505ee66afaa513305de0c1895a224e52c45. Removing only 3/3 (qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()) the assertion triggers after a few seconds, leaving behind a few leaked clusters but no errors in the image.
(qemu) qemu-system-x86_64:qemu/include/qemu/coroutine.h:175:qemu_co_mutex_assert_locked: Assertion `mutex->locked && mutex->holder == qemu_coroutine_self()' failed.
Aborted (core dumped)
$ qemu-img check qtest.qcow2 Leaked cluster 169257 refcount=3 reference=2 Leaked cluster 172001 refcount=1 reference=0 Leaked cluster 172002 refcount=1 reference=0 Leaked cluster 172003 refcount=1 reference=0 Leaked cluster 172004 refcount=1 reference=0 Leaked cluster 172005 refcount=1 reference=0 Leaked cluster 172006 refcount=1 reference=0 Leaked cluster 172007 refcount=1 reference=0 Leaked cluster 172008 refcount=1 reference=0 Leaked cluster 172009 refcount=1 reference=0 Leaked cluster 172010 refcount=1 reference=0 Leaked cluster 172011 refcount=1 reference=0 Leaked cluster 172012 refcount=1 reference=0
13 leaked clusters were found on the image.
This means waste of disk space, but no harm to data. 255525/327680 = 77.98% allocated, 3.22% fragmented, 0.00% compressed clusters
Image end offset: 17106403328
I was going to test with master as well but got overtaken by v2. Will move on to test v2 now. :)
Series:
No biggie but if there's a chance could you switch my address to the above?
Signature · 2 lines
Thanks, Michael
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Am 25.10.2019 um 12:35 hat Michael Weiser geschrieben:
Quoted 5 lines
I was going to test with master as well but got overtaken by v2. Will move on to test v2 now. :)
Series:
Tested-by Michael Weiser <[email protected]>
Thanks for testing!
The fix itself is unchanged in v2, so I assume the result will be the same, but testing it explicitly can't hurt. I'm going to send a pull request today, but if you're quick, I can wait for your results.
Quoted 2 lines
No biggie but if there's a chance could you switch my address to the above?
No problem, I've updated the address in the Reported-by tag.
Kevin
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
In reply to Kevin WolfQuoted 19 lines · Kevin Wolf
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0
block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 6 lines
block/qcow2-cache.c +3
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }
that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
Den
Quoted 12 lines
block/qcow2-cache.c +2
-0trace_qcow2_cache_get (qemu_coroutine_self (), c == s->l2_table_cache, offset, read_from_disk);@@ -386,6 +389,8 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, } } +assert (c->entries[i].ref ==0 ); +assert (c->entries[i].offset ==0 ); c->entries[i].offset = offset;/* And return the right table */
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:
Quoted 22 lines · Kevin Wolf
Quoted 21 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;
Quoted 8 lines
block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 4 lines
block/qcow2-cache.c +3
-0assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.
Kevin
Denis Lunev
Reviewed-by Denis V. LunevRe: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Quoted 30 lines · Kevin Wolf
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:
Quoted 22 lines · Kevin Wolf
Quoted 21 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;Quoted 7 lines
block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 4 lines
block/qcow2-cache.c +3
-0assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
Quoted 6 lines
As the commit message explains, outside of coroutine context, we can't yield and
bdrv_preadandbdrv_flushbecome synchronous operations instead, so there is nothing else that we need to protect against.Kevin
Hmm. It seems I was not careful enough with reading entire message. I am fine with this though it looks a bit tricky to me as such things can change in the future.
Anyway, you could consider this as
Den
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Am 24.10.2019 um 13:14 hat Denis Lunev geschrieben:
Quoted 31 lines · Kevin Wolf
Quoted 30 lines
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:
Quoted 29 lines · Kevin Wolf
Quoted 19 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 6 lines
block/qcow2-cache.c +3
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
Quoted 7 lines
Quoted 4 lines
As the commit message explains, outside of coroutine context, we can't yield and
bdrv_preadandbdrv_flushbecome synchronous operations instead, so there is nothing else that we need to protect against.Hmm. It seems I was not careful enough with reading entire message. I am fine with this though it looks a bit tricky to me as such things can change in the future.
In which way do you think this could change? It's a pretty fundamental fact about non-coroutine code that it can't yield.
What could change, of course, is that some code switches from being synchronous to using a coroutine. The assertion would automatically apply then and catch the bug if adding proper locking is forgotten.
Quoted 3 lines
Anyway, you could consider this as
Reviewed-by Denis V. Lunev <[email protected]>
Thanks!
Kevin
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
In reply to Kevin WolfQuoted 32 lines · Kevin Wolf
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:
Quoted 22 lines · Kevin Wolf
Quoted 21 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;Quoted 8 lines
block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 4 lines
block/qcow2-cache.c +3
-0assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
Quoted 4 lines
As the commit message explains, outside of coroutine context, we can't yield and
bdrv_preadandbdrv_flushbecome synchronous operations instead, so there is nothing else that we need to protect against.
Recently we discussed similar problems about block-dirty-bitmap-* qmp commands, which wanted to update qcow2 metadata about bitmaps from non-coroutine context. "qcow2 lock"
<[email protected]>
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg01419.html
And, as I understand, the correct way is to convert to coroutine and lock mutex appropriately. Or we want to lock aio context and to be in drained section to avoid parallel requests accessing critical section. Should we assert such conditions in case of !qemu_in_coroutine() ?
--
Final commit to fix bug about bitmaps:
commit d2c3080e41fd2c9bc36c996cc9d33804462ba803
Author: Vladimir Sementsov-Ogievskiy <[email protected]> Date: Fri Sep 20 11:25:43 2019 +0300
block/qcow2: proper locking on bitmap add/remove paths
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock
Am 24.10.2019 um 15:03 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 38 lines
Quoted 32 lines · Kevin Wolf
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:
Quoted 22 lines · Kevin Wolf
Quoted 21 lines
qcow2_cache_do_get()requires thats->lockis locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where
bdrv_preadand flushes become synchronous operations. The lock cannot and need not be taken in this case.Signed-off-by Kevin Wolf <[email protected]>--- block/qcow2-cache.c | 5 +++++ 1 file changed, 5 insertions(+)block/qcow2-cache.c +0
-0@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,int min_lru_index =-1 ;Quoted 8 lines
block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332
Quoted 4 lines
block/qcow2-cache.c +3
-0assert (offset !=0 ); +if (qemu_in_coroutine ()) { +qemu_co_mutex_assert_locked (&s->lock); + }that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.
Quoted 4 lines
As the commit message explains, outside of coroutine context, we can't yield and
bdrv_preadandbdrv_flushbecome synchronous operations instead, so there is nothing else that we need to protect against.
Quoted 5 lines
Recently we discussed similar problems about block-dirty-bitmap-* qmp commands, which wanted to update qcow2 metadata about bitmaps from non-coroutine context. "qcow2 lock" <[email protected]>
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg01419.html
Hm, right, I already forgot about the nested event loop again...
Quoted 4 lines
And, as I understand, the correct way is to convert to coroutine and lock mutex appropriately. Or we want to lock aio context and to be in drained section to avoid parallel requests accessing critical section. Should we assert such conditions in case of !qemu_in_coroutine() ?
The AioContext lock must be held anyway, so I don't think this would be a new requirement. As for draining, I'll have to see.
I'm currently still auditing all the callers of qcow2_cache_do_get(). The synchronous callers I already know are the snapshot functions. I think these happen to be in a drained section anyway (or should be at least), so AioContext lock + drain seems like a very reasonable option for them.
For other synchronous callers, if any, maybe conversion to a coroutine would make more sense.
Kevin
PATCH 3/3 qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
In reply to Kevin Wolfqcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.
As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the qcow2_get_refcount() calls, but around the whole function.
This patch takes the lock in qcow2_co_block_status() earlier and asserts in qcow2_detect_metadata_preallocation() that we hold the lock.
--- block/qcow2-refcount.c | 2 ++ block/qcow2.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +2 -0
@@ -3455,6 +3455,8 @@ int qcow2_detect_metadata_preallocation(BlockDriverState *bs)int64_t i, end_cluster, cluster_count =0 , threshold;int64_t file_length, real_allocation, real_clusters; +qemu_co_mutex_assert_locked (&s->lock); + file_length =bdrv_getlength (bs->file->bs);if (file_length <0 ) {return file_length;
block/qcow2.c +2 -1
@@ -1916,6 +1916,8 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,unsigned int bytes;int status =0 ; +qemu_co_mutex_lock (&s->lock); +if (!s->metadata_preallocation_checked) { ret =qcow2_detect_metadata_preallocation (bs); s->metadata_preallocation = (ret ==1 );
@@ -1923,7 +1925,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs, } bytes =MIN (INT_MAX , count); -qemu_co_mutex_lock (&s->lock); ret =qcow2_get_cluster_offset (bs, offset, &bytes, &cluster_offset);qemu_co_mutex_unlock (&s->lock);if (ret <0 ) {
Signature · 1 line
2.20.1
Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
Quoted 11 lines · Kevin Wolf
qcow2_detect_metadata_preallocation()callsqcow2_get_refcount()which requiress->lockto be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the
qcow2_get_refcount()calls, but around the whole function.
Quoted 7 lines
This patch takes the lock in
qcow2_co_block_status()earlier and asserts inqcow2_detect_metadata_preallocation()that we hold the lock.Fixes 69f47505ee66afaa513305de0c1895a224e52c45Reported-by Michael Weiser <[email protected]>Signed-off-by Kevin Wolf <[email protected]>
Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(
Quoted 40 lines
--- block/qcow2-refcount.c | 2 ++ block/qcow2.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)block/qcow2-refcount.c +2
-0@@ -3455,6 +3455,8 @@ int qcow2_detect_metadata_preallocation(BlockDriverState *bs)int64_t i, end_cluster, cluster_count =0 , threshold;int64_t file_length, real_allocation, real_clusters; +qemu_co_mutex_assert_locked (&s->lock); + file_length =bdrv_getlength (bs->file->bs);if (file_length <0 ) {return file_length;block/qcow2.c +2
-1@@ -1916,6 +1916,8 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,unsigned int bytes;int status =0 ; +qemu_co_mutex_lock (&s->lock); +if (!s->metadata_preallocation_checked) { ret =qcow2_detect_metadata_preallocation (bs); s->metadata_preallocation = (ret ==1 );@@ -1923,7 +1925,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs, } bytes =MIN (INT_MAX , count); -qemu_co_mutex_lock (&s->lock); ret =qcow2_get_cluster_offset (bs, offset, &bytes, &cluster_offset);qemu_co_mutex_unlock (&s->lock);if (ret <0 ) {
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 15 lines
Quoted 14 lines · Kevin Wolf
qcow2_detect_metadata_preallocation()callsqcow2_get_refcount()which requiress->lockto be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the
qcow2_get_refcount()calls, but around the whole function.This patch takes the lock in
qcow2_co_block_status()earlier and asserts inqcow2_detect_metadata_preallocation()that we hold the lock.
Quoted 7 lines
Quoted 4 lines
Fixes 69f47505ee66afaa513305de0c1895a224e52c45Reported-by Michael Weiser <[email protected]>Signed-off-by Kevin Wolf <[email protected]>Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(
Seriously? Is your downstream QEMU so divergent from upstream that you wouldn't notice something like this? I think you really have to improve something there.
I mean, we have a serious data corruptor in the 4.1 release and I wasted days to debug this, and you've been sitting on a fix for months without telling anyone? This isn't a memory leak or something, this kills people's images. It's eating their data.
Modifying an image format driver requires utmost care and I think I'll try to make sure to allow only few qcow2 changes per release in the future. Too many changes were made in too short time, and with too little care, and there are even more qcow2 patches pending. Please check whether these series actually match your downstream code. Anyway, we'll tread very carefully now with the pending patches, even if it means delaying them for another release or two. Stability is way more important for an image format driver than new features and optimisations.
Do whatever you need to fix your downstream process, but seriously, this must not ever happen again.
Kevin
Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
Quoted 21 lines · Kevin Wolf
Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 20 lines
Quoted 18 lines · Kevin Wolf
qcow2_detect_metadata_preallocation()callsqcow2_get_refcount()which requiress->lockto be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the
qcow2_get_refcount()calls, but around the whole function.This patch takes the lock in
qcow2_co_block_status()earlier and asserts inqcow2_detect_metadata_preallocation()that we hold the lock.Fixes 69f47505ee66afaa513305de0c1895a224e52c45Reported-by Michael Weiser <[email protected]>Signed-off-by Kevin Wolf <[email protected]>
Quoted 6 lines
Quoted 2 lines
Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(
Seriously? Is your downstream QEMU so divergent from upstream that you wouldn't notice something like this? I think you really have to improve something there.
How to improve it? Months and years are passed to bring patches into upstream, of course we have to live with a bunch (now it's nearer to 300 and this is a lot better then 500+ in the recent past) of patches above Rhel release.
Quoted 4 lines
I mean, we have a serious data corruptor in the 4.1 release and I wasted days to debug this, and you've been sitting on a fix for months without telling anyone?
Of course, it was not my goal to hide the fix, I'll do my best to avoid this in future. Very sorry for your time wasted.
Quoted 2 lines
This isn't a memory leak or something, this kills
people's images. It's eating their data.
Possibly, I didn't know about data corruption. For some reason I decided that lock should be taken here, I don't remember. Still I should have applied it to upstream version too, of course.
Quoted 6 lines
Modifying an image format driver requires utmost care and I think I'll try to make sure to allow only few qcow2 changes per release in the future. Too many changes were made in too short time, and with too little care, and there are even more qcow2 patches pending. Please check whether these series actually match your downstream code.
The difference is that I didn't move the lock() but add separate lock()/unlock() pair around qcow2_detect_metadata_preallocation(). I think there is no serious difference.
Quoted 3 lines
Anyway, we'll
tread very carefully now with the pending patches, even if it means delaying them for another release or two.
What do you mean? What to do with sent patches during several months?
Quoted 3 lines
Stability is way more
important for an image format driver than new features and optimisations.
Agree. But I think, the main source of stability is testing.
Quoted 4 lines
Do whatever you need to fix your downstream process, but seriously, this must not ever happen again.
I don't see how to improve the process to exclude such problems. But I'll of course will do my best to avoid them.
Signature · 2 lines
Best regards, Vladimir