Skip to contentSkip to conversation

qcow2: Fix image corruption bug in 4.1

PATCH Superseded Kevin Wolf 4 participants 19 messages Oct 23, 2019 – Oct 25, 2019 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

cover qcow2: Fix image corruption bug in 4.1

This series fixes an image corruption bug that was introduced in commit 69f47505e ('block: avoid recursive block_status call if possible'), first contained in the QEMU 4.1.0 release.

This bug was reported by Michael Weiser on Launchpad:
https://bugs.launchpad.net/qemu/+bug/1846427

Kevin Wolf (3):
coroutine: Add qemu_co_mutex_assert_locked()
qcow2: Assert that qcow2_cache_get() callers hold s->lock
qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

 include/qemu/coroutine.h | 7 +++++++
 block/qcow2-cache.c      | 5 +++++
 block/qcow2-refcount.c   | 2 ++
 block/qcow2.c            | 3 ++-
 4 files changed, 16 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/3 coroutine: Add qemu_co_mutex_assert_locked()

Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.

---
 include/qemu/coroutine.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/qemu/coroutine.h +7 -0
index 9801e7f5a4..a36bcfe5c8 100644
@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
  */
 void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
 
+/**
+ * Assert that the current coroutine holds @mutex.
+ */
+static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex)
+{
+    assert(mutex->locked && mutex->holder == qemu_coroutine_self());
+}
 
 /**
  * CoQueues are a mechanism to queue coroutines in order to continue executing
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

Quoted 18 lines · Kevin Wolf

Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.

---
 include/qemu/coroutine.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/qemu/coroutine.h +0 -0
index 9801e7f5a4..a36bcfe5c8 100644
@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
  */
 void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
 

include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179

Quoted 10 lines
include/qemu/coroutine.h +7 -0
+/**
+ * Assert that the current coroutine holds @mutex.
+ */
+static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex)
+{
+    assert(mutex->locked && mutex->holder == qemu_coroutine_self());
+}
 
 /**
  * CoQueues are a mechanism to queue coroutines in order to continue executing

I think that we should use atomic_read(&mutex->locked) and require barriers working with holder.

Den

author
#3 lore raw

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben:

Quoted 27 lines · Kevin Wolf
Quoted 26 lines

Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.

---
 include/qemu/coroutine.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/qemu/coroutine.h +7 -0
index 9801e7f5a4..a36bcfe5c8 100644
@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
  */
 void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
 
+/**
+ * Assert that the current coroutine holds @mutex.
+ */
+static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex)
+{
+    assert(mutex->locked && mutex->holder == qemu_coroutine_self());
+}
 
Quoted 4 lines

include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179

Quoted 2 lines
include/qemu/coroutine.h +0 -0
 /**
  * CoQueues are a mechanism to queue coroutines in order to continue executing

I think that we should use atomic_read(&mutex->locked) and require barriers working with holder.

Hm, this would only be necessary for the case that the assertion doesn't hold true. I'll do the atomic_read() because it's easy enough, but I don't think we need or want barriers here. If another thread modifies this concurrently, the condition is false either way.

Kevin

#4 lore raw

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

Quoted 32 lines · Kevin Wolf

Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben:

Quoted 27 lines · Kevin Wolf
Quoted 26 lines

Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held.

---
 include/qemu/coroutine.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/qemu/coroutine.h +7 -0
index 9801e7f5a4..a36bcfe5c8 100644
@@ -167,6 +167,13 @@ void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
  */
 void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
 
+/**
+ * Assert that the current coroutine holds @mutex.
+ */
+static inline coroutine_fn void qemu_co_mutex_assert_locked(CoMutex *mutex)
+{
+    assert(mutex->locked && mutex->holder == qemu_coroutine_self());
+}
 
Quoted 4 lines

include/qemu/coroutine.h void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex); · line 179

Quoted 2 lines
include/qemu/coroutine.h +0 -0
 /**
  * CoQueues are a mechanism to queue coroutines in order to continue executing

I think that we should use atomic_read(&mutex->locked) and require barriers working with holder.

Quoted 7 lines

Hm, this would only be necessary for the case that the assertion doesn't hold true. I'll do the atomic_read() because it's easy enough, but I don't think we need or want barriers here. If another thread modifies this concurrently, the condition is false either way.

Kevin

It looks like you are correct. We will see either NULL or something other if we are in the process of lock taking. Does this worth a comment? :)

Den

author
#5 lore raw

PATCH 2/3 qcow2: Assert that qcow2_cache_get() callers hold s->lock

In reply to Kevin Wolf

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
 block/qcow2-cache.c | 5 +++++
 1 file changed, 5 insertions(+)
block/qcow2-cache.c +5 -0
index d29b038a67..75b13dad99 100644
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
     int min_lru_index = -1;
 
     assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }
 
     trace_qcow2_cache_get(qemu_coroutine_self(), c == s->l2_table_cache,
                           offset, read_from_disk);
@@ -386,6 +389,8 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
         }
     }
 
+    assert(c->entries[i].ref == 0);
+    assert(c->entries[i].offset == 0);
     c->entries[i].offset = offset;
 
     /* And return the right table */
Signature · 1 line
2.20.1
author
#6 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Am 23.10.2019 um 17:26 hat Kevin Wolf geschrieben:

Quoted 10 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

Oops, this one was a bit too optimistic. :-)

I'm still running tests to see if any other code paths trigger the assertion, but image creation calls this without the lock held (which is harmless because nobody else knows about the image so there won't be concurrent requests). The following patch is needed additionally to make image creation work with the new assertion.

Kevin

block/qcow2.c +6 -1
index 0bc69e6996..7761cf3e07 100644
@@ -3213,6 +3213,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
     BlockBackend *blk = NULL;
     BlockDriverState *bs = NULL;
     BlockDriverState *data_bs = NULL;
+    BDRVQcow2State *s;
     QCowHeader *header;
     size_t cluster_size;
     int version;
@@ -3424,7 +3425,12 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         goto out;
     }
 
+    s = blk_bs(blk)->opaque;
+
+    qemu_co_mutex_lock(&s->lock);
     ret = qcow2_alloc_clusters(blk_bs(blk), 3 * cluster_size);
+    qemu_co_mutex_unlock(&s->lock);
+
     if (ret < 0) {
         error_setg_errno(errp, -ret, "Could not allocate clusters for qcow2 "
                          "header and refcount table");
@@ -3437,7 +3443,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
 
     /* Set the external data file if necessary */
     if (data_bs) {
-        BDRVQcow2State *s = blk_bs(blk)->opaque;
         s->image_data_file = g_strdup(data_bs->filename);
     }
 
Tested-by
#7 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Hi Kevin,

Quoted 8 lines · Kevin Wolf
Quoted 8 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

Quoted 5 lines

I'm still running tests to see if any other code paths trigger the assertion, but image creation calls this without the lock held (which is harmless because nobody else knows about the image so there won't be concurrent requests). The following patch is needed additionally to make image creation work with the new assertion.

I can confirm that with all four patches corruption does no longer occur as of commit 69f47505ee66afaa513305de0c1895a224e52c45. Removing only 3/3 (qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()) the assertion triggers after a few seconds, leaving behind a few leaked clusters but no errors in the image.

(qemu) qemu-system-x86_64:qemu/include/qemu/coroutine.h:175:
qemu_co_mutex_assert_locked: Assertion `mutex->locked && mutex->holder == qemu_coroutine_self()' failed.
Aborted (core dumped)

$ qemu-img check qtest.qcow2 
Leaked cluster 169257 refcount=3 reference=2
Leaked cluster 172001 refcount=1 reference=0
Leaked cluster 172002 refcount=1 reference=0
Leaked cluster 172003 refcount=1 reference=0
Leaked cluster 172004 refcount=1 reference=0
Leaked cluster 172005 refcount=1 reference=0
Leaked cluster 172006 refcount=1 reference=0
Leaked cluster 172007 refcount=1 reference=0
Leaked cluster 172008 refcount=1 reference=0
Leaked cluster 172009 refcount=1 reference=0
Leaked cluster 172010 refcount=1 reference=0
Leaked cluster 172011 refcount=1 reference=0
Leaked cluster 172012 refcount=1 reference=0

13 leaked clusters were found on the image.
This means waste of disk space, but no harm to data. 255525/327680 = 77.98% allocated, 3.22% fragmented, 0.00% compressed clusters
Image end offset: 17106403328

I was going to test with master as well but got overtaken by v2. Will move on to test v2 now. :)

Series:

No biggie but if there's a chance could you switch my address to the above?

Signature · 2 lines
Thanks,
Michael
author
#8 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Am 25.10.2019 um 12:35 hat Michael Weiser geschrieben:

Quoted 5 lines

I was going to test with master as well but got overtaken by v2. Will move on to test v2 now. :)

Series:

Thanks for testing!

The fix itself is unchanged in v2, so I assume the result will be the same, but testing it explicitly can't hurt. I'm going to send a pull request today, but if you're quick, I can wait for your results.

Quoted 2 lines

No biggie but if there's a chance could you switch my address to the above?

No problem, I've updated the address in the Reported-by tag.

Kevin

#9 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

In reply to Kevin Wolf
Quoted 19 lines · Kevin Wolf

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
 block/qcow2-cache.c | 5 +++++
 1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 6 lines
block/qcow2-cache.c +3 -0
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
     int min_lru_index = -1;
 
     assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

Den

Quoted 12 lines
block/qcow2-cache.c +2 -0
     trace_qcow2_cache_get(qemu_coroutine_self(), c == s->l2_table_cache,
                           offset, read_from_disk);
@@ -386,6 +389,8 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
         }
     }
 
+    assert(c->entries[i].ref == 0);
+    assert(c->entries[i].offset == 0);
     c->entries[i].offset = offset;
 
     /* And return the right table */
author
#10 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:

Quoted 22 lines · Kevin Wolf
Quoted 21 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
 block/qcow2-cache.c | 5 +++++
 1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
     int min_lru_index = -1;
 
Quoted 8 lines

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 4 lines
block/qcow2-cache.c +3 -0
     assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.

Kevin

Reviewed-by Denis V. Lunev
#11 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Quoted 30 lines · Kevin Wolf

Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:

Quoted 22 lines · Kevin Wolf
Quoted 21 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
 block/qcow2-cache.c | 5 +++++
 1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
     int min_lru_index = -1;
 
Quoted 7 lines

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 4 lines
block/qcow2-cache.c +3 -0
     assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

Quoted 6 lines

As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.

Kevin

Hmm. It seems I was not careful enough with reading entire message. I am fine with this though it looks a bit tricky to me as such things can change in the future.

Anyway, you could consider this as

Den

author
#12 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Am 24.10.2019 um 13:14 hat Denis Lunev geschrieben:

Quoted 31 lines · Kevin Wolf
Quoted 30 lines

Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:

Quoted 29 lines · Kevin Wolf
Quoted 19 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
 block/qcow2-cache.c | 5 +++++
 1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 6 lines
block/qcow2-cache.c +3 -0
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
     int min_lru_index = -1;
 
     assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

Quoted 7 lines
Quoted 4 lines

As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.

Hmm. It seems I was not careful enough with reading entire message. I am fine with this though it looks a bit tricky to me as such things can change in the future.

In which way do you think this could change? It's a pretty fundamental fact about non-coroutine code that it can't yield.

What could change, of course, is that some code switches from being synchronous to using a coroutine. The assertion would automatically apply then and catch the bug if adding proper locking is forgotten.

Quoted 3 lines

Anyway, you could consider this as

Thanks!

Kevin

#13 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

In reply to Kevin Wolf
Quoted 32 lines · Kevin Wolf

Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:

Quoted 22 lines · Kevin Wolf
Quoted 21 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
  block/qcow2-cache.c | 5 +++++
  1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
      int min_lru_index = -1;
 
Quoted 8 lines

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 4 lines
block/qcow2-cache.c +3 -0
      assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

Quoted 4 lines

As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.

Recently we discussed similar problems about block-dirty-bitmap-* qmp commands, which wanted to update qcow2 metadata about bitmaps from non-coroutine context. "qcow2 lock"
<[email protected]>
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg01419.html

And, as I understand, the correct way is to convert to coroutine and lock mutex appropriately. Or we want to lock aio context and to be in drained section to avoid parallel requests accessing critical section. Should we assert such conditions in case of !qemu_in_coroutine() ?

--
Final commit to fix bug about bitmaps:

commit d2c3080e41fd2c9bc36c996cc9d33804462ba803

Author: Vladimir Sementsov-Ogievskiy <[email protected]>
Date:   Fri Sep 20 11:25:43 2019 +0300

block/qcow2: proper locking on bitmap add/remove paths

Signature · 2 lines
Best regards,
Vladimir
author
#14 lore raw

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

Am 24.10.2019 um 15:03 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 38 lines
Quoted 32 lines · Kevin Wolf

Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben:

Quoted 22 lines · Kevin Wolf
Quoted 21 lines

qcow2_cache_do_get() requires that s->lock is locked because it can yield between picking a cache entry and actually taking ownership of it by setting offset and increasing the reference count.

Add an assertion to make sure the caller really holds the lock. The function can be called outside of coroutine context, where bdrv_pread and flushes become synchronous operations. The lock cannot and need not be taken in this case.

---
  block/qcow2-cache.c | 5 +++++
  1 file changed, 5 insertions(+)
block/qcow2-cache.c +0 -0
index d29b038a67..75b13dad99 100644
@@ -327,6 +327,9 @@ static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
      int min_lru_index = -1;
 
Quoted 8 lines

block/qcow2-cache.c static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c, · line 332

Quoted 4 lines
block/qcow2-cache.c +3 -0
      assert(offset != 0);
+    if (qemu_in_coroutine()) {
+        qemu_co_mutex_assert_locked(&s->lock);
+    }

that is looking not good to me. If this is really requires lock, we should check for the lock always. In the other hand we could face missed lock out of coroutine.

Quoted 4 lines

As the commit message explains, outside of coroutine context, we can't yield and bdrv_pread and bdrv_flush become synchronous operations instead, so there is nothing else that we need to protect against.

Quoted 5 lines

Recently we discussed similar problems about block-dirty-bitmap-* qmp commands, which wanted to update qcow2 metadata about bitmaps from non-coroutine context. "qcow2 lock" <[email protected]>
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg01419.html

Hm, right, I already forgot about the nested event loop again...

Quoted 4 lines

And, as I understand, the correct way is to convert to coroutine and lock mutex appropriately. Or we want to lock aio context and to be in drained section to avoid parallel requests accessing critical section. Should we assert such conditions in case of !qemu_in_coroutine() ?

The AioContext lock must be held anyway, so I don't think this would be a new requirement. As for draining, I'll have to see.

I'm currently still auditing all the callers of qcow2_cache_do_get(). The synchronous callers I already know are the snapshot functions. I think these happen to be in a drained section anyway (or should be at least), so AioContext lock + drain seems like a very reasonable option for them.

For other synchronous callers, if any, maybe conversion to a coroutine would make more sense.

Kevin

author
#15 lore raw

PATCH 3/3 qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

In reply to Kevin Wolf

qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.

As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the qcow2_get_refcount() calls, but around the whole function.

This patch takes the lock in qcow2_co_block_status() earlier and asserts in qcow2_detect_metadata_preallocation() that we hold the lock.

Fixes 69f47505ee66afaa513305de0c1895a224e52c45
---
 block/qcow2-refcount.c | 2 ++
 block/qcow2.c          | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +2 -0
index ef965d7895..0d64bf5a5e 100644
@@ -3455,6 +3455,8 @@ int qcow2_detect_metadata_preallocation(BlockDriverState *bs)
     int64_t i, end_cluster, cluster_count = 0, threshold;
     int64_t file_length, real_allocation, real_clusters;
 
+    qemu_co_mutex_assert_locked(&s->lock);
+
     file_length = bdrv_getlength(bs->file->bs);
     if (file_length < 0) {
         return file_length;
block/qcow2.c +2 -1
index 8b05933565..0bc69e6996 100644
@@ -1916,6 +1916,8 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
     unsigned int bytes;
     int status = 0;
 
+    qemu_co_mutex_lock(&s->lock);
+
     if (!s->metadata_preallocation_checked) {
         ret = qcow2_detect_metadata_preallocation(bs);
         s->metadata_preallocation = (ret == 1);
@@ -1923,7 +1925,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
     }
 
     bytes = MIN(INT_MAX, count);
-    qemu_co_mutex_lock(&s->lock);
     ret = qcow2_get_cluster_offset(bs, offset, &bytes, &cluster_offset);
     qemu_co_mutex_unlock(&s->lock);
     if (ret < 0) {
Signature · 1 line
2.20.1
Reviewed-by on v1 3/3
#16 lore raw

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

Quoted 11 lines · Kevin Wolf

qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.

As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the qcow2_get_refcount() calls, but around the whole function.

Quoted 7 lines

This patch takes the lock in qcow2_co_block_status() earlier and asserts in qcow2_detect_metadata_preallocation() that we hold the lock.

Fixes 69f47505ee66afaa513305de0c1895a224e52c45

Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(

Quoted 40 lines
---
  block/qcow2-refcount.c | 2 ++
  block/qcow2.c          | 3 ++-
  2 files changed, 4 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +2 -0
index ef965d7895..0d64bf5a5e 100644
@@ -3455,6 +3455,8 @@ int qcow2_detect_metadata_preallocation(BlockDriverState *bs)
      int64_t i, end_cluster, cluster_count = 0, threshold;
      int64_t file_length, real_allocation, real_clusters;
 
+    qemu_co_mutex_assert_locked(&s->lock);
+
      file_length = bdrv_getlength(bs->file->bs);
      if (file_length < 0) {
          return file_length;
block/qcow2.c +2 -1
index 8b05933565..0bc69e6996 100644
@@ -1916,6 +1916,8 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
      unsigned int bytes;
      int status = 0;
 
+    qemu_co_mutex_lock(&s->lock);
+
      if (!s->metadata_preallocation_checked) {
          ret = qcow2_detect_metadata_preallocation(bs);
          s->metadata_preallocation = (ret == 1);
@@ -1923,7 +1925,6 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
      }
 
      bytes = MIN(INT_MAX, count);
-    qemu_co_mutex_lock(&s->lock);
      ret = qcow2_get_cluster_offset(bs, offset, &bytes, &cluster_offset);
      qemu_co_mutex_unlock(&s->lock);
      if (ret < 0) {
Signature · 2 lines
Best regards,
Vladimir
author
#17 lore raw

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 15 lines
Quoted 14 lines · Kevin Wolf

qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.

As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the qcow2_get_refcount() calls, but around the whole function.

This patch takes the lock in qcow2_co_block_status() earlier and asserts in qcow2_detect_metadata_preallocation() that we hold the lock.

Quoted 7 lines
Quoted 4 lines
Fixes 69f47505ee66afaa513305de0c1895a224e52c45

Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(

Seriously? Is your downstream QEMU so divergent from upstream that you wouldn't notice something like this? I think you really have to improve something there.

I mean, we have a serious data corruptor in the 4.1 release and I wasted days to debug this, and you've been sitting on a fix for months without telling anyone? This isn't a memory leak or something, this kills people's images. It's eating their data.

Modifying an image format driver requires utmost care and I think I'll try to make sure to allow only few qcow2 changes per release in the future. Too many changes were made in too short time, and with too little care, and there are even more qcow2 patches pending. Please check whether these series actually match your downstream code. Anyway, we'll tread very carefully now with the pending patches, even if it means delaying them for another release or two. Stability is way more important for an image format driver than new features and optimisations.

Do whatever you need to fix your downstream process, but seriously, this must not ever happen again.

Kevin

#18 lore raw

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

Quoted 21 lines · Kevin Wolf

Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 20 lines
Quoted 18 lines · Kevin Wolf

qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the same time, for different tables at different offsets, resulting in image corruption.

As it would be preferable to base the detection on consistent data (even though it's just heuristics), let's take the lock not only around the qcow2_get_refcount() calls, but around the whole function.

This patch takes the lock in qcow2_co_block_status() earlier and asserts in qcow2_detect_metadata_preallocation() that we hold the lock.

Fixes 69f47505ee66afaa513305de0c1895a224e52c45
Quoted 6 lines
Quoted 2 lines

Oh, I'm very sorry. I was going to backport this patch, and found that it's fixed in our downstream long ago, even before last upstream version patch sent :(

Seriously? Is your downstream QEMU so divergent from upstream that you wouldn't notice something like this? I think you really have to improve something there.

How to improve it? Months and years are passed to bring patches into upstream, of course we have to live with a bunch (now it's nearer to 300 and this is a lot better then 500+ in the recent past) of patches above Rhel release.

Quoted 4 lines

I mean, we have a serious data corruptor in the 4.1 release and I wasted days to debug this, and you've been sitting on a fix for months without telling anyone?

Of course, it was not my goal to hide the fix, I'll do my best to avoid this in future. Very sorry for your time wasted.

Quoted 2 lines

This isn't a memory leak or something, this kills
people's images. It's eating their data.

Possibly, I didn't know about data corruption. For some reason I decided that lock should be taken here, I don't remember. Still I should have applied it to upstream version too, of course.

Quoted 6 lines

Modifying an image format driver requires utmost care and I think I'll try to make sure to allow only few qcow2 changes per release in the future. Too many changes were made in too short time, and with too little care, and there are even more qcow2 patches pending. Please check whether these series actually match your downstream code.

The difference is that I didn't move the lock() but add separate lock()/unlock() pair around qcow2_detect_metadata_preallocation(). I think there is no serious difference.

Quoted 3 lines

Anyway, we'll
tread very carefully now with the pending patches, even if it means delaying them for another release or two.

What do you mean? What to do with sent patches during several months?

Quoted 3 lines

Stability is way more
important for an image format driver than new features and optimisations.

Agree. But I think, the main source of stability is testing.

Quoted 4 lines

Do whatever you need to fix your downstream process, but seriously, this must not ever happen again.

I don't see how to improve the process to exclude such problems. But I'll of course will do my best to avoid them.

Signature · 2 lines
Best regards,
Vladimir