Skip to contentSkip to conversation

qcow2: Bring synchronous read/write back to life

PATCH In review Kevin Wolf 4 participants 8 messages Oct 8, 2009 Block layer

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

PATCH 1/1 qcow2: Bring synchronous read/write back to life

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read. Which in turn means that (meta)data read earlier could have changed and be invalid now. qcow2 is not prepared to work in this way and it's just scary how many places there are where other requests could run.

I'm not sure yet where exactly it breaks, but you'll see breakage with virtio on qcow2 with a backing file. Providing synchronous functions again fixes the problem for me.

---
 block/qcow2-cluster.c |    6 ++--
 block/qcow2.c         |   51 +++++++++++++++++++++++++++++++++++++++++++++++-
 block/qcow2.h         |    3 ++
 3 files changed, 55 insertions(+), 5 deletions(-)
block/qcow2-cluster.c +3 -3
index e444e53..a7de820 100644
@@ -306,8 +306,8 @@ void qcow2_encrypt_sectors(BDRVQcowState *s, int64_t sector_num,
 }
 
 
-static int qcow_read(BlockDriverState *bs, int64_t sector_num,
-                     uint8_t *buf, int nb_sectors)
+int qcow2_read(BlockDriverState *bs, int64_t sector_num, uint8_t *buf,
+    int nb_sectors)
 {
     BDRVQcowState *s = bs->opaque;
     int ret, index_in_cluster, n, n1;
@@ -358,7 +358,7 @@ static int copy_sectors(BlockDriverState *bs, uint64_t start_sect,
     n = n_end - n_start;
     if (n <= 0)
         return 0;
-    ret = qcow_read(bs, start_sect + n_start, s->cluster_data, n);
+    ret = qcow2_read(bs, start_sect + n_start, s->cluster_data, n);
     if (ret < 0)
         return ret;
     if (s->crypt_method) {
block/qcow2.c +49 -2

63 lines not on this page · raw message

block/qcow2.h +3 -0
index 26ab5d9..d3f690a 100644
@@ -202,6 +202,9 @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
 int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, uint64_t cluster_offset,
     QCowL2Meta *m);
 
+int qcow2_read(BlockDriverState *bs, int64_t sector_num, uint8_t *buf,
+    int nb_sectors);
+
 /* qcow2-snapshot.c functions */
 int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info);
 int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id);
Signature · 1 line
1.6.2.5
#1 lore raw
Quoted 9 lines · Kevin Wolf

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read.

Perhaps you could create a mechanism to freeze the qcow2 image by queuing all completions within qcow2 until the image was unfrozen. This would have the same effect switching to synchronous read/write.

You may also have to queue new read/write requests...

Introducing sync read/write seems like a major step backwards to me.

Regards,

Anthony Liguori

author
#2 lore raw

Am 08.10.2009 16:30, schrieb Anthony Liguori:

Quoted 11 lines
Quoted 9 lines · Kevin Wolf

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read.

Quoted 7 lines

Perhaps you could create a mechanism to freeze the qcow2 image by queuing all completions within qcow2 until the image was unfrozen. This would have the same effect switching to synchronous read/write.

You may also have to queue new read/write requests...

Introducing sync read/write seems like a major step backwards to me.

Right, I was expecting your reaction. ;-) I do even agree that it's not nice to have the synchronous functions back. But removing them caused a regression, so the removal should be reverted until it is done right.

I just want to make clear that we're talking about data corruption here. This is not just something that we can care about when we are bored some time in the future.

For stable, I think taking this patch is the only reasonable thing to do. Any other solution would be way too invasive. For master we can discuss other solutions. However, I really don't feel like hacking around it with a quick fix and breaking other stuff, so this takes a bit more time. For the meantime I would prefer committing this to master, too.

By the way, I don't think queuing things in qcow2 is the right thing to do. It probably wouldn't even help if, say, VMDK implemented AIO and I used a VMDK image as a backing file for qcow2. The real solution is to fix the generic bdrv_read/write emulation. Now we just need to find the best way to do this.

Kevin

#3 lore raw
Quoted 21 lines · Kevin Wolf

Am 08.10.2009 16:30, schrieb Anthony Liguori:

Quoted 12 lines
Quoted 10 lines · Kevin Wolf

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read.

Quoted 6 lines

Perhaps you could create a mechanism to freeze the qcow2 image by queuing all completions within qcow2 until the image was unfrozen. This would have the same effect switching to synchronous read/write.

You may also have to queue new read/write requests...

Introducing sync read/write seems like a major step backwards to me.

Quoted 8 lines

Right, I was expecting your reaction. ;-) I do even agree that it's not nice to have the synchronous functions back. But removing them caused a regression, so the removal should be reverted until it is done right.

I just want to make clear that we're talking about data corruption here. This is not just something that we can care about when we are bored some time in the future.

Yeah, okay. Can we do a more direct revert though so that it's clearer in the commit log?

Quoted 6 lines

By the way, I don't think queuing things in qcow2 is the right thing to do. It probably wouldn't even help if, say, VMDK implemented AIO and I used a VMDK image as a backing file for qcow2. The real solution is to fix the generic bdrv_read/write emulation. Now we just need to find the best way to do this.

Ideally, you could say qemu_aio_wait(aiocb) and it would only wait for that particular request. I'm not convinced though that there aren't dependency issues though since we can generate synthetic aiocbs.

Regards,

Anthony Liguori

author
#4 lore raw

Am 08.10.2009 17:01, schrieb Anthony Liguori:

Quoted 28 lines
Quoted 27 lines · Kevin Wolf

Am 08.10.2009 16:30, schrieb Anthony Liguori:

Quoted 12 lines
Quoted 10 lines · Kevin Wolf

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read.

Quoted 8 lines

Perhaps you could create a mechanism to freeze the qcow2 image by queuing all completions within qcow2 until the image was unfrozen. This would have the same effect switching to synchronous read/write.

You may also have to queue new read/write requests...

Introducing sync read/write seems like a major step backwards to me.

Right, I was expecting your reaction. ;-) I do even agree that it's not nice to have the synchronous functions back. But removing them caused a regression, so the removal should be reverted until it is done right.

Quoted 7 lines
Quoted 4 lines

I just want to make clear that we're talking about data corruption here. This is not just something that we can care about when we are bored some time in the future.

Yeah, okay. Can we do a more direct revert though so that it's clearer in the commit log?

Hm, it's not a single commit to revert. The main part is that we need qcow_write back which was removed in ade40677. The removal of the synchronous functions from the BlockDriver must have happened longer ago.

And then I needed to put one or two changes on top to make it work with the changes since it was dropped.

Quoted 9 lines
Quoted 5 lines

By the way, I don't think queuing things in qcow2 is the right thing to do. It probably wouldn't even help if, say, VMDK implemented AIO and I used a VMDK image as a backing file for qcow2. The real solution is to fix the generic bdrv_read/write emulation. Now we just need to find the best way to do this.

Ideally, you could say qemu_aio_wait(aiocb) and it would only wait for that particular request. I'm not convinced though that there aren't dependency issues though since we can generate synthetic aiocbs.

You can't only wait for this single request but you also need to allow waiting for all requests that were submitted while processing the request you're waiting for. I think you need to create something like a new context for all requests that are newly started.

Kevin

#5 lore raw
In reply to Anthony Liguori
Quoted 28 lines · Anthony Liguori
Quoted 27 lines · Kevin Wolf

Am 08.10.2009 16:30, schrieb Anthony Liguori:

Quoted 12 lines
Quoted 10 lines · Kevin Wolf

When the synchronous read and write functions were dropped, they were replaced by generic emulation functions. Unfortunately, these emulation functions don't provide the same semantics as the original functions did.

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read.

Quoted 8 lines

Perhaps you could create a mechanism to freeze the qcow2 image by queuing all completions within qcow2 until the image was unfrozen. This would have the same effect switching to synchronous read/write.

You may also have to queue new read/write requests...

Introducing sync read/write seems like a major step backwards to me.

Right, I was expecting your reaction. ;-) I do even agree that it's not nice to have the synchronous functions back. But removing them caused a regression, so the removal should be reverted until it is done right.

Quoted 7 lines
Quoted 4 lines

I just want to make clear that we're talking about data corruption here. This is not just something that we can care about when we are bored some time in the future.

Yeah, okay. Can we do a more direct revert though so that it's clearer in the commit log?

FWIW, here's the Fedora 12 (qemu-kvm-0.11.0) report on this:

https://bugzilla.redhat.com/524734

Cheers,
Mark.

#6 lore raw
In reply to Kevin Wolf
Quoted 7 lines · Kevin Wolf

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read. Which in turn means that (meta)data read earlier could have changed and be invalid now.

I'm not sure if I understand your description, specifically "(meta)data read earlier could have changed and be invalid now".

Do you mean:

Async call into qcow2 #2
------------------------

issues a request with bdrv_read/write

Async call into qcow2 #1
------------------------
reads some metadata from memory (**)
does some calculations
issues a request with bdrv_read/write

the request completes updates some metadata in memory async call finished with result

the request completes
updates some metadata in memory
.... ERROR, memory isn't what it was at point (**)

Thanks,
-- Jamie

author
#7 lore raw

Am 08.10.2009 17:28, schrieb Jamie Lokier:

Quoted 9 lines
Quoted 7 lines · Kevin Wolf

The original bdrv_read would mean that we read some data synchronously and that we won't be interrupted during this read. The latter assumption is no longer true with the emulation function which needs to use qemu_aio_poll and therefore allows the callback of any other concurrent AIO request to be run during the read. Which in turn means that (meta)data read earlier could have changed and be invalid now.

Quoted 4 lines

I'm not sure if I understand your description, specifically "(meta)data read earlier could have changed and be invalid now".

Do you mean:

No, it's not exactly what I meant. But you're right, your scenario could happen, too. If global state is changed in an AIO callback called during bdrv_read/write (e.g. a new L2 table is loaded into the cache), bad things are almost guaranteed to happen.

What I was thinking of is:

Quoted 9 lines

Async call into qcow2 #2
------------------------

issues a request with bdrv_read/write

Async call into qcow2 #1
------------------------
reads some metadata from memory (**)

#1 reads some metadata from disk (from the image file)

Quoted 5 lines

does some calculations
issues a request with bdrv_read/write

the request completes updates some metadata in memory

#2 updates the metadata in the file

Quoted 5 lines

async call finished with result

the request completes
updates some metadata in memory
.... ERROR, memory isn't what it was at point (**)

#1 still uses the old metadata which it had loaded into memory (specifically those parts on the stack).

Also, I was thinking of #2 as being a regular AIO write (no bdrv_write involved), but again your version could work as well. As I said I don't know yet which of them really happens.

Kevin