make sure disk writes actually hit disk

This is the simple approach to making sure that disk writes actually hit disk before we tell the guest OS that IO has completed. Thanks to DMA_MULTI_THREAD the performance still seems to be adequate.

This is the simple approach to making sure that disk writes actually hit disk before we tell the guest OS that IO has completed. Thanks to DMA_MULTI_THREAD the performance still seems to be adequate.

A fancier solution would be to make the sync/non-sync behaviour of the qemu disk backing store tunable from the guest OS, by tuning the IDE disk write cache on/off with hdparm, and having hw/ide.c call ->fsync functions in the block backends.

I'm willing to code up the fancy solution if people prefer that.

Right now Fabrice is working on rewriting the block API to be asynchronous. There's been quite a lot of discussion about why using threads isn't a good idea for this

With a proper async API, is there any reason why we would want this to be tunable? I don't think there's much of a benefit of prematurely claiming a write is complete especially once the SCSI emulation can support multiple simultaneous requests.

Quoted 4 lines

With a proper async API, is there any reason why we would want this to be tunable? I don't think there's much of a benefit of prematurely claiming a write is complete especially once the SCSI emulation can support multiple simultaneous requests.

You're right. This O_SYNC bandaid should probably stay in place to prevent data corruption, until the AIO framework is ready to be used.

Quoted 7 lines

Quoted 4 lines

With a proper async API, is there any reason why we would want this to be tunable? I don't think there's much of a benefit of prematurely claiming a write is complete especially once the SCSI emulation can support multiple simultaneous requests.

You're right. This O_SYNC bandaid should probably stay in place to prevent data corruption, until the AIO framework is ready to be used.

It's arguable whether O_SYNC is needed at all. Qemu doesn't claim data is written to disk, and provides facilities for the guest OS to flush the cache, just like real hardware does.

Have you measured the impact of O_SYNC? I wouldn't be surprised if it was significant.

Quoted 2 lines

Have you measured the impact of O_SYNC? I wouldn't be surprised if it was significant.

I suspect it'll be horrific in the qemu codebase (blocking execution of the guest OS until disk IO is complete), but it's fine in the Xen qemu-dm situation, where IO completion happens asynchronously.

The recent commit message on the Xen side did not suggest there was that much of a difference between both qemu code bases. Obviously I was wrong, and the O_SYNC bandaid should probably be kept out for now.

This is the simple approach to making sure that disk writes actually hit disk before we tell the guest OS that IO has completed. Thanks to DMA_MULTI_THREAD the performance still seems to be adequate.

A fancier solution would be to make the sync/non-sync behaviour of the qemu disk backing store tunable from the guest OS, by tuning the IDE disk write cache on/off with hdparm, and having hw/ide.c call ->fsync functions in the block backends.

I'm willing to code up the fancy solution if people prefer that.

------------------------------------------------------------------------

Make sure disk writes really made it to disk before we report I/O completion to the guest domain. The DMA_MULTI_THREAD functionality from the qemu-dm IDE emulation should make the performance overhead of synchronous writes bearable, or at least comparable to native hardware.

Signed-off-by Rik van Riel <[email protected]>

tools/ioemu/block-bochs.c renamed from tools/ioemu/block-bochs.c.osync +1 -1

@@ -91,7 +91,7 @@
     int fd, i;
     struct bochs_header bochs;
 
-    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE);
+    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE | O_SYNC);
     if (fd < 0) {
         fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
         if (fd < 0)

tools/ioemu/block.c renamed from tools/ioemu/block.c.osync +1 -1

@@ -677,7 +677,7 @@
     int rv;
 #endif
 
-    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE);
+    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE | O_SYNC);
     if (fd < 0) {
         fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
         if (fd < 0)

tools/ioemu/block-cloop.c renamed from tools/ioemu/block-cloop.c.osync +1 -1

@@ -55,7 +55,7 @@
     BDRVCloopState *s = bs->opaque;
     uint32_t offsets_size,max_compressed_block_size=1,i;
 
-    s->fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
+    s->fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE | O_SYNC);
     if (s->fd < 0)
         return -1;
     bs->read_only = 1;

tools/ioemu/block-cow.c renamed from tools/ioemu/block-cow.c.osync +1 -1

@@ -69,7 +69,7 @@
     struct cow_header_v2 cow_header;
     int64_t size;
 
-    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE);
+    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE | O_SYNC);
     if (fd < 0) {
         fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
         if (fd < 0)

tools/ioemu/block-qcow.c renamed from tools/ioemu/block-qcow.c.osync +1 -1

@@ -95,7 +95,7 @@
     int fd, len, i, shift;
     QCowHeader header;
 
-    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE);
+    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE | O_SYNC);
     if (fd < 0) {
         fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
         if (fd < 0)

tools/ioemu/block-vmdk.c renamed from tools/ioemu/block-vmdk.c.osync +1 -2

@@ -96,7 +96,7 @@
     uint32_t magic;
     int l1_size;
 
-    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE);
+    fd = open(filename, O_RDWR | O_BINARY | O_LARGEFILE | O_SYNC);
     if (fd < 0) {
         fd = open(filename, O_RDONLY | O_BINARY | O_LARGEFILE);
         if (fd < 0)
 
 
------------------------------------------------------------------------

_______________________________________________
Qemu-devel mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/qemu-devel

Hi,

Using O_SYNC for disk image access is not acceptable: QEMU relies on the host OS to ensure that the data is written correctly.

Quoted 4 lines · Fabrice Bellard

Hi,

Using O_SYNC for disk image access is not acceptable: QEMU relies on the host OS to ensure that the data is written correctly.

This means that write ordering is not preserved, and on a power failure any data written by qemu (or Xen fully virt) guests may not be preserved.

Quoted 8 lines · Rik van Riel

Quoted 4 lines · Fabrice Bellard

Hi,

Using O_SYNC for disk image access is not acceptable: QEMU relies on the host OS to ensure that the data is written correctly.

This means that write ordering is not preserved, and on a power failure any data written by qemu (or Xen fully virt) guests may not be preserved.

I might be willing to accept this (or similar) patch if you made it conditional on the guest having disabled write caching. I agree with Fabrice that the performance impact it too severe to consider turning it on by default.

Easy to do with the fsync infrastructure, but probably not worth doing since people are working on the AIO I/O backend, which would allow multiple outstanding writes from a guest. That, in turn, means I/O completion in the guest can be done when the data really hits disk, but without a performance impact.

Quoted 4 lines · Fabrice Bellard

Hi,

Using O_SYNC for disk image access is not acceptable: QEMU relies on the host OS to ensure that the data is written correctly.

This means that write ordering is not preserved, and on a power failure any data written by qemu (or Xen fully virt) guests may not be preserved.

Applications running on the host can count on fsync doing the right thing, meaning that if they call fsync, the data *will* have made it to disk. Applications running inside a guest have no guarantees that their data is actually going to make it anywhere when fsync returns...

This may look like hair splitting, but so far I've lost a (test) postgresql database to this 3 times already. Not getting the guest application's data to disk when the application calls fsync is a recipe for disaster.

--
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan

_______________________________________________
Qemu-devel mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/qemu-devel

This may look like hair splitting, but so far I've lost a (test) postgresql database to this 3 times already. Not getting the guest application's data to disk when the application calls fsync is a recipe for disaster.

How about compromising, and making the patch a run time option. Presumably this is only a problem when the virtual machine is not
properly shutdown. For those ho want the extra security of knowing
the data will be written regardless of the shutdown status they can enable the flag. By default it could be turned off. Then everybody can be happy.

Quoted 3 lines · Anthony Liguori

Right now Fabrice is working on rewriting the block API to be asynchronous. There's been quite a lot of discussion about why using threads isn't a good idea for this

Agreed, AIO is the way to go in the long run.

Quoted 4 lines

With a proper async API, is there any reason why we would want this to be tunable? I don't think there's much of a benefit of prematurely claiming a write is complete especially once the SCSI emulation can support multiple simultaneous requests.

You're right. This O_SYNC bandaid should probably stay in place to prevent data corruption, until the AIO framework is ready to be used.

Quoted 5 lines

Easy to do with the fsync infrastructure, but probably not worth doing since people are working on the AIO I/O backend, which would allow multiple outstanding writes from a guest. That, in turn, means I/O completion in the guest can be done when the data really hits disk, but without a performance impact.

Not entirely true. That only works if you allow multiple guest IO requests in parallel, ie. some form of tagged command queueing. This requires either improving the SCSI emulation, or implementing SATA emulation. AFAIK parallel IDE doesn't support command queueing.

My impression what that the initial AIO implementation is just straight serial async operation. IO wouldn't actually go any faster, it just means the guest can do something else while it's waiting.

Quoted 4 lines · Fabrice Bellard

Hi,

Using O_SYNC for disk image access is not acceptable: QEMU relies on the host OS to ensure that the data is written correctly.

This means that write ordering is not preserved, and on a power failure any data written by qemu (or Xen fully virt) guests may not be preserved.

Applications running on the host can count on fsync doing the right thing, meaning that if they call fsync, the data *will* have made it to disk. Applications running inside a guest have no guarantees that their data is actually going to make it anywhere when fsync returns...

Quoted 5 lines

Applications running on the host can count on fsync doing the right thing, meaning that if they call fsync, the data *will* have made it to disk. Applications running inside a guest have no guarantees that their data is actually going to make it anywhere when fsync returns...

Then the guest OS is broken.

Quoted 6 lines · Jens Axboe

Quoted 5 lines

Applications running on the host can count on fsync doing the right thing, meaning that if they call fsync, the data *will* have made it to disk. Applications running inside a guest have no guarantees that their data is actually going to make it anywhere when fsync returns...

Quoted 1 line

Then the guest OS is broken.

The problem is that supposedly many OS'es are broken in this way. See
http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html

Quoted 4 lines · Rik van Riel

This may look like hair splitting, but so far I've lost a (test) postgresql database to this 3 times already. Not getting the guest application's data to disk when the application calls fsync is a recipe for disaster.

Exactly the same thing happens with real IDE disks if IDE write caching (on the drive itself) is enabled, which it is by default. It is rarer, but it happens.

I've seen this with Linux 2.4 kernels writing to ext3 (real, not virtual). Filesystem metadata gets corrupted from time to time if power is removed, because write ordering is not preserved. Disabling IDE write caching fixes it, but the performance impact is huge on some systems.

Linux 2.6 kernels will issue IDE cache flush commands, at least with ext3, to commit data to disk when fsync is called, and to preserve journal/metadata ordering.

Doesn't qemu fsync the host file corresponding to the emulated disk, when the guest OS issues an IDE cache flush?

For IDE emulation to be as reliable for data storage as a real disk, it should:

- fsync the host file whenever the guest OS issues an IDE cache flush command.

- use O_SYNC (or fsync after each write or aio equivalent, etc.) _only_ when the guest OS disables the IDE disk cache (not done by default).

-- JAmie

_______________________________________________
Qemu-devel mailing list
[email protected]
http://lists.nongnu.org/mailman/listinfo/qemu-devel

Quoted 9 lines · Jamie Lokier

Quoted 4 lines · Rik van Riel

This may look like hair splitting, but so far I've lost a (test) postgresql database to this 3 times already. Not getting the guest application's data to disk when the application calls fsync is a recipe for disaster.

Exactly the same thing happens with real IDE disks if IDE write caching (on the drive itself) is enabled, which it is by default. It is rarer, but it happens.

The little difference with QEMU is that there are two caches above it:
the host OS'es software cache and the IDE hardware cache. When a guest OS flushes its own software cache its precious data goes to the host's software cache while the guest thinks it's already the IDE cache. This is ofcourse of less importance because data in both caches (hard- and software) is lost when the power is cut off.

IMHO what really makes IO unreliable in QEMU is that IO errors on the host are not reported to the guest by the IDE emulation and there's an exact place in hw/ide.c where they are arrogantly ignored.

Quoted 11 lines · andrzej zaborowski

Quoted 9 lines · Jamie Lokier

Quoted 4 lines · Rik van Riel

This may look like hair splitting, but so far I've lost a (test) postgresql database to this 3 times already. Not getting the guest application's data to disk when the application calls fsync is a recipe for disaster.

Exactly the same thing happens with real IDE disks if IDE write caching (on the drive itself) is enabled, which it is by default. It is rarer, but it happens.

Quoted 6 lines

The little difference with QEMU is that there are two caches above it:
the host OS'es software cache and the IDE hardware cache. When a guest OS flushes its own software cache its precious data goes to the host's software cache while the guest thinks it's already the IDE cache. This is ofcourse of less importance because data in both caches (hard- and software) is lost when the power is cut off.

But the drive cache does not let the dirty data linger for as long as wht OS page/buffer cache.

Quoted 4 lines

IMHO what really makes IO unreliable in QEMU is that IO errors on the host are not reported to the guest by the IDE emulation and there's an exact place in hw/ide.c where they are arrogantly ignored.

Send a patch, I'm pretty sure nobody would disagree :-)

--
Jens Axboe