data integrity fixes

We would have to claim one for data=writeback to be safe, but for I will follow Avi's opinion that it is a useless mode and should be our dedicated unsafe mode. If anyone disagrees please start the flame thrower now and I will change it. Otherwise a documentation patch will follow to explicitly document data=writeback as unsafe.

Right now, on a Linux host O_SYNC is unsafe with hardware that has a volatile write cache. That might not be changed, but if it is than performance with cache=writethrough will plummet (due to issuing a CACHE FLUSH to the hardware after every write), while performance with cache=writeback will be reasonable.

If an unsafe mode is desired (I think it is, for those throwaway testing VMs, or during OS installs), I suggest adding cache=volatile:

cache=none
O_DIRECT, fdatasync, advertise volatile write cache

cache=writethrough
O_SYNC, do not advertise

cache=writeback
fdatasync, advertise volatile write cache

cache=volatile
nothing (perhaps fdatasync on QEMU blockdev close)

When using guests OSes which issue CACHE FLUSH commands (that's a guest config issue), why would you ever use cache=writethrough? cache=writeback should be faster and equally safe - provided you do actually advertise the write cache!

Quoted 5 lines · Jamie Lokier

Right now, on a Linux host O_SYNC is unsafe with hardware that has a volatile write cache. That might not be changed, but if it is than performance with cache=writethrough will plummet (due to issuing a CACHE FLUSH to the hardware after every write), while performance with cache=writeback will be reasonable.

Currenly all modes are more or less unsafe with volatile write caches at least when using ext3 or raw block device accesses. XFS is safe two thirds due to doing the right thing and one third due to sheer luck.

Quoted 1 line

nothing (perhaps fdatasync on QEMU blockdev close)

Fine withe me, let the flame war begin :)

Quoted 4 lines

When using guests OSes which issue CACHE FLUSH commands (that's a guest config issue), why would you ever use cache=writethrough? cache=writeback should be faster and equally safe - provided you do actually advertise the write cache!

And provided the guest OS actually issues cache flushes when it should, something that at least Linux historicaly utterly failed at, and some other operating systems haven't even tried.

cache=writethrough is the equivalent of turning off the volatile write cache of the real disk. It might be slower (which isn't even always the case for real disks), but it is much safer.

E.g. if you want to move your old SCO Unix box into a VM it's the only safe option.

Quoted 11 lines · Jamie Lokier

Quoted 5 lines

Right now, on a Linux host O_SYNC is unsafe with hardware that has a volatile write cache. That might not be changed, but if it is than performance with cache=writethrough will plummet (due to issuing a CACHE FLUSH to the hardware after every write), while performance with cache=writeback will be reasonable.

Currenly all modes are more or less unsafe with volatile write caches at least when using ext3 or raw block device accesses. XFS is safe two thirds due to doing the right thing and one third due to sheer luck.

Right, but now you've made it worse. By not calling fdatasync at all, you've reduced the integrity. Previously it would reach the drive's cache, and take whatever (short) time it took to reach the platter. Now you're leaving data in the host cache which can stay for much longer, and is vulnerable to host kernel crashes.

Oh, and QEMU could call whatever "hdparm -F" does when using raw block devices ;-)

Well I'd like to start by pointing out your patch introduces a regression in the combination cache=writeback with emulated SCSI, because it effectively removes the fdatasync calls in that case :-)

It goes to show no matter how hard we try, data integrity is a slippery thing where getting it wrong does not show up under normal circumstances, only during catastrophic system failures.

It failed with fsync, which
is also important to applications, but filesystem integrity is the most important thing and it's been good at that for many years.

Quoted 2 lines

E.g. if you want to move your old SCO Unix box into a VM it's the only safe option.

I agree, and for that reason, cache=writethrough or cache=none are the only reasonable defaults.

Quoted 2 lines

Oh, and QEMU could call whatever "hdparm -F" does when using raw block devices ;-)

Actually for ide/scsi implementing cache control is on my todo list. Not sure about virtio yet.

Quoted 3 lines

It goes to show no matter how hard we try, data integrity is a slippery thing where getting it wrong does not show up under normal circumstances, only during catastrophic system failures.

Honestly, it should not. Digging through all this was a bit of work, but I was extremly how carelessly most people that touched it before were. It's not rocket sciense and can be tested quite easily using various tools - qemu beeing the easiest nowdays but scsi_debug or an instrumented iscsi target would do the same thing.

Quoted 3 lines

It failed with fsync, which is also important to applications, but filesystem integrity is the most important thing and it's been good at that for many years.

Users might disagree with that. With my user hat on I couldn't care less on what state the internal metadata is as long as I get back at my data which the OS has guaranteed me to reach the disk after a successfull fsync/fdatasync/O_SYNC write.

Quoted 5 lines

Quoted 2 lines

E.g. if you want to move your old SCO Unix box into a VM it's the only safe option.

I agree, and for that reason, cache=writethrough or cache=none are the only reasonable defaults.

despite the extremly misleading name cache=none is _NOT_ an alternative, unless we make it open the image using O_DIRECT|O_SYNC.

Quoted 9 lines

If an unsafe mode is desired (I think it is, for those throwaway testing VMs, or during OS installs), I suggest adding cache=volatile:

cache=none
O_DIRECT, fdatasync, advertise volatile write cache

cache=writethrough
O_SYNC, do not advertise

Quoted 6 lines

cache=writeback
fdatasync, advertise volatile write cache

cache=volatile
nothing (perhaps fdatasync on QEMU blockdev close)

Fine withe me, let the flame war begin :)

Quoted 9 lines · Christoph Hellwig

Quoted 9 lines

If an unsafe mode is desired (I think it is, for those throwaway testing VMs, or during OS installs), I suggest adding cache=volatile:

cache=none
O_DIRECT, fdatasync, advertise volatile write cache

cache=writethrough
O_SYNC, do not advertise

Quoted 9 lines

Quoted 6 lines

cache=writeback
fdatasync, advertise volatile write cache

cache=volatile
nothing (perhaps fdatasync on QEMU blockdev close)

Fine withe me, let the flame war begin :)

I think we should pity our poor users and avoid adding yet another obscure option that is likely to be misunderstood.

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

I think the two reasonable options are 1) make cache=writeback safe, avoid a massive perf decrease in the process 2) keep cache=writeback as a no-guarantees option.

I think we should pity our poor users and avoid adding yet another obscure option that is likely to be misunderstood.

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Quoted 2 lines · Anthony Liguori

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Unfortunately we can't yet quantify the impact on the hardware I care about (ordinary consumer PCs with non-NCQ SATA disks), because Linux hosts don't *yet* implement O_SYNC or fdatasync properly.

Quoted 7 lines · Jamie Lokier

Quoted 2 lines · Anthony Liguori

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Unfortunately we can't yet quantify the impact on the hardware I care about (ordinary consumer PCs with non-NCQ SATA disks), because Linux hosts don't *yet* implement O_SYNC or fdatasync properly.

They do if you use XFS.

I think we should pity our poor users and avoid adding yet another obscure option that is likely to be misunderstood.

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Quoted 4 lines · Anthony Liguori

I think we should pity our poor users and avoid adding yet another obscure option that is likely to be misunderstood.

Can someone do some benchmarking with cache=writeback and fdatasync first and quantify what the real performance impact is?

Some preliminary numbers because they are very interesting. Note that his is on a raid controller, not cheap ide disks. To make up for that I used an image file on ext3, which due to it's horrible fsync performance should be kind of a worst case. All these patches are with Linux 2.6.31-rc8 + my various barrier fixes on guest and host, using ext3 with barrier=1 on both.

A kernel defconfig compile takes between 9m40s and 9m42s with data=writeback and barrieres disabled, and with fdatasync barriers enabled it actually is minimally faster,

between 9m38s and 9m39s
(given that I've only done three runs each this might fall under the boundary for measurement tolerances).

For comparism the raw block device nodes with cache=none (just one run) is 9m36.759s, which is not far apart. A completely native run is 7m39.326, btw - and I fear much of the slowdown in KVM isn't I/O related.

Does barrier=0 make a performance difference? IOW, would the typical default ext3 deployment show worse behavior?

Quoted 3 lines

A kernel defconfig compile takes between 9m40s and 9m42s with data=writeback and barrieres disabled, and with fdatasync barriers enabled it actually is minimally faster,

If fdatasync different than fsync on ext3? Does it result in a full metadata commit?

Quoted 4 lines

performance should be kind of a worst case. All these patches are with Linux 2.6.31-rc8 + my various barrier fixes on guest and host, using ext3 with barrier=1 on both.

Does barrier=0 make a performance difference? IOW, would the typical default ext3 deployment show worse behavior?

Quoted 5 lines · Christoph Hellwig

If we are flushing the caches for our image files we only care about the data (including the metadata required for accessing it) but not things like timestamp updates. So use fdatasync instead of fsync to implement the flush operations.

Review

Quoted 2 lines

(quoted diff) +1 -1

-    fsync(s->fd);
+    fdatasync(s->fd);

I believe fsync was used because of uncertainty about whether fdatasync reliably flushes the necessary metadata to access the data on all hosts, after things like writing to holes and extending a file.

I'm still not sure if fdatasync provides that guarantee on Linux.

Quoted 14 lines · Jamie Lokier

Review

Quoted 8 lines · Christoph Hellwig

If we are flushing the caches for our image files we only care about the data (including the metadata required for accessing it) but not things like timestamp updates. So use fdatasync instead of fsync to implement the flush operations.

(quoted diff) +1 -1

-    fsync(s->fd);
+    fdatasync(s->fd);

I believe fsync was used because of uncertainty about whether fdatasync reliably flushes the necessary metadata to access the data on all hosts, after things like writing to holes and extending a file.

Quoted 1 line

I'm still not sure if fdatasync provides that guarantee on Linux.

fdatasync is defined to provide that guarantee, and modulo the whole clusterfuck around volatile write caches it does the right thing thing on common Linux filesystems.

Quoted 3 lines

fdatasync is defined to provide that guarantee, and modulo the whole clusterfuck around volatile write caches it does the right thing thing on common Linux filesystems.

That's good to know. I'm not sure about other hosts though. It's specified to update the metadata, but I think I've encountered people saying it doesn't - specifically when extending a file, that it doesn't guarantee the file size is extended on some OSes. But that might be an urban myth.

If we are flushing the caches for our image files we only care about the data (including the metadata required for accessing it) but not things like timestamp updates.  So use fdatasync instead of fsync to implement the flush operations.

-    fsync(s->fd);
+    fdatasync(s->fd);

Quoted 5 lines · Christoph Hellwig

If we are flushing the caches for our image files we only care about the data (including the metadata required for accessing it) but not things like timestamp updates. ??So use fdatasync instead of fsync to implement the flush operations.

Quoted 2 lines

- ?? ??fsync(s->fd);
+ ?? ??fdatasync(s->fd);

There is no fdatasync outside of Linux, not in OpenBSD and mingw32 at least. But you could add a probe for it to configure.

Quoted 7 lines · Blue Swirl

Quoted 5 lines · Christoph Hellwig

If we are flushing the caches for our image files we only care about the data (including the metadata required for accessing it) but not things like timestamp updates. ??So use fdatasync instead of fsync to implement the flush operations.

Quoted 5 lines

Quoted 2 lines

- ?? ??fsync(s->fd);
+ ?? ??fdatasync(s->fd);

There is no fdatasync outside of Linux, not in OpenBSD and mingw32 at least. But you could add a probe for it to configure.

Oh, okay. Kinda sad that people still don't implement useful inteface that have been in Posix for a long time.

fdatasync is common among the big commercial unixes which were still being actively developed a few years ago, but fsync is universal - it's very old.

Neither OpenBSD nor FreeBSD have fdatasync.

mingw32 is a thin wrapper around Windows. Windows as far as I can tell doesn't have an equivalent of fdatasync, although it does have an equivalent of O_DIRECT|O_SYNC (but documentation contradicts itself regarding whether the combination flushes metadata on each write).

Quoted 10 lines · Jamie Lokier

fdatasync is common among the big commercial unixes which were still being actively developed a few years ago, but fsync is universal - it's very old.

Neither OpenBSD nor FreeBSD have fdatasync.

mingw32 is a thin wrapper around Windows. Windows as far as I can tell doesn't have an equivalent of fdatasync, although it does have an equivalent of O_DIRECT|O_SYNC (but documentation contradicts itself regarding whether the combination flushes metadata on each write).

Yeah, I'll just add a

#ifndef _POSIX_SYNCHRONIZED_IO

# define fdatasync(fd)	fsync(fd)
#endif

Now I'm pretty sure we'll find various fuckups where headers define fdatasync but not _POSIX_SYNCHRONIZED_IO or vice versa, but I can outsource adding workarounds for that to people with those broken setups.

Quoted 10 lines · Jamie Lokier

fdatasync is common among the big commercial unixes which were still being actively developed a few years ago, but fsync is universal - it's very old.

Neither OpenBSD nor FreeBSD have fdatasync.

mingw32 is a thin wrapper around Windows.  Windows as far as I can tell doesn't have an equivalent of fdatasync, although it does have an equivalent of O_DIRECT|O_SYNC (but documentation contradicts itself regarding whether the combination flushes metadata on each write).

Yeah,  I'll just add a

#ifndef _POSIX_SYNCHRONIZED_IO

# define fdatasync(fd)  fsync(fd)
#endif

Now I'm pretty sure we'll find various fuckups where headers define fdatasync but not _POSIX_SYNCHRONIZED_IO or vice versa, but I can outsource adding workarounds for that to people with those broken setups.

Quoted 4 lines

Now I'm pretty sure we'll find various fuckups where headers define fdatasync but not _POSIX_SYNCHRONIZED_IO or vice versa, but I can outsource adding workarounds for that to people with those broken setups.

I'd use a real function, like qemu_fdatasync(). Especially if the Windows equivalent will be complex.

Instead stalling the VCPU while serving a cache flush try to do it asynchronously. Use our good old helper thread pool to issue an asynchronous fdatasync for raw-posix. Note that while Linux AIO implements a fdatasync operation it is not useful for us because it isn't actually implement in asynchronous fashion.

For now only use it in IDE because virtio-blk doesn't implement cache flusing yet (will be fixed in patch 4/4) and the interface between the HBA emulation and scsi-disk will need some changes to accomodate it for scsi (will be a separate patch series).

Doesn't aio_flush() need to wait for all previously issued requests to complete?

We can require callers to do that, but it seems quite a burden.

Patch has some tabs.