JFYI: ext4 bug triggerable by kvm
https://bugzilla.kernel.org/show_bug.cgi?id=16165
When a (raw) guest image is placed on an ext4 filesystem, it is possible to get data corruption, now due to ext4 bug, not kvm bug.
Also, ext4 is _very_ slow on O_SYNC writes (which is used in kvm with default cache).
JFYI.
/mjt
Quoted 6 lines · Michael Tokarev
https://bugzilla.kernel.org/show_bug.cgi?id=16165
When a (raw) guest image is placed on an ext4 filesystem, it is possible to get data corruption, now due to ext4 bug, not kvm bug.
Yeah, there appears to be a few O_DIRECT related issues with ext4. AFAIK, a preallocated raw image should be safe though which is probably the only time you should use O_DIRECT.
Quoted 3 lines
Also, ext4 is _very_ slow on
O_SYNCwrites (which is used in kvm with default cache).
Yeah, we probably need to switch to sync_file_range() to avoid the journal commit on every write.
Regards,
Anthony Liguori
Quoted 8 lines
JFYI.
/mjt
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Anthony Liguori
Quoted 2 lines
Also, ext4 is _very_ slow on
O_SYNCwrites (which is used in kvm with default cache).Yeah, we probably need to switch to
sync_file_range()to avoid the journal commit on every write.
No, we don't. sync_file_range does not actually provide any data integrity.
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Anthony Liguori
Quoted 3 lines
Also, ext4 is _very_ slow on
O_SYNCwrites (which is used in kvm with default cache).
Quoted 7 lines
Quoted 4 lines
Yeah, we probably need to switch to
sync_file_range()to avoid the journal commit on every write.No, we don't.
sync_file_rangedoes not actually provide any data integrity.
What do you mean by data integrity?
For each write in cache=writethrough, we don't have to ensure the data is on the platter. We really just need to ensure that the data has been sent to next level in the storage hierarchy and that it has been acknowledged as having been written. We don't need to actually inject a barrier.
My understanding is that on ext4/btrfs, an O_SYNC write injects a barrier for every write which is not the behavior we're looking for. As I understand it, sync_file_range() would give us the above guarantee without the barrier and for explicit barriers, we would use fsync.
Regards,
Anthony Liguori
Quoted 11 lines · Anthony Liguori
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Anthony Liguori
Quoted 2 lines
Also, ext4 is _very_ slow on
O_SYNCwrites (which is used in kvm with default cache).Yeah, we probably need to switch to
sync_file_range()to avoid the journal commit on every write.No, we don't.
sync_file_rangedoes not actually provide any data integrity.What do you mean by data integrity?
sync_file_range only does pagecache-level writeout of the file data. It nevers calls into the actual filesystem, that means any block allocations (for filling holes / converting preallocated space in normal filesystems, or every write in COW-based filesstems like qcow2) never get flushes to disk, and even more importantly the disk write cache is never flushed.
In short it's completely worthless for any real filesystem.
Quoted 3 lines · Christoph Hellwig
In short it's completely worthless for any real filesystem.
The documentation should be updated then. It suggests that it is usable for data integrity.
(or maybe, it should be fixed?)
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 3 lines · Christoph Hellwig
In short it's completely worthless for any real filesystem.
The documentation should be updated then. It suggests that it is usable for data integrity.
The manpage has a "warning" section documenting what I said above since I added it in January.
Quoted 16 lines · Christoph Hellwig
Quoted 14 lines · Anthony Liguori
Quoted 12 lines · Christoph Hellwig
Quoted 7 lines · Anthony Liguori
Quoted 3 lines
Also, ext4 is _very_ slow on
O_SYNCwrites (which is used in kvm with default cache).Yeah, we probably need to switch to
sync_file_range()to avoid the journal commit on every write.No, we don't.
sync_file_rangedoes not actually provide any data integrity.
Quoted 7 lines
Quoted 2 lines
What do you mean by data integrity?
sync_file_rangeonly does pagecache-level writeout of the file data. It nevers calls into the actual filesystem, that means any block allocations (for filling holes / converting preallocated space in normal filesystems, or every write in COW-based filesstems like qcow2) never get flushes to disk,
But assuming that you had a preallocated disk image, it would effectively flush the page cache so it sounds like the only real issue is sparse and growable files.
Quoted 3 lines
and even more importantly the disk write cache is
never flushed.
The point is that we don't want to flush the disk write cache. The intention of writethrough is not to make the disk cache writethrough but to treat the host's cache as writethrough.
Regards,
Anthony Liguori
Quoted 3 lines
In short it's completely worthless for any real filesystem.
Quoted 3 lines · Anthony Liguori
But assuming that you had a preallocated disk image, it would effectively flush the page cache so it sounds like the only real issue is sparse and growable files.
For preallocated as in using fallocate() we still converting unwritten to regular extents and do have metadata updates. For preallocated as in writining zeroes into the whole image earlier we do indeed only care about the data, and will not have metadata for most filesystems. That still leaves COW based filesystems that need to allocate new blocks on every write, and from my reading NFS also needs the ->fsync callout to actually commit unstable data to disk.
Quoted 6 lines
Quoted 2 lines
and even more importantly the disk write cache is
never flushed.The point is that we don't want to flush the disk write cache. The intention of writethrough is not to make the disk cache writethrough but to treat the host's cache as writethrough.
We need to make sure data is not in the disk write cache if want to provide data integrity. It has nothing to do with the qemu caching mode - for data=writeback or none it's commited as part of the fdatasync call, and for data=writethrough it's commited as part of the O_SYNC write. Note that both these path end up calling the filesystems ->fsync method which is what's require to make writes stable. That's exactly what is missing out in sync_file_range, and that's why that API is not useful at all for data integrity operations. It's also what makes fsync slow on extN - but the fix to that is not to not provide data integrity but rather to make fsync fast. There's various other filesystems that can already do it, and if you insist on using those that are slow for this operation you'll have to suffer until that issue is fixed for them.
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines
The point is that we don't want to flush the disk write cache. The intention of writethrough is not to make the disk cache writethrough but to treat the host's cache as writethrough.
We need to make sure data is not in the disk write cache if want to provide data integrity.
When the guest explicitly flushes the emulated disk's write cache. Not on every single write completion.
Quoted 7 lines
It has nothing to do with the qemu caching
mode - for data=writeback or none it's commited as part of the fdatasync call, and for data=writethrough it's commited as part of theO_SYNCwrite. Note that both these path end up calling the filesystems ->fsync method which is what's require to make writes stable. That's exactly what is missing out insync_file_range, and that's why that API is not useful at all for data integrity operations.
For normal writes from a guest, we don't need to follow the write with an fsync(). We should only need to issue an fsync() given an explicit flush from the guest.
Quoted 7 lines
It's also what makes
fsync slow on extN - but the fix to that is not to not provide data integrity but rather to make fsync fast. There's various other filesystems that can already do it, and if you insist on using those that are slow for this operation you'll have to suffer until that issue is fixed for them.
fsync() being slow is orthogonal to my point. I don't see why we need to do an fsync() on *every* write. It should only be necessary when a guest injects an actual barrier.
Regards,
Anthony Liguori
Quoted 10 lines · Anthony Liguori
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines
The point is that we don't want to flush the disk write cache. The intention of writethrough is not to make the disk cache writethrough but to treat the host's cache as writethrough.
We need to make sure data is not in the disk write cache if want to provide data integrity.
When the guest explicitly flushes the emulated disk's write cache. Not on every single write completion.
That depends on the cache= mode. For cache=none and cache=writeback we present a write-back cache to the guest, and the guest does explicit cache flushes. For cache=writethrough we present a writethrough cache to the guest, and we need to make sure data actually has hit the disk before returning I/O completion to the guest.
Quoted 11 lines
Quoted 7 lines
It has nothing to do with the qemu caching
mode - for data=writeback or none it's commited as part of the fdatasync call, and for data=writethrough it's commited as part of theO_SYNCwrite. Note that both these path end up calling the filesystems ->fsync method which is what's require to make writes stable. That's exactly what is missing out insync_file_range, and that's why that API is not useful at all for data integrity operations.For normal writes from a guest, we don't need to follow the write with an
fsync(). We should only need to issue anfsync()given an explicit flush from the guest.
Define normal writes. For cache=none and cache=writeback we don't have to, and instead do explicit calls to fsync()/fdatasync() calls when a we a cache flush from the guest. For data=writethrough we guarantee data has made it to disk, and we implement this using O_DSYNC/O_SYNC when opening the file. That tells the operating system
to not return until data has hit the disk. For Linux this is
internally implement using a range-fsync/fdatasync after the actual write.
Quoted 3 lines
fsync()being slow is orthogonal to my point. I don't see why we need to do anfsync()on *every* write. It should only be necessary when a guest injects an actual barrier.
See above.
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Anthony Liguori
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines
The point is that we don't want to flush the disk write cache. The intention of writethrough is not to make the disk cache writethrough but to treat the host's cache as writethrough.
We need to make sure data is not in the disk write cache if want to provide data integrity.
Quoted 9 lines
Quoted 3 lines
When the guest explicitly flushes the emulated disk's write cache. Not on every single write completion.
That depends on the cache= mode. For cache=none and cache=writeback we present a write-back cache to the guest, and the guest does explicit cache flushes. For cache=writethrough we present a writethrough cache to the guest, and we need to make sure data actually has hit the disk before returning I/O completion to the guest.
Why?
The type of cache we present to the guest only should relate to how the hypervisor caches the storage. It should be independent of how data is cached by the disk.
There can be many levels of caching in a storage hierarchy and each hierarchy cached independently of the next level.
If the user has a disk with a writeback cache, if we expose a writethrough cache to the guest, it's not our responsibility to make sure that we break through the writeback cache on the disk.
Regards,
Anthony Liguori
Quoted 3 lines · Anthony Liguori
The type of cache we present to the guest only should relate to how the hypervisor caches the storage. It should be independent of how data is cached by the disk.
It is.
Quoted 6 lines
There can be many levels of caching in a storage hierarchy and each hierarchy cached independently of the next level.
If the user has a disk with a writeback cache, if we expose a writethrough cache to the guest, it's not our responsibility to make sure that we break through the writeback cache on the disk.
The users doesn't know or have to care about the caching. The users uses O_SYNC/fsync to tell it wants data on disk, and it's the
operating systems job to make that happen. The situation with qemu
is the same - if we tell the guest that we do not have a volatile write cache that needs explicit management the guest can rely on the fact that it does not have to do manual cache management.
Quoted 8 lines · Christoph Hellwig
The users doesn't know or have to care about the caching. The users uses O_SYNC/fsync to tell it wants data on disk, and it's the
operating systems job to make that happen. The situation with qemu
is the same - if we tell the guest that we do not have a volatile write cache that needs explicit management the guest can rely on the fact that it does not have to do manual cache management.
In the general case this is correct, however sometimes we want to explicitly lie (cache=unsafe, or say that we have a write-back cache when we don't to preserve the guest's view of things after a migration).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Christoph Hellwig
Quoted 4 lines · Anthony Liguori
The type of cache we present to the guest only should relate to how the hypervisor caches the storage. It should be independent of how data is cached by the disk.
It is.
Quoted 7 lines
There can be many levels of caching in a storage hierarchy and each hierarchy cached independently of the next level.
If the user has a disk with a writeback cache, if we expose a writethrough cache to the guest, it's not our responsibility to make sure that we break through the writeback cache on the disk.
Quoted 7 lines
The users doesn't know or have to care about the caching. The users uses O_SYNC/fsync to tell it wants data on disk, and it's the
operating systems job to make that happen. The situation with qemu
is the same - if we tell the guest that we do not have a volatile write cache that needs explicit management the guest can rely on the fact that it does not have to do manual cache management.
This is simply unrealistic. O_SYNC might force data to be on a platter when using a directly attached disk but many NAS's actually do writeback caching and relying on having an UPS to preserve data integrity. There's really no way in the general case to ensure that data is actually on a platter once you've involved a complex storage setup or you assume FUA
Let me put it another way. If an admin knows the disks on a machine have battery backed cache, he's likely to leave writeback caching enabled.
We are currently giving the admin two choices with QEMU, either ignore the fact that the disk is battery backed and do write through caching of the disk or do writeback caching in the host which expands the disk cache from something very small and non-volatile (the on-disk cache) to something very large and volatile (the page cache). To make the page cache non-volatile, you would need to have an UPS for the hypervisor with enough power to flush the page cache.
So basically, we're not presenting a model that makes sensible use of reliable disks.
cache=none does the right thing here but doesn't benefit from the host's page cache for reads. This is really the missing behavior.
Regards,
Anthony Liguori
Quoted 2 lines · Anthony Liguori
This is simply unrealistic.
O_SYNCmight force data to be on a platter when using a directly attached disk but many NAS's actually do writeback caching and relying on having an UPS to preserve data integrity. There's really no way in the general case to ensure that data is actually on a platter once you've involved a complex storage setup or you assume FUA
That's fine. Memory backed up by a UPS is a disk platter as far as the user is concerned, if the NAS is reliable.
Quoted 2 lines
Let me put it another way. If an admin knows the disks on a machine have battery backed cache, he's likely to leave writeback caching enabled.
In this case, as far as the host is concerned, there is no cache. Data written is guaranteed to reach the disk eventually even without a flush. Hopefully the disk advertises itself as not having a volatile cache.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Anthony Liguori
This is simply unrealistic.
O_SYNCmight force data to be on a platter when using a directly attached disk but many NAS's actually do writeback caching and relying on having an UPS to preserve data integrity. There's really no way in the general case to ensure that data is actually on a platter once you've involved a complex storage setup or you assume FUA
Yes, there is. If you have an array that has batter backup it handles this internally. The normal case is to not set the WCE bit in the mode page, which tells the operating system not ever send SYNCHRONIZE_CACHE commands. I have one array that sets a WCE bit neveless, but it also doesn't flush it's non-volatile cache in SYNCHRONIZE_CACHE, but rather implements it as an effective no-op.
Quoted 7 lines
Let me put it another way. If an admin knows the disks on a machine have battery backed cache, he's likely to leave writeback caching enabled.
We are currently giving the admin two choices with QEMU, either ignore the fact that the disk is battery backed and do write through caching of the disk or do writeback caching in the host which
Again, this is not qemu's business at all. Qemu is not different from any other application requiring data integrity. If that admin really thinks he needs to overide the storage provided settings he can mount the filesystem using -o nobarrier and we will not send cache flushes. I would in general recommend against this, as an external UPS still has lots of failure modes that this doesn't account for. Arrays with internal non-volatile memory already do the right thing anyway.
Quoted 1 line · Christoph Hellwig
On Tue, Aug 17, 2010 at 09:20:37AM -0500, Anthony Liguori wrote:
[]
Quoted 4 lines
Quoted 3 lines
For normal writes from a guest, we don't need to follow the write with an
fsync(). We should only need to issue anfsync()given an explicit flush from the guest.
Quoted 8 lines
Define normal writes. For cache=none and cache=writeback we don't have to, and instead do explicit calls to fsync()/fdatasync() calls when a we a cache flush from the guest. For data=writethrough we guarantee data has made it to disk, and we implement this using O_DSYNC/O_SYNC when opening the file. That tells the operating system
to not return until data has hit the disk. For Linux this is
internally implement using a range-fsync/fdatasync after the actual write.
And this is actually what I mentioned in the very beginning, in a hopefully-single-thread-email I've sent. Mentioned that ext4 is very slow when using with O_SYNC (without O_DIRECT).
I still had no opportunity to collect more info on this, and yes, I've seen your (Christoph's) speed tests of a few FSes in the famous "BTRFS: Unbelievably slow with kvm/qemu" thread. A few users reported _insane_ write speeds of qcow2 files with default cache mode on ext4.
And this is what prompted all this discussion (which actually has nothing to do with the $subject line ;), -- an attempt to think about replacing O_SYNC/fsync() with something "lighter"...
Quoted 3 lines
Quoted 3 lines
fsync()being slow is orthogonal to my point. I don't see why we need to do anfsync()on *every* write. It should only be necessary when a guest injects an actual barrier.
We don't do sync on every write, but O_SYNC implies that. And apparently it is what happening behind the scenes in ext4 O_SYNC case.
But ok....
/mjt
Quoted 9 lines · Michael Tokarev
Quoted 5 lines
Quoted 5 lines
fsync()being slow is orthogonal to my point. I don't see why we need to do anfsync()on *every* write. It should only be necessary when a guest injects an actual barrier.We don't do sync on every write, but
O_SYNCimplies that. And apparently it is what happening behind the scenes in ext4O_SYNCcase.
I think the real issue is we're mixing host configuration with guest visible state.
With O_SYNC, we're causing cache=writethrough to do writethrough through two layers of the storage heirarchy. I don't think that's necessary or desirable though.
Regards,
Anthony Liguori
Quoted 4 lines
But ok....
/mjt
Quoted 2 lines · Anthony Liguori
I think the real issue is we're mixing host configuration with guest visible state.
The last time I proposed to decouple the two you and Avi were heavily opposed to it..
Quoted 3 lines
With
O_SYNC, we're causing cache=writethrough to do writethrough through two layers of the storage heirarchy. I don't think that's necessary or desirable though.
It's absolutely nessecary if we tell the guest that we do not have a volatile write cache. Which is the only good reason to use data=writethrough anyway - except for dealing with old guests that can't handle volatile writecache it's an absolutely stupid mode of operation.
Quoted 13 lines · Christoph Hellwig
Quoted 3 lines · Anthony Liguori
I think the real issue is we're mixing host configuration with guest visible state.
The last time I proposed to decouple the two you and Avi were heavily opposed to it..
Quoted 4 lines
With
O_SYNC, we're causing cache=writethrough to do writethrough through two layers of the storage heirarchy. I don't think that's necessary or desirable though.
Quoted 6 lines
It's absolutely nessecary if we tell the guest that we do not have a volatile write cache. Which is the only good reason to use data=writethrough anyway - except for dealing with old guests that can't handle volatile writecache it's an absolutely stupid mode of operation.
You can lose an awful lot of data with cache=writeback because the host page cache is volatile. In a perfect world, this would only be non-critical data because everyone would be using fsync() properly but 1) even non-critical data is important when there's a lot of it 2) we don't live in a perfect world. The fact of the matter is, there is a huge amount of crappy filesystems and applications today that don't submit barriers appropriately.
We make the situation much worse with virtualization because of the shear size of the cache we introduce.
Regards,
Anthony Liguori
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines · Anthony Liguori
I think the real issue is we're mixing host configuration with guest visible state.
The last time I proposed to decouple the two you and Avi were heavily opposed to it..
I wasn't that I can recall.
Quoted 8 lines
Quoted 3 lines
With
O_SYNC, we're causing cache=writethrough to do writethrough through two layers of the storage heirarchy. I don't think that's necessary or desirable though.It's absolutely nessecary if we tell the guest that we do not have a volatile write cache. Which is the only good reason to use data=writethrough anyway - except for dealing with old guests that can't handle volatile writecache it's an absolutely stupid mode of operation.
I agree, but there's another case: tell the guest that we have a write cache, use O_DSYNC, but only flush the disk cache on guest flushes.
The reason for this is that if we don't use O_DSYNC the page cache can grow to huge proportions. While this is allowed by the contract between virtual drive and guest, guest software and users won't expect a huge data loss on power fail, only a minor data loss from the last fraction of a second before the failure.
I believe this can be approximated by mounting the host filesystem with barrier=0?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
I agree, but there's another case: tell the guest that we have a write cache, use
O_DSYNC, but only flush the disk cache on guest flushes.
O_DSYNC flushes the disk write cache and any filesystem that supports
non-volatile cache. The disk cache is not an abstraction
exposed to applications.
Quoted 2 lines
I believe this can be approximated by mounting the host filesystem with barrier=0?
Mounting the host filesystem with nobarrier means we will never explicit flush the volatile write cache on the disk.